cleveragents/cleveragents-core

Fork 3

feat(ci): add nox-based PR validation workflow #213

New issue

Closed

opened 2026-02-22 23:40:10 +00:00 by freemo · 1 comment

freemo commented

2026-02-22 23:40:10 +00:00

Owner

Metadata

Commit: feat(ci): add nox-based PR validation workflow
Branch: feature/q0-min-ci

Subtasks

Update .forgejo/workflows/ci.yml to install dependencies via Hatch and run nox (unit + integration + typecheck + lint + coverage_report); do not add GitHub workflows.
Ensure CI uses Python 3.13, caches pip/Hatch artifacts, and uploads nox logs on failure.
Fail pipeline if any nox session fails or coverage <97% (explicit coverage gate).
Add CI usage notes in docs/development/ci-cd.md, including local repro commands and cache notes.
Tests (Behave): Add a scenario that validates the workflow file exists and references required nox sessions. (features/ci_workflow_validation.feature, 11 scenarios)
Tests (Robot): Add a Robot smoke test that runs the same nox session matrix locally and asserts zero failures. (robot/ci_nox_validation.robot)
Tests (ASV): Add benchmarks/ci_yaml_parse_bench.py to benchmark workflow parsing and key lookup. (benchmarks/ci_yaml_parse_bench.py)
Run nox (all default sessions, including benchmark). (1673 scenarios passed, 0 failed)
Verify coverage >=97% via nox -s coverage_report. If coverage is <97% then review the current unit test coverage report at build/coverage.xml and use it to write new Behave based unit tests to improve code coverage. Specifically, write Behave style unit tests that are descriptively named and specifically improves coverage on whichever file has the most uncovered lines by writing tests that will target the uncovered lines in the report. Once that is done rerun nox -s coverage_report to verify all tests pass and coverage is above >=97%. Only mark this as complete once coverage is >=97%, if not repeat this task as many times as is needed until coverage reaches >=97%. (97% total, fail-under=97 passes)

Section: ### Section 0: Quality Automation Setup [WORKSTREAM Q - Brent Lead]
Status: Completed

## Metadata - **Commit**: `feat(ci): add nox-based PR validation workflow` - **Branch**: `feature/q0-min-ci` ## Subtasks - [x] Update `.forgejo/workflows/ci.yml` to install dependencies via Hatch and run `nox` (unit + integration + typecheck + lint + coverage_report); do not add GitHub workflows. - [x] Ensure CI uses Python 3.13, caches pip/Hatch artifacts, and uploads `nox` logs on failure. - [x] Fail pipeline if any `nox` session fails or coverage <97% (explicit coverage gate). - [x] Add CI usage notes in `docs/development/ci-cd.md`, including local repro commands and cache notes. - [x] Tests (Behave): Add a scenario that validates the workflow file exists and references required `nox` sessions. (features/ci_workflow_validation.feature, 11 scenarios) - [x] Tests (Robot): Add a Robot smoke test that runs the same `nox` session matrix locally and asserts zero failures. (robot/ci_nox_validation.robot) - [x] Tests (ASV): Add `benchmarks/ci_yaml_parse_bench.py` to benchmark workflow parsing and key lookup. (benchmarks/ci_yaml_parse_bench.py) - [x] Run `nox` (all default sessions, including benchmark). (1673 scenarios passed, 0 failed) - [x] Verify coverage >=97% via `nox -s coverage_report`. If coverage is <97% then review the current unit test coverage report at `build/coverage.xml` and use it to write new Behave based unit tests to improve code coverage. Specifically, write Behave style unit tests that are descriptively named and specifically improves coverage on whichever file has the most uncovered lines by writing tests that will target the uncovered lines in the report. Once that is done rerun `nox -s coverage_report` to verify all tests pass and coverage is above >=97%. Only mark this as complete once coverage is >=97%, if not repeat this task as many times as is needed until coverage reaches >=97%. (97% total, fail-under=97 passes) **Section**: ### Section 0: Quality Automation Setup [WORKSTREAM Q - Brent Lead] **Status**: Completed

freemo added this to the (deleted) milestone

2026-02-22 23:40:10 +00:00

freemo added the

labels

2026-02-22 23:40:10 +00:00

brent.edwards was assigned by freemo

2026-02-22 23:40:10 +00:00

freemo closed this issue

2026-02-22 23:40:10 +00:00

freemo added a new dependency

2026-02-22 23:54:07 +00:00

#349 Epic: Quality Automation Setup (Q0)

freemo modified the milestone from (deleted) to v3.0.0

2026-02-23 00:07:10 +00:00

freemo added a new dependency

2026-02-23 01:11:10 +00:00

#45 feature/q0-min-ci

freemo referenced this issue from a pull request that will close it,

2026-02-23 17:25:53 +00:00

feature/q0-min-ci #45

freemo commented

2026-02-24 02:29:14 +00:00

Author

Owner

Implementation Notes — CI/CD Pipeline Stabilization

2026-02-12: Task Q0-min-ci In Progress - Nox-Based PR Validation Workflow

Rewrote .forgejo/workflows/ci.yml to route ALL jobs through nox sessions instead of direct tool invocations.
Coverage boost 92%->97%: Wrote 108 new Behave scenarios covering 6 largest coverage gaps.
Key discovery: Behave loads ALL step definition files globally; any @given/@when/@then pattern matching an existing one causes AmbiguousStep errors.

2026-02-12: Bugfix - Integration Test Failure in Robot.Actor Configuration

Root cause: robot/ci_nox_validation.robot ran nox --list during integration tests. When pabot executed all Robot suites in parallel, the nox --list stdout leaked into the sibling robot/actor_configuration.robot process, causing JSONDecodeError.
Fix: Added slow tag to interfering test cases; replaced hardcoded venv paths with python.

2026-02-12: Bugfix - CI security Job Failure (session name mismatch)

Root cause: CI referenced nox -s security_scan and nox -s dead_code, but the noxfile only had a session named security.
Fix: Renamed security to security_scan, added standalone dead_code session.

2026-02-12: Bugfix - CI unit_tests Failure: Rich Console Line-Wrapping

Root cause: Rich's Console wraps output at 80 columns. On CI, temp directory paths exceed 80 chars, splitting filenames across lines.
Fix: Collapse Rich line-wraps via clean_output.replace("\n", "") before assertions.

2026-02-12: Bugfix - CI integration_tests Failures (3 categories)

Issue 1: ModuleNotFoundError — hardcoded /app path doesn't exist on CI. Fixed with ${CURDIR}/.. relative paths.
Issue 2: --load-context help text assertion failures — Rich output goes to stderr when no TTY. Fixed with stderr=STDOUT merge.
Issue 3: FileNotFoundError: robot — pabot subprocess spawning exhausts container resources. Restored --exclude discovery flag.

2026-02-13: Bugfix - CI security Job Failure (missing build/ directory)

Root cause: build/ directory does not exist in fresh CI checkout. Bandit cannot create intermediate directories.
Fix: Added os.makedirs("build", exist_ok=True) before bandit invocation.

2026-02-13: Bugfix - CI integration_tests Failures (3 categories)

Issue 1: FileNotFoundError: robot after ~17 suites — container resource exhaustion. Capped pabot parallelism to min(default, 2), explicitly prepend venv bin/ to PATH.
Issue 2: Rich ANSI escape codes split --load-context string. Added env:NO_COLOR=1.
Issue 3: initial_next_command_test.robot exit code 3 — missing OPENAI_API_KEY. Tagged as slow.

2026-02-13: Bugfix - CI integration_tests Failures Round 2 (pabot resource exhaustion + ANSI codes)

robot is findable at session start yet pabot still fails after ~24 suites even with --processes 2.
Root cause confirmed: FileNotFoundError from subprocess.Popen on Linux occurs when the system cannot execve() due to exhausted PIDs or file descriptors, not because the binary is missing.
Fix: Replaced pabot with robot (sequential execution). CI values reliability over speed.
NO_COLOR=1 insufficient — Rich still embeds non-color ANSI sequences (bold, reset). Added env:TERM=dumb.

2026-02-13: Bugfix - CI integration_tests Failures Round 3 (venv PATH, Resource paths, hanging tests)

Issue 1: python -m cleveragents uses system Python — venv PATH line was accidentally removed. Restored.
Issue 2: Robot Resource file resolution fails on CI — bare relative references. Converted all to ${CURDIR}/ absolute paths across 30 robot files.
Issue 3: rxpy_route_validation.robot tests hang — missing timeout parameters on Run Process calls. Added timeout=30s to all 9 calls.

2026-02-13: Bugfix - CI integration_tests Failures Round 4 (duplicate Settings blocks, venv Python injection, CI debug)

Issue 1: Duplicate *** Settings *** blocks in 14 robot files created by sed-based conversion. Merged into single blocks.
Issue 2: Bare python in Run Process calls resolves to system Python on CI. Noxfile now passes --variable PYTHON:<venv-python-path> to robot. All 14 files updated to use ${PYTHON}.
Issue 3: Hardcoded /app/src path in system_prompt_template_rendering.robot. Removed (venv Python already has package installed).

2026-02-13: Bugfix - CI integration_tests Failures Round 5 (guard cleanup when OpenAI key missing)

Root cause: robot/scientific_paper_basic.robot suite setup skips on missing OPENAI_API_KEY, leaving ${CONTEXT_DIR} empty. Teardown's Remove Directory ${CONTEXT_DIR} recursive=True resolves to CWD, deleting the repo root (including .nox venv and robot/*.resource files), causing all downstream suites to fail.
Fix: Initialize ${CONTEXT_DIR} to ${TEMPDIR}/paper_basic_contexts and guard cleanup with empty check.

2026-02-13: Enhancement - Restore parallel Robot runs + silence discovery resource warning

Reintroduced pabot with conservative parallelism (max 2 processes) now that teardown bug is fixed.
Added robot/discovery_common.resource stub to silence non-fatal import warning.

(Migrated from docs/implementation-notes.md)

## Implementation Notes — CI/CD Pipeline Stabilization **2026-02-12**: Task Q0-min-ci In Progress - Nox-Based PR Validation Workflow - Rewrote `.forgejo/workflows/ci.yml` to route ALL jobs through nox sessions instead of direct tool invocations. - **Coverage boost 92%->97%**: Wrote 108 new Behave scenarios covering 6 largest coverage gaps. - **Key discovery**: Behave loads ALL step definition files globally; any `@given/@when/@then` pattern matching an existing one causes `AmbiguousStep` errors. **2026-02-12**: Bugfix - Integration Test Failure in Robot.Actor Configuration - **Root cause**: `robot/ci_nox_validation.robot` ran `nox --list` during integration tests. When `pabot` executed all Robot suites in parallel, the `nox --list` stdout leaked into the sibling `robot/actor_configuration.robot` process, causing `JSONDecodeError`. - **Fix**: Added `slow` tag to interfering test cases; replaced hardcoded venv paths with `python`. **2026-02-12**: Bugfix - CI `security` Job Failure (session name mismatch) - **Root cause**: CI referenced `nox -s security_scan` and `nox -s dead_code`, but the noxfile only had a session named `security`. - **Fix**: Renamed `security` to `security_scan`, added standalone `dead_code` session. **2026-02-12**: Bugfix - CI `unit_tests` Failure: Rich Console Line-Wrapping - **Root cause**: Rich's `Console` wraps output at 80 columns. On CI, temp directory paths exceed 80 chars, splitting filenames across lines. - **Fix**: Collapse Rich line-wraps via `clean_output.replace("\n", "")` before assertions. **2026-02-12**: Bugfix - CI `integration_tests` Failures (3 categories) - **Issue 1**: `ModuleNotFoundError` — hardcoded `/app` path doesn't exist on CI. Fixed with `${CURDIR}/..` relative paths. - **Issue 2**: `--load-context` help text assertion failures — Rich output goes to stderr when no TTY. Fixed with `stderr=STDOUT` merge. - **Issue 3**: `FileNotFoundError: robot` — pabot subprocess spawning exhausts container resources. Restored `--exclude discovery` flag. **2026-02-13**: Bugfix - CI `security` Job Failure (missing `build/` directory) - **Root cause**: `build/` directory does not exist in fresh CI checkout. Bandit cannot create intermediate directories. - **Fix**: Added `os.makedirs("build", exist_ok=True)` before bandit invocation. **2026-02-13**: Bugfix - CI `integration_tests` Failures (3 categories) - **Issue 1**: `FileNotFoundError: robot` after ~17 suites — container resource exhaustion. Capped pabot parallelism to `min(default, 2)`, explicitly prepend venv `bin/` to PATH. - **Issue 2**: Rich ANSI escape codes split `--load-context` string. Added `env:NO_COLOR=1`. - **Issue 3**: `initial_next_command_test.robot` exit code 3 — missing `OPENAI_API_KEY`. Tagged as `slow`. **2026-02-13**: Bugfix - CI `integration_tests` Failures Round 2 (pabot resource exhaustion + ANSI codes) - `robot` is findable at session start yet pabot still fails after ~24 suites even with `--processes 2`. - **Root cause confirmed**: `FileNotFoundError` from `subprocess.Popen` on Linux occurs when the system cannot `execve()` due to exhausted PIDs or file descriptors, not because the binary is missing. - **Fix**: Replaced `pabot` with `robot` (sequential execution). CI values reliability over speed. - `NO_COLOR=1` insufficient — Rich still embeds non-color ANSI sequences (bold, reset). Added `env:TERM=dumb`. **2026-02-13**: Bugfix - CI `integration_tests` Failures Round 3 (venv PATH, Resource paths, hanging tests) - **Issue 1**: `python -m cleveragents` uses system Python — venv PATH line was accidentally removed. Restored. - **Issue 2**: Robot `Resource` file resolution fails on CI — bare relative references. Converted all to `${CURDIR}/` absolute paths across 30 robot files. - **Issue 3**: `rxpy_route_validation.robot` tests hang — missing `timeout` parameters on `Run Process` calls. Added `timeout=30s` to all 9 calls. **2026-02-13**: Bugfix - CI `integration_tests` Failures Round 4 (duplicate Settings blocks, venv Python injection, CI debug) - **Issue 1**: Duplicate `*** Settings ***` blocks in 14 robot files created by sed-based conversion. Merged into single blocks. - **Issue 2**: Bare `python` in `Run Process` calls resolves to system Python on CI. Noxfile now passes `--variable PYTHON:<venv-python-path>` to `robot`. All 14 files updated to use `${PYTHON}`. - **Issue 3**: Hardcoded `/app/src` path in `system_prompt_template_rendering.robot`. Removed (venv Python already has package installed). **2026-02-13**: Bugfix - CI `integration_tests` Failures Round 5 (guard cleanup when OpenAI key missing) - **Root cause**: `robot/scientific_paper_basic.robot` suite setup skips on missing `OPENAI_API_KEY`, leaving `${CONTEXT_DIR}` empty. Teardown's `Remove Directory ${CONTEXT_DIR} recursive=True` resolves to CWD, deleting the repo root (including `.nox` venv and `robot/*.resource` files), causing all downstream suites to fail. - **Fix**: Initialize `${CONTEXT_DIR}` to `${TEMPDIR}/paper_basic_contexts` and guard cleanup with empty check. **2026-02-13**: Enhancement - Restore parallel Robot runs + silence discovery resource warning - Reintroduced `pabot` with conservative parallelism (max 2 processes) now that teardown bug is fixed. - Added `robot/discovery_common.resource` stub to silence non-fatal import warning. *(Migrated from `docs/implementation-notes.md`)*

freemo removed a dependency

2026-03-03 22:54:41 +00:00

#45 feature/q0-min-ci

freemo added a new dependency

2026-03-03 22:54:41 +00:00

#45 feature/q0-min-ci

freemo referenced this issue

2026-04-03 09:42:37 +00:00

TEST-INFRA: [ci-pipeline-design] Implement Conditional Test Execution to Run Only Affected Tests #2232