[AUTO-INF-7] Improve accuracy of coverage reporting #8318

New issue

Closed

opened 2026-04-13 08:30:41 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-13 08:30:41 +00:00

Owner

Summary

Slipcover-based coverage session only tracks src/ while the project coverage settings expect both src/ and scripts/, so results under-report exercised CLI helpers.
The CI coverage summary step still uses a temporary 50% gate and limited parsing, causing misleading "OK" messages when the main nox session fails at the 97% threshold.
Coverage configuration lives in pyproject.toml, but the nox session duplicates values; the two sources have already drifted (missing scripts/), risking future omissions.

Findings

Slipcover session excludes the `scripts` package

noxfile.py::coverage_report hard-codes source_paths = "src" and builds its own omit list.
pyproject.toml defines [tool.coverage.run] source = ["src", "scripts"], so coverage.py-based tooling counts the CLI scripts; slipcover currently ignores them.
Behave workflows install the custom runner from scripts/run_behave_parallel.py, so excluding scripts/ suppresses legitimately executed lines and lowers reported totals.

CI summary still gates at 50%

.forgejo/workflows/ci.yml step "Surface coverage summary" sets threshold = 50, a leftover temporary override.
The step runs with if: always(), so even when nox -s coverage_report fails at <97%, the follow-up script still prints COVERAGE OK at ≥50%, sending mixed signals to reviewers.
The parser also only reads percent_covered, whereas the nox session already handles alternate keys (percent_covered_display, line_coverage_percent).

Coverage configuration duplication

The omit patterns in coverage_report are copy/pasted instead of sourced from pyproject.toml. Any future edits risk silent drift; we already have one divergence (scripts/ missing from sources).
There is no .coveragerc; the canonical settings are in pyproject.toml, but the slipcover path bypasses them entirely.

Recommendations

Update coverage_report to include the scripts directory when invoking slipcover (e.g. --source src --source scripts) and, ideally, load omit patterns from the shared coverage config to keep parity.
Refactor the coverage job summary script to consume the same COVERAGE_THRESHOLD (97) and reuse the tolerant parsing logic from coverage_report so the CI message matches the gate result.
Consider extracting coverage include/omit settings to a single module (or loading from pyproject.toml) so both slipcover and any coverage.py tooling stay aligned.

Duplicate Check

Searched open issues for [AUTO-INF-7] via GET /api/v1/repos/cleveragents/cleveragents-core/issues?state=open (response: []).

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-worker

## Summary - Slipcover-based coverage session only tracks `src/` while the project coverage settings expect both `src/` and `scripts/`, so results under-report exercised CLI helpers. - The CI coverage summary step still uses a temporary 50% gate and limited parsing, causing misleading "OK" messages when the main nox session fails at the 97% threshold. - Coverage configuration lives in `pyproject.toml`, but the nox session duplicates values; the two sources have already drifted (missing `scripts/`), risking future omissions. ## Findings ### Slipcover session excludes the `scripts` package - `noxfile.py::coverage_report` hard-codes `source_paths = "src"` and builds its own omit list. - `pyproject.toml` defines `[tool.coverage.run] source = ["src", "scripts"]`, so coverage.py-based tooling counts the CLI scripts; slipcover currently ignores them. - Behave workflows install the custom runner from `scripts/run_behave_parallel.py`, so excluding `scripts/` suppresses legitimately executed lines and lowers reported totals. ### CI summary still gates at 50% - `.forgejo/workflows/ci.yml` step "Surface coverage summary" sets `threshold = 50`, a leftover temporary override. - The step runs with `if: always()`, so even when `nox -s coverage_report` fails at `<97%`, the follow-up script still prints `COVERAGE OK` at ≥50%, sending mixed signals to reviewers. - The parser also only reads `percent_covered`, whereas the nox session already handles alternate keys (`percent_covered_display`, `line_coverage_percent`). ### Coverage configuration duplication - The omit patterns in `coverage_report` are copy/pasted instead of sourced from `pyproject.toml`. Any future edits risk silent drift; we already have one divergence (`scripts/` missing from sources). - There is no `.coveragerc`; the canonical settings are in `pyproject.toml`, but the slipcover path bypasses them entirely. ## Recommendations 1. Update `coverage_report` to include the `scripts` directory when invoking slipcover (e.g. `--source src --source scripts`) and, ideally, load omit patterns from the shared coverage config to keep parity. 2. Refactor the coverage job summary script to consume the same `COVERAGE_THRESHOLD` (97) and reuse the tolerant parsing logic from `coverage_report` so the CI message matches the gate result. 3. Consider extracting coverage include/omit settings to a single module (or loading from `pyproject.toml`) so both slipcover and any coverage.py tooling stay aligned. ### Duplicate Check - Searched open issues for `[AUTO-INF-7]` via `GET /api/v1/repos/cleveragents/cleveragents-core/issues?state=open` (response: `[]`). --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-worker

Rows
Columns