[AUTO-INF-5] Harden CI quality gates (coverage parity, docs build, nightly enforcement) #9697

Open
opened 2026-04-15 03:17:20 +00:00 by HAL9000 · 0 comments
Owner

Metadata

  • Commit message: chore(ci): harden quality gates — coverage parity, docs job, nightly enforcement
  • Branch name: chore/harden-ci-quality-gates-inf-5

Background and Context

The coverage job in .forgejo/workflows/ci.yml hard-codes threshold = 50 in the "Surface coverage summary" step even though noxfile.py keeps COVERAGE_THRESHOLD = 97. This duplication caused the regression tracked in #9597 and will resurface unless the workflow consumes the value emitted by the nox session.

CI never runs the docs build despite having a dedicated nox -s docs session (noxfile.py#L93-L118), so the missing CHANGELOG.md (docs link rot) slips through. .forgejo/workflows/ci.yml only defines lint/typecheck/security/quality/tests/build jobs and skips docs entirely.

The nightly quality workflow still wraps the quality gate script with || echo ... skipping (.forgejo/workflows/nightly-quality.yml#L41-L72), so coverage drops or script failures do not fail the nightly run.

Duplicate Check

  • Open issues: inspected #9597 (restores the literal threshold) and #9683 (parallelism fix); neither addresses sharing the threshold, adding docs gating, or fixing nightly quality exits.
  • Closed issues: reviewed #9605 (watchdog alert about threshold drift) to confirm the regression history — this proposal closes the loop.
  • No existing issue covers these CI design fixes; coverage threshold restoration alone is already tracked in #9597.

Expected Behavior

  1. The CI coverage job derives its threshold from the value emitted by nox -s coverage_report (e.g., parsed from build/coverage.json or exported via COVERAGE_THRESHOLD env var) so that a single source of truth governs enforcement and future drift like the threshold = 50 stanza is impossible.
  2. A docs CI job runs nox -s docs, asserts CHANGELOG.md exists in the built output, and uploads the built site/logs — broken links (e.g., missing changelog) are caught pre-merge.
  3. The nightly quality "Run quality gates script" step fails the workflow when the script is missing or exits non-zero, instead of swallowing the exit code with || echo ... skipping.

Acceptance Criteria

  • .forgejo/workflows/ci.yml coverage job reads the threshold from build/coverage.json (or equivalent nox-emitted artifact) rather than a hard-coded literal; CI fails when coverage is below the nox-defined threshold.
  • .forgejo/workflows/ci.yml includes a docs job that runs nox -s docs, asserts CHANGELOG.md is present in the built output, and uploads the docs artifact/logs.
  • .forgejo/workflows/nightly-quality.yml "Run quality gates script" step no longer uses || echo ... skipping; a missing script or non-zero exit fails the nightly workflow.
  • No regression in existing CI jobs (lint, typecheck, security, tests, build).
  • The @tdd_expected_fail scenario in features/ci_workflow_validation.feature (if applicable) is updated to reflect the new threshold-sourcing mechanism.

Subtasks

  • Audit nox -s coverage_report output to identify the artifact or env var that carries COVERAGE_THRESHOLD = 97 (e.g., build/coverage.json, coverage.xml, or a printed env export).
  • Refactor the "Surface coverage summary" step in .forgejo/workflows/ci.yml to parse the threshold from the nox-emitted artifact instead of the hard-coded threshold = 50.
  • Add a docs job to .forgejo/workflows/ci.yml that runs nox -s docs, asserts CHANGELOG.md exists, and uploads the built site as a workflow artifact.
  • Update .forgejo/workflows/nightly-quality.yml to remove the || echo ... skipping guard from the quality gate script step; ensure the step fails on non-zero exit.
  • Consider extracting a shared coverage-summary parser (composite action or script) reused by both the main CI coverage job and the nightly quality job.
  • Run the updated workflows in a feature branch to confirm all jobs pass and the docs assertion catches a missing changelog.
  • Open a PR and get it merged.

Definition of Done

This issue should be closed when:

  1. The CI coverage threshold is sourced from the nox session artifact (not a hard-coded literal) and enforces 97% without manual edits.
  2. A docs CI job runs on every PR and fails when CHANGELOG.md is absent from the built output.
  3. The nightly quality workflow fails the run when the quality gate script is missing or exits non-zero.
  4. All changes are merged to the main branch with CI green.

Automated by CleverAgents Bot
Agent: new-issue-creator

## Metadata - **Commit message:** `chore(ci): harden quality gates — coverage parity, docs job, nightly enforcement` - **Branch name:** `chore/harden-ci-quality-gates-inf-5` ## Background and Context The coverage job in [.forgejo/workflows/ci.yml](https://git.cleverthis.com/cleveragents/cleveragents-core/src/branch/master/.forgejo/workflows/ci.yml#L190-L238) hard-codes `threshold = 50` in the "Surface coverage summary" step even though [noxfile.py](https://git.cleverthis.com/cleveragents/cleveragents-core/src/branch/master/noxfile.py#L210-L308) keeps `COVERAGE_THRESHOLD = 97`. This duplication caused the regression tracked in #9597 and will resurface unless the workflow consumes the value emitted by the nox session. CI never runs the docs build despite having a dedicated `nox -s docs` session ([noxfile.py#L93-L118](https://git.cleverthis.com/cleveragents/cleveragents-core/src/branch/master/noxfile.py#L93-L118)), so the missing `CHANGELOG.md` (docs link rot) slips through. `.forgejo/workflows/ci.yml` only defines lint/typecheck/security/quality/tests/build jobs and skips docs entirely. The nightly quality workflow still wraps the quality gate script with `|| echo ... skipping` ([.forgejo/workflows/nightly-quality.yml#L41-L72](https://git.cleverthis.com/cleveragents/cleveragents-core/src/branch/master/.forgejo/workflows/nightly-quality.yml#L41-L72)), so coverage drops or script failures do not fail the nightly run. ### Duplicate Check - Open issues: inspected #9597 (restores the literal threshold) and #9683 (parallelism fix); neither addresses sharing the threshold, adding docs gating, or fixing nightly quality exits. - Closed issues: reviewed #9605 (watchdog alert about threshold drift) to confirm the regression history — this proposal closes the loop. - No existing issue covers these CI design fixes; coverage threshold restoration alone is already tracked in #9597. ## Expected Behavior 1. The CI coverage job derives its threshold from the value emitted by `nox -s coverage_report` (e.g., parsed from `build/coverage.json` or exported via `COVERAGE_THRESHOLD` env var) so that a single source of truth governs enforcement and future drift like the `threshold = 50` stanza is impossible. 2. A `docs` CI job runs `nox -s docs`, asserts `CHANGELOG.md` exists in the built output, and uploads the built site/logs — broken links (e.g., missing changelog) are caught pre-merge. 3. The nightly quality "Run quality gates script" step fails the workflow when the script is missing or exits non-zero, instead of swallowing the exit code with `|| echo ... skipping`. ## Acceptance Criteria - [ ] `.forgejo/workflows/ci.yml` coverage job reads the threshold from `build/coverage.json` (or equivalent nox-emitted artifact) rather than a hard-coded literal; CI fails when coverage is below the nox-defined threshold. - [ ] `.forgejo/workflows/ci.yml` includes a `docs` job that runs `nox -s docs`, asserts `CHANGELOG.md` is present in the built output, and uploads the docs artifact/logs. - [ ] `.forgejo/workflows/nightly-quality.yml` "Run quality gates script" step no longer uses `|| echo ... skipping`; a missing script or non-zero exit fails the nightly workflow. - [ ] No regression in existing CI jobs (lint, typecheck, security, tests, build). - [ ] The `@tdd_expected_fail` scenario in `features/ci_workflow_validation.feature` (if applicable) is updated to reflect the new threshold-sourcing mechanism. ## Subtasks - [ ] Audit `nox -s coverage_report` output to identify the artifact or env var that carries `COVERAGE_THRESHOLD = 97` (e.g., `build/coverage.json`, `coverage.xml`, or a printed env export). - [ ] Refactor the "Surface coverage summary" step in `.forgejo/workflows/ci.yml` to parse the threshold from the nox-emitted artifact instead of the hard-coded `threshold = 50`. - [ ] Add a `docs` job to `.forgejo/workflows/ci.yml` that runs `nox -s docs`, asserts `CHANGELOG.md` exists, and uploads the built site as a workflow artifact. - [ ] Update `.forgejo/workflows/nightly-quality.yml` to remove the `|| echo ... skipping` guard from the quality gate script step; ensure the step fails on non-zero exit. - [ ] Consider extracting a shared coverage-summary parser (composite action or script) reused by both the main CI coverage job and the nightly quality job. - [ ] Run the updated workflows in a feature branch to confirm all jobs pass and the docs assertion catches a missing changelog. - [ ] Open a PR and get it merged. ## Definition of Done This issue should be closed when: 1. The CI coverage threshold is sourced from the nox session artifact (not a hard-coded literal) and enforces 97% without manual edits. 2. A docs CI job runs on every PR and fails when `CHANGELOG.md` is absent from the built output. 3. The nightly quality workflow fails the run when the quality gate script is missing or exits non-zero. 4. All changes are merged to the main branch with CI green. --- **Automated by CleverAgents Bot** Agent: new-issue-creator
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleveragents/cleveragents-core#9697
No description provided.