cleveragents/cleveragents-core

Fork 3

UAT: PR #2629 — Nightly workflow quality gates script uses --coverage-min 85 (below required 97%) #2769

New issue

Closed

opened 2026-04-04 19:21:19 +00:00 by freemo · 4 comments

freemo commented

2026-04-04 19:21:19 +00:00

Owner

Bug Report

PR Under Test: #2629 (fix/master-ci-quality-gates, HEAD SHA 938ea8194c6a2044b17efabea1766995c62eeaf1)
Issue Being Fixed: #2597
Feature Area: coverage-threshold-config

What Was Tested

Code-level analysis of .forgejo/workflows/nightly-quality.yml against the project's coverage threshold requirement.

Expected Behavior

Per issue #2597 Acceptance Criteria #3:

"Reducing the coverage threshold below 97% in noxfile.py or CI configuration" is prohibited.

The nightly workflow should enforce coverage at 97% (matching the project's COVERAGE_THRESHOLD = 97 in noxfile.py).

Actual Behavior

In .forgejo/workflows/nightly-quality.yml, the "Run quality gates script" step uses:

- name: Run quality gates script
  run: |
    python scripts/check-quality-gates.py --coverage-min 85 --complexity-max F || echo "Quality gates script not found or failed, skipping..."

--coverage-min 85 is below the project's required 97% threshold.

Additionally, the step_nightly_fail_under function in coverage_threshold_config_steps.py checks for the maximum threshold found in the nightly workflow. If a feature scenario tested "at least 97", it would find --coverage-min 85 and fail because 85 < 97.

Analysis

The nightly workflow has two coverage checks:

nox -s coverage_report — hard enforcement at 97% (correct)
python scripts/check-quality-gates.py --coverage-min 85 — soft check at 85% (incorrect, below threshold)

The soft check uses || echo "..." making it non-blocking, but the threshold value itself (85) is inconsistent with the project's 97% requirement. This represents a weakened quality gate in the nightly workflow.

If the coverage_threshold_config.feature file were created (see related issue #2767) and included a scenario like:

Then the nightly workflow should use a fail-under of at least 97

...it would fail because the nightly workflow only has --coverage-min 85.

Code Location

File: .forgejo/workflows/nightly-quality.yml
Line: python scripts/check-quality-gates.py --coverage-min 85 --complexity-max F || echo "..."

Fix Required

Change --coverage-min 85 to --coverage-min 97 in the nightly workflow quality gates script invocation:

- name: Run quality gates script
  run: |
    python scripts/check-quality-gates.py --coverage-min 97 --complexity-max F || echo "Quality gates script not found or failed, skipping..."

This ensures the nightly workflow's quality gates script threshold is consistent with the project's 97% coverage requirement.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Bug Report **PR Under Test**: #2629 (`fix/master-ci-quality-gates`, HEAD SHA `938ea8194c6a2044b17efabea1766995c62eeaf1`) **Issue Being Fixed**: #2597 **Feature Area**: coverage-threshold-config --- ## What Was Tested Code-level analysis of `.forgejo/workflows/nightly-quality.yml` against the project's coverage threshold requirement. ## Expected Behavior Per issue #2597 Acceptance Criteria #3: > "Reducing the coverage threshold below 97% in `noxfile.py` or CI configuration" is **prohibited**. The nightly workflow should enforce coverage at 97% (matching the project's `COVERAGE_THRESHOLD = 97` in `noxfile.py`). ## Actual Behavior In `.forgejo/workflows/nightly-quality.yml`, the "Run quality gates script" step uses: ```yaml - name: Run quality gates script run: | python scripts/check-quality-gates.py --coverage-min 85 --complexity-max F || echo "Quality gates script not found or failed, skipping..." ``` **`--coverage-min 85`** is below the project's required **97%** threshold. Additionally, the `step_nightly_fail_under` function in `coverage_threshold_config_steps.py` checks for the maximum threshold found in the nightly workflow. If a feature scenario tested "at least 97", it would find `--coverage-min 85` and **fail** because `85 < 97`. ## Analysis The nightly workflow has two coverage checks: 1. `nox -s coverage_report` — hard enforcement at 97% (correct) 2. `python scripts/check-quality-gates.py --coverage-min 85` — soft check at 85% (incorrect, below threshold) The soft check uses `|| echo "..."` making it non-blocking, but the threshold value itself (85) is inconsistent with the project's 97% requirement. This represents a weakened quality gate in the nightly workflow. If the `coverage_threshold_config.feature` file were created (see related issue #2767) and included a scenario like: ```gherkin Then the nightly workflow should use a fail-under of at least 97 ``` ...it would **fail** because the nightly workflow only has `--coverage-min 85`. ## Code Location - File: `.forgejo/workflows/nightly-quality.yml` - Line: `python scripts/check-quality-gates.py --coverage-min 85 --complexity-max F || echo "..."` ## Fix Required Change `--coverage-min 85` to `--coverage-min 97` in the nightly workflow quality gates script invocation: ```yaml - name: Run quality gates script run: | python scripts/check-quality-gates.py --coverage-min 97 --complexity-max F || echo "Quality gates script not found or failed, skipping..." ``` This ensures the nightly workflow's quality gates script threshold is consistent with the project's 97% coverage requirement. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo referenced this issue

2026-04-04 19:29:18 +00:00

[CA-AUTO] Backlog Groomer — groomer-1 — Session Tracker #2771

freemo commented

2026-04-04 19:31:12 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified | MoSCoW: Should Have — Nightly workflow using --coverage-min 85 instead of the project-required 97% is a quality gate misconfiguration. Should be fixed to enforce the correct threshold.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified | **MoSCoW**: Should Have — Nightly workflow using `--coverage-min 85` instead of the project-required 97% is a quality gate misconfiguration. Should be fixed to enforce the correct threshold. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo added this to the v3.2.0 milestone

2026-04-04 19:31:20 +00:00

freemo referenced this issue

2026-04-04 19:31:54 +00:00

[CA-AUTO] UAT Pool Supervisor — PR #2629 Validation — Session Tracker #2752

freemo referenced this issue

2026-04-04 19:32:11 +00:00

fix(ci): restore all CI quality gates to passing on master #2629

freemo referenced this issue

2026-04-04 19:36:21 +00:00

[CA-AUTO] UAT Pool Supervisor — PR #2629 Validation — Session Tracker #2752

freemo commented

2026-04-05 06:45:39 +00:00

Author

Owner

Starting implementation on branch fix/nightly-coverage-min-threshold.

Fix: Change --coverage-min 85 to --coverage-min 97 in .forgejo/workflows/nightly-quality.yml line 80.

This is a straightforward single-line configuration fix to align the nightly workflow quality gates script with the project's required 97% coverage threshold.

Automated by CleverAgents Bot
Supervisor: Implementation | Agent: ca-issue-worker

Starting implementation on branch `fix/nightly-coverage-min-threshold`. **Fix**: Change `--coverage-min 85` to `--coverage-min 97` in `.forgejo/workflows/nightly-quality.yml` line 80. This is a straightforward single-line configuration fix to align the nightly workflow quality gates script with the project's required 97% coverage threshold. --- **Automated by CleverAgents Bot** Supervisor: Implementation | Agent: ca-issue-worker

freemo referenced this issue from a commit

2026-04-05 07:06:03 +00:00

fix(ci): update nightly workflow coverage minimum to 97% per project requirements

freemo referenced this issue from a pull request that will close it,

2026-04-05 07:06:41 +00:00

fix(ci): update nightly workflow coverage minimum to 97% per project requirements #3161

freemo commented

2026-04-05 07:08:28 +00:00

Author

Owner

All subtasks complete. Quality gates passed. Creating PR.

Implementation Summary:

Changed --coverage-min 85 → --coverage-min 97 in .forgejo/workflows/nightly-quality.yml
Updated features/consolidated_config.feature BDD scenario to assert "at least 97 percent" (was "at least 85 percent")
Lint and format quality gates: ✅ PASS
Manual verification of fix logic: ✅ PASS (97 >= 97)

PR #3161 created on branch fix/nightly-coverage-min-threshold. PR review and merge handled by continuous review stream.

Automated by CleverAgents Bot
Supervisor: Implementation | Agent: ca-issue-worker

All subtasks complete. Quality gates passed. Creating PR. **Implementation Summary:** - Changed `--coverage-min 85` → `--coverage-min 97` in `.forgejo/workflows/nightly-quality.yml` - Updated `features/consolidated_config.feature` BDD scenario to assert "at least 97 percent" (was "at least 85 percent") - Lint and format quality gates: ✅ PASS - Manual verification of fix logic: ✅ PASS (97 >= 97) PR #3161 created on branch `fix/nightly-coverage-min-threshold`. PR review and merge handled by continuous review stream. --- **Automated by CleverAgents Bot** Supervisor: Implementation | Agent: ca-issue-worker

freemo referenced this issue

2026-04-05 07:21:47 +00:00

[CA-AUTO] UAT Pool Supervisor — v3.2.0–v3.7.0 — Session Tracker #3094

freemo commented

2026-04-05 08:01:25 +00:00

Author

Owner

PR #3161 reviewed, approved, and merged.

The nightly workflow quality gates script now correctly uses --coverage-min 97 (was 85), aligning with the project's COVERAGE_THRESHOLD = 97 in noxfile.py. The corresponding BDD scenario was also updated.

Automated by CleverAgents Bot
Supervisor: PR Review | Agent: ca-pr-self-reviewer

PR #3161 reviewed, approved, and merged. The nightly workflow quality gates script now correctly uses `--coverage-min 97` (was `85`), aligning with the project's `COVERAGE_THRESHOLD = 97` in `noxfile.py`. The corresponding BDD scenario was also updated. --- **Automated by CleverAgents Bot** Supervisor: PR Review | Agent: ca-pr-self-reviewer

freemo closed this issue

2026-04-05 08:14:28 +00:00