feat(ci): implement TDD bug tag quality gate for bug fix PRs
CI / push-validation (pull_request) Successful in 45s
CI / helm (pull_request) Successful in 57s
CI / build (pull_request) Successful in 1m8s
CI / lint (pull_request) Successful in 1m31s
CI / tdd_quality_gate (pull_request) Failing after 1m26s
CI / quality (pull_request) Successful in 1m30s
CI / typecheck (pull_request) Successful in 1m43s
CI / security (pull_request) Successful in 1m42s
CI / e2e_tests (pull_request) Failing after 4m10s
CI / integration_tests (pull_request) Successful in 5m2s
CI / unit_tests (pull_request) Successful in 6m6s
CI / docker (pull_request) Successful in 1m32s
CI / coverage (pull_request) Successful in 12m12s
CI / status-check (push) Blocked by required conditions
CI / status-check (pull_request) Failing after 3s
CI / tdd_quality_gate (push) Has been skipped
CI / benchmark-regression (push) Failing after 1m8s
CI / build (push) Successful in 1m2s
CI / lint (push) Successful in 1m16s
CI / helm (push) Successful in 45s
CI / push-validation (push) Successful in 45s
CI / quality (push) Successful in 1m37s
CI / typecheck (push) Successful in 2m1s
CI / security (push) Successful in 2m1s
CI / integration_tests (push) Successful in 3m34s
CI / unit_tests (push) Successful in 5m14s
CI / docker (push) Successful in 1m37s
CI / coverage (push) Failing after 19m17s
CI / benchmark-publish (push) Successful in 1h20m43s
CI / e2e_tests (push) Successful in 4m13s

Add an automated quality gate that enforces TDD bug fix workflow rules
on pull requests. The gate parses PR descriptions for bug-closing
keywords (Closes/Fixes/Resolves #N, ISSUES CLOSED: #N), searches the
codebase for corresponding TDD tests tagged @tdd_bug_N, and verifies
that @tdd_expected_fail tags have been removed.

Key components:
- scripts/tdd_quality_gate.py: Main quality gate script with PR
  description parsing, TDD test discovery, and tag removal verification.
  All public functions validate arguments fail-fast and are statically
  typed.
- noxfile.py: New tdd_quality_gate session that reads PR_DESCRIPTION
  from the environment and runs the quality gate script.
- .forgejo/workflows/ci.yml: New tdd_quality_gate CI job that runs
  only on pull_request events, passing the PR body as PR_DESCRIPTION.
- features/tdd_quality_gate.feature: 46 Behave scenarios covering PR
  parsing, TDD test search, tag removal verification, full gate logic,
  robot diff handling, edge cases, argument validation, bool guards,
  co-located bug false-positive guard, and main() CLI entry point.
- features/steps/tdd_quality_gate_steps.py: Step definitions for all
  Behave scenarios using temporary directories for isolation.
- robot/tdd_quality_gate.robot: 15 Robot Framework integration tests
  exercising the gate end-to-end via a helper subprocess.
- robot/helper_tdd_quality_gate.py: Helper script for Robot tests with
  sentinel-based sub-commands.

Review-round fixes applied:
- check_expected_fail_removed now uses _contains_tag_token for
  word-boundary matching (avoids false positives on partial tag names)
- Diff expected-fail removal detection tracks flags at file level
  instead of per-hunk (fixes false negatives when tags span hunks)
- parse_bug_refs filters out issue number zero
- Redundant double error reporting eliminated (file-level check
  short-circuits the diff-level check)
- run_quality_gate returns (errors, bug_refs) tuple to avoid
  redundant re-parsing in main()
- Regex compilation cached via functools.lru_cache
- Nox session no longer installs the full project (stdlib only)
- CI checkout uses fetch-depth: 0 for reliable merge-base resolution

Review-round 2 fixes applied:
- _diff_has_expected_fail_removal_for_bug now requires the removed
  line to contain both the expected-fail tag and the specific bug tag
  (fixes false positives when two bugs share the same test file)
- check_expected_fail_removed error messages use the correct tag
  prefix per file type (@tdd_bug_N for .feature, tdd_bug_N for .robot)
- bool values rejected by bug-number validation guards in
  find_tdd_tests, check_expected_fail_removed, and
  _diff_has_expected_fail_removal_for_bug
- File-read error handling catches UnicodeDecodeError alongside OSError
  (root-safe unreadable-file handling via invalid-UTF-8 test fixture)
- Temp directory cleanup added to after_scenario hook in environment.py
- 8 new Behave scenarios: bool type guards (2), co-located bug
  false-positive regression (1), run_quality_gate argument validation
  (3), and main() CLI entry point exit codes (2)

Review-round 3 fixes applied:
- Synthetic PR diff helper (_default_pr_diff_for_bug_refs) now
  auto-detects .robot vs .feature file type from the temp search
  tree and generates the matching diff format (fixes under-tested
  robot-format diff code path in multi-bug integration scenarios)
- check_expected_fail_removed test step now filters files by bug
  tag via find_tdd_tests before checking (matches production path
  in run_quality_gate)
- after_scenario temp directory cleanup no longer sets
  context.temp_dir = None (fixes cleanup conflict with
  cli_init_yes_flag_steps.py cleanup functions that run after hooks)
- 2 new Behave scenarios: multi-line PR description parsing, and
  non-string pr_diff type guard for run_quality_gate

ISSUES CLOSED: #629
This commit was merged in pull request #1155.
This commit is contained in:
CoreRasurae
2026-03-25 00:43:02 +00:00
committed by Luis Mendes
parent dd763f50d9
commit e8d2f76466
11 changed files with 1938 additions and 9 deletions
+23
View File
@@ -882,6 +882,29 @@ def benchmark_regression(session: nox.Session):
session.run("asv", "publish", f"--config={config_path}")
@nox.session(python=DEFAULT_PYTHON, reuse_venv=True, venv_backend="uv")
def tdd_quality_gate(session: nox.Session):
"""Enforce TDD bug fix workflow rules on PRs.
Reads the PR description from the ``PR_DESCRIPTION`` environment
variable and verifies that:
1. Every bug referenced via closing keywords (``Fixes #N``,
``Closes #N``, ``Resolves #N``, ``ISSUES CLOSED: #N``) has
a corresponding TDD test tagged ``@tdd_bug_N``.
2. The ``@tdd_expected_fail`` / ``tdd_expected_fail`` tag has been
removed from each of those tests in the PR diff.
If no bug references are found the gate passes trivially.
"""
# The quality gate script uses only standard library; no install needed.
pr_description = os.environ.get("PR_DESCRIPTION", "")
pr_base_ref = os.environ.get("PR_BASE_REF", "master")
session.env["PR_DESCRIPTION"] = pr_description
session.env["PR_BASE_REF"] = pr_base_ref
session.run("python", "scripts/tdd_quality_gate.py")
# Sessions to run by default when running `nox` without arguments
nox.options.sessions = [
"lint", # ~5-10 seconds