feat(ci): implement TDD bug tag quality gate for bug fix PRs
CI / push-validation (pull_request) Successful in 45s
CI / helm (pull_request) Successful in 57s
CI / build (pull_request) Successful in 1m8s
CI / lint (pull_request) Successful in 1m31s
CI / tdd_quality_gate (pull_request) Failing after 1m26s
CI / quality (pull_request) Successful in 1m30s
CI / typecheck (pull_request) Successful in 1m43s
CI / security (pull_request) Successful in 1m42s
CI / e2e_tests (pull_request) Failing after 4m10s
CI / integration_tests (pull_request) Successful in 5m2s
CI / unit_tests (pull_request) Successful in 6m6s
CI / docker (pull_request) Successful in 1m32s
CI / coverage (pull_request) Successful in 12m12s
CI / status-check (push) Blocked by required conditions
CI / status-check (pull_request) Failing after 3s
CI / tdd_quality_gate (push) Has been skipped
CI / benchmark-regression (push) Failing after 1m8s
CI / build (push) Successful in 1m2s
CI / lint (push) Successful in 1m16s
CI / helm (push) Successful in 45s
CI / push-validation (push) Successful in 45s
CI / quality (push) Successful in 1m37s
CI / typecheck (push) Successful in 2m1s
CI / security (push) Successful in 2m1s
CI / integration_tests (push) Successful in 3m34s
CI / unit_tests (push) Successful in 5m14s
CI / docker (push) Successful in 1m37s
CI / coverage (push) Failing after 19m17s
CI / benchmark-publish (push) Successful in 1h20m43s
CI / e2e_tests (push) Successful in 4m13s
CI / push-validation (pull_request) Successful in 45s
CI / helm (pull_request) Successful in 57s
CI / build (pull_request) Successful in 1m8s
CI / lint (pull_request) Successful in 1m31s
CI / tdd_quality_gate (pull_request) Failing after 1m26s
CI / quality (pull_request) Successful in 1m30s
CI / typecheck (pull_request) Successful in 1m43s
CI / security (pull_request) Successful in 1m42s
CI / e2e_tests (pull_request) Failing after 4m10s
CI / integration_tests (pull_request) Successful in 5m2s
CI / unit_tests (pull_request) Successful in 6m6s
CI / docker (pull_request) Successful in 1m32s
CI / coverage (pull_request) Successful in 12m12s
CI / status-check (push) Blocked by required conditions
CI / status-check (pull_request) Failing after 3s
CI / tdd_quality_gate (push) Has been skipped
CI / benchmark-regression (push) Failing after 1m8s
CI / build (push) Successful in 1m2s
CI / lint (push) Successful in 1m16s
CI / helm (push) Successful in 45s
CI / push-validation (push) Successful in 45s
CI / quality (push) Successful in 1m37s
CI / typecheck (push) Successful in 2m1s
CI / security (push) Successful in 2m1s
CI / integration_tests (push) Successful in 3m34s
CI / unit_tests (push) Successful in 5m14s
CI / docker (push) Successful in 1m37s
CI / coverage (push) Failing after 19m17s
CI / benchmark-publish (push) Successful in 1h20m43s
CI / e2e_tests (push) Successful in 4m13s
Add an automated quality gate that enforces TDD bug fix workflow rules on pull requests. The gate parses PR descriptions for bug-closing keywords (Closes/Fixes/Resolves #N, ISSUES CLOSED: #N), searches the codebase for corresponding TDD tests tagged @tdd_bug_N, and verifies that @tdd_expected_fail tags have been removed. Key components: - scripts/tdd_quality_gate.py: Main quality gate script with PR description parsing, TDD test discovery, and tag removal verification. All public functions validate arguments fail-fast and are statically typed. - noxfile.py: New tdd_quality_gate session that reads PR_DESCRIPTION from the environment and runs the quality gate script. - .forgejo/workflows/ci.yml: New tdd_quality_gate CI job that runs only on pull_request events, passing the PR body as PR_DESCRIPTION. - features/tdd_quality_gate.feature: 46 Behave scenarios covering PR parsing, TDD test search, tag removal verification, full gate logic, robot diff handling, edge cases, argument validation, bool guards, co-located bug false-positive guard, and main() CLI entry point. - features/steps/tdd_quality_gate_steps.py: Step definitions for all Behave scenarios using temporary directories for isolation. - robot/tdd_quality_gate.robot: 15 Robot Framework integration tests exercising the gate end-to-end via a helper subprocess. - robot/helper_tdd_quality_gate.py: Helper script for Robot tests with sentinel-based sub-commands. Review-round fixes applied: - check_expected_fail_removed now uses _contains_tag_token for word-boundary matching (avoids false positives on partial tag names) - Diff expected-fail removal detection tracks flags at file level instead of per-hunk (fixes false negatives when tags span hunks) - parse_bug_refs filters out issue number zero - Redundant double error reporting eliminated (file-level check short-circuits the diff-level check) - run_quality_gate returns (errors, bug_refs) tuple to avoid redundant re-parsing in main() - Regex compilation cached via functools.lru_cache - Nox session no longer installs the full project (stdlib only) - CI checkout uses fetch-depth: 0 for reliable merge-base resolution Review-round 2 fixes applied: - _diff_has_expected_fail_removal_for_bug now requires the removed line to contain both the expected-fail tag and the specific bug tag (fixes false positives when two bugs share the same test file) - check_expected_fail_removed error messages use the correct tag prefix per file type (@tdd_bug_N for .feature, tdd_bug_N for .robot) - bool values rejected by bug-number validation guards in find_tdd_tests, check_expected_fail_removed, and _diff_has_expected_fail_removal_for_bug - File-read error handling catches UnicodeDecodeError alongside OSError (root-safe unreadable-file handling via invalid-UTF-8 test fixture) - Temp directory cleanup added to after_scenario hook in environment.py - 8 new Behave scenarios: bool type guards (2), co-located bug false-positive regression (1), run_quality_gate argument validation (3), and main() CLI entry point exit codes (2) Review-round 3 fixes applied: - Synthetic PR diff helper (_default_pr_diff_for_bug_refs) now auto-detects .robot vs .feature file type from the temp search tree and generates the matching diff format (fixes under-tested robot-format diff code path in multi-bug integration scenarios) - check_expected_fail_removed test step now filters files by bug tag via find_tdd_tests before checking (matches production path in run_quality_gate) - after_scenario temp directory cleanup no longer sets context.temp_dir = None (fixes cleanup conflict with cli_init_yes_flag_steps.py cleanup functions that run after hooks) - 2 new Behave scenarios: multi-line PR description parsing, and non-string pr_diff type guard for run_quality_gate ISSUES CLOSED: #629
This commit was merged in pull request #1155.
This commit is contained in:
+23
@@ -882,6 +882,29 @@ def benchmark_regression(session: nox.Session):
|
||||
session.run("asv", "publish", f"--config={config_path}")
|
||||
|
||||
|
||||
@nox.session(python=DEFAULT_PYTHON, reuse_venv=True, venv_backend="uv")
|
||||
def tdd_quality_gate(session: nox.Session):
|
||||
"""Enforce TDD bug fix workflow rules on PRs.
|
||||
|
||||
Reads the PR description from the ``PR_DESCRIPTION`` environment
|
||||
variable and verifies that:
|
||||
|
||||
1. Every bug referenced via closing keywords (``Fixes #N``,
|
||||
``Closes #N``, ``Resolves #N``, ``ISSUES CLOSED: #N``) has
|
||||
a corresponding TDD test tagged ``@tdd_bug_N``.
|
||||
2. The ``@tdd_expected_fail`` / ``tdd_expected_fail`` tag has been
|
||||
removed from each of those tests in the PR diff.
|
||||
|
||||
If no bug references are found the gate passes trivially.
|
||||
"""
|
||||
# The quality gate script uses only standard library; no install needed.
|
||||
pr_description = os.environ.get("PR_DESCRIPTION", "")
|
||||
pr_base_ref = os.environ.get("PR_BASE_REF", "master")
|
||||
session.env["PR_DESCRIPTION"] = pr_description
|
||||
session.env["PR_BASE_REF"] = pr_base_ref
|
||||
session.run("python", "scripts/tdd_quality_gate.py")
|
||||
|
||||
|
||||
# Sessions to run by default when running `nox` without arguments
|
||||
nox.options.sessions = [
|
||||
"lint", # ~5-10 seconds
|
||||
|
||||
Reference in New Issue
Block a user