CoreRasurae
e8d2f76466
CI / push-validation (pull_request) Successful in 45s
CI / helm (pull_request) Successful in 57s
CI / build (pull_request) Successful in 1m8s
CI / lint (pull_request) Successful in 1m31s
CI / tdd_quality_gate (pull_request) Failing after 1m26s
CI / quality (pull_request) Successful in 1m30s
CI / typecheck (pull_request) Successful in 1m43s
CI / security (pull_request) Successful in 1m42s
CI / e2e_tests (pull_request) Failing after 4m10s
CI / integration_tests (pull_request) Successful in 5m2s
CI / unit_tests (pull_request) Successful in 6m6s
CI / docker (pull_request) Successful in 1m32s
CI / coverage (pull_request) Successful in 12m12s
CI / status-check (push) Blocked by required conditions
CI / status-check (pull_request) Failing after 3s
CI / tdd_quality_gate (push) Has been skipped
CI / benchmark-regression (push) Failing after 1m8s
CI / build (push) Successful in 1m2s
CI / lint (push) Successful in 1m16s
CI / helm (push) Successful in 45s
CI / push-validation (push) Successful in 45s
CI / quality (push) Successful in 1m37s
CI / typecheck (push) Successful in 2m1s
CI / security (push) Successful in 2m1s
CI / integration_tests (push) Successful in 3m34s
CI / unit_tests (push) Successful in 5m14s
CI / docker (push) Successful in 1m37s
CI / coverage (push) Failing after 19m17s
CI / benchmark-publish (push) Successful in 1h20m43s
CI / e2e_tests (push) Successful in 4m13s
Add an automated quality gate that enforces TDD bug fix workflow rules on pull requests. The gate parses PR descriptions for bug-closing keywords (Closes/Fixes/Resolves #N, ISSUES CLOSED: #N), searches the codebase for corresponding TDD tests tagged @tdd_bug_N, and verifies that @tdd_expected_fail tags have been removed. Key components: - scripts/tdd_quality_gate.py: Main quality gate script with PR description parsing, TDD test discovery, and tag removal verification. All public functions validate arguments fail-fast and are statically typed. - noxfile.py: New tdd_quality_gate session that reads PR_DESCRIPTION from the environment and runs the quality gate script. - .forgejo/workflows/ci.yml: New tdd_quality_gate CI job that runs only on pull_request events, passing the PR body as PR_DESCRIPTION. - features/tdd_quality_gate.feature: 46 Behave scenarios covering PR parsing, TDD test search, tag removal verification, full gate logic, robot diff handling, edge cases, argument validation, bool guards, co-located bug false-positive guard, and main() CLI entry point. - features/steps/tdd_quality_gate_steps.py: Step definitions for all Behave scenarios using temporary directories for isolation. - robot/tdd_quality_gate.robot: 15 Robot Framework integration tests exercising the gate end-to-end via a helper subprocess. - robot/helper_tdd_quality_gate.py: Helper script for Robot tests with sentinel-based sub-commands. Review-round fixes applied: - check_expected_fail_removed now uses _contains_tag_token for word-boundary matching (avoids false positives on partial tag names) - Diff expected-fail removal detection tracks flags at file level instead of per-hunk (fixes false negatives when tags span hunks) - parse_bug_refs filters out issue number zero - Redundant double error reporting eliminated (file-level check short-circuits the diff-level check) - run_quality_gate returns (errors, bug_refs) tuple to avoid redundant re-parsing in main() - Regex compilation cached via functools.lru_cache - Nox session no longer installs the full project (stdlib only) - CI checkout uses fetch-depth: 0 for reliable merge-base resolution Review-round 2 fixes applied: - _diff_has_expected_fail_removal_for_bug now requires the removed line to contain both the expected-fail tag and the specific bug tag (fixes false positives when two bugs share the same test file) - check_expected_fail_removed error messages use the correct tag prefix per file type (@tdd_bug_N for .feature, tdd_bug_N for .robot) - bool values rejected by bug-number validation guards in find_tdd_tests, check_expected_fail_removed, and _diff_has_expected_fail_removal_for_bug - File-read error handling catches UnicodeDecodeError alongside OSError (root-safe unreadable-file handling via invalid-UTF-8 test fixture) - Temp directory cleanup added to after_scenario hook in environment.py - 8 new Behave scenarios: bool type guards (2), co-located bug false-positive regression (1), run_quality_gate argument validation (3), and main() CLI entry point exit codes (2) Review-round 3 fixes applied: - Synthetic PR diff helper (_default_pr_diff_for_bug_refs) now auto-detects .robot vs .feature file type from the temp search tree and generates the matching diff format (fixes under-tested robot-format diff code path in multi-bug integration scenarios) - check_expected_fail_removed test step now filters files by bug tag via find_tdd_tests before checking (matches production path in run_quality_gate) - after_scenario temp directory cleanup no longer sets context.temp_dir = None (fixes cleanup conflict with cli_init_yes_flag_steps.py cleanup functions that run after hooks) - 2 new Behave scenarios: multi-line PR description parsing, and non-string pr_diff type guard for run_quality_gate ISSUES CLOSED: #629
266 lines
11 KiB
Gherkin
266 lines
11 KiB
Gherkin
Feature: TDD bug tag quality gate for bug fix PRs
|
|
As a CI system
|
|
I want to enforce TDD bug fix workflow rules on PRs
|
|
So that bug fix PRs follow the required TDD workflow
|
|
|
|
# --- PR description parsing ---
|
|
|
|
Scenario: Parse single Fixes reference
|
|
Given a PR description "Fixes #42"
|
|
When I parse the bug references
|
|
Then the bug references should be [42]
|
|
|
|
Scenario: Parse single Closes reference
|
|
Given a PR description "Closes #100"
|
|
When I parse the bug references
|
|
Then the bug references should be [100]
|
|
|
|
Scenario: Parse single Resolves reference
|
|
Given a PR description "Resolves #7"
|
|
When I parse the bug references
|
|
Then the bug references should be [7]
|
|
|
|
Scenario: Parse case-insensitive closing keywords
|
|
Given a PR description "fixes #10 and CLOSES #20"
|
|
When I parse the bug references
|
|
Then the bug references should be [10, 20]
|
|
|
|
Scenario: Parse ISSUES CLOSED block
|
|
Given a PR description "ISSUES CLOSED: #5, #10"
|
|
When I parse the bug references
|
|
Then the bug references should be [5, 10]
|
|
|
|
Scenario: Parse multiple mixed references
|
|
Given a PR description "Fixes #42, also closes #99. ISSUES CLOSED: #7"
|
|
When I parse the bug references
|
|
Then the bug references should be [7, 42, 99]
|
|
|
|
Scenario: No bug references returns empty list
|
|
Given a PR description "Add new feature for users"
|
|
When I parse the bug references
|
|
Then the bug references should be []
|
|
|
|
Scenario: Deduplicate repeated bug references
|
|
Given a PR description "Fixes #42. Also closes #42"
|
|
When I parse the bug references
|
|
Then the bug references should be [42]
|
|
|
|
Scenario: Parse past tense closing keywords
|
|
Given a PR description "Fixed #15, Closed #20, Resolved #25"
|
|
When I parse the bug references
|
|
Then the bug references should be [15, 20, 25]
|
|
|
|
Scenario: Ignore non-closing words that end with keyword substrings
|
|
Given a PR description "prefixes #12 and hotfixes #34"
|
|
When I parse the bug references
|
|
Then the bug references should be []
|
|
|
|
Scenario: Parse bug reference in multi-line PR description
|
|
Given a multiline PR description
|
|
"""
|
|
feat(ci): implement quality gate
|
|
|
|
This PR adds the TDD quality gate.
|
|
|
|
Fixes #42
|
|
"""
|
|
When I parse the bug references
|
|
Then the bug references should be [42]
|
|
|
|
# --- TDD test search ---
|
|
|
|
Scenario: Find TDD test in .feature file
|
|
Given a temporary directory with a file "tests/bug.feature" containing "@tdd_bug_42"
|
|
When I search for TDD tests for bug 42
|
|
Then the search should find 1 test file
|
|
|
|
Scenario: Find TDD test in .robot file
|
|
Given a temporary directory with a file "tests/bug.robot" containing "tdd_bug_42"
|
|
When I search for TDD tests for bug 42
|
|
Then the search should find 1 test file
|
|
|
|
Scenario: Find TDD tests in both .feature and .robot files
|
|
Given a temporary directory with a file "features/bug.feature" containing "@tdd_bug_42"
|
|
And a temporary directory also has a file "robot/bug.robot" containing "tdd_bug_42"
|
|
When I search for TDD tests for bug 42
|
|
Then the search should find 2 test files
|
|
|
|
Scenario: No TDD test found for bug number
|
|
Given a temporary directory with a file "tests/other.feature" containing "@tdd_bug_99"
|
|
When I search for TDD tests for bug 42
|
|
Then the search should find 0 test files
|
|
|
|
Scenario: Do not match partial TDD bug tags
|
|
Given a temporary directory with a file "tests/partial.feature" containing "@tdd_bug_420"
|
|
When I search for TDD tests for bug 42
|
|
Then the search should find 0 test files
|
|
|
|
# --- Tag removal verification ---
|
|
|
|
Scenario: Expected fail tag still present in .feature file
|
|
Given a temporary directory with a file "tests/bug.feature" containing "@tdd_expected_fail @tdd_bug @tdd_bug_42"
|
|
When I check expected fail removal for bug 42
|
|
Then there should be 1 removal error
|
|
And the removal error should mention "@tdd_expected_fail"
|
|
And the removal error should mention "@tdd_bug_42"
|
|
|
|
Scenario: Expected fail tag removed from .feature file
|
|
Given a temporary directory with a file "tests/bug.feature" containing "@tdd_bug @tdd_bug_42"
|
|
When I check expected fail removal for bug 42
|
|
Then there should be 0 removal errors
|
|
|
|
Scenario: Expected fail tag still present in .robot file
|
|
Given a temporary directory with a file "tests/bug.robot" containing "tdd_expected_fail tdd_bug_42"
|
|
When I check expected fail removal for bug 42
|
|
Then there should be 1 removal error
|
|
|
|
Scenario: Expected fail tag removed from .robot file
|
|
Given a temporary directory with a file "tests/bug.robot" containing "tdd_bug tdd_bug_42"
|
|
When I check expected fail removal for bug 42
|
|
Then there should be 0 removal errors
|
|
|
|
# --- Full quality gate ---
|
|
|
|
Scenario: Quality gate passes when no bug refs in PR
|
|
Given a temporary search root
|
|
And a PR description "Add new feature"
|
|
When I run the quality gate
|
|
Then the quality gate should pass
|
|
|
|
Scenario: Quality gate fails when no TDD test exists for referenced bug
|
|
Given a temporary search root
|
|
And a PR description "Fixes #42"
|
|
When I run the quality gate
|
|
Then the quality gate should fail
|
|
And the quality gate errors should mention "No TDD test found for bug #42"
|
|
|
|
Scenario: Quality gate fails when expected fail tag is still present
|
|
Given a temporary search root with file "features/bug.feature" containing "@tdd_expected_fail @tdd_bug @tdd_bug_42"
|
|
And a PR description "Fixes #42"
|
|
When I run the quality gate
|
|
Then the quality gate should fail
|
|
And the quality gate errors should mention "@tdd_expected_fail"
|
|
|
|
Scenario: Quality gate passes when expected fail tag has been removed
|
|
Given a temporary search root with file "features/bug.feature" containing "@tdd_bug @tdd_bug_42"
|
|
And a PR description "Fixes #42"
|
|
When I run the quality gate
|
|
Then the quality gate should pass
|
|
|
|
Scenario: Quality gate handles multiple bug references
|
|
Given a temporary search root with file "features/bug10.feature" containing "@tdd_bug @tdd_bug_10"
|
|
And the search root also has file "features/bug20.feature" containing "@tdd_expected_fail @tdd_bug @tdd_bug_20"
|
|
And a PR description "Fixes #10 and fixes #20"
|
|
When I run the quality gate
|
|
Then the quality gate should fail
|
|
And the quality gate errors should mention "@tdd_bug_20"
|
|
|
|
Scenario: Quality gate passes when all bugs have clean TDD tests
|
|
Given a temporary search root with file "features/bug10.feature" containing "@tdd_bug @tdd_bug_10"
|
|
And the search root also has file "robot/bug20.robot" containing "tdd_bug tdd_bug_20"
|
|
And a PR description "Fixes #10 and fixes #20"
|
|
When I run the quality gate
|
|
Then the quality gate should pass
|
|
|
|
Scenario: Quality gate fails when PR diff does not remove expected fail tags
|
|
Given a temporary search root with file "features/bug.feature" containing "@tdd_bug @tdd_bug_42"
|
|
And a PR description "Fixes #42"
|
|
And the PR diff does not remove expected fail tags
|
|
When I run the quality gate
|
|
Then the quality gate should fail
|
|
And the quality gate errors should mention "No removal of @tdd_expected_fail / tdd_expected_fail detected"
|
|
|
|
Scenario: Quality gate passes for robot diff with expected fail removed across hunks
|
|
Given a temporary search root with file "robot/bug.robot" containing "tdd_bug tdd_bug_42"
|
|
And a PR description "Fixes #42"
|
|
And the PR diff removes expected fail for robot bug 42
|
|
When I run the quality gate
|
|
Then the quality gate should pass
|
|
|
|
Scenario: Issue number zero is silently ignored
|
|
Given a temporary search root
|
|
And a PR description "Fixes #0"
|
|
When I run the quality gate
|
|
Then the quality gate should pass
|
|
|
|
Scenario: find_tdd_tests skips unreadable files
|
|
Given a temporary directory with an unreadable feature file for bug 42
|
|
When I search for TDD tests for bug 42
|
|
Then the search should find 0 test files
|
|
|
|
Scenario: check_expected_fail_removed skips unreadable files
|
|
Given a temporary directory with an unreadable feature file for bug 42
|
|
When I check expected fail removal for bug 42
|
|
Then there should be 0 removal errors
|
|
|
|
# --- Argument validation ---
|
|
|
|
Scenario: parse_bug_refs rejects non-string input
|
|
When I call parse_bug_refs with a non-string argument
|
|
Then a TypeError should be raised by the quality gate
|
|
|
|
Scenario: find_tdd_tests rejects invalid bug number
|
|
When I call find_tdd_tests with bug number 0
|
|
Then a ValueError should be raised by the quality gate
|
|
|
|
Scenario: find_tdd_tests rejects non-Path search root
|
|
When I call find_tdd_tests with a non-Path search root
|
|
Then a TypeError should be raised by the quality gate
|
|
|
|
Scenario: check_expected_fail_removed rejects non-list input
|
|
When I call check_expected_fail_removed with a non-list argument
|
|
Then a TypeError should be raised by the quality gate
|
|
|
|
Scenario: check_expected_fail_removed rejects invalid bug number
|
|
When I call check_expected_fail_removed with bug number 0
|
|
Then a ValueError should be raised by the quality gate
|
|
|
|
# --- Bool type guard ---
|
|
|
|
Scenario: find_tdd_tests rejects boolean True as bug number
|
|
When I call find_tdd_tests with boolean True as bug number
|
|
Then a ValueError should be raised by the quality gate
|
|
|
|
Scenario: check_expected_fail_removed rejects boolean True as bug number
|
|
When I call check_expected_fail_removed with boolean True as bug number
|
|
Then a ValueError should be raised by the quality gate
|
|
|
|
# --- Co-located bug false positive guard (M1) ---
|
|
|
|
Scenario: Diff detection does not false-positive for co-located bug tests
|
|
Given a temporary search root with file "features/bugs.feature" containing "@tdd_bug @tdd_bug_42"
|
|
And the search root also has file "features/bugs99.feature" containing "@tdd_bug @tdd_bug_99"
|
|
And a PR description "Fixes #42"
|
|
And the PR diff only removes expected fail for bug 99 not bug 42
|
|
When I run the quality gate
|
|
Then the quality gate should fail
|
|
And the quality gate errors should mention "No removal of @tdd_expected_fail / tdd_expected_fail detected"
|
|
|
|
# --- run_quality_gate argument validation (L4) ---
|
|
|
|
Scenario: run_quality_gate rejects non-string PR description
|
|
When I call run_quality_gate with a non-string PR description
|
|
Then a TypeError should be raised by the quality gate
|
|
|
|
Scenario: run_quality_gate rejects non-Path search root
|
|
When I call run_quality_gate with a non-Path search root
|
|
Then a TypeError should be raised by the quality gate
|
|
|
|
Scenario: run_quality_gate rejects empty base_ref
|
|
When I call run_quality_gate with an empty base_ref
|
|
Then a ValueError should be raised by the quality gate
|
|
|
|
Scenario: run_quality_gate rejects non-string pr_diff
|
|
When I call run_quality_gate with a non-string pr_diff
|
|
Then a TypeError should be raised by the quality gate
|
|
|
|
# --- main() CLI entry point (M2) ---
|
|
|
|
Scenario: main returns 0 when no bug refs in PR description
|
|
When I call main with PR_DESCRIPTION "Add new feature"
|
|
Then the main exit code should be 0
|
|
|
|
Scenario: main returns 1 when TDD test is missing
|
|
When I call main with PR_DESCRIPTION "Fixes #99999" in an empty search root
|
|
Then the main exit code should be 1
|