test(agents/plan_generation): fix BDD step ambiguity and add tdd_bug tag (#10480)
CI / push-validation (pull_request) Successful in 19s
CI / helm (pull_request) Successful in 28s
CI / build (pull_request) Successful in 42s
CI / lint (pull_request) Successful in 48s
CI / quality (pull_request) Successful in 49s
CI / tdd_quality_gate (pull_request) Failing after 59s
CI / typecheck (pull_request) Successful in 1m14s
CI / security (pull_request) Successful in 1m15s
CI / e2e_tests (pull_request) Successful in 4m2s
CI / integration_tests (pull_request) Failing after 4m8s
CI / unit_tests (pull_request) Failing after 4m56s
CI / coverage (pull_request) Has been skipped
CI / docker (pull_request) Has been skipped
CI / status-check (pull_request) Failing after 3s

The compliance tests added in 80fa1329 had three issues that broke
multiple CI gates:

1. unit_tests (Behave): the new
   features/steps/validation_bypass_issue_10480_steps.py redefined
   `@given("the plan generation coverage module is imported")`, which
   was already defined in plan_generation_coverage_boost_steps.py.
   Behave raised AmbiguousStep and 32 scenarios errored on import.
   The duplicate has been removed; the feature reuses the existing
   step via behave's global step discovery.

2. unit_tests (Behave): three `@then` decorators included a literal
   "and " prefix (e.g. `@then("and no artificial length threshold ...")`),
   which never matches because Behave normalises "And" into the
   preceding step's keyword and strips it from the match text. The
   prefixes have been removed so the three previously-undefined steps
   now resolve.

3. tdd_quality_gate: scenarios lacked the `@tdd_bug_10480` tag that
   scripts/tdd_quality_gate.py requires to recognise the regression
   guard for the bug. All 5 scenarios now carry the tag.

Also fixes a grammatical error in the CHANGELOG entry
("responsible for determine" → "responsible for determining") noted
by the reviewer.

Refs #10480
This commit is contained in:
2026-06-15 00:51:20 -04:00
parent 80fa132921
commit e5cf1a4528
3 changed files with 13 additions and 8 deletions
+1 -1
View File
@@ -7,7 +7,7 @@
- Fixed a validation bypass in `PlanGenerationGraph._validate()` (issue #10480):
the artificial ``or len(all_code) > 10`` condition that caused any code longer
than 10 characters to auto-pass LLM validation has been removed. LLM verdicts
are now solely responsible for determine pass/fail status, enabling the retry
are now solely responsible for determining pass/fail status, enabling the retry
logic and broken-code rejection to operate correctly.
- Hardened the TDD bug-fix quality gate for issue #629: PR parsing now
@@ -40,10 +40,10 @@ def make_fake_llm(responses: list[str]) -> FakeListLLM:
# ---------------------------------------------------------------------------
# Background steps
# ---------------------------------------------------------------------------
@given("the plan generation coverage module is imported")
def step_coverage_module_imported(context):
"""Ensure the plan_generation module is importable."""
assert PlanGenerationGraph is not None
# Note: "the plan generation coverage module is imported" is defined in
# features/steps/plan_generation_coverage_boost_steps.py and reused here
# (behave discovers steps globally across features/steps/). Defining it
# again would raise AmbiguousStep.
@given("the validation bypass test helper is ready")
@@ -168,7 +168,7 @@ def step_validation_message_contains_feedback(context):
)
@then("and no artificial length threshold bypassed the rejection")
@then("no artificial length threshold bypassed the rejection")
def step_no_length_threshold_bypass(context):
"""Confirm that code length did not override a FAIL verdict.
@@ -185,7 +185,7 @@ def step_no_length_threshold_bypass(context):
)
@then("and the validation message should preserve the full LLM feedback")
@then("the validation message should preserve the full LLM feedback")
def step_validation_preserves_feedback(context):
"""Full message must contain the original LLM response."""
assert context.validation_result is not None
@@ -196,7 +196,7 @@ def step_validation_preserves_feedback(context):
)
@then("and the code-length condition must NOT override the FAIL verdict")
@then("the code-length condition must NOT override the FAIL verdict")
def step_code_length_not_override(context):
"""Definitive regression test: code length > 10 did not cause auto-pass.
@@ -7,30 +7,35 @@ Feature: Validation bypass fix (issue #10480)
Given the plan generation coverage module is imported
And the validation bypass test helper is ready
@tdd_bug_10480
Scenario: LLM returns FAIL for short code and validation correctly rejects it
Given an LLM mock that responds with "FAIL: syntax error"
When a PlanGenerationGraph _validate is called with short generated code
Then the validation result status should be "FAIL"
And the validation message should contain the LLM feedback
@tdd_bug_10480
Scenario: LLM returns FAIL for long code and validation correctly rejects it
Given an LLM mock that responds with "FAIL: unsafe pattern detected"
When a PlanGenerationGraph _validate is called with longer generated code
Then the validation result status should be "FAIL"
And no artificial length threshold bypassed the rejection
@tdd_bug_10480
Scenario: LLM returns PASS and validation correctly accepts it
Given an LLM mock that responds with "PASS"
When a PlanGenerationGraph _validate is called with any generated code
Then the validation result status should be "PASS"
And the validation message should contain the LLM feedback
@tdd_bug_10480
Scenario: LLM returns FAIL with additional details and validation correctly rejects
Given an LLM mock that responds with "FAIL: missing error handling in auth module"
When a PlanGenerationGraph _validate is called with generated code
Then the validation result status should be "FAIL"
And the validation message should preserve the full LLM feedback
@tdd_bug_10480
Scenario: Validation ignores code length and does not auto-pass long code
Given an LLM mock that responds with "FAIL: security vulnerability"
When a PlanGenerationGraph _validate is called with code longer than 10 characters