test(agents/plan_generation): fix BDD step ambiguity and add tdd_bug tag (#10480)
CI / push-validation (pull_request) Successful in 19s
CI / helm (pull_request) Successful in 28s
CI / build (pull_request) Successful in 42s
CI / lint (pull_request) Successful in 48s
CI / quality (pull_request) Successful in 49s
CI / tdd_quality_gate (pull_request) Failing after 59s
CI / typecheck (pull_request) Successful in 1m14s
CI / security (pull_request) Successful in 1m15s
CI / e2e_tests (pull_request) Successful in 4m2s
CI / integration_tests (pull_request) Failing after 4m8s
CI / unit_tests (pull_request) Failing after 4m56s
CI / coverage (pull_request) Has been skipped
CI / docker (pull_request) Has been skipped
CI / status-check (pull_request) Failing after 3s
CI / push-validation (pull_request) Successful in 19s
CI / helm (pull_request) Successful in 28s
CI / build (pull_request) Successful in 42s
CI / lint (pull_request) Successful in 48s
CI / quality (pull_request) Successful in 49s
CI / tdd_quality_gate (pull_request) Failing after 59s
CI / typecheck (pull_request) Successful in 1m14s
CI / security (pull_request) Successful in 1m15s
CI / e2e_tests (pull_request) Successful in 4m2s
CI / integration_tests (pull_request) Failing after 4m8s
CI / unit_tests (pull_request) Failing after 4m56s
CI / coverage (pull_request) Has been skipped
CI / docker (pull_request) Has been skipped
CI / status-check (pull_request) Failing after 3s
The compliance tests added in 80fa1329 had three issues that broke
multiple CI gates:
1. unit_tests (Behave): the new
features/steps/validation_bypass_issue_10480_steps.py redefined
`@given("the plan generation coverage module is imported")`, which
was already defined in plan_generation_coverage_boost_steps.py.
Behave raised AmbiguousStep and 32 scenarios errored on import.
The duplicate has been removed; the feature reuses the existing
step via behave's global step discovery.
2. unit_tests (Behave): three `@then` decorators included a literal
"and " prefix (e.g. `@then("and no artificial length threshold ...")`),
which never matches because Behave normalises "And" into the
preceding step's keyword and strips it from the match text. The
prefixes have been removed so the three previously-undefined steps
now resolve.
3. tdd_quality_gate: scenarios lacked the `@tdd_bug_10480` tag that
scripts/tdd_quality_gate.py requires to recognise the regression
guard for the bug. All 5 scenarios now carry the tag.
Also fixes a grammatical error in the CHANGELOG entry
("responsible for determine" → "responsible for determining") noted
by the reviewer.
Refs #10480
This commit is contained in:
+1
-1
@@ -7,7 +7,7 @@
|
||||
- Fixed a validation bypass in `PlanGenerationGraph._validate()` (issue #10480):
|
||||
the artificial ``or len(all_code) > 10`` condition that caused any code longer
|
||||
than 10 characters to auto-pass LLM validation has been removed. LLM verdicts
|
||||
are now solely responsible for determine pass/fail status, enabling the retry
|
||||
are now solely responsible for determining pass/fail status, enabling the retry
|
||||
logic and broken-code rejection to operate correctly.
|
||||
|
||||
- Hardened the TDD bug-fix quality gate for issue #629: PR parsing now
|
||||
|
||||
@@ -40,10 +40,10 @@ def make_fake_llm(responses: list[str]) -> FakeListLLM:
|
||||
# ---------------------------------------------------------------------------
|
||||
# Background steps
|
||||
# ---------------------------------------------------------------------------
|
||||
@given("the plan generation coverage module is imported")
|
||||
def step_coverage_module_imported(context):
|
||||
"""Ensure the plan_generation module is importable."""
|
||||
assert PlanGenerationGraph is not None
|
||||
# Note: "the plan generation coverage module is imported" is defined in
|
||||
# features/steps/plan_generation_coverage_boost_steps.py and reused here
|
||||
# (behave discovers steps globally across features/steps/). Defining it
|
||||
# again would raise AmbiguousStep.
|
||||
|
||||
|
||||
@given("the validation bypass test helper is ready")
|
||||
@@ -168,7 +168,7 @@ def step_validation_message_contains_feedback(context):
|
||||
)
|
||||
|
||||
|
||||
@then("and no artificial length threshold bypassed the rejection")
|
||||
@then("no artificial length threshold bypassed the rejection")
|
||||
def step_no_length_threshold_bypass(context):
|
||||
"""Confirm that code length did not override a FAIL verdict.
|
||||
|
||||
@@ -185,7 +185,7 @@ def step_no_length_threshold_bypass(context):
|
||||
)
|
||||
|
||||
|
||||
@then("and the validation message should preserve the full LLM feedback")
|
||||
@then("the validation message should preserve the full LLM feedback")
|
||||
def step_validation_preserves_feedback(context):
|
||||
"""Full message must contain the original LLM response."""
|
||||
assert context.validation_result is not None
|
||||
@@ -196,7 +196,7 @@ def step_validation_preserves_feedback(context):
|
||||
)
|
||||
|
||||
|
||||
@then("and the code-length condition must NOT override the FAIL verdict")
|
||||
@then("the code-length condition must NOT override the FAIL verdict")
|
||||
def step_code_length_not_override(context):
|
||||
"""Definitive regression test: code length > 10 did not cause auto-pass.
|
||||
|
||||
|
||||
@@ -7,30 +7,35 @@ Feature: Validation bypass fix (issue #10480)
|
||||
Given the plan generation coverage module is imported
|
||||
And the validation bypass test helper is ready
|
||||
|
||||
@tdd_bug_10480
|
||||
Scenario: LLM returns FAIL for short code and validation correctly rejects it
|
||||
Given an LLM mock that responds with "FAIL: syntax error"
|
||||
When a PlanGenerationGraph _validate is called with short generated code
|
||||
Then the validation result status should be "FAIL"
|
||||
And the validation message should contain the LLM feedback
|
||||
|
||||
@tdd_bug_10480
|
||||
Scenario: LLM returns FAIL for long code and validation correctly rejects it
|
||||
Given an LLM mock that responds with "FAIL: unsafe pattern detected"
|
||||
When a PlanGenerationGraph _validate is called with longer generated code
|
||||
Then the validation result status should be "FAIL"
|
||||
And no artificial length threshold bypassed the rejection
|
||||
|
||||
@tdd_bug_10480
|
||||
Scenario: LLM returns PASS and validation correctly accepts it
|
||||
Given an LLM mock that responds with "PASS"
|
||||
When a PlanGenerationGraph _validate is called with any generated code
|
||||
Then the validation result status should be "PASS"
|
||||
And the validation message should contain the LLM feedback
|
||||
|
||||
@tdd_bug_10480
|
||||
Scenario: LLM returns FAIL with additional details and validation correctly rejects
|
||||
Given an LLM mock that responds with "FAIL: missing error handling in auth module"
|
||||
When a PlanGenerationGraph _validate is called with generated code
|
||||
Then the validation result status should be "FAIL"
|
||||
And the validation message should preserve the full LLM feedback
|
||||
|
||||
@tdd_bug_10480
|
||||
Scenario: Validation ignores code length and does not auto-pass long code
|
||||
Given an LLM mock that responds with "FAIL: security vulnerability"
|
||||
When a PlanGenerationGraph _validate is called with code longer than 10 characters
|
||||
|
||||
Reference in New Issue
Block a user