diff --git a/CHANGELOG.md b/CHANGELOG.md index 414fe53f8..5becaa0a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ - Fixed a validation bypass in `PlanGenerationGraph._validate()` (issue #10480): the artificial ``or len(all_code) > 10`` condition that caused any code longer than 10 characters to auto-pass LLM validation has been removed. LLM verdicts - are now solely responsible for determine pass/fail status, enabling the retry + are now solely responsible for determining pass/fail status, enabling the retry logic and broken-code rejection to operate correctly. - Hardened the TDD bug-fix quality gate for issue #629: PR parsing now diff --git a/features/steps/validation_bypass_issue_10480_steps.py b/features/steps/validation_bypass_issue_10480_steps.py index c3565bd60..0eb4ee09a 100644 --- a/features/steps/validation_bypass_issue_10480_steps.py +++ b/features/steps/validation_bypass_issue_10480_steps.py @@ -40,10 +40,10 @@ def make_fake_llm(responses: list[str]) -> FakeListLLM: # --------------------------------------------------------------------------- # Background steps # --------------------------------------------------------------------------- -@given("the plan generation coverage module is imported") -def step_coverage_module_imported(context): - """Ensure the plan_generation module is importable.""" - assert PlanGenerationGraph is not None +# Note: "the plan generation coverage module is imported" is defined in +# features/steps/plan_generation_coverage_boost_steps.py and reused here +# (behave discovers steps globally across features/steps/). Defining it +# again would raise AmbiguousStep. @given("the validation bypass test helper is ready") @@ -168,7 +168,7 @@ def step_validation_message_contains_feedback(context): ) -@then("and no artificial length threshold bypassed the rejection") +@then("no artificial length threshold bypassed the rejection") def step_no_length_threshold_bypass(context): """Confirm that code length did not override a FAIL verdict. @@ -185,7 +185,7 @@ def step_no_length_threshold_bypass(context): ) -@then("and the validation message should preserve the full LLM feedback") +@then("the validation message should preserve the full LLM feedback") def step_validation_preserves_feedback(context): """Full message must contain the original LLM response.""" assert context.validation_result is not None @@ -196,7 +196,7 @@ def step_validation_preserves_feedback(context): ) -@then("and the code-length condition must NOT override the FAIL verdict") +@then("the code-length condition must NOT override the FAIL verdict") def step_code_length_not_override(context): """Definitive regression test: code length > 10 did not cause auto-pass. diff --git a/features/validation_bypass_issue_10480.feature b/features/validation_bypass_issue_10480.feature index c953220cf..c653ca03d 100644 --- a/features/validation_bypass_issue_10480.feature +++ b/features/validation_bypass_issue_10480.feature @@ -7,30 +7,35 @@ Feature: Validation bypass fix (issue #10480) Given the plan generation coverage module is imported And the validation bypass test helper is ready + @tdd_bug_10480 Scenario: LLM returns FAIL for short code and validation correctly rejects it Given an LLM mock that responds with "FAIL: syntax error" When a PlanGenerationGraph _validate is called with short generated code Then the validation result status should be "FAIL" And the validation message should contain the LLM feedback + @tdd_bug_10480 Scenario: LLM returns FAIL for long code and validation correctly rejects it Given an LLM mock that responds with "FAIL: unsafe pattern detected" When a PlanGenerationGraph _validate is called with longer generated code Then the validation result status should be "FAIL" And no artificial length threshold bypassed the rejection + @tdd_bug_10480 Scenario: LLM returns PASS and validation correctly accepts it Given an LLM mock that responds with "PASS" When a PlanGenerationGraph _validate is called with any generated code Then the validation result status should be "PASS" And the validation message should contain the LLM feedback + @tdd_bug_10480 Scenario: LLM returns FAIL with additional details and validation correctly rejects Given an LLM mock that responds with "FAIL: missing error handling in auth module" When a PlanGenerationGraph _validate is called with generated code Then the validation result status should be "FAIL" And the validation message should preserve the full LLM feedback + @tdd_bug_10480 Scenario: Validation ignores code length and does not auto-pass long code Given an LLM mock that responds with "FAIL: security vulnerability" When a PlanGenerationGraph _validate is called with code longer than 10 characters