HAL9000
e8e76702a9
Remove the len(all_code) > 10 fallback in the _validate method that was overriding the LLM validation response. Previously, any code longer than 10 characters would cause validation to automatically pass regardless of the LLM's assessment, making the validation check ineffective. The fix ensures validation status is determined solely by whether the LLM response contains 'PASS', making the validation meaningful. A regression test was added to verify that FAIL/REJECTED LLM responses are properly handled even for long code blocks. ISSUES CLOSED: #10746
26 lines
1.2 KiB
Gherkin
26 lines
1.2 KiB
Gherkin
Feature: Plan Generation Validation Fix
|
|
Regression test for bug where _validate always passed for code longer than 10 characters.
|
|
|
|
Scenario: Validation properly fails when LLM response contains FAIL and code is long
|
|
Given a PlanGenerationGraph instance
|
|
And generated code longer than 10 characters
|
|
And the LLM validation response is "FAIL: issues found"
|
|
When the validation node runs
|
|
Then the validation status should be "FAIL"
|
|
And the bug where length over 10 characters forced PASS should be fixed
|
|
|
|
Scenario: Validation properly fails when LLM response contains REJECTED
|
|
Given a PlanGenerationGraph instance
|
|
And generated code longer than 10 characters
|
|
And the LLM validation response is "REJECTED: unsafe patterns detected"
|
|
When the validation node runs
|
|
Then the validation status should be "FAIL"
|
|
And the validation should respect LLM rejection regardless of code length
|
|
|
|
Scenario: Validation properly passes when LLM response contains PASS
|
|
Given a PlanGenerationGraph instance
|
|
And generated code longer than 10 characters
|
|
And the LLM validation response is "PASS: all checks successful"
|
|
When the validation node runs
|
|
Then the validation status should be "PASS"
|