HAL9000
6141d2a36d
CI / lint (pull_request) Successful in 42s
CI / quality (pull_request) Successful in 1m2s
CI / security (pull_request) Successful in 1m12s
CI / typecheck (pull_request) Successful in 1m19s
CI / build (pull_request) Successful in 47s
CI / helm (pull_request) Successful in 51s
CI / push-validation (pull_request) Successful in 25s
CI / unit_tests (pull_request) Successful in 5m57s
CI / docker (pull_request) Successful in 1m40s
CI / integration_tests (pull_request) Successful in 9m35s
CI / coverage (pull_request) Successful in 12m42s
CI / status-check (pull_request) Successful in 4s
- features/steps/tdd_plan_generation_validate_steps.py: drop duplicate
@given/@then registrations (live in plan_generation_langgraph_coverage_
steps.py; redefinition caused AmbiguousStep errors crashing all 8
behave-parallel workers); have the @when step feed the docstring as the
FakeListLLM response so each scenario tests _validate()'s parsing of a
specific PASS/FAIL signal rather than the input code.
- features/tdd_plan_generation_validate_logic.feature: add the required
@tdd_issue tag alongside @tdd_issue_10746 (enforced by
features/environment.py); tighten scenario 4 wording to remove the
reference to the obsolete length guard.
- robot/plan_generation_graph.robot: drop assertion for handle_retry node
(retry is a conditional edge, not a fifth node) and update node-count
check from 5 to 4; update Should Retry test to assert _should_retry does
NOT mutate state (read-only contract for LangGraph conditional edges).
- src/cleveragents/agents/graphs/plan_generation.py:
* _validate: persist retry_count increment in return dict (FAIL path and
exception path) so LangGraph propagates it through the state graph.
* _should_retry: remove state mutation (conditional-edge functions are
read-only in LangGraph; mutations were silently dropped, causing
retry_count to remain 0 forever and the graph to loop infinitely).
Adjust comparison to retry_count <= max_retries because _validate has
already incremented before _should_retry runs.
* __init__: add max_context_files parameter (default 5, validated > 0)
and wire it into _format_context_summary in place of the hardcoded 5,
implementing the configurable-limits contract tested by
features/agent_configurable_limits.feature.
ISSUES CLOSED: #10746
52 lines
2.2 KiB
Gherkin
52 lines
2.2 KiB
Gherkin
Feature: TDD — PlanGenerationGraph validation logic
|
|
As a developer fixing a validation regression
|
|
I want the PlanGenerationGraph._validate() logic to respect explicit LLM FAIL responses
|
|
So that generated code is rejected when the LLM indicates failure, regardless of code length
|
|
|
|
@tdd_issue @tdd_issue_10746
|
|
Scenario: FAIL response is respected for code longer than 10 characters
|
|
Given I have a langgraph PlanGenerationGraph instance
|
|
When I execute the langgraph validate node with generated code:
|
|
"""
|
|
FAIL: Security vulnerability found in generated code that exceeds length threshold
|
|
"""
|
|
Then the langgraph validation status should be "FAIL"
|
|
|
|
@tdd_issue @tdd_issue_10746
|
|
Scenario: PASS response is accepted when no FAIL present
|
|
Given I have a langgraph PlanGenerationGraph instance
|
|
When I execute the langgraph validate node with generated code:
|
|
"""
|
|
PASS: Code appears correct and follows best practices
|
|
"""
|
|
Then the langgraph validation status should be "PASS"
|
|
|
|
@tdd_issue @tdd_issue_10746
|
|
Scenario: Mixed PASS and FAIL prefers FAIL
|
|
Given I have a langgraph PlanGenerationGraph instance
|
|
When I execute the langgraph validate node with generated code:
|
|
"""
|
|
PASS: Looks okay
|
|
FAIL: But contains a logic bug in edge case handling
|
|
"""
|
|
Then the langgraph validation status should be "FAIL"
|
|
|
|
@tdd_issue @tdd_issue_10746
|
|
Scenario: No explicit PASS or FAIL keyword should fail
|
|
Given I have a langgraph PlanGenerationGraph instance
|
|
When I execute the langgraph validate node with generated code:
|
|
"""
|
|
The validation response contains only neutral observations about the code
|
|
structure and design choices with no verdict keyword present.
|
|
"""
|
|
Then the langgraph validation status should be "FAIL"
|
|
|
|
@tdd_issue @tdd_issue_10746
|
|
Scenario: PASS keyword in mixed-case is accepted (case-insensitive)
|
|
Given I have a langgraph PlanGenerationGraph instance
|
|
When I execute the langgraph validate node with generated code:
|
|
"""
|
|
pass: minimal checks passed, nothing critical found
|
|
"""
|
|
Then the langgraph validation status should be "PASS"
|