cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

fix(agents/graphs/plan_generation): _validate always passes for code longer than 10 characters #10731

Closed
HAL9000 wants to merge 2 commits from fix/plan-generation-validate-logic into master
pull from: fix/plan-generation-validate-logic
merge into: cleveragents:master
Conversation 5 Commits 2 Files Changed 4
+206 -4

2 Commits

Author SHA1 Message Date
HAL9000 d89a1c7098 fix(agents/graphs/plan_generation): _validate always passes for code longer than 10 characters
CI / push-validation (pull_request) Successful in 39s
Details
CI / helm (pull_request) Successful in 56s
Details
CI / lint (pull_request) Successful in 4m26s
Details
CI / build (pull_request) Successful in 4m28s
Details
CI / quality (pull_request) Successful in 4m47s
Details
CI / typecheck (pull_request) Successful in 5m15s
Details
CI / security (pull_request) Successful in 5m28s
Details
CI / unit_tests (pull_request) Failing after 6m5s
Details
CI / docker (pull_request) Has been skipped
Details
CI / e2e_tests (pull_request) Successful in 7m31s
Details
CI / integration_tests (pull_request) Successful in 8m17s
Details
CI / coverage (pull_request) Successful in 19m2s
Details
CI / status-check (pull_request) Failing after 5s
Details 
Applied a Ruff format fix in features/steps/tdd_plan_generation_validate_logic_steps.py by merging two f-string lines into one.

Updated two integration tests in robot/plan_generation_graph.robot to use FakeListLLM(responses=['PASS: code looks good']*10) instead of FakeListLLM(responses=['test']*3). This is necessary because the validation logic fix makes 'test' responses fail validation (they don't contain 'PASS'), causing the workflow to retry indefinitely. The tests now provide passing responses to reflect the new behavior.
 2026-04-19 12:26:29 +00:00
HAL9000 cab2646543 fix(agents/graphs/plan_generation): _validate always passes for code longer than 10 characters
CI / helm (pull_request) Successful in 41s
Details
CI / lint (pull_request) Failing after 1m34s
Details
CI / build (pull_request) Successful in 3m59s
Details
CI / quality (pull_request) Successful in 4m40s
Details
CI / typecheck (pull_request) Successful in 4m57s
Details
CI / security (pull_request) Successful in 5m8s
Details
CI / coverage (pull_request) Has been skipped
Details
CI / integration_tests (pull_request) Failing after 7m20s
Details
CI / e2e_tests (pull_request) Successful in 7m32s
Details
CI / unit_tests (pull_request) Successful in 11m37s
Details
CI / docker (pull_request) Has been skipped
Details
CI / push-validation (pull_request) Successful in 23s
Details
CI / status-check (pull_request) Failing after 3s
Details 
Fix PlanGenerationGraph._validate logic by removing the erroneous
or len(all_code) > 10 condition that caused validation to always pass
for any generated code longer than 10 characters, completely bypassing
the LLM validation response.

The fix changes the is_valid calculation from:
  is_valid = 'PASS' in validation.upper() or len(all_code) > 10
to:
  is_valid = 'PASS' in validation.upper() and 'FAIL' not in validation.upper()

This ensures the LLM's judgment is respected regardless of code length,
restoring the retry logic and preventing invalid/broken code from passing
validation.

Also adds TDD tests (feature file and steps) for issue #10477 as the
TDD counterpart, with @tdd_issue and @tdd_issue_10477 tags.

Closes #10480
 2026-04-19 07:19:06 +00:00