Files
cleveragents-core/features/fix_then_revalidate.feature
Luis Mendes 3cf3f1f69e
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 15s
CI / lint (pull_request) Successful in 3m40s
CI / quality (pull_request) Successful in 4m13s
CI / security (pull_request) Successful in 4m16s
CI / typecheck (pull_request) Successful in 4m16s
CI / integration_tests (pull_request) Successful in 9m11s
CI / unit_tests (pull_request) Successful in 9m23s
CI / docker (pull_request) Successful in 1m18s
CI / coverage (pull_request) Successful in 11m48s
CI / status-check (pull_request) Successful in 7s
CI / e2e_tests (pull_request) Successful in 8m24s
CI / lint (push) Successful in 3m18s
CI / build (push) Failing after 13s
CI / typecheck (push) Successful in 3m54s
CI / benchmark-regression (push) Has been skipped
CI / quality (push) Successful in 4m15s
CI / security (push) Successful in 4m39s
CI / integration_tests (push) Successful in 7m21s
CI / unit_tests (push) Successful in 7m44s
CI / docker (push) Successful in 1m9s
CI / e2e_tests (push) Successful in 11m9s
CI / coverage (push) Successful in 13m41s
CI / status-check (push) Successful in 2s
CI / benchmark-regression (pull_request) Successful in 48m49s
CI / benchmark-publish (push) Successful in 26m25s
feat(validation): implement Fix-then-Revalidate orchestration loop for required validations
Implemented FixThenRevalidateOrchestrator with full diagnosis → self-fix →
re-validation → retry limit → strategy revision → user escalation → terminal
failure flow. Added retry counting per validation per plan, configurable retry
limit (default 3), auto_strategy_revision flag support, and validation_fix_history
recording in plan execution metadata.

Review fixes applied (round 1):
- Added try/except around fix_callback and revalidate_callback (spec: validation
  errors treated as required failure regardless of mode)
- Wired optional EventBus for VALIDATION_FIX_ATTEMPTED/SUCCEEDED/EXHAUSTED events
- Fixed total_attempts derivation from fix_history length instead of cumulative
  retry counts
- Updated module docstring to reflect implemented vs caller-responsible steps
- Added exhausted-retry logging on re-invocation
- Replaced bare assert in Robot helper with explicit _check() function
- Added 10 new Behave scenarios covering exception paths, multi-failure, reset,
  field validation, boundary values, event_bus property, and exhausted re-invocation

Review fixes applied (round 2 — PR #711):
- B1+B7: Fixed TOCTOU race in _fix_single_validation; atomic claim-per-iteration
  under RLock; eliminated defaultdict auto-vivification with .get() reads
- B2: Added validation_name mismatch check on revalidate_callback return
- B3: Added fix_description truncation to 2000 chars before FixAttemptRecord
- T1: Switched threading.Lock to threading.RLock for defensive reentrancy
- A1: Added model_validator preventing escalated+terminal_failure both True
- R1: Added max_length=255 to FixAttemptRecord.validation_name
- S7: Fixed misleading docstring about automation_profile integration
- D4: Fixed timestamp field description to specify UTC
- D1: Renamed misleading benchmark time_fix_after_two_retries
- Added 5 new Behave scenarios for truncation, model constraints, name mismatch

ISSUES CLOSED: #583
2026-03-24 22:01:48 +00:00

62 lines
3.1 KiB
Gherkin

Feature: Fix-then-Revalidate orchestration loop
As an execution actor
I want a bounded retry loop that diagnoses, fixes, and re-validates
So that recoverable validation failures are resolved automatically
Background:
Given a fix-revalidate orchestrator with max_retries 3
# Happy path: first fix attempt resolves the validation failure
Scenario: Fix succeeds on first attempt
Given a required validation "check-syntax" that fails initially
And a fix callback that always succeeds
And a revalidate callback that passes after 1 fix
When the fix-revalidate loop runs for plan "PLAN001"
Then the fix-revalidate result should have validation_attempts 1
And the fix-revalidate result should have final_passed True
And the fix-revalidate result should not be escalated
And the fix-revalidate result should not be terminal failure
# Multiple attempts needed before the fix works
Scenario: Fix succeeds after multiple attempts
Given a required validation "check-lint" that fails initially
And a fix callback that always succeeds
And a revalidate callback that passes after 2 fixes
When the fix-revalidate loop runs for plan "PLAN002"
Then the fix-revalidate result should have validation_attempts 2
And the fix-revalidate result should have final_passed True
And the validation fix history should have 2 records
# Retry limit reached, auto_strategy_revision triggers escalation
Scenario: Retry limit exhausted with strategy revision
Given a fix-revalidate orchestrator with auto_strategy_revision enabled
And a required validation "check-security" that fails initially
And a fix callback that always succeeds
And a revalidate callback that never passes
When the fix-revalidate loop runs for plan "PLAN003"
Then the fix-revalidate result should have final_passed False
And the fix-revalidate result should be escalated
And the fix-revalidate result should not be terminal failure
And the fix-revalidate result should have validation_attempts 3
# Terminal failure when all attempts fail and no strategy revision
Scenario: Terminal failure when all attempts fail
Given a required validation "check-deploy" that fails initially
And a fix callback that always succeeds
And a revalidate callback that never passes
When the fix-revalidate loop runs for plan "PLAN004"
Then the fix-revalidate result should have final_passed False
And the fix-revalidate result should need user escalation
And the fix-revalidate result should not be terminal failure
And the validation fix history should have 3 records
# Informational validations do not trigger the fix loop
Scenario: Informational validations do not trigger fix loop
Given an informational validation "check-style" that fails
And a fix callback that always succeeds
And a revalidate callback that passes after 1 fix
When the fix-revalidate loop runs for plan "PLAN005"
Then the fix-revalidate result should have validation_attempts 0
And the fix-revalidate result should have final_passed True
And the validation fix history should have 0 records