3cf3f1f69e
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 15s
CI / lint (pull_request) Successful in 3m40s
CI / quality (pull_request) Successful in 4m13s
CI / security (pull_request) Successful in 4m16s
CI / typecheck (pull_request) Successful in 4m16s
CI / integration_tests (pull_request) Successful in 9m11s
CI / unit_tests (pull_request) Successful in 9m23s
CI / docker (pull_request) Successful in 1m18s
CI / coverage (pull_request) Successful in 11m48s
CI / status-check (pull_request) Successful in 7s
CI / e2e_tests (pull_request) Successful in 8m24s
CI / lint (push) Successful in 3m18s
CI / build (push) Failing after 13s
CI / typecheck (push) Successful in 3m54s
CI / benchmark-regression (push) Has been skipped
CI / quality (push) Successful in 4m15s
CI / security (push) Successful in 4m39s
CI / integration_tests (push) Successful in 7m21s
CI / unit_tests (push) Successful in 7m44s
CI / docker (push) Successful in 1m9s
CI / e2e_tests (push) Successful in 11m9s
CI / coverage (push) Successful in 13m41s
CI / status-check (push) Successful in 2s
CI / benchmark-regression (pull_request) Successful in 48m49s
CI / benchmark-publish (push) Successful in 26m25s
Implemented FixThenRevalidateOrchestrator with full diagnosis → self-fix → re-validation → retry limit → strategy revision → user escalation → terminal failure flow. Added retry counting per validation per plan, configurable retry limit (default 3), auto_strategy_revision flag support, and validation_fix_history recording in plan execution metadata. Review fixes applied (round 1): - Added try/except around fix_callback and revalidate_callback (spec: validation errors treated as required failure regardless of mode) - Wired optional EventBus for VALIDATION_FIX_ATTEMPTED/SUCCEEDED/EXHAUSTED events - Fixed total_attempts derivation from fix_history length instead of cumulative retry counts - Updated module docstring to reflect implemented vs caller-responsible steps - Added exhausted-retry logging on re-invocation - Replaced bare assert in Robot helper with explicit _check() function - Added 10 new Behave scenarios covering exception paths, multi-failure, reset, field validation, boundary values, event_bus property, and exhausted re-invocation Review fixes applied (round 2 — PR #711): - B1+B7: Fixed TOCTOU race in _fix_single_validation; atomic claim-per-iteration under RLock; eliminated defaultdict auto-vivification with .get() reads - B2: Added validation_name mismatch check on revalidate_callback return - B3: Added fix_description truncation to 2000 chars before FixAttemptRecord - T1: Switched threading.Lock to threading.RLock for defensive reentrancy - A1: Added model_validator preventing escalated+terminal_failure both True - R1: Added max_length=255 to FixAttemptRecord.validation_name - S7: Fixed misleading docstring about automation_profile integration - D4: Fixed timestamp field description to specify UTC - D1: Renamed misleading benchmark time_fix_after_two_retries - Added 5 new Behave scenarios for truncation, model constraints, name mismatch ISSUES CLOSED: #583
62 lines
3.1 KiB
Gherkin
62 lines
3.1 KiB
Gherkin
Feature: Fix-then-Revalidate orchestration loop
|
|
As an execution actor
|
|
I want a bounded retry loop that diagnoses, fixes, and re-validates
|
|
So that recoverable validation failures are resolved automatically
|
|
|
|
Background:
|
|
Given a fix-revalidate orchestrator with max_retries 3
|
|
|
|
# Happy path: first fix attempt resolves the validation failure
|
|
Scenario: Fix succeeds on first attempt
|
|
Given a required validation "check-syntax" that fails initially
|
|
And a fix callback that always succeeds
|
|
And a revalidate callback that passes after 1 fix
|
|
When the fix-revalidate loop runs for plan "PLAN001"
|
|
Then the fix-revalidate result should have validation_attempts 1
|
|
And the fix-revalidate result should have final_passed True
|
|
And the fix-revalidate result should not be escalated
|
|
And the fix-revalidate result should not be terminal failure
|
|
|
|
# Multiple attempts needed before the fix works
|
|
Scenario: Fix succeeds after multiple attempts
|
|
Given a required validation "check-lint" that fails initially
|
|
And a fix callback that always succeeds
|
|
And a revalidate callback that passes after 2 fixes
|
|
When the fix-revalidate loop runs for plan "PLAN002"
|
|
Then the fix-revalidate result should have validation_attempts 2
|
|
And the fix-revalidate result should have final_passed True
|
|
And the validation fix history should have 2 records
|
|
|
|
# Retry limit reached, auto_strategy_revision triggers escalation
|
|
Scenario: Retry limit exhausted with strategy revision
|
|
Given a fix-revalidate orchestrator with auto_strategy_revision enabled
|
|
And a required validation "check-security" that fails initially
|
|
And a fix callback that always succeeds
|
|
And a revalidate callback that never passes
|
|
When the fix-revalidate loop runs for plan "PLAN003"
|
|
Then the fix-revalidate result should have final_passed False
|
|
And the fix-revalidate result should be escalated
|
|
And the fix-revalidate result should not be terminal failure
|
|
And the fix-revalidate result should have validation_attempts 3
|
|
|
|
# Terminal failure when all attempts fail and no strategy revision
|
|
Scenario: Terminal failure when all attempts fail
|
|
Given a required validation "check-deploy" that fails initially
|
|
And a fix callback that always succeeds
|
|
And a revalidate callback that never passes
|
|
When the fix-revalidate loop runs for plan "PLAN004"
|
|
Then the fix-revalidate result should have final_passed False
|
|
And the fix-revalidate result should need user escalation
|
|
And the fix-revalidate result should not be terminal failure
|
|
And the validation fix history should have 3 records
|
|
|
|
# Informational validations do not trigger the fix loop
|
|
Scenario: Informational validations do not trigger fix loop
|
|
Given an informational validation "check-style" that fails
|
|
And a fix callback that always succeeds
|
|
And a revalidate callback that passes after 1 fix
|
|
When the fix-revalidate loop runs for plan "PLAN005"
|
|
Then the fix-revalidate result should have validation_attempts 0
|
|
And the fix-revalidate result should have final_passed True
|
|
And the validation fix history should have 0 records
|