Luis Mendes
3cf3f1f69e
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 15s
CI / lint (pull_request) Successful in 3m40s
CI / quality (pull_request) Successful in 4m13s
CI / security (pull_request) Successful in 4m16s
CI / typecheck (pull_request) Successful in 4m16s
CI / integration_tests (pull_request) Successful in 9m11s
CI / unit_tests (pull_request) Successful in 9m23s
CI / docker (pull_request) Successful in 1m18s
CI / coverage (pull_request) Successful in 11m48s
CI / status-check (pull_request) Successful in 7s
CI / e2e_tests (pull_request) Successful in 8m24s
CI / lint (push) Successful in 3m18s
CI / build (push) Failing after 13s
CI / typecheck (push) Successful in 3m54s
CI / benchmark-regression (push) Has been skipped
CI / quality (push) Successful in 4m15s
CI / security (push) Successful in 4m39s
CI / integration_tests (push) Successful in 7m21s
CI / unit_tests (push) Successful in 7m44s
CI / docker (push) Successful in 1m9s
CI / e2e_tests (push) Successful in 11m9s
CI / coverage (push) Successful in 13m41s
CI / status-check (push) Successful in 2s
CI / benchmark-regression (pull_request) Successful in 48m49s
CI / benchmark-publish (push) Successful in 26m25s
Implemented FixThenRevalidateOrchestrator with full diagnosis → self-fix → re-validation → retry limit → strategy revision → user escalation → terminal failure flow. Added retry counting per validation per plan, configurable retry limit (default 3), auto_strategy_revision flag support, and validation_fix_history recording in plan execution metadata. Review fixes applied (round 1): - Added try/except around fix_callback and revalidate_callback (spec: validation errors treated as required failure regardless of mode) - Wired optional EventBus for VALIDATION_FIX_ATTEMPTED/SUCCEEDED/EXHAUSTED events - Fixed total_attempts derivation from fix_history length instead of cumulative retry counts - Updated module docstring to reflect implemented vs caller-responsible steps - Added exhausted-retry logging on re-invocation - Replaced bare assert in Robot helper with explicit _check() function - Added 10 new Behave scenarios covering exception paths, multi-failure, reset, field validation, boundary values, event_bus property, and exhausted re-invocation Review fixes applied (round 2 — PR #711): - B1+B7: Fixed TOCTOU race in _fix_single_validation; atomic claim-per-iteration under RLock; eliminated defaultdict auto-vivification with .get() reads - B2: Added validation_name mismatch check on revalidate_callback return - B3: Added fix_description truncation to 2000 chars before FixAttemptRecord - T1: Switched threading.Lock to threading.RLock for defensive reentrancy - A1: Added model_validator preventing escalated+terminal_failure both True - R1: Added max_length=255 to FixAttemptRecord.validation_name - S7: Fixed misleading docstring about automation_profile integration - D4: Fixed timestamp field description to specify UTC - D1: Renamed misleading benchmark time_fix_after_two_retries - Added 5 new Behave scenarios for truncation, model constraints, name mismatch ISSUES CLOSED: #583
60 lines
2.9 KiB
Plaintext
60 lines
2.9 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration tests for Fix-then-Revalidate orchestration loop
|
|
Library Process
|
|
Library OperatingSystem
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER_SCRIPT} robot/helper_fix_then_revalidate.py
|
|
|
|
*** Test Cases ***
|
|
Fix Then Revalidate Models Are Valid
|
|
[Documentation] Verify FixAttemptRecord and FixThenRevalidateResult Pydantic models
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} models cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} models-ok
|
|
|
|
Fix Succeeds On First Attempt
|
|
[Documentation] Fix loop resolves validation on first fix attempt
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} fix_first cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} fix-first-ok
|
|
|
|
Fix Succeeds After Multiple Attempts
|
|
[Documentation] Fix loop resolves validation after multiple fix attempts
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} fix_multiple cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} fix-multiple-ok
|
|
|
|
Terminal Failure When All Attempts Exhausted
|
|
[Documentation] Plan reaches terminal failure when retries exhausted
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} terminal_failure cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} terminal-failure-ok
|
|
|
|
Escalation To Strategy Revision
|
|
[Documentation] Loop escalates to strategy revision when auto_strategy_revision enabled
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} escalation cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} escalation-ok
|
|
|
|
Informational Validations Skipped
|
|
[Documentation] Informational validations do not trigger fix loop
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} informational_skipped cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} informational-skipped-ok
|
|
|
|
Retry Counting Per Validation Per Plan
|
|
[Documentation] Verify retry counting and reset per validation per plan
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} retry_counting cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} retry-counting-ok
|
|
|
|
Unknown Command Returns Error
|
|
[Documentation] Running with unknown command exits with code 1
|
|
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} nonexistent_cmd cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 1
|
|
Should Contain ${result.stderr} Unknown command
|