CoreRasurae
332 lines
16 KiB
Gherkin
332 lines
16 KiB
Gherkin
Feature: Error recovery patterns and CLI hints
|
|
As a plan executor
|
|
I want structured error recovery with retry logic and actionable hints
|
|
So that failed plans can be recovered automatically or with clear guidance
|
|
|
|
Background:
|
|
Given an error recovery test environment
|
|
|
|
# -- Error classification ------------------------------------------------
|
|
|
|
Scenario: Classify transient error by message
|
|
When I classify the error message "Connection timed out after 30s"
|
|
Then the classified error category should be "transient"
|
|
|
|
Scenario: Classify validation error by message
|
|
When I classify the error message "Schema validation failed for field X"
|
|
Then the classified error category should be "validation"
|
|
|
|
Scenario: Classify configuration error by message
|
|
When I classify the error message "Missing configuration for provider"
|
|
Then the classified error category should be "configuration"
|
|
|
|
Scenario: Classify merge conflict error by message
|
|
When I classify the error message "Merge conflict in src/main.py"
|
|
Then the classified error category should be "merge_conflict"
|
|
|
|
Scenario: Classify authentication error by message
|
|
When I classify the error message "Authentication failed: invalid token"
|
|
Then the classified error category should be "authentication"
|
|
|
|
Scenario: Classify provider error by message
|
|
When I classify the error message "Model not available: gpt-4-turbo"
|
|
Then the classified error category should be "provider"
|
|
|
|
Scenario: Classify internal error by message
|
|
When I classify the error message "Unexpected internal error in pipeline"
|
|
Then the classified error category should be "internal"
|
|
|
|
Scenario: Classify unknown error by message
|
|
When I classify the error message "Something completely novel happened"
|
|
Then the classified error category should be "unknown"
|
|
|
|
Scenario: Classify error by exception type
|
|
When I classify error text "failed" using exception type "RateLimitError"
|
|
Then the classified error category should be "transient"
|
|
|
|
Scenario: Classify error by exception type overrides message
|
|
When I classify error text "validation issue" using exception type "NetworkError"
|
|
Then the classified error category should be "transient"
|
|
|
|
# -- Recovery hints ------------------------------------------------------
|
|
|
|
Scenario: Transient error recovery hints include retry command
|
|
When I get recovery hints for category "transient" and plan "01PLAN001"
|
|
Then the first recovery hint action should be "retry"
|
|
And the first recovery hint should have CLI command containing "01PLAN001"
|
|
|
|
Scenario: Validation error recovery hints include correct command
|
|
When I get recovery hints for category "validation" and plan "01PLAN001"
|
|
Then the first recovery hint action should be "correct"
|
|
And the first recovery hint should have CLI command containing "correct"
|
|
|
|
Scenario: Configuration error recovery hints suggest cancel
|
|
When I get recovery hints for category "configuration" and plan "01PLAN001"
|
|
Then the first recovery hint action should be "cancel"
|
|
|
|
Scenario: Merge conflict hints suggest revert to strategize
|
|
When I get recovery hints for category "merge_conflict" and plan "01PLAN001"
|
|
Then the first recovery hint action should be "revert_strategize"
|
|
|
|
# -- Error recording -----------------------------------------------------
|
|
|
|
Scenario: Record error creates error record with metadata
|
|
Given a plan in errored state for error recovery
|
|
When I record an error with phase "execute" and message "Timeout talking to provider"
|
|
Then the recorded error should have category "transient"
|
|
And the recorded error should have phase "execute"
|
|
And the recorded error should have recovery hints
|
|
And the error history should have 1 record
|
|
|
|
Scenario: Record error with explicit category
|
|
Given a plan in errored state for error recovery
|
|
When I record a categorized error with phase "apply" and message "custom issue" and category "merge_conflict"
|
|
Then the recorded error should have category "merge_conflict"
|
|
|
|
Scenario: Record multiple errors tracks retry count
|
|
Given a plan in errored state for error recovery
|
|
When I record an error with phase "execute" and message "Timeout 1"
|
|
And I record an error with phase "execute" and message "Timeout 2"
|
|
Then the error history should have 2 records
|
|
And the latest error retry count should be 1
|
|
|
|
# -- Retry policy --------------------------------------------------------
|
|
|
|
Scenario: Transient error with auto-retry threshold 0 is retriable
|
|
Given a plan in errored state for error recovery
|
|
And the auto retry threshold is 0.0
|
|
When I record an error with phase "execute" and message "Connection timed out"
|
|
Then the service should recommend retry
|
|
|
|
Scenario: Transient error with auto-retry threshold 1 requires escalation
|
|
Given a plan in errored state for error recovery
|
|
And the auto retry threshold is 1.0
|
|
When I record an error with phase "execute" and message "Connection timed out"
|
|
Then the service should recommend escalation
|
|
|
|
Scenario: Configuration error is never retriable
|
|
Given a plan in errored state for error recovery
|
|
And the auto retry threshold is 0.0
|
|
When I record an error with phase "strategize" and message "Missing configuration"
|
|
Then the service should recommend escalation
|
|
|
|
Scenario: Retry exhaustion after max retries
|
|
Given a plan in errored state for error recovery
|
|
And the max retries is 2
|
|
When I record an error with phase "execute" and message "Timeout 1"
|
|
And I record an error with phase "execute" and message "Timeout 2"
|
|
And I record an error with phase "execute" and message "Timeout 3"
|
|
Then the latest error should have retries exhausted
|
|
And the service should recommend escalation
|
|
|
|
# -- Error history -------------------------------------------------------
|
|
|
|
Scenario: Error history shows category breakdown
|
|
Given a plan in errored state for error recovery
|
|
When I record an error with phase "execute" and message "Timeout occurred"
|
|
And I record an error with phase "apply" and message "Merge conflict in file.py"
|
|
Then the error history category breakdown should include "transient"
|
|
And the error history category breakdown should include "merge_conflict"
|
|
|
|
# -- CLI output formatting -----------------------------------------------
|
|
|
|
Scenario: Format error output as plain text
|
|
Given a plan in errored state for error recovery
|
|
When I record an error with phase "execute" and message "Provider timed out"
|
|
And I format the error output as "plain"
|
|
Then the error recovery output should contain "transient"
|
|
And the error recovery output should contain "Provider timed out"
|
|
And the error recovery output should contain "execute"
|
|
|
|
Scenario: Format error output as JSON
|
|
Given a plan in errored state for error recovery
|
|
When I record an error with phase "execute" and message "Provider timed out"
|
|
And I format the error output as "json"
|
|
Then the error recovery output should contain "transient"
|
|
And the error recovery output should contain "plan_id"
|
|
|
|
Scenario: Format empty error history
|
|
When I format error output as "plain" for separate plan "01EMPTYPLAN00000000001"
|
|
Then the error recovery output should contain "No errors recorded"
|
|
|
|
# -- ErrorRecord model checks --------------------------------------------
|
|
|
|
Scenario: ErrorRecord is_retriable for transient errors
|
|
Given an error record with category "transient" and retry count 0 and max retries 3
|
|
Then the error record should be retriable
|
|
|
|
Scenario: ErrorRecord is not retriable when exhausted
|
|
Given an error record with category "transient" and retry count 3 and max retries 3
|
|
Then the error record should not be retriable
|
|
And the error record should have retries exhausted
|
|
|
|
Scenario: ErrorRecord is not retriable for configuration errors
|
|
Given an error record with category "configuration" and retry count 0 and max retries 3
|
|
Then the error record should not be retriable
|
|
|
|
Scenario: ErrorRecord to_cli_dict has expected fields
|
|
Given an error recovery test environment
|
|
Given an error record with category "transient" and retry count 1 and max retries 3
|
|
Then the error record CLI dict should have key "error_id"
|
|
And the error record CLI dict should have key "category"
|
|
And the error record CLI dict should have key "is_retriable"
|
|
|
|
# -- Exception-based recording -------------------------------------------
|
|
|
|
Scenario: Record error with actual exception captures stack trace
|
|
Given a plan in errored state for error recovery
|
|
When I record an error with exception for phase "execute" and message "timeout"
|
|
Then the recorded error should have a stack summary
|
|
|
|
# -- Edge cases -----------------------------------------------------------
|
|
|
|
Scenario: should_retry returns false when no errors exist
|
|
Given a plan in errored state for error recovery
|
|
Then the service should not recommend retry for a clean plan
|
|
|
|
Scenario: should_escalate returns false when no errors exist
|
|
Given a plan in errored state for error recovery
|
|
Then the service should not recommend escalation for a clean plan
|
|
|
|
Scenario: Format recovery hints for CLI
|
|
Given a plan in errored state for error recovery
|
|
When I record an error with phase "execute" and message "Connection timed out"
|
|
Then the CLI recovery hints should contain "transient"
|
|
|
|
# -- ErrorRecoveryPolicy formatting --------------------------------------
|
|
|
|
Scenario: Policy formats output with actor and tool info
|
|
Given a plan in errored state for error recovery
|
|
When I record an actor-error in phase "execute" with message "Timeout" actor "test-actor" tool "test-tool"
|
|
And I format the error output as "plain"
|
|
Then the error recovery output should contain "test-actor"
|
|
And the error recovery output should contain "test-tool"
|
|
|
|
Scenario: Policy format shows automatic retry available
|
|
Given a plan in errored state for error recovery
|
|
And the auto retry threshold is 0.0
|
|
When I record an error with phase "execute" and message "Connection timed out"
|
|
And I format the error output as "plain"
|
|
Then the error recovery output should contain "Automatic retry available"
|
|
|
|
Scenario: Policy format shows human intervention required
|
|
Given a plan in errored state for error recovery
|
|
And the auto retry threshold is 1.0
|
|
When I record an error with phase "execute" and message "Connection timed out"
|
|
And I format the error output as "plain"
|
|
Then the error recovery output should contain "Human intervention required"
|
|
|
|
Scenario: Policy format shows retries exhausted message
|
|
Given a plan in errored state for error recovery
|
|
And the max retries is 1
|
|
When I record an error with phase "execute" and message "Timeout 1"
|
|
And I record an error with phase "execute" and message "Timeout 2"
|
|
And I format the error output as "plain"
|
|
Then the error recovery output should contain "retry attempts exhausted"
|
|
|
|
# -- ErrorHistory with no retriable errors --------------------------------
|
|
|
|
Scenario: Error history has_retriable is false when only non-retriable
|
|
Given a plan in errored state for error recovery
|
|
When I record an error with phase "strategize" and message "Missing configuration"
|
|
Then the error history should not have retriable errors
|
|
|
|
# -- ErrorRecord with actor and tool for CLI dict -------------------------
|
|
|
|
Scenario: ErrorRecord to_cli_dict includes actor and tool when present
|
|
Given an error record with actor "my-actor" and tool "my-tool"
|
|
Then the error record CLI dict should have key "actor"
|
|
And the error record CLI dict should have key "tool_call"
|
|
|
|
# -- Recovery hints for empty plan ----------------------------------------
|
|
|
|
Scenario: Format recovery hints returns empty for plan with no errors
|
|
Given a plan in errored state for error recovery
|
|
Then the CLI recovery hints for a clean plan should be empty
|
|
|
|
# -- ErrorRecord to_cli_dict with stack summary ---------------------------
|
|
|
|
Scenario: ErrorRecord to_cli_dict includes stack summary
|
|
Given an error record with stack summary "File test.py line 10"
|
|
Then the error record CLI dict should have key "stack_summary"
|
|
|
|
# -- ErrorRecoveryPolicy should_retry on non-retriable --------------------
|
|
|
|
Scenario: Policy should_retry returns false for non-retriable category
|
|
Given a policy with threshold 0.0 and max retries 3
|
|
And an error record with category "configuration" and retry count 0 and max retries 3
|
|
Then the policy should not recommend retry
|
|
|
|
Scenario: Policy should_retry returns false for exhausted retries
|
|
Given a policy with threshold 0.0 and max retries 3
|
|
And an error record with category "transient" and retry count 3 and max retries 3
|
|
Then the policy should not recommend retry
|
|
|
|
# -- T1: Persistence failure path ----------------------------------------
|
|
|
|
Scenario: Persist error metadata handles lifecycle errors gracefully
|
|
Given a plan in errored state for error recovery
|
|
And the lifecycle service update_error_details raises an error
|
|
When I record an error with phase "execute" and message "transient failure"
|
|
Then the error should still be recorded in memory
|
|
|
|
# -- T2: Branch coverage for policy/format --------------------------------
|
|
|
|
Scenario: Policy should_escalate returns true for exhausted retries
|
|
Given a policy with threshold 0.0 and max retries 3
|
|
And an error record with category "transient" and retry count 3 and max retries 3
|
|
Then the policy should recommend escalation
|
|
|
|
Scenario: Policy should_escalate returns true for non-retriable category
|
|
Given a policy with threshold 0.0 and max retries 3
|
|
And an error record with category "configuration" and retry count 0 and max retries 3
|
|
Then the policy should recommend escalation
|
|
|
|
Scenario: Format recovery output includes hints with CLI commands
|
|
Given a policy with threshold 0.0 and max retries 3
|
|
And a hinted error record with category "transient" and retry count 0 and max retries 3
|
|
Then the formatted recovery output should contain CLI command text
|
|
|
|
Scenario: Format recovery output shows exhausted message
|
|
Given a policy with threshold 0.0 and max retries 3
|
|
And a hinted error record with category "transient" and retry count 3 and max retries 3
|
|
Then the formatted recovery output should contain "exhausted"
|
|
|
|
Scenario: Format recovery output shows automatic retry available
|
|
Given a policy with threshold 0.0 and max retries 3
|
|
And an error record with category "transient" and retry count 0 and max retries 3
|
|
Then the formatted recovery output should contain "Automatic retry"
|
|
|
|
Scenario: Policy should_retry returns false when threshold is 1.0
|
|
Given a policy with threshold 1.0 and max retries 3
|
|
And an error record with category "transient" and retry count 0 and max retries 3
|
|
Then the policy should not recommend retry
|
|
|
|
# -- T4: Integration test with executor retry loop ------------------------
|
|
|
|
Scenario: Executor retries execute phase on recoverable errors
|
|
Given a plan executor with error recovery service
|
|
And the execute actor fails twice then succeeds
|
|
When I run the executor for the plan
|
|
Then the execute should complete after retries
|
|
And the error recovery service should have recorded 2 errors
|
|
|
|
Scenario: Executor fails after exhausting retries
|
|
Given a plan executor with error recovery service max retries 2
|
|
And the execute actor always fails with "persistent failure"
|
|
When I run the executor expecting failure
|
|
Then the lifecycle should record the plan as failed
|
|
And the error recovery service should have recorded 3 errors
|
|
|
|
# -- B5: Bare conflict no longer matches MERGE_CONFLICT -------------------
|
|
|
|
Scenario: Classify resource conflict does not match merge_conflict
|
|
When I classify the error message "resource conflict detected"
|
|
Then the classified error category should not be "merge_conflict"
|
|
|
|
# -- S1: Format string safe substitution ----------------------------------
|
|
|
|
Scenario: Recovery hints handle plan_id with braces safely
|
|
When I get recovery hints for category "transient" with plan_id "{evil_id}"
|
|
Then the hints should contain "{evil_id}" in CLI commands
|