# Error Recovery Reference This document describes the error recovery patterns, error classification, recovery hint system, and retry policy logic added in D1b.recovery. ## Error Categories Errors are classified into categories that determine recovery behaviour: | Category | Retriable | Description | |----------|-----------|-------------| | `transient` | Yes | Temporary failures (network, timeout, rate-limit) | | `validation` | Yes | Validation failures during Execute or Apply | | `configuration` | No | Bad config, missing required fields | | `resource` | No | Missing or conflicting resources | | `merge_conflict` | Yes | Sandbox merge failures | | `authentication` | No | Auth/authz failures | | `provider` | Yes | LLM provider errors (model unavailable, token limit) | | `internal` | No | Unexpected internal errors (bugs) | | `unknown` | No | Unclassified errors | ## Recovery Actions Each error category maps to one or more recovery actions with CLI commands: | Action | CLI Command | When Used | |--------|-------------|-----------| | `retry` | `agents plan prompt ` | Transient / provider errors | | `correct` | `agents plan correct ` | Validation failures | | `revert_strategize` | `agents plan revert --to-phase strategize` | Resource / merge conflicts | | `prompt` | `agents plan prompt ` | Resume with guidance | | `cancel` | `agents plan cancel ` | Non-recoverable errors | | `manual` | — | Internal errors (check logs) | ## Retry Policy The retry policy is controlled by the automation profile's `modify_config` threshold (named per spec § Automatable Tasks): - **0.0** (auto): Automatic retry for retriable errors - **1.0** (human): Always escalate to human ``` Error occurs → classify_error(message, exception_type) | +→ retriable category? → retry_count < max_retries? → threshold < 1.0? | | | | | No → escalate No → exhausted, escalate No → escalate | | | Yes → auto retry | +→ non-retriable → escalate immediately ``` ### Retry Limits Default `max_retries = 3`. Configured via `SubplanConfig.max_retries` for subplans (max: 5). ### Retry Exhaustion When all retry attempts are used, the error is escalated with: - Clear message indicating retries exhausted - Recovery hints for manual intervention - Error history preserved for debugging ## Structured Error Metadata When an error is recorded, the following metadata is stored in the plan's `error_details` field: | Key | Type | Description | |-----|------|-------------| | `error_category` | string | Classification category | | `error_phase` | string | Phase when error occurred | | `retry_count` | string | Number of retries attempted | | `max_retries` | string | Maximum retries allowed | | `is_retriable` | string | Whether error can be retried | | `error_actor` | string | Actor involved (if any) | | `error_tool_call` | string | Tool call that failed (if any) | | `stack_summary` | string | Abbreviated stack trace (last 3 frames) | | `recovery_action` | string | Suggested recovery action | | `recovery_hint` | string | Human-readable recovery message | | `recovery_command` | string | CLI command for recovery | ## Execute Phase Integration The `PlanExecutor` accepts an optional `ErrorRecoveryService` and uses it during the Execute phase to record errors and drive automatic retries: ``` PlanExecutor ├── StrategizeStubActor (records errors via ErrorRecoveryService) ├── ExecuteStubActor (retry loop driven by ErrorRecoveryService) ├── PlanLifecycleService (phase transitions, persistence) └── ErrorRecoveryService (optional — error recording + retry policy) ``` ### Retry Loop When `error_recovery_service` is provided, `run_execute()` wraps the execute actor in a retry loop: 1. The actor runs. 2. On failure the error is recorded via `ErrorRecoveryService.record_error()`. 3. `should_retry()` is checked — if the error is retriable and retries remain, the actor runs again. 4. If retries are exhausted or the error is non-retriable, the plan transitions to `ERRORED` via `fail_execute()`. Strategize failures are also recorded through the service (but not retried — the strategize phase is read-only and failures are deterministic). ```python from cleveragents.application.services.plan_executor import PlanExecutor from cleveragents.application.services.error_recovery_service import ( ErrorRecoveryService, ) er_service = ErrorRecoveryService( lifecycle_service=lifecycle, auto_retry_threshold=0.0, max_retries=3, ) executor = PlanExecutor( lifecycle_service=lifecycle, tool_runner=runner, error_recovery_service=er_service, ) # Execute with automatic retry on transient errors result = executor.run_execute(plan_id) ``` ## CLI Commands ### `agents plan errors ` Shows error decisions with recovery hints and retry history for a plan. ```bash # Rich (default) output agents plan errors 01HXYZ... # JSON output agents plan errors 01HXYZ... --format json # Plain text agents plan errors 01HXYZ... --format plain ``` Example rich output: ``` ╭─ Plan Errors ──────────────────────────────────────────╮ │ Plan: 01HXYZ... │ │ Phase: execute │ │ State: errored │ │ Error: RuntimeError: Connection timed out │ │ │ │ Error Category: transient │ │ Error Phase: execute │ │ Retry Count: 3/3 │ │ Retriable: false │ │ │ │ Recovery Suggestions: │ │ → Retry the operation — the failure was transient. │ │ $ agents plan prompt 01HXYZ... │ ╰────────────────────────────────────────────────────────╯ ``` When no errors have been recorded the command shows a green confirmation message instead. ## CLI Output Recovery hints are included in CLI error output: ``` Error [transient]: Connection timed out after 30s Phase: execute Retry: 0/3 Recovery suggestions: → Retry the operation — the failure was transient. $ agents plan prompt 01HXYZ... ✓ Automatic retry available. ``` ### Error History The error history tracks all errors for a plan, including: - Total error count - Retry count per error - Category breakdown - Chronological error records with recovery hints > **Note:** Error history is held in-memory within the > `ErrorRecoveryService` instance. If the service is restarted or a > new instance is created, in-memory history is lost. The latest > error metadata is also persisted to the plan's `error_details` > field for durability across restarts. ## Service API ### `ErrorRecoveryService` ```python from cleveragents.application.services.error_recovery_service import ( ErrorRecoveryService, ) service = ErrorRecoveryService( lifecycle_service=lifecycle, auto_retry_threshold=0.0, # from automation profile max_retries=3, ) # Record an error record = service.record_error( plan_id="01HXYZ...", phase="execute", message="Connection timed out", exception=exc, # optional, for stack trace ) # Check retry decision if service.should_retry(plan_id): # auto-retry ... elif service.should_escalate(plan_id): # show recovery hints to user hints = service.format_recovery_hints_for_cli(plan_id) print(hints) # Get full error history history = service.get_error_history(plan_id) print(f"Total errors: {history.total_errors}") # Format for CLI output = service.format_error_output(plan_id, fmt="json") ``` ### `PlanLifecycleService.update_error_details` External services use `update_error_details()` to merge structured error metadata into a plan's `error_details` field without accessing internal persistence methods: ```python lifecycle.update_error_details(plan_id, { "error_category": "transient", "retry_count": "2", }) ``` ### Domain Models ```python from cleveragents.domain.models.core.error_recovery import ( ErrorCategory, ErrorRecord, ErrorHistory, ErrorRecoveryPolicy, RecoveryAction, RecoveryHint, classify_error, get_recovery_hints, ) # Classify an error category = classify_error("Connection timed out", "NetworkError") # Get recovery hints hints = get_recovery_hints(ErrorCategory.TRANSIENT, "01HXYZ...") # Create a policy policy = ErrorRecoveryPolicy(auto_retry_threshold=0.0, max_retries=3) if policy.should_retry(record): ... ```