cleveragents-core/docs/reference/error_recovery.md

Error Recovery Reference

This document describes the error recovery patterns, error classification, recovery hint system, and retry policy logic added in D1b.recovery.

Error Categories

Errors are classified into categories that determine recovery behaviour:

Category	Retriable	Description
transient	Yes	Temporary failures (network, timeout, rate-limit)
validation	Yes	Validation failures during Execute or Apply
configuration	No	Bad config, missing required fields
resource	No	Missing or conflicting resources
merge_conflict	Yes	Sandbox merge failures
authentication	No	Auth/authz failures
provider	Yes	LLM provider errors (model unavailable, token limit)
internal	No	Unexpected internal errors (bugs)
unknown	No	Unclassified errors

Recovery Actions

Each error category maps to one or more recovery actions with CLI commands:

Action	CLI Command	When Used
retry	agents plan prompt <plan_id>	Transient / provider errors
correct	agents plan correct <plan_id>	Validation failures
revert_strategize	agents plan revert <plan_id> --to-phase strategize	Resource / merge conflicts
prompt	agents plan prompt <plan_id>	Resume with guidance
cancel	agents plan cancel <plan_id>	Non-recoverable errors
manual	—	Internal errors (check logs)

Retry Policy

The retry policy is controlled by the automation profile's auto_retry_transient threshold:

• 0.0 (auto): Automatic retry for retriable errors
• 1.0 (human): Always escalate to human

Error occurs → classify_error(message, exception_type)
    |
    +→ retriable category? → retry_count < max_retries? → threshold < 1.0?
    |     |                     |                            |
    |     No → escalate         No → exhausted, escalate     No → escalate
    |                                                        |
    |                                                        Yes → auto retry
    |
    +→ non-retriable → escalate immediately

Retry Limits

Default max_retries = 3. Configured via SubplanConfig.max_retries for subplans (max: 5).

Retry Exhaustion

When all retry attempts are used, the error is escalated with:

• Clear message indicating retries exhausted
• Recovery hints for manual intervention
• Error history preserved for debugging

Structured Error Metadata

When an error is recorded, the following metadata is stored in the plan's error_details field:

Key	Type	Description
error_category	string	Classification category
error_phase	string	Phase when error occurred
retry_count	string	Number of retries attempted
max_retries	string	Maximum retries allowed
is_retriable	string	Whether error can be retried
error_actor	string	Actor involved (if any)
error_tool_call	string	Tool call that failed (if any)
stack_summary	string	Abbreviated stack trace (last 3 frames)
recovery_action	string	Suggested recovery action
recovery_hint	string	Human-readable recovery message
recovery_command	string	CLI command for recovery

Execute Phase Integration

The PlanExecutor accepts an optional ErrorRecoveryService and uses it during the Execute phase to record errors and drive automatic retries:

PlanExecutor
├── StrategizeStubActor   (records errors via ErrorRecoveryService)
├── ExecuteStubActor      (retry loop driven by ErrorRecoveryService)
├── PlanLifecycleService  (phase transitions, persistence)
└── ErrorRecoveryService  (optional — error recording + retry policy)

Retry Loop

When error_recovery_service is provided, run_execute() wraps the execute actor in a retry loop:

1. The actor runs.
2. On failure the error is recorded via ErrorRecoveryService.record_error().
3. should_retry() is checked — if the error is retriable and retries remain, the actor runs again.
4. If retries are exhausted or the error is non-retriable, the plan transitions to ERRORED via fail_execute().

Strategize failures are also recorded through the service (but not retried — the strategize phase is read-only and failures are deterministic).

from cleveragents.application.services.plan_executor import PlanExecutor
from cleveragents.application.services.error_recovery_service import (
    ErrorRecoveryService,
)

er_service = ErrorRecoveryService(
    lifecycle_service=lifecycle,
    auto_retry_threshold=0.0,
    max_retries=3,
)

executor = PlanExecutor(
    lifecycle_service=lifecycle,
    tool_runner=runner,
    error_recovery_service=er_service,
)

# Execute with automatic retry on transient errors
result = executor.run_execute(plan_id)

CLI Commands

agents plan errors <plan_id>

Shows error decisions with recovery hints and retry history for a plan.

# Rich (default) output
agents plan errors 01HXYZ...

# JSON output
agents plan errors 01HXYZ... --format json

# Plain text
agents plan errors 01HXYZ... --format plain

Example rich output:

╭─ Plan Errors ──────────────────────────────────────────╮
│ Plan: 01HXYZ...                                        │
│ Phase: execute                                         │
│ State: errored                                         │
│ Error: RuntimeError: Connection timed out              │
│                                                        │
│ Error Category: transient                              │
│ Error Phase: execute                                   │
│ Retry Count: 3/3                                       │
│ Retriable: false                                       │
│                                                        │
│ Recovery Suggestions:                                  │
│   → Retry the operation — the failure was transient.   │
│     $ agents plan prompt 01HXYZ...                     │
╰────────────────────────────────────────────────────────╯

When no errors have been recorded the command shows a green confirmation message instead.

CLI Output

Recovery hints are included in CLI error output:

Error [transient]: Connection timed out after 30s
  Phase: execute
  Retry: 0/3
  Recovery suggestions:
    → Retry the operation — the failure was transient.
      $ agents plan prompt 01HXYZ...
  ✓ Automatic retry available.

Error History

The error history tracks all errors for a plan, including:

• Total error count
• Retry count per error
• Category breakdown
• Chronological error records with recovery hints

Note: Error history is held in-memory within the ErrorRecoveryService instance. If the service is restarted or a new instance is created, in-memory history is lost. The latest error metadata is also persisted to the plan's error_details field for durability across restarts.

Service API

ErrorRecoveryService

from cleveragents.application.services.error_recovery_service import (
    ErrorRecoveryService,
)

service = ErrorRecoveryService(
    lifecycle_service=lifecycle,
    auto_retry_threshold=0.0,  # from automation profile
    max_retries=3,
)

# Record an error
record = service.record_error(
    plan_id="01HXYZ...",
    phase="execute",
    message="Connection timed out",
    exception=exc,  # optional, for stack trace
)

# Check retry decision
if service.should_retry(plan_id):
    # auto-retry
    ...
elif service.should_escalate(plan_id):
    # show recovery hints to user
    hints = service.format_recovery_hints_for_cli(plan_id)
    print(hints)

# Get full error history
history = service.get_error_history(plan_id)
print(f"Total errors: {history.total_errors}")

# Format for CLI
output = service.format_error_output(plan_id, fmt="json")

PlanLifecycleService.update_error_details

External services use update_error_details() to merge structured error metadata into a plan's error_details field without accessing internal persistence methods:

lifecycle.update_error_details(plan_id, {
    "error_category": "transient",
    "retry_count": "2",
})

Domain Models

from cleveragents.domain.models.core.error_recovery import (
    ErrorCategory,
    ErrorRecord,
    ErrorHistory,
    ErrorRecoveryPolicy,
    RecoveryAction,
    RecoveryHint,
    classify_error,
    get_recovery_hints,
)

# Classify an error
category = classify_error("Connection timed out", "NetworkError")

# Get recovery hints
hints = get_recovery_hints(ErrorCategory.TRANSIENT, "01HXYZ...")

# Create a policy
policy = ErrorRecoveryPolicy(auto_retry_threshold=0.0, max_retries=3)
if policy.should_retry(record):
    ...