temp/docs/reference/error_recovery.md

# Error Recovery Reference

This document describes the error recovery patterns, error classification,
recovery hint system, and retry policy logic added in D1b.recovery.

## Error Categories

Errors are classified into categories that determine recovery behaviour:

| Category | Retriable | Description |
|----------|-----------|-------------|
| `transient` | Yes | Temporary failures (network, timeout, rate-limit) |
| `validation` | Yes | Validation failures during Execute or Apply |
| `configuration` | No | Bad config, missing required fields |
| `resource` | No | Missing or conflicting resources |
| `merge_conflict` | Yes | Sandbox merge failures |
| `authentication` | No | Auth/authz failures |
| `provider` | Yes | LLM provider errors (model unavailable, token limit) |
| `internal` | No | Unexpected internal errors (bugs) |
| `unknown` | No | Unclassified errors |

## Recovery Actions

Each error category maps to one or more recovery actions with CLI commands:

| Action | CLI Command | When Used |
|--------|-------------|-----------|
| `retry` | `agents plan prompt <plan_id>` | Transient / provider errors |
| `correct` | `agents plan correct <plan_id>` | Validation failures |
| `revert_strategize` | `agents plan revert <plan_id> --to-phase strategize` | Resource / merge conflicts |
| `prompt` | `agents plan prompt <plan_id>` | Resume with guidance |
| `cancel` | `agents plan cancel <plan_id>` | Non-recoverable errors |
| `manual` | — | Internal errors (check logs) |

## Retry Policy

The retry policy is controlled by the automation profile's
`modify_config` threshold (named per spec § Automatable Tasks):

- **0.0** (auto): Automatic retry for retriable errors
- **1.0** (human): Always escalate to human

```
Error occurs → classify_error(message, exception_type)
    |
    +→ retriable category? → retry_count < max_retries? → threshold < 1.0?
    |     |                     |                            |
    |     No → escalate         No → exhausted, escalate     No → escalate
    |                                                        |
    |                                                        Yes → auto retry
    |
    +→ non-retriable → escalate immediately
```

### Retry Limits

Default `max_retries = 3`. Configured via `SubplanConfig.max_retries`
for subplans (max: 5).

### Retry Exhaustion

When all retry attempts are used, the error is escalated with:
- Clear message indicating retries exhausted
- Recovery hints for manual intervention
- Error history preserved for debugging

## Structured Error Metadata

When an error is recorded, the following metadata is stored in the
plan's `error_details` field:

| Key | Type | Description |
|-----|------|-------------|
| `error_category` | string | Classification category |
| `error_phase` | string | Phase when error occurred |
| `retry_count` | string | Number of retries attempted |
| `max_retries` | string | Maximum retries allowed |
| `is_retriable` | string | Whether error can be retried |
| `error_actor` | string | Actor involved (if any) |
| `error_tool_call` | string | Tool call that failed (if any) |
| `stack_summary` | string | Abbreviated stack trace (last 3 frames) |
| `recovery_action` | string | Suggested recovery action |
| `recovery_hint` | string | Human-readable recovery message |
| `recovery_command` | string | CLI command for recovery |

## Execute Phase Integration

The `PlanExecutor` accepts an optional `ErrorRecoveryService` and uses
it during the Execute phase to record errors and drive automatic
retries:

```
PlanExecutor
├── StrategizeStubActor   (records errors via ErrorRecoveryService)
├── ExecuteStubActor      (retry loop driven by ErrorRecoveryService)
├── PlanLifecycleService  (phase transitions, persistence)
└── ErrorRecoveryService  (optional — error recording + retry policy)
```

### Retry Loop

When `error_recovery_service` is provided, `run_execute()` wraps the
execute actor in a retry loop:

1. The actor runs.
2. On failure the error is recorded via `ErrorRecoveryService.record_error()`.
3. `should_retry()` is checked — if the error is retriable and retries
   remain, the actor runs again.
4. If retries are exhausted or the error is non-retriable, the plan
   transitions to `ERRORED` via `fail_execute()`.

Strategize failures are also recorded through the service (but not
retried — the strategize phase is read-only and failures are
deterministic).

```python
from cleveragents.application.services.plan_executor import PlanExecutor
from cleveragents.application.services.error_recovery_service import (
    ErrorRecoveryService,
)

er_service = ErrorRecoveryService(
    lifecycle_service=lifecycle,
    auto_retry_threshold=0.0,
    max_retries=3,
)

executor = PlanExecutor(
    lifecycle_service=lifecycle,
    tool_runner=runner,
    error_recovery_service=er_service,
)

# Execute with automatic retry on transient errors
result = executor.run_execute(plan_id)
```

## CLI Commands

### `agents plan errors <plan_id>`

Shows error decisions with recovery hints and retry history for a plan.

```bash
# Rich (default) output
agents plan errors 01HXYZ...

# JSON output
agents plan errors 01HXYZ... --format json

# Plain text
agents plan errors 01HXYZ... --format plain
```

Example rich output:

```
╭─ Plan Errors ──────────────────────────────────────────╮
│ Plan: 01HXYZ...                                        │
│ Phase: execute                                         │
│ State: errored                                         │
│ Error: RuntimeError: Connection timed out              │
│                                                        │
│ Error Category: transient                              │
│ Error Phase: execute                                   │
│ Retry Count: 3/3                                       │
│ Retriable: false                                       │
│                                                        │
│ Recovery Suggestions:                                  │
│   → Retry the operation — the failure was transient.   │
│     $ agents plan prompt 01HXYZ...                     │
╰────────────────────────────────────────────────────────╯
```

When no errors have been recorded the command shows a green
confirmation message instead.

## CLI Output

Recovery hints are included in CLI error output:

```
Error [transient]: Connection timed out after 30s
  Phase: execute
  Retry: 0/3
  Recovery suggestions:
    → Retry the operation — the failure was transient.
      $ agents plan prompt 01HXYZ...
  ✓ Automatic retry available.
```

### Error History

The error history tracks all errors for a plan, including:
- Total error count
- Retry count per error
- Category breakdown
- Chronological error records with recovery hints

> **Note:** Error history is held in-memory within the
> `ErrorRecoveryService` instance. If the service is restarted or a
> new instance is created, in-memory history is lost. The latest
> error metadata is also persisted to the plan's `error_details`
> field for durability across restarts.

## Service API

### `ErrorRecoveryService`

```python
from cleveragents.application.services.error_recovery_service import (
    ErrorRecoveryService,
)

service = ErrorRecoveryService(
    lifecycle_service=lifecycle,
    auto_retry_threshold=0.0,  # from automation profile
    max_retries=3,
)

# Record an error
record = service.record_error(
    plan_id="01HXYZ...",
    phase="execute",
    message="Connection timed out",
    exception=exc,  # optional, for stack trace
)

# Check retry decision
if service.should_retry(plan_id):
    # auto-retry
    ...
elif service.should_escalate(plan_id):
    # show recovery hints to user
    hints = service.format_recovery_hints_for_cli(plan_id)
    print(hints)

# Get full error history
history = service.get_error_history(plan_id)
print(f"Total errors: {history.total_errors}")

# Format for CLI
output = service.format_error_output(plan_id, fmt="json")
```

### `PlanLifecycleService.update_error_details`

External services use `update_error_details()` to merge structured
error metadata into a plan's `error_details` field without accessing
internal persistence methods:

```python
lifecycle.update_error_details(plan_id, {
    "error_category": "transient",
    "retry_count": "2",
})
```

### Domain Models

```python
from cleveragents.domain.models.core.error_recovery import (
    ErrorCategory,
    ErrorRecord,
    ErrorHistory,
    ErrorRecoveryPolicy,
    RecoveryAction,
    RecoveryHint,
    classify_error,
    get_recovery_hints,
)

# Classify an error
category = classify_error("Connection timed out", "NetworkError")

# Get recovery hints
hints = get_recovery_hints(ErrorCategory.TRANSIENT, "01HXYZ...")

# Create a policy
policy = ErrorRecoveryPolicy(auto_retry_threshold=0.0, max_retries=3)
if policy.should_retry(record):
    ...
```