cleveragents-core/docs/reference/retry_policy.md

# Retry Policy Configuration

## Overview

CleverAgents integrates retry policies and circuit breakers into service
layer operations.  Each service has a default resilience policy that can
be overridden via environment variables or a JSON config string.

Retry is applied **only to idempotent operations** (repository reads,
validation calls).  Apply/write operations are never retried.

Both synchronous and asynchronous service methods are supported.  Use
`ServiceRetryWiring.execute()` for sync callables and
`ServiceRetryWiring.async_execute()` for async callables.  The
`retry_service_operation` decorator auto-detects coroutine functions
and uses `AsyncRetrying` internally.

Passing an async callable to `ServiceRetryWiring.execute()` raises
`TypeError` — use `async_execute()` instead.  Similarly,
`RetryContext.execute()` raises `TypeError` for async callables.

All state transitions in `CircuitBreaker` are thread-safe — an internal
lock guards failure counts and state changes without holding the lock
during the actual function execution.

## Configuration Keys

| Key | Env Var | Default | Description |
|-----|---------|---------|-------------|
| `retry_max_attempts` | `CLEVERAGENTS_RETRY_MAX_ATTEMPTS` | 3 | Maximum retry attempts |
| `retry_base_delay` | `CLEVERAGENTS_RETRY_BASE_DELAY` | 1.0 | Base delay (seconds) |
| `retry_max_delay` | `CLEVERAGENTS_RETRY_MAX_DELAY` | 60.0 | Max delay (seconds) |
| `retry_jitter` | `CLEVERAGENTS_RETRY_JITTER` | true | Add random jitter |
| `retry_backoff_strategy` | `CLEVERAGENTS_RETRY_BACKOFF_STRATEGY` | exponential | Backoff: exponential, linear, fixed, jitter, none |
| `circuit_breaker_failure_threshold` | `CLEVERAGENTS_CB_FAILURE_THRESHOLD` | 5 | Failures before circuit opens |
| `circuit_breaker_recovery_timeout` | `CLEVERAGENTS_CB_RECOVERY_TIMEOUT` | 60.0 | Seconds before half-open |
| `circuit_breaker_cooldown` | `CLEVERAGENTS_CB_COOLDOWN` | 30.0 | Min seconds between recovery |
| `retry_service_overrides` | `CLEVERAGENTS_RETRY_SERVICE_OVERRIDES` | {} | JSON per-service overrides |

## Per-Service Override Example

```bash
export CLEVERAGENTS_RETRY_SERVICE_OVERRIDES='{
  "plan_service": {
    "retry": {"max_attempts": 5, "base_delay": 2.0},
    "circuit_breaker": {"failure_threshold": 3}
  },
  "vector_store_service": {
    "retry": {"max_attempts": 7, "backoff_strategy": "jitter"}
  }
}'
```

## Default Per-Service Policies

| Service | Category | Max Attempts | Backoff | Jitter |
|---------|----------|-------------|---------|--------|
| `plan_service` | provider | 3 | jitter | yes |
| `plan_lifecycle_service` | database | 3 | fixed | no |
| `context_service` | database | 3 | fixed | no |
| `session_service` | database | 3 | fixed | no |
| `checkpoint_service` | database | 3 | fixed | no |
| `trace_service` | database | 3 | fixed | no |
| `actor_service` | database | 3 | fixed | no |
| `resource_registry_service` | database | 3 | fixed | no |
| `decision_service` | database | 3 | fixed | no |
| `vector_store_service` | network | 5 | exponential | yes |
| `error_recovery_service` | database | 3 | fixed | no |

## Circuit Breaker States

```
closed -> open (after failure_threshold consecutive failures)
open -> half-open (after recovery_timeout seconds AND cooldown_seconds
                   since last half-open attempt; failure_count is reset
                   to 0 on entry to prevent stale counts from
                   immediately re-opening the circuit)
half-open -> closed (after half_open_max_successes consecutive successes)
half-open -> open (on any failure)
```

Both `cooldown_seconds` and `half_open_max_successes` are wired from
per-service JSON overrides into the `CircuitBreaker` constructor.
Global environment variable overrides (`CLEVERAGENTS_CB_*`) do not
currently include `half_open_max_successes` — this field is only
configurable via the JSON per-service override path.

The `reset_circuit()` helper atomically clears all counters **and
timestamps** (including `last_failure_time`) under the circuit breaker's
internal lock.

### Half-Open Probe Limit

When the circuit enters half-open state, only `half_open_max_successes`
concurrent requests are admitted as probes.  Additional requests receive
`CircuitBreakerOpen` immediately.  This prevents a flood of requests
from hitting a recovering service.

### Failure Tracking

All exceptions (not just the configured `expected_exception` type) are
counted towards the failure threshold.  This ensures the circuit breaker
opens for any type of recurring failure.

### Concurrent Safety in Half-Open State

A successful probe in half-open state only counts towards recovery if
the circuit is still half-open at the time the lock is acquired.  If a
concurrent failure has already re-opened the circuit, the stale success
is ignored — the circuit remains open.

### Per-Service Circuit Breaker Toggle

Each service's `CircuitBreakerConfig` includes an `enabled` boolean
(default `True`).  Setting `enabled: false` in a per-service override
disables circuit breaker protection for that service without removing
it from the registry.  When disabled, operations run without any
circuit breaker wrapping.

## Structured Logs

Every retry attempt emits a structured log with:

- `service`: Service name
- `operation`: Operation name
- `attempt`: Attempt number
- `error`: Error message (if applicable)

Circuit breaker opens emit:

- `service`: Service name
- `failure_count`: Number of consecutive failures

## Idempotency Guard

The `retry_on_idempotent_only` flag (default `True`) ensures that
non-idempotent operations (writes, applies) are never retried.  The
`is_read_only_plan_operation()` guard additionally prevents retries on
tool execution writes in read-only plans.  Only string values of
`plan_phase` are evaluated; non-string truthy values (e.g. `True`, `1`)
are treated as *not* read-only to prevent false positives.

## Thread Safety

`CircuitBreaker` uses an internal `threading.Lock` to guard all state
transitions.  The lock is *not* held during the actual function call,
so long-running operations do not block other threads.  Logging that
the circuit has opened is also emitted *after* the lock is released to
avoid holding the lock during potentially slow I/O.

The `ServiceRetryWiring._decorator_cache` is guarded by a separate
`threading.Lock` to prevent TOCTOU races under concurrent access.  The
entire lookup-or-create operation runs under the lock.

`is_circuit_open()` reads the circuit state under the breaker's lock,
making it safe to call from any thread.

## Backoff Floor

When the backoff strategy is `"exponential"` and `base_delay` is below
`0.1`, the effective delay is clamped to `0.1` seconds to prevent
instant retry hammering.

When the strategy is `"fixed"`, a minimum floor of `0.1` seconds is
enforced.

When the strategy is `"linear"`, a minimum floor of `2.0` seconds is
enforced per the specification.  Note that the "linear" strategy uses
`wait_fixed` (constant delay), not a linearly increasing delay —
every retry waits the same duration.

When the strategy is `"jitter"`, a minimum floor of `0.1` seconds is
enforced, consistent with the exponential and fixed floors.

## Retry Amplification Guard

A `contextvars`-based nesting guard prevents exponential retry
multiplication when service A (retrying) calls service B (also
retrying).  At nesting depth >= 1, the inner call executes exactly once
(no retry), while still routing through the circuit breaker if present.

The nesting guard is applied in both `execute()`/`async_execute()` and
the `retry_service_operation` decorator, ensuring consistent protection
regardless of the calling pattern.

## Total Timeout Cap

All retry attempts are bounded by a total wall-clock timeout of
`MAX_RETRY_TOTAL_TIMEOUT` (300 seconds / 5 minutes).  This prevents
pathological configurations from blocking a thread indefinitely.

The `retry_service_operation` decorator accepts a `total_timeout`
parameter (default 300.0 seconds) providing the same wall-clock cap.

## Error Sanitization

Exception messages logged during retry attempts are sanitized to
prevent leaking secrets.  The following patterns are redacted:

- URL credentials: `://user:pass@host` → `://***@host`
- Key-value secrets: `api_key=...`, `password=...`, `secret=...`,
  `credential=...`, `token=...`, `private_key=...`,
  `connection_string=...`, `access_key=...`, `bearer=...`,
  `session_id=...`, `auth_token=...`, `refresh_token=...`,
  `client_secret=...` → `<key>=***`
- Authorization headers: `Authorization: Bearer ...`,
  `x-api-key: ...` → `<header>: ***`
- Dict-literal secrets: `{'api_key': '...', 'token': '...'}` →
  `{'api_key': '***', 'token': '***'}` (Python dict repr format)

Messages longer than 200 characters are truncated with `...` appended.

## Async Callable Detection

The `retry_service_operation` decorator detects async callables using
`_is_async_callable()`, which handles `asyncio` coroutine functions,
`functools.partial` wrapping async functions, and callable objects with
an async `__call__` method.

## Override Safety

Per-service JSON overrides (`retry_service_overrides`) have a maximum
length of 10 000 characters.  The `service_name` key is ignored in
overrides to prevent key/value mismatches in the policy registry.
Invalid overrides that fail Pydantic validation are logged and skipped.
Non-dict override values are logged and ignored.  Unrecognised
top-level keys in override dicts emit a warning so typos are not
silently lost.

## Wait Strategy Caching

Wait strategies (tenacity wait objects) are pre-built during
`ServiceRetryWiring.__init__()` and cached per service name.
`execute()` and `async_execute()` reuse the cached strategy via
`_get_wait_strategy()` instead of rebuilding per call.  Unknown
services that are not registered at init time are lazily built and
cached on first access.

## Default Isolation

Per-service default policies are deep-copied on registry creation, so
mutation of one service's sub-models never corrupts shared defaults or
other policies.  Auto-generated policies for unknown services (created
on first ``get()`` call) are likewise deep-copied from the module-level
``DEFAULT_DATABASE_RETRY`` and ``DEFAULT_CIRCUIT_BREAKER`` constants.