cleveragents-core/docs/reference/retry_policy.md

Retry Policy Configuration

Overview

CleverAgents integrates retry policies and circuit breakers into service layer operations. Each service has a default resilience policy that can be overridden via environment variables or a JSON config string.

Retry is applied only to idempotent operations (repository reads, validation calls). Apply/write operations are never retried.

Both synchronous and asynchronous service methods are supported. Use ServiceRetryWiring.execute() for sync callables and ServiceRetryWiring.async_execute() for async callables. The retry_service_operation decorator auto-detects coroutine functions and uses AsyncRetrying internally.

Passing an async callable to ServiceRetryWiring.execute() raises TypeError — use async_execute() instead. Similarly, RetryContext.execute() raises TypeError for async callables.

All state transitions in CircuitBreaker are thread-safe — an internal lock guards failure counts and state changes without holding the lock during the actual function execution.

Configuration Keys

Key	Env Var	Default	Description
retry_max_attempts	CLEVERAGENTS_RETRY_MAX_ATTEMPTS	3	Maximum retry attempts
retry_base_delay	CLEVERAGENTS_RETRY_BASE_DELAY	1.0	Base delay (seconds)
retry_max_delay	CLEVERAGENTS_RETRY_MAX_DELAY	60.0	Max delay (seconds)
retry_jitter	CLEVERAGENTS_RETRY_JITTER	true	Add random jitter
retry_backoff_strategy	CLEVERAGENTS_RETRY_BACKOFF_STRATEGY	exponential	Backoff: exponential, linear, fixed, jitter, none
circuit_breaker_failure_threshold	CLEVERAGENTS_CB_FAILURE_THRESHOLD	5	Failures before circuit opens
circuit_breaker_recovery_timeout	CLEVERAGENTS_CB_RECOVERY_TIMEOUT	60.0	Seconds before half-open
circuit_breaker_cooldown	CLEVERAGENTS_CB_COOLDOWN	30.0	Min seconds between recovery
retry_service_overrides	CLEVERAGENTS_RETRY_SERVICE_OVERRIDES	{}	JSON per-service overrides

Per-Service Override Example

export CLEVERAGENTS_RETRY_SERVICE_OVERRIDES='{
  "plan_service": {
    "retry": {"max_attempts": 5, "base_delay": 2.0},
    "circuit_breaker": {"failure_threshold": 3}
  },
  "vector_store_service": {
    "retry": {"max_attempts": 7, "backoff_strategy": "jitter"}
  }
}'

Default Per-Service Policies

Service	Category	Max Attempts	Backoff	Jitter
plan_service	provider	3	jitter	yes
plan_lifecycle_service	database	3	fixed	no
context_service	database	3	fixed	no
session_service	database	3	fixed	no
checkpoint_service	database	3	fixed	no
trace_service	database	3	fixed	no
actor_service	database	3	fixed	no
resource_registry_service	database	3	fixed	no
decision_service	database	3	fixed	no
vector_store_service	network	5	exponential	yes
error_recovery_service	database	3	fixed	no

Circuit Breaker States

closed -> open (after failure_threshold consecutive failures)
open -> half-open (after recovery_timeout seconds AND cooldown_seconds
                   since last half-open attempt; failure_count is reset
                   to 0 on entry to prevent stale counts from
                   immediately re-opening the circuit)
half-open -> closed (after half_open_max_successes consecutive successes)
half-open -> open (on any failure)

Both cooldown_seconds and half_open_max_successes are wired from per-service JSON overrides into the CircuitBreaker constructor. Global environment variable overrides (CLEVERAGENTS_CB_*) do not currently include half_open_max_successes — this field is only configurable via the JSON per-service override path.

The reset_circuit() helper atomically clears all counters and timestamps (including last_failure_time) under the circuit breaker's internal lock.

Half-Open Probe Limit

When the circuit enters half-open state, only half_open_max_successes concurrent requests are admitted as probes. Additional requests receive CircuitBreakerOpen immediately. This prevents a flood of requests from hitting a recovering service.

Failure Tracking

All exceptions (not just the configured expected_exception type) are counted towards the failure threshold. This ensures the circuit breaker opens for any type of recurring failure.

Concurrent Safety in Half-Open State

A successful probe in half-open state only counts towards recovery if the circuit is still half-open at the time the lock is acquired. If a concurrent failure has already re-opened the circuit, the stale success is ignored — the circuit remains open.

Per-Service Circuit Breaker Toggle

Each service's CircuitBreakerConfig includes an enabled boolean (default True). Setting enabled: false in a per-service override disables circuit breaker protection for that service without removing it from the registry. When disabled, operations run without any circuit breaker wrapping.

Structured Logs

Every retry attempt emits a structured log with:

• service: Service name
• operation: Operation name
• attempt: Attempt number
• error: Error message (if applicable)

Circuit breaker opens emit:

• service: Service name
• failure_count: Number of consecutive failures

Idempotency Guard

The retry_on_idempotent_only flag (default True) ensures that non-idempotent operations (writes, applies) are never retried. The is_read_only_plan_operation() guard additionally prevents retries on tool execution writes in read-only plans. Only string values of plan_phase are evaluated; non-string truthy values (e.g. True, 1) are treated as not read-only to prevent false positives.

Thread Safety

CircuitBreaker uses an internal threading.Lock to guard all state transitions. The lock is not held during the actual function call, so long-running operations do not block other threads. Logging that the circuit has opened is also emitted after the lock is released to avoid holding the lock during potentially slow I/O.

The ServiceRetryWiring._decorator_cache is guarded by a separate threading.Lock to prevent TOCTOU races under concurrent access. The entire lookup-or-create operation runs under the lock.

is_circuit_open() reads the circuit state under the breaker's lock, making it safe to call from any thread.

Backoff Floor

When the backoff strategy is "exponential" and base_delay is below 0.1, the effective delay is clamped to 0.1 seconds to prevent instant retry hammering.

When the strategy is "fixed", a minimum floor of 0.1 seconds is enforced.

When the strategy is "linear", a minimum floor of 2.0 seconds is enforced per the specification. Note that the "linear" strategy uses wait_fixed (constant delay), not a linearly increasing delay — every retry waits the same duration.

When the strategy is "jitter", a minimum floor of 0.1 seconds is enforced, consistent with the exponential and fixed floors.

Retry Amplification Guard

A contextvars-based nesting guard prevents exponential retry multiplication when service A (retrying) calls service B (also retrying). At nesting depth >= 1, the inner call executes exactly once (no retry), while still routing through the circuit breaker if present.

The nesting guard is applied in both execute()/async_execute() and the retry_service_operation decorator, ensuring consistent protection regardless of the calling pattern.

Total Timeout Cap

All retry attempts are bounded by a total wall-clock timeout of MAX_RETRY_TOTAL_TIMEOUT (300 seconds / 5 minutes). This prevents pathological configurations from blocking a thread indefinitely.

The retry_service_operation decorator accepts a total_timeout parameter (default 300.0 seconds) providing the same wall-clock cap.

Error Sanitization

Exception messages logged during retry attempts are sanitized to prevent leaking secrets. The following patterns are redacted:

• URL credentials: ://user:pass@host → ://***@host
• Key-value secrets: api_key=..., password=..., secret=..., credential=..., token=..., private_key=..., connection_string=..., access_key=..., bearer=..., session_id=..., auth_token=..., refresh_token=..., client_secret=... → <key>=***
• Authorization headers: Authorization: Bearer ..., x-api-key: ... → <header>: ***
• Dict-literal secrets: {'api_key': '...', 'token': '...'} → {'api_key': '***', 'token': '***'} (Python dict repr format)

Messages longer than 200 characters are truncated with ... appended.

Async Callable Detection

The retry_service_operation decorator detects async callables using _is_async_callable(), which handles asyncio coroutine functions, functools.partial wrapping async functions, and callable objects with an async __call__ method.

Override Safety

Per-service JSON overrides (retry_service_overrides) have a maximum length of 10 000 characters. The service_name key is ignored in overrides to prevent key/value mismatches in the policy registry. Invalid overrides that fail Pydantic validation are logged and skipped. Non-dict override values are logged and ignored. Unrecognised top-level keys in override dicts emit a warning so typos are not silently lost.

Wait Strategy Caching

Wait strategies (tenacity wait objects) are pre-built during ServiceRetryWiring.__init__() and cached per service name. execute() and async_execute() reuse the cached strategy via _get_wait_strategy() instead of rebuilding per call. Unknown services that are not registered at init time are lazily built and cached on first access.

Default Isolation

Per-service default policies are deep-copied on registry creation, so mutation of one service's sub-models never corrupts shared defaults or other policies. Auto-generated policies for unknown services (created on first get() call) are likewise deep-copied from the module-level DEFAULT_DATABASE_RETRY and DEFAULT_CIRCUIT_BREAKER constants.