Files
cleveragents-core/features/steps/circuit_breaker_steps.py
CoreRasurae 4d3499dcfb
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 16s
CI / build (pull_request) Successful in 17s
CI / quality (pull_request) Successful in 18s
CI / security (pull_request) Successful in 35s
CI / typecheck (pull_request) Successful in 38s
CI / unit_tests (pull_request) Successful in 2m43s
CI / integration_tests (pull_request) Successful in 3m20s
CI / docker (pull_request) Successful in 51s
CI / coverage (pull_request) Successful in 6m18s
CI / build (push) Successful in 14s
CI / lint (push) Successful in 21s
CI / quality (push) Successful in 29s
CI / typecheck (push) Successful in 35s
CI / security (push) Successful in 36s
CI / benchmark-regression (push) Has been skipped
CI / unit_tests (push) Successful in 3m17s
CI / docker (push) Successful in 9s
CI / integration_tests (push) Successful in 3m29s
CI / coverage (push) Successful in 7m3s
CI / benchmark-publish (push) Successful in 19m50s
CI / benchmark-regression (pull_request) Successful in 40m6s
feat(async): wire retry policies into services
Wire per-service retry policies and circuit breakers into the service
layer via ServiceRetryWiring, backed by ServiceRetryPolicyRegistry and
configurable through Settings environment variables.

Production hardening from code review:
- Fix TOCTOU race in CircuitBreaker._on_success (half-open state)
- Add half-open probe limit to prevent unbounded concurrent requests
- Track all exception types for circuit breaker failure counting
- Detect async callables wrapped in functools.partial and callable objects
- Enforce spec-compliant 2s minimum for linear backoff strategy
- Enforce 0.1s floor for fixed backoff strategy
- Add retry amplification guard via contextvars nesting depth tracking
- Cap total retry wall-clock time at 300s (MAX_RETRY_TOTAL_TIMEOUT)
- Sanitize exception messages in retry logs to prevent secret leakage
- Fix wrap_service_method TOCTOU by holding cache lock for full operation
- Deep-copy default policies to prevent cross-policy mutation
- Warn on unknown override keys in apply_overrides
- Guard apply_overrides against non-dict and deeply nested JSON values
- Read circuit breaker state under lock in is_circuit_open
- Catch RecursionError in JSON config parsing
- Add total_timeout + nesting guard to retry_service_operation decorator
- Extend secret sanitization to Authorization headers, private_key,
  connection_string, and access_key patterns
- Enforce 0.1s floor on jitter backoff strategy
- Cache wait strategies per service in ServiceRetryWiring (M3)
- Reset failure_count to 0 when entering half-open from open (M6)
- Use cached _get_wait_strategy() in execute()/async_execute()
- Move circuit-open logging out of _on_failure lock scope to prevent
  holding the lock during potentially slow I/O (F1)
- Pass total_timeout=MAX_RETRY_TOTAL_TIMEOUT to wrap_service_method
  retry_service_operation call for consistency with execute() (F4)
- Capture failure_count into local variable inside lock scope before
  logging outside the lock, preventing stale reads from concurrent
  threads in CircuitBreaker.call() and async_call() (F1)
- Deep-copy module-level DEFAULT_DATABASE_RETRY and DEFAULT_CIRCUIT_BREAKER
  in ServiceRetryPolicyRegistry.get() for auto-generated unknown service
  policies, preventing shared mutable state corruption (F1)
- Unify CircuitBreaker to a single threading.Lock for sync and async (P1-1)
- Restore BaseException permit in half-open path to prevent permit leak (P1-6)
- Prevent CircuitBreakerOpen cascading into failure_count (S2)
- Protect all logger calls with contextlib.suppress (S3, S4)
- Replace time.time() with time.monotonic() for monotonic timing (S5)
- Add distinct log events for half-open and closed transitions (S11, S12)
- Track pre-existing services so second apply_settings_defaults only
  targets newly registered services (P1-2)
- Lazy circuit breaker creation via _get_or_create_cb() (P1-3)
- Reject async callables in sync execute() with TypeError (P1-5)
- Strengthen retry predicate to retry_if_exception_type(Exception) &
  retry_if_not_exception_type(CircuitBreakerOpen) (S1)
- Add lock on _get_wait_strategy cache access (P2-16)
- Truncate raw JSON to 80 chars in override warning (P2-17)
- Warn on non-dict JSON overrides (P2-29)
- Debug log for nesting guard bypass (S13)
- Deep-copy from get() and all_policies() in registry (P1-4)
- Thread-safe registry with threading.Lock (P2-15)
- Robust exception handling in apply_overrides get() (P2-18)
- Log ValidationError details on override failure (P2-19)
- Sanitize service_name via _safe_service_name() (P2-28)
- Warn on non-dict sub-key values in overrides (P2-30)
- Allowlist for is_read_only_plan_operation phases (P2-10)
- Cap retry_auto_debug sleep at 60s (P2-11)
- Use is-not-None instead of falsy checks for error values (P2-12)
- Extend secret regex with bearer, session_id, auth_token,
  refresh_token, client_secret patterns (P2-25)
- Pre-truncate error messages to 2000 chars before regex (P2-26)
- Add upper bounds on retry Settings fields (P2-7)
- Add cross-field validator max_delay >= base_delay (P2-8)
- Case-insensitive backoff strategy validation (P2-21)
- Add half_open_max_successes setting (S10)
- Remove phantom ContextFragment from services __all__ (ImportError fix)
- Export ServiceRetryWiring from application.services package
- Include sanitised error context in TypeError logging fallback
- Initialise RetryContext.attempt_count to 1 for bare context-manager usage
- Introduce CircuitBreakerState StrEnum replacing raw string literals
- Fix vacuous CircuitBreakerOpen propagation assertions in BDD steps
- Replace tautological logging test with structlog capture verification
- Assert circuit breaker existence instead of silently skipping on None
- Add Unicode control-char rejection validator to ServiceRetryPolicy.service_name
- Add name parameter with service= in all log calls
- Add extra="forbid" to all 3 Pydantic models
- Deep-copy _SERVICE_DEFAULTS construction
- Key normalisation (.strip()) in get() and apply_overrides()
- Add cooldown <= recovery_timeout validator
- Async guard on RetryContext.execute()
- Nesting guard on RetryContext.execute()/async_execute()
- stop_after_delay(300.0) on RetryContext
- retry_auto_debug async-only guard, dict result fix, sleep guard
- Retry-attempt logging in RetryContext
- Module-level docs for contextlib.suppress(TypeError) rationale
- Exhaustion log on retry failure
- Startup log in __init__; name=service_name to CircuitBreaker
- log_after_retry guarded to not fire on first-attempt success
- get_retry_decorator now includes logging callbacks
- Changed retry_backoff_strategy from str to RetryStrategy StrEnum

Closes #313
2026-03-11 17:42:13 +00:00

421 lines
15 KiB
Python

"""Step definitions for circuit breaker behavior tests."""
from __future__ import annotations
import contextlib
from typing import Any
from behave import given, then, when
from cleveragents.core.retry_patterns import CircuitBreakerOpen, CircuitBreakerState
# ---------------------------------------------------------------------------
# Circuit breaker wiring steps
# ---------------------------------------------------------------------------
@given('the circuit breaker for "{name}" is open')
def step_set_cb_open(context: Any, name: str) -> None:
cb = context.wiring.get_circuit_breaker(name)
assert cb is not None, f"No circuit breaker for {name}"
cb.state = CircuitBreakerState.OPEN
cb.failure_count = cb.failure_threshold
import time
cb.last_failure_time = time.monotonic()
@then("a CircuitBreakerOpen exception should be raised")
def step_check_cb_open_error(context: Any) -> None:
assert isinstance(context.exec_error, CircuitBreakerOpen)
@when('I reset the circuit breaker for "{name}"')
def step_reset_cb(context: Any, name: str) -> None:
context.wiring.reset_circuit(name)
@then('the circuit breaker for "{name}" should be closed')
def step_check_cb_closed(context: Any, name: str) -> None:
cb = context.wiring.get_circuit_breaker(name)
assert cb is not None
assert cb.state == "closed"
@when('I check the circuit status for "{name}"')
def step_check_circuit_status(context: Any, name: str) -> None:
context.circuit_open = context.wiring.is_circuit_open(name)
@then("the circuit should not be open")
def step_check_circuit_not_open(context: Any) -> None:
assert context.circuit_open is False
@given('the circuit breaker for "{name}" is in half-open state')
def step_set_cb_half_open(context: Any, name: str) -> None:
cb = context.wiring.get_circuit_breaker(name)
assert cb is not None, f"No circuit breaker for {name}"
cb.state = CircuitBreakerState.HALF_OPEN
cb.success_count_half_open = 0
cb.failure_count = 0
cb._half_open_permits = cb.half_open_max_successes
@when('I execute two successful calls through the wiring for "{name}"')
def step_execute_two_successes(context: Any, name: str) -> None:
# Execute directly through the circuit breaker to avoid retry decorator
cb = context.wiring.get_circuit_breaker(name)
if cb is not None:
cb.call(lambda: "ok")
cb.call(lambda: "ok")
# ---------------------------------------------------------------------------
# Circuit breaker config wiring (C2, C3)
# ---------------------------------------------------------------------------
@then('the circuit breaker for "{name}" should have cooldown_seconds {value:g}')
def step_check_cb_cooldown_wired(context: Any, name: str, value: float) -> None:
cb = context.wiring.get_circuit_breaker(name)
assert cb is not None
assert cb.cooldown_seconds == value
@then('the circuit breaker for "{name}" should have half_open_max_successes {value:d}')
def step_check_cb_half_open_max_wired(context: Any, name: str, value: int) -> None:
cb = context.wiring.get_circuit_breaker(name)
assert cb is not None
assert cb.half_open_max_successes == value
# ---------------------------------------------------------------------------
# Circuit breaker last_failure_time cleared
# ---------------------------------------------------------------------------
@then('the circuit breaker for "{name}" last_failure_time should be None')
def step_check_cb_last_failure_cleared(context: Any, name: str) -> None:
cb = context.wiring.get_circuit_breaker(name)
assert cb is not None
assert cb.last_failure_time is None
# ---------------------------------------------------------------------------
# B1: _on_success does not close open circuit
# ---------------------------------------------------------------------------
@given("I have a standalone CircuitBreaker instance")
def step_create_standalone_cb(context: Any) -> None:
from cleveragents.core.retry_patterns import CircuitBreaker
context.standalone_cb = CircuitBreaker(failure_threshold=3)
@given('the standalone circuit breaker state is set to "{state}"')
def step_set_standalone_cb_state(context: Any, state: str) -> None:
context.standalone_cb.state = CircuitBreakerState(state)
context.standalone_cb.failure_count = context.standalone_cb.failure_threshold
@when("I call _on_success on the standalone circuit breaker")
def step_call_on_success_standalone(context: Any) -> None:
with context.standalone_cb._lock:
context.standalone_cb._on_success()
@then('the standalone circuit breaker state should remain "{state}"')
def step_check_standalone_cb_state_remain(context: Any, state: str) -> None:
assert context.standalone_cb.state == state
# ---------------------------------------------------------------------------
# B2: Half-open probe limit
# ---------------------------------------------------------------------------
@given("I have a standalone CircuitBreaker instance in half-open with 0 permits")
def step_create_cb_half_open_no_permits(context: Any) -> None:
from cleveragents.core.retry_patterns import CircuitBreaker
context.standalone_cb = CircuitBreaker(failure_threshold=3)
context.standalone_cb.state = CircuitBreakerState.HALF_OPEN
context.standalone_cb._half_open_permits = 0
@when("I call the standalone circuit breaker with a test function")
def step_call_standalone_cb(context: Any) -> None:
from cleveragents.core.retry_patterns import CircuitBreakerOpen
try:
context.standalone_cb.call(lambda: "ok")
context.standalone_cb_error = None
except CircuitBreakerOpen as exc:
context.standalone_cb_error = exc
@then("a CircuitBreakerOpen exception should be raised for probe limit")
def step_check_probe_limit_error(context: Any) -> None:
assert context.standalone_cb_error is not None
assert "probe limit" in str(context.standalone_cb_error)
# ---------------------------------------------------------------------------
# B4: Non-expected exceptions tracked by circuit breaker
# ---------------------------------------------------------------------------
@given("I have a standalone CircuitBreaker with expected_exception IOError")
def step_create_cb_with_ioerror(context: Any) -> None:
from cleveragents.core.retry_patterns import CircuitBreaker
context.standalone_cb = CircuitBreaker(
failure_threshold=5, expected_exception=IOError
)
@when("I call it with a function that raises RuntimeError")
def step_call_cb_with_runtime_error(context: Any) -> None:
def raise_runtime() -> str:
raise RuntimeError("boom")
with contextlib.suppress(RuntimeError):
context.standalone_cb.call(raise_runtime)
@when("I call it with a function that raises IOError")
def step_call_cb_with_ioerror(context: Any) -> None:
def raise_io() -> str:
raise OSError("boom")
with contextlib.suppress(OSError):
context.standalone_cb.call(raise_io)
@then("the standalone circuit breaker failure_count should be {count:d}")
def step_check_standalone_cb_failure_count(context: Any, count: int) -> None:
assert context.standalone_cb.failure_count == count
# ---------------------------------------------------------------------------
# T-A2: Half-open failure re-opens circuit
# ---------------------------------------------------------------------------
@given("I have a standalone CircuitBreaker instance in half-open state with permits")
def step_create_cb_half_open_with_permits(context: Any) -> None:
from cleveragents.core.retry_patterns import CircuitBreaker
context.standalone_cb = CircuitBreaker(failure_threshold=3)
context.standalone_cb.state = CircuitBreakerState.HALF_OPEN
context.standalone_cb._half_open_permits = 2
context.standalone_cb.success_count_half_open = 0
@when("I call it with a function that raises an expected exception")
def step_call_cb_with_expected_exception(context: Any) -> None:
def failing_func() -> str:
raise RuntimeError("half-open probe failed")
with contextlib.suppress(RuntimeError):
context.standalone_cb.call(failing_func)
@then('the standalone circuit breaker state should be "{state}"')
def step_check_standalone_state(context: Any, state: str) -> None:
assert context.standalone_cb.state == state
# ---------------------------------------------------------------------------
# T-A3: Cooldown prevents premature reset
# ---------------------------------------------------------------------------
@given("I have a standalone CircuitBreaker with a recent half-open attempt")
def step_create_cb_with_recent_half_open(context: Any) -> None:
import time
from cleveragents.core.retry_patterns import CircuitBreaker
context.standalone_cb = CircuitBreaker(
failure_threshold=3,
recovery_timeout=1.0,
cooldown_seconds=9999.0, # Very long cooldown
)
context.standalone_cb.state = CircuitBreakerState.OPEN
context.standalone_cb.failure_count = 3
context.standalone_cb.last_failure_time = time.monotonic() - 10.0 # Past recovery
context.standalone_cb._last_half_open_time = time.monotonic() # Just attempted
@when("I check _should_attempt_reset on the standalone circuit breaker")
def step_check_should_attempt_reset(context: Any) -> None:
context.reset_allowed = context.standalone_cb._should_attempt_reset()
@then("the reset should be denied due to cooldown")
def step_check_reset_denied(context: Any) -> None:
assert context.reset_allowed is False
# ---------------------------------------------------------------------------
# M6: failure_count reset on half-open entry
# ---------------------------------------------------------------------------
@given("I have a standalone CircuitBreaker that transitions to half-open")
def step_create_cb_for_half_open_transition(context: Any) -> None:
import time as _time
from cleveragents.core.retry_patterns import CircuitBreaker
context.standalone_cb = CircuitBreaker(
failure_threshold=3,
recovery_timeout=0.01,
cooldown_seconds=0.0,
)
# Simulate an open circuit with failures
# S5: circuit breaker now uses time.monotonic() internally
context.standalone_cb.state = CircuitBreakerState.OPEN
context.standalone_cb.failure_count = 5
context.standalone_cb.last_failure_time = _time.monotonic() - 1.0
@when("the circuit breaker transitions from open to half-open")
def step_transition_to_half_open(context: Any) -> None:
# Calling a successful function should trigger open -> half-open transition
result = context.standalone_cb.call(lambda: "probe_ok")
context.half_open_transition_result = result
@then("the circuit breaker failure_count should be {count:d}")
def step_check_cb_failure_count_after_transition(context: Any, count: int) -> None:
assert context.standalone_cb.failure_count == count
# ---------------------------------------------------------------------------
# Constructor validation: half_open_max_successes
# ---------------------------------------------------------------------------
@when("I create a CircuitBreaker with half_open_max_successes {value:d}")
def step_create_cb_invalid_half_open(context: Any, value: int) -> None:
from cleveragents.core.retry_patterns import CircuitBreaker
try:
CircuitBreaker(failure_threshold=3, half_open_max_successes=value)
context.cb_ctor_error = None
except ValueError as exc:
context.cb_ctor_error = exc
@then("a ValueError should be raised for invalid half_open_max_successes")
def step_check_cb_ctor_value_error(context: Any) -> None:
assert isinstance(context.cb_ctor_error, ValueError)
assert "half_open_max_successes" in str(context.cb_ctor_error)
# ---------------------------------------------------------------------------
# _should_attempt_reset with None last_failure_time
# ---------------------------------------------------------------------------
@given("I have a fresh circuit breaker with no failures")
def step_create_fresh_cb(context: Any) -> None:
from cleveragents.core.retry_patterns import CircuitBreaker
context.fresh_cb = CircuitBreaker(failure_threshold=3)
# Ensure no failures have been recorded
assert context.fresh_cb.last_failure_time is None
@when("I check _should_attempt_reset on the fresh circuit breaker")
def step_check_fresh_should_attempt_reset(context: Any) -> None:
context.fresh_reset_result = context.fresh_cb._should_attempt_reset()
@then("_should_attempt_reset should return False")
def step_verify_fresh_reset_false(context: Any) -> None:
assert context.fresh_reset_result is False
# ---------------------------------------------------------------------------
# Async half-open probe limit
# ---------------------------------------------------------------------------
@given("I have an async circuit breaker in half-open with 0 permits")
def step_create_async_cb_half_open_no_permits(context: Any) -> None:
from cleveragents.core.retry_patterns import CircuitBreaker
context.async_cb = CircuitBreaker(failure_threshold=3)
context.async_cb.state = CircuitBreakerState.HALF_OPEN
context.async_cb._half_open_permits = 0
@when("I make an async call on the half-open circuit breaker")
def step_async_call_half_open_no_permits(context: Any) -> None:
import asyncio as _asyncio
from cleveragents.core.retry_patterns import CircuitBreakerOpen
async def _do() -> None:
await context.async_cb.async_call(lambda: "ok")
loop = _asyncio.new_event_loop()
try:
loop.run_until_complete(_do())
context.async_probe_error = None
except CircuitBreakerOpen as exc:
context.async_probe_error = exc
finally:
loop.close()
@then("a CircuitBreakerOpen should be raised for async probe limit")
def step_check_async_probe_limit(context: Any) -> None:
from cleveragents.core.retry_patterns import CircuitBreakerOpen
assert isinstance(context.async_probe_error, CircuitBreakerOpen)
assert "probe limit" in str(context.async_probe_error)
# ---------------------------------------------------------------------------
# Async CancelledError restores permits
# ---------------------------------------------------------------------------
@given("I have an async circuit breaker in half-open with 1 permit")
def step_create_async_cb_half_open_one_permit(context: Any) -> None:
from cleveragents.core.retry_patterns import CircuitBreaker
context.async_cb = CircuitBreaker(failure_threshold=3)
context.async_cb.state = CircuitBreakerState.HALF_OPEN
context.async_cb._half_open_permits = 1
@when("the async call is cancelled mid-execution")
def step_async_call_cancelled(context: Any) -> None:
import asyncio as _asyncio
async def cancelling_func() -> str:
raise _asyncio.CancelledError()
loop = _asyncio.new_event_loop()
try:
loop.run_until_complete(context.async_cb.async_call(cancelling_func))
context.cancel_error = None
except _asyncio.CancelledError:
context.cancel_error = "cancelled"
finally:
loop.close()
@then("the half-open permit should be restored")
def step_check_permit_restored(context: Any) -> None:
assert context.cancel_error == "cancelled"
# The permit was decremented on entry and restored on CancelledError
assert context.async_cb._half_open_permits == 1