Files
temp/features/steps/retry_patterns_coverage_boost_steps.py
CoreRasurae 4d3499dcfb feat(async): wire retry policies into services
Wire per-service retry policies and circuit breakers into the service
layer via ServiceRetryWiring, backed by ServiceRetryPolicyRegistry and
configurable through Settings environment variables.

Production hardening from code review:
- Fix TOCTOU race in CircuitBreaker._on_success (half-open state)
- Add half-open probe limit to prevent unbounded concurrent requests
- Track all exception types for circuit breaker failure counting
- Detect async callables wrapped in functools.partial and callable objects
- Enforce spec-compliant 2s minimum for linear backoff strategy
- Enforce 0.1s floor for fixed backoff strategy
- Add retry amplification guard via contextvars nesting depth tracking
- Cap total retry wall-clock time at 300s (MAX_RETRY_TOTAL_TIMEOUT)
- Sanitize exception messages in retry logs to prevent secret leakage
- Fix wrap_service_method TOCTOU by holding cache lock for full operation
- Deep-copy default policies to prevent cross-policy mutation
- Warn on unknown override keys in apply_overrides
- Guard apply_overrides against non-dict and deeply nested JSON values
- Read circuit breaker state under lock in is_circuit_open
- Catch RecursionError in JSON config parsing
- Add total_timeout + nesting guard to retry_service_operation decorator
- Extend secret sanitization to Authorization headers, private_key,
  connection_string, and access_key patterns
- Enforce 0.1s floor on jitter backoff strategy
- Cache wait strategies per service in ServiceRetryWiring (M3)
- Reset failure_count to 0 when entering half-open from open (M6)
- Use cached _get_wait_strategy() in execute()/async_execute()
- Move circuit-open logging out of _on_failure lock scope to prevent
  holding the lock during potentially slow I/O (F1)
- Pass total_timeout=MAX_RETRY_TOTAL_TIMEOUT to wrap_service_method
  retry_service_operation call for consistency with execute() (F4)
- Capture failure_count into local variable inside lock scope before
  logging outside the lock, preventing stale reads from concurrent
  threads in CircuitBreaker.call() and async_call() (F1)
- Deep-copy module-level DEFAULT_DATABASE_RETRY and DEFAULT_CIRCUIT_BREAKER
  in ServiceRetryPolicyRegistry.get() for auto-generated unknown service
  policies, preventing shared mutable state corruption (F1)
- Unify CircuitBreaker to a single threading.Lock for sync and async (P1-1)
- Restore BaseException permit in half-open path to prevent permit leak (P1-6)
- Prevent CircuitBreakerOpen cascading into failure_count (S2)
- Protect all logger calls with contextlib.suppress (S3, S4)
- Replace time.time() with time.monotonic() for monotonic timing (S5)
- Add distinct log events for half-open and closed transitions (S11, S12)
- Track pre-existing services so second apply_settings_defaults only
  targets newly registered services (P1-2)
- Lazy circuit breaker creation via _get_or_create_cb() (P1-3)
- Reject async callables in sync execute() with TypeError (P1-5)
- Strengthen retry predicate to retry_if_exception_type(Exception) &
  retry_if_not_exception_type(CircuitBreakerOpen) (S1)
- Add lock on _get_wait_strategy cache access (P2-16)
- Truncate raw JSON to 80 chars in override warning (P2-17)
- Warn on non-dict JSON overrides (P2-29)
- Debug log for nesting guard bypass (S13)
- Deep-copy from get() and all_policies() in registry (P1-4)
- Thread-safe registry with threading.Lock (P2-15)
- Robust exception handling in apply_overrides get() (P2-18)
- Log ValidationError details on override failure (P2-19)
- Sanitize service_name via _safe_service_name() (P2-28)
- Warn on non-dict sub-key values in overrides (P2-30)
- Allowlist for is_read_only_plan_operation phases (P2-10)
- Cap retry_auto_debug sleep at 60s (P2-11)
- Use is-not-None instead of falsy checks for error values (P2-12)
- Extend secret regex with bearer, session_id, auth_token,
  refresh_token, client_secret patterns (P2-25)
- Pre-truncate error messages to 2000 chars before regex (P2-26)
- Add upper bounds on retry Settings fields (P2-7)
- Add cross-field validator max_delay >= base_delay (P2-8)
- Case-insensitive backoff strategy validation (P2-21)
- Add half_open_max_successes setting (S10)
- Remove phantom ContextFragment from services __all__ (ImportError fix)
- Export ServiceRetryWiring from application.services package
- Include sanitised error context in TypeError logging fallback
- Initialise RetryContext.attempt_count to 1 for bare context-manager usage
- Introduce CircuitBreakerState StrEnum replacing raw string literals
- Fix vacuous CircuitBreakerOpen propagation assertions in BDD steps
- Replace tautological logging test with structlog capture verification
- Assert circuit breaker existence instead of silently skipping on None
- Add Unicode control-char rejection validator to ServiceRetryPolicy.service_name
- Add name parameter with service= in all log calls
- Add extra="forbid" to all 3 Pydantic models
- Deep-copy _SERVICE_DEFAULTS construction
- Key normalisation (.strip()) in get() and apply_overrides()
- Add cooldown <= recovery_timeout validator
- Async guard on RetryContext.execute()
- Nesting guard on RetryContext.execute()/async_execute()
- stop_after_delay(300.0) on RetryContext
- retry_auto_debug async-only guard, dict result fix, sleep guard
- Retry-attempt logging in RetryContext
- Module-level docs for contextlib.suppress(TypeError) rationale
- Exhaustion log on retry failure
- Startup log in __init__; name=service_name to CircuitBreaker
- log_after_retry guarded to not fire on first-attempt success
- get_retry_decorator now includes logging callbacks
- Changed retry_backoff_strategy from str to RetryStrategy StrEnum

Closes #313
2026-03-11 17:42:13 +00:00

410 lines
15 KiB
Python

"""Step definitions for retry_patterns_coverage_boost.feature.
These steps target specific uncovered lines in retry_patterns.py:
- Lines 592-599: get_retry_decorator with a known category
- RetryContext.execute / async_execute accepts None as valid return value
(the _UNSET sentinel replaced None as the uninitialised marker)
- Line 551/553: auto-debug retry returns dict without error key
- Line 555: auto-debug retry returns non-dict result
- S12 fix: auto-debug returns error dict directly when no debug_callback
- Line 98: log_after_retry success-path TypeError fallback
- Line 82: else None branch in exception ternary of log_after_retry
"""
import asyncio
from types import SimpleNamespace
from behave import given, then, when
from cleveragents.core import retry_patterns as retry_patterns_module
from cleveragents.core.retry_patterns import (
RETRY_CATEGORIES,
RetryContext,
get_retry_decorator,
log_after_retry,
retry_auto_debug,
)
# ---------------------------------------------------------------------------
# Background
# ---------------------------------------------------------------------------
@given("the retry patterns coverage module is imported")
def step_coverage_module_imported(context):
"""Ensure the module is importable."""
assert get_retry_decorator is not None
assert RetryContext is not None
assert retry_auto_debug is not None
# ---------------------------------------------------------------------------
# get_retry_decorator with known category (lines 592-599)
# ---------------------------------------------------------------------------
@when('I request the retry decorator for known category "{category}"')
def step_request_known_category_decorator(context, category):
"""Request a retry decorator for a known category."""
context.known_category = category
context.known_decorator = get_retry_decorator(category)
@then("the returned decorator should use the network category configuration")
def step_verify_network_decorator(context):
"""Verify the decorator was built from the network config."""
# The decorator should be callable (a tenacity retry wrapper)
assert callable(context.known_decorator)
# Confirm the category exists and has the expected max_attempts
config = RETRY_CATEGORIES["network"]
assert config["max_attempts"] == 5
@then("calling the decorated function should succeed")
def step_call_known_decorated_function(context):
"""Decorate and call a simple function to verify it works."""
@context.known_decorator
def sample():
return "known_category_ok"
result = sample()
assert result == "known_category_ok"
@then("the returned decorator should use the database category configuration")
def step_verify_database_decorator(context):
"""Verify the decorator was built from the database config."""
assert callable(context.known_decorator)
config = RETRY_CATEGORIES["database"]
assert config["max_attempts"] == 3
@context.known_decorator
def sample():
return "db_ok"
assert sample() == "db_ok"
# ---------------------------------------------------------------------------
# RetryContext.execute raises RuntimeError for None result (line 469)
# ---------------------------------------------------------------------------
@given('I have a retry context named "{name}"')
def step_create_named_retry_context(context, name):
"""Create a RetryContext with the given operation name."""
context.coverage_retry_ctx = RetryContext(
operation_name=name,
max_attempts=1,
)
@given("I have a function that always returns None")
def step_function_returning_none(context):
"""Create a sync function that returns None."""
def none_func():
return None
context.none_returning_func = none_func
@when("I execute the None-returning function with the retry context")
def step_execute_none_returning(context):
"""Execute the None-returning function via RetryContext.execute."""
try:
context.coverage_retry_ctx.execute(context.none_returning_func)
context.none_exec_error = None
except RuntimeError as exc:
context.none_exec_error = exc
@given("I have an async function that always returns None")
def step_async_function_returning_none(context):
"""Create an async function that returns None."""
async def async_none_func():
return None
context.async_none_returning_func = async_none_func
@when("I execute the async None-returning function with the retry context")
def step_execute_async_none_returning(context):
"""Execute the async None-returning function via RetryContext.async_execute."""
loop = asyncio.new_event_loop()
try:
loop.run_until_complete(
context.coverage_retry_ctx.async_execute(context.async_none_returning_func)
)
context.none_exec_error = None
except RuntimeError as exc:
context.none_exec_error = exc
finally:
loop.close()
@then("None should be accepted as a valid return value")
def step_verify_none_accepted_sync(context):
"""Verify that None is returned without raising RuntimeError.
Since the _UNSET sentinel replaced None as the uninitialized marker,
a function returning None is perfectly valid.
"""
assert context.none_exec_error is None, (
f"Expected no error, got {context.none_exec_error!r}"
)
@then("None should be accepted as a valid async return value")
def step_verify_none_accepted_async(context):
"""Verify that None is returned from async path without RuntimeError."""
assert context.none_exec_error is None, (
f"Expected no error, got {context.none_exec_error!r}"
)
# ---------------------------------------------------------------------------
# Auto-debug retry: dict result without error key (line 551/553)
# ---------------------------------------------------------------------------
@given("I have an async function that returns a dict without an error key")
def step_async_func_dict_no_error(context):
"""Create async func returning a dict with no 'error' key."""
async def success_dict_func():
return {"status": "ok", "data": 42}
context.auto_debug_target = success_dict_func
@when("I apply auto-debug retry for dict success")
def step_apply_auto_debug_dict_success(context):
"""Apply auto-debug and run the function."""
decorated = retry_auto_debug(max_debug_attempts=3)(context.auto_debug_target)
loop = asyncio.new_event_loop()
try:
context.auto_debug_output = loop.run_until_complete(decorated())
finally:
loop.close()
@then("the auto-debug result should be the success dict")
def step_verify_auto_debug_dict(context):
"""Verify the dict was returned directly."""
assert isinstance(context.auto_debug_output, dict)
assert context.auto_debug_output["status"] == "ok"
assert context.auto_debug_output["data"] == 42
# ---------------------------------------------------------------------------
# Auto-debug retry: non-dict result (line 555)
# ---------------------------------------------------------------------------
@given("I have an async function that returns a non-dict result")
def step_async_func_non_dict(context):
"""Create async func returning a plain string."""
async def string_func():
return "plain_string_result"
context.auto_debug_target = string_func
@when("I apply auto-debug retry for non-dict success")
def step_apply_auto_debug_non_dict(context):
"""Apply auto-debug and run the function."""
decorated = retry_auto_debug(max_debug_attempts=3)(context.auto_debug_target)
loop = asyncio.new_event_loop()
try:
context.auto_debug_output = loop.run_until_complete(decorated())
finally:
loop.close()
@then("the auto-debug result should be the non-dict value")
def step_verify_auto_debug_non_dict(context):
"""Verify the non-dict value was returned."""
assert context.auto_debug_output == "plain_string_result"
# ---------------------------------------------------------------------------
# Auto-debug retry: returns error dict when no debug callback (S12 fix)
# ---------------------------------------------------------------------------
@given("I have an async function that returns a dict with an error key")
def step_async_func_returns_error_dict(context):
"""Create a func returning a dict with an ``error`` key.
With the S12 fix, when no ``debug_callback`` is provided the dict
is returned immediately — the loop does **not** exhaust.
"""
async def error_dict_func():
return {"error": "something went wrong", "detail": "transient"}
context.auto_debug_target = error_dict_func
@when("I apply auto-debug retry without debug callback")
def step_apply_auto_debug_no_callback(context):
"""Apply auto-debug with no callback; expect the dict to be returned."""
decorated = retry_auto_debug(max_debug_attempts=2, debug_callback=None)(
context.auto_debug_target
)
loop = asyncio.new_event_loop()
try:
context.auto_debug_output = loop.run_until_complete(decorated())
context.auto_debug_none_error = None
except Exception as exc:
context.auto_debug_output = None
context.auto_debug_none_error = exc
finally:
loop.close()
@then("the auto-debug result should be the error dict returned directly")
def step_verify_auto_debug_error_dict_returned(context):
"""Verify the error dict is returned directly per S12 fix."""
assert context.auto_debug_none_error is None, (
f"Expected no error, got {context.auto_debug_none_error!r}"
)
assert isinstance(context.auto_debug_output, dict)
assert context.auto_debug_output["error"] == "something went wrong"
# ---------------------------------------------------------------------------
# log_after_retry success TypeError fallback (line 98)
# ---------------------------------------------------------------------------
@given("a logger that rejects keyword arguments is installed for coverage boost")
def step_install_keyword_rejecting_logger_boost(context):
"""Install a logger that rejects kwargs to force the TypeError fallback."""
context.boost_logged = []
class KwargRejectingLogger:
def __init__(self, messages):
self._messages = messages
def debug(self, message, *args, **kwargs):
if kwargs:
raise TypeError("kwargs not supported")
self._messages.append(message)
def info(self, *a, **kw):
pass
def warning(self, *a, **kw):
pass
def error(self, *a, **kw):
pass
context.boost_original_logger = retry_patterns_module.logger
retry_patterns_module.logger = KwargRejectingLogger(context.boost_logged)
def restore():
retry_patterns_module.logger = context.boost_original_logger
context.add_cleanup(restore)
@when("I call log_after_retry with a successful outcome")
def step_call_log_after_retry_success(context):
"""Call log_after_retry with outcome.failed=False to hit the success TypeError fallback."""
success_state = SimpleNamespace(
attempt_number=7,
outcome=SimpleNamespace(failed=False),
next_action=None,
)
log_after_retry(success_state)
@then("the fallback success message should be logged")
def step_verify_fallback_success_logged(context):
"""Verify the fallback message was recorded."""
assert "Attempt 7 succeeded" in context.boost_logged, (
f"Expected 'Attempt 7 succeeded' in {context.boost_logged}"
)
# ---------------------------------------------------------------------------
# log_after_retry: else None branch (line 82)
# ---------------------------------------------------------------------------
@given("a logger that accepts all arguments is installed for coverage boost")
def step_install_accepting_logger(context):
"""Install a logger that records all calls without raising."""
context.boost_debug_calls = []
class AcceptingLogger:
def __init__(self, calls):
self._calls = calls
def debug(self, message, *args, **kwargs):
self._calls.append({"msg": message, "kwargs": kwargs})
def info(self, *a, **kw):
pass
def warning(self, *a, **kw):
pass
def error(self, *a, **kw):
pass
context.boost_original_logger = retry_patterns_module.logger
retry_patterns_module.logger = AcceptingLogger(context.boost_debug_calls)
def restore():
retry_patterns_module.logger = context.boost_original_logger
context.add_cleanup(restore)
@given("I have a retry state where outcome.failed toggles between checks")
def step_create_toggling_outcome(context):
"""Create a retry state whose outcome.failed is True first, then False.
Line 71 checks ``retry_state.outcome.failed`` to enter the failure branch.
Line 81 checks it again as part of a ternary for the exception= kwarg.
If the second check returns False, line 82 (``else None``) executes.
We use a property that toggles after the first access.
"""
class TogglingOutcome:
def __init__(self):
self._call_count = 0
@property
def failed(self):
self._call_count += 1
# Line 71: ``retry_state.outcome and retry_state.outcome.failed``
# - outcome truthiness doesn't call .failed
# - first .failed access → call_count=1, returns True (enters if block)
# Line 81: ternary condition checks .failed again
# - second .failed access → call_count=2, returns False → else None
return self._call_count <= 1
def exception(self):
return ValueError("should not be called")
context.toggling_retry_state = SimpleNamespace(
attempt_number=1,
outcome=TogglingOutcome(),
next_action=SimpleNamespace(sleep=0.1),
)
@when("I call log_after_retry with the toggling outcome")
def step_call_log_after_retry_toggling(context):
"""Invoke log_after_retry with the toggling outcome."""
log_after_retry(context.toggling_retry_state)
@then("the exception keyword should resolve to None")
def step_verify_exception_none(context):
"""Verify the logger received exception=None from the else branch."""
# Find the 'Attempt failed, will retry' call
failed_calls = [
c for c in context.boost_debug_calls if c["msg"] == "Attempt failed, will retry"
]
assert len(failed_calls) == 1, (
f"Expected exactly 1 'Attempt failed' log call, got {len(failed_calls)}: "
f"{context.boost_debug_calls}"
)
assert failed_calls[0]["kwargs"].get("exception") is None, (
f"Expected exception=None, got {failed_calls[0]['kwargs'].get('exception')}"
)