4d3499dcfb
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 16s
CI / build (pull_request) Successful in 17s
CI / quality (pull_request) Successful in 18s
CI / security (pull_request) Successful in 35s
CI / typecheck (pull_request) Successful in 38s
CI / unit_tests (pull_request) Successful in 2m43s
CI / integration_tests (pull_request) Successful in 3m20s
CI / docker (pull_request) Successful in 51s
CI / coverage (pull_request) Successful in 6m18s
CI / build (push) Successful in 14s
CI / lint (push) Successful in 21s
CI / quality (push) Successful in 29s
CI / typecheck (push) Successful in 35s
CI / security (push) Successful in 36s
CI / benchmark-regression (push) Has been skipped
CI / unit_tests (push) Successful in 3m17s
CI / docker (push) Successful in 9s
CI / integration_tests (push) Successful in 3m29s
CI / coverage (push) Successful in 7m3s
CI / benchmark-publish (push) Successful in 19m50s
CI / benchmark-regression (pull_request) Successful in 40m6s
Wire per-service retry policies and circuit breakers into the service layer via ServiceRetryWiring, backed by ServiceRetryPolicyRegistry and configurable through Settings environment variables. Production hardening from code review: - Fix TOCTOU race in CircuitBreaker._on_success (half-open state) - Add half-open probe limit to prevent unbounded concurrent requests - Track all exception types for circuit breaker failure counting - Detect async callables wrapped in functools.partial and callable objects - Enforce spec-compliant 2s minimum for linear backoff strategy - Enforce 0.1s floor for fixed backoff strategy - Add retry amplification guard via contextvars nesting depth tracking - Cap total retry wall-clock time at 300s (MAX_RETRY_TOTAL_TIMEOUT) - Sanitize exception messages in retry logs to prevent secret leakage - Fix wrap_service_method TOCTOU by holding cache lock for full operation - Deep-copy default policies to prevent cross-policy mutation - Warn on unknown override keys in apply_overrides - Guard apply_overrides against non-dict and deeply nested JSON values - Read circuit breaker state under lock in is_circuit_open - Catch RecursionError in JSON config parsing - Add total_timeout + nesting guard to retry_service_operation decorator - Extend secret sanitization to Authorization headers, private_key, connection_string, and access_key patterns - Enforce 0.1s floor on jitter backoff strategy - Cache wait strategies per service in ServiceRetryWiring (M3) - Reset failure_count to 0 when entering half-open from open (M6) - Use cached _get_wait_strategy() in execute()/async_execute() - Move circuit-open logging out of _on_failure lock scope to prevent holding the lock during potentially slow I/O (F1) - Pass total_timeout=MAX_RETRY_TOTAL_TIMEOUT to wrap_service_method retry_service_operation call for consistency with execute() (F4) - Capture failure_count into local variable inside lock scope before logging outside the lock, preventing stale reads from concurrent threads in CircuitBreaker.call() and async_call() (F1) - Deep-copy module-level DEFAULT_DATABASE_RETRY and DEFAULT_CIRCUIT_BREAKER in ServiceRetryPolicyRegistry.get() for auto-generated unknown service policies, preventing shared mutable state corruption (F1) - Unify CircuitBreaker to a single threading.Lock for sync and async (P1-1) - Restore BaseException permit in half-open path to prevent permit leak (P1-6) - Prevent CircuitBreakerOpen cascading into failure_count (S2) - Protect all logger calls with contextlib.suppress (S3, S4) - Replace time.time() with time.monotonic() for monotonic timing (S5) - Add distinct log events for half-open and closed transitions (S11, S12) - Track pre-existing services so second apply_settings_defaults only targets newly registered services (P1-2) - Lazy circuit breaker creation via _get_or_create_cb() (P1-3) - Reject async callables in sync execute() with TypeError (P1-5) - Strengthen retry predicate to retry_if_exception_type(Exception) & retry_if_not_exception_type(CircuitBreakerOpen) (S1) - Add lock on _get_wait_strategy cache access (P2-16) - Truncate raw JSON to 80 chars in override warning (P2-17) - Warn on non-dict JSON overrides (P2-29) - Debug log for nesting guard bypass (S13) - Deep-copy from get() and all_policies() in registry (P1-4) - Thread-safe registry with threading.Lock (P2-15) - Robust exception handling in apply_overrides get() (P2-18) - Log ValidationError details on override failure (P2-19) - Sanitize service_name via _safe_service_name() (P2-28) - Warn on non-dict sub-key values in overrides (P2-30) - Allowlist for is_read_only_plan_operation phases (P2-10) - Cap retry_auto_debug sleep at 60s (P2-11) - Use is-not-None instead of falsy checks for error values (P2-12) - Extend secret regex with bearer, session_id, auth_token, refresh_token, client_secret patterns (P2-25) - Pre-truncate error messages to 2000 chars before regex (P2-26) - Add upper bounds on retry Settings fields (P2-7) - Add cross-field validator max_delay >= base_delay (P2-8) - Case-insensitive backoff strategy validation (P2-21) - Add half_open_max_successes setting (S10) - Remove phantom ContextFragment from services __all__ (ImportError fix) - Export ServiceRetryWiring from application.services package - Include sanitised error context in TypeError logging fallback - Initialise RetryContext.attempt_count to 1 for bare context-manager usage - Introduce CircuitBreakerState StrEnum replacing raw string literals - Fix vacuous CircuitBreakerOpen propagation assertions in BDD steps - Replace tautological logging test with structlog capture verification - Assert circuit breaker existence instead of silently skipping on None - Add Unicode control-char rejection validator to ServiceRetryPolicy.service_name - Add name parameter with service= in all log calls - Add extra="forbid" to all 3 Pydantic models - Deep-copy _SERVICE_DEFAULTS construction - Key normalisation (.strip()) in get() and apply_overrides() - Add cooldown <= recovery_timeout validator - Async guard on RetryContext.execute() - Nesting guard on RetryContext.execute()/async_execute() - stop_after_delay(300.0) on RetryContext - retry_auto_debug async-only guard, dict result fix, sleep guard - Retry-attempt logging in RetryContext - Module-level docs for contextlib.suppress(TypeError) rationale - Exhaustion log on retry failure - Startup log in __init__; name=service_name to CircuitBreaker - log_after_retry guarded to not fire on first-attempt success - get_retry_decorator now includes logging callbacks - Changed retry_backoff_strategy from str to RetryStrategy StrEnum Closes #313
265 lines
8.4 KiB
Python
265 lines
8.4 KiB
Python
"""Step definitions for ServiceRetryWiring settings override tests."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
from typing import Any
|
|
|
|
from behave import given, then, when
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Settings individual override scenarios
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given("I have Settings with non-default retry_max_attempts {value:d}")
|
|
def step_settings_max_attempts(context: Any, value: int) -> None:
|
|
import os
|
|
|
|
from cleveragents.config.settings import Settings
|
|
|
|
os.environ["CLEVERAGENTS_RETRY_MAX_ATTEMPTS"] = str(value)
|
|
try:
|
|
context.settings = Settings(
|
|
database_url="sqlite:///test_retry.db",
|
|
mock_providers=True,
|
|
env="test",
|
|
)
|
|
finally:
|
|
os.environ.pop("CLEVERAGENTS_RETRY_MAX_ATTEMPTS", None)
|
|
|
|
|
|
@when("I create a ServiceRetryWiring from those max-attempts Settings")
|
|
def step_create_wiring_max_attempts(context: Any) -> None:
|
|
from cleveragents.application.services.service_retry_wiring import (
|
|
ServiceRetryWiring,
|
|
)
|
|
|
|
context.wiring = ServiceRetryWiring(context.settings)
|
|
|
|
|
|
@given("I have Settings with non-default circuit_breaker_failure_threshold {value:d}")
|
|
def step_settings_cb_threshold(context: Any, value: int) -> None:
|
|
import os
|
|
|
|
from cleveragents.config.settings import Settings
|
|
|
|
os.environ["CLEVERAGENTS_CB_FAILURE_THRESHOLD"] = str(value)
|
|
try:
|
|
context.settings = Settings(
|
|
database_url="sqlite:///test_retry.db",
|
|
mock_providers=True,
|
|
env="test",
|
|
)
|
|
finally:
|
|
os.environ.pop("CLEVERAGENTS_CB_FAILURE_THRESHOLD", None)
|
|
|
|
|
|
@when("I create a ServiceRetryWiring from those cb-threshold Settings")
|
|
def step_create_wiring_cb_threshold(context: Any) -> None:
|
|
from cleveragents.application.services.service_retry_wiring import (
|
|
ServiceRetryWiring,
|
|
)
|
|
|
|
context.wiring = ServiceRetryWiring(context.settings)
|
|
|
|
|
|
@then("the plan_service circuit breaker should have failure_threshold {value:d}")
|
|
def step_check_cb_threshold_override(context: Any, value: int) -> None:
|
|
cb = context.wiring.get_circuit_breaker("plan_service")
|
|
assert cb is not None
|
|
assert cb.failure_threshold == value
|
|
|
|
|
|
@when('I check is_circuit_open for "{name}"')
|
|
def step_check_is_circuit_open_unknown(context: Any, name: str) -> None:
|
|
context.unknown_circuit_open = context.wiring.is_circuit_open(name)
|
|
|
|
|
|
@then("the circuit should report not open")
|
|
def step_verify_circuit_not_open(context: Any) -> None:
|
|
assert context.unknown_circuit_open is False
|
|
|
|
|
|
@when("I async_execute the function with circuit breaker enabled")
|
|
def step_async_execute_with_cb(context: Any) -> None:
|
|
loop = asyncio.new_event_loop()
|
|
try:
|
|
context.async_result = loop.run_until_complete(
|
|
context.wiring.async_execute(
|
|
service_name="plan_service",
|
|
operation_name="test_async_cb_retry",
|
|
func=context.async_test_func,
|
|
idempotent=True,
|
|
)
|
|
)
|
|
context.async_exec_error = None
|
|
except Exception as exc:
|
|
context.async_exec_error = exc
|
|
context.async_result = None
|
|
finally:
|
|
loop.close()
|
|
|
|
|
|
@given("I have Settings with non-default retry_base_delay {value:g}")
|
|
def step_settings_base_delay(context: Any, value: float) -> None:
|
|
import os
|
|
|
|
from cleveragents.config.settings import Settings
|
|
|
|
os.environ["CLEVERAGENTS_RETRY_BASE_DELAY"] = str(value)
|
|
try:
|
|
context.settings = Settings(
|
|
database_url="sqlite:///test_retry.db",
|
|
mock_providers=True,
|
|
env="test",
|
|
)
|
|
finally:
|
|
os.environ.pop("CLEVERAGENTS_RETRY_BASE_DELAY", None)
|
|
|
|
|
|
@when("I create a ServiceRetryWiring from those base-delay Settings")
|
|
def step_create_wiring_base_delay(context: Any) -> None:
|
|
from cleveragents.application.services.service_retry_wiring import (
|
|
ServiceRetryWiring,
|
|
)
|
|
|
|
context.wiring = ServiceRetryWiring(context.settings)
|
|
|
|
|
|
@then("the plan_service policy should have base_delay {value:g}")
|
|
def step_check_base_delay_override(context: Any, value: float) -> None:
|
|
policy = context.wiring.get_policy("plan_service")
|
|
assert policy.retry.base_delay == value
|
|
|
|
|
|
@given("I have Settings with non-default circuit_breaker_recovery_timeout {value:g}")
|
|
def step_settings_cb_recovery_timeout(context: Any, value: float) -> None:
|
|
import os
|
|
|
|
from cleveragents.config.settings import Settings
|
|
|
|
os.environ["CLEVERAGENTS_CB_RECOVERY_TIMEOUT"] = str(value)
|
|
try:
|
|
context.settings = Settings(
|
|
database_url="sqlite:///test_retry.db",
|
|
mock_providers=True,
|
|
env="test",
|
|
)
|
|
finally:
|
|
os.environ.pop("CLEVERAGENTS_CB_RECOVERY_TIMEOUT", None)
|
|
|
|
|
|
@when("I create a ServiceRetryWiring from those recovery-timeout Settings")
|
|
def step_create_wiring_recovery_timeout(context: Any) -> None:
|
|
from cleveragents.application.services.service_retry_wiring import (
|
|
ServiceRetryWiring,
|
|
)
|
|
|
|
context.wiring = ServiceRetryWiring(context.settings)
|
|
|
|
|
|
@then("the plan_service circuit breaker should have recovery_timeout {value:g}")
|
|
def step_check_recovery_timeout_override(context: Any, value: float) -> None:
|
|
cb = context.wiring.get_circuit_breaker("plan_service")
|
|
assert cb is not None
|
|
assert cb.recovery_timeout == value
|
|
|
|
|
|
@given("I have Settings with non-default circuit_breaker_cooldown {value:g}")
|
|
def step_settings_cb_cooldown(context: Any, value: float) -> None:
|
|
import os
|
|
|
|
from cleveragents.config.settings import Settings
|
|
|
|
os.environ["CLEVERAGENTS_CB_COOLDOWN"] = str(value)
|
|
try:
|
|
context.settings = Settings(
|
|
database_url="sqlite:///test_retry.db",
|
|
mock_providers=True,
|
|
env="test",
|
|
)
|
|
finally:
|
|
os.environ.pop("CLEVERAGENTS_CB_COOLDOWN", None)
|
|
|
|
|
|
@when("I create a ServiceRetryWiring from those cooldown Settings")
|
|
def step_create_wiring_cooldown(context: Any) -> None:
|
|
from cleveragents.application.services.service_retry_wiring import (
|
|
ServiceRetryWiring,
|
|
)
|
|
|
|
context.wiring = ServiceRetryWiring(context.settings)
|
|
|
|
|
|
@then("the plan_service circuit breaker should have cooldown_seconds {value:g}")
|
|
def step_check_cooldown_override(context: Any, value: float) -> None:
|
|
cb = context.wiring.get_circuit_breaker("plan_service")
|
|
assert cb is not None
|
|
assert cb.cooldown_seconds == value
|
|
|
|
|
|
@given("I have Settings with non-default retry_max_delay {value:g}")
|
|
def step_settings_max_delay(context: Any, value: float) -> None:
|
|
import os
|
|
|
|
from cleveragents.config.settings import Settings
|
|
|
|
os.environ["CLEVERAGENTS_RETRY_MAX_DELAY"] = str(value)
|
|
try:
|
|
context.settings = Settings(
|
|
database_url="sqlite:///test_retry.db",
|
|
mock_providers=True,
|
|
env="test",
|
|
)
|
|
finally:
|
|
os.environ.pop("CLEVERAGENTS_RETRY_MAX_DELAY", None)
|
|
|
|
|
|
@when("I create a ServiceRetryWiring from those max-delay Settings")
|
|
def step_create_wiring_max_delay(context: Any) -> None:
|
|
from cleveragents.application.services.service_retry_wiring import (
|
|
ServiceRetryWiring,
|
|
)
|
|
|
|
context.wiring = ServiceRetryWiring(context.settings)
|
|
|
|
|
|
@then("the plan_service policy should have max_delay {value:g}")
|
|
def step_check_max_delay_override(context: Any, value: float) -> None:
|
|
policy = context.wiring.get_policy("plan_service")
|
|
assert policy.retry.max_delay == value
|
|
|
|
|
|
@given("I have Settings with non-default retry_jitter {value}")
|
|
def step_settings_jitter(context: Any, value: str) -> None:
|
|
import os
|
|
|
|
from cleveragents.config.settings import Settings
|
|
|
|
os.environ["CLEVERAGENTS_RETRY_JITTER"] = value
|
|
try:
|
|
context.settings = Settings(
|
|
database_url="sqlite:///test_retry.db",
|
|
mock_providers=True,
|
|
env="test",
|
|
)
|
|
finally:
|
|
os.environ.pop("CLEVERAGENTS_RETRY_JITTER", None)
|
|
|
|
|
|
@when("I create a ServiceRetryWiring from those jitter Settings")
|
|
def step_create_wiring_jitter(context: Any) -> None:
|
|
from cleveragents.application.services.service_retry_wiring import (
|
|
ServiceRetryWiring,
|
|
)
|
|
|
|
context.wiring = ServiceRetryWiring(context.settings)
|
|
|
|
|
|
@then("the plan_service policy should have jitter {value}")
|
|
def step_check_jitter_override(context: Any, value: str) -> None:
|
|
policy = context.wiring.get_policy("plan_service")
|
|
expected = value == "True"
|
|
assert policy.retry.jitter == expected
|