4d3499dcfb
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 16s
CI / build (pull_request) Successful in 17s
CI / quality (pull_request) Successful in 18s
CI / security (pull_request) Successful in 35s
CI / typecheck (pull_request) Successful in 38s
CI / unit_tests (pull_request) Successful in 2m43s
CI / integration_tests (pull_request) Successful in 3m20s
CI / docker (pull_request) Successful in 51s
CI / coverage (pull_request) Successful in 6m18s
CI / build (push) Successful in 14s
CI / lint (push) Successful in 21s
CI / quality (push) Successful in 29s
CI / typecheck (push) Successful in 35s
CI / security (push) Successful in 36s
CI / benchmark-regression (push) Has been skipped
CI / unit_tests (push) Successful in 3m17s
CI / docker (push) Successful in 9s
CI / integration_tests (push) Successful in 3m29s
CI / coverage (push) Successful in 7m3s
CI / benchmark-publish (push) Successful in 19m50s
CI / benchmark-regression (pull_request) Successful in 40m6s
Wire per-service retry policies and circuit breakers into the service layer via ServiceRetryWiring, backed by ServiceRetryPolicyRegistry and configurable through Settings environment variables. Production hardening from code review: - Fix TOCTOU race in CircuitBreaker._on_success (half-open state) - Add half-open probe limit to prevent unbounded concurrent requests - Track all exception types for circuit breaker failure counting - Detect async callables wrapped in functools.partial and callable objects - Enforce spec-compliant 2s minimum for linear backoff strategy - Enforce 0.1s floor for fixed backoff strategy - Add retry amplification guard via contextvars nesting depth tracking - Cap total retry wall-clock time at 300s (MAX_RETRY_TOTAL_TIMEOUT) - Sanitize exception messages in retry logs to prevent secret leakage - Fix wrap_service_method TOCTOU by holding cache lock for full operation - Deep-copy default policies to prevent cross-policy mutation - Warn on unknown override keys in apply_overrides - Guard apply_overrides against non-dict and deeply nested JSON values - Read circuit breaker state under lock in is_circuit_open - Catch RecursionError in JSON config parsing - Add total_timeout + nesting guard to retry_service_operation decorator - Extend secret sanitization to Authorization headers, private_key, connection_string, and access_key patterns - Enforce 0.1s floor on jitter backoff strategy - Cache wait strategies per service in ServiceRetryWiring (M3) - Reset failure_count to 0 when entering half-open from open (M6) - Use cached _get_wait_strategy() in execute()/async_execute() - Move circuit-open logging out of _on_failure lock scope to prevent holding the lock during potentially slow I/O (F1) - Pass total_timeout=MAX_RETRY_TOTAL_TIMEOUT to wrap_service_method retry_service_operation call for consistency with execute() (F4) - Capture failure_count into local variable inside lock scope before logging outside the lock, preventing stale reads from concurrent threads in CircuitBreaker.call() and async_call() (F1) - Deep-copy module-level DEFAULT_DATABASE_RETRY and DEFAULT_CIRCUIT_BREAKER in ServiceRetryPolicyRegistry.get() for auto-generated unknown service policies, preventing shared mutable state corruption (F1) - Unify CircuitBreaker to a single threading.Lock for sync and async (P1-1) - Restore BaseException permit in half-open path to prevent permit leak (P1-6) - Prevent CircuitBreakerOpen cascading into failure_count (S2) - Protect all logger calls with contextlib.suppress (S3, S4) - Replace time.time() with time.monotonic() for monotonic timing (S5) - Add distinct log events for half-open and closed transitions (S11, S12) - Track pre-existing services so second apply_settings_defaults only targets newly registered services (P1-2) - Lazy circuit breaker creation via _get_or_create_cb() (P1-3) - Reject async callables in sync execute() with TypeError (P1-5) - Strengthen retry predicate to retry_if_exception_type(Exception) & retry_if_not_exception_type(CircuitBreakerOpen) (S1) - Add lock on _get_wait_strategy cache access (P2-16) - Truncate raw JSON to 80 chars in override warning (P2-17) - Warn on non-dict JSON overrides (P2-29) - Debug log for nesting guard bypass (S13) - Deep-copy from get() and all_policies() in registry (P1-4) - Thread-safe registry with threading.Lock (P2-15) - Robust exception handling in apply_overrides get() (P2-18) - Log ValidationError details on override failure (P2-19) - Sanitize service_name via _safe_service_name() (P2-28) - Warn on non-dict sub-key values in overrides (P2-30) - Allowlist for is_read_only_plan_operation phases (P2-10) - Cap retry_auto_debug sleep at 60s (P2-11) - Use is-not-None instead of falsy checks for error values (P2-12) - Extend secret regex with bearer, session_id, auth_token, refresh_token, client_secret patterns (P2-25) - Pre-truncate error messages to 2000 chars before regex (P2-26) - Add upper bounds on retry Settings fields (P2-7) - Add cross-field validator max_delay >= base_delay (P2-8) - Case-insensitive backoff strategy validation (P2-21) - Add half_open_max_successes setting (S10) - Remove phantom ContextFragment from services __all__ (ImportError fix) - Export ServiceRetryWiring from application.services package - Include sanitised error context in TypeError logging fallback - Initialise RetryContext.attempt_count to 1 for bare context-manager usage - Introduce CircuitBreakerState StrEnum replacing raw string literals - Fix vacuous CircuitBreakerOpen propagation assertions in BDD steps - Replace tautological logging test with structlog capture verification - Assert circuit breaker existence instead of silently skipping on None - Add Unicode control-char rejection validator to ServiceRetryPolicy.service_name - Add name parameter with service= in all log calls - Add extra="forbid" to all 3 Pydantic models - Deep-copy _SERVICE_DEFAULTS construction - Key normalisation (.strip()) in get() and apply_overrides() - Add cooldown <= recovery_timeout validator - Async guard on RetryContext.execute() - Nesting guard on RetryContext.execute()/async_execute() - stop_after_delay(300.0) on RetryContext - retry_auto_debug async-only guard, dict result fix, sleep guard - Retry-attempt logging in RetryContext - Module-level docs for contextlib.suppress(TypeError) rationale - Exhaustion log on retry failure - Startup log in __init__; name=service_name to CircuitBreaker - log_after_retry guarded to not fire on first-attempt success - get_retry_decorator now includes logging callbacks - Changed retry_backoff_strategy from str to RetryStrategy StrEnum Closes #313
445 lines
16 KiB
Python
445 lines
16 KiB
Python
"""Step definitions for retry policy model and registry tests."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Any
|
|
|
|
from behave import given, then, when
|
|
from pydantic import ValidationError
|
|
|
|
from cleveragents.domain.models.core.retry_policy import (
|
|
CATEGORY_DEFAULTS,
|
|
CircuitBreakerConfig,
|
|
RetryCategory,
|
|
RetryPolicyConfig,
|
|
RetryStrategy,
|
|
ServiceRetryPolicy,
|
|
ServiceRetryPolicyRegistry,
|
|
)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Background
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given("I have the retry policy wiring module imported")
|
|
def step_import_retry_policy_wiring(context: Any) -> None:
|
|
context.modules_loaded = True
|
|
assert RetryPolicyConfig is not None
|
|
assert CircuitBreakerConfig is not None
|
|
assert ServiceRetryPolicy is not None
|
|
assert ServiceRetryPolicyRegistry is not None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# RetryPolicyConfig
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when("I create a default RetryPolicyConfig")
|
|
def step_create_default_retry_policy(context: Any) -> None:
|
|
context.retry_policy = RetryPolicyConfig()
|
|
|
|
|
|
@then("the retry policy should have max_attempts {value:d}")
|
|
def step_check_max_attempts(context: Any, value: int) -> None:
|
|
assert context.retry_policy.max_attempts == value
|
|
|
|
|
|
@then("the retry policy should have base_delay {value:g}")
|
|
def step_check_base_delay(context: Any, value: float) -> None:
|
|
assert context.retry_policy.base_delay == value
|
|
|
|
|
|
@then("the retry policy should have max_delay {value:g}")
|
|
def step_check_max_delay(context: Any, value: float) -> None:
|
|
assert context.retry_policy.max_delay == value
|
|
|
|
|
|
@then("the retry policy should have jitter {value}")
|
|
def step_check_jitter(context: Any, value: str) -> None:
|
|
assert context.retry_policy.jitter == (value == "True")
|
|
|
|
|
|
@then("the retry policy backoff_strategy should be exponential")
|
|
def step_check_backoff_strategy_exp(context: Any) -> None:
|
|
assert context.retry_policy.backoff_strategy == RetryStrategy.EXPONENTIAL
|
|
|
|
|
|
@when("I create a RetryPolicyConfig with base_delay {base:g} and max_delay {mx:g}")
|
|
def step_create_invalid_retry_policy(context: Any, base: float, mx: float) -> None:
|
|
context.validation_error = None
|
|
try:
|
|
context.retry_policy = RetryPolicyConfig(base_delay=base, max_delay=mx)
|
|
except ValidationError as exc:
|
|
context.validation_error = exc
|
|
|
|
|
|
@then("a validation error should be raised for max_delay")
|
|
def step_check_validation_error(context: Any) -> None:
|
|
assert context.validation_error is not None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# CircuitBreakerConfig
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when("I create a default CircuitBreakerConfig")
|
|
def step_create_default_cb_config(context: Any) -> None:
|
|
context.cb_config = CircuitBreakerConfig()
|
|
|
|
|
|
@then("the circuit breaker config should have failure_threshold {value:d}")
|
|
def step_check_cb_threshold(context: Any, value: int) -> None:
|
|
assert context.cb_config.failure_threshold == value
|
|
|
|
|
|
@then("the circuit breaker config should have recovery_timeout {value:g}")
|
|
def step_check_cb_recovery(context: Any, value: float) -> None:
|
|
assert context.cb_config.recovery_timeout == value
|
|
|
|
|
|
@then("the circuit breaker config should have half_open_max_successes {value:d}")
|
|
def step_check_cb_half_open(context: Any, value: int) -> None:
|
|
assert context.cb_config.half_open_max_successes == value
|
|
|
|
|
|
@then("the circuit breaker config should have cooldown_seconds {value:g}")
|
|
def step_check_cb_cooldown(context: Any, value: float) -> None:
|
|
assert context.cb_config.cooldown_seconds == value
|
|
|
|
|
|
@then("the circuit breaker config should be enabled")
|
|
def step_check_cb_enabled(context: Any) -> None:
|
|
assert context.cb_config.enabled is True
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ServiceRetryPolicy
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when('I create a ServiceRetryPolicy for "{name}"')
|
|
def step_create_service_policy(context: Any, name: str) -> None:
|
|
context.service_policy = ServiceRetryPolicy(service_name=name)
|
|
|
|
|
|
@then('the policy service_name should be "{name}"')
|
|
def step_check_policy_name(context: Any, name: str) -> None:
|
|
assert context.service_policy.service_name == name
|
|
|
|
|
|
@then("the policy should have a retry config")
|
|
def step_check_policy_retry(context: Any) -> None:
|
|
assert context.service_policy.retry is not None
|
|
|
|
|
|
@then("the policy should have a circuit breaker config")
|
|
def step_check_policy_cb(context: Any) -> None:
|
|
assert context.service_policy.circuit_breaker is not None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ServiceRetryPolicyRegistry
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given("I have a ServiceRetryPolicyRegistry")
|
|
def step_create_registry(context: Any) -> None:
|
|
context.registry = ServiceRetryPolicyRegistry()
|
|
|
|
|
|
@when('I query the policy for "{name}"')
|
|
def step_query_policy(context: Any, name: str) -> None:
|
|
context.queried_policy = context.registry.get(name)
|
|
|
|
|
|
@then('the returned policy should have service_name "{name}"')
|
|
def step_check_queried_name(context: Any, name: str) -> None:
|
|
assert context.queried_policy.service_name == name
|
|
|
|
|
|
@then("the returned policy should have a non-empty description")
|
|
def step_check_queried_desc(context: Any) -> None:
|
|
assert context.queried_policy.description != ""
|
|
|
|
|
|
@then('the returned policy description should contain "{text}"')
|
|
def step_check_desc_contains(context: Any, text: str) -> None:
|
|
assert text in context.queried_policy.description
|
|
|
|
|
|
@when("I apply overrides setting plan_service max_attempts to {value:d}")
|
|
def step_apply_overrides(context: Any, value: int) -> None:
|
|
context.registry.apply_overrides(
|
|
{"plan_service": {"retry": {"max_attempts": value}}}
|
|
)
|
|
|
|
|
|
@then("the plan_service policy should have max_attempts {value:d}")
|
|
def step_check_plan_service_max(context: Any, value: int) -> None:
|
|
registry = getattr(context, "registry", None) or context.wiring.registry
|
|
policy = registry.get("plan_service")
|
|
assert policy.retry.max_attempts == value
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Enums
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when("I list all RetryCategory values")
|
|
def step_list_categories(context: Any) -> None:
|
|
context.category_values = [c.value for c in RetryCategory]
|
|
|
|
|
|
@then('the categories should include "{value}"')
|
|
def step_check_category(context: Any, value: str) -> None:
|
|
assert value in context.category_values
|
|
|
|
|
|
@when("I list all RetryStrategy values")
|
|
def step_list_strategies(context: Any) -> None:
|
|
context.strategy_values = [s.value for s in RetryStrategy]
|
|
|
|
|
|
@then('the strategies should include "{value}"')
|
|
def step_check_strategy(context: Any, value: str) -> None:
|
|
assert value in context.strategy_values
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# CATEGORY_DEFAULTS
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when("I inspect CATEGORY_DEFAULTS")
|
|
def step_inspect_defaults(context: Any) -> None:
|
|
context.category_defaults = CATEGORY_DEFAULTS
|
|
|
|
|
|
@then("CATEGORY_DEFAULTS should have entries for all RetryCategory values")
|
|
def step_check_category_defaults(context: Any) -> None:
|
|
for cat in RetryCategory:
|
|
assert cat in context.category_defaults
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Registry register and all_policies
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when('I register a custom policy for "{name}"')
|
|
def step_register_custom(context: Any, name: str) -> None:
|
|
policy = ServiceRetryPolicy(
|
|
service_name=name,
|
|
description="Custom test policy",
|
|
)
|
|
context.registry.register(policy)
|
|
|
|
|
|
@then('"{name}" should appear in registered_services')
|
|
def step_check_registered(context: Any, name: str) -> None:
|
|
assert name in context.registry.registered_services()
|
|
|
|
|
|
@when("I call all_policies")
|
|
def step_call_all_policies(context: Any) -> None:
|
|
context.all_policies_result = context.registry.all_policies()
|
|
|
|
|
|
@then("the all_policies snapshot should contain known service names")
|
|
def step_check_all_policies_dict(context: Any) -> None:
|
|
assert isinstance(context.all_policies_result, dict)
|
|
assert "plan_service" in context.all_policies_result
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Negative validation on domain models
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when("I attempt to create a RetryPolicyConfig with max_attempts {value:d}")
|
|
def step_create_invalid_max_attempts(context: Any, value: int) -> None:
|
|
context.neg_validation_error = None
|
|
try:
|
|
RetryPolicyConfig(max_attempts=value)
|
|
except ValidationError as exc:
|
|
context.neg_validation_error = exc
|
|
|
|
|
|
@then("a validation error should be raised for invalid retry config")
|
|
def step_check_neg_retry_validation(context: Any) -> None:
|
|
assert context.neg_validation_error is not None
|
|
|
|
|
|
@when("I attempt to create a CircuitBreakerConfig with failure_threshold {value:d}")
|
|
def step_create_invalid_cb_threshold(context: Any, value: int) -> None:
|
|
context.neg_cb_validation_error = None
|
|
try:
|
|
CircuitBreakerConfig(failure_threshold=value)
|
|
except ValidationError as exc:
|
|
context.neg_cb_validation_error = exc
|
|
|
|
|
|
@when("I attempt to create a CircuitBreakerConfig with recovery_timeout {value:g}")
|
|
def step_create_invalid_cb_recovery(context: Any, value: float) -> None:
|
|
context.neg_cb_validation_error = None
|
|
try:
|
|
CircuitBreakerConfig(recovery_timeout=value)
|
|
except ValidationError as exc:
|
|
context.neg_cb_validation_error = exc
|
|
|
|
|
|
@then("a validation error should be raised for invalid circuit breaker config")
|
|
def step_check_neg_cb_validation(context: Any) -> None:
|
|
assert context.neg_cb_validation_error is not None
|
|
|
|
|
|
@when("I attempt to create a ServiceRetryPolicy with empty service_name")
|
|
def step_create_empty_service_name(context: Any) -> None:
|
|
context.neg_policy_validation_error = None
|
|
try:
|
|
ServiceRetryPolicy(service_name="")
|
|
except ValidationError as exc:
|
|
context.neg_policy_validation_error = exc
|
|
|
|
|
|
@then("a validation error should be raised for invalid service policy")
|
|
def step_check_neg_policy_validation(context: Any) -> None:
|
|
assert context.neg_policy_validation_error is not None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# apply_overrides safety (H4, M4)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when('I apply overrides with service_name mutation for "{name}"')
|
|
def step_apply_overrides_with_name_mutation(context: Any, name: str) -> None:
|
|
context.registry.apply_overrides(
|
|
{name: {"service_name": "hijacked_name", "retry": {"max_attempts": 4}}}
|
|
)
|
|
|
|
|
|
@then('the plan_service policy service_name should still be "{name}"')
|
|
def step_check_no_name_mutation(context: Any, name: str) -> None:
|
|
policy = context.registry.get(name)
|
|
assert policy.service_name == name
|
|
|
|
|
|
@when('I apply overrides with invalid data for "{name}"')
|
|
def step_apply_invalid_overrides(context: Any, name: str) -> None:
|
|
original = context.registry.get(name)
|
|
context.original_max_attempts = original.retry.max_attempts
|
|
# max_attempts=0 violates ge=1 constraint -> ValidationError
|
|
context.registry.apply_overrides({name: {"retry": {"max_attempts": 0}}})
|
|
|
|
|
|
@then("the plan_service policy should retain its original max_attempts")
|
|
def step_check_original_retained(context: Any) -> None:
|
|
policy = context.registry.get("plan_service")
|
|
assert policy.retry.max_attempts == context.original_max_attempts
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Unknown service caching (L2)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when('I query the policy for "{name}" twice')
|
|
def step_query_twice(context: Any, name: str) -> None:
|
|
context.first_query = context.registry.get(name)
|
|
context.second_query = context.registry.get(name)
|
|
|
|
|
|
@then("both queries should return the same policy object")
|
|
def step_check_same_object(context: Any) -> None:
|
|
# P1-4: get() now returns deep copies, so identity check is replaced
|
|
# with equality. Two consecutive gets for the same service must yield
|
|
# structurally identical policies.
|
|
assert context.first_query == context.second_query
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# X5: Non-dict override values
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when('I apply non-dict overrides for "{name}"')
|
|
def step_apply_non_dict_overrides(context: Any, name: str) -> None:
|
|
context.original_policy = context.registry.get(name).model_copy(deep=True)
|
|
context.registry.apply_overrides({name: "not_a_dict"}) # type: ignore[dict-item]
|
|
|
|
|
|
@then("the plan_service policy should be unchanged after non-dict override")
|
|
def step_check_policy_unchanged_after_non_dict(context: Any) -> None:
|
|
policy = context.registry.get("plan_service")
|
|
assert policy.retry.max_attempts == context.original_policy.retry.max_attempts
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# B6: Deep-copied defaults prevent mutation
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given("I have a fresh ServiceRetryPolicyRegistry")
|
|
def step_create_fresh_registry(context: Any) -> None:
|
|
context.fresh_registry = ServiceRetryPolicyRegistry()
|
|
|
|
|
|
@when("I mutate one service policy retry config directly")
|
|
def step_mutate_policy(context: Any) -> None:
|
|
policy = context.fresh_registry.get("plan_service")
|
|
context.original_session_max = context.fresh_registry.get(
|
|
"session_service"
|
|
).retry.max_attempts
|
|
# Mutate plan_service's retry config
|
|
policy.retry.max_attempts = 99
|
|
|
|
|
|
@then("other service policies should not be affected by the mutation")
|
|
def step_check_no_cross_mutation(context: Any) -> None:
|
|
session_policy = context.fresh_registry.get("session_service")
|
|
assert session_policy.retry.max_attempts == context.original_session_max
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# F1 (round 6): Auto-generated unknown service policies isolation
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when("I get auto-generated policies for two unknown services")
|
|
def step_get_two_unknown_policies(context: Any) -> None:
|
|
context.unknown_a = context.fresh_registry.get("auto_gen_svc_alpha")
|
|
context.unknown_b = context.fresh_registry.get("auto_gen_svc_beta")
|
|
|
|
|
|
@when("I mutate the first unknown service retry config")
|
|
def step_mutate_first_unknown(context: Any) -> None:
|
|
from cleveragents.domain.models.core.retry_policy import DEFAULT_DATABASE_RETRY
|
|
|
|
context.default_max_before = DEFAULT_DATABASE_RETRY.max_attempts
|
|
context.unknown_b_max_before = context.unknown_b.retry.max_attempts
|
|
# Mutate only the first unknown service's retry config
|
|
context.unknown_a.retry.max_attempts = 77
|
|
|
|
|
|
@then("the second unknown service retry config should be unaffected")
|
|
def step_check_second_unknown_unaffected(context: Any) -> None:
|
|
assert context.unknown_b.retry.max_attempts == context.unknown_b_max_before, (
|
|
f"Second unknown service was corrupted: "
|
|
f"{context.unknown_b.retry.max_attempts} != {context.unknown_b_max_before}"
|
|
)
|
|
|
|
|
|
@then("the module-level DEFAULT_DATABASE_RETRY should be unaffected")
|
|
def step_check_default_database_retry_unaffected(context: Any) -> None:
|
|
from cleveragents.domain.models.core.retry_policy import DEFAULT_DATABASE_RETRY
|
|
|
|
assert DEFAULT_DATABASE_RETRY.max_attempts == context.default_max_before, (
|
|
f"DEFAULT_DATABASE_RETRY was corrupted: "
|
|
f"{DEFAULT_DATABASE_RETRY.max_attempts} != {context.default_max_before}"
|
|
)
|