Files
cleveragents-core/features/retry_policy_wiring.feature
Dev User a3ba3c3eaf fix(tests): resolve pre-existing AmbiguousStep collisions in step definitions
Rename step texts to avoid case-sensitive conflicts between different step modules:

- edge_case_plan_steps.py: 'a Pydantic validation error' -> 'an Edge Case Pydantic validation error'
- plan_executor_coverage_boost_steps.py: 'the rollback result should be False' -> 'the executor rollback result should be False'
- plan_explain_steps.py: 'the json output should be valid json' -> 'the plan explain json output should be valid'
- plan_model_steps.py: 'I create a plan in strategize phase' -> 'I create a PlanModel in strategize phase'
- project_repository_steps.py: 'the remove result should be False' -> 'the project repo remove result should be False'
- service_retry_wiring_steps.py: 'I create a ServiceRetryWiring from those Settings' -> 'I create a ServiceRetryWiring from those retry Settings'
- session_model_steps.py: 'I get the session CLI dict' -> 'I get the session model CLI dict'

These pre-existing bugs prevented all behave tests from loading.
2026-05-07 20:05:33 +00:00

590 lines
26 KiB
Gherkin

@retry @wiring
Feature: Retry Policy Wiring for Services
As a developer
I want retry policies wired into service layer operations
So that services are resilient to transient failures
Background:
Given I have the retry policy wiring module imported
# -------------------------------------------------------------------
# Registry Configuration
# -------------------------------------------------------------------
@registry
Scenario: RetryPolicyConfig validates default construction
When I create a default RetryPolicyConfig
Then the retry policy should have max_attempts 3
And the retry policy should have base_delay 1.0
And the retry policy should have max_delay 60.0
And the retry policy should have jitter True
And the retry policy backoff_strategy should be exponential
@registry
Scenario: RetryPolicyConfig enforces max_delay >= base_delay
When I create a RetryPolicyConfig with base_delay 10.0 and max_delay 5.0
Then a validation error should be raised for max_delay
@registry
Scenario: CircuitBreakerConfig validates default construction
When I create a default CircuitBreakerConfig
Then the circuit breaker config should have failure_threshold 5
And the circuit breaker config should have recovery_timeout 60.0
And the circuit breaker config should have half_open_max_successes 2
And the circuit breaker config should have cooldown_seconds 30.0
And the circuit breaker config should be enabled
@registry
Scenario: ServiceRetryPolicy binds retry and circuit breaker to service
When I create a ServiceRetryPolicy for "plan_service"
Then the policy service_name should be "plan_service"
And the policy should have a retry config
And the policy should have a circuit breaker config
@registry
Scenario: ServiceRetryPolicyRegistry returns defaults for known services
Given I have a ServiceRetryPolicyRegistry
When I query the policy for "plan_service"
Then the returned policy should have service_name "plan_service"
And the returned policy should have a non-empty description
@registry
Scenario: ServiceRetryPolicyRegistry returns auto-generated default for unknown services
Given I have a ServiceRetryPolicyRegistry
When I query the policy for "unknown_service"
Then the returned policy should have service_name "unknown_service"
And the returned policy description should contain "Auto-generated"
@registry
Scenario: ServiceRetryPolicyRegistry applies overrides
Given I have a ServiceRetryPolicyRegistry
When I apply overrides setting plan_service max_attempts to 7
Then the plan_service policy should have max_attempts 7
@registry
Scenario: ServiceRetryPolicyRegistry register adds new policy
Given I have a ServiceRetryPolicyRegistry
When I register a custom policy for "custom_service"
Then "custom_service" should appear in registered_services
@registry
Scenario: ServiceRetryPolicyRegistry all_policies returns snapshot
Given I have a ServiceRetryPolicyRegistry
When I call all_policies
Then the all_policies snapshot should contain known service names
@registry
Scenario: Per-service policy defaults with config overrides
Given I have Settings with retry_service_overrides setting plan_service max_attempts to 10
When I create a ServiceRetryWiring from those retry Settings
Then the plan_service policy should have max_attempts 10
@registry
Scenario: Unknown service policy is cached on first get
Given I have a ServiceRetryPolicyRegistry
When I query the policy for "brand_new_service" twice
Then both queries should return the same policy object
@registry
Scenario: apply_overrides rejects service_name mutation
Given I have a ServiceRetryPolicyRegistry
When I apply overrides with service_name mutation for "plan_service"
Then the plan_service policy service_name should still be "plan_service"
@registry
Scenario: apply_overrides catches invalid override gracefully
Given I have a ServiceRetryPolicyRegistry
When I apply overrides with invalid data for "plan_service"
Then the plan_service policy should retain its original max_attempts
@registry
Scenario: apply_overrides handles non-dict override values gracefully
Given I have a ServiceRetryPolicyRegistry
When I apply non-dict overrides for "plan_service"
Then the plan_service policy should be unchanged after non-dict override
@registry
Scenario: Deep-copied defaults prevent cross-policy mutation
Given I have a fresh ServiceRetryPolicyRegistry
When I mutate one service policy retry config directly
Then other service policies should not be affected by the mutation
@registry
Scenario: Auto-generated unknown service policies do not share mutable defaults
Given I have a fresh ServiceRetryPolicyRegistry
When I get auto-generated policies for two unknown services
And I mutate the first unknown service retry config
Then the second unknown service retry config should be unaffected
And the module-level DEFAULT_DATABASE_RETRY should be unaffected
# -------------------------------------------------------------------
# Settings Integration
# -------------------------------------------------------------------
@settings
Scenario: ServiceRetryWiring initialises from Settings
Given I have default Settings
When I create a ServiceRetryWiring from Settings
Then the wiring should have circuit breakers for known services
And the wiring registry should contain known service policies
@settings
Scenario: Settings contain retry configuration keys
Given I have default Settings
Then the settings should have retry_max_attempts field
And the settings should have retry_base_delay field
And the settings should have retry_max_delay field
And the settings should have retry_jitter field
And the settings should have retry_backoff_strategy field
And the settings should have circuit_breaker_failure_threshold field
And the settings should have circuit_breaker_recovery_timeout field
And the settings should have circuit_breaker_cooldown field
And the settings should have retry_service_overrides field
@settings
Scenario: Invalid JSON in retry_service_overrides is ignored gracefully
Given I have Settings with retry_service_overrides set to "not-valid-json"
When I create a ServiceRetryWiring from those gracefully-invalid Settings
Then the wiring should initialise without errors
@settings
Scenario: Non-dict JSON in retry_service_overrides is ignored gracefully
Given I have Settings with retry_service_overrides set to "[1, 2, 3]"
When I create a ServiceRetryWiring from those gracefully-invalid Settings
Then the wiring should initialise without errors
@settings
Scenario: Global retry_backoff_strategy propagated via settings defaults
Given I have Settings with non-default retry_backoff_strategy "fixed"
When I create a ServiceRetryWiring from those backoff-strategy Settings
Then all service policies should have backoff_strategy "fixed"
# -------------------------------------------------------------------
# Retry Execution
# -------------------------------------------------------------------
@retry-execution
Scenario: ServiceRetryWiring executes idempotent operation with retry
Given I have a ServiceRetryWiring instance
And I have a function that fails once then succeeds
When I execute the function through the wiring as an idempotent operation
Then the function should succeed after retry
And the function should have been called 2 times
@retry-execution
Scenario: ServiceRetryWiring skips retry for non-idempotent operations
Given I have a ServiceRetryWiring instance
And I have a function that fails on first call
When I execute the function through the wiring as a non-idempotent operation
Then the function should fail immediately
And the function should have been called exactly 1 time
@retry-execution
Scenario: Structured logging emitted on retry attempts
Given I have a ServiceRetryWiring instance
And I have a function that fails once then succeeds
When I execute the function through the wiring with logging captured
Then structured retry log messages should be present
@retry-execution
Scenario: Retry exhaustion propagates the last exception
Given I have a ServiceRetryWiring instance
And I have a function that always fails with RuntimeError
When I execute the always-failing function through the wiring as idempotent
Then the RuntimeError should propagate after exhausting retries
And the always-failing function should have been called max_attempts times
@retry-execution
Scenario: Retry amplification guard prevents nested retries
Given I have a ServiceRetryWiring instance
When I execute a nested retry operation through the wiring
Then the inner function should have been called only once per outer attempt
@retry-execution
Scenario: Decorator nesting guard prevents retry amplification
Given I have a ServiceRetryWiring instance
When I call a decorated function that nests another decorated call
Then the inner decorated call should execute without retries
@retry-execution
Scenario: Decorator total timeout parameter is honoured
Given I have a ServiceRetryWiring instance
When I create a retry_service_operation decorator with total_timeout 1.0
Then the decorator should accept the total_timeout parameter
@retry-execution
Scenario: wrap_service_method decorator executes the wrapped function
Given I have a ServiceRetryWiring instance
When I decorate and call a function via wrap_service_method
Then the wrapped function should return its expected value
@retry-execution
Scenario: wrap_service_method returns cached decorator on second call
Given I have a ServiceRetryWiring instance
When I call wrap_service_method twice with the same arguments
Then both wrap_service_method calls should return the same object
@retry-execution
Scenario: _get_wait_strategy lazy-builds for unregistered service name
Given I have a ServiceRetryWiring instance
When I execute a succeeding function for an unregistered service name
Then the wait strategy cache should contain the unregistered service
# -------------------------------------------------------------------
# Circuit Breaker
# -------------------------------------------------------------------
@circuit-breaker
Scenario: ServiceRetryWiring respects circuit breaker open state
Given I have a ServiceRetryWiring instance
And the circuit breaker for "plan_service" is open
When I execute a function through the wiring for "plan_service"
Then a CircuitBreakerOpen exception should be raised
@circuit-breaker
Scenario: ServiceRetryWiring resets circuit breaker
Given I have a ServiceRetryWiring instance
And the circuit breaker for "plan_service" is open
When I reset the circuit breaker for "plan_service"
Then the circuit breaker for "plan_service" should be closed
@circuit-breaker
Scenario: ServiceRetryWiring checks circuit open status
Given I have a ServiceRetryWiring instance
When I check the circuit status for "plan_service"
Then the circuit should not be open
@circuit-breaker
Scenario: Circuit breaker half-open recovery with consecutive successes
Given I have a ServiceRetryWiring instance
And the circuit breaker for "plan_service" is in half-open state
When I execute two successful calls through the wiring for "plan_service"
Then the circuit breaker for "plan_service" should be closed
@circuit-breaker
Scenario: Circuit breaker wires cooldown_seconds from config
Given I have a ServiceRetryWiring instance
Then the circuit breaker for "plan_service" should have cooldown_seconds 30.0
@circuit-breaker
Scenario: Circuit breaker wires half_open_max_successes from config
Given I have a ServiceRetryWiring instance
Then the circuit breaker for "plan_service" should have half_open_max_successes 2
@circuit-breaker
Scenario: Circuit breaker with circuit_breaker=None path executes without retry protection
Given I have a ServiceRetryWiring instance for a service without circuit breaker
When I execute a function through the wiring for the unprotected service
Then the function should execute successfully without circuit breaker
@circuit-breaker
Scenario: reset_circuit clears last_failure_time
Given I have a ServiceRetryWiring instance
And the circuit breaker for "plan_service" is open
When I reset the circuit breaker for "plan_service"
Then the circuit breaker for "plan_service" last_failure_time should be None
@circuit-breaker
Scenario: _on_success does not close circuit opened by concurrent failure
Given I have a standalone CircuitBreaker instance
And the standalone circuit breaker state is set to "open"
When I call _on_success on the standalone circuit breaker
Then the standalone circuit breaker state should remain "open"
@circuit-breaker
Scenario: Half-open probe limit blocks excess concurrent requests
Given I have a standalone CircuitBreaker instance in half-open with 0 permits
When I call the standalone circuit breaker with a test function
Then a CircuitBreakerOpen exception should be raised for probe limit
@circuit-breaker
Scenario: Circuit breaker only counts expected_exception for failure tracking
Given I have a standalone CircuitBreaker with expected_exception IOError
When I call it with a function that raises RuntimeError
Then the standalone circuit breaker failure_count should be 0
@circuit-breaker
Scenario: Circuit breaker counts matching expected_exception failures
Given I have a standalone CircuitBreaker with expected_exception IOError
When I call it with a function that raises IOError
Then the standalone circuit breaker failure_count should be 1
@circuit-breaker
Scenario: Half-open failure re-opens the circuit breaker
Given I have a standalone CircuitBreaker instance in half-open state with permits
When I call it with a function that raises an expected exception
Then the standalone circuit breaker state should be "open"
@circuit-breaker
Scenario: Cooldown prevents premature half-open transitions
Given I have a standalone CircuitBreaker with a recent half-open attempt
When I check _should_attempt_reset on the standalone circuit breaker
Then the reset should be denied due to cooldown
@circuit-breaker
Scenario: failure_count is reset when entering half-open from open
Given I have a standalone CircuitBreaker that transitions to half-open
When the circuit breaker transitions from open to half-open
Then the circuit breaker failure_count should be 0
@circuit-breaker @negative
Scenario: Disabled circuit breaker does not block calls
Given I have a ServiceRetryWiring instance with disabled circuit breaker
When I execute a function through the wiring for the disabled-CB service
Then the function should execute successfully without circuit breaker protection
@circuit-breaker @negative
Scenario: Concurrent failures open the circuit breaker
Given I have a standalone CircuitBreaker with threshold 3
When 3 concurrent threads trigger failures on the circuit breaker
Then the standalone circuit breaker state should be "open"
@circuit-breaker @negative
Scenario: Retry exhaustion opens the circuit breaker
Given I have a ServiceRetryWiring instance with low CB threshold
And I have a function that always fails with RuntimeError
When I execute the always-failing function through the wiring as idempotent
Then the circuit breaker for "plan_service" should be open after exhaustion
# -------------------------------------------------------------------
# Sanitization
# -------------------------------------------------------------------
@sanitization
Scenario Outline: Sensitive values are sanitized from error messages
When I sanitize an error containing <sensitive_type> with secret "<secret>"
Then the sanitized output should not contain "<secret>"
And the sanitized output should contain "<redacted_marker>"
Examples:
| sensitive_type | secret | redacted_marker |
| database_url | s3cret | ***@ |
| api_key | sk-12345abcdef | api_key=*** |
| auth_header | sk-live-abc123xyz | Authorization: *** |
| private_key | AAAA-BBBB-CCCC-DDDD | private_key=*** |
| connection_string| Server=db;Password=s3cret| connection_string=*** |
| access_key | AKIAIOSFODNN7EXAMPLE | access_key=*** |
@sanitization
Scenario: Long error messages are truncated at 200 characters
When I sanitize an error message longer than 200 characters
Then the sanitized message should be truncated with ellipsis
And the sanitized message length should not exceed 203
@sanitization
Scenario: _sanitize_error_message returns None for None input
When I sanitize a None error
Then the sanitized result should be None
# -------------------------------------------------------------------
# Read-Only Operations
# -------------------------------------------------------------------
@read-only
Scenario: Guard prevents retries on tool writes in read-only plans
Given I have the read-only plan operation guard
When I check a read_only=True operation
Then the retry guard should indicate read-only
When I check a read_only=False operation
Then the retry guard should indicate not read-only
@read-only
Scenario: plan_phase guard returns True for strategize phase
Given I have the read-only plan operation guard
When I check an operation with plan_phase "strategize"
Then the retry guard should indicate read-only
@read-only
Scenario: plan_phase guard returns False for execute phase
Given I have the read-only plan operation guard
When I check an operation with plan_phase "execute"
Then the retry guard should indicate not read-only
@read-only
Scenario: plan_phase guard returns False for empty phase
Given I have the read-only plan operation guard
When I check an operation with empty plan_phase
Then the retry guard should indicate not read-only
@read-only
Scenario: plan_phase guard returns False for apply phase
Given I have the read-only plan operation guard
When I check an operation with plan_phase "apply"
Then the retry guard should indicate not read-only
@read-only
Scenario: plan_phase guard returns False for non-string truthy values
Given I have the read-only plan operation guard
When I check an operation with non-string plan_phase True
Then the retry guard should indicate not read-only
# -------------------------------------------------------------------
# Enums and Defaults
# -------------------------------------------------------------------
@enums
Scenario: RetryCategory enum covers all categories
When I list all RetryCategory values
Then the categories should include "network"
And the categories should include "provider"
And the categories should include "database"
And the categories should include "file_operation"
@enums
Scenario: RetryStrategy enum covers all strategies
When I list all RetryStrategy values
Then the strategies should include "exponential"
And the strategies should include "linear"
And the strategies should include "fixed"
And the strategies should include "jitter"
And the strategies should include "none"
@enums
Scenario: Category defaults map all categories
When I inspect CATEGORY_DEFAULTS
Then CATEGORY_DEFAULTS should have entries for all RetryCategory values
# -------------------------------------------------------------------
# Backoff Strategy Builders
# -------------------------------------------------------------------
@backoff
Scenario: Exponential backoff with jitter builds correctly
When I build a wait strategy with strategy "exponential" and jitter True
Then the wait strategy should be an exponential jitter strategy
@backoff
Scenario: Exponential backoff without jitter builds correctly
When I build a wait strategy with strategy "exponential" and jitter False
Then the wait strategy should be an exponential strategy
@backoff
Scenario: Exponential backoff enforces minimum base delay
When I build a wait strategy with strategy "exponential" base_delay 0.0 and jitter False
Then the effective base delay should be at least 0.1
@backoff
Scenario: Linear backoff enforces 2-second minimum delay
When I build a linear wait strategy with base_delay 0.5
Then the linear wait value should be at least 2.0
@backoff
Scenario: Fixed backoff enforces 0.1-second minimum delay
When I build a fixed wait strategy with base_delay 0.0
Then the fixed wait value should be at least 0.1
@backoff
Scenario: _build_wait_strategy handles jitter strategy
When I build a jitter wait strategy with base_delay 1.0
Then the jitter wait strategy should not be None
@backoff
Scenario: _build_wait_strategy handles none strategy
When I build a none wait strategy
Then the none wait strategy should produce zero-second wait
@backoff
Scenario: Jitter strategy enforces minimum delay floor
When I build a jitter wait strategy with base_delay 0.0
Then the jitter wait value should be at least 0.1
# -------------------------------------------------------------------
# Async Operations
# -------------------------------------------------------------------
@async
Scenario: Async retry_service_operation retries correctly
Given I have a ServiceRetryWiring instance
And I have an async function that fails once then succeeds
When I execute the async function through async_execute as idempotent
Then the async function should succeed after retry
And the async function should have been called 2 times
@async
Scenario: Async execute skips retry for non-idempotent operations
Given I have a ServiceRetryWiring instance
And I have an async function that always fails
When I execute the async function through async_execute as non-idempotent
Then the async function should fail immediately with RuntimeError
And the async function call count should be 1
@async
Scenario: async_execute raises CircuitBreakerOpen for open circuit
Given I have a ServiceRetryWiring instance
And the circuit breaker for "plan_service" is open
When I async_execute a function for "plan_service" with open circuit
Then a CircuitBreakerOpen exception should be raised for async_execute
@async
Scenario: async_execute nesting guard prevents retry amplification
Given I have a ServiceRetryWiring instance
When I async_execute a nested retry operation through the wiring
Then the async inner function should have been called only once
@async
Scenario: Async retry exhaustion propagates the last exception
Given I have a ServiceRetryWiring instance
And I have an async function that always fails
When I async_execute the always-failing function as idempotent
Then the async RuntimeError should propagate after exhausting retries
And the async always-failing function should have been called max_attempts times
# -------------------------------------------------------------------
# Async Callable Detection
# -------------------------------------------------------------------
@async
Scenario: _is_async_callable detects partial-wrapped async functions
When I check _is_async_callable with a partial-wrapped async function
Then _is_async_callable should return True
@async
Scenario: _is_async_callable detects callable objects with async __call__
When I check _is_async_callable with an async callable object
Then _is_async_callable should return True
@async
Scenario: _is_async_callable returns False for sync functions
When I check _is_async_callable with a regular sync function
Then _is_async_callable should return False
@async
Scenario: _is_async_callable returns False for non-callable objects
When I check _is_async_callable with a non-callable object
Then _is_async_callable should return False
# -------------------------------------------------------------------
# Error Handling & Validation
# -------------------------------------------------------------------
@error-handling @negative
Scenario: RetryPolicyConfig rejects max_attempts below 1
When I attempt to create a RetryPolicyConfig with max_attempts 0
Then a validation error should be raised for invalid retry config
@error-handling @negative
Scenario: CircuitBreakerConfig rejects failure_threshold below 1
When I attempt to create a CircuitBreakerConfig with failure_threshold 0
Then a validation error should be raised for invalid circuit breaker config
@error-handling @negative
Scenario: CircuitBreakerConfig rejects recovery_timeout below 1
When I attempt to create a CircuitBreakerConfig with recovery_timeout 0.0
Then a validation error should be raised for invalid circuit breaker config
@error-handling @negative
Scenario: ServiceRetryPolicy rejects empty service_name
When I attempt to create a ServiceRetryPolicy with empty service_name
Then a validation error should be raised for invalid service policy
@error-handling @negative
Scenario: Negative base_delay is rejected
When I attempt to create a RetryPolicyConfig with base_delay -1.0
Then a validation error should be raised for invalid retry config
@retry-execution
Scenario: ServiceRetryWiring wrap_service_method produces a decorator
Given I have a ServiceRetryWiring instance
When I get a decorator from wrap_service_method for "session_service"
Then the decorator should be callable