cleveragents-core/features/retry_patterns.feature

@retry @patterns
Feature: Retry Patterns Implementation
  As a developer
  I want retry patterns implemented with tenacity
  So that operations can be resilient to transient failures

  Background:
    Given I have the retry patterns module imported

  # -------------------------------------------------------------------
  # Basic Retry Strategies
  # -------------------------------------------------------------------

  @exponential
  Scenario: Basic exponential backoff retry works
    Given I have a function that fails 2 times then succeeds
    When I apply exponential backoff retry with max 3 attempts
    Then the function should eventually succeed
    And the function should be called 3 times

  @exponential @async
  Scenario: Async exponential backoff retry works
    Given I have an async function that fails 2 times then succeeds
    When I apply async exponential backoff retry with max 3 attempts
    Then the async function should eventually succeed
    And the function should be called 3 times

  @timeout
  Scenario: Retry with timeout retries transient failures without waiting
    Given I have a function that fails 1 times then succeeds
    When I apply retry with timeout of 3 attempts and 0.01 seconds
    Then the function should eventually succeed
    And the function should be called 2 times

  @jitter
  Scenario: Retry with jitter prevents thundering herd
    Given I have multiple concurrent operations
    When I apply retry with jitter
    Then the retries should have random delays
    And operations should not retry simultaneously

  @result-retry
  Scenario: Retry on result retries when predicate flags response
    Given I have sequential result payloads requiring a retry
    And I have a retry predicate that checks for retry flag
    When I apply retry on result with max 3 attempts
    Then the decorated function should return the successful payload
    And the function should be called 2 times

  # -------------------------------------------------------------------
  # Network / Provider / File Retry Patterns
  # -------------------------------------------------------------------

  @network
  Scenario: Network operation retry handles connection errors
    Given I have a function that raises NetworkError twice
    When I apply network retry pattern
    Then the function should retry on NetworkError
    And the function should be called 3 times maximum

  @provider
  Scenario: Provider operation retry handles rate limits
    Given I have a function that raises RateLimitError
    When I apply provider retry pattern
    Then the function should retry with exponential jitter
    And the function should respect rate limit backoff

  @file
  Scenario: File operation retry recovers from transient OSError
    Given I have a file operation that fails with OSError once
    When I apply file retry pattern
    Then the file operation should eventually succeed
    And the file operation should be invoked 2 times

  # -------------------------------------------------------------------
  # Circuit Breaker
  # -------------------------------------------------------------------

  @circuit-breaker
  Scenario: Circuit breaker opens after threshold
    Given I have a circuit breaker with threshold 3
    When a function fails 3 times
    Then the circuit breaker should open
    And subsequent calls should fail immediately

  @circuit-breaker
  Scenario: Circuit breaker resets after timeout
    Given I have an open circuit breaker
    When the recovery timeout expires
    Then the circuit breaker should enter half-open state
    And successful calls should close the circuit

  @circuit-breaker
  Scenario: Circuit breaker resets failure counters after closed success
    Given I have a circuit breaker with threshold 2
    And the circuit breaker has recorded failures
    When I execute a successful call while the circuit is closed
    Then the circuit breaker should reset failure tracking after success

  @circuit-breaker
  Scenario: Circuit breaker half-open closes after consecutive successes
    Given I have a half-open circuit breaker with threshold 2
    When I record a successful half-open call
    And I record another successful half-open call
    Then the circuit breaker should close after consecutive successes

  @circuit-breaker
  Scenario: Circuit breaker half-open reopens after failure
    Given I have a half-open circuit breaker with threshold 2
    When I record a successful half-open call
    And I simulate a half-open failure
    Then the circuit breaker should reopen after half-open failure

  @circuit-breaker
  Scenario: Circuit breaker rejects invalid half_open_max_successes
    When I create a CircuitBreaker with half_open_max_successes 0
    Then a ValueError should be raised for invalid half_open_max_successes

  @circuit-breaker
  Scenario: Circuit breaker _should_attempt_reset returns False with no prior failures
    Given I have a fresh circuit breaker with no failures
    When I check _should_attempt_reset on the fresh circuit breaker
    Then _should_attempt_reset should return False

  # -------------------------------------------------------------------
  # Async Circuit Breaker
  # -------------------------------------------------------------------

  @circuit-breaker @async
  Scenario: Async circuit breaker raises immediately while open
    Given I have an async circuit breaker with threshold 1
    When I trigger an async failure on the circuit breaker
    Then the async circuit breaker should raise immediately while open

  @circuit-breaker @async
  Scenario: Async circuit breaker closes after recovery and successful calls
    Given I have an async circuit breaker with threshold 1
    When I trigger an async failure on the circuit breaker
    And the recovery timeout expires
    And I perform two async successful calls after recovery
    Then the async circuit breaker should close after async successes

  @circuit-breaker @async
  Scenario: Async circuit breaker half-open probe limit blocks excess calls
    Given I have an async circuit breaker in half-open with 0 permits
    When I make an async call on the half-open circuit breaker
    Then a CircuitBreakerOpen should be raised for async probe limit

  @circuit-breaker @async
  Scenario: Async circuit breaker restores permits on CancelledError
    Given I have an async circuit breaker in half-open with 1 permit
    When the async call is cancelled mid-execution
    Then the half-open permit should be restored

  # -------------------------------------------------------------------
  # Retry Context
  # -------------------------------------------------------------------

  @retry-context
  Scenario: Retry context tracks attempts
    Given I have a retry context for "test operation"
    When I execute a function that fails twice
    Then the retry context should track 3 attempts
    And the errors should be recorded

  @retry-context
  Scenario: Retry context manager records exceptions raised in block
    Given I have a retry context for "context block"
    When I execute a failing block under the retry context manager
    Then the retry context manager should capture the exception

  @retry-context @async
  Scenario: Async retry context manager records exceptions raised in block
    Given I have a retry context for "async context block"
    When I execute a failing block under the async retry context manager
    Then the async retry context manager should capture the exception

  @retry-context
  Scenario: Retry context manager leaves errors untouched on success
    Given I have a retry context for "successful context block"
    When I execute a successful block under the retry context manager
    Then the retry context manager should not record errors

  @retry-context @async
  Scenario: Async retry context manager leaves errors untouched on success
    Given I have a retry context for "async successful context block"
    When I execute a successful block under the async retry context manager
    Then the async retry context manager should not record errors

  @retry-context @async
  Scenario: Retry context async execute handles transient async failures
    Given I have a retry context for "async execute operation"
    And I have an async function that fails 2 times then succeeds
    When I execute the async function with the retry context
    Then the async retry context execute should succeed after 3 attempts

  # -------------------------------------------------------------------
  # should_retry_result
  # -------------------------------------------------------------------

  @result-retry
  Scenario: should_retry_result handles error payloads
    Given I have a response payload with error "timeout detected"
    When I evaluate the retry decision
    Then the retry decision should be True

  @result-retry
  Scenario: should_retry_result handles server error status codes
    Given I have a response payload with status code 503
    When I evaluate the retry decision
    Then the retry decision should be True

  @result-retry
  Scenario: should_retry_result returns False for None payloads
    Given I have a result payload set to None
    When I evaluate the retry decision
    Then the retry decision should be False

  @result-retry
  Scenario: should_retry_result returns False for non-error dictionaries
    Given I have a response payload with status code 200
    When I evaluate the retry decision
    Then the retry decision should be False

  @result-retry
  Scenario: should_retry_result returns False for non-dictionary results
    Given I have a result payload set to the string "ok"
    When I evaluate the retry decision
    Then the retry decision should be False

  # -------------------------------------------------------------------
  # Retry Categories & Logging
  # -------------------------------------------------------------------

  @categories
  Scenario: Retry categories have correct configurations
    Given I have the retry categories defined
    Then network operations should retry 5 times
    And provider operations should retry 3 times
    And database operations should retry 3 times
    And file operations should retry 3 times

  @categories
  Scenario: Unknown retry category falls back to default decorator
    When I request a retry decorator for "unknown-category"
    And I decorate a function returning "decorated result"
    Then the decorated function should execute successfully

  @categories
  Scenario: Known retry category returns a configured decorator
    When I request a retry decorator for "network"
    And I decorate a function returning "network result"
    Then the decorated function should execute successfully

  @logging
  Scenario: Retry logging handles loggers without keyword support
    Given the retry logger rejects keyword arguments
    When I invoke logging callbacks for failed and successful attempts
    Then the fallback logging messages should be recorded

  @logging
  Scenario: configure_retry_logging sets tenacity logger levels
    When I configure retry logging to warning level
    Then the tenacity loggers should be set to warning

  # -------------------------------------------------------------------
  # Auto-Debug Retry
  # -------------------------------------------------------------------

  @auto-debug
  Scenario: Auto-debug retry pattern works
    Given I have a function that fails with errors
    And I have a debug callback that fixes issues
    When I apply auto-debug retry
    Then the function should retry with debug fixes
    And eventually succeed after debugging

  @auto-debug
  Scenario: Auto-debug retry raises the last error message when unresolved
    Given I have a function that returns persistent error payloads
    And I have a debug callback that never fixes issues
    When I apply auto-debug retry expecting failure
    Then the auto-debug retry should raise the last error message

  @auto-debug
  Scenario: Auto-debug retry logs debug callback failures
    Given I have a function that returns persistent error payloads
    And I have a debug callback that raises errors
    When I apply auto-debug retry expecting failure
    Then the auto-debug retry should raise the debug callback failure
    And the debug callback failures should be counted

  @auto-debug
  Scenario: Auto-debug retry returns result for non-dict success
    Given I have an async function that returns a non-dict value
    When I apply auto-debug retry to the non-dict function
    Then the auto-debug should return the non-dict value directly

  @auto-debug
  Scenario: Auto-debug retry returns None when no errors occur and no last_error
    Given I have an async function that returns error-free dict then stops
    And I have no debug callback
    When I apply auto-debug retry to the error-free function
    Then the auto-debug should return the dict result

  @auto-debug
  Scenario: Auto-debug with fix callback that resolves the error
    Given I have an async function that returns error then succeeds after fix
    And I have a debug callback that returns fixed
    When I apply auto-debug retry with fix callback
    Then the auto-debug should eventually succeed

  # -------------------------------------------------------------------
  # Retry Service Operation Decorators
  # -------------------------------------------------------------------

  @decorator @async
  Scenario: Async retry_service_operation decorator retries correctly
    Given I have an async function that fails once then returns ok
    When I apply retry_service_operation as async and call
    Then the async decorated function should succeed
    And the async decorated function should have been called 2 times

  @decorator @async
  Scenario: Async retry_service_operation nesting guard skips retries
    When I call nested async retry_service_operation decorators
    Then the inner async decorator should execute without retries

  @decorator
  Scenario: Sync decorator with circuit breaker handles open
    Given I have a sync function guarded by retry_service_operation with CB
    When the circuit breaker opens during sync decorated retry
    Then the sync decorator should propagate CircuitBreakerOpen

  @decorator @async
  Scenario: Async retry_service_operation with circuit breaker and CB open
    Given I have an async function guarded by retry_service_operation with CB
    When the circuit breaker opens during async decorated retry
    Then the async decorator should propagate CircuitBreakerOpen

  @decorator
  Scenario: Sync decorator nesting guard delegates to circuit breaker
    When I call a sync decorated function at max nesting depth with CB
    Then the sync decorator nesting guard should delegate to CB

  @decorator
  Scenario: retry_service_operation non-idempotent returns function as-is
    When I create a non-idempotent retry_service_operation decorator
    Then the decorator should return the original function unchanged

  @decorator @async
  Scenario: Async decorator nesting guard delegates to circuit breaker
    When I call an async decorated function at max nesting depth with CB
    Then the async nesting CB guard should delegate successfully

  # -------------------------------------------------------------------
  # _is_async_callable
  # -------------------------------------------------------------------

  @async
  Scenario: _is_async_callable detects partial wrapping async callable object
    When I check _is_async_callable with partial wrapping an async callable object
    Then _is_async_callable should return True for partial async callable

  @async
  Scenario: _is_async_callable detects partial wrapping plain async function
    When I check _is_async_callable with partial wrapping a plain async function
    Then _is_async_callable should return True for partial plain async

  # -------------------------------------------------------------------
  # Structured Log Fallbacks
  # -------------------------------------------------------------------

  @logging
  Scenario: Structured log TypeError fallback for service retry
    When I trigger the structlog TypeError fallback for service retry logging
    Then the fallback logging should not raise any exception