Flaky tests: non-deterministic retry pattern validation in retry_patterns_steps.py #10079

Open
opened 2026-04-16 21:42:39 +00:00 by HAL9000 · 0 comments
Owner

Metadata

  • Commit Message: Fix non-deterministic test ordering in retry_patterns_steps.py
  • Branch Name: bugfix/m1-flaky-retry-patterns
  • Type: Bug

Background and Context

Tests in features/steps/retry_patterns_steps.py verify random jitter behavior in retry delays, but the verification logic relies on detecting distinct timestamps. On systems with coarse clock granularity or in parallel test execution, multiple operations may receive identical timestamps, causing the assertion to fail intermittently.

Evidence:

  • step_verify_no_simultaneous_retry() in retry_patterns_steps.py asserts unique_times >= 2
  • The step uses monotonic_ns() for high-resolution timestamps but still depends on loop overhead
  • A comment in the code acknowledges this: "On every platform (even fast CI runners with time.sleep patched to a no-op in a preceding scenario) the sequential loop overhead ensures at least two distinct nanosecond readings"
  • This assumption may not hold on all systems

Root Cause:
The test relies on implementation details (loop overhead) rather than explicit synchronization. The jitter behavior is validated by tenacity's own test suite, so this test is redundant and fragile.

Expected Behavior

Retry pattern tests pass consistently across all CI runners and environments, regardless of system clock granularity or parallel test execution. The test logic is decoupled from timing assumptions and does not rely on loop overhead to produce distinct timestamps.

Acceptance Criteria

  • Retry pattern tests pass consistently on all CI runners
  • No flaky failures in parallel test execution
  • Test logic is decoupled from system clock granularity
  • All retry pattern scenarios pass 100 consecutive runs

Subtasks

  • Reproduce the flaky failure locally or in CI to confirm the root cause
  • Evaluate the three candidate solutions:
    • Replace the assertion with a more robust check that does not depend on timing
    • Add explicit delays between operations to guarantee distinct timestamps
    • Mock the time module to control timestamps explicitly
  • Implement the chosen solution in features/steps/retry_patterns_steps.py
  • Add or update BDD scenarios to cover the fixed behaviour
  • Run the full retry pattern test suite 100 consecutive times to confirm stability
  • Ensure nox -s unit_tests passes with coverage ≥ 97%
  • Update any relevant documentation or comments

Definition of Done

This issue should be closed when:

  • The flaky assertion in step_verify_no_simultaneous_retry() has been replaced with a deterministic, clock-independent check
  • All retry pattern BDD scenarios pass reliably in CI (including parallel runs)
  • nox -s unit_tests and nox -s coverage_report pass with coverage ≥ 97%
  • A PR has been merged on branch bugfix/m1-flaky-retry-patterns with commit message Fix non-deterministic test ordering in retry_patterns_steps.py

Automated by CleverAgents Bot
Agent: new-issue-creator

## Metadata - **Commit Message:** `Fix non-deterministic test ordering in retry_patterns_steps.py` - **Branch Name:** `bugfix/m1-flaky-retry-patterns` - **Type:** Bug ## Background and Context Tests in `features/steps/retry_patterns_steps.py` verify random jitter behavior in retry delays, but the verification logic relies on detecting distinct timestamps. On systems with coarse clock granularity or in parallel test execution, multiple operations may receive identical timestamps, causing the assertion to fail intermittently. **Evidence:** - `step_verify_no_simultaneous_retry()` in `retry_patterns_steps.py` asserts `unique_times >= 2` - The step uses `monotonic_ns()` for high-resolution timestamps but still depends on loop overhead - A comment in the code acknowledges this: *"On every platform (even fast CI runners with `time.sleep` patched to a no-op in a preceding scenario) the sequential loop overhead ensures at least two distinct nanosecond readings"* - This assumption may not hold on all systems **Root Cause:** The test relies on implementation details (loop overhead) rather than explicit synchronization. The jitter behavior is validated by tenacity's own test suite, so this test is redundant and fragile. ## Expected Behavior Retry pattern tests pass consistently across all CI runners and environments, regardless of system clock granularity or parallel test execution. The test logic is decoupled from timing assumptions and does not rely on loop overhead to produce distinct timestamps. ## Acceptance Criteria - [ ] Retry pattern tests pass consistently on all CI runners - [ ] No flaky failures in parallel test execution - [ ] Test logic is decoupled from system clock granularity - [ ] All retry pattern scenarios pass 100 consecutive runs ## Subtasks - [ ] Reproduce the flaky failure locally or in CI to confirm the root cause - [ ] Evaluate the three candidate solutions: - [ ] Replace the assertion with a more robust check that does not depend on timing - [ ] Add explicit delays between operations to guarantee distinct timestamps - [ ] Mock the time module to control timestamps explicitly - [ ] Implement the chosen solution in `features/steps/retry_patterns_steps.py` - [ ] Add or update BDD scenarios to cover the fixed behaviour - [ ] Run the full retry pattern test suite 100 consecutive times to confirm stability - [ ] Ensure `nox -s unit_tests` passes with coverage ≥ 97% - [ ] Update any relevant documentation or comments ## Definition of Done This issue should be closed when: - The flaky assertion in `step_verify_no_simultaneous_retry()` has been replaced with a deterministic, clock-independent check - All retry pattern BDD scenarios pass reliably in CI (including parallel runs) - `nox -s unit_tests` and `nox -s coverage_report` pass with coverage ≥ 97% - A PR has been merged on branch `bugfix/m1-flaky-retry-patterns` with commit message `Fix non-deterministic test ordering in retry_patterns_steps.py` --- **Automated by CleverAgents Bot** Agent: new-issue-creator
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleveragents/cleveragents-core#10079
No description provided.