TEST-INFRA: [flaky-tests] Replace time.sleep() with more robust synchronization in async_execution tests #3734

New issue

Open

opened 2026-04-05 22:21:30 +00:00 by freemo · 0 comments

freemo commented

2026-04-05 22:21:30 +00:00

Owner

Metadata

Branch: fix/flaky-tests/async-execution-sleep-replacement
Commit Message: test(async_execution): replace time.sleep() with robust synchronization in Behave steps
Milestone: N/A — Backlog
Parent Epic: #1678

Background and Context

The Behave tests for the async execution service in features/steps/async_execution_steps.py use time.sleep() in several places to wait for asynchronous operations to complete. This can lead to flaky tests, especially when running on heavily loaded systems.

Tests that rely on fixed-time waits are often unreliable because the timing of operations can vary depending on the system load and other factors. A better approach is to use more robust synchronization mechanisms, such as threading.Event or concurrent.futures.Future, to wait for asynchronous operations to complete.

The following instances of time.sleep() have been identified as potential sources of flakiness:

time.sleep(0.001) in a busy-wait loop to ensure that the clock advances before recording a heartbeat.
time.sleep(0.05) to wait for asynchronous operations to complete.
time.sleep() in step_worker_processes_queued and step_all_created_jobs_terminal to poll the status of a job until it reaches a terminal state.

Expected Behavior

All time.sleep() calls in features/steps/async_execution_steps.py are replaced with deterministic synchronization primitives (threading.Event, concurrent.futures.Future, or equivalent), making the tests reliable under varying system load conditions.

Subtasks

Replace the busy-wait loop (with time.sleep(0.001)) with a more robust mechanism for ensuring that the clock has advanced (e.g., monotonic clock comparison or a threading.Event signalled by the worker)
Replace time.sleep(0.05) fixed-time waits with threading.Event.wait() or Future.result() with a reasonable timeout
Replace polling loops in step_worker_processes_queued and step_all_created_jobs_terminal with event-driven or future-based waiting
Ensure all modified tests still pass: nox -s behave (or equivalent)
Run nox (all default sessions), fix any errors
Verify coverage >= 97% via nox -s coverage_report

Definition of Done

This issue is complete when:

All time.sleep() calls in features/steps/async_execution_steps.py have been replaced with more robust synchronization mechanisms.
The tests in features/steps/async_execution_steps.py are still passing.
The tests are more robust and less prone to flakiness on loaded systems.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage >= 97%.

Backlog note: This issue was discovered during autonomous operation
on milestone v3.7.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/flaky-tests/async-execution-sleep-replacement` - **Commit Message**: `test(async_execution): replace time.sleep() with robust synchronization in Behave steps` - **Milestone**: N/A — Backlog - **Parent Epic**: #1678 ## Background and Context The Behave tests for the async execution service in `features/steps/async_execution_steps.py` use `time.sleep()` in several places to wait for asynchronous operations to complete. This can lead to flaky tests, especially when running on heavily loaded systems. Tests that rely on fixed-time waits are often unreliable because the timing of operations can vary depending on the system load and other factors. A better approach is to use more robust synchronization mechanisms, such as `threading.Event` or `concurrent.futures.Future`, to wait for asynchronous operations to complete. The following instances of `time.sleep()` have been identified as potential sources of flakiness: - `time.sleep(0.001)` in a busy-wait loop to ensure that the clock advances before recording a heartbeat. - `time.sleep(0.05)` to wait for asynchronous operations to complete. - `time.sleep()` in `step_worker_processes_queued` and `step_all_created_jobs_terminal` to poll the status of a job until it reaches a terminal state. ## Expected Behavior All `time.sleep()` calls in `features/steps/async_execution_steps.py` are replaced with deterministic synchronization primitives (`threading.Event`, `concurrent.futures.Future`, or equivalent), making the tests reliable under varying system load conditions. ## Subtasks - [ ] Replace the busy-wait loop (with `time.sleep(0.001)`) with a more robust mechanism for ensuring that the clock has advanced (e.g., monotonic clock comparison or a `threading.Event` signalled by the worker) - [ ] Replace `time.sleep(0.05)` fixed-time waits with `threading.Event.wait()` or `Future.result()` with a reasonable timeout - [ ] Replace polling loops in `step_worker_processes_queued` and `step_all_created_jobs_terminal` with event-driven or future-based waiting - [ ] Ensure all modified tests still pass: `nox -s behave` (or equivalent) - [ ] Run `nox` (all default sessions), fix any errors - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done This issue is complete when: - All `time.sleep()` calls in `features/steps/async_execution_steps.py` have been replaced with more robust synchronization mechanisms. - The tests in `features/steps/async_execution_steps.py` are still passing. - The tests are more robust and less prone to flakiness on loaded systems. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.7.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added a new dependency

2026-04-05 22:21:43 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing