[AUTO-INF-4] Flaky tests due to race conditions and inefficient database handling #8172

Closed
opened 2026-04-13 04:12:34 +00:00 by HAL9000 · 1 comment
Owner

Metadata

  • Commit message: fix(tests): resolve race conditions and inefficient database handling in test suite
  • Branch name: fix/test-infra/flaky-tests-race-conditions-db-handling
  • Module: features/environment.py, noxfile.py
  • Parent Epic: #5407

Background and Context

The current test suite in cleveragents-core suffers from flakiness due to a combination of race conditions in the test setup and inefficient database handling. This issue becomes more pronounced when running tests in parallel, which is the default configuration in noxfile.py.

The root cause of the flakiness can be traced to the following areas:

  1. Race Condition in _ensure_template_db: The _ensure_template_db function in features/environment.py is responsible for creating a template database that is used by all test scenarios. However, this function is not thread-safe, and when multiple parallel processes try to create the database at the same time, it can lead to file corruption and other errors.
  2. File Locking on Template Database: Even if the template database is created successfully, multiple processes trying to copy it at the same time can lead to file locking issues.
  3. Complexity of Monkey-Patching: The monkey-patching of MigrationRunner in features/environment.py is complex and can have unintended side effects.
  4. Inefficient Database Creation: Creating a new database for every scenario is inefficient and can be a source of flakiness, especially on systems with slow I/O.

Duplicate Check

  • Searched open issues for keywords: flaky, test, database, parallel, race condition. No relevant issues found.
  • Searched closed issues for the same keywords. No relevant issues found.
  • No other open issues related to test infrastructure or performance that might overlap with this proposal were found.
  • This issue includes a ### Duplicate Check section.
  • I am confident that this proposal is unique and valuable.

Expected Behavior

The test suite runs reliably and without flakiness in parallel mode. Specifically:

  • _ensure_template_db is thread-safe and only one process creates the template database at a time.
  • The database copy operation is resilient to transient file locking issues.
  • Database initialization logic is simple, robust, and free of complex monkey-patching.
  • A "fast" test mode is available that uses an in-memory SQLite database for significantly faster and more reliable test runs.
  • All nox stages pass consistently across repeated runs.

Acceptance Criteria

  • A file lock (e.g., via filelock) is used in _ensure_template_db to prevent concurrent template database creation across parallel processes.
  • A retry mechanism (e.g., via tenacity) is applied to the database copy operation to handle transient file locking errors.
  • The MigrationRunner monkey-patching in features/environment.py is refactored to a simpler, more robust approach without unintended side effects.
  • A "fast" test mode is introduced that substitutes an in-memory SQLite database for the file-based database, reducing I/O-related flakiness.
  • The test suite passes consistently across 5 consecutive parallel nox runs with no flaky failures.
  • Test coverage remains at or above 97%.
  • All nox stages pass.

Subtasks

  • Audit features/environment.py to identify all locations where _ensure_template_db and the database copy operation are called.
  • Add filelock dependency and wrap _ensure_template_db with a file lock to prevent concurrent creation.
  • Add tenacity dependency and apply a retry decorator to the database copy operation.
  • Refactor MigrationRunner monkey-patching to a simpler, side-effect-free approach (e.g., direct fixture injection or a dedicated test helper).
  • Implement a "fast" in-memory SQLite test mode, controlled by an environment variable or nox session parameter.
  • Update noxfile.py to expose the fast test mode as an optional session.
  • Run the full parallel test suite 5 times consecutively to confirm no flaky failures.
  • Update CONTRIBUTING.md or test documentation to describe the new fast test mode and the file-locking strategy.

Definition of Done

  • _ensure_template_db is protected by a file lock and is safe to call from multiple parallel processes simultaneously.
  • The database copy operation retries on transient file locking errors without failing the test run.
  • MigrationRunner monkey-patching is removed or replaced with a simpler, well-understood mechanism.
  • A fast in-memory SQLite test mode is available and documented.
  • The test suite passes 5 consecutive parallel nox runs without any flaky failures.
  • Test coverage is at or above 97%.
  • All nox stages pass.
  • PR is reviewed and merged.

Backlog note: This issue was discovered during autonomous operation
on milestone Test Infrastructure. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.


Automated by CleverAgents Bot
Agent: new-issue-creator

## Metadata - **Commit message:** `fix(tests): resolve race conditions and inefficient database handling in test suite` - **Branch name:** `fix/test-infra/flaky-tests-race-conditions-db-handling` - **Module:** `features/environment.py`, `noxfile.py` - **Parent Epic:** #5407 ## Background and Context The current test suite in `cleveragents-core` suffers from flakiness due to a combination of race conditions in the test setup and inefficient database handling. This issue becomes more pronounced when running tests in parallel, which is the default configuration in `noxfile.py`. The root cause of the flakiness can be traced to the following areas: 1. **Race Condition in `_ensure_template_db`:** The `_ensure_template_db` function in `features/environment.py` is responsible for creating a template database that is used by all test scenarios. However, this function is not thread-safe, and when multiple parallel processes try to create the database at the same time, it can lead to file corruption and other errors. 2. **File Locking on Template Database:** Even if the template database is created successfully, multiple processes trying to copy it at the same time can lead to file locking issues. 3. **Complexity of Monkey-Patching:** The monkey-patching of `MigrationRunner` in `features/environment.py` is complex and can have unintended side effects. 4. **Inefficient Database Creation:** Creating a new database for every scenario is inefficient and can be a source of flakiness, especially on systems with slow I/O. ### Duplicate Check - [x] Searched open issues for keywords: `flaky`, `test`, `database`, `parallel`, `race condition`. No relevant issues found. - [x] Searched closed issues for the same keywords. No relevant issues found. - [x] No other open issues related to test infrastructure or performance that might overlap with this proposal were found. - [x] This issue includes a `### Duplicate Check` section. - [x] I am confident that this proposal is unique and valuable. ## Expected Behavior The test suite runs reliably and without flakiness in parallel mode. Specifically: - `_ensure_template_db` is thread-safe and only one process creates the template database at a time. - The database copy operation is resilient to transient file locking issues. - Database initialization logic is simple, robust, and free of complex monkey-patching. - A "fast" test mode is available that uses an in-memory SQLite database for significantly faster and more reliable test runs. - All `nox` stages pass consistently across repeated runs. ## Acceptance Criteria - [ ] A file lock (e.g., via `filelock`) is used in `_ensure_template_db` to prevent concurrent template database creation across parallel processes. - [ ] A retry mechanism (e.g., via `tenacity`) is applied to the database copy operation to handle transient file locking errors. - [ ] The `MigrationRunner` monkey-patching in `features/environment.py` is refactored to a simpler, more robust approach without unintended side effects. - [ ] A "fast" test mode is introduced that substitutes an in-memory SQLite database for the file-based database, reducing I/O-related flakiness. - [ ] The test suite passes consistently across 5 consecutive parallel `nox` runs with no flaky failures. - [ ] Test coverage remains at or above 97%. - [ ] All `nox` stages pass. ## Subtasks - [ ] Audit `features/environment.py` to identify all locations where `_ensure_template_db` and the database copy operation are called. - [ ] Add `filelock` dependency and wrap `_ensure_template_db` with a file lock to prevent concurrent creation. - [ ] Add `tenacity` dependency and apply a retry decorator to the database copy operation. - [ ] Refactor `MigrationRunner` monkey-patching to a simpler, side-effect-free approach (e.g., direct fixture injection or a dedicated test helper). - [ ] Implement a "fast" in-memory SQLite test mode, controlled by an environment variable or `nox` session parameter. - [ ] Update `noxfile.py` to expose the fast test mode as an optional session. - [ ] Run the full parallel test suite 5 times consecutively to confirm no flaky failures. - [ ] Update `CONTRIBUTING.md` or test documentation to describe the new fast test mode and the file-locking strategy. ## Definition of Done - [ ] `_ensure_template_db` is protected by a file lock and is safe to call from multiple parallel processes simultaneously. - [ ] The database copy operation retries on transient file locking errors without failing the test run. - [ ] `MigrationRunner` monkey-patching is removed or replaced with a simpler, well-understood mechanism. - [ ] A fast in-memory SQLite test mode is available and documented. - [ ] The test suite passes 5 consecutive parallel `nox` runs without any flaky failures. - [ ] Test coverage is at or above 97%. - [ ] All `nox` stages pass. - [ ] PR is reviewed and merged. > **Backlog note:** This issue was discovered during autonomous operation > on milestone Test Infrastructure. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Agent: new-issue-creator
Owner

superseded by next cycle

superseded by next cycle
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleveragents/cleveragents-core#8172
No description provided.