TEST-INFRA: [ci-pipeline-design] Add a Flaky Test Detection and Retry Mechanism #1885

Open
opened 2026-04-03 00:06:19 +00:00 by freemo · 2 comments
Owner

Metadata

  • Branch: feat/ci-pipeline-design-flaky-test-detection
  • Commit Message: feat(ci): add flaky test detection and retry mechanism to CI pipeline
  • Milestone: v3.2.0
  • Parent Epic: #1678

Description

The current CI pipeline does not have a mechanism for detecting or handling flaky tests. Flaky tests can be a major source of frustration for developers and can lead to a loss of confidence in the test suite. We should add a mechanism to automatically detect and handle flaky tests.

Per the project specification, the CI/CD pipeline is the single source of truth for build and test artifacts and must run all quality gates in isolated, containerized environments. A flaky test detection and retry mechanism strengthens this guarantee by ensuring transient failures do not produce false negatives, while also surfacing genuinely unreliable tests for remediation.

Subtasks

  • Research and select a tool or library for flaky test detection and retries (e.g., pytest-rerunfailures for pytest, or a custom script for Robot Framework / Behave).
  • Integrate the selected tool into noxfile.py and .forgejo/workflows/ci.yml.
  • Configure the CI pipeline to automatically retry failed tests a configurable number of times.
  • Implement a mechanism to report flaky tests to a dedicated channel or issue tracker.
  • Update features/ci_workflow_validation.feature (Behave) to assert retry and flaky-detection steps are present.
  • Update Robot Framework integration tests in robot/ to cover the retry behaviour end-to-end.

Definition of Done

  • The CI pipeline automatically retries failed tests up to the configured retry limit.
  • Flaky tests are automatically detected and reported (e.g., via a dedicated issue or notification).
  • Changes are documented in the project's development documentation.
  • All nox stages pass.
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `feat/ci-pipeline-design-flaky-test-detection` - **Commit Message**: `feat(ci): add flaky test detection and retry mechanism to CI pipeline` - **Milestone**: v3.2.0 - **Parent Epic**: #1678 ## Description The current CI pipeline does not have a mechanism for detecting or handling flaky tests. Flaky tests can be a major source of frustration for developers and can lead to a loss of confidence in the test suite. We should add a mechanism to automatically detect and handle flaky tests. Per the project specification, the CI/CD pipeline is the single source of truth for build and test artifacts and must run all quality gates in isolated, containerized environments. A flaky test detection and retry mechanism strengthens this guarantee by ensuring transient failures do not produce false negatives, while also surfacing genuinely unreliable tests for remediation. ## Subtasks - [ ] Research and select a tool or library for flaky test detection and retries (e.g., `pytest-rerunfailures` for pytest, or a custom script for Robot Framework / Behave). - [ ] Integrate the selected tool into `noxfile.py` and `.forgejo/workflows/ci.yml`. - [ ] Configure the CI pipeline to automatically retry failed tests a configurable number of times. - [ ] Implement a mechanism to report flaky tests to a dedicated channel or issue tracker. - [ ] Update `features/ci_workflow_validation.feature` (Behave) to assert retry and flaky-detection steps are present. - [ ] Update Robot Framework integration tests in `robot/` to cover the retry behaviour end-to-end. ## Definition of Done - [ ] The CI pipeline automatically retries failed tests up to the configured retry limit. - [ ] Flaky tests are automatically detected and reported (e.g., via a dedicated issue or notification). - [ ] Changes are documented in the project's development documentation. - [ ] All nox stages pass. - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.2.0 milestone 2026-04-03 00:07:53 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1885
No description provided.