TEST-INFRA: [Flaky Tests] Analysis of flaky tests blocked by missing tool #1798

New issue

Open

opened 2026-04-02 23:52:15 +00:00 by freemo · 1 comment

freemo commented

2026-04-02 23:52:15 +00:00

Owner

Metadata

Branch: task/flaky-tests-analysis-blocked-missing-tool
Commit Message: docs(ci): record flaky-test analysis findings blocked by missing check-run-history tool
Milestone: v3.7.0
Parent Epic: #1678

Summary

This issue reports the findings of the flaky tests analysis. The analysis was blocked by the lack of a tool to get the check run history for a repository.

Steps Taken

Cloned the repository.
Attempted to query Forgejo for CI run history on recent PRs.
Tried various approaches to get the check run history, including:
- Listing pull requests.
- Getting individual pull request details.
- Listing commits.
- Reading the CI configuration file.
All attempts to get the check run history failed due to tool limitations and truncated output.

Conclusion

It is not possible to identify flaky tests without the ability to get the check run history for a repository.

Recommendation

Prioritize the development of a tool to get the check run history for a repository. See issue #1742.

Subtasks

Document the analysis findings in the project wiki or a dedicated ADR
Confirm that issue #1742 (check-run-history tool) is prioritised to unblock this work
Re-run the flaky test analysis once the tool from #1742 is available
Run nox (all default sessions), fix any errors

Definition of Done

Analysis findings are documented and linked to this issue
Issue #1742 is confirmed as a blocker and prioritised accordingly
Flaky test analysis is re-executed successfully once the check-run-history tool is available
All nox stages pass
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/flaky-tests-analysis-blocked-missing-tool` - **Commit Message**: `docs(ci): record flaky-test analysis findings blocked by missing check-run-history tool` - **Milestone**: v3.7.0 - **Parent Epic**: #1678 ## Summary This issue reports the findings of the flaky tests analysis. The analysis was blocked by the lack of a tool to get the check run history for a repository. ## Steps Taken 1. Cloned the repository. 2. Attempted to query Forgejo for CI run history on recent PRs. 3. Tried various approaches to get the check run history, including: - Listing pull requests. - Getting individual pull request details. - Listing commits. - Reading the CI configuration file. 4. All attempts to get the check run history failed due to tool limitations and truncated output. ## Conclusion It is not possible to identify flaky tests without the ability to get the check run history for a repository. ## Recommendation Prioritize the development of a tool to get the check run history for a repository. See issue #1742. ## Subtasks - [ ] Document the analysis findings in the project wiki or a dedicated ADR - [ ] Confirm that issue #1742 (check-run-history tool) is prioritised to unblock this work - [ ] Re-run the flaky test analysis once the tool from #1742 is available - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done - [ ] Analysis findings are documented and linked to this issue - [ ] Issue #1742 is confirmed as a blocker and prioritised accordingly - [ ] Flaky test analysis is re-executed successfully once the check-run-history tool is available - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.7.0 milestone

2026-04-02 23:52:31 +00:00

freemo added a new dependency

2026-04-02 23:54:37 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo commented

2026-04-03 00:27:16 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo added the

MoSCoW

Could have

label

2026-04-03 00:27:16 +00:00