TEST-INFRA: [flaky-tests] Blocked by tool limitations #3361

Open
opened 2026-04-05 14:33:00 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: fix/test-infra-flaky-tests-tool-access
  • Commit Message: fix(test-infra): expose repository clone and check-run access to flaky-test analysis tooling
  • Milestone: (none — see backlog note below)
  • Parent Epic: #3204

Background

The flaky-test analysis worker is unable to perform its analysis because two critical capabilities are unavailable in the current tool environment:

  1. git clone is not permitted — The worker cannot clone the repository to inspect test history, run tests locally, or examine test output artifacts. Without repository access, it is impossible to identify which tests are flaky, reproduce failures, or correlate test results across runs.

  2. No tool to retrieve check run / CI job information from the Forgejo API — The worker cannot query CI check run results (e.g., /repos/{owner}/{repo}/actions/runs or equivalent check-run endpoints) to identify which tests have historically failed intermittently. Without this data, flaky test detection is entirely blocked.

Impact

The flaky-test analysis agent is completely blocked. It cannot:

  • Identify tests that fail non-deterministically across CI runs
  • Correlate test failures with specific commits or branches
  • Produce a ranked list of flaky tests for remediation
  • Provide any actionable output to developers

Required Remediation

To unblock flaky-test analysis, at least one of the following must be provided:

  • Allow git clone — Permit the worker to clone the repository (read-only) so it can inspect test history and run tests locally
  • Provide a check-run tool — Add a tool (or document the correct API call) that retrieves CI check run results from the Forgejo API, including per-job pass/fail status and test output for historical runs

Subtasks

  • Determine which remediation path is preferred: git clone access or a Forgejo check-run API tool
  • Implement the chosen remediation (allow git clone or add check-run tool to the worker's tool set)
  • Verify the flaky-test worker can successfully retrieve CI run data end-to-end
  • Re-run the flaky-test analysis once access is confirmed
  • Document the tool/API path in the test infrastructure runbook

Definition of Done

  • The flaky-test analysis worker can access repository data (via git clone) or CI check run results (via Forgejo API tool)
  • The worker successfully produces a ranked list of flaky tests without tool-access errors
  • The chosen access method is documented in the test infrastructure runbook
  • All nox stages pass
  • Coverage >= 97%

Backlog note: This issue was discovered during autonomous operation
on milestone v3.7.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.


Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/test-infra-flaky-tests-tool-access` - **Commit Message**: `fix(test-infra): expose repository clone and check-run access to flaky-test analysis tooling` - **Milestone**: *(none — see backlog note below)* - **Parent Epic**: #3204 ## Background The flaky-test analysis worker is unable to perform its analysis because two critical capabilities are unavailable in the current tool environment: 1. **`git clone` is not permitted** — The worker cannot clone the repository to inspect test history, run tests locally, or examine test output artifacts. Without repository access, it is impossible to identify which tests are flaky, reproduce failures, or correlate test results across runs. 2. **No tool to retrieve check run / CI job information from the Forgejo API** — The worker cannot query CI check run results (e.g., `/repos/{owner}/{repo}/actions/runs` or equivalent check-run endpoints) to identify which tests have historically failed intermittently. Without this data, flaky test detection is entirely blocked. ## Impact The flaky-test analysis agent is completely blocked. It cannot: - Identify tests that fail non-deterministically across CI runs - Correlate test failures with specific commits or branches - Produce a ranked list of flaky tests for remediation - Provide any actionable output to developers ## Required Remediation To unblock flaky-test analysis, **at least one** of the following must be provided: - [ ] **Allow `git clone`** — Permit the worker to clone the repository (read-only) so it can inspect test history and run tests locally - [ ] **Provide a check-run tool** — Add a tool (or document the correct API call) that retrieves CI check run results from the Forgejo API, including per-job pass/fail status and test output for historical runs ## Subtasks - [ ] Determine which remediation path is preferred: `git clone` access or a Forgejo check-run API tool - [ ] Implement the chosen remediation (allow `git clone` or add check-run tool to the worker's tool set) - [ ] Verify the flaky-test worker can successfully retrieve CI run data end-to-end - [ ] Re-run the flaky-test analysis once access is confirmed - [ ] Document the tool/API path in the test infrastructure runbook ## Definition of Done - [ ] The flaky-test analysis worker can access repository data (via `git clone`) **or** CI check run results (via Forgejo API tool) - [ ] The worker successfully produces a ranked list of flaky tests without tool-access errors - [ ] The chosen access method is documented in the test infrastructure runbook - [ ] All nox stages pass - [ ] Coverage >= 97% > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.7.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: Backlog (unchanged)
  • Story Points: 1 — XS — This is a tracking issue for a known tool limitation. No implementation work needed until the limitation is resolved.
  • MoSCoW: Could Have — Blocked by external tool limitations. Track for future resolution.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Backlog (unchanged) - **Story Points**: 1 — XS — This is a tracking issue for a known tool limitation. No implementation work needed until the limitation is resolved. - **MoSCoW**: Could Have — Blocked by external tool limitations. Track for future resolution. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#3361
No description provided.