TEST-INFRA: [flaky-tests] Unable to analyze CI history #2207

Open
opened 2026-04-03 09:31:14 +00:00 by freemo · 0 comments
Owner

Metadata

  • Branch: task/test-infra-ci-history-analysis-blocked
  • Commit Message: chore(ci): document blocked CI history analysis for flaky test detection
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

An automated agent (ca-test-infra-improver) attempted to analyze CI history to identify flaky tests but was unable to complete the task due to environmental and tooling constraints. This issue documents the blockers so they can be resolved, enabling future automated or manual CI history analysis.

Current Behavior

The CI history analysis task fails entirely due to the following discovered blockers:

  • git clone is not available in the agent execution environment.
  • The read tool consistently fails with a "Maximum call stack size exceeded" error, preventing output from other tools from being consumed.
  • The curl tool fails with an SSL error, preventing direct API calls to the Forgejo server.
  • The available forgejo MCP tools do not expose check runs or commit statuses for pull requests or commits, which are required to identify flaky tests from CI history.

Expected Behavior

An agent (or human) should be able to:

  1. Query CI run history for a given branch or set of commits.
  2. Identify tests that pass on some runs and fail on others (flaky tests).
  3. Produce a report of flaky tests with failure frequency and affected commits.

Acceptance Criteria

  • The environment provides a mechanism (tool or API access) to retrieve CI check run results per commit/PR
  • The read tool stack overflow issue is investigated and resolved or a workaround is documented
  • SSL connectivity from the agent environment to the Forgejo host is verified and fixed if broken
  • A repeatable process (manual or automated) exists for CI history analysis to detect flaky tests

Supporting Information

  • Reported by: ca-test-infra-improver agent
  • Parent Epic: #1678 (CI Execution Time Optimization)
  • Related concern: flaky tests inflate CI wall-clock time and cause false failures, directly impacting the goals of Epic #1678

Subtasks

  • Investigate and resolve the read tool "Maximum call stack size exceeded" error in the agent environment
  • Verify and fix SSL connectivity from the agent execution environment to git.cleverthis.com
  • Expose CI check run / commit status data via an available MCP tool or document the API endpoint to use
  • Re-run CI history analysis once blockers are resolved
  • Document the flaky test findings and open follow-up issues for each confirmed flaky test

Definition of Done

  • All environmental blockers listed above are resolved or have documented workarounds
  • CI history analysis can be performed (automated or manual) without hitting the identified blockers
  • At least one successful CI history analysis run is completed and its findings are recorded
  • All nox stages pass
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/test-infra-ci-history-analysis-blocked` - **Commit Message**: `chore(ci): document blocked CI history analysis for flaky test detection` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context An automated agent (`ca-test-infra-improver`) attempted to analyze CI history to identify flaky tests but was unable to complete the task due to environmental and tooling constraints. This issue documents the blockers so they can be resolved, enabling future automated or manual CI history analysis. ## Current Behavior The CI history analysis task fails entirely due to the following discovered blockers: - `git clone` is not available in the agent execution environment. - The `read` tool consistently fails with a "Maximum call stack size exceeded" error, preventing output from other tools from being consumed. - The `curl` tool fails with an SSL error, preventing direct API calls to the Forgejo server. - The available `forgejo` MCP tools do not expose check runs or commit statuses for pull requests or commits, which are required to identify flaky tests from CI history. ## Expected Behavior An agent (or human) should be able to: 1. Query CI run history for a given branch or set of commits. 2. Identify tests that pass on some runs and fail on others (flaky tests). 3. Produce a report of flaky tests with failure frequency and affected commits. ## Acceptance Criteria - [ ] The environment provides a mechanism (tool or API access) to retrieve CI check run results per commit/PR - [ ] The `read` tool stack overflow issue is investigated and resolved or a workaround is documented - [ ] SSL connectivity from the agent environment to the Forgejo host is verified and fixed if broken - [ ] A repeatable process (manual or automated) exists for CI history analysis to detect flaky tests ## Supporting Information - Reported by: `ca-test-infra-improver` agent - Parent Epic: #1678 (CI Execution Time Optimization) - Related concern: flaky tests inflate CI wall-clock time and cause false failures, directly impacting the goals of Epic #1678 ## Subtasks - [ ] Investigate and resolve the `read` tool "Maximum call stack size exceeded" error in the agent environment - [ ] Verify and fix SSL connectivity from the agent execution environment to `git.cleverthis.com` - [ ] Expose CI check run / commit status data via an available MCP tool or document the API endpoint to use - [ ] Re-run CI history analysis once blockers are resolved - [ ] Document the flaky test findings and open follow-up issues for each confirmed flaky test ## Definition of Done - [ ] All environmental blockers listed above are resolved or have documented workarounds - [ ] CI history analysis can be performed (automated or manual) without hitting the identified blockers - [ ] At least one successful CI history analysis run is completed and its findings are recorded - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 09:31:25 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2207
No description provided.