TEST-INFRA: [Tool] Add a tool to get check run history for a repository #1742

Open
opened 2026-04-02 23:39:51 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: task/add-check-run-history-tool
  • Commit Message: feat(tools): add tool to retrieve check run history for a repository
  • Milestone: v3.8.0
  • Parent Epic: #1678

Summary

Currently, it is not possible to get the check run history for a repository using the available tools. This makes it impossible to identify flaky tests, as we cannot see the history of check runs for a given PR or commit.

Proposal

Add a new tool that can get the check run history for a repository. The tool should be able to filter by PR, commit, and status.

Background and Context

The CleverAgents platform relies on a rich set of tools (via the MCP Tool Registry) to enable agents to interact with external systems. The CI/CD pipeline and test infrastructure are critical systems that agents must be able to introspect. Without the ability to query check run history, agents cannot identify patterns of flaky tests, cannot correlate failures across commits or PRs, and cannot make informed decisions about test reliability.

This gap was identified as a blocker for the flaky-test tracking work tracked in #1706, which requires historical check run data to detect non-deterministic test behaviour.

Expected Behavior

A new tool is available in the tool registry that:

  • Accepts a repository identifier as input
  • Optionally filters results by PR number, commit SHA, and/or check run status
  • Returns a structured list of check run records including run ID, name, status, conclusion, started/completed timestamps, and associated commit/PR references

Subtasks

  • Design the tool interface: define input parameters (repo, pr, commit, status filter) and output schema
  • Implement the tool using the Forgejo API /repos/{owner}/{repo}/statuses/{sha} and /repos/{owner}/{repo}/commits/{ref}/statuses endpoints
  • Register the new tool in the MCP Tool Registry
  • Tests (Behave): Add BDD scenarios covering successful retrieval, filtering by PR, filtering by commit SHA, filtering by status, and error cases (invalid repo, no results)
  • Tests (Robot): Add integration test verifying the tool can be invoked end-to-end and returns valid check run data
  • Verify coverage ≥ 97% via nox -s coverage_report
  • Run nox (all default sessions), fix any errors

Definition of Done

  • A new tool is available to get the check run history for a repository.
  • The tool can be used to identify flaky tests.
  • The tool supports filtering by PR, commit SHA, and status.
  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/add-check-run-history-tool` - **Commit Message**: `feat(tools): add tool to retrieve check run history for a repository` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Summary Currently, it is not possible to get the check run history for a repository using the available tools. This makes it impossible to identify flaky tests, as we cannot see the history of check runs for a given PR or commit. ## Proposal Add a new tool that can get the check run history for a repository. The tool should be able to filter by PR, commit, and status. ## Background and Context The CleverAgents platform relies on a rich set of tools (via the MCP Tool Registry) to enable agents to interact with external systems. The CI/CD pipeline and test infrastructure are critical systems that agents must be able to introspect. Without the ability to query check run history, agents cannot identify patterns of flaky tests, cannot correlate failures across commits or PRs, and cannot make informed decisions about test reliability. This gap was identified as a blocker for the flaky-test tracking work tracked in #1706, which requires historical check run data to detect non-deterministic test behaviour. ## Expected Behavior A new tool is available in the tool registry that: - Accepts a repository identifier as input - Optionally filters results by PR number, commit SHA, and/or check run status - Returns a structured list of check run records including run ID, name, status, conclusion, started/completed timestamps, and associated commit/PR references ## Subtasks - [ ] Design the tool interface: define input parameters (repo, pr, commit, status filter) and output schema - [ ] Implement the tool using the Forgejo API `/repos/{owner}/{repo}/statuses/{sha}` and `/repos/{owner}/{repo}/commits/{ref}/statuses` endpoints - [ ] Register the new tool in the MCP Tool Registry - [ ] Tests (Behave): Add BDD scenarios covering successful retrieval, filtering by PR, filtering by commit SHA, filtering by status, and error cases (invalid repo, no results) - [ ] Tests (Robot): Add integration test verifying the tool can be invoked end-to-end and returns valid check run data - [ ] Verify coverage ≥ 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done - [ ] A new tool is available to get the check run history for a repository. - [ ] The tool can be used to identify flaky tests. - [ ] The tool supports filtering by PR, commit SHA, and status. - [ ] All subtasks above are completed and checked off. - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-02 23:41:40 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1742
No description provided.