TEST-INFRA: [ci-execution-time] Run benchmark regression on a less frequent schedule #7432

Open
opened 2026-04-10 19:14:21 +00:00 by HAL9000 · 3 comments
Owner

Metadata

  • Branch: task/ci-execution-time-benchmark-regression-schedule
  • Commit Message: chore(ci): run benchmark-regression on a nightly schedule instead of every PR
  • Milestone: N/A — Backlog (see note below)
  • Parent Epic: #1678

Backlog note: This issue was discovered during autonomous operation
on milestone v3.2.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

Summary

The benchmark-regression CI job runs on every pull request, comparing the performance of the PR branch against master. While this is a valuable check, it is time-consuming and adds unnecessary overhead to every PR's CI cycle, slowing developer feedback loops.

Current Behavior

The benchmark-regression job runs on every pull request, regardless of whether the change is likely to affect performance. This results in:

  • Increased CI execution time for every PR.
  • Increased resource consumption on CI runners.
  • Slower feedback loops for developers working on non-performance-sensitive changes.

Expected Behavior

The benchmark-regression job should run on a less frequent schedule to reduce CI execution time on pull requests:

  • On a nightly schedule (e.g., cron: '0 3 * * *').
  • On demand (e.g., via workflow_dispatch trigger).
  • Optionally, on pull requests targeting master only.
  • Not on every pull request to every feature branch.

Acceptance Criteria

  • The benchmark-regression job is not run on every pull request.
  • The benchmark-regression job is run on a nightly schedule (e.g., via schedule trigger in the workflow).
  • The benchmark-regression job is optionally triggerable on demand via workflow_dispatch.
  • The overall CI execution time for pull requests is measurably reduced.
  • The change is documented in the workflow file with a comment explaining the scheduling rationale.

Supporting Information

  • Related issues: #7421 (Run E2E tests on a less frequent schedule), #7216 (Review and optimize CI job dependencies), #7195 (Parallelize unit and integration test suites), #7187 (Consolidate redundant CI setup steps).
  • The benchmark-regression job depends on lint and typecheck and runs ASV benchmark regression via nox -s benchmark_regression.
  • The benchmark-publish job already runs only on push to master/develop — the same pattern should be applied to benchmark-regression.

Subtasks

  • Review the current benchmark-regression job trigger configuration in .forgejo/workflows/ci.yml.
  • Define the new schedule trigger (nightly cron) and optional workflow_dispatch condition.
  • Update .forgejo/workflows/ci.yml to apply the new conditional execution logic for benchmark-regression.
  • Add a comment in the workflow file explaining why benchmark regression runs on a reduced schedule.
  • Verify the updated workflow triggers correctly in a test branch.
  • Run nox (all default sessions), fix any errors.
  • Verify coverage >= 97% via nox -s coverage_report.

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: new-issue-creator

## Metadata - **Branch**: `task/ci-execution-time-benchmark-regression-schedule` - **Commit Message**: `chore(ci): run benchmark-regression on a nightly schedule instead of every PR` - **Milestone**: N/A — Backlog (see note below) - **Parent Epic**: #1678 > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.2.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. ## Summary The `benchmark-regression` CI job runs on every pull request, comparing the performance of the PR branch against `master`. While this is a valuable check, it is time-consuming and adds unnecessary overhead to every PR's CI cycle, slowing developer feedback loops. ## Current Behavior The `benchmark-regression` job runs on every pull request, regardless of whether the change is likely to affect performance. This results in: - Increased CI execution time for every PR. - Increased resource consumption on CI runners. - Slower feedback loops for developers working on non-performance-sensitive changes. ## Expected Behavior The `benchmark-regression` job should run on a less frequent schedule to reduce CI execution time on pull requests: - On a nightly schedule (e.g., `cron: '0 3 * * *'`). - On demand (e.g., via `workflow_dispatch` trigger). - Optionally, on pull requests targeting `master` only. - Not on every pull request to every feature branch. ## Acceptance Criteria - [ ] The `benchmark-regression` job is not run on every pull request. - [ ] The `benchmark-regression` job is run on a nightly schedule (e.g., via `schedule` trigger in the workflow). - [ ] The `benchmark-regression` job is optionally triggerable on demand via `workflow_dispatch`. - [ ] The overall CI execution time for pull requests is measurably reduced. - [ ] The change is documented in the workflow file with a comment explaining the scheduling rationale. ## Supporting Information - Related issues: #7421 (Run E2E tests on a less frequent schedule), #7216 (Review and optimize CI job dependencies), #7195 (Parallelize unit and integration test suites), #7187 (Consolidate redundant CI setup steps). - The `benchmark-regression` job depends on `lint` and `typecheck` and runs ASV benchmark regression via `nox -s benchmark_regression`. - The `benchmark-publish` job already runs only on push to `master`/`develop` — the same pattern should be applied to `benchmark-regression`. ## Subtasks - [ ] Review the current `benchmark-regression` job trigger configuration in `.forgejo/workflows/ci.yml`. - [ ] Define the new schedule trigger (nightly cron) and optional `workflow_dispatch` condition. - [ ] Update `.forgejo/workflows/ci.yml` to apply the new conditional execution logic for `benchmark-regression`. - [ ] Add a comment in the workflow file explaining why benchmark regression runs on a reduced schedule. - [ ] Verify the updated workflow triggers correctly in a test branch. - [ ] Run `nox` (all default sessions), fix any errors. - [ ] Verify coverage >= 97% via `nox -s coverage_report`. ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: new-issue-creator
Author
Owner

Verified — CI optimization: benchmark regression scheduling. MoSCoW: Could-have. Priority: Low.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — CI optimization: benchmark regression scheduling. MoSCoW: Could-have. Priority: Low. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Author
Owner

Verified — CI optimization: benchmark regression scheduling. MoSCoW: Could-have. Priority: Low.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — CI optimization: benchmark regression scheduling. MoSCoW: Could-have. Priority: Low. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Author
Owner

Verified — CI optimization: benchmark regression scheduling. MoSCoW: Could-have. Priority: Low.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — CI optimization: benchmark regression scheduling. MoSCoW: Could-have. Priority: Low. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#7432
No description provided.