CI: Move benchmark regression job out of the default PR workflow #9040

New issue

Closed

opened 2026-04-14 06:21:29 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-14 06:21:29 +00:00

Owner

Summary

The benchmark-regression job in .forgejo/workflows/ci.yml runs on every pull request and depends on the same docker runners as the rest of the pipeline.
Recent pulls (runs 4821 and 4427) clocked 132 and 99 minutes respectively; the rest of the pipeline finished much earlier, but the benchmark job kept status-check from succeeding.
Benchmarking requires full-history checkout, S3 sync, and ASV runs — all long-lived tasks that do not provide immediate validation for most code changes.

Impact

Developers wait more than an hour for CI feedback even when lint/typecheck/tests pass in under 20 minutes.
Slow runs occupy runners, exacerbating the 20–38 minute queue times observed elsewhere.

Proposal

Extract benchmark-regression into a scheduled workflow (nightly/weekly) that publishes results to S3, and provide a manual dispatch trigger for ad-hoc validation.
Gate benchmark execution behind an opt-in label (for example needs-benchmark) or path filter so only performance-sensitive changes invoke it.
Update CI docs to explain how to trigger the benchmark workflow and how results are reported.
Leave a lightweight smoke benchmark (if needed) in the main workflow with a strict timeout (≤ 5 minutes).

Acceptance Criteria

Pull-request CI completes without waiting on the ASV benchmark job; median turnaround time drops under 30 minutes.
A scheduled or dispatchable benchmark workflow runs at least once successfully and publishes results.
Documentation in docs/development/ci-cd.md (or similar) covers the new trigger process and reporting channel.

## Summary - The `benchmark-regression` job in `.forgejo/workflows/ci.yml` runs on every pull request and depends on the same `docker` runners as the rest of the pipeline. - Recent pulls (runs **4821** and **4427**) clocked **132** and **99** minutes respectively; the rest of the pipeline finished much earlier, but the benchmark job kept `status-check` from succeeding. - Benchmarking requires full-history checkout, S3 sync, and ASV runs — all long-lived tasks that do not provide immediate validation for most code changes. ## Impact - Developers wait more than an hour for CI feedback even when lint/typecheck/tests pass in under 20 minutes. - Slow runs occupy runners, exacerbating the 20–38 minute queue times observed elsewhere. ## Proposal 1. Extract `benchmark-regression` into a scheduled workflow (nightly/weekly) that publishes results to S3, and provide a manual dispatch trigger for ad-hoc validation. 2. Gate benchmark execution behind an opt-in label (for example `needs-benchmark`) or path filter so only performance-sensitive changes invoke it. 3. Update CI docs to explain how to trigger the benchmark workflow and how results are reported. 4. Leave a lightweight smoke benchmark (if needed) in the main workflow with a strict timeout (≤ 5 minutes). ## Acceptance Criteria - Pull-request CI completes without waiting on the ASV benchmark job; median turnaround time drops under 30 minutes. - A scheduled or dispatchable benchmark workflow runs at least once successfully and publishes results. - Documentation in `docs/development/ci-cd.md` (or similar) covers the new trigger process and reporting channel.

HAL9000 referenced this issue

2026-04-14 06:26:37 +00:00

[AUTO-WATCHDOG] Status: System Watchdog Pool Supervisor (Cycle 4) #9043

HAL9000 added this to the v3.9.0 milestone

2026-04-14 06:43:42 +00:00

HAL9000 added the

labels

2026-04-14 06:43:42 +00:00

HAL9000 commented

2026-04-14 07:09:51 +00:00

Author

Owner

✅ Verified — CI improvement: move benchmark regression job out of default PR workflow. MoSCoW: Should-have. Priority: Medium.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — CI improvement: move benchmark regression job out of default PR workflow. MoSCoW: Should-have. Priority: Medium. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

Rows
Columns