ci: move benchmark regression job out of default PR workflow
CI / helm (pull_request) Successful in 31s
CI / push-validation (pull_request) Successful in 32s
CI / build (pull_request) Successful in 1m2s
CI / lint (pull_request) Successful in 1m7s
CI / quality (pull_request) Successful in 1m21s
CI / typecheck (pull_request) Successful in 1m29s
CI / security (pull_request) Successful in 1m36s
CI / integration_tests (pull_request) Successful in 3m55s
CI / e2e_tests (pull_request) Successful in 4m26s
CI / unit_tests (pull_request) Successful in 5m56s
CI / docker (pull_request) Successful in 1m32s
CI / coverage (pull_request) Successful in 11m8s
CI / status-check (pull_request) Successful in 3s
CI / push-validation (push) Successful in 25s
CI / helm (push) Successful in 30s
CI / build (push) Successful in 51s
CI / lint (push) Successful in 55s
CI / quality (push) Successful in 1m10s
CI / typecheck (push) Successful in 1m31s
CI / security (push) Successful in 1m36s
CI / e2e_tests (push) Successful in 4m10s
CI / integration_tests (push) Successful in 4m20s
CI / unit_tests (push) Successful in 4m28s
CI / docker (push) Successful in 1m42s
CI / coverage (push) Successful in 11m23s
CI / status-check (push) Successful in 4s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (push) Has been skipped
CI / benchmark-publish (push) Failing after 21m14s
CI / benchmark-regression (pull_request) Failing after 31m46s

Extract the benchmark-regression job from the default PR CI workflow
into a dedicated scheduled workflow (.forgejo/workflows/benchmark-scheduled.yml).

The benchmark job was blocking PR feedback for 99-132 minutes even when
lint/typecheck/tests passed in under 20 minutes. The new workflow runs:
- Nightly regression tests (2 AM UTC) comparing HEAD against master
- Weekly full benchmark suite (3 AM UTC Sundays) for trend analysis
- Manual dispatch trigger for ad-hoc benchmark validation

AWS S3 integration stores benchmark results for historical trend analysis.
Documentation updated in docs/development/ci-cd.md.
CONTRIBUTORS.md updated with benchmark workflow separation contribution.

ISSUES CLOSED: #9040

# Conflicts:
#	CONTRIBUTORS.md

# Conflicts:
#	CONTRIBUTORS.md
This commit was merged in pull request #9233.
This commit is contained in:
2026-04-23 14:24:09 +00:00
parent 0e130e39c3
commit 44fc7411f9
3 changed files with 266 additions and 0 deletions
+71
View File
@@ -399,3 +399,74 @@ bash scripts/setup-dev.sh
pre-commit install
pre-commit install --hook-type commit-msg
```
## Benchmark Workflow
### Why Benchmarks Are Separate from PR Validation
The `benchmark-regression` job was removed from the default PR workflow because:
- **Long runtime**: ASV benchmark runs take 99-132 minutes, blocking PR feedback
even when lint/typecheck/tests pass in under 20 minutes.
- **Runner contention**: Long-running ASV tasks occupy docker runners, exacerbating
queue times for other PRs.
- **Low signal for most changes**: The vast majority of code changes do not affect
performance; running benchmarks on every PR is wasteful.
The benchmark workflow now runs on a schedule and can be triggered manually for
performance-sensitive changes.
### Benchmark Workflow Triggers
The benchmark workflow (`.forgejo/workflows/benchmark-scheduled.yml`) runs:
| Trigger | Schedule | Job | Purpose |
|---------|----------|-----|---------|
| Nightly cron | 2 AM UTC daily | `benchmark-regression` | Compare HEAD against master for regressions |
| Weekly cron | 3 AM UTC Sundays | `benchmark-full` | Full benchmark suite for trend analysis |
| Manual dispatch | On demand | Either | Ad-hoc validation for performance-sensitive PRs |
### How to Trigger Benchmarks Manually
To run benchmarks for a specific PR or branch:
1. Navigate to **Actions** > **Benchmark Regression** in the Forgejo UI.
2. Click **Run workflow**.
3. Select the branch to benchmark.
4. Set `base_sha` to the SHA or branch to compare against (default: `master`).
5. Set `run_full_suite` to `true` for the full suite, or `false` (default) for
regression-only comparison.
Alternatively, use the Forgejo API:
```bash
curl -X POST \
-H "Authorization: token ${FORGEJO_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"ref": "your-branch", "inputs": {"base_sha": "master", "run_full_suite": "false"}}' \
"${FORGEJO_URL}/api/v1/repos/cleveragents/cleveragents-core/actions/workflows/benchmark-scheduled.yml/dispatches"
```
### AWS Credentials Required for S3 Benchmark Storage
Benchmark results are stored in S3 for historical trend analysis. The following
secrets must be configured in **Repository Settings > Actions > Secrets**:
| Secret | Purpose |
|--------|---------|
| `AWS_ACCESS_KEY_ID` | AWS credentials for S3 benchmark storage |
| `AWS_SECRET_ACCESS_KEY` | AWS credentials for S3 benchmark storage |
| `AWS_DEFAULT_REGION` | AWS region for S3 benchmark storage |
| `ASV_S3_BUCKET` | S3 bucket name for ASV benchmark storage |
If these secrets are not configured, benchmark runs will still execute locally
but results will not be published to S3 (a warning is emitted instead of failing).
### Benchmark Results
Benchmark results are available as:
- **Forgejo Actions artifacts**: Uploaded as `benchmark-regression-results` or
`benchmark-full-results` with 90-day retention.
- **S3 storage**: Published to `s3://${ASV_S3_BUCKET}/asv/` when AWS credentials
are configured. The HTML report is at `s3://${ASV_S3_BUCKET}/asv/html/`.