ci: move benchmark regression job out of default PR workflow
CI / helm (pull_request) Successful in 31s
CI / push-validation (pull_request) Successful in 32s
CI / build (pull_request) Successful in 1m2s
CI / lint (pull_request) Successful in 1m7s
CI / quality (pull_request) Successful in 1m21s
CI / typecheck (pull_request) Successful in 1m29s
CI / security (pull_request) Successful in 1m36s
CI / integration_tests (pull_request) Successful in 3m55s
CI / e2e_tests (pull_request) Successful in 4m26s
CI / unit_tests (pull_request) Successful in 5m56s
CI / docker (pull_request) Successful in 1m32s
CI / coverage (pull_request) Successful in 11m8s
CI / status-check (pull_request) Successful in 3s
CI / push-validation (push) Successful in 25s
CI / helm (push) Successful in 30s
CI / build (push) Successful in 51s
CI / lint (push) Successful in 55s
CI / quality (push) Successful in 1m10s
CI / typecheck (push) Successful in 1m31s
CI / security (push) Successful in 1m36s
CI / e2e_tests (push) Successful in 4m10s
CI / integration_tests (push) Successful in 4m20s
CI / unit_tests (push) Successful in 4m28s
CI / docker (push) Successful in 1m42s
CI / coverage (push) Successful in 11m23s
CI / status-check (push) Successful in 4s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (push) Has been skipped
CI / benchmark-publish (push) Failing after 21m14s
CI / benchmark-regression (pull_request) Failing after 31m46s
CI / helm (pull_request) Successful in 31s
CI / push-validation (pull_request) Successful in 32s
CI / build (pull_request) Successful in 1m2s
CI / lint (pull_request) Successful in 1m7s
CI / quality (pull_request) Successful in 1m21s
CI / typecheck (pull_request) Successful in 1m29s
CI / security (pull_request) Successful in 1m36s
CI / integration_tests (pull_request) Successful in 3m55s
CI / e2e_tests (pull_request) Successful in 4m26s
CI / unit_tests (pull_request) Successful in 5m56s
CI / docker (pull_request) Successful in 1m32s
CI / coverage (pull_request) Successful in 11m8s
CI / status-check (pull_request) Successful in 3s
CI / push-validation (push) Successful in 25s
CI / helm (push) Successful in 30s
CI / build (push) Successful in 51s
CI / lint (push) Successful in 55s
CI / quality (push) Successful in 1m10s
CI / typecheck (push) Successful in 1m31s
CI / security (push) Successful in 1m36s
CI / e2e_tests (push) Successful in 4m10s
CI / integration_tests (push) Successful in 4m20s
CI / unit_tests (push) Successful in 4m28s
CI / docker (push) Successful in 1m42s
CI / coverage (push) Successful in 11m23s
CI / status-check (push) Successful in 4s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (push) Has been skipped
CI / benchmark-publish (push) Failing after 21m14s
CI / benchmark-regression (pull_request) Failing after 31m46s
Extract the benchmark-regression job from the default PR CI workflow into a dedicated scheduled workflow (.forgejo/workflows/benchmark-scheduled.yml). The benchmark job was blocking PR feedback for 99-132 minutes even when lint/typecheck/tests passed in under 20 minutes. The new workflow runs: - Nightly regression tests (2 AM UTC) comparing HEAD against master - Weekly full benchmark suite (3 AM UTC Sundays) for trend analysis - Manual dispatch trigger for ad-hoc benchmark validation AWS S3 integration stores benchmark results for historical trend analysis. Documentation updated in docs/development/ci-cd.md. CONTRIBUTORS.md updated with benchmark workflow separation contribution. ISSUES CLOSED: #9040 # Conflicts: # CONTRIBUTORS.md # Conflicts: # CONTRIBUTORS.md
This commit was merged in pull request #9233.
This commit is contained in:
@@ -399,3 +399,74 @@ bash scripts/setup-dev.sh
|
||||
pre-commit install
|
||||
pre-commit install --hook-type commit-msg
|
||||
```
|
||||
|
||||
## Benchmark Workflow
|
||||
|
||||
### Why Benchmarks Are Separate from PR Validation
|
||||
|
||||
The `benchmark-regression` job was removed from the default PR workflow because:
|
||||
|
||||
- **Long runtime**: ASV benchmark runs take 99-132 minutes, blocking PR feedback
|
||||
even when lint/typecheck/tests pass in under 20 minutes.
|
||||
- **Runner contention**: Long-running ASV tasks occupy docker runners, exacerbating
|
||||
queue times for other PRs.
|
||||
- **Low signal for most changes**: The vast majority of code changes do not affect
|
||||
performance; running benchmarks on every PR is wasteful.
|
||||
|
||||
The benchmark workflow now runs on a schedule and can be triggered manually for
|
||||
performance-sensitive changes.
|
||||
|
||||
### Benchmark Workflow Triggers
|
||||
|
||||
The benchmark workflow (`.forgejo/workflows/benchmark-scheduled.yml`) runs:
|
||||
|
||||
| Trigger | Schedule | Job | Purpose |
|
||||
|---------|----------|-----|---------|
|
||||
| Nightly cron | 2 AM UTC daily | `benchmark-regression` | Compare HEAD against master for regressions |
|
||||
| Weekly cron | 3 AM UTC Sundays | `benchmark-full` | Full benchmark suite for trend analysis |
|
||||
| Manual dispatch | On demand | Either | Ad-hoc validation for performance-sensitive PRs |
|
||||
|
||||
### How to Trigger Benchmarks Manually
|
||||
|
||||
To run benchmarks for a specific PR or branch:
|
||||
|
||||
1. Navigate to **Actions** > **Benchmark Regression** in the Forgejo UI.
|
||||
2. Click **Run workflow**.
|
||||
3. Select the branch to benchmark.
|
||||
4. Set `base_sha` to the SHA or branch to compare against (default: `master`).
|
||||
5. Set `run_full_suite` to `true` for the full suite, or `false` (default) for
|
||||
regression-only comparison.
|
||||
|
||||
Alternatively, use the Forgejo API:
|
||||
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: token ${FORGEJO_TOKEN}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"ref": "your-branch", "inputs": {"base_sha": "master", "run_full_suite": "false"}}' \
|
||||
"${FORGEJO_URL}/api/v1/repos/cleveragents/cleveragents-core/actions/workflows/benchmark-scheduled.yml/dispatches"
|
||||
```
|
||||
|
||||
### AWS Credentials Required for S3 Benchmark Storage
|
||||
|
||||
Benchmark results are stored in S3 for historical trend analysis. The following
|
||||
secrets must be configured in **Repository Settings > Actions > Secrets**:
|
||||
|
||||
| Secret | Purpose |
|
||||
|--------|---------|
|
||||
| `AWS_ACCESS_KEY_ID` | AWS credentials for S3 benchmark storage |
|
||||
| `AWS_SECRET_ACCESS_KEY` | AWS credentials for S3 benchmark storage |
|
||||
| `AWS_DEFAULT_REGION` | AWS region for S3 benchmark storage |
|
||||
| `ASV_S3_BUCKET` | S3 bucket name for ASV benchmark storage |
|
||||
|
||||
If these secrets are not configured, benchmark runs will still execute locally
|
||||
but results will not be published to S3 (a warning is emitted instead of failing).
|
||||
|
||||
### Benchmark Results
|
||||
|
||||
Benchmark results are available as:
|
||||
|
||||
- **Forgejo Actions artifacts**: Uploaded as `benchmark-regression-results` or
|
||||
`benchmark-full-results` with 90-day retention.
|
||||
- **S3 storage**: Published to `s3://${ASV_S3_BUCKET}/asv/` when AWS credentials
|
||||
are configured. The HTML report is at `s3://${ASV_S3_BUCKET}/asv/html/`.
|
||||
|
||||
Reference in New Issue
Block a user