cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

perf(ci): optimize benchmark-regression test suite to reduce CI execution time #10846

Open
HAL9000 wants to merge 5 commits from test/ci-execution-time-optimize-benchmark-regression into master
pull from: test/ci-execution-time-optimize-benchmark-regression
merge into: cleveragents:master
Conversation 8 Commits 5 Files Changed 7
+113 -9

5 Commits

Author SHA1 Message Date
controller-ci-rerun 398186a23a chore: re-trigger CI [controller]
CI / e2e_tests (pull_request) Has been cancelled
Details
CI / build (pull_request) Has been cancelled
Details
CI / helm (pull_request) Has been cancelled
Details
CI / push-validation (pull_request) Has been cancelled
Details
CI / lint (pull_request) Successful in 45s
Details
CI / security (pull_request) Successful in 1m15s
Details
CI / quality (pull_request) Successful in 59s
Details
CI / typecheck (pull_request) Successful in 1m35s
Details
CI / integration_tests (pull_request) Failing after 3m1s
Details
CI / unit_tests (pull_request) Failing after 6m41s
Details
CI / docker (pull_request) Has been skipped
Details
CI / coverage (pull_request) Successful in 10m52s
Details
CI / status-check (pull_request) Failing after 3s
Details
2026-06-18 11:01:19 -04:00
CleverAgents Bot 79a59eacb2 ci: stop master workflow on PR updates
CI / lint (pull_request) Has been cancelled
Details
CI / typecheck (pull_request) Has been cancelled
Details
CI / security (pull_request) Has been cancelled
Details
CI / quality (pull_request) Has been cancelled
Details
CI / unit_tests (pull_request) Has been cancelled
Details
CI / integration_tests (pull_request) Has been cancelled
Details
CI / e2e_tests (pull_request) Has been cancelled
Details
CI / coverage (pull_request) Has been cancelled
Details
CI / build (pull_request) Has been cancelled
Details
CI / docker (pull_request) Has been cancelled
Details
CI / helm (pull_request) Has been cancelled
Details
CI / push-validation (pull_request) Has been cancelled
Details
CI / status-check (pull_request) Has been cancelled
Details 
Remove the stale pull_request trigger from master.yml so PR branch commits do not launch the master workflow.

Maintenance patch for PR #10846.
 2026-06-10 20:22:48 -04:00
HAL9000 d9f2201355 fix(ci): remove benchmark_regression from ci.yml status-check requirements
CI / push-validation (pull_request) Successful in 34s
Details
CI / helm (pull_request) Successful in 47s
Details
CI / build (pull_request) Successful in 58s
Details
CI / lint (pull_request) Successful in 1m7s
Details
CI / quality (pull_request) Successful in 1m17s
Details
CI / benchmark-publish (pull_request) Has been skipped
Details
CI / typecheck (pull_request) Successful in 1m37s
Details
CI / security (pull_request) Successful in 1m38s
Details
CI / integration_tests (pull_request) Successful in 4m31s
Details
CI / e2e_tests (pull_request) Successful in 4m48s
Details
CI / coverage (pull_request) Failing after 56s
Details
CI / unit_tests (pull_request) Successful in 6m45s
Details
CI / docker (pull_request) Successful in 2m30s
Details
CI / status-check (pull_request) Failing after 3s
Details
CI / benchmark-regression (pull_request) Failing after 35m5s
Details 
The benchmark_regression job added to ci.yml runs on the standard docker
runner which lacks the performance headroom needed for asv continuous.
The master.yml workflow already runs benchmark_regression_fast on the
dedicated docker-benchmark runner for every PR — duplicating it in ci.yml
on the wrong runner caused both jobs to time out at 35 minutes.

Remove the benchmark_regression job from ci.yml entirely and drop it from
the status-check required-needs list. The master.yml benchmark-regression
job (docker-benchmark, timeout-minutes: 35) remains the authoritative PR
regression gate.
 2026-05-03 00:22:50 +00:00
HAL9000 a49d07a4d6 fix(ci): use benchmark_regression_fast in master.yml and raise timeout to 35 min
CI / helm (pull_request) Successful in 30s
Details
CI / push-validation (pull_request) Successful in 30s
Details
CI / build (pull_request) Successful in 1m0s
Details
CI / quality (pull_request) Successful in 1m14s
Details
CI / lint (pull_request) Successful in 1m19s
Details
CI / benchmark-publish (pull_request) Has been skipped
Details
CI / typecheck (pull_request) Successful in 1m47s
Details
CI / security (pull_request) Successful in 2m7s
Details
CI / integration_tests (pull_request) Successful in 3m57s
Details
CI / e2e_tests (pull_request) Successful in 4m2s
Details
CI / unit_tests (pull_request) Successful in 5m1s
Details
CI / docker (pull_request) Successful in 1m29s
Details
CI / coverage (pull_request) Successful in 11m19s
Details
CI / benchmark_regression (pull_request) Failing after 35m5s
Details
CI / benchmark-regression (pull_request) Failing after 35m5s
Details
CI / status-check (pull_request) Failing after 3s
Details 
- master.yml benchmark-regression job was still calling nox -s benchmark_regression
  (the full suite) on pull_request events; update it to call
  nox -s benchmark_regression_fast so PRs use the fast subset
- Add timeout-minutes: 35 to master.yml benchmark-regression job (was unbounded)
- Raise timeout-minutes in ci.yml benchmark_regression job from 20 to 35 to
  accommodate the actual wall-clock time of asv continuous running both base
  and HEAD commits through the fast subset
 2026-04-24 09:47:14 +00:00
HAL9000 ea4998ba61 perf(ci): optimize benchmark-regression test suite to reduce CI execution time
CI / status-check (pull_request) Blocked by required conditions
Details
CI / benchmark-publish (pull_request) Has been skipped
Details
CI / push-validation (pull_request) Successful in 25s
Details
CI / helm (pull_request) Successful in 33s
Details
CI / build (pull_request) Successful in 54s
Details
CI / lint (pull_request) Successful in 1m17s
Details
CI / typecheck (pull_request) Successful in 1m28s
Details
CI / quality (pull_request) Successful in 1m33s
Details
CI / security (pull_request) Successful in 1m39s
Details
CI / integration_tests (pull_request) Successful in 3m46s
Details
CI / e2e_tests (pull_request) Successful in 4m2s
Details
CI / unit_tests (pull_request) Successful in 4m42s
Details
CI / docker (pull_request) Successful in 1m49s
Details
CI / coverage (pull_request) Successful in 11m51s
Details
CI / benchmark_regression (pull_request) Failing after 28m22s
Details
CI / benchmark-regression (pull_request) Failing after 28m30s
Details 
Added benchmark_regression_fast nox session that excludes the three slowest benchmark suites (IndexingScalingSuite, ContextAssemblyScalingSuite, ExecutionThroughputSuite) from PR regression checks. These suites have timeouts of 300-600 s each and were the primary contributors to the 50+ minute CI execution time.

Added benchmark_regression CI job to ci.yml using the fast session with a 20-minute timeout. Added full benchmark_regression run to the nightly quality workflow so the complete suite still runs on a schedule.

Documented the excluded suites and their timeout characteristics in each benchmark file for future maintainers.

ISSUES CLOSED: #1668
 2026-04-23 17:49:42 +00:00