cleveragents/cleveractors-core
0
0
Fork 0
Code Issues 17 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

fix(ci): make benchmark_regression job fast and correctly fail on regressions #87

Open
CoreRasurae wants to merge 1 commits from task/m2-quick-benchmark-regression-ci into master
pull from: task/m2-quick-benchmark-regression-ci
merge into: cleveragents:master
Conversation 1 Commits 1 Files Changed 3
+27 -5

1 Commits

Author SHA1 Message Date
CoreRasurae a1fd36932f fix(ci): make benchmark_regression job fast and correctly fail on regressions
CI / lint (pull_request) Successful in 44s
Details
CI / typecheck (pull_request) Successful in 1m32s
Details
CI / security (pull_request) Successful in 1m15s
Details
CI / quality (pull_request) Successful in 47s
Details
CI / build (pull_request) Successful in 2m4s
Details
CI / integration_tests (pull_request) Successful in 4m18s
Details
CI / unit_tests (pull_request) Successful in 5m41s
Details
CI / coverage (pull_request) Successful in 4m36s
Details
CI / status-check (pull_request) Successful in 7s
Details
CI / benchmark (pull_request) Failing after 17m20s
Details 
Three compounding issues in the benchmark_regression CI check, found
while investigating a 41m13s hang/timeout on PR #85's run (job 205876):

1. `asv continuous` ran with no speed-oriented flags, fully calibrating
   and repeating every one of 138 benchmark runs (69 benchmarks x 2
   commits). Per-benchmark ASV overhead (calibration + repeat + process
   averaging) cost 20-90s of wall clock per benchmark, dwarfing the
   actual measured microsecond-to-millisecond costs. `noxfile.py`'s
   `benchmark_regression` session now forwards *session.posargs to
   `asv continuous`, so its own default (full, accurate) behavior is
   unchanged when called without extra args; `.gitea/workflows/ci.yml`'s
   `benchmark` job now invokes `nox -s benchmark_regression -- --quick`,
   scoping ASV's single-run mode to this informational job only.

2. With --quick applied, a new bottleneck surfaced: asv.conf.json's
   build/install/uninstall commands used plain `python -m build`/`pip`,
   which silently spent ~11 minutes per run in pip's resolver (build
   isolation + a full, uncached dependency install per compared commit,
   no progress output during backtracking). Switched to `uv build`/
   `uv pip install`/`uv pip uninstall`. `uv` isn't installed inside each
   ASV-managed virtualenv by default (asv's find_executable only
   searches that venv's own bin/, not $PATH), so added `uv` to asv's
   `matrix` config to have it pip-installed during venv bootstrap
   (already-fast for a single small package), and pass `--python
   {env_dir}/bin/python` explicitly to each command since standalone
   `uv` (unlike `python -m pip`) has no unambiguous way to infer which
   of asv's several per-commit venvs to target otherwise. Also added
   `--force-reinstall` to the install command per asv's own documented
   rationale (compared commits may share the same package version).

3. `asv continuous` returns the boolean `worsened` as its process exit
   code (0 = no significant regression, 1 = a benchmark measurably
   worsened past --factor) -- there is no exit code 2 in this asv
   version. `noxfile.py`'s `success_codes=[0, 2]` therefore never
   actually matched the real "worsened" code, silently making this
   session fail on *any* regression signal regardless of significance.
   Corrected to `success_codes=[0]`, so the session (and CI job step)
   now fails specifically when asv reports a real, significant
   regression -- the CI job's existing `continue-on-error: true` is
   what keeps this from blocking the PR, not this success list.

Verified locally end-to-end (rm -rf .asv/env .asv/results between
runs): completes in ~3-11 minutes (vs. the prior 40+ minute
hang/timeout), and the session now correctly reports "failed" when
asv detects a significant regression between the compared commits.

ISSUES CLOSED: #86
 2026-07-31 14:12:50 +00:00