cleveragents/cleveractors-core

Fork 0

fix(ci): benchmark_regression job intermittently hangs/times out and never signals real regressions correctly #86

New issue

Closed

opened 2026-07-31 11:54:35 +00:00 by CoreRasurae · 0 comments

CoreRasurae commented

2026-07-31 11:54:35 +00:00

Member

Metadata

Field	Value
Commit Message	fix(ci): make benchmark_regression job fast and correctly fail on regressions
Branch	task/m2-quick-benchmark-regression-ci

Background and context

While investigating a hung benchmark CI job on PR #85's run (https://git.cleverthis.com/cleveragents/cleveractors-core/actions/runs/31568/jobs/205876), the "Run ASV benchmark regression check via nox" step ran for 41m13s and ended in failure, with continue-on-error: true so it did not block the PR. Fixing this uncovered three compounding issues rather than one.

Issue 1 — ASV calibration overhead

noxfile.py's benchmark_regression session ran asv continuous origin/master HEAD --show-stderr with none of ASV's speed-oriented flags. By default, ASV auto-calibrates the iteration count per benchmark, runs multiple timed repeats, and repeats the whole cycle across processes (default 2) — for 2 commits × 69 benchmarks = 138 full calibrated runs. Individual benchmarks cost 20-90s of wall clock between log entries, while their actual measured cost was microseconds-to-low-milliseconds — ASV's own calibration/repeat overhead, not the code under test, dominated total time.

Issue 2 — packaging bottleneck (surfaced after fixing #1)

After scoping ASV's --quick flag to CI (fixing issue 1), a second, independent run (https://git.cleverthis.com/cleveragents/cleveractors-core/actions/runs/31578/jobs/205924) still took ~14 minutes. asv.conf.json's build_command/install_command/uninstall_command used plain python -m build/pip, which silently spent ~11 minutes per run — PEP 517 build isolation plus a full, uncached pip install of the whole dependency tree (langchain/langgraph, etc.) per compared commit, with no console output during pip's resolver work. uv (already installed and used elsewhere in this CI job) doesn't share pip's slow/silent resolver behavior, but a naive swap to bare uv build/uv pip install/uv pip uninstall initially failed with ModuleNotFoundError: No module named 'cleveractors', because:

ASV's find_executable() only searches each managed virtualenv's own bin/ directory, never $PATH — so a system-level uv binary is invisible to it unless installed into that venv.
Standalone uv pip install/uv pip uninstall (unlike python -m pip install, which is unambiguous because python itself is resolved from inside the target venv) has no reliable way to infer which of ASV's several per-commit venvs to target without an explicit --python argument.

Issue 3 — success_codes never matched ASV's real exit code

asv continuous returns the boolean worsened as its process exit code: 0 = no significant regression, 1 = a benchmark measurably worsened past --factor. There is no exit code 2 in this ASV version. noxfile.py's pre-existing success_codes=[0, 2] therefore never matched the real "worsened" code (1), so the session — and by extension the CI job step — failed unconditionally on any regression signal, real or noise, ever since this job was first wired into CI. This was independent of issues 1 and 2 and pre-dated all of this work.

Current behavior (before this fix)

noxfile.py's benchmark_regression session does not forward extra CLI args to asv continuous, always running full (non-quick) calibration.
asv.conf.json uses plain python -m build/pip for its own per-commit environment build/install/uninstall.
noxfile.py's success_codes=[0, 2] for the asv continuous call does not include 1, the actual "worsened" exit code.
Net effect: the job intermittently ran 40+ minutes and was killed externally with no diagnostic message, and even after becoming fast, would still report failure regardless of whether a real regression occurred.

Expected behavior

The CI-invoked benchmark_regression run completes reliably in a small fraction of the previous time (--quick, scoped to CI only via *session.posargs).
ASV's own per-commit environment build/install/uninstall uses uv (fast, no silent resolver backtracking), correctly targeting each per-commit venv via matrix-bootstrapped uv + explicit --python {env_dir}/bin/python.
The session (and CI job step) fails specifically when asv continuous reports a real, significant regression (success_codes=[0]) — not on every run regardless of significance. continue-on-error: true on the job (unchanged) is what keeps this from blocking the PR merge.
No change to the shared statistical/accuracy behavior of noxfile.py's benchmark_regression session when invoked without extra args, for use at other CI stages later.

Acceptance criteria

noxfile.py's benchmark_regression session forwards *session.posargs to asv continuous.
.gitea/workflows/ci.yml's benchmark job invokes nox -s benchmark_regression -- --quick.
asv.conf.json's build_command/install_command/uninstall_command use uv, with uv bootstrapped via matrix and --python {env_dir}/bin/python passed explicitly.
noxfile.py's success_codes for the asv continuous call corrected to [0].
Verified locally end-to-end, repeatedly (rm -rf .asv/env .asv/results between runs): completes in ~3-11 minutes; correctly reports failed when a real regression is detected between the compared commits.
Confirm on the next PR's CI run that the benchmark job completes in a small fraction of the previous ~41 minutes, and that its conclusion matches whether a real regression was detected.
nox -s coverage_report unaffected (this change touches CI workflow/noxfile.py/asv.conf.json only, no src/ changes).

Supporting information

Evidence run 1 (original hang): https://git.cleverthis.com/cleveragents/cleveractors-core/actions/runs/31568/jobs/205876 — 41m13s, killed externally with no diagnostic message.
Evidence run 2 (after --quick, before uv fix): https://git.cleverthis.com/cleveragents/cleveractors-core/actions/runs/31578/jobs/205924 — ~14 minutes, log goes silent for ~11.5 minutes right after "Building ... for virtualenv-py3.13".
asv continuous --help: --quick, -q Do a "quick" run, where each benchmark function is run only once...
ASV source (asv/commands/continuous.py): return worsened — confirms the real exit-code semantics (0/1, never 2).
ASV source (asv/environment.py: find_executable): confirms the search is restricted to the managed venv's own bin/, explaining why a system-level uv wasn't found until added via matrix.
Parent Epic note: no currently open Epic cleanly covers CI/tooling reliability — Epic #77 is scoped to LLM Agent Runtime Stabilization and explicitly excludes new-capability/build-out work; this is CI infra, not application runtime. Left unlinked pending a maintainer decision on whether to create a "CI/CD & Developer Tooling" Epic, following the same precedent set in issue #83.

Subtasks

Forward *session.posargs in noxfile.py's benchmark_regression session
Scope --quick to .gitea/workflows/ci.yml's benchmark job invocation only
Switch asv.conf.json's build/install/uninstall commands to uv, bootstrapped via matrix and targeted via explicit --python {env_dir}/bin/python
Correct noxfile.py's success_codes for asv continuous to [0]
Verify coverage >=97% via nox -s coverage_report (expected unaffected; confirm no regression)
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line matches the Commit Message in Metadata exactly.
The commit is pushed to the branch matching the Branch in Metadata exactly.
The commit is submitted as a PR to master, reviewed, and merged.

🤖 Generated with Claude Code

## Metadata | Field | Value | |-------|-------| | Commit Message | fix(ci): make benchmark_regression job fast and correctly fail on regressions | | Branch | task/m2-quick-benchmark-regression-ci | ## Background and context While investigating a hung `benchmark` CI job on PR #85's run (https://git.cleverthis.com/cleveragents/cleveractors-core/actions/runs/31568/jobs/205876), the "Run ASV benchmark regression check via nox" step ran for **41m13s** and ended in `failure`, with `continue-on-error: true` so it did not block the PR. Fixing this uncovered three compounding issues rather than one. ### Issue 1 — ASV calibration overhead `noxfile.py`'s `benchmark_regression` session ran `asv continuous origin/master HEAD --show-stderr` with none of ASV's speed-oriented flags. By default, ASV auto-calibrates the iteration count per benchmark, runs multiple timed repeats, and repeats the whole cycle across `processes` (default 2) — for 2 commits × 69 benchmarks = 138 full calibrated runs. Individual benchmarks cost 20-90s of *wall clock* between log entries, while their actual *measured* cost was microseconds-to-low-milliseconds — ASV's own calibration/repeat overhead, not the code under test, dominated total time. ### Issue 2 — packaging bottleneck (surfaced after fixing #1) After scoping ASV's `--quick` flag to CI (fixing issue 1), a second, independent run (https://git.cleverthis.com/cleveragents/cleveractors-core/actions/runs/31578/jobs/205924) still took ~14 minutes. `asv.conf.json`'s `build_command`/`install_command`/`uninstall_command` used plain `python -m build`/`pip`, which silently spent ~11 minutes per run — PEP 517 build isolation plus a full, uncached `pip install` of the whole dependency tree (langchain/langgraph, etc.) per compared commit, with no console output during pip's resolver work. `uv` (already installed and used elsewhere in this CI job) doesn't share pip's slow/silent resolver behavior, but a naive swap to bare `uv build`/`uv pip install`/`uv pip uninstall` initially failed with `ModuleNotFoundError: No module named 'cleveractors'`, because: - ASV's `find_executable()` only searches each managed virtualenv's own `bin/` directory, never `$PATH` — so a system-level `uv` binary is invisible to it unless installed *into* that venv. - Standalone `uv pip install`/`uv pip uninstall` (unlike `python -m pip install`, which is unambiguous because `python` itself is resolved from inside the target venv) has no reliable way to infer which of ASV's several per-commit venvs to target without an explicit `--python` argument. ### Issue 3 — success_codes never matched ASV's real exit code `asv continuous` returns the boolean `worsened` as its process exit code: `0` = no significant regression, `1` = a benchmark measurably worsened past `--factor`. There is no exit code `2` in this ASV version. `noxfile.py`'s pre-existing `success_codes=[0, 2]` therefore never matched the real "worsened" code (`1`), so the session — and by extension the CI job step — failed **unconditionally** on any regression signal, real or noise, ever since this job was first wired into CI. This was independent of issues 1 and 2 and pre-dated all of this work. ## Current behavior (before this fix) 1. `noxfile.py`'s `benchmark_regression` session does not forward extra CLI args to `asv continuous`, always running full (non-quick) calibration. 2. `asv.conf.json` uses plain `python -m build`/`pip` for its own per-commit environment build/install/uninstall. 3. `noxfile.py`'s `success_codes=[0, 2]` for the `asv continuous` call does not include `1`, the actual "worsened" exit code. 4. Net effect: the job intermittently ran 40+ minutes and was killed externally with no diagnostic message, and even after becoming fast, would still report `failure` regardless of whether a real regression occurred. ## Expected behavior 1. The CI-invoked `benchmark_regression` run completes reliably in a small fraction of the previous time (`--quick`, scoped to CI only via `*session.posargs`). 2. ASV's own per-commit environment build/install/uninstall uses `uv` (fast, no silent resolver backtracking), correctly targeting each per-commit venv via `matrix`-bootstrapped `uv` + explicit `--python {env_dir}/bin/python`. 3. The session (and CI job step) fails specifically when `asv continuous` reports a real, significant regression (`success_codes=[0]`) — not on every run regardless of significance. `continue-on-error: true` on the job (unchanged) is what keeps this from blocking the PR merge. 4. No change to the shared statistical/accuracy behavior of `noxfile.py`'s `benchmark_regression` session when invoked without extra args, for use at other CI stages later. ## Acceptance criteria - [x] `noxfile.py`'s `benchmark_regression` session forwards `*session.posargs` to `asv continuous`. - [x] `.gitea/workflows/ci.yml`'s `benchmark` job invokes `nox -s benchmark_regression -- --quick`. - [x] `asv.conf.json`'s `build_command`/`install_command`/`uninstall_command` use `uv`, with `uv` bootstrapped via `matrix` and `--python {env_dir}/bin/python` passed explicitly. - [x] `noxfile.py`'s `success_codes` for the `asv continuous` call corrected to `[0]`. - [x] Verified locally end-to-end, repeatedly (`rm -rf .asv/env .asv/results` between runs): completes in ~3-11 minutes; correctly reports `failed` when a real regression is detected between the compared commits. - [ ] Confirm on the next PR's CI run that the `benchmark` job completes in a small fraction of the previous ~41 minutes, and that its conclusion matches whether a real regression was detected. - [ ] `nox -s coverage_report` unaffected (this change touches CI workflow/noxfile.py/asv.conf.json only, no `src/` changes). ## Supporting information - Evidence run 1 (original hang): https://git.cleverthis.com/cleveragents/cleveractors-core/actions/runs/31568/jobs/205876 — 41m13s, killed externally with no diagnostic message. - Evidence run 2 (after --quick, before uv fix): https://git.cleverthis.com/cleveragents/cleveractors-core/actions/runs/31578/jobs/205924 — ~14 minutes, log goes silent for ~11.5 minutes right after "Building ... for virtualenv-py3.13". - `asv continuous --help`: `--quick, -q Do a "quick" run, where each benchmark function is run only once...` - ASV source (`asv/commands/continuous.py`): `return worsened` — confirms the real exit-code semantics (0/1, never 2). - ASV source (`asv/environment.py: find_executable`): confirms the search is restricted to the managed venv's own `bin/`, explaining why a system-level `uv` wasn't found until added via `matrix`. - **Parent Epic note**: no currently open Epic cleanly covers CI/tooling reliability — Epic #77 is scoped to LLM Agent Runtime Stabilization and explicitly excludes new-capability/build-out work; this is CI infra, not application runtime. Left unlinked pending a maintainer decision on whether to create a "CI/CD & Developer Tooling" Epic, following the same precedent set in issue #83. ## Subtasks - [x] Forward `*session.posargs` in `noxfile.py`'s `benchmark_regression` session - [x] Scope `--quick` to `.gitea/workflows/ci.yml`'s `benchmark` job invocation only - [x] Switch `asv.conf.json`'s build/install/uninstall commands to `uv`, bootstrapped via `matrix` and targeted via explicit `--python {env_dir}/bin/python` - [x] Correct `noxfile.py`'s `success_codes` for `asv continuous` to `[0]` - [ ] Verify coverage >=97% via `nox -s coverage_report` (expected unaffected; confirm no regression) - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the first line matches the Commit Message in Metadata exactly. - The commit is pushed to the branch matching the Branch in Metadata exactly. - The commit is submitted as a PR to master, reviewed, and merged. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

CoreRasurae added this to the v2.1.0 milestone

2026-07-31 11:54:35 +00:00

CoreRasurae added the

labels

2026-07-31 11:54:35 +00:00

CoreRasurae referenced this issue from a commit

2026-07-31 12:02:18 +00:00

fix(ci): scope asv --quick to the benchmark_regression CI job only

CoreRasurae referenced this issue from a pull request that will close it,

2026-07-31 12:02:46 +00:00

fix(ci): make benchmark_regression job fast and correctly fail on regressions #87

CoreRasurae added a new dependency

2026-07-31 12:24:31 +00:00

#87 fix(ci): make benchmark_regression job fast and correctly fail on regressions

CoreRasurae referenced this issue from a commit

2026-07-31 14:59:28 +00:00

fix(ci): make benchmark_regression job fast and correctly fail on regressions

CoreRasurae changed title from ~~fix(ci): benchmark_regression job intermittently hangs/times out under full ASV calibration~~ to fix(ci): benchmark_regression job intermittently hangs/times out and never signals real regressions correctly

2026-07-31 15:00:54 +00:00