cleveragents/cleveragents-core

Fork 3

perf(tests): add ASV benchmarks to track test suite runtime regressions #486

New issue

Closed

opened 2026-03-01 03:12:24 +00:00 by freemo · 1 comment

freemo commented

2026-03-01 03:12:24 +00:00

Owner

Metadata

Commit Message: perf(tests): add ASV benchmarks for test suite runtime regression tracking
Branch: perf/asv-test-runtime-benchmarks

Background and Context

Part of #478.

After the test suite optimizations from the sibling issues in this Epic are complete, we need a mechanism to detect future runtime regressions before they accumulate back to the current 24-minute / 75-minute state. Without continuous benchmarking, it is common for test suites to gradually slow down as new features add scenarios, imports, and fixtures.

ASV (Airspeed Velocity) is a Python benchmarking framework that:

Tracks benchmark results over time (per commit)
Generates HTML dashboards showing performance trends
Can be integrated into CI to fail builds on regressions
Stores results in a Git-based database for history

What to Benchmark

Wall-clock time of nox -e unit_tests (target: <2m 30s)
Wall-clock time of nox -e coverage_report (target: <7m 30s)
Per-tier feature timing — aggregate time for the top-8, medium-20, and consolidated-small tiers
Subprocess count — total number of subprocesses spawned by behave-parallel
Coverage reporting overhead — time spent in post-test coverage combine/report steps

Acceptance Criteria

ASV is configured in the repository with asv.conf.json
At least 5 benchmarks are defined covering the metrics listed above
Benchmarks can be run locally via asv run
Baseline results are recorded for the current (post-optimization) state
CI pipeline runs benchmarks and flags regressions exceeding 20% of baseline
HTML dashboard is generated and accessible (either as CI artifact or published)

Subtasks

Setup Phase

Add asv as a development dependency in pyproject.toml (or noxfile.py dev session)
Create asv.conf.json at the repository root with appropriate settings: Python version, environment configuration, benchmark directory, result storage location
Create benchmarks/ directory for benchmark modules
Verify asv check passes with the initial configuration

Benchmark Implementation Phase

Create benchmarks/bench_unit_tests.py with a benchmark class:
- time_unit_tests_wall_clock() — runs nox -e unit_tests and measures wall-clock time
- time_unit_tests_top8_features() — runs only the 8 slowest features and measures aggregate time
- time_unit_tests_medium20_features() — runs only the 20 medium-slow features and measures aggregate time
- time_unit_tests_small_consolidated() — runs only the consolidated small features and measures aggregate time
Create benchmarks/bench_coverage_report.py with a benchmark class:
- time_coverage_wall_clock() — runs nox -e coverage_report and measures wall-clock time
- time_coverage_combine() — measures time for coverage combine step only
- time_coverage_html() — measures time for coverage html step only
- time_coverage_report() — measures time for coverage report step only
Create benchmarks/bench_subprocess_overhead.py with a benchmark class:
- track_subprocess_count() — counts total subprocesses spawned during nox -e unit_tests (tracking benchmark, not timing)
- time_single_feature_subprocess() — measures time to run a single trivial feature via subprocess (overhead baseline)
Add setup() and teardown() methods to each benchmark class for proper environment initialization
Set appropriate timeout, rounds, repeat, and warmup_time parameters for each benchmark (test suite benchmarks should use rounds=1, repeat=1 since they are inherently slow)

Baseline Phase

Run asv run to record baseline results on the current commit (post-optimization)
Run asv publish to generate the HTML dashboard
Verify the dashboard shows reasonable values for all benchmarks
Store baseline results in the repository (results/ directory or equivalent)

CI Integration Phase

Add a nox session nox -e benchmarks that runs asv run --quick (single iteration, no statistical rigor — just regression detection)
Configure CI to run nox -e benchmarks on pull requests
Configure asv compare to fail if any benchmark regresses more than 20% from the baseline
Add benchmark results as a CI artifact (HTML dashboard)
Document the benchmark workflow in CONTRIBUTING.md or a benchmarks/README.md

Verification Phase

Run asv run locally and confirm all benchmarks complete successfully
Verify asv compare correctly detects a synthetic regression (temporarily add a time.sleep(1) to a benchmark to test)
Verify CI correctly runs benchmarks and reports results
Verify the HTML dashboard renders correctly

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.

## Metadata - **Commit Message**: `perf(tests): add ASV benchmarks for test suite runtime regression tracking` - **Branch**: `perf/asv-test-runtime-benchmarks` ## Background and Context Part of #478. After the test suite optimizations from the sibling issues in this Epic are complete, we need a mechanism to **detect future runtime regressions** before they accumulate back to the current 24-minute / 75-minute state. Without continuous benchmarking, it is common for test suites to gradually slow down as new features add scenarios, imports, and fixtures. [ASV (Airspeed Velocity)](https://asv.readthedocs.io/) is a Python benchmarking framework that: - Tracks benchmark results over time (per commit) - Generates HTML dashboards showing performance trends - Can be integrated into CI to fail builds on regressions - Stores results in a Git-based database for history ### What to Benchmark 1. **Wall-clock time** of `nox -e unit_tests` (target: <2m 30s) 2. **Wall-clock time** of `nox -e coverage_report` (target: <7m 30s) 3. **Per-tier feature timing** — aggregate time for the top-8, medium-20, and consolidated-small tiers 4. **Subprocess count** — total number of subprocesses spawned by behave-parallel 5. **Coverage reporting overhead** — time spent in post-test coverage combine/report steps ## Acceptance Criteria - [x] ASV is configured in the repository with `asv.conf.json` - [x] At least 5 benchmarks are defined covering the metrics listed above - [x] Benchmarks can be run locally via `asv run` - [x] Baseline results are recorded for the current (post-optimization) state - [x] CI pipeline runs benchmarks and flags regressions exceeding 20% of baseline - [x] HTML dashboard is generated and accessible (either as CI artifact or published) ## Subtasks ### Setup Phase - [x] Add `asv` as a development dependency in `pyproject.toml` (or `noxfile.py` dev session) - [x] Create `asv.conf.json` at the repository root with appropriate settings: Python version, environment configuration, benchmark directory, result storage location - [x] Create `benchmarks/` directory for benchmark modules - [x] Verify `asv check` passes with the initial configuration ### Benchmark Implementation Phase - [x] Create `benchmarks/bench_unit_tests.py` with a benchmark class: - `time_unit_tests_wall_clock()` — runs `nox -e unit_tests` and measures wall-clock time - `time_unit_tests_top8_features()` — runs only the 8 slowest features and measures aggregate time - `time_unit_tests_medium20_features()` — runs only the 20 medium-slow features and measures aggregate time - `time_unit_tests_small_consolidated()` — runs only the consolidated small features and measures aggregate time - [x] Create `benchmarks/bench_coverage_report.py` with a benchmark class: - `time_coverage_wall_clock()` — runs `nox -e coverage_report` and measures wall-clock time - `time_coverage_combine()` — measures time for `coverage combine` step only - `time_coverage_html()` — measures time for `coverage html` step only - `time_coverage_report()` — measures time for `coverage report` step only - [x] Create `benchmarks/bench_subprocess_overhead.py` with a benchmark class: - `track_subprocess_count()` — counts total subprocesses spawned during `nox -e unit_tests` (tracking benchmark, not timing) - `time_single_feature_subprocess()` — measures time to run a single trivial feature via subprocess (overhead baseline) - [x] Add `setup()` and `teardown()` methods to each benchmark class for proper environment initialization - [x] Set appropriate `timeout`, `rounds`, `repeat`, and `warmup_time` parameters for each benchmark (test suite benchmarks should use `rounds=1, repeat=1` since they are inherently slow) ### Baseline Phase - [x] Run `asv run` to record baseline results on the current commit (post-optimization) - [x] Run `asv publish` to generate the HTML dashboard - [x] Verify the dashboard shows reasonable values for all benchmarks - [x] Store baseline results in the repository (`results/` directory or equivalent) ### CI Integration Phase - [x] Add a nox session `nox -e benchmarks` that runs `asv run --quick` (single iteration, no statistical rigor — just regression detection) - [x] Configure CI to run `nox -e benchmarks` on pull requests - [x] Configure `asv compare` to fail if any benchmark regresses more than 20% from the baseline - [x] Add benchmark results as a CI artifact (HTML dashboard) - [x] Document the benchmark workflow in CONTRIBUTING.md or a `benchmarks/README.md` ### Verification Phase - [x] Run `asv run` locally and confirm all benchmarks complete successfully - [x] Verify `asv compare` correctly detects a synthetic regression (temporarily add a `time.sleep(1)` to a benchmark to test) - [x] Verify CI correctly runs benchmarks and reports results - [x] Verify the HTML dashboard renders correctly ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.

freemo added the

Session	Status
lint	PASS
typecheck	PASS
unit_tests	PASS — 7510 scenarios
integration_tests	PASS (pre-existing failures only)
coverage_report	PASS — 98%

Rows
Columns

perf(tests): add ASV benchmarks to track test suite runtime regressions #486

Metadata

Background and Context

What to Benchmark

Acceptance Criteria

Subtasks

Setup Phase

Benchmark Implementation Phase

Baseline Phase

CI Integration Phase

Verification Phase

Definition of Done

Implementation Notes

Commit: 8683f36 on perf/asv-test-runtime-benchmarks

What was done

Design decisions

Quality gate results

Fixes applied

Commit: `8683f36` on `perf/asv-test-runtime-benchmarks`