UAT: Orphaned benchmark files in asv/benchmarks/ are never discovered by ASV #3945

New issue

Open

opened 2026-04-06 07:41:45 +00:00 by freemo · 0 comments

freemo commented

2026-04-06 07:41:45 +00:00

Owner

Bug Report

Feature Area: Benchmarks and Performance
Severity: High
Found by: UAT automated testing (instance: uat-benchmarks-perf-001)

What Was Tested

The asv.conf.json configuration and the asv/benchmarks/ directory were inspected to verify that all benchmark files are correctly discovered by ASV.

Expected Behavior (from spec/CONTRIBUTING.md)

All ASV benchmark files should be discoverable by the asv run command. The asv.conf.json benchmark_dir field specifies where ASV looks for benchmarks.

Actual Behavior

There are two separate benchmark directories:

benchmarks/ — the directory configured in asv.conf.json as "benchmark_dir": "benchmarks" (200+ files, correctly discovered)
asv/benchmarks/ — a separate directory containing 2 benchmark files that are never discovered by ASV:
- asv/benchmarks/action_model_bench.py — benchmarks for Action domain model operations (argument parsing, coercion, from_config(), as_cli_dict(), template rendering)
- asv/benchmarks/tui_reference_fuzzy_bench.py — benchmarks for TUI fuzzy reference ranking

Evidence

asv.conf.json (root of repo):

{
  "benchmark_dir": "benchmarks",
  ...
}

asv/benchmarks/ directory (never referenced by asv.conf.json):

asv/
  benchmarks/
    __init__.py
    action_model_bench.py       ← orphaned, never run
    tui_reference_fuzzy_bench.py ← orphaned, never run

The asv/benchmarks/action_model_bench.py file contains valid ASV benchmark suites (TimeArgumentParsing, TimeArgumentCoercion, TimeFromConfig, TimeAsCliDict, TimeTemplateRendering, TimeFromMapping) that measure performance of the Action domain model. These benchmarks will never be executed by nox -s benchmark or nox -s benchmark_regression because ASV only looks in benchmarks/ (the root-level directory).

Steps to Reproduce

Run asv run --config asv.conf.json HEAD
Observe that action_model_bench.py and tui_reference_fuzzy_bench.py from asv/benchmarks/ are not included in the benchmark run
Confirm: cat asv.conf.json | python3 -c "import json,sys; print(json.load(sys.stdin)['benchmark_dir'])" → outputs benchmarks (not asv/benchmarks)

Impact

Performance regressions in Action model operations and TUI fuzzy ranking will go undetected
The asv/benchmarks/ directory is dead code — it exists but serves no purpose
Developers who add benchmarks to asv/benchmarks/ will believe they are being tracked, but they are not

Recommended Fix

Either:

Move asv/benchmarks/action_model_bench.py and asv/benchmarks/tui_reference_fuzzy_bench.py into the root benchmarks/ directory (and remove the empty asv/benchmarks/ directory), OR
Update asv.conf.json to point benchmark_dir to asv/benchmarks (but this would break the 200+ existing benchmarks in benchmarks/)

Option 1 is strongly preferred: move the orphaned files into benchmarks/ and delete asv/benchmarks/.

Code Locations

asv.conf.json — benchmark_dir field (line 8)
asv/benchmarks/action_model_bench.py — orphaned benchmark file
asv/benchmarks/tui_reference_fuzzy_bench.py — orphaned benchmark file

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Bug Report **Feature Area:** Benchmarks and Performance **Severity:** High **Found by:** UAT automated testing (instance: uat-benchmarks-perf-001) ### What Was Tested The `asv.conf.json` configuration and the `asv/benchmarks/` directory were inspected to verify that all benchmark files are correctly discovered by ASV. ### Expected Behavior (from spec/CONTRIBUTING.md) All ASV benchmark files should be discoverable by the `asv run` command. The `asv.conf.json` `benchmark_dir` field specifies where ASV looks for benchmarks. ### Actual Behavior There are **two separate benchmark directories**: 1. `benchmarks/` — the directory configured in `asv.conf.json` as `"benchmark_dir": "benchmarks"` (200+ files, correctly discovered) 2. `asv/benchmarks/` — a separate directory containing 2 benchmark files that are **never discovered by ASV**: - `asv/benchmarks/action_model_bench.py` — benchmarks for `Action` domain model operations (argument parsing, coercion, `from_config()`, `as_cli_dict()`, template rendering) - `asv/benchmarks/tui_reference_fuzzy_bench.py` — benchmarks for TUI fuzzy reference ranking ### Evidence **`asv.conf.json`** (root of repo): ```json { "benchmark_dir": "benchmarks", ... } ``` **`asv/benchmarks/` directory** (never referenced by `asv.conf.json`): ``` asv/ benchmarks/ __init__.py action_model_bench.py ← orphaned, never run tui_reference_fuzzy_bench.py ← orphaned, never run ``` The `asv/benchmarks/action_model_bench.py` file contains valid ASV benchmark suites (`TimeArgumentParsing`, `TimeArgumentCoercion`, `TimeFromConfig`, `TimeAsCliDict`, `TimeTemplateRendering`, `TimeFromMapping`) that measure performance of the `Action` domain model. These benchmarks will never be executed by `nox -s benchmark` or `nox -s benchmark_regression` because ASV only looks in `benchmarks/` (the root-level directory). ### Steps to Reproduce 1. Run `asv run --config asv.conf.json HEAD` 2. Observe that `action_model_bench.py` and `tui_reference_fuzzy_bench.py` from `asv/benchmarks/` are not included in the benchmark run 3. Confirm: `cat asv.conf.json | python3 -c "import json,sys; print(json.load(sys.stdin)['benchmark_dir'])"` → outputs `benchmarks` (not `asv/benchmarks`) ### Impact - Performance regressions in `Action` model operations and TUI fuzzy ranking will go undetected - The `asv/benchmarks/` directory is dead code — it exists but serves no purpose - Developers who add benchmarks to `asv/benchmarks/` will believe they are being tracked, but they are not ### Recommended Fix Either: 1. **Move** `asv/benchmarks/action_model_bench.py` and `asv/benchmarks/tui_reference_fuzzy_bench.py` into the root `benchmarks/` directory (and remove the empty `asv/benchmarks/` directory), OR 2. **Update** `asv.conf.json` to point `benchmark_dir` to `asv/benchmarks` (but this would break the 200+ existing benchmarks in `benchmarks/`) Option 1 is strongly preferred: move the orphaned files into `benchmarks/` and delete `asv/benchmarks/`. ### Code Locations - `asv.conf.json` — `benchmark_dir` field (line 8) - `asv/benchmarks/action_model_bench.py` — orphaned benchmark file - `asv/benchmarks/tui_reference_fuzzy_bench.py` — orphaned benchmark file --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo referenced this issue

2026-04-06 07:43:14 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 07:49:16 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775