TEST-INFRA: [test-architecture] Organize helper files in benchmarks directory #2233

New issue

Open

opened 2026-04-03 09:43:43 +00:00 by freemo · 0 comments

freemo commented

2026-04-03 09:43:43 +00:00

Owner

Metadata

Branch: refactor/test-arch-organize-benchmarks-helper-files
Commit Message: refactor(tests): organize helper files in benchmarks directory into a dedicated subdirectory
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The benchmarks/ directory currently contains 227 Python files all in a single flat directory. This includes:

209 *_bench.py files (benchmark suites following the <name>_bench.py naming convention)
15 bench_*.py files (benchmark suites following the alternative bench_<name>.py naming convention)
2 files with non-standard naming: cli_benchmark.py and plan_generation_benchmark.py
1 shared helper module: _session_bench_common.py (shared utilities extracted from tdd_session_create_di_bench.py and tdd_session_list_di_bench.py)
1 __init__.py

Per the project's File Organization guidelines in CONTRIBUTING.md, each directory should have a clear, single purpose — do not mix concerns. The current flat layout mixes benchmark suite files with shared helper/utility modules, making it difficult to navigate, understand the directory's structure at a glance, and locate specific support code.

Organizing the helper files into a dedicated benchmarks/helpers/ subdirectory (or equivalent) would:

Clearly separate ASV benchmark suite files from their shared Python utility/helper modules
Make the benchmarks/ directory scannable at a glance
Align with the single-purpose directory principle from CONTRIBUTING.md

This issue is related to and complements:

#2220 — Organize helper files in robot directory (same pattern, different test layer)

Current Behavior

All 227 Python files coexist in the same flat benchmarks/ directory. There is no structural separation between benchmark suite files and their shared helper/utility modules. The _session_bench_common.py file provides shared utilities used by multiple benchmark suites but lives alongside the benchmark files themselves.

Expected Behavior

Shared helper and utility modules are moved into a dedicated subdirectory (e.g., benchmarks/helpers/) so that the benchmarks/ directory contains only ASV benchmark suite files and the __init__.py. All imports within benchmark files that reference moved helper modules are updated to reflect the new path.

Acceptance Criteria

All shared helper/utility modules (e.g., _session_bench_common.py) are relocated to a dedicated subdirectory within benchmarks/
All other non-benchmark Python support files are evaluated and either co-located in the helpers subdirectory or placed in an appropriate location
All benchmark files that import from moved helper modules are updated with the correct new paths
The full ASV benchmark suite runs without import errors after the reorganization
nox (all default sessions) passes cleanly

Supporting Information

Current file counts in benchmarks/:

*_bench.py benchmark files: 209
bench_*.py benchmark files: 15
Non-standard named benchmark files: 2 (cli_benchmark.py, plan_generation_benchmark.py)
Shared helper modules: 1 (_session_bench_common.py)
__init__.py: 1

The _session_bench_common.py file is currently imported by at least:

tdd_session_create_di_bench.py
tdd_session_list_di_bench.py

Related issues: #2220 (robot directory equivalent)

Subtasks

Audit all Python files in benchmarks/ and categorize them (benchmark suite, shared helper, utility, etc.)
Decide on the target subdirectory structure (e.g., benchmarks/helpers/ for all shared helper modules)
Move all shared helper/utility modules to the chosen subdirectory
Update all import paths in benchmark files to reflect the new locations
Verify ASV can resolve all imports after the move
Run nox -s benchmarks (or equivalent) and confirm all benchmark suites load without errors
Run nox (all default sessions) and fix any errors
Verify coverage >= 97% via nox -s coverage_report

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `refactor/test-arch-organize-benchmarks-helper-files` - **Commit Message**: `refactor(tests): organize helper files in benchmarks directory into a dedicated subdirectory` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The `benchmarks/` directory currently contains 227 Python files all in a single flat directory. This includes: - 209 `*_bench.py` files (benchmark suites following the `<name>_bench.py` naming convention) - 15 `bench_*.py` files (benchmark suites following the alternative `bench_<name>.py` naming convention) - 2 files with non-standard naming: `cli_benchmark.py` and `plan_generation_benchmark.py` - 1 shared helper module: `_session_bench_common.py` (shared utilities extracted from `tdd_session_create_di_bench.py` and `tdd_session_list_di_bench.py`) - 1 `__init__.py` Per the project's **File Organization** guidelines in `CONTRIBUTING.md`, each directory should have a clear, single purpose — do not mix concerns. The current flat layout mixes benchmark suite files with shared helper/utility modules, making it difficult to navigate, understand the directory's structure at a glance, and locate specific support code. Organizing the helper files into a dedicated `benchmarks/helpers/` subdirectory (or equivalent) would: - Clearly separate ASV benchmark suite files from their shared Python utility/helper modules - Make the `benchmarks/` directory scannable at a glance - Align with the single-purpose directory principle from `CONTRIBUTING.md` This issue is related to and complements: - #2220 — Organize helper files in robot directory (same pattern, different test layer) ## Current Behavior All 227 Python files coexist in the same flat `benchmarks/` directory. There is no structural separation between benchmark suite files and their shared helper/utility modules. The `_session_bench_common.py` file provides shared utilities used by multiple benchmark suites but lives alongside the benchmark files themselves. ## Expected Behavior Shared helper and utility modules are moved into a dedicated subdirectory (e.g., `benchmarks/helpers/`) so that the `benchmarks/` directory contains only ASV benchmark suite files and the `__init__.py`. All imports within benchmark files that reference moved helper modules are updated to reflect the new path. ## Acceptance Criteria - All shared helper/utility modules (e.g., `_session_bench_common.py`) are relocated to a dedicated subdirectory within `benchmarks/` - All other non-benchmark Python support files are evaluated and either co-located in the helpers subdirectory or placed in an appropriate location - All benchmark files that import from moved helper modules are updated with the correct new paths - The full ASV benchmark suite runs without import errors after the reorganization - `nox` (all default sessions) passes cleanly ## Supporting Information Current file counts in `benchmarks/`: - `*_bench.py` benchmark files: 209 - `bench_*.py` benchmark files: 15 - Non-standard named benchmark files: 2 (`cli_benchmark.py`, `plan_generation_benchmark.py`) - Shared helper modules: 1 (`_session_bench_common.py`) - `__init__.py`: 1 The `_session_bench_common.py` file is currently imported by at least: - `tdd_session_create_di_bench.py` - `tdd_session_list_di_bench.py` Related issues: #2220 (robot directory equivalent) ## Subtasks - [ ] Audit all Python files in `benchmarks/` and categorize them (benchmark suite, shared helper, utility, etc.) - [ ] Decide on the target subdirectory structure (e.g., `benchmarks/helpers/` for all shared helper modules) - [ ] Move all shared helper/utility modules to the chosen subdirectory - [ ] Update all import paths in benchmark files to reflect the new locations - [ ] Verify ASV can resolve all imports after the move - [ ] Run `nox -s benchmarks` (or equivalent) and confirm all benchmark suites load without errors - [ ] Run `nox` (all default sessions) and fix any errors - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-03 09:44:17 +00:00

freemo added a new dependency

2026-04-03 10:09:22 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo removed this from the v3.8.0 milestone

2026-04-07 01:13:36 +00:00