TEST-INFRA: [ci-execution-time] Parallelize coverage report generation #2263

Closed
opened 2026-04-03 11:57:16 +00:00 by freemo · 2 comments
Owner

Metadata

  • Branch: task/ci-parallelize-coverage-report-generation
  • Commit Message: perf(ci): parallelize coverage data collection and report generation
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The coverage_report nox session is a known bottleneck in the CI pipeline. Currently, behave is invoked in a single process to collect coverage data, meaning all feature files are executed sequentially before a report can be generated. On a large test suite this serialised execution inflates wall-clock time significantly and delays feedback for every CI run.

Problem Statement

The coverage_report nox session runs behave in a single process. Because Behave does not natively distribute feature-file execution across workers, the entire suite must complete before coverage combine and coverage report can be invoked. This single-process bottleneck prevents the coverage stage from benefiting from the multi-core runners available in CI.

Proposed Approach

Explore strategies to parallelize coverage data collection and then merge the results:

  1. Parallel Behave execution with pytest-xdist-style sharding — partition the features/ directory into N shards and run each shard in a separate subprocess with its own .coverage.<shard> data file, then merge with coverage combine.
  2. behave-parallel or custom multiprocessing runner — investigate third-party runners (e.g., behave-parallel) or a custom multiprocessing.Pool-based harness that spawns one worker per feature file or tag group.
  3. Coverage subprocess tracing — ensure each worker is launched with coverage run --parallel-mode so that partial .coverage.* files are written atomically and can be safely combined afterwards.
  4. Merge and report step — after all workers complete, run coverage combine followed by coverage report / coverage html in a single serial step.

The chosen approach must keep total coverage at ≥ 97 % and must not break the existing nox -e coverage_report interface consumed by other nox sessions and CI jobs.

Subtasks

  • Profile the current coverage_report session to establish a baseline wall-clock time
  • Research and evaluate parallel Behave execution strategies (sharding, behave-parallel, multiprocessing harness)
  • Prototype the chosen strategy locally and verify coverage data integrity after coverage combine
  • Update the noxfile.py coverage_report session to use the parallel strategy
  • Validate that overall coverage remains ≥ 97 % after parallelisation
  • Update CI configuration if the session interface or artefact paths change
  • Add or update Behave scenarios covering the new parallel execution path
  • Document the approach and any tuning knobs (e.g., worker count) in CONTRIBUTING.md or inline nox comments

Definition of Done

  • nox -e coverage_report completes measurably faster than the single-process baseline
  • Coverage remains ≥ 97 % after parallelisation
  • coverage combine produces a single .coverage file from all parallel workers without data loss
  • No existing nox session or CI job interface is broken
  • Behave scenarios for the new execution path are present and passing
  • All nox stages pass
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-parallelize-coverage-report-generation` - **Commit Message**: `perf(ci): parallelize coverage data collection and report generation` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The `coverage_report` nox session is a known bottleneck in the CI pipeline. Currently, `behave` is invoked in a single process to collect coverage data, meaning all feature files are executed sequentially before a report can be generated. On a large test suite this serialised execution inflates wall-clock time significantly and delays feedback for every CI run. ## Problem Statement The `coverage_report` nox session runs `behave` in a single process. Because Behave does not natively distribute feature-file execution across workers, the entire suite must complete before `coverage combine` and `coverage report` can be invoked. This single-process bottleneck prevents the coverage stage from benefiting from the multi-core runners available in CI. ## Proposed Approach Explore strategies to parallelize coverage data collection and then merge the results: 1. **Parallel Behave execution with `pytest-xdist`-style sharding** — partition the `features/` directory into N shards and run each shard in a separate subprocess with its own `.coverage.<shard>` data file, then merge with `coverage combine`. 2. **`behave-parallel` or custom multiprocessing runner** — investigate third-party runners (e.g., `behave-parallel`) or a custom `multiprocessing.Pool`-based harness that spawns one worker per feature file or tag group. 3. **Coverage subprocess tracing** — ensure each worker is launched with `coverage run --parallel-mode` so that partial `.coverage.*` files are written atomically and can be safely combined afterwards. 4. **Merge and report step** — after all workers complete, run `coverage combine` followed by `coverage report` / `coverage html` in a single serial step. The chosen approach must keep total coverage at ≥ 97 % and must not break the existing `nox -e coverage_report` interface consumed by other nox sessions and CI jobs. ## Subtasks - [ ] Profile the current `coverage_report` session to establish a baseline wall-clock time - [ ] Research and evaluate parallel Behave execution strategies (sharding, `behave-parallel`, multiprocessing harness) - [ ] Prototype the chosen strategy locally and verify coverage data integrity after `coverage combine` - [ ] Update the `noxfile.py` `coverage_report` session to use the parallel strategy - [ ] Validate that overall coverage remains ≥ 97 % after parallelisation - [ ] Update CI configuration if the session interface or artefact paths change - [ ] Add or update Behave scenarios covering the new parallel execution path - [ ] Document the approach and any tuning knobs (e.g., worker count) in `CONTRIBUTING.md` or inline nox comments ## Definition of Done - [ ] `nox -e coverage_report` completes measurably faster than the single-process baseline - [ ] Coverage remains ≥ 97 % after parallelisation - [ ] `coverage combine` produces a single `.coverage` file from all parallel workers without data loss - [ ] No existing nox session or CI job interface is broken - [ ] Behave scenarios for the new execution path are present and passing - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 11:58:56 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: Medium (confirmed)
  • Milestone: v3.8.0 (confirmed — CI infrastructure)
  • MoSCoW: Could Have — Parallelizing coverage report generation is a CI performance optimization. Not blocking any deliverables.
  • Parent Epic: #1678 (confirmed correct)

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium (confirmed) - **Milestone**: v3.8.0 (confirmed — CI infrastructure) - **MoSCoW**: Could Have — Parallelizing coverage report generation is a CI performance optimization. Not blocking any deliverables. - **Parent Epic**: #1678 (confirmed correct) --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Author
Owner

Closing as duplicate of #2299.

This issue ("Parallelize coverage report generation") covers the same scope as #2299 ("Optimize coverage report generation to avoid a separate sequential test run"). Consolidating to avoid duplicate work.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Closing as duplicate of #2299. This issue ("Parallelize coverage report generation") covers the same scope as #2299 ("Optimize coverage report generation to avoid a separate sequential test run"). Consolidating to avoid duplicate work. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2263
No description provided.