TEST-INFRA: [ci-execution-time] Optimize `coverage_report` session by running tests in parallel #2340

New issue

Closed

opened 2026-04-03 14:54:04 +00:00 by freemo · 1 comment

freemo commented

2026-04-03 14:54:04 +00:00

Owner

Metadata

Branch: task/ci-coverage-report-parallel-execution
Commit Message: perf(ci): optimize coverage_report nox session with parallel test execution
Milestone: v3.8.0
Parent Epic: #1678

Description

The coverage_report nox session currently runs the entire Behave test suite sequentially under slipcover to generate a coverage report. This is a significant bottleneck in the CI pipeline, as it does not take advantage of the parallel execution capabilities of the test suite.

Analysis

The noxfile.py defines the coverage_report session as follows:

@nox.session(python=DEFAULT_PYTHON, reuse_venv=True, venv_backend="uv")
def coverage_report(session: nox.Session):
    """Generate coverage report from Behave tests.
    ...
    """
    ...
    # Force sequential mode so slipcover wraps the entire process.
    session.env["BEHAVE_PARALLEL_COVERAGE"] = "1"

    # Wrap the entire behave-parallel run under slipcover.
    session.run(
        "python",
        "-m",
        "slipcover",
        ...
        "--",
        *behave_args,
        success_codes=[0, 1],
    )
    ...

The comment explicitly states that the session is forced to run in sequential mode. This is the root cause of the performance issue.

Proposed Solution

To improve the performance of the coverage_report session, we should investigate methods for collecting coverage data from parallel test runs. Here are a few potential approaches:

Use slipcover with parallel execution: Research if slipcover can be configured to work with the behave-parallel runner. This would be the ideal solution as it would require minimal changes to the existing setup.
Use coverage.py with parallel execution: The standard coverage package supports parallel execution. We can configure it to collect data from each test process and then use coverage combine to merge the results. This would require replacing slipcover with coverage.
Custom solution: If neither of the above options is feasible, we could develop a custom solution to collect and merge coverage data from the parallel test runs.

Subtasks

Research slipcover compatibility with behave-parallel parallel execution mode
Evaluate coverage.py parallel execution as an alternative to slipcover
Implement the chosen parallel coverage collection approach in noxfile.py
Update BEHAVE_PARALLEL_COVERAGE env var logic to support parallel mode
Tests (Behave): Add/update scenarios for coverage_report session behaviour
Verify the generated coverage report is accurate and complete
Verify coverage >= 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors
Update documentation if the coverage tooling changes

Definition of Done

The coverage_report session runs tests in parallel.
The generated coverage report is accurate and complete.
The overall execution time of the coverage_report session is significantly reduced.
All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-coverage-report-parallel-execution` - **Commit Message**: `perf(ci): optimize coverage_report nox session with parallel test execution` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Description The `coverage_report` nox session currently runs the entire Behave test suite sequentially under `slipcover` to generate a coverage report. This is a significant bottleneck in the CI pipeline, as it does not take advantage of the parallel execution capabilities of the test suite. ## Analysis The `noxfile.py` defines the `coverage_report` session as follows: ```python @nox.session(python=DEFAULT_PYTHON, reuse_venv=True, venv_backend="uv") def coverage_report(session: nox.Session): """Generate coverage report from Behave tests. ... """ ... # Force sequential mode so slipcover wraps the entire process. session.env["BEHAVE_PARALLEL_COVERAGE"] = "1" # Wrap the entire behave-parallel run under slipcover. session.run( "python", "-m", "slipcover", ... "--", *behave_args, success_codes=[0, 1], ) ... ``` The comment explicitly states that the session is forced to run in sequential mode. This is the root cause of the performance issue. ## Proposed Solution To improve the performance of the `coverage_report` session, we should investigate methods for collecting coverage data from parallel test runs. Here are a few potential approaches: 1. **Use `slipcover` with parallel execution:** Research if `slipcover` can be configured to work with the `behave-parallel` runner. This would be the ideal solution as it would require minimal changes to the existing setup. 2. **Use `coverage.py` with parallel execution:** The standard `coverage` package supports parallel execution. We can configure it to collect data from each test process and then use `coverage combine` to merge the results. This would require replacing `slipcover` with `coverage`. 3. **Custom solution:** If neither of the above options is feasible, we could develop a custom solution to collect and merge coverage data from the parallel test runs. ## Subtasks - [ ] Research `slipcover` compatibility with `behave-parallel` parallel execution mode - [ ] Evaluate `coverage.py` parallel execution as an alternative to `slipcover` - [ ] Implement the chosen parallel coverage collection approach in `noxfile.py` - [ ] Update `BEHAVE_PARALLEL_COVERAGE` env var logic to support parallel mode - [ ] Tests (Behave): Add/update scenarios for `coverage_report` session behaviour - [ ] Verify the generated coverage report is accurate and complete - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors - [ ] Update documentation if the coverage tooling changes ## Definition of Done - [ ] The `coverage_report` session runs tests in parallel. - [ ] The generated coverage report is accurate and complete. - [ ] The overall execution time of the `coverage_report` session is significantly reduced. - [ ] All subtasks above are completed and checked off. - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-03 14:54:08 +00:00

freemo added a new dependency

2026-04-03 14:55:31 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo commented

2026-04-03 17:23:44 +00:00

Author

Owner

Closing as duplicate of #2299.

This issue ("Optimize coverage_report session by running tests in parallel") covers the same scope as #2299 ("Optimize coverage report generation to avoid a separate sequential test run"). Both address the same root cause — the coverage report re-running tests unnecessarily. Consolidating to avoid duplicate work.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Closing as duplicate of #2299. This issue ("Optimize coverage_report session by running tests in parallel") covers the same scope as #2299 ("Optimize coverage report generation to avoid a separate sequential test run"). Both address the same root cause — the coverage report re-running tests unnecessarily. Consolidating to avoid duplicate work. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo closed this issue

2026-04-03 17:24:02 +00:00

freemo referenced this issue

2026-04-03 17:24:37 +00:00

TEST-INFRA: [ci-execution-time] Optimize coverage report generation to avoid a separate sequential test run #2299