Optimize coverage report generation to eliminate sequential test re-execution #1808

Open
opened 2026-04-02 23:53:38 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: task/ci-optimize-coverage-report-generation
  • Commit Message: perf(ci): parallelize coverage data collection and eliminate sequential test re-execution
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The coverage_report nox session currently runs the entire Behave suite sequentially from scratch in order to generate a coverage report. This is a major bottleneck in the CI pipeline: the unit_tests job has already executed the same tests, yet the coverage job duplicates that work entirely by re-running them. This is the third structural gap identified in Epic #1678 (CI Execution Time Optimization).

Current Behavior

The coverage CI job calls nox -s coverage_report, which invokes the full Behave test suite sequentially. This means every test is run twice per CI pipeline: once in unit_tests and once in coverage. The duplication wastes runner time and increases overall pipeline wall-clock time significantly.

Expected Behavior

The unit_tests nox session should run tests with coverage enabled for each parallel process, producing per-process .coverage.* data files. After all parallel processes complete, the coverage data files are merged into a single .coverage file. The coverage_report session then generates the final report from the pre-merged data file — without re-running any tests.

Acceptance Criteria

  • The unit_tests nox session runs Behave with coverage run (or equivalent) per parallel process, producing individual .coverage.* data files.
  • A merge step within unit_tests (or a dedicated nox session) combines all .coverage.* files into a single .coverage artifact.
  • The coverage_report nox session generates the HTML/XML/terminal report from the merged .coverage file without invoking the test suite.
  • The generated coverage report is numerically identical (or within acceptable floating-point rounding) to the report produced by the previous sequential approach.
  • The CI pipeline unit_tests job uploads the merged .coverage file as a workflow artifact.
  • The CI pipeline coverage job downloads the artifact instead of re-running tests.
  • All nox default sessions pass without error.
  • Coverage remains ≥ 97%.

Supporting Information

  • Parent Epic: #1678 — "Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing"
  • Related child issues in the same Epic: concurrency groups (#child-1) and job-level timeouts (#child-2).
  • Python coverage tool supports coverage combine for merging parallel data files; pytest-cov and behave both support --cov / COVERAGE_PROCESS_START patterns for parallel coverage collection.

Subtasks

  • Modify the unit_tests nox session to invoke Behave with coverage enabled per parallel worker (e.g., via COVERAGE_PROCESS_START or coverage run wrapper).
  • Add a coverage combine step (within unit_tests or a new coverage_merge session) to merge all .coverage.* data files into a single .coverage file.
  • Modify the coverage_report nox session to generate the report from the merged .coverage file without re-running the test suite.
  • Update .forgejo/workflows/ci.yml: add an upload-artifact step in unit_tests to publish the merged .coverage file.
  • Update .forgejo/workflows/ci.yml: add a download-artifact step in the coverage job to consume the uploaded .coverage file.
  • Update features/ci_workflow_validation.feature to assert the artifact upload/download steps are present for coverage data.
  • Verify the generated coverage report matches the expected output from the previous sequential approach.
  • Run nox (all default sessions) and fix any errors.
  • Verify coverage ≥ 97% via nox -s coverage_report.

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • The coverage_report nox session no longer re-runs the test suite; it only generates the report from a pre-existing .coverage file.
  • The CI coverage job downloads the .coverage artifact produced by unit_tests rather than re-executing tests.
  • features/ci_workflow_validation.feature is updated to cover the artifact sharing steps.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-optimize-coverage-report-generation` - **Commit Message**: `perf(ci): parallelize coverage data collection and eliminate sequential test re-execution` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The `coverage_report` nox session currently runs the entire Behave suite sequentially from scratch in order to generate a coverage report. This is a major bottleneck in the CI pipeline: the `unit_tests` job has already executed the same tests, yet the `coverage` job duplicates that work entirely by re-running them. This is the third structural gap identified in Epic #1678 (CI Execution Time Optimization). ## Current Behavior The `coverage` CI job calls `nox -s coverage_report`, which invokes the full Behave test suite sequentially. This means every test is run twice per CI pipeline: once in `unit_tests` and once in `coverage`. The duplication wastes runner time and increases overall pipeline wall-clock time significantly. ## Expected Behavior The `unit_tests` nox session should run tests with coverage enabled for each parallel process, producing per-process `.coverage.*` data files. After all parallel processes complete, the coverage data files are merged into a single `.coverage` file. The `coverage_report` session then generates the final report from the pre-merged data file — without re-running any tests. ## Acceptance Criteria - The `unit_tests` nox session runs Behave with `coverage run` (or equivalent) per parallel process, producing individual `.coverage.*` data files. - A merge step within `unit_tests` (or a dedicated nox session) combines all `.coverage.*` files into a single `.coverage` artifact. - The `coverage_report` nox session generates the HTML/XML/terminal report from the merged `.coverage` file without invoking the test suite. - The generated coverage report is numerically identical (or within acceptable floating-point rounding) to the report produced by the previous sequential approach. - The CI pipeline `unit_tests` job uploads the merged `.coverage` file as a workflow artifact. - The CI pipeline `coverage` job downloads the artifact instead of re-running tests. - All nox default sessions pass without error. - Coverage remains ≥ 97%. ## Supporting Information - Parent Epic: #1678 — "Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing" - Related child issues in the same Epic: concurrency groups (#child-1) and job-level timeouts (#child-2). - Python `coverage` tool supports `coverage combine` for merging parallel data files; `pytest-cov` and `behave` both support `--cov` / `COVERAGE_PROCESS_START` patterns for parallel coverage collection. ## Subtasks - [ ] Modify the `unit_tests` nox session to invoke Behave with coverage enabled per parallel worker (e.g., via `COVERAGE_PROCESS_START` or `coverage run` wrapper). - [ ] Add a `coverage combine` step (within `unit_tests` or a new `coverage_merge` session) to merge all `.coverage.*` data files into a single `.coverage` file. - [ ] Modify the `coverage_report` nox session to generate the report from the merged `.coverage` file without re-running the test suite. - [ ] Update `.forgejo/workflows/ci.yml`: add an `upload-artifact` step in `unit_tests` to publish the merged `.coverage` file. - [ ] Update `.forgejo/workflows/ci.yml`: add a `download-artifact` step in the `coverage` job to consume the uploaded `.coverage` file. - [ ] Update `features/ci_workflow_validation.feature` to assert the artifact upload/download steps are present for coverage data. - [ ] Verify the generated coverage report matches the expected output from the previous sequential approach. - [ ] Run `nox` (all default sessions) and fix any errors. - [ ] Verify coverage ≥ 97% via `nox -s coverage_report`. ## Definition of Done This issue is complete when: - [ ] All subtasks above are completed and checked off. - [ ] The `coverage_report` nox session no longer re-runs the test suite; it only generates the report from a pre-existing `.coverage` file. - [ ] The CI `coverage` job downloads the `.coverage` artifact produced by `unit_tests` rather than re-executing tests. - [ ] `features/ci_workflow_validation.feature` is updated to cover the artifact sharing steps. - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - [ ] All nox stages pass. - [ ] Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-02 23:54:04 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1808
No description provided.