TEST-INFRA: [ci-pipeline-design] Implement Matrix Test Strategy for Parallel Python Version Testing #2223

Open
opened 2026-04-03 09:38:09 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: chore/ci-matrix-python-version-testing
  • Commit Message: chore(ci): implement matrix test strategy for parallel python version testing
  • Milestone: v3.7.0
  • Parent Epic: #1678

Background and Context

The CI pipeline currently runs tests for different Python versions sequentially. This means each Python version's test suite must complete before the next one begins, resulting in unnecessarily long total pipeline execution times. A matrix test strategy would run these tests in parallel across all supported Python versions simultaneously, significantly reducing the overall CI execution time and providing faster feedback to developers.

This is particularly important as the project scales and the test suite grows — sequential execution will become an increasingly significant bottleneck.

Current Behavior

Tests for different Python versions (e.g., Python 3.10, 3.11, 3.12) are executed one after another in the CI pipeline. The total execution time is the sum of all individual version test runs.

Expected Behavior

The CI pipeline should use a matrix strategy to run tests for all supported Python versions in parallel. The total execution time should be approximately equal to the longest single-version test run, rather than the sum of all runs.

Acceptance Criteria

  • CI pipeline configuration defines a matrix of all supported Python versions
  • Tests for each Python version run in parallel (not sequentially)
  • All matrix jobs must pass for the overall CI check to succeed
  • Matrix configuration is DRY — Python version list is defined once and reused
  • CI pipeline correctly reports per-version results in the job summary
  • No regression in test coverage reporting (coverage artifacts are correctly aggregated or reported per-version)

Supporting Information

  • Parent Epic: #1678 (CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing)
  • Related to overall CI execution time reduction efforts tracked in the parent epic
  • Refer to the project's nox configuration for the list of supported Python versions

Subtasks

  • Audit current CI pipeline configuration to identify sequential Python version test steps
  • Determine the full list of supported Python versions from nox / pyproject.toml
  • Refactor CI pipeline YAML to use a matrix strategy for Python version testing
  • Validate that all matrix jobs run in parallel in a test pipeline run
  • Ensure coverage reporting is not broken by parallelisation (aggregate or per-version)
  • Update CI documentation / comments to reflect the new matrix strategy
  • Tests (Behave/Robot): Verify CI pipeline configuration is valid and matrix is correctly defined
  • Verify coverage >= 97% via nox -s coverage_report
  • Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly (chore(ci): implement matrix test strategy for parallel python version testing), followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly (chore/ci-matrix-python-version-testing).
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `chore/ci-matrix-python-version-testing` - **Commit Message**: `chore(ci): implement matrix test strategy for parallel python version testing` - **Milestone**: v3.7.0 - **Parent Epic**: #1678 ## Background and Context The CI pipeline currently runs tests for different Python versions sequentially. This means each Python version's test suite must complete before the next one begins, resulting in unnecessarily long total pipeline execution times. A matrix test strategy would run these tests in parallel across all supported Python versions simultaneously, significantly reducing the overall CI execution time and providing faster feedback to developers. This is particularly important as the project scales and the test suite grows — sequential execution will become an increasingly significant bottleneck. ## Current Behavior Tests for different Python versions (e.g., Python 3.10, 3.11, 3.12) are executed one after another in the CI pipeline. The total execution time is the sum of all individual version test runs. ## Expected Behavior The CI pipeline should use a matrix strategy to run tests for all supported Python versions in parallel. The total execution time should be approximately equal to the longest single-version test run, rather than the sum of all runs. ## Acceptance Criteria - [ ] CI pipeline configuration defines a matrix of all supported Python versions - [ ] Tests for each Python version run in parallel (not sequentially) - [ ] All matrix jobs must pass for the overall CI check to succeed - [ ] Matrix configuration is DRY — Python version list is defined once and reused - [ ] CI pipeline correctly reports per-version results in the job summary - [ ] No regression in test coverage reporting (coverage artifacts are correctly aggregated or reported per-version) ## Supporting Information - Parent Epic: #1678 (CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing) - Related to overall CI execution time reduction efforts tracked in the parent epic - Refer to the project's nox configuration for the list of supported Python versions ## Subtasks - [ ] Audit current CI pipeline configuration to identify sequential Python version test steps - [ ] Determine the full list of supported Python versions from `nox` / `pyproject.toml` - [ ] Refactor CI pipeline YAML to use a matrix strategy for Python version testing - [ ] Validate that all matrix jobs run in parallel in a test pipeline run - [ ] Ensure coverage reporting is not broken by parallelisation (aggregate or per-version) - [ ] Update CI documentation / comments to reflect the new matrix strategy - [ ] Tests (Behave/Robot): Verify CI pipeline configuration is valid and matrix is correctly defined - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly (`chore(ci): implement matrix test strategy for parallel python version testing`), followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly (`chore/ci-matrix-python-version-testing`). - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.7.0 milestone 2026-04-03 09:38:20 +00:00
freemo self-assigned this 2026-04-03 16:57:58 +00:00
Author
Owner

MoSCoW classification: Should Have

Rationale: This issue addresses an important spec requirement or quality improvement. The project should include this fix but it is not strictly essential for the milestone.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

MoSCoW classification: **Should Have** Rationale: This issue addresses an important spec requirement or quality improvement. The project should include this fix but it is not strictly essential for the milestone. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2223
No description provided.