TEST-INFRA: [ci-pipeline-design] Implement matrix test strategies for broader coverage #3068

Open
opened 2026-04-05 04:52:06 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: task/ci-matrix-test-strategies
  • Commit Message: feat(ci): implement matrix test strategies for broader configuration coverage
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

Our current CI pipeline does not utilise matrix testing, which limits our ability to validate the codebase against different runtime configurations and environments. A matrix strategy allows a single workflow job to fan out across multiple combinations of variables (Python version, OS, dependency version), catching regressions that only surface in specific environments before they reach production.

Expected Behavior

The CI pipeline should run the full test suite across a defined matrix of configurations so that failures in any supported environment are caught automatically on every push and pull request.

Acceptance Criteria

  • A matrix strategy is defined in .forgejo/workflows/ci.yml covering at minimum: Python versions 3.9, 3.10, and 3.11.
  • The matrix job runs nox (all default sessions) for each combination.
  • features/ci_workflow_validation.feature is updated to assert that the matrix configuration is present and correct.
  • All nox stages pass across every matrix leg.
  • Coverage ≥ 97% is maintained.

Supporting Information

  • Existing CI workflow: .forgejo/workflows/ci.yml
  • Behave unit tests for CI validation: features/ci_workflow_validation.feature
  • Related issue (dependency caching): #3062
  • Parent Epic (CI Execution Time Optimization): #1678

Subtasks

  • Identify the most important configurations to test against (Python versions, OS variants, key dependency versions).
  • Add a matrix strategy block to the relevant job(s) in .forgejo/workflows/ci.yml.
  • Ensure fail-fast: false is set so all matrix legs are reported even when one fails.
  • Update features/ci_workflow_validation.feature to assert matrix configuration steps are present.
  • Run nox (all default sessions) locally and fix any errors surfaced by the matrix changes.
  • Verify coverage ≥ 97% after changes.

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A matrix test strategy is implemented in .forgejo/workflows/ci.yml and the CI pipeline tests against a wider range of configurations.
  • features/ci_workflow_validation.feature covers the new matrix configuration.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage ≥ 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-matrix-test-strategies` - **Commit Message**: `feat(ci): implement matrix test strategies for broader configuration coverage` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context Our current CI pipeline does not utilise matrix testing, which limits our ability to validate the codebase against different runtime configurations and environments. A matrix strategy allows a single workflow job to fan out across multiple combinations of variables (Python version, OS, dependency version), catching regressions that only surface in specific environments before they reach production. ## Expected Behavior The CI pipeline should run the full test suite across a defined matrix of configurations so that failures in any supported environment are caught automatically on every push and pull request. ## Acceptance Criteria - A `matrix` strategy is defined in `.forgejo/workflows/ci.yml` covering at minimum: Python versions 3.9, 3.10, and 3.11. - The matrix job runs `nox` (all default sessions) for each combination. - `features/ci_workflow_validation.feature` is updated to assert that the matrix configuration is present and correct. - All nox stages pass across every matrix leg. - Coverage ≥ 97% is maintained. ## Supporting Information - Existing CI workflow: `.forgejo/workflows/ci.yml` - Behave unit tests for CI validation: `features/ci_workflow_validation.feature` - Related issue (dependency caching): #3062 - Parent Epic (CI Execution Time Optimization): #1678 ## Subtasks - [ ] Identify the most important configurations to test against (Python versions, OS variants, key dependency versions). - [ ] Add a `matrix` strategy block to the relevant job(s) in `.forgejo/workflows/ci.yml`. - [ ] Ensure `fail-fast: false` is set so all matrix legs are reported even when one fails. - [ ] Update `features/ci_workflow_validation.feature` to assert matrix configuration steps are present. - [ ] Run `nox` (all default sessions) locally and fix any errors surfaced by the matrix changes. - [ ] Verify coverage ≥ 97% after changes. ## Definition of Done This issue is complete when: - [ ] All subtasks above are completed and checked off. - [ ] A matrix test strategy is implemented in `.forgejo/workflows/ci.yml` and the CI pipeline tests against a wider range of configurations. - [ ] `features/ci_workflow_validation.feature` covers the new matrix configuration. - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - [ ] All nox stages pass. - [ ] Coverage ≥ 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-05 04:52:22 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: Medium (retained)
  • Milestone: v3.8.0 (retained)
  • MoSCoW: Could Have — matrix test strategies for broader coverage are desirable but not essential. Current test coverage meets the ≥97% threshold.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium (retained) - **Milestone**: v3.8.0 (retained) - **MoSCoW**: Could Have — matrix test strategies for broader coverage are desirable but not essential. Current test coverage meets the ≥97% threshold. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#3068
No description provided.