TEST-INFRA: [ci-pipeline-design] Implement matrix test strategies #1753

Open
opened 2026-04-02 23:42:06 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: feature/ci-pipeline-matrix-test-strategies
  • Commit Message: feat(ci): implement matrix test strategies for multi-version Python compatibility
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The CI pipeline currently runs tests against a single Python version. Implementing a matrix strategy will allow running tests against multiple Python versions, ensuring broader compatibility and catching version-specific issues early.

Without a matrix strategy, version-specific regressions can go undetected until they surface in production or on a developer's local environment running a different Python version. A matrix build provides a safety net that validates the codebase across the full range of supported interpreter versions on every CI run.

Expected Behavior

The CI pipeline should execute the full test suite in parallel across all supported Python versions (e.g., 3.11, 3.12, 3.13) using a matrix strategy defined in .forgejo/workflows/ci.yml. Each matrix leg should be independently reported, and a failure in any leg should fail the overall CI run.

Acceptance Criteria

  • .forgejo/workflows/ci.yml defines a matrix strategy covering all supported Python versions
  • Each matrix leg runs the full test suite independently
  • A failure in any matrix leg causes the overall CI run to fail
  • Matrix configuration is documented in the CI workflow file with inline comments
  • features/ci_workflow_validation.feature is updated to assert matrix strategy is present and correct
  • All nox stages pass after changes
  • Coverage >= 97%

Supporting Information

  • Parent Epic: #1678 — CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing
  • Related: Forgejo Actions matrix strategy documentation

Subtasks

  • Identify all supported Python versions for the project
  • Add strategy.matrix block to the relevant job(s) in .forgejo/workflows/ci.yml
  • Parameterize the Python version in the setup-python step using the matrix variable
  • Ensure each matrix leg uploads its own coverage artifact with a version-specific name
  • Update features/ci_workflow_validation.feature to assert matrix strategy presence
  • Tests (Behave): Add/update scenarios for CI matrix validation
  • Verify coverage >= 97% via nox -s coverage_report
  • Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly (feat(ci): implement matrix test strategies for multi-version Python compatibility), followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly (feature/ci-pipeline-matrix-test-strategies).
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `feature/ci-pipeline-matrix-test-strategies` - **Commit Message**: `feat(ci): implement matrix test strategies for multi-version Python compatibility` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The CI pipeline currently runs tests against a single Python version. Implementing a matrix strategy will allow running tests against multiple Python versions, ensuring broader compatibility and catching version-specific issues early. Without a matrix strategy, version-specific regressions can go undetected until they surface in production or on a developer's local environment running a different Python version. A matrix build provides a safety net that validates the codebase across the full range of supported interpreter versions on every CI run. ## Expected Behavior The CI pipeline should execute the full test suite in parallel across all supported Python versions (e.g., 3.11, 3.12, 3.13) using a matrix strategy defined in `.forgejo/workflows/ci.yml`. Each matrix leg should be independently reported, and a failure in any leg should fail the overall CI run. ## Acceptance Criteria - [ ] `.forgejo/workflows/ci.yml` defines a `matrix` strategy covering all supported Python versions - [ ] Each matrix leg runs the full test suite independently - [ ] A failure in any matrix leg causes the overall CI run to fail - [ ] Matrix configuration is documented in the CI workflow file with inline comments - [ ] `features/ci_workflow_validation.feature` is updated to assert matrix strategy is present and correct - [ ] All nox stages pass after changes - [ ] Coverage >= 97% ## Supporting Information - Parent Epic: #1678 — CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing - Related: Forgejo Actions matrix strategy documentation ## Subtasks - [ ] Identify all supported Python versions for the project - [ ] Add `strategy.matrix` block to the relevant job(s) in `.forgejo/workflows/ci.yml` - [ ] Parameterize the Python version in the `setup-python` step using the matrix variable - [ ] Ensure each matrix leg uploads its own coverage artifact with a version-specific name - [ ] Update `features/ci_workflow_validation.feature` to assert matrix strategy presence - [ ] Tests (Behave): Add/update scenarios for CI matrix validation - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly (`feat(ci): implement matrix test strategies for multi-version Python compatibility`), followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly (`feature/ci-pipeline-matrix-test-strategies`). - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.7.0 milestone 2026-04-02 23:42:40 +00:00
freemo modified the milestone from v3.7.0 to v3.8.0 2026-04-02 23:43:22 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1753
No description provided.