TEST-INFRA: [ci-pipeline-design] Implement matrix strategy for multi-version testing #2452

New issue

Open

opened 2026-04-03 18:25:06 +00:00 by freemo · 1 comment

freemo commented

2026-04-03 18:25:06 +00:00

Owner

Metadata

Branch: feature/ci-matrix-multi-version-testing
Commit Message: test(ci): implement matrix strategy for multi-version Python testing
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The CI pipeline currently only runs tests on a single Python version (3.13). This means that version-specific regressions — such as behaviour differences in the standard library, type annotation handling, or third-party dependency compatibility — can go undetected until they surface in production or in a user's environment.

The project targets multiple Python versions (3.11, 3.12, 3.13). Without a matrix strategy, the CI pipeline provides an incomplete quality signal and cannot guarantee that the codebase is compatible with all supported runtimes.

Expected Behavior

The .forgejo/workflows/ci.yml workflow should use a GitHub Actions-style matrix strategy to run the full test suite (lint, typecheck, unit tests, integration tests, coverage) across all supported Python versions (3.11, 3.12, 3.13) on every pull request and push to master.

Acceptance Criteria

The ci.yml workflow defines a matrix with python-version: [3.11, 3.12, 3.13].
All test jobs (at minimum: unit_tests, integration_tests, coverage) are parameterised by the matrix.
The CI pipeline passes on all three Python versions.
The features/ci_workflow_validation.feature Behave scenario is updated to assert that the matrix strategy is present and covers all supported versions.

Supporting Information

Related Epic: #1678 — CI Execution Time Optimization
Related issue: #2438 — Consolidate redundant Docker setup in CI jobs
Workflow file: .forgejo/workflows/ci.yml

Subtasks

Update .forgejo/workflows/ci.yml to add a strategy.matrix block with python-version: [3.11, 3.12, 3.13]
Parameterise all relevant CI jobs to use ${{ matrix.python-version }} for the Python setup step
Verify that all nox sessions pass locally against Python 3.11 and 3.12 (in addition to 3.13)
Update features/ci_workflow_validation.feature to assert the matrix strategy is present and covers all supported versions
Run nox (all default sessions) and fix any errors
Verify coverage >= 97% via nox -s coverage_report

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `feature/ci-matrix-multi-version-testing` - **Commit Message**: `test(ci): implement matrix strategy for multi-version Python testing` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The CI pipeline currently only runs tests on a single Python version (3.13). This means that version-specific regressions — such as behaviour differences in the standard library, type annotation handling, or third-party dependency compatibility — can go undetected until they surface in production or in a user's environment. The project targets multiple Python versions (3.11, 3.12, 3.13). Without a matrix strategy, the CI pipeline provides an incomplete quality signal and cannot guarantee that the codebase is compatible with all supported runtimes. ## Expected Behavior The `.forgejo/workflows/ci.yml` workflow should use a GitHub Actions-style matrix strategy to run the full test suite (lint, typecheck, unit tests, integration tests, coverage) across all supported Python versions (3.11, 3.12, 3.13) on every pull request and push to `master`. ## Acceptance Criteria - The `ci.yml` workflow defines a `matrix` with `python-version: [3.11, 3.12, 3.13]`. - All test jobs (at minimum: `unit_tests`, `integration_tests`, `coverage`) are parameterised by the matrix. - The CI pipeline passes on all three Python versions. - The `features/ci_workflow_validation.feature` Behave scenario is updated to assert that the matrix strategy is present and covers all supported versions. ## Supporting Information - Related Epic: #1678 — CI Execution Time Optimization - Related issue: #2438 — Consolidate redundant Docker setup in CI jobs - Workflow file: `.forgejo/workflows/ci.yml` ## Subtasks - [ ] Update `.forgejo/workflows/ci.yml` to add a `strategy.matrix` block with `python-version: [3.11, 3.12, 3.13]` - [ ] Parameterise all relevant CI jobs to use `${{ matrix.python-version }}` for the Python setup step - [ ] Verify that all nox sessions pass locally against Python 3.11 and 3.12 (in addition to 3.13) - [ ] Update `features/ci_workflow_validation.feature` to assert the matrix strategy is present and covers all supported versions - [ ] Run `nox` (all default sessions) and fix any errors - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-03 18:25:10 +00:00

freemo added a new dependency

2026-04-03 18:26:07 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo added the

MoSCoW

Could have

label

2026-04-03 18:34:45 +00:00

freemo commented

2026-04-03 18:35:12 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
MoSCoW: Could Have — Desirable improvement but not necessary for the milestone. Include only if time permits.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: Could Have — Desirable improvement but not necessary for the milestone. Include only if time permits. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo referenced this issue

2026-04-03 18:37:27 +00:00

[CA-AUTO] UAT Pool Supervisor — v3.x — Session Tracker #2435

freemo removed the

MoSCoW

Could have

label

2026-04-05 03:45:46 +00:00