TEST-INFRA: [ci-pipeline-design] Run tests on multiple Python versions #1948

Open
opened 2026-04-03 00:20:39 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: task/ci-multi-python-version-matrix
  • Commit Message: feat(ci): add multi-Python-version test matrix to CI pipeline
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The CleverAgents CI pipeline currently runs all test suites against a single, hardcoded Python version. This creates a blind spot: regressions introduced by Python minor-version differences (e.g., typing behaviour changes, stdlib additions/removals, deprecation warnings promoted to errors) go undetected until a user or contributor encounters them on a different interpreter.

The project targets a broad contributor base and is expected to run in diverse environments. Per the specification's quality mandate, the CI pipeline must enforce correctness across the full range of supported Python versions before any PR is merged.

Current Behaviour

.forgejo/workflows/ci.yml pins a single Python version (e.g., python-version: "3.12") for all jobs. No matrix strategy is applied. A change that silently breaks Python 3.11 or 3.13 compatibility will pass CI and be merged without warning.

Expected Behaviour

The unit_tests, integration_tests, and e2e_tests jobs (at minimum) should run in a matrix across all supported Python versions (e.g., 3.11, 3.12, 3.13). The pipeline should:

  1. Execute the full test suite on each version in the matrix.
  2. Fail the PR if any version fails.
  3. Report per-version results clearly in the CI summary.

Non-test jobs (lint, typecheck, security, build, docker, helm) may continue to run on a single canonical version to keep wall-clock time reasonable.

Acceptance Criteria

  • .forgejo/workflows/ci.yml defines a strategy.matrix for python-version covering at minimum ["3.11", "3.12", "3.13"] on the unit_tests job.
  • integration_tests and e2e_tests jobs also use the same matrix (or a documented subset with justification).
  • All matrix legs must pass for the CI run to be considered green.
  • Non-test jobs (lint, typecheck, security, coverage, build, docker, helm) continue to run on a single canonical version.
  • features/ci_workflow_validation.feature is updated to assert the matrix configuration is present and correct.
  • All nox stages pass after the changes.
  • Coverage ≥ 97% is maintained.

Supporting Information

  • Parent Epic: #1678 — CI Execution Time Optimization (this issue extends the CI hardening effort to version coverage).
  • Relevant CI file: .forgejo/workflows/ci.yml
  • Relevant BDD feature: features/ci_workflow_validation.feature
  • Python version support policy should be documented in CONTRIBUTING.md or the project README once decided.

Subtasks

  • Decide and document the supported Python version range (e.g., 3.11, 3.12, 3.13)
  • Add strategy.matrix.python-version to the unit_tests job in .forgejo/workflows/ci.yml
  • Extend the matrix to integration_tests and e2e_tests jobs (or document why a subset is used)
  • Ensure setup-python action uses ${{ matrix.python-version }} in all matrix jobs
  • Confirm non-test jobs remain pinned to a single canonical version
  • Update features/ci_workflow_validation.feature with scenarios asserting matrix presence
  • Run nox (all default sessions) locally and fix any errors
  • Verify coverage ≥ 97% via nox -s coverage_report

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly (feat(ci): add multi-Python-version test matrix to CI pipeline), followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly (task/ci-multi-python-version-matrix).
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-multi-python-version-matrix` - **Commit Message**: `feat(ci): add multi-Python-version test matrix to CI pipeline` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The CleverAgents CI pipeline currently runs all test suites against a single, hardcoded Python version. This creates a blind spot: regressions introduced by Python minor-version differences (e.g., typing behaviour changes, stdlib additions/removals, deprecation warnings promoted to errors) go undetected until a user or contributor encounters them on a different interpreter. The project targets a broad contributor base and is expected to run in diverse environments. Per the specification's quality mandate, the CI pipeline must enforce correctness across the full range of supported Python versions before any PR is merged. ## Current Behaviour `.forgejo/workflows/ci.yml` pins a single Python version (e.g., `python-version: "3.12"`) for all jobs. No matrix strategy is applied. A change that silently breaks Python 3.11 or 3.13 compatibility will pass CI and be merged without warning. ## Expected Behaviour The `unit_tests`, `integration_tests`, and `e2e_tests` jobs (at minimum) should run in a matrix across all supported Python versions (e.g., `3.11`, `3.12`, `3.13`). The pipeline should: 1. Execute the full test suite on each version in the matrix. 2. Fail the PR if any version fails. 3. Report per-version results clearly in the CI summary. Non-test jobs (lint, typecheck, security, build, docker, helm) may continue to run on a single canonical version to keep wall-clock time reasonable. ## Acceptance Criteria - [ ] `.forgejo/workflows/ci.yml` defines a `strategy.matrix` for `python-version` covering at minimum `["3.11", "3.12", "3.13"]` on the `unit_tests` job. - [ ] `integration_tests` and `e2e_tests` jobs also use the same matrix (or a documented subset with justification). - [ ] All matrix legs must pass for the CI run to be considered green. - [ ] Non-test jobs (lint, typecheck, security, coverage, build, docker, helm) continue to run on a single canonical version. - [ ] `features/ci_workflow_validation.feature` is updated to assert the matrix configuration is present and correct. - [ ] All nox stages pass after the changes. - [ ] Coverage ≥ 97% is maintained. ## Supporting Information - Parent Epic: #1678 — CI Execution Time Optimization (this issue extends the CI hardening effort to version coverage). - Relevant CI file: `.forgejo/workflows/ci.yml` - Relevant BDD feature: `features/ci_workflow_validation.feature` - Python version support policy should be documented in `CONTRIBUTING.md` or the project README once decided. ## Subtasks - [ ] Decide and document the supported Python version range (e.g., `3.11`, `3.12`, `3.13`) - [ ] Add `strategy.matrix.python-version` to the `unit_tests` job in `.forgejo/workflows/ci.yml` - [ ] Extend the matrix to `integration_tests` and `e2e_tests` jobs (or document why a subset is used) - [ ] Ensure `setup-python` action uses `${{ matrix.python-version }}` in all matrix jobs - [ ] Confirm non-test jobs remain pinned to a single canonical version - [ ] Update `features/ci_workflow_validation.feature` with scenarios asserting matrix presence - [ ] Run `nox` (all default sessions) locally and fix any errors - [ ] Verify coverage ≥ 97% via `nox -s coverage_report` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly (`feat(ci): add multi-Python-version test matrix to CI pipeline`), followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly (`task/ci-multi-python-version-matrix`). - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 00:22:44 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1948
No description provided.