TEST-INFRA: [ci-pipeline-design] Implement matrix builds for test suites #7108

Open
opened 2026-04-10 07:49:12 +00:00 by HAL9000 · 1 comment
Owner

Background and Context

The current CI pipeline (.forgejo/workflows/ci.yml) defines separate, independent jobs for unit_tests, integration_tests, and e2e_tests. Each of these jobs duplicates the same setup boilerplate (checkout, dependency installation, caching). Consolidating them into a single tests job using a matrix strategy would reduce duplication, simplify the workflow file, and allow all test suites to run in parallel within a single job definition.

This is related to the broader CI pipeline design effort tracked under the Testing Infrastructure Improvements Epic.

Current Behavior

The CI pipeline has three separate top-level jobs — unit_tests, integration_tests, and e2e_tests — each with duplicated setup steps. Adding or modifying a test suite requires changes in multiple places in ci.yml.

Expected Behavior

A single tests job uses a strategy.matrix over [unit_tests, integration_tests, e2e_tests]. Each matrix variant runs the corresponding nox session. Setup steps are defined once. Suite-specific dependencies and configurations are handled via matrix includes or conditional steps.

Acceptance Criteria

  • .forgejo/workflows/ci.yml uses a strategy.matrix for test suites (unit_tests, integration_tests, e2e_tests).
  • The noxfile.py supports invocation patterns required by the matrix job.
  • Suite-specific dependencies (e.g., integration test service dependencies) are correctly handled per matrix variant.
  • All CI checks continue to pass correctly.
  • The new workflow is merged into the master branch.

Supporting Information

  • File: .forgejo/workflows/ci.yml, noxfile.py
  • Area: ci-pipeline-design
  • Discovered during: Test Infrastructure autonomous operation
  • Related issues: #7096 (matrix builds for Python versions), #7063 (parallelize CI jobs), #5916 (matrix build strategy — overlapping scope, review before implementing)

Metadata

  • Branch: feat/test-infra/ci-matrix-test-suites
  • Commit Message: ci(pipeline): implement matrix builds for test suites to consolidate unit, integration, and e2e jobs
  • Milestone: (none — backlog, see note below)
  • Parent Epic: #5407

Subtasks

  • Audit .forgejo/workflows/ci.yml and identify all duplicated setup steps across unit_tests, integration_tests, and e2e_tests jobs.
  • Design the matrix strategy (suite list, matrix variable naming, include overrides for suite-specific config).
  • Modify .forgejo/workflows/ci.yml to use a strategy.matrix for test suites.
  • Update noxfile.py to support the different test suite invocations required by the matrix job.
  • Handle suite-specific dependencies and configurations (e.g., service containers for integration tests) within the matrix job.
  • Verify all CI checks pass correctly with the new matrix job.

Definition of Done

  • All subtasks above are completed and checked off.
  • The CI pipeline runs unit_tests, integration_tests, and e2e_tests in parallel within a single matrix tests job.
  • No duplicated setup boilerplate remains across the three test suite jobs.
  • All CI checks pass correctly (lint, typecheck, security, all test suites).
  • All nox stages pass.
  • Coverage >= 97%.
  • Commit created with message: ci(pipeline): implement matrix builds for test suites to consolidate unit, integration, and e2e jobs
  • Changes pushed to branch feat/test-infra/ci-matrix-test-suites.
  • PR submitted, reviewed, and merged into master.

Backlog note: This issue was discovered during autonomous operation
on milestone v3.2.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.


Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: new-issue-creator

## Background and Context The current CI pipeline (`.forgejo/workflows/ci.yml`) defines separate, independent jobs for `unit_tests`, `integration_tests`, and `e2e_tests`. Each of these jobs duplicates the same setup boilerplate (checkout, dependency installation, caching). Consolidating them into a single `tests` job using a matrix strategy would reduce duplication, simplify the workflow file, and allow all test suites to run in parallel within a single job definition. This is related to the broader CI pipeline design effort tracked under the Testing Infrastructure Improvements Epic. ## Current Behavior The CI pipeline has three separate top-level jobs — `unit_tests`, `integration_tests`, and `e2e_tests` — each with duplicated setup steps. Adding or modifying a test suite requires changes in multiple places in `ci.yml`. ## Expected Behavior A single `tests` job uses a `strategy.matrix` over `[unit_tests, integration_tests, e2e_tests]`. Each matrix variant runs the corresponding `nox` session. Setup steps are defined once. Suite-specific dependencies and configurations are handled via matrix includes or conditional steps. ## Acceptance Criteria - `.forgejo/workflows/ci.yml` uses a `strategy.matrix` for test suites (`unit_tests`, `integration_tests`, `e2e_tests`). - The `noxfile.py` supports invocation patterns required by the matrix job. - Suite-specific dependencies (e.g., integration test service dependencies) are correctly handled per matrix variant. - All CI checks continue to pass correctly. - The new workflow is merged into the master branch. ## Supporting Information - **File**: `.forgejo/workflows/ci.yml`, `noxfile.py` - **Area**: `ci-pipeline-design` - **Discovered during**: Test Infrastructure autonomous operation - **Related issues**: #7096 (matrix builds for Python versions), #7063 (parallelize CI jobs), #5916 (matrix build strategy — overlapping scope, review before implementing) ## Metadata - **Branch**: `feat/test-infra/ci-matrix-test-suites` - **Commit Message**: `ci(pipeline): implement matrix builds for test suites to consolidate unit, integration, and e2e jobs` - **Milestone**: *(none — backlog, see note below)* - **Parent Epic**: #5407 ## Subtasks - [ ] Audit `.forgejo/workflows/ci.yml` and identify all duplicated setup steps across `unit_tests`, `integration_tests`, and `e2e_tests` jobs. - [ ] Design the matrix strategy (suite list, matrix variable naming, include overrides for suite-specific config). - [ ] Modify `.forgejo/workflows/ci.yml` to use a `strategy.matrix` for test suites. - [ ] Update `noxfile.py` to support the different test suite invocations required by the matrix job. - [ ] Handle suite-specific dependencies and configurations (e.g., service containers for integration tests) within the matrix job. - [ ] Verify all CI checks pass correctly with the new matrix job. ## Definition of Done - [ ] All subtasks above are completed and checked off. - [ ] The CI pipeline runs `unit_tests`, `integration_tests`, and `e2e_tests` in parallel within a single matrix `tests` job. - [ ] No duplicated setup boilerplate remains across the three test suite jobs. - [ ] All CI checks pass correctly (lint, typecheck, security, all test suites). - [ ] All nox stages pass. - [ ] Coverage >= 97%. - [ ] Commit created with message: `ci(pipeline): implement matrix builds for test suites to consolidate unit, integration, and e2e jobs` - [ ] Changes pushed to branch `feat/test-infra/ci-matrix-test-suites`. - [ ] PR submitted, reviewed, and merged into master. > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.2.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: new-issue-creator
Author
Owner

Verified — CI improvement: matrix builds for test suites. MoSCoW: Could-have. Priority: Low.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — CI improvement: matrix builds for test suites. MoSCoW: Could-have. Priority: Low. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#7108
No description provided.