TEST-INFRA: [ci-execution-time] Investigate parallelizing the integration test suite #2328

New issue

Closed

opened 2026-04-03 14:28:35 +00:00 by freemo · 1 comment

freemo commented

2026-04-03 14:28:35 +00:00

Owner

Metadata

Branch: investigation/ci-execution-time-parallel-integration-tests
Commit Message: test(ci): investigate parallelizing integration test suite to reduce CI execution time
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The CI pipeline currently runs the integration test suite as a single sequential job. Similar to the E2E test suite — which has already been identified as a potential bottleneck in the CI execution time Epic (#1678) — the integration test suite may also contribute meaningfully to overall wall-clock time on shared runners.

As the project grows and the number of integration tests increases, a monolithic sequential integration job will become an increasingly significant bottleneck. Proactively investigating parallelization strategies now will allow the team to make an informed decision before the suite becomes a blocking constraint.

Expected Behavior

The integration test suite should be investigated for feasibility of splitting into parallel CI jobs (e.g., using a matrix strategy or explicit job splitting), with the goal of reducing the total CI execution time contributed by integration tests.

Acceptance Criteria

A clear analysis of the current integration test suite structure is documented (number of tests, approximate runtime, logical groupings).
Feasibility of parallelization is assessed — including any shared state, ordering dependencies, or fixture constraints that could prevent safe parallel execution.
At least one concrete parallelization strategy is proposed (e.g., pytest-xdist, job matrix by test module/group, or explicit split jobs in the CI workflow).
A recommendation is made: either proceed with parallelization (with a follow-up implementation issue) or document why it is not yet warranted.

Supporting Information

Parent Epic: #1678 — CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing
The E2E suite has been identified as a similar bottleneck; this issue applies the same lens to the integration suite.
Relevant CI workflow: .forgejo/workflows/ci.yml, job integration_tests.

Subtasks

Profile the current integration test suite: measure total runtime and identify the slowest tests/modules
Audit integration tests for shared state, ordering dependencies, or fixture constraints that affect parallelization safety
Research parallelization options: pytest-xdist, CI job matrix strategy, or explicit job splitting in .forgejo/workflows/ci.yml
Document findings and produce a written recommendation (proceed with parallelization or defer with rationale)
If proceeding: open a follow-up implementation issue linked to this investigation
Run nox (all default sessions), fix any errors
Verify coverage >= 97% via nox -s coverage_report

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A written investigation report (in the issue comments or a linked document) covers runtime profiling, parallelization feasibility, and a clear recommendation.
If parallelization is recommended, a follow-up implementation issue has been created and linked.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the investigation findings.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `investigation/ci-execution-time-parallel-integration-tests` - **Commit Message**: `test(ci): investigate parallelizing integration test suite to reduce CI execution time` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The CI pipeline currently runs the integration test suite as a single sequential job. Similar to the E2E test suite — which has already been identified as a potential bottleneck in the CI execution time Epic (#1678) — the integration test suite may also contribute meaningfully to overall wall-clock time on shared runners. As the project grows and the number of integration tests increases, a monolithic sequential integration job will become an increasingly significant bottleneck. Proactively investigating parallelization strategies now will allow the team to make an informed decision before the suite becomes a blocking constraint. ## Expected Behavior The integration test suite should be investigated for feasibility of splitting into parallel CI jobs (e.g., using a matrix strategy or explicit job splitting), with the goal of reducing the total CI execution time contributed by integration tests. ## Acceptance Criteria - A clear analysis of the current integration test suite structure is documented (number of tests, approximate runtime, logical groupings). - Feasibility of parallelization is assessed — including any shared state, ordering dependencies, or fixture constraints that could prevent safe parallel execution. - At least one concrete parallelization strategy is proposed (e.g., pytest-xdist, job matrix by test module/group, or explicit split jobs in the CI workflow). - A recommendation is made: either proceed with parallelization (with a follow-up implementation issue) or document why it is not yet warranted. ## Supporting Information - Parent Epic: #1678 — CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing - The E2E suite has been identified as a similar bottleneck; this issue applies the same lens to the integration suite. - Relevant CI workflow: `.forgejo/workflows/ci.yml`, job `integration_tests`. ## Subtasks - [ ] Profile the current integration test suite: measure total runtime and identify the slowest tests/modules - [ ] Audit integration tests for shared state, ordering dependencies, or fixture constraints that affect parallelization safety - [ ] Research parallelization options: pytest-xdist, CI job matrix strategy, or explicit job splitting in `.forgejo/workflows/ci.yml` - [ ] Document findings and produce a written recommendation (proceed with parallelization or defer with rationale) - [ ] If proceeding: open a follow-up implementation issue linked to this investigation - [ ] Run `nox` (all default sessions), fix any errors - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A written investigation report (in the issue comments or a linked document) covers runtime profiling, parallelization feasibility, and a clear recommendation. - If parallelization is recommended, a follow-up implementation issue has been created and linked. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the investigation findings. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-03 14:28:47 +00:00

freemo added a new dependency

2026-04-03 14:30:10 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo commented

2026-04-03 17:23:30 +00:00

Author

Owner

Closing as duplicate of #2315.

This issue ("Investigate parallelizing the integration test suite") covers the same scope as #2315 ("Investigate and optimize slow integration_tests suite (22m+ runtime)"), which is more comprehensive with specific performance targets. Consolidating to avoid duplicate work.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Closing as duplicate of #2315. This issue ("Investigate parallelizing the integration test suite") covers the same scope as #2315 ("Investigate and optimize slow integration_tests suite (22m+ runtime)"), which is more comprehensive with specific performance targets. Consolidating to avoid duplicate work. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo closed this issue

2026-04-03 17:23:56 +00:00

freemo referenced this issue

2026-04-03 17:23:59 +00:00

Proposal: improve ca-test-infra-improver — strengthen duplicate avoidance to prevent near-identical issues #2370

freemo referenced this issue

2026-04-03 17:24:36 +00:00

TEST-INFRA: [ci-execution-time] Investigate and optimize slow integration_tests suite (22m+ runtime) #2315