TEST-INFRA: [ci-pipeline-design] Split integration tests for better parallelism #1968

New issue

Open

opened 2026-04-03 00:26:24 +00:00 by freemo · 0 comments

freemo commented

2026-04-03 00:26:24 +00:00

Owner

Metadata

Branch: chore/ci-split-integration-tests-parallelism
Commit Message: chore(ci): split integration tests into granular chunks for improved parallelism
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The current CI pipeline runs integration tests in parallel using pabot. However, the tests are not split into sufficiently granular chunks, which limits the degree of parallelism achievable and results in longer build times. Improving the test splitting strategy would allow more parallel workers to be utilized effectively, reducing overall pipeline wall-clock time.

Current Behavior

Integration tests are executed via pabot but without fine-grained test splitting. All tests are distributed at a coarse level, leaving potential parallelism gains unrealized.

Expected Behavior

Integration tests are split into smaller, more granular chunks — either via pabot's --testlevelsplit option or by manually organizing tests into smaller suites — so that more parallel workers can be utilized and build time is significantly reduced.

Acceptance Criteria

The --testlevelsplit option of pabot has been investigated and its suitability assessed.
Manual test suite splitting has been investigated as an alternative or complement.
A test splitting strategy has been chosen and implemented.
The chosen strategy demonstrably reduces CI build time for the integration test stage.
All integration tests continue to pass after the split.

Supporting Information

Related to the CI Execution Time Optimization Epic: #1678
pabot docs: https://github.com/mkorpela/pabot
Similar CI pipeline design issues: #1911, #1905, #1888

Subtasks

Investigate pabot's --testlevelsplit option and document findings
Investigate manually splitting integration tests into smaller suites
Evaluate trade-offs between --testlevelsplit and manual splitting
Implement the chosen test splitting strategy in the CI pipeline configuration
Verify that all integration tests still pass after the split
Measure and document the reduction in build time
Run nox (all default sessions), fix any errors
Verify coverage >= 97% via nox -s coverage_report

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly (chore(ci): split integration tests into granular chunks for improved parallelism), followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly (chore/ci-split-integration-tests-parallelism).
The integration tests are split into smaller, more granular chunks.
The build time is measurably and significantly reduced.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `chore/ci-split-integration-tests-parallelism` - **Commit Message**: `chore(ci): split integration tests into granular chunks for improved parallelism` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The current CI pipeline runs integration tests in parallel using `pabot`. However, the tests are not split into sufficiently granular chunks, which limits the degree of parallelism achievable and results in longer build times. Improving the test splitting strategy would allow more parallel workers to be utilized effectively, reducing overall pipeline wall-clock time. ## Current Behavior Integration tests are executed via `pabot` but without fine-grained test splitting. All tests are distributed at a coarse level, leaving potential parallelism gains unrealized. ## Expected Behavior Integration tests are split into smaller, more granular chunks — either via `pabot`'s `--testlevelsplit` option or by manually organizing tests into smaller suites — so that more parallel workers can be utilized and build time is significantly reduced. ## Acceptance Criteria - The `--testlevelsplit` option of `pabot` has been investigated and its suitability assessed. - Manual test suite splitting has been investigated as an alternative or complement. - A test splitting strategy has been chosen and implemented. - The chosen strategy demonstrably reduces CI build time for the integration test stage. - All integration tests continue to pass after the split. ## Supporting Information - Related to the CI Execution Time Optimization Epic: #1678 - `pabot` docs: https://github.com/mkorpela/pabot - Similar CI pipeline design issues: #1911, #1905, #1888 ## Subtasks - [ ] Investigate `pabot`'s `--testlevelsplit` option and document findings - [ ] Investigate manually splitting integration tests into smaller suites - [ ] Evaluate trade-offs between `--testlevelsplit` and manual splitting - [ ] Implement the chosen test splitting strategy in the CI pipeline configuration - [ ] Verify that all integration tests still pass after the split - [ ] Measure and document the reduction in build time - [ ] Run `nox` (all default sessions), fix any errors - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly (`chore(ci): split integration tests into granular chunks for improved parallelism`), followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly (`chore/ci-split-integration-tests-parallelism`). - The integration tests are split into smaller, more granular chunks. - The build time is measurably and significantly reduced. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-03 00:26:50 +00:00

freemo added a new dependency

2026-04-03 00:30:23 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo referenced this issue

2026-04-03 09:31:27 +00:00

TEST-INFRA: [ci-execution-time] Increase default parallelism for tests #2209

freemo referenced this issue

2026-04-05 08:49:48 +00:00

[CA-AUTO] Project Owner — project-owner-1 — Session Tracker #2793

freemo referenced this issue