TEST-INFRA: [ci-execution-time] Parallelize the E2E test suite #2322

Closed
opened 2026-04-03 14:20:32 +00:00 by freemo · 2 comments
Owner

Metadata

  • Branch: chore/ci-parallelize-e2e-test-suite
  • Commit Message: chore(ci): parallelize the E2E test suite to reduce pipeline execution time
  • Milestone: v3.2.0
  • Parent Epic: #1678

Background and Context

The E2E test suite is a significant bottleneck in the CI pipeline. With a 45-minute timeout, it is the longest-running stage and blocks all downstream pipeline steps from completing. As the codebase grows and more E2E scenarios are added, this problem will only worsen.

Parallelizing the E2E suite by splitting tests into multiple concurrent jobs is the most direct way to reduce wall-clock CI time for this stage. This aligns with the broader CI Execution Time Optimization effort tracked in Epic #1678.

Current Behavior

The E2E test suite runs as a single sequential job with a 45-minute timeout. All E2E scenarios execute one after another, meaning the total CI time for this stage is the sum of every individual test's runtime.

Expected Behavior

The E2E test suite should be split into multiple parallel jobs (e.g., by feature area, tag, or file-based sharding). Each parallel job runs a subset of the E2E scenarios concurrently, reducing the total wall-clock time for the E2E stage to approximately total_runtime / N where N is the number of parallel workers.

Acceptance Criteria

  • E2E tests are split into at least 2 parallel CI jobs using a sharding or tagging strategy
  • All E2E scenarios continue to pass after parallelization
  • Total wall-clock time for the E2E stage is measurably reduced compared to the current sequential run
  • No test isolation issues are introduced (each shard runs independently without shared mutable state)
  • Coverage data from all parallel shards is merged correctly before the coverage report is generated
  • The CI configuration documents the parallelization strategy

Supporting Information

  • The E2E suite currently has a 45-minute timeout, making it the primary CI bottleneck
  • Related issues: #2319 (Parallelize E2E and integration tests), #2278 (Parallelize E2E tests for faster execution), #2298 (Parallelize E2E tests for faster CI), #2273 (Investigate and optimize long-running E2E tests)
  • Parent Epic: #1678 — CI Execution Time Optimization

Subtasks

  • Audit the E2E test suite to identify natural sharding boundaries (by feature, tag, or file)
  • Design a parallelization strategy (e.g., pytest-split, manual tag-based sharding, or matrix jobs)
  • Update the CI pipeline configuration to run E2E shards as parallel jobs
  • Ensure coverage artifacts from all shards are collected and merged
  • Validate that all E2E scenarios pass across all shards
  • Measure and document the wall-clock time improvement
  • Run nox (all default sessions), fix any errors
  • Verify coverage >= 97% via nox -s coverage_report

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `chore/ci-parallelize-e2e-test-suite` - **Commit Message**: `chore(ci): parallelize the E2E test suite to reduce pipeline execution time` - **Milestone**: v3.2.0 - **Parent Epic**: #1678 ## Background and Context The E2E test suite is a significant bottleneck in the CI pipeline. With a 45-minute timeout, it is the longest-running stage and blocks all downstream pipeline steps from completing. As the codebase grows and more E2E scenarios are added, this problem will only worsen. Parallelizing the E2E suite by splitting tests into multiple concurrent jobs is the most direct way to reduce wall-clock CI time for this stage. This aligns with the broader CI Execution Time Optimization effort tracked in Epic #1678. ## Current Behavior The E2E test suite runs as a single sequential job with a 45-minute timeout. All E2E scenarios execute one after another, meaning the total CI time for this stage is the sum of every individual test's runtime. ## Expected Behavior The E2E test suite should be split into multiple parallel jobs (e.g., by feature area, tag, or file-based sharding). Each parallel job runs a subset of the E2E scenarios concurrently, reducing the total wall-clock time for the E2E stage to approximately `total_runtime / N` where `N` is the number of parallel workers. ## Acceptance Criteria - [ ] E2E tests are split into at least 2 parallel CI jobs using a sharding or tagging strategy - [ ] All E2E scenarios continue to pass after parallelization - [ ] Total wall-clock time for the E2E stage is measurably reduced compared to the current sequential run - [ ] No test isolation issues are introduced (each shard runs independently without shared mutable state) - [ ] Coverage data from all parallel shards is merged correctly before the coverage report is generated - [ ] The CI configuration documents the parallelization strategy ## Supporting Information - The E2E suite currently has a 45-minute timeout, making it the primary CI bottleneck - Related issues: #2319 (Parallelize E2E and integration tests), #2278 (Parallelize E2E tests for faster execution), #2298 (Parallelize E2E tests for faster CI), #2273 (Investigate and optimize long-running E2E tests) - Parent Epic: #1678 — CI Execution Time Optimization ## Subtasks - [ ] Audit the E2E test suite to identify natural sharding boundaries (by feature, tag, or file) - [ ] Design a parallelization strategy (e.g., pytest-split, manual tag-based sharding, or matrix jobs) - [ ] Update the CI pipeline configuration to run E2E shards as parallel jobs - [ ] Ensure coverage artifacts from all shards are collected and merged - [ ] Validate that all E2E scenarios pass across all shards - [ ] Measure and document the wall-clock time improvement - [ ] Run `nox` (all default sessions), fix any errors - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.2.0 milestone 2026-04-03 14:20:45 +00:00
Author
Owner

Duplicate of #1961 and #2278.

This issue describes the same optimization as:

  • #1961 ("Parallelize E2E test execution in CI to reduce pipeline wall-clock time", v3.8.0)
  • #2278 ("Parallelize E2E tests for faster execution", v3.8.0)

The issue body itself acknowledges the overlap: "Related issues: #2319, #2278, #2298, #2273". All of these request E2E test parallelization.

Closing as a duplicate.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

**Duplicate of #1961 and #2278.** This issue describes the same optimization as: - #1961 ("Parallelize E2E test execution in CI to reduce pipeline wall-clock time", v3.8.0) - #2278 ("Parallelize E2E tests for faster execution", v3.8.0) The issue body itself acknowledges the overlap: "Related issues: #2319, #2278, #2298, #2273". All of these request E2E test parallelization. Closing as a duplicate. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
freemo reopened this issue 2026-04-03 14:21:54 +00:00
Author
Owner

Closing as duplicate of #2336.

This issue ("Parallelize the E2E test suite") covers the same scope as #2336 ("Parallelize E2E tests using pabot for faster CI execution"). Issue #2336 is more specific and actionable — it identifies pabot as the tool and references the existing integration_tests pattern. Consolidating to avoid duplicate work.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Closing as duplicate of #2336. This issue ("Parallelize the E2E test suite") covers the same scope as #2336 ("Parallelize E2E tests using pabot for faster CI execution"). Issue #2336 is more specific and actionable — it identifies `pabot` as the tool and references the existing `integration_tests` pattern. Consolidating to avoid duplicate work. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
freemo 2026-04-03 17:22:29 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2322
No description provided.