TEST-INFRA: [ci-execution-time] Optimize e2e_tests CI job #1820

New issue

Open

opened 2026-04-02 23:55:37 +00:00 by freemo · 2 comments

freemo commented

2026-04-02 23:55:37 +00:00

Owner

Metadata

Branch: task/ci-execution-time-optimize-e2e-tests-job
Commit Message: perf(ci): optimize e2e_tests CI job execution time
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The e2e_tests CI job is one of the slowest jobs in the pipeline, taking over 15 minutes to complete. This significantly slows down the developer feedback loop and contributes to overall CI wall-clock time exceeding acceptable thresholds.

This issue tracks a holistic investigation and implementation of optimizations across the e2e_tests job — covering test analysis, parallelization, environment setup/teardown, and test data quality.

Current Behavior

The e2e_tests CI job runs sequentially and takes over 15 minutes to complete, creating a bottleneck in the CI pipeline and delaying developer feedback.

Expected Behavior

The e2e_tests CI job execution time is reduced by at least 50% (to under ~7.5 minutes), without degrading test coverage or reliability.

Acceptance Criteria

The e2e_tests CI job completes in under 7.5 minutes (≥50% reduction from current baseline).
No existing test scenarios are removed or skipped to achieve the time reduction.
Test coverage remains at or above the project threshold (≥97%).
All nox stages continue to pass after the changes.
Changes are documented in the relevant documentation and communicated to the team.

Supporting Information

Parent Epic: #1678 — CI Execution Time Optimization
Related issues: #1786 (Parallelize e2e_tests), #1759 (Optimize e2e_tests suite)
Affected area: test-infra / noxfile.py / CI pipeline configuration

Subtasks

Analyze test suite: Profile and identify the slowest tests within the e2e_tests suite; document findings.
Implement parallelization: Configure the CI job to run tests in parallel across multiple runners (see also #1786).
Optimize test setup/teardown: Review and streamline E2E environment setup and teardown for unnecessary overhead.
Optimize test data: Audit test data for realism and eliminate any data patterns causing unnecessary delays.
Verify coverage: Run nox -s coverage_report and confirm coverage ≥97%.
Run full nox suite: Run nox (all default sessions) and fix any errors introduced.
Document changes: Update relevant documentation and communicate optimizations to the team.

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
The e2e_tests CI job execution time is reduced by at least 50%.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly (perf(ci): optimize e2e_tests CI job execution time), followed by a blank line, then additional lines providing relevant implementation details.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly (task/ci-execution-time-optimize-e2e-tests-job).
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage ≥ 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-execution-time-optimize-e2e-tests-job` - **Commit Message**: `perf(ci): optimize e2e_tests CI job execution time` - **Milestone**: `v3.8.0` - **Parent Epic**: #1678 ## Background and Context The `e2e_tests` CI job is one of the slowest jobs in the pipeline, taking over 15 minutes to complete. This significantly slows down the developer feedback loop and contributes to overall CI wall-clock time exceeding acceptable thresholds. This issue tracks a holistic investigation and implementation of optimizations across the `e2e_tests` job — covering test analysis, parallelization, environment setup/teardown, and test data quality. ## Current Behavior The `e2e_tests` CI job runs sequentially and takes over 15 minutes to complete, creating a bottleneck in the CI pipeline and delaying developer feedback. ## Expected Behavior The `e2e_tests` CI job execution time is reduced by at least 50% (to under ~7.5 minutes), without degrading test coverage or reliability. ## Acceptance Criteria - The `e2e_tests` CI job completes in under 7.5 minutes (≥50% reduction from current baseline). - No existing test scenarios are removed or skipped to achieve the time reduction. - Test coverage remains at or above the project threshold (≥97%). - All nox stages continue to pass after the changes. - Changes are documented in the relevant documentation and communicated to the team. ## Supporting Information - Parent Epic: #1678 — CI Execution Time Optimization - Related issues: #1786 (Parallelize e2e_tests), #1759 (Optimize e2e_tests suite) - Affected area: `test-infra` / `noxfile.py` / CI pipeline configuration ## Subtasks - [ ] **Analyze test suite**: Profile and identify the slowest tests within the `e2e_tests` suite; document findings. - [ ] **Implement parallelization**: Configure the CI job to run tests in parallel across multiple runners (see also #1786). - [ ] **Optimize test setup/teardown**: Review and streamline E2E environment setup and teardown for unnecessary overhead. - [ ] **Optimize test data**: Audit test data for realism and eliminate any data patterns causing unnecessary delays. - [ ] **Verify coverage**: Run `nox -s coverage_report` and confirm coverage ≥97%. - [ ] **Run full nox suite**: Run `nox` (all default sessions) and fix any errors introduced. - [ ] **Document changes**: Update relevant documentation and communicate optimizations to the team. ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - The `e2e_tests` CI job execution time is reduced by at least 50%. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly (`perf(ci): optimize e2e_tests CI job execution time`), followed by a blank line, then additional lines providing relevant implementation details. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly (`task/ci-execution-time-optimize-e2e-tests-job`). - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage ≥ 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-02 23:57:12 +00:00

freemo commented

2026-04-02 23:57:27 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo added the

MoSCoW

Could have

label

2026-04-02 23:57:27 +00:00

freemo added a new dependency

2026-04-02 23:58:50 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo commented

2026-04-03 00:27:12 +00:00

Author

Owner