TEST-INFRA: [ci-execution-time] Analyze and split slow E2E tests #2341

New issue

Closed

opened 2026-04-03 14:56:10 +00:00 by freemo · 2 comments

freemo commented

2026-04-03 14:56:10 +00:00

Owner

Metadata

Branch: task/ci-execution-time-split-slow-e2e-tests
Commit Message: perf(ci): analyze and split slow E2E tests to reduce pipeline execution time
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The e2e_tests job in the CI pipeline has a 45-minute timeout, indicating it is a significant bottleneck. This long execution time slows down the feedback loop for developers and increases the overall cost of CI. The E2E test suite has grown organically and has not been profiled or restructured for parallelism, resulting in a monolithic job that cannot be optimized without first understanding which tests are the slowest and why.

This issue is a child of Epic #1678 (CI Execution Time Optimization), which addresses structural CI pipeline gaps. Splitting and parallelizing slow E2E tests directly reduces the wall-clock time of the e2e_tests job.

Current Behavior

The e2e_tests CI job runs all E2E tests sequentially in a single job with a 45-minute timeout. There is no test performance profiling in place, no visibility into which individual tests are the slowest, and no parallelism configured for the E2E suite.

Expected Behavior

The slowest E2E tests are identified via profiling, refactored into smaller independent tests, and the CI configuration is updated to run them in parallel — significantly reducing the overall e2e_tests job execution time.

Acceptance Criteria

A test performance profiling mechanism is set up for the E2E suite and produces a ranked list of the slowest tests.
The slowest tests are analyzed and a documented plan exists for splitting them into smaller, independent tests.
The identified slow tests are refactored into smaller, parallelizable tests with no loss of coverage.
The CI configuration is updated to run the new smaller E2E tests in parallel.
The overall execution time of the e2e_tests job is measurably and significantly reduced compared to the baseline.

Supporting Information

Parent Epic: #1678 — CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing
The e2e_tests job currently has timeout-minutes: 45, which is the only job in the pipeline with an explicit timeout (see Epic #1678 child issue for adding timeouts to all jobs).
Parallelizing E2E tests complements the concurrency group and artifact-sharing work tracked in Epic #1678.

Subtasks

Set up test performance profiling for the E2E suite to identify the slowest tests (e.g., via pytest-benchmark, pytest-timeout reporting, or Robot Framework's built-in elapsed time output).
Run the profiling and produce a ranked list of the top slowest E2E tests with their individual execution times.
Analyze the identified bottlenecks and create a documented plan for splitting them into smaller, independent, parallelizable tests.
Refactor the slow E2E tests into smaller, focused tests per the documented plan.
Update the CI configuration (.forgejo/workflows/ci.yml) to run the new smaller E2E tests in parallel.
Update features/ci_workflow_validation.feature to assert E2E parallelism configuration is present.
Verify all nox stages pass after changes.
Confirm coverage ≥ 97% via nox -s coverage_report.

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
The slowest E2E tests are identified, profiled, and documented.
The identified tests are split into smaller, parallelizable tests with no regression in coverage.
The CI configuration runs the new E2E tests in parallel and the e2e_tests job execution time is measurably reduced.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-execution-time-split-slow-e2e-tests` - **Commit Message**: `perf(ci): analyze and split slow E2E tests to reduce pipeline execution time` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The `e2e_tests` job in the CI pipeline has a 45-minute timeout, indicating it is a significant bottleneck. This long execution time slows down the feedback loop for developers and increases the overall cost of CI. The E2E test suite has grown organically and has not been profiled or restructured for parallelism, resulting in a monolithic job that cannot be optimized without first understanding which tests are the slowest and why. This issue is a child of Epic #1678 (CI Execution Time Optimization), which addresses structural CI pipeline gaps. Splitting and parallelizing slow E2E tests directly reduces the wall-clock time of the `e2e_tests` job. ## Current Behavior The `e2e_tests` CI job runs all E2E tests sequentially in a single job with a 45-minute timeout. There is no test performance profiling in place, no visibility into which individual tests are the slowest, and no parallelism configured for the E2E suite. ## Expected Behavior The slowest E2E tests are identified via profiling, refactored into smaller independent tests, and the CI configuration is updated to run them in parallel — significantly reducing the overall `e2e_tests` job execution time. ## Acceptance Criteria - A test performance profiling mechanism is set up for the E2E suite and produces a ranked list of the slowest tests. - The slowest tests are analyzed and a documented plan exists for splitting them into smaller, independent tests. - The identified slow tests are refactored into smaller, parallelizable tests with no loss of coverage. - The CI configuration is updated to run the new smaller E2E tests in parallel. - The overall execution time of the `e2e_tests` job is measurably and significantly reduced compared to the baseline. ## Supporting Information - Parent Epic: #1678 — CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing - The `e2e_tests` job currently has `timeout-minutes: 45`, which is the only job in the pipeline with an explicit timeout (see Epic #1678 child issue for adding timeouts to all jobs). - Parallelizing E2E tests complements the concurrency group and artifact-sharing work tracked in Epic #1678. ## Subtasks - [ ] Set up test performance profiling for the E2E suite to identify the slowest tests (e.g., via pytest-benchmark, pytest-timeout reporting, or Robot Framework's built-in elapsed time output). - [ ] Run the profiling and produce a ranked list of the top slowest E2E tests with their individual execution times. - [ ] Analyze the identified bottlenecks and create a documented plan for splitting them into smaller, independent, parallelizable tests. - [ ] Refactor the slow E2E tests into smaller, focused tests per the documented plan. - [ ] Update the CI configuration (`.forgejo/workflows/ci.yml`) to run the new smaller E2E tests in parallel. - [ ] Update `features/ci_workflow_validation.feature` to assert E2E parallelism configuration is present. - [ ] Verify all nox stages pass after changes. - [ ] Confirm coverage ≥ 97% via `nox -s coverage_report`. ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - The slowest E2E tests are identified, profiled, and documented. - The identified tests are split into smaller, parallelizable tests with no regression in coverage. - The CI configuration runs the new E2E tests in parallel and the `e2e_tests` job execution time is measurably reduced. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-03 14:56:15 +00:00

freemo added a new dependency

2026-04-03 14:56:52 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo commented

2026-04-03 14:57:03 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: High (confirmed)
Milestone: v3.8.0 (confirmed — CI infrastructure)
MoSCoW: Could Have — E2E test profiling and splitting is a CI optimization. Related to but distinct from the parallelization issues (#1961, #2278). This focuses on identifying and splitting slow tests, while those focus on running them in parallel.
Parent Epic: #1678 (confirmed correct)

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: High (confirmed) - **Milestone**: v3.8.0 (confirmed — CI infrastructure) - **MoSCoW**: Could Have — E2E test profiling and splitting is a CI optimization. Related to but distinct from the parallelization issues (#1961, #2278). This focuses on identifying and splitting slow tests, while those focus on running them in parallel. - **Parent Epic**: #1678 (confirmed correct) --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo added the

MoSCoW

Could have

label

2026-04-03 14:57:03 +00:00

freemo commented

2026-04-03 17:22:24 +00:00

Author

Owner

Closing as duplicate of #2336.

This issue ("Analyze and split slow E2E tests") is a subset of the work described in #2336 ("Parallelize E2E tests using pabot"). The analysis and splitting of slow E2E tests is a natural subtask of the parallelization effort. Consolidating to avoid duplicate work.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Closing as duplicate of #2336. This issue ("Analyze and split slow E2E tests") is a subset of the work described in #2336 ("Parallelize E2E tests using pabot"). The analysis and splitting of slow E2E tests is a natural subtask of the parallelization effort. Consolidating to avoid duplicate work. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo

2026-04-03 17:22:31 +00:00