Parallelize E2E test execution in CI to reduce pipeline wall-clock time #1961

Open
opened 2026-04-03 00:24:18 +00:00 by freemo · 2 comments
Owner

Metadata

  • Branch: task/parallelize-e2e-tests
  • Commit Message: perf(ci): parallelize E2E test execution to reduce pipeline wall-clock time
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The E2E test suite currently runs sequentially in the e2e_tests CI job. As the number of
end-to-end scenarios grows, this becomes a significant bottleneck in the CI pipeline's
wall-clock time. Parallelizing the E2E tests — either by splitting them across multiple
runner jobs or by enabling concurrent execution within a single job — will reduce feedback
latency for contributors and make the CI pipeline more efficient overall.

This issue is a child of Epic #1678 (CI Execution Time Optimization — Timeouts, Concurrency,
and Coverage Artifact Sharing), which tracks broader CI performance improvements.

Current Behavior

All E2E test scenarios execute sequentially in a single e2e_tests job. There is no
sharding, matrix strategy, or intra-job concurrency applied to the E2E suite.

Expected Behavior

E2E tests are split across parallel workers (via a matrix strategy or equivalent mechanism)
so that the total wall-clock time for the E2E stage is reduced proportionally to the number
of parallel workers used.

Acceptance Criteria

  • E2E tests are distributed across multiple parallel workers (e.g., via a GitHub/Forgejo
    Actions matrix strategy or a test-runner sharding flag).
  • All E2E scenarios continue to pass after parallelization.
  • No test scenario is skipped or duplicated across workers.
  • The e2e_tests job in .forgejo/workflows/ci.yml reflects the parallel configuration.
  • features/ci_workflow_validation.feature is updated to assert the parallel/matrix
    configuration is present.
  • Total E2E wall-clock time is measurably reduced compared to the sequential baseline.

Supporting Information

  • Parent Epic: #1678 — CI Execution Time Optimization
  • Related: concurrency groups, job-level timeouts, and coverage artifact sharing tracked in
    sibling issues under Epic #1678.
  • Forgejo Actions supports strategy.matrix for job-level parallelism; the test runner
    (Robot Framework / Behave) may also support --processes or shard flags for intra-job
    concurrency.

Subtasks

  • Investigate available parallelism mechanisms: Forgejo Actions matrix vs. test-runner
    sharding (e.g., --processes N for Robot Framework, --parallel for Behave)
  • Choose and document the parallelism strategy (update relevant ADR or add inline
    comments in the workflow file)
  • Update .forgejo/workflows/ci.yml to implement the chosen parallel strategy for the
    e2e_tests job
  • Ensure each parallel worker receives a non-overlapping, complete subset of test
    scenarios (no duplication, no gaps)
  • Update features/ci_workflow_validation.feature to assert the parallel configuration
    is present in the workflow
  • Run the full nox suite locally to confirm all stages pass
  • Verify coverage ≥ 97% via nox -s coverage_report
  • Run nox (all default sessions) and fix any errors

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit
    Message in Metadata exactly (perf(ci): parallelize E2E test execution to reduce pipeline wall-clock time), followed by a blank line, then additional lines providing relevant
    details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata
    exactly (task/parallelize-e2e-tests).
  • The commit is submitted as a pull request to master, reviewed, and merged before
    this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/parallelize-e2e-tests` - **Commit Message**: `perf(ci): parallelize E2E test execution to reduce pipeline wall-clock time` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The E2E test suite currently runs sequentially in the `e2e_tests` CI job. As the number of end-to-end scenarios grows, this becomes a significant bottleneck in the CI pipeline's wall-clock time. Parallelizing the E2E tests — either by splitting them across multiple runner jobs or by enabling concurrent execution within a single job — will reduce feedback latency for contributors and make the CI pipeline more efficient overall. This issue is a child of Epic #1678 (CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing), which tracks broader CI performance improvements. ## Current Behavior All E2E test scenarios execute sequentially in a single `e2e_tests` job. There is no sharding, matrix strategy, or intra-job concurrency applied to the E2E suite. ## Expected Behavior E2E tests are split across parallel workers (via a matrix strategy or equivalent mechanism) so that the total wall-clock time for the E2E stage is reduced proportionally to the number of parallel workers used. ## Acceptance Criteria - [ ] E2E tests are distributed across multiple parallel workers (e.g., via a GitHub/Forgejo Actions matrix strategy or a test-runner sharding flag). - [ ] All E2E scenarios continue to pass after parallelization. - [ ] No test scenario is skipped or duplicated across workers. - [ ] The `e2e_tests` job in `.forgejo/workflows/ci.yml` reflects the parallel configuration. - [ ] `features/ci_workflow_validation.feature` is updated to assert the parallel/matrix configuration is present. - [ ] Total E2E wall-clock time is measurably reduced compared to the sequential baseline. ## Supporting Information - Parent Epic: #1678 — CI Execution Time Optimization - Related: concurrency groups, job-level timeouts, and coverage artifact sharing tracked in sibling issues under Epic #1678. - Forgejo Actions supports `strategy.matrix` for job-level parallelism; the test runner (Robot Framework / Behave) may also support `--processes` or shard flags for intra-job concurrency. ## Subtasks - [ ] Investigate available parallelism mechanisms: Forgejo Actions matrix vs. test-runner sharding (e.g., `--processes N` for Robot Framework, `--parallel` for Behave) - [ ] Choose and document the parallelism strategy (update relevant ADR or add inline comments in the workflow file) - [ ] Update `.forgejo/workflows/ci.yml` to implement the chosen parallel strategy for the `e2e_tests` job - [ ] Ensure each parallel worker receives a non-overlapping, complete subset of test scenarios (no duplication, no gaps) - [ ] Update `features/ci_workflow_validation.feature` to assert the parallel configuration is present in the workflow - [ ] Run the full nox suite locally to confirm all stages pass - [ ] Verify coverage ≥ 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions) and fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly (`perf(ci): parallelize E2E test execution to reduce pipeline wall-clock time`), followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly (`task/parallelize-e2e-tests`). - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 00:24:45 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Author
Owner

Label compliance fix applied:

  • Removed conflicting label: State/Unverified
  • Reason: This issue had both State/Unverified and State/Verified simultaneously, which is contradictory. Per CONTRIBUTING.md, an issue should have exactly one State/* label. Since State/Verified is the more advanced state, State/Unverified has been removed.

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: ca-backlog-groomer

Label compliance fix applied: - Removed conflicting label: `State/Unverified` - Reason: This issue had both `State/Unverified` and `State/Verified` simultaneously, which is contradictory. Per CONTRIBUTING.md, an issue should have exactly one `State/*` label. Since `State/Verified` is the more advanced state, `State/Unverified` has been removed. --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: ca-backlog-groomer
freemo removed this from the v3.8.0 milestone 2026-04-07 01:31:21 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1961
No description provided.