TEST-INFRA: [ci-execution-time] Investigate and optimize slow e2e_tests suite (14m+ runtime) #2318

Closed
opened 2026-04-03 14:12:04 +00:00 by freemo · 2 comments
Owner

Metadata

  • Branch: chore/ci-e2e-tests-optimization
  • Commit Message: perf(ci): investigate and optimize slow e2e_tests suite runtime
  • Milestone: v3.8.0
  • Parent Epic: #1678

Summary

The e2e_tests suite takes over 14 minutes to run. While E2E tests are expected to be slower, this runtime can be improved.

Analysis

The noxfile.py shows that these tests run against real LLM API keys, which introduces network latency.

Subtasks

  • Investigate the possibility of using mocked services for a faster E2E suite that can run on every PR.
  • Profile the existing E2E tests to identify any non-network-related bottlenecks.
  • Consider running the full E2E suite less frequently (e.g., nightly) if a faster, mocked version is available.
  • Document findings and proposed strategy in the relevant docs.
  • Run nox (all default sessions), fix any errors.
  • Verify coverage >= 97% via nox -s coverage_report.

Definition of Done

  • A strategy for faster E2E testing is implemented.
  • The runtime of the PR-blocking E2E tests is significantly reduced.
  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `chore/ci-e2e-tests-optimization` - **Commit Message**: `perf(ci): investigate and optimize slow e2e_tests suite runtime` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Summary The `e2e_tests` suite takes over 14 minutes to run. While E2E tests are expected to be slower, this runtime can be improved. ### Analysis The `noxfile.py` shows that these tests run against real LLM API keys, which introduces network latency. ## Subtasks - [ ] Investigate the possibility of using mocked services for a faster E2E suite that can run on every PR. - [ ] Profile the existing E2E tests to identify any non-network-related bottlenecks. - [ ] Consider running the full E2E suite less frequently (e.g., nightly) if a faster, mocked version is available. - [ ] Document findings and proposed strategy in the relevant docs. - [ ] Run `nox` (all default sessions), fix any errors. - [ ] Verify coverage >= 97% via `nox -s coverage_report`. ## Definition of Done - [ ] A strategy for faster E2E testing is implemented. - [ ] The runtime of the PR-blocking E2E tests is significantly reduced. - [ ] All subtasks above are completed and checked off. - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - [ ] All nox stages pass. - [ ] Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 14:12:12 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: Medium (confirmed)
  • Milestone: v3.8.0 (confirmed — CI infrastructure)
  • MoSCoW: Could Have — E2E test optimization is a CI performance improvement. Not blocking any deliverables.
  • Parent Epic: #1678 (confirmed correct)

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium (confirmed) - **Milestone**: v3.8.0 (confirmed — CI infrastructure) - **MoSCoW**: Could Have — E2E test optimization is a CI performance improvement. Not blocking any deliverables. - **Parent Epic**: #1678 (confirmed correct) --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Author
Owner

Closing as duplicate of #2336.

This issue ("Investigate and optimize slow e2e_tests suite") covers the same scope as #2336 ("Parallelize E2E tests using pabot"). The investigation and optimization of the E2E suite is the core of what #2336 addresses. Consolidating to avoid duplicate work.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Closing as duplicate of #2336. This issue ("Investigate and optimize slow e2e_tests suite") covers the same scope as #2336 ("Parallelize E2E tests using pabot"). The investigation and optimization of the E2E suite is the core of what #2336 addresses. Consolidating to avoid duplicate work. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
freemo 2026-04-03 17:22:32 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2318
No description provided.