TEST-INFRA: [ci-execution-time] Investigate and optimize slow integration_tests suite (22m+ runtime) #2315

Open
opened 2026-04-03 14:08:42 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: chore/ci-integration-tests-optimization
  • Commit Message: perf(ci): investigate and optimize slow integration_tests suite runtime
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The integration_tests suite currently takes over 22 minutes to run, making it a significant bottleneck in the CI/CD pipeline. This slows down development feedback loops and contributes to excessive CI wall-clock time — a problem already tracked at the Epic level in #1678.

The noxfile.py shows that the tests are run in parallel using pabot, but with a conservative default of only 2 processes. The tests themselves may also contain inefficiencies that compound the runtime.

Current Behavior

The integration_tests CI job runs for 22+ minutes, blocking downstream jobs and consuming runner capacity for an extended period.

Expected Behavior

The integration_tests suite runtime is reduced by at least 50% (target: ≤ 11 minutes), achieved through a combination of profiling, parallelism tuning, and test-level optimizations.

Acceptance Criteria

  • The integration_tests suite completes in ≤ 11 minutes on CI runners (≥ 50% reduction from current 22m+ baseline).
  • Any increase in pabot parallelism is validated to not cause test flakiness or resource contention.
  • All changes are documented (noxfile comments, CI workflow notes, or a brief ADR entry if architectural).

Supporting Information

  • Parent Epic: #1678 — CI Execution Time Optimization
  • noxfile.py uses pabot with a default of 2 parallel processes for integration_tests.
  • Profiling should identify the slowest individual tests and suites before optimizations are applied.

Subtasks

  • Profile the integration_tests suite to identify the slowest tests and suites.
  • Investigate increasing the number of parallel pabot processes on CI runners and validate stability.
  • Optimize the slowest-running individual tests identified during profiling.
  • Consider splitting the test suite into smaller, independently parallelizable CI jobs.
  • Update noxfile.py and/or .forgejo/workflows/ci.yml with the chosen optimizations.
  • Document all changes (inline comments, CI workflow notes, or ADR entry as appropriate).
  • Verify all nox stages pass after changes via nox.
  • Confirm coverage ≥ 97% via nox -s coverage_report.

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • The integration_tests suite runtime on CI is reduced by at least 50% from the 22m+ baseline.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `chore/ci-integration-tests-optimization` - **Commit Message**: `perf(ci): investigate and optimize slow integration_tests suite runtime` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The `integration_tests` suite currently takes over 22 minutes to run, making it a significant bottleneck in the CI/CD pipeline. This slows down development feedback loops and contributes to excessive CI wall-clock time — a problem already tracked at the Epic level in #1678. The `noxfile.py` shows that the tests are run in parallel using `pabot`, but with a conservative default of only 2 processes. The tests themselves may also contain inefficiencies that compound the runtime. ## Current Behavior The `integration_tests` CI job runs for 22+ minutes, blocking downstream jobs and consuming runner capacity for an extended period. ## Expected Behavior The `integration_tests` suite runtime is reduced by at least 50% (target: ≤ 11 minutes), achieved through a combination of profiling, parallelism tuning, and test-level optimizations. ## Acceptance Criteria - The `integration_tests` suite completes in ≤ 11 minutes on CI runners (≥ 50% reduction from current 22m+ baseline). - Any increase in `pabot` parallelism is validated to not cause test flakiness or resource contention. - All changes are documented (noxfile comments, CI workflow notes, or a brief ADR entry if architectural). ## Supporting Information - Parent Epic: #1678 — CI Execution Time Optimization - `noxfile.py` uses `pabot` with a default of 2 parallel processes for `integration_tests`. - Profiling should identify the slowest individual tests and suites before optimizations are applied. ## Subtasks - [ ] Profile the `integration_tests` suite to identify the slowest tests and suites. - [ ] Investigate increasing the number of parallel `pabot` processes on CI runners and validate stability. - [ ] Optimize the slowest-running individual tests identified during profiling. - [ ] Consider splitting the test suite into smaller, independently parallelizable CI jobs. - [ ] Update `noxfile.py` and/or `.forgejo/workflows/ci.yml` with the chosen optimizations. - [ ] Document all changes (inline comments, CI workflow notes, or ADR entry as appropriate). - [ ] Verify all nox stages pass after changes via `nox`. - [ ] Confirm coverage ≥ 97% via `nox -s coverage_report`. ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - The `integration_tests` suite runtime on CI is reduced by at least 50% from the 22m+ baseline. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 14:08:51 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: High (already set, correct — 22m+ integration test runtime is a significant bottleneck)
  • Milestone: v3.8.0 (correct)
  • MoSCoW: Could Have — CI optimization is valuable but not blocking milestone delivery.
  • Parent Epic: #1678

This is the canonical issue for integration test optimization. Duplicates #2328, #2330, #2332, #2303, #2309 have been closed and consolidated here.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified ✅ - **Priority**: High (already set, correct — 22m+ integration test runtime is a significant bottleneck) - **Milestone**: v3.8.0 (correct) - **MoSCoW**: Could Have — CI optimization is valuable but not blocking milestone delivery. - **Parent Epic**: #1678 This is the canonical issue for integration test optimization. Duplicates #2328, #2330, #2332, #2303, #2309 have been closed and consolidated here. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2315
No description provided.