ci(integration_tests): optimize integration_tests suite execution time #1765

Open
opened 2026-04-02 23:45:42 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: chore/ci-optimize-integration-tests-suite
  • Commit Message: perf(ci): optimize integration_tests suite to reduce CI wall-clock time
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The integration_tests suite is a significant bottleneck in the CI pipeline, with an average execution time of over 20 minutes. This slows down the development feedback loop and increases the time contributors wait for pull request validation. As part of the broader CI Execution Time Optimization Epic (#1678), this issue targets the integration_tests job specifically.

Expected Behavior

The integration_tests suite should complete in a materially shorter wall-clock time, achieved through a combination of parallelization, test data reduction, and targeted optimization of the slowest individual tests.

Acceptance Criteria

  • Integration tests are split into parallel jobs (or shards) that run concurrently in CI.
  • Test data used by integration tests is analyzed and reduced where possible without sacrificing coverage.
  • The top 3 slowest integration tests are identified via profiling and optimized.
  • The measured wall-clock time of the integration_tests suite is reduced compared to the baseline.
  • All nox stages pass after changes.
  • Coverage remains ≥ 97%.

Subtasks

  • Establish a baseline measurement: record the current average wall-clock time of the integration_tests suite in CI.
  • Analyze the integration tests and identify opportunities for parallelization (job sharding or parallel steps).
  • Implement parallel execution for the integration tests in .forgejo/workflows/ci.yml.
  • Profile the integration tests to identify the top 3 slowest individual tests.
  • Propose and implement targeted optimizations for the identified slow tests.
  • Analyze test data used in integration tests; reduce data volume where it does not impact coverage.
  • Measure and document the post-optimization wall-clock time and compare to baseline.
  • Update features/ci_workflow_validation.feature to assert parallel execution steps are present.
  • Verify coverage ≥ 97% via nox -s coverage_report.
  • Run nox (all default sessions) and fix any errors.

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly (perf(ci): optimize integration_tests suite to reduce CI wall-clock time), followed by a blank line, then additional lines providing relevant implementation details.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly (chore/ci-optimize-integration-tests-suite).
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `chore/ci-optimize-integration-tests-suite` - **Commit Message**: `perf(ci): optimize integration_tests suite to reduce CI wall-clock time` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The `integration_tests` suite is a significant bottleneck in the CI pipeline, with an average execution time of over 20 minutes. This slows down the development feedback loop and increases the time contributors wait for pull request validation. As part of the broader CI Execution Time Optimization Epic (#1678), this issue targets the `integration_tests` job specifically. ## Expected Behavior The `integration_tests` suite should complete in a materially shorter wall-clock time, achieved through a combination of parallelization, test data reduction, and targeted optimization of the slowest individual tests. ## Acceptance Criteria - [ ] Integration tests are split into parallel jobs (or shards) that run concurrently in CI. - [ ] Test data used by integration tests is analyzed and reduced where possible without sacrificing coverage. - [ ] The top 3 slowest integration tests are identified via profiling and optimized. - [ ] The measured wall-clock time of the `integration_tests` suite is reduced compared to the baseline. - [ ] All nox stages pass after changes. - [ ] Coverage remains ≥ 97%. ## Subtasks - [ ] Establish a baseline measurement: record the current average wall-clock time of the `integration_tests` suite in CI. - [ ] Analyze the integration tests and identify opportunities for parallelization (job sharding or parallel steps). - [ ] Implement parallel execution for the integration tests in `.forgejo/workflows/ci.yml`. - [ ] Profile the integration tests to identify the top 3 slowest individual tests. - [ ] Propose and implement targeted optimizations for the identified slow tests. - [ ] Analyze test data used in integration tests; reduce data volume where it does not impact coverage. - [ ] Measure and document the post-optimization wall-clock time and compare to baseline. - [ ] Update `features/ci_workflow_validation.feature` to assert parallel execution steps are present. - [ ] Verify coverage ≥ 97% via `nox -s coverage_report`. - [ ] Run `nox` (all default sessions) and fix any errors. ## Definition of Done This issue is complete when: - [ ] All subtasks above are completed and checked off. - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly (`perf(ci): optimize integration_tests suite to reduce CI wall-clock time`), followed by a blank line, then additional lines providing relevant implementation details. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly (`chore/ci-optimize-integration-tests-suite`). - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - [ ] All nox stages pass. - [ ] Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-02 23:48:12 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1765
No description provided.