TEST-INFRA: [ci-execution-time] Optimize `CI / integration_tests (pull_request)` #1707

New issue

Closed

opened 2026-04-02 23:31:43 +00:00 by freemo · 1 comment

freemo commented

2026-04-02 23:31:43 +00:00

Owner

Metadata

Branch: task/ci-optimize-integration-tests-execution-time
Commit Message: perf(ci): reduce integration_tests execution time via parallelization and caching
Milestone: v3.8.0
Parent Epic: #1678

Background and context

The CI / integration_tests (pull_request) test suite is one of the slowest jobs in the CI pipeline, with an average execution time of 1301.00 seconds (~21 minutes). This significantly slows down the developer feedback loop and increases the time required to merge pull requests, creating a bottleneck across all active milestones.

Current behavior

The integration_tests job runs sequentially as a single job with no parallelization, no test splitting, and no caching of dependencies or setup artifacts. The full suite must complete end-to-end on every PR, regardless of which areas of the codebase were changed.

Expected behavior

The CI / integration_tests (pull_request) job should complete in significantly less time (target: ≥30% reduction) by leveraging parallelization, test splitting, dependency caching, and setup optimization — without compromising test coverage or reliability.

Acceptance criteria

The average execution time of CI / integration_tests (pull_request) is reduced by at least 30% (target: ≤910 seconds).
All integration tests continue to pass with no reduction in coverage.
The CI configuration changes are validated by updated features/ci_workflow_validation.feature scenarios.
No flakiness is introduced by the parallelization strategy.

Subtasks

Profile the integration_tests Robot Framework suite to identify the slowest test files and setup/teardown bottlenecks.
Identify candidates for parallelization (independent test suites with no shared mutable state).
Design a matrix/parallel job strategy in .forgejo/workflows/ci.yml for splitting integration tests.
Implement dependency and asset caching for the integration_tests job (e.g., pip cache, Robot Framework library cache).
Implement the parallelization plan in the CI configuration.
Update features/ci_workflow_validation.feature to assert the new parallel job structure and caching steps are present.
Monitor and record execution time after changes to verify the ≥30% improvement.
Run nox (all default sessions), fix any errors.
Verify coverage ≥ 97% via nox -s coverage_report.

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
The average execution time of CI / integration_tests (pull_request) is reduced by at least 30%.
The changes are implemented without compromising test coverage or reliability.
All nox stages pass.
Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-optimize-integration-tests-execution-time` - **Commit Message**: `perf(ci): reduce integration_tests execution time via parallelization and caching` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and context The `CI / integration_tests (pull_request)` test suite is one of the slowest jobs in the CI pipeline, with an average execution time of **1301.00 seconds** (~21 minutes). This significantly slows down the developer feedback loop and increases the time required to merge pull requests, creating a bottleneck across all active milestones. ## Current behavior The `integration_tests` job runs sequentially as a single job with no parallelization, no test splitting, and no caching of dependencies or setup artifacts. The full suite must complete end-to-end on every PR, regardless of which areas of the codebase were changed. ## Expected behavior The `CI / integration_tests (pull_request)` job should complete in significantly less time (target: ≥30% reduction) by leveraging parallelization, test splitting, dependency caching, and setup optimization — without compromising test coverage or reliability. ## Acceptance criteria - The average execution time of `CI / integration_tests (pull_request)` is reduced by at least 30% (target: ≤910 seconds). - All integration tests continue to pass with no reduction in coverage. - The CI configuration changes are validated by updated `features/ci_workflow_validation.feature` scenarios. - No flakiness is introduced by the parallelization strategy. ## Subtasks - [ ] Profile the `integration_tests` Robot Framework suite to identify the slowest test files and setup/teardown bottlenecks. - [ ] Identify candidates for parallelization (independent test suites with no shared mutable state). - [ ] Design a matrix/parallel job strategy in `.forgejo/workflows/ci.yml` for splitting integration tests. - [ ] Implement dependency and asset caching for the `integration_tests` job (e.g., pip cache, Robot Framework library cache). - [ ] Implement the parallelization plan in the CI configuration. - [ ] Update `features/ci_workflow_validation.feature` to assert the new parallel job structure and caching steps are present. - [ ] Monitor and record execution time after changes to verify the ≥30% improvement. - [ ] Run `nox` (all default sessions), fix any errors. - [ ] Verify coverage ≥ 97% via `nox -s coverage_report`. ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - The average execution time of `CI / integration_tests (pull_request)` is reduced by at least 30%. - The changes are implemented without compromising test coverage or reliability. - All nox stages pass. - Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-02 23:32:00 +00:00

freemo added a new dependency

2026-04-02 23:36:37 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo commented

2026-04-02 23:41:33 +00:00

Author

Owner

Closing as duplicate. CI parallelization and execution time improvements are tracked in #1604 (setup consolidation), #1536 (parallelize static analysis), and #1632 (split integration tests).

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Closing as duplicate. CI parallelization and execution time improvements are tracked in #1604 (setup consolidation), #1536 (parallelize static analysis), and #1632 (split integration tests). --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo

2026-04-02 23:41:34 +00:00