TEST-INFRA: [ci-execution-time] Optimize parallel execution of Robot Framework integration tests #5662

Open
opened 2026-04-09 08:19:05 +00:00 by HAL9000 · 0 comments
Owner

Metadata

  • Branch: test-infra/optimize-robot-parallel-execution
  • Commit Message: perf(ci): optimize parallel execution of Robot Framework integration tests
  • Milestone: (none — see backlog note below)
  • Parent Epic: #5407

Overview

The Robot Framework integration test suite (nox -s integration_tests) is a significant contributor to the overall CI execution time. While the tests are already parallelized using pabot, there are opportunities to further optimize the parallel execution strategy to reduce the test run time.

Problem

The current pabot configuration uses a default strategy for splitting tests among parallel processes. This can lead to suboptimal load balancing, where some processes finish much earlier than others, leaving resources idle. Additionally, the setup and teardown of test environments for each parallel process can add significant overhead.

Proposal

  1. Implement a more granular test splitting strategy: Instead of relying on the default test splitting, explore splitting tests by suite or by file using the --testlevel and --ordering options of pabot.
  2. Optimize test environment setup: Investigate ways to optimize the setup and teardown of the test environment for each parallel process. This could involve using a shared test database, pre-building test data, or using a faster test environment provisioning mechanism.
  3. Introduce test execution profiles: Define different test execution profiles for different scenarios (e.g., a "smoke test" profile for a small subset of critical tests, and a "full" profile for all tests).

Duplicate Check

  • Search query: "integration tests", "pabot", "parallelization", "performance", "speed"
  • Results: No existing open or closed issues were found that specifically address the optimization of pabot or the parallel execution of integration tests.

Subtasks

  • Analyze the current pabot configuration and test execution times to establish a baseline.
  • Experiment with different test splitting strategies (--testlevel, --ordering) and measure impact.
  • Profile the test environment setup and teardown process to identify bottlenecks.
  • Implement a shared test database or other environment optimization techniques where applicable.
  • Define and implement test execution profiles (e.g., "smoke", "full") in noxfile.py.
  • Integrate the new test execution profiles into the CI pipeline (.forgejo/workflows/ci.yml).
  • Document the new parallel execution strategy and profiles in CONTRIBUTING.md.

Definition of Done

  • The average execution time of the integration_tests nox session is reduced by at least 20%.
  • The new parallel execution strategy is documented in CONTRIBUTING.md.
  • The new test execution profiles are integrated into the CI pipeline.
  • All nox stages pass.
  • Coverage >= 97%.

Backlog note: This issue was discovered during autonomous operation
on the current active milestone. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.


Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: new-issue-creator

## Metadata - **Branch**: `test-infra/optimize-robot-parallel-execution` - **Commit Message**: `perf(ci): optimize parallel execution of Robot Framework integration tests` - **Milestone**: *(none — see backlog note below)* - **Parent Epic**: #5407 ## Overview The Robot Framework integration test suite (`nox -s integration_tests`) is a significant contributor to the overall CI execution time. While the tests are already parallelized using `pabot`, there are opportunities to further optimize the parallel execution strategy to reduce the test run time. ### Problem The current `pabot` configuration uses a default strategy for splitting tests among parallel processes. This can lead to suboptimal load balancing, where some processes finish much earlier than others, leaving resources idle. Additionally, the setup and teardown of test environments for each parallel process can add significant overhead. ### Proposal 1. **Implement a more granular test splitting strategy**: Instead of relying on the default test splitting, explore splitting tests by suite or by file using the `--testlevel` and `--ordering` options of `pabot`. 2. **Optimize test environment setup**: Investigate ways to optimize the setup and teardown of the test environment for each parallel process. This could involve using a shared test database, pre-building test data, or using a faster test environment provisioning mechanism. 3. **Introduce test execution profiles**: Define different test execution profiles for different scenarios (e.g., a "smoke test" profile for a small subset of critical tests, and a "full" profile for all tests). ### Duplicate Check - Search query: "integration tests", "pabot", "parallelization", "performance", "speed" - Results: No existing open or closed issues were found that specifically address the optimization of `pabot` or the parallel execution of integration tests. ## Subtasks - [ ] Analyze the current `pabot` configuration and test execution times to establish a baseline. - [ ] Experiment with different test splitting strategies (`--testlevel`, `--ordering`) and measure impact. - [ ] Profile the test environment setup and teardown process to identify bottlenecks. - [ ] Implement a shared test database or other environment optimization techniques where applicable. - [ ] Define and implement test execution profiles (e.g., "smoke", "full") in `noxfile.py`. - [ ] Integrate the new test execution profiles into the CI pipeline (`.forgejo/workflows/ci.yml`). - [ ] Document the new parallel execution strategy and profiles in `CONTRIBUTING.md`. ## Definition of Done - [ ] The average execution time of the `integration_tests` nox session is reduced by at least 20%. - [ ] The new parallel execution strategy is documented in `CONTRIBUTING.md`. - [ ] The new test execution profiles are integrated into the CI pipeline. - [ ] All nox stages pass. - [ ] Coverage >= 97%. > **Backlog note:** This issue was discovered during autonomous operation > on the current active milestone. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: new-issue-creator
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#5662
No description provided.