TEST-INFRA: [ci-execution-time] Investigate and increase test parallelism #2333

Open
opened 2026-04-03 14:38:10 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: test/ci-execution-time-increase-test-parallelism
  • Commit Message: test(ci-execution-time): increase test parallelism via configurable _default_processes()
  • Milestone: v3.8.0
  • Parent Epic: #1678

Summary

The noxfile.py currently limits the parallelism for both unit and integration tests to a maximum of 2 processes via _default_processes() which is hardcoded to min(os.cpu_count(), 2). This is a conservative setting that may be under-utilizing the available resources on the CI runners, leading to longer than necessary CI execution times.

Finding

The _default_processes() function in noxfile.py is hardcoded to min(os.cpu_count(), 2). This means that even if the CI runners have many CPU cores, the test suites will only ever run on a maximum of two of them in parallel.

While the comment in the code mentions avoiding OOM errors on shared runners, this setting may be overly cautious and could be a significant bottleneck in the CI pipeline.

Proposed Solution

  1. Benchmark Higher Parallelism:

    • Modify the _default_processes() function to allow for a higher number of processes.
    • Run a series of benchmark tests on the CI infrastructure with different levels of parallelism (e.g., 4, 8, and the full number of available CPUs).
    • Monitor the execution time and memory usage for each benchmark run to find the optimal level of parallelism that balances speed and stability.
  2. Make Parallelism Configurable:

    • Instead of a hardcoded limit, the number of processes should be configurable via an environment variable (e.g., TEST_PROCESSES).
    • The _default_processes() function should be updated to use this environment variable if it is set, otherwise falling back to a safe default.
    • This will allow for easier tuning of the CI pipeline in the future without requiring code changes.

Subtasks

  • Audit the current _default_processes() function in noxfile.py and document the rationale for the min(os.cpu_count(), 2) cap
  • Benchmark CI runner resource availability (CPU count, available memory) to determine safe upper bounds for parallelism
  • Run benchmark tests with parallelism levels of 4, 8, and os.cpu_count() — record wall-clock time and peak memory usage for each
  • Update _default_processes() to read from a TEST_PROCESSES environment variable, falling back to a safe default derived from benchmarks
  • Update .forgejo/workflows/ci.yml to set TEST_PROCESSES to the optimal value determined by benchmarking
  • Update features/ci_workflow_validation.feature to assert the TEST_PROCESSES env var is set in the relevant CI jobs
  • Verify all nox stages pass after changes
  • Confirm coverage ≥ 97%

Definition of Done

  • The _default_processes() function in noxfile.py is updated to allow for a configurable level of parallelism via TEST_PROCESSES env var
  • The default parallelism for the unit_tests and integration_tests nox sessions is increased to a value that is proven to be both faster and stable on the CI infrastructure
  • The overall execution time for the CI pipeline is measurably reduced
  • The CI pipeline remains stable and does not experience an increase in OOM errors or other resource-related failures
  • All nox stages pass
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `test/ci-execution-time-increase-test-parallelism` - **Commit Message**: `test(ci-execution-time): increase test parallelism via configurable _default_processes()` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Summary The `noxfile.py` currently limits the parallelism for both unit and integration tests to a maximum of 2 processes via `_default_processes()` which is hardcoded to `min(os.cpu_count(), 2)`. This is a conservative setting that may be under-utilizing the available resources on the CI runners, leading to longer than necessary CI execution times. ## Finding The `_default_processes()` function in `noxfile.py` is hardcoded to `min(os.cpu_count(), 2)`. This means that even if the CI runners have many CPU cores, the test suites will only ever run on a maximum of two of them in parallel. While the comment in the code mentions avoiding OOM errors on shared runners, this setting may be overly cautious and could be a significant bottleneck in the CI pipeline. ## Proposed Solution 1. **Benchmark Higher Parallelism:** - Modify the `_default_processes()` function to allow for a higher number of processes. - Run a series of benchmark tests on the CI infrastructure with different levels of parallelism (e.g., 4, 8, and the full number of available CPUs). - Monitor the execution time and memory usage for each benchmark run to find the optimal level of parallelism that balances speed and stability. 2. **Make Parallelism Configurable:** - Instead of a hardcoded limit, the number of processes should be configurable via an environment variable (e.g., `TEST_PROCESSES`). - The `_default_processes()` function should be updated to use this environment variable if it is set, otherwise falling back to a safe default. - This will allow for easier tuning of the CI pipeline in the future without requiring code changes. ## Subtasks - [ ] Audit the current `_default_processes()` function in `noxfile.py` and document the rationale for the `min(os.cpu_count(), 2)` cap - [ ] Benchmark CI runner resource availability (CPU count, available memory) to determine safe upper bounds for parallelism - [ ] Run benchmark tests with parallelism levels of 4, 8, and `os.cpu_count()` — record wall-clock time and peak memory usage for each - [ ] Update `_default_processes()` to read from a `TEST_PROCESSES` environment variable, falling back to a safe default derived from benchmarks - [ ] Update `.forgejo/workflows/ci.yml` to set `TEST_PROCESSES` to the optimal value determined by benchmarking - [ ] Update `features/ci_workflow_validation.feature` to assert the `TEST_PROCESSES` env var is set in the relevant CI jobs - [ ] Verify all nox stages pass after changes - [ ] Confirm coverage ≥ 97% ## Definition of Done - [ ] The `_default_processes()` function in `noxfile.py` is updated to allow for a configurable level of parallelism via `TEST_PROCESSES` env var - [ ] The default parallelism for the `unit_tests` and `integration_tests` nox sessions is increased to a value that is proven to be both faster and stable on the CI infrastructure - [ ] The overall execution time for the CI pipeline is measurably reduced - [ ] The CI pipeline remains stable and does not experience an increase in OOM errors or other resource-related failures - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 14:38:17 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: High (already set, correct)
  • Milestone: v3.8.0 (correct)
  • MoSCoW: Could Have — Increasing test parallelism is a performance optimization, not blocking milestone delivery.
  • Parent Epic: #1678

This is the canonical issue for increasing test parallelism. Duplicates #2296, #2283, #2285, #2326, #2301 have been closed and consolidated here.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified ✅ - **Priority**: High (already set, correct) - **Milestone**: v3.8.0 (correct) - **MoSCoW**: Could Have — Increasing test parallelism is a performance optimization, not blocking milestone delivery. - **Parent Epic**: #1678 This is the canonical issue for increasing test parallelism. Duplicates #2296, #2283, #2285, #2326, #2301 have been closed and consolidated here. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
freemo removed this from the v3.8.0 milestone 2026-04-07 01:01:21 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2333
No description provided.