TEST-INFRA: [ci-execution-time] Increase default parallelism for unit and integration tests #2277

New issue

Closed

opened 2026-04-03 12:39:56 +00:00 by freemo · 1 comment

freemo commented

2026-04-03 12:39:56 +00:00

Owner

Metadata

Branch: task/ci-execution-time-increase-default-parallelism
Commit Message: perf(nox): increase default test parallelism and make process count easily configurable
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The _default_processes() function in noxfile.py caps the number of parallel test workers at min(cpus, 2), meaning the effective default is always 2 processes regardless of how many CPU cores the CI runner has available. This is intentionally conservative (the comment reads: "Keep default parallelism conservative to avoid timeout/OOM flakes under heavy Robot/pabot subprocess fan-out in CI and shared runners"), but it significantly underutilizes modern CI runners that routinely provide 4–8+ cores.

While the TEST_PROCESSES environment variable already exists as an override mechanism, it is not documented, not set in the CI workflow, and not surfaced in any contributor-facing configuration. As a result, contributors and CI pipelines alike silently run at 2 processes even on capable hardware.

This issue tracks increasing the hard cap from 2 to a higher, more appropriate default, and making the configuration easy to discover and adjust without diving into noxfile.py internals.

Current Behavior

In noxfile.py, _default_processes() returns min(cpus, 2). Both the unit_tests and integration_tests (pabot/Robot) nox sessions call this function to determine parallelism. On a CI runner with 8 cores, only 2 processes are used, leaving 75% of available CPU idle during test execution.

Expected Behavior

The default process cap should be raised to a more aggressive value (e.g., 4 or auto based on available CPUs with a reasonable upper bound).
The TEST_PROCESSES environment variable should be documented in README.md (or a dedicated CI/testing configuration guide) so contributors know how to tune it.
The CI workflow (.forgejo/workflows/ci.yml) should explicitly set TEST_PROCESSES to the optimal value determined by profiling the actual runner hardware.
The change should be easy to revert or tune without modifying noxfile.py source code.

Acceptance Criteria

The default parallelism cap in _default_processes() is raised above 2 (exact value to be determined by experimentation on CI runners).
TEST_PROCESSES is documented as the canonical override mechanism.
The CI workflow sets TEST_PROCESSES explicitly so the value is visible and auditable in version control.
No existing nox sessions are broken by the change.
Test execution wall-clock time on CI is measurably reduced compared to the 2-process baseline.

Supporting Information

noxfile.py — _default_processes() function (line ~17), _behave_parallel_args(), _pabot_parallel_args()
Parent Epic: #1678 (CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing)
Related: #2209 (Increase default parallelism for tests — broader scope), #2268 (Investigate increasing parallel processes for Behave and Robot tests), #2269 (Investigate test-level parallelization in CI jobs)

Subtasks

Profile CI runner hardware to determine available CPU count and memory headroom
Experiment with process counts (4, 6, 8, auto) on both unit_tests and integration_tests sessions to find a stable, fast default
Update _default_processes() in noxfile.py to use the new default cap
Add TEST_PROCESSES documentation to README.md or the contributing/testing guide
Update .forgejo/workflows/ci.yml to set TEST_PROCESSES explicitly for unit_tests and integration_tests jobs
Update or add a Behave scenario in features/ to assert the CI workflow sets TEST_PROCESSES
Verify coverage >= 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly: perf(nox): increase default test parallelism and make process count easily configurable
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly: task/ci-execution-time-increase-default-parallelism
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
The default process cap in noxfile.py is raised above 2.
TEST_PROCESSES is documented and set explicitly in the CI workflow.
All nox stages pass.
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-execution-time-increase-default-parallelism` - **Commit Message**: `perf(nox): increase default test parallelism and make process count easily configurable` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The `_default_processes()` function in `noxfile.py` caps the number of parallel test workers at `min(cpus, 2)`, meaning the effective default is always **2 processes** regardless of how many CPU cores the CI runner has available. This is intentionally conservative (the comment reads: *"Keep default parallelism conservative to avoid timeout/OOM flakes under heavy Robot/pabot subprocess fan-out in CI and shared runners"*), but it significantly underutilizes modern CI runners that routinely provide 4–8+ cores. While the `TEST_PROCESSES` environment variable already exists as an override mechanism, it is not documented, not set in the CI workflow, and not surfaced in any contributor-facing configuration. As a result, contributors and CI pipelines alike silently run at 2 processes even on capable hardware. This issue tracks increasing the hard cap from `2` to a higher, more appropriate default, and making the configuration easy to discover and adjust without diving into `noxfile.py` internals. ## Current Behavior In `noxfile.py`, `_default_processes()` returns `min(cpus, 2)`. Both the `unit_tests` and `integration_tests` (pabot/Robot) nox sessions call this function to determine parallelism. On a CI runner with 8 cores, only 2 processes are used, leaving 75% of available CPU idle during test execution. ## Expected Behavior - The default process cap should be raised to a more aggressive value (e.g., `4` or `auto` based on available CPUs with a reasonable upper bound). - The `TEST_PROCESSES` environment variable should be documented in `README.md` (or a dedicated CI/testing configuration guide) so contributors know how to tune it. - The CI workflow (`.forgejo/workflows/ci.yml`) should explicitly set `TEST_PROCESSES` to the optimal value determined by profiling the actual runner hardware. - The change should be easy to revert or tune without modifying `noxfile.py` source code. ## Acceptance Criteria - The default parallelism cap in `_default_processes()` is raised above `2` (exact value to be determined by experimentation on CI runners). - `TEST_PROCESSES` is documented as the canonical override mechanism. - The CI workflow sets `TEST_PROCESSES` explicitly so the value is visible and auditable in version control. - No existing nox sessions are broken by the change. - Test execution wall-clock time on CI is measurably reduced compared to the 2-process baseline. ## Supporting Information - `noxfile.py` — `_default_processes()` function (line ~17), `_behave_parallel_args()`, `_pabot_parallel_args()` - Parent Epic: #1678 (CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing) - Related: #2209 (Increase default parallelism for tests — broader scope), #2268 (Investigate increasing parallel processes for Behave and Robot tests), #2269 (Investigate test-level parallelization in CI jobs) ## Subtasks - [ ] Profile CI runner hardware to determine available CPU count and memory headroom - [ ] Experiment with process counts (`4`, `6`, `8`, `auto`) on both `unit_tests` and `integration_tests` sessions to find a stable, fast default - [ ] Update `_default_processes()` in `noxfile.py` to use the new default cap - [ ] Add `TEST_PROCESSES` documentation to `README.md` or the contributing/testing guide - [ ] Update `.forgejo/workflows/ci.yml` to set `TEST_PROCESSES` explicitly for `unit_tests` and `integration_tests` jobs - [ ] Update or add a Behave scenario in `features/` to assert the CI workflow sets `TEST_PROCESSES` - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly: `perf(nox): increase default test parallelism and make process count easily configurable` - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly: `task/ci-execution-time-increase-default-parallelism` - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - The default process cap in `noxfile.py` is raised above `2`. - `TEST_PROCESSES` is documented and set explicitly in the CI workflow. - All nox stages pass. - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-03 12:40:21 +00:00

freemo added a new dependency

2026-04-03 12:42:49 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo referenced this issue

2026-04-03 12:48:31 +00:00

TEST-INFRA: [ci-execution-time] Optimize default nox run #2282

freemo referenced this issue

2026-04-03 12:58:16 +00:00

TEST-INFRA: [ci-execution-time] Increase default parallelization for unit and integration tests #2285

freemo referenced this issue

2026-04-03 12:58:39 +00:00

[Automated] Product Build Session State — v3.7.0 Production Sprint #2006

freemo referenced this issue

2026-04-03 17:23:59 +00:00

Proposal: improve ca-test-infra-improver — strengthen duplicate avoidance to prevent near-identical issues #2370

freemo commented

2026-04-03 17:30:42 +00:00

Author

Owner

Closing as duplicate of #2333.

This issue ("Increase default parallelism for unit and integration tests") covers the same scope as #2333 ("Investigate and increase test parallelism"). Consolidating to avoid duplicate work.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Closing as duplicate of #2333. This issue ("Increase default parallelism for unit and integration tests") covers the same scope as #2333 ("Investigate and increase test parallelism"). Consolidating to avoid duplicate work. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo closed this issue

2026-04-03 17:30:59 +00:00