feat(tests): increase parallelism across all test suites (pytest, Behave, Robot Framework) #1987

Open
opened 2026-04-03 00:31:24 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: task/increase-test-suite-parallelism
  • Commit Message: feat(tests): increase parallelism across all test suites
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background

The CI pipeline runs three distinct test suites — pytest (unit), Behave (BDD), and Robot Framework (integration/e2e) — each with limited or no intra-suite parallelism. As the test corpus grows (currently 97%+ coverage target across a large codebase), sequential execution is a significant bottleneck on CI wall-clock time.

Specific gaps identified:

  1. pytest (unit_tests nox session) — runs with no -n worker flag; pytest-xdist is not configured. All unit tests execute sequentially on a single process.
  2. Behave (integration_tests nox session) — the run_behave_parallel.py script exists (extracted in #1538) but the worker count is not tuned; it defaults to a low value and does not adapt to available CPU cores on the CI runner.
  3. Robot Framework (e2e_tests nox session) — runs with --processes 1 (or no --processes flag), executing all suites serially. pabot (Parallel Robot Framework executor) is not used.
  4. noxfile.py session definitions — none of the test sessions pass a parallelism argument derived from os.cpu_count() or a configurable TEST_WORKERS environment variable.

Subtasks

  • Add pytest-xdist to pyproject.toml dev dependencies and configure pytest.ini / pyproject.toml [tool.pytest.ini_options] with addopts = "-n auto"
  • Update the unit_tests nox session in noxfile.py to pass -n auto (or TEST_WORKERS env override) to pytest
  • Tune run_behave_parallel.py worker count to use os.cpu_count() as the default (with BEHAVE_WORKERS env override)
  • Update the integration_tests nox session to pass the worker count through to run_behave_parallel.py
  • Add robotframework-pabot to dev dependencies and update the e2e_tests nox session to invoke pabot --processes <N> instead of robot
  • Add a TEST_WORKERS / BEHAVE_WORKERS / PABOT_PROCESSES env var section to docs/development/ci-cd.md documenting how to tune parallelism
  • Update features/ci_workflow_validation.feature to assert that test jobs use parallel execution flags
  • Verify all nox stages pass after changes
  • Confirm coverage ≥ 97%

Definition of Done

  • pytest unit tests run with pytest-xdist -n auto parallelism in CI and locally
  • Behave BDD tests run with worker count defaulting to os.cpu_count() (overridable via BEHAVE_WORKERS)
  • Robot Framework e2e tests run via pabot --processes <N> (overridable via PABOT_PROCESSES)
  • All three test sessions accept a TEST_WORKERS / per-suite env var to override parallelism for debugging
  • docs/development/ci-cd.md documents the parallelism knobs
  • features/ci_workflow_validation.feature covers parallel execution flags
  • All nox stages pass
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/increase-test-suite-parallelism` - **Commit Message**: `feat(tests): increase parallelism across all test suites` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background The CI pipeline runs three distinct test suites — pytest (unit), Behave (BDD), and Robot Framework (integration/e2e) — each with limited or no intra-suite parallelism. As the test corpus grows (currently 97%+ coverage target across a large codebase), sequential execution is a significant bottleneck on CI wall-clock time. Specific gaps identified: 1. **pytest (`unit_tests` nox session)** — runs with no `-n` worker flag; `pytest-xdist` is not configured. All unit tests execute sequentially on a single process. 2. **Behave (`integration_tests` nox session)** — the `run_behave_parallel.py` script exists (extracted in #1538) but the worker count is not tuned; it defaults to a low value and does not adapt to available CPU cores on the CI runner. 3. **Robot Framework (`e2e_tests` nox session)** — runs with `--processes 1` (or no `--processes` flag), executing all suites serially. `pabot` (Parallel Robot Framework executor) is not used. 4. **noxfile.py session definitions** — none of the test sessions pass a parallelism argument derived from `os.cpu_count()` or a configurable `TEST_WORKERS` environment variable. ## Subtasks - [ ] Add `pytest-xdist` to `pyproject.toml` dev dependencies and configure `pytest.ini` / `pyproject.toml` `[tool.pytest.ini_options]` with `addopts = "-n auto"` - [ ] Update the `unit_tests` nox session in `noxfile.py` to pass `-n auto` (or `TEST_WORKERS` env override) to pytest - [ ] Tune `run_behave_parallel.py` worker count to use `os.cpu_count()` as the default (with `BEHAVE_WORKERS` env override) - [ ] Update the `integration_tests` nox session to pass the worker count through to `run_behave_parallel.py` - [ ] Add `robotframework-pabot` to dev dependencies and update the `e2e_tests` nox session to invoke `pabot --processes <N>` instead of `robot` - [ ] Add a `TEST_WORKERS` / `BEHAVE_WORKERS` / `PABOT_PROCESSES` env var section to `docs/development/ci-cd.md` documenting how to tune parallelism - [ ] Update `features/ci_workflow_validation.feature` to assert that test jobs use parallel execution flags - [ ] Verify all nox stages pass after changes - [ ] Confirm coverage ≥ 97% ## Definition of Done - [ ] `pytest` unit tests run with `pytest-xdist` `-n auto` parallelism in CI and locally - [ ] Behave BDD tests run with worker count defaulting to `os.cpu_count()` (overridable via `BEHAVE_WORKERS`) - [ ] Robot Framework e2e tests run via `pabot --processes <N>` (overridable via `PABOT_PROCESSES`) - [ ] All three test sessions accept a `TEST_WORKERS` / per-suite env var to override parallelism for debugging - [ ] `docs/development/ci-cd.md` documents the parallelism knobs - [ ] `features/ci_workflow_validation.feature` covers parallel execution flags - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 00:31:39 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: Medium (confirmed)
  • Milestone: v3.8.0 (already assigned — CI infrastructure improvements)
  • MoSCoW: Could Have — Increasing test parallelism is a CI optimization that improves developer experience but does not block any milestone deliverables. The tests run correctly today; this just makes them faster.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium (confirmed) - **Milestone**: v3.8.0 (already assigned — CI infrastructure improvements) - **MoSCoW**: Could Have — Increasing test parallelism is a CI optimization that improves developer experience but does not block any milestone deliverables. The tests run correctly today; this just makes them faster. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1987
No description provided.