Split Large Test Suites into Parallel CI Shards #2852

Open
opened 2026-04-04 20:55:47 +00:00 by freemo · 0 comments
Owner

Metadata

  • Branch: task/ci-split-large-test-suites
  • Commit Message: perf(ci): split large test suites into parallel shards to reduce wall-clock time
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background

The Behave (unit) and Robot Framework (integration) test suites have grown large enough that running them as a single sequential job is a significant contributor to overall CI wall-clock time. Splitting each suite into parallel shards — where each shard runs a subset of tests concurrently — will reduce the critical-path duration of the CI pipeline and provide faster feedback to contributors.

This work is a direct sub-task of Epic #1678 (CI Execution Time Optimization).

Subtasks

  • Profile the current Behave unit test suite (nox -e unit_tests) to identify the slowest feature files and measure total runtime
  • Profile the current Robot Framework integration test suite (nox -e integration_tests) to identify the slowest robot suites and measure total runtime
  • Design a sharding strategy for Behave: partition .feature files into N balanced shards using a matrix strategy in the CI workflow
  • Design a sharding strategy for Robot Framework: partition .robot suite files into N balanced shards using a matrix strategy in the CI workflow
  • Update the CI workflow YAML to introduce a matrix job for unit_tests with configurable shard count
  • Update the CI workflow YAML to introduce a matrix job for integration_tests with configurable shard count
  • Ensure coverage artifacts from all shards are uploaded and merged correctly before the coverage_report nox session runs
  • Validate that the nox -e coverage_report gate still enforces ≥ 97% coverage after shard merging
  • Update CONTRIBUTING.md to document the sharding approach and how to run a specific shard locally
  • Write Behave feature scenarios covering the shard-selection logic (if any Python helper code is introduced)

Definition of Done

  • Behave unit test suite is split into at least 2 parallel shards in CI with no test omissions
  • Robot Framework integration test suite is split into at least 2 parallel shards in CI with no test omissions
  • Total CI wall-clock time for the test stage is measurably reduced (target: ≥ 25% reduction)
  • Coverage artifacts from all shards are correctly merged; nox -e coverage_report passes with ≥ 97% coverage
  • No mocks or fakes are introduced into integration test shards
  • CONTRIBUTING.md documents the sharding strategy and local shard execution
  • All nox stages pass (nox -e lint, nox -e typecheck, nox -e unit_tests, nox -e integration_tests, nox -e coverage_report)
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-split-large-test-suites` - **Commit Message**: `perf(ci): split large test suites into parallel shards to reduce wall-clock time` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background The Behave (unit) and Robot Framework (integration) test suites have grown large enough that running them as a single sequential job is a significant contributor to overall CI wall-clock time. Splitting each suite into parallel shards — where each shard runs a subset of tests concurrently — will reduce the critical-path duration of the CI pipeline and provide faster feedback to contributors. This work is a direct sub-task of Epic #1678 (CI Execution Time Optimization). ## Subtasks - [ ] Profile the current Behave unit test suite (`nox -e unit_tests`) to identify the slowest feature files and measure total runtime - [ ] Profile the current Robot Framework integration test suite (`nox -e integration_tests`) to identify the slowest robot suites and measure total runtime - [ ] Design a sharding strategy for Behave: partition `.feature` files into N balanced shards using a matrix strategy in the CI workflow - [ ] Design a sharding strategy for Robot Framework: partition `.robot` suite files into N balanced shards using a matrix strategy in the CI workflow - [ ] Update the CI workflow YAML to introduce a `matrix` job for `unit_tests` with configurable shard count - [ ] Update the CI workflow YAML to introduce a `matrix` job for `integration_tests` with configurable shard count - [ ] Ensure coverage artifacts from all shards are uploaded and merged correctly before the `coverage_report` nox session runs - [ ] Validate that the `nox -e coverage_report` gate still enforces ≥ 97% coverage after shard merging - [ ] Update `CONTRIBUTING.md` to document the sharding approach and how to run a specific shard locally - [ ] Write Behave feature scenarios covering the shard-selection logic (if any Python helper code is introduced) ## Definition of Done - [ ] Behave unit test suite is split into at least 2 parallel shards in CI with no test omissions - [ ] Robot Framework integration test suite is split into at least 2 parallel shards in CI with no test omissions - [ ] Total CI wall-clock time for the test stage is measurably reduced (target: ≥ 25% reduction) - [ ] Coverage artifacts from all shards are correctly merged; `nox -e coverage_report` passes with ≥ 97% coverage - [ ] No mocks or fakes are introduced into integration test shards - [ ] `CONTRIBUTING.md` documents the sharding strategy and local shard execution - [ ] All nox stages pass (`nox -e lint`, `nox -e typecheck`, `nox -e unit_tests`, `nox -e integration_tests`, `nox -e coverage_report`) - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-04 20:55:52 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2852
No description provided.