TEST-INFRA: [ci-pipeline-design] Parallelize CI jobs #1724

Open
opened 2026-04-02 23:36:08 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: chore/ci-parallelize-jobs
  • Commit Message: chore(ci): parallelize independent CI jobs to reduce pipeline wall-clock time
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The current CI pipeline in .forgejo/workflows/ci.yml runs most jobs sequentially. This means each job waits for the previous one to complete before starting, even when there is no logical dependency between them. This results in unnecessarily long pipeline wall-clock times and inefficient use of CI runner capacity.

Current Behavior

Jobs such as lint, typecheck, security, and quality run one after another in sequence. Similarly, unit_tests, integration_tests, and e2e_tests are serialized. None of these groups have inter-dependencies that require sequential execution.

Expected Behavior

Independent jobs should run concurrently:

  • lint, typecheck, security, and quality should execute in parallel.
  • unit_tests, integration_tests, and e2e_tests should execute in parallel.
  • The status-check job must correctly depend on all parallel jobs so it only passes when every parallel job has succeeded.

Acceptance Criteria

  • lint, typecheck, security, and quality jobs run in parallel (no sequential needs chain between them).
  • unit_tests, integration_tests, and e2e_tests jobs run in parallel.
  • status-check job lists all parallel jobs in its needs array.
  • CI pipeline wall-clock time is measurably reduced.
  • No regressions introduced — all jobs still execute and report results correctly.

Subtasks

  • Audit .forgejo/workflows/ci.yml to map current needs dependencies between jobs.
  • Update the needs section of lint, typecheck, security, and quality jobs to remove sequential chaining and allow parallel execution.
  • Update the needs section of unit_tests, integration_tests, and e2e_tests jobs to allow parallel execution.
  • Update the status-check job's needs array to include all parallel jobs.
  • Update features/ci_workflow_validation.feature to assert the new parallel job structure.
  • Run nox (all default sessions) and fix any errors.
  • Verify coverage >= 97% via nox -s coverage_report.

Definition of Done

  • All subtasks above are completed and checked off.
  • lint, typecheck, security, and quality jobs run in parallel in .forgejo/workflows/ci.yml.
  • unit_tests, integration_tests, and e2e_tests jobs run in parallel in .forgejo/workflows/ci.yml.
  • status-check correctly depends on all parallel jobs.
  • features/ci_workflow_validation.feature is updated to cover the new parallel structure.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `chore/ci-parallelize-jobs` - **Commit Message**: `chore(ci): parallelize independent CI jobs to reduce pipeline wall-clock time` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The current CI pipeline in `.forgejo/workflows/ci.yml` runs most jobs sequentially. This means each job waits for the previous one to complete before starting, even when there is no logical dependency between them. This results in unnecessarily long pipeline wall-clock times and inefficient use of CI runner capacity. ## Current Behavior Jobs such as `lint`, `typecheck`, `security`, and `quality` run one after another in sequence. Similarly, `unit_tests`, `integration_tests`, and `e2e_tests` are serialized. None of these groups have inter-dependencies that require sequential execution. ## Expected Behavior Independent jobs should run concurrently: - `lint`, `typecheck`, `security`, and `quality` should execute in parallel. - `unit_tests`, `integration_tests`, and `e2e_tests` should execute in parallel. - The `status-check` job must correctly depend on **all** parallel jobs so it only passes when every parallel job has succeeded. ## Acceptance Criteria - [ ] `lint`, `typecheck`, `security`, and `quality` jobs run in parallel (no sequential `needs` chain between them). - [ ] `unit_tests`, `integration_tests`, and `e2e_tests` jobs run in parallel. - [ ] `status-check` job lists all parallel jobs in its `needs` array. - [ ] CI pipeline wall-clock time is measurably reduced. - [ ] No regressions introduced — all jobs still execute and report results correctly. ## Subtasks - [ ] Audit `.forgejo/workflows/ci.yml` to map current `needs` dependencies between jobs. - [ ] Update the `needs` section of `lint`, `typecheck`, `security`, and `quality` jobs to remove sequential chaining and allow parallel execution. - [ ] Update the `needs` section of `unit_tests`, `integration_tests`, and `e2e_tests` jobs to allow parallel execution. - [ ] Update the `status-check` job's `needs` array to include all parallel jobs. - [ ] Update `features/ci_workflow_validation.feature` to assert the new parallel job structure. - [ ] Run `nox` (all default sessions) and fix any errors. - [ ] Verify coverage >= 97% via `nox -s coverage_report`. ## Definition of Done - [ ] All subtasks above are completed and checked off. - [ ] `lint`, `typecheck`, `security`, and `quality` jobs run in parallel in `.forgejo/workflows/ci.yml`. - [ ] `unit_tests`, `integration_tests`, and `e2e_tests` jobs run in parallel in `.forgejo/workflows/ci.yml`. - [ ] `status-check` correctly depends on all parallel jobs. - [ ] `features/ci_workflow_validation.feature` is updated to cover the new parallel structure. - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - [ ] All nox stages pass. - [ ] Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-02 23:36:44 +00:00
Author
Owner

Closing as duplicate. CI parallelization and execution time improvements are tracked in #1604 (setup consolidation), #1536 (parallelize static analysis), and #1632 (split integration tests).


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Closing as duplicate. CI parallelization and execution time improvements are tracked in #1604 (setup consolidation), #1536 (parallelize static analysis), and #1632 (split integration tests). --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
freemo 2026-04-02 23:41:33 +00:00
freemo 2026-04-02 23:48:29 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1724
No description provided.