TEST-INFRA: [ci-pipeline-design] Implement Conditional Test Execution to Run Only Affected Tests #2232

Open
opened 2026-04-03 09:42:37 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: task/ci-pipeline-design-conditional-test-execution
  • Commit Message: feat(ci): implement conditional test execution to run only affected tests
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The CI pipeline currently runs the full test suite on every commit, regardless of which files were changed. This results in unnecessarily long CI run times and wastes compute resources — especially as the test suite grows. Implementing conditional test execution would detect which modules or packages are affected by a given change and run only the relevant tests, significantly reducing feedback loop times for contributors.

This is particularly impactful given the current project state: 84 open PRs, 500 open issues, and a growing test suite across M3–M9. Faster CI cycles directly improve developer throughput and reduce the time-to-merge for the open PR backlog.

Current Behavior

Every commit triggers a full test suite run via nox, regardless of the scope of the change. A one-line documentation fix triggers the same CI workload as a core domain model change.

Expected Behavior

The CI pipeline detects which source files were changed in a commit or PR and runs only the test sessions that cover the affected modules. Unaffected test suites are skipped, with a clear log message indicating why they were skipped. Full suite runs are still triggered on merges to master and on release branches.

Acceptance Criteria

  • Changed files are detected per-commit/PR using git diff against the base branch
  • A mapping exists from source modules/packages to their corresponding nox test sessions
  • Only the nox sessions covering affected modules are executed for feature branch pushes
  • Full suite (nox all default sessions) always runs on master and release branches
  • CI logs clearly indicate which sessions were skipped and why
  • Coverage reporting still aggregates correctly even when partial sessions run
  • No regression in test coverage enforcement (≥97% threshold maintained)

Supporting Information

  • Parent Epic: #1678 (CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing)
  • Related CI issues: #1792–#1797 (CI pipeline improvements from Day 54 agent wave)
  • Existing nox-based PR validation: #213 (feat(ci): add nox-based PR validation workflow)
  • Coverage enforcement: #214 (feat(qa): enforce coverage >=97% in CI)

Subtasks

  • Audit existing nox sessions and map each to its source module(s)/package(s)
  • Implement a CI script (or nox session) that computes the set of affected modules from git diff
  • Integrate the affected-module detection into the CI workflow (.forgejo/workflows/ or equivalent)
  • Add logic to skip unaffected nox sessions with an informative log message
  • Ensure full suite always runs on master / release branch pushes
  • Validate coverage aggregation still works correctly with partial session runs
  • Tests (Behave): Add BDD scenarios for the affected-module detection logic
  • Tests (Robot): Add integration test verifying correct session selection for sample changesets
  • Verify coverage ≥97% via nox -s coverage_report
  • Run nox (all default sessions), fix any errors
  • Update CI documentation / README to describe the conditional execution behaviour

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly (feat(ci): implement conditional test execution to run only affected tests), followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly (task/ci-pipeline-design-conditional-test-execution).
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage ≥ 97%.

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-pipeline-design-conditional-test-execution` - **Commit Message**: `feat(ci): implement conditional test execution to run only affected tests` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The CI pipeline currently runs the full test suite on every commit, regardless of which files were changed. This results in unnecessarily long CI run times and wastes compute resources — especially as the test suite grows. Implementing conditional test execution would detect which modules or packages are affected by a given change and run only the relevant tests, significantly reducing feedback loop times for contributors. This is particularly impactful given the current project state: 84 open PRs, 500 open issues, and a growing test suite across M3–M9. Faster CI cycles directly improve developer throughput and reduce the time-to-merge for the open PR backlog. ## Current Behavior Every commit triggers a full test suite run via `nox`, regardless of the scope of the change. A one-line documentation fix triggers the same CI workload as a core domain model change. ## Expected Behavior The CI pipeline detects which source files were changed in a commit or PR and runs only the test sessions that cover the affected modules. Unaffected test suites are skipped, with a clear log message indicating why they were skipped. Full suite runs are still triggered on merges to `master` and on release branches. ## Acceptance Criteria - [ ] Changed files are detected per-commit/PR using git diff against the base branch - [ ] A mapping exists from source modules/packages to their corresponding nox test sessions - [ ] Only the nox sessions covering affected modules are executed for feature branch pushes - [ ] Full suite (`nox` all default sessions) always runs on `master` and release branches - [ ] CI logs clearly indicate which sessions were skipped and why - [ ] Coverage reporting still aggregates correctly even when partial sessions run - [ ] No regression in test coverage enforcement (≥97% threshold maintained) ## Supporting Information - Parent Epic: #1678 (CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing) - Related CI issues: #1792–#1797 (CI pipeline improvements from Day 54 agent wave) - Existing nox-based PR validation: #213 (`feat(ci): add nox-based PR validation workflow`) - Coverage enforcement: #214 (`feat(qa): enforce coverage >=97% in CI`) ## Subtasks - [ ] Audit existing nox sessions and map each to its source module(s)/package(s) - [ ] Implement a CI script (or nox session) that computes the set of affected modules from `git diff` - [ ] Integrate the affected-module detection into the CI workflow (`.forgejo/workflows/` or equivalent) - [ ] Add logic to skip unaffected nox sessions with an informative log message - [ ] Ensure full suite always runs on `master` / release branch pushes - [ ] Validate coverage aggregation still works correctly with partial session runs - [ ] Tests (Behave): Add BDD scenarios for the affected-module detection logic - [ ] Tests (Robot): Add integration test verifying correct session selection for sample changesets - [ ] Verify coverage ≥97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors - [ ] Update CI documentation / README to describe the conditional execution behaviour ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly (`feat(ci): implement conditional test execution to run only affected tests`), followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly (`task/ci-pipeline-design-conditional-test-execution`). - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage ≥ 97%. --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 09:42:57 +00:00
Author
Owner

⚠️ Potential duplicate: Issues #1984 and #2043 both describe "Implement conditional test execution" for the CI pipeline. Please check if this issue is a duplicate before proceeding.


Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: ca-backlog-groomer

⚠️ **Potential duplicate**: Issues #1984 and #2043 both describe "Implement conditional test execution" for the CI pipeline. Please check if this issue is a duplicate before proceeding. --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: ca-backlog-groomer
freemo removed this from the v3.8.0 milestone 2026-04-07 01:13:40 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2232
No description provided.