TEST-INFRA: [ci-pipeline-design] Implement conditional test execution #7784

Open
opened 2026-04-12 03:34:44 +00:00 by HAL9000 · 3 comments
Owner

Metadata

  • Branch: task/ci-pipeline-design-conditional-test-execution
  • Commit Message: chore(ci): implement conditional test execution based on changed files
  • Milestone: N/A — Backlog (see note below)
  • Parent Epic: #1678

Backlog note: This issue was discovered during autonomous operation
on milestone v3.2.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

Background and Context

The CI pipeline currently runs all tests for every change, which can be inefficient. Implementing conditional test execution would allow the pipeline to only run tests relevant to the changes, significantly reducing CI run times. This is distinct from job-level path filtering (see #7466) — this issue focuses on test-level conditional execution within jobs (e.g., selecting which Behave scenarios or Robot suites to run based on changed source modules).

Current Behavior

All tests (unit, integration, benchmarks) are executed on every push and pull request, regardless of which source files have changed. A change to a single module triggers the full test suite across all modules.

Expected Behavior

The CI pipeline should execute only the tests relevant to the files changed in a pull request:

  • Changes to a specific module should trigger only the tests for that module (and its dependents).
  • CI run time is measurably reduced for changes that only affect a subset of the codebase.
  • The correct set of tests is always run — no regressions are introduced by skipping unrelated tests.

Acceptance Criteria

  • Investigate and implement a mechanism for conditional test execution based on the files changed in a pull request.
  • Ensure that the correct set of tests is run for each change.
  • Verify that the CI run time is reduced for changes that only affect a subset of the codebase.

Supporting Information

  • Related issues: #7466 (CI job-level conditional execution via path filters), #7123 (investigation spike for conditional test execution), #7430 (parallelize CI jobs), Epic #1678 (CI Execution Time Optimization).
  • Forgejo Actions supports paths and paths-ignore filters on workflow triggers; test-level filtering may require a custom script or pytest/behave plugin.
  • Consider tools such as pytest-testmon, behave tag filtering, or a custom changed-files script to determine which test suites to invoke.

Subtasks

  • Investigate available mechanisms for test-level conditional execution (e.g., changed-files detection, pytest-testmon, Behave tag filtering).
  • Design and document the mapping from changed source files to relevant test suites.
  • Implement the conditional test execution mechanism in the CI workflow (.forgejo/workflows/ci.yml).
  • Add inline comments to the workflow file explaining the conditional logic.
  • Verify that the full test suite still runs on changes to shared/core modules.
  • Measure and document CI run time improvement.

Definition of Done

  • Conditional test execution is implemented and active in the CI pipeline.
  • The correct tests are selected for representative change sets (unit-tested via dry-run or CI log inspection).
  • CI run time is demonstrably reduced for single-module changes.
  • Workflow file is documented with inline comments.
  • All nox stages pass.
  • Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: new-issue-creator

## Metadata - **Branch**: `task/ci-pipeline-design-conditional-test-execution` - **Commit Message**: `chore(ci): implement conditional test execution based on changed files` - **Milestone**: N/A — Backlog (see note below) - **Parent Epic**: #1678 > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.2.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. ## Background and Context The CI pipeline currently runs all tests for every change, which can be inefficient. Implementing conditional test execution would allow the pipeline to only run tests relevant to the changes, significantly reducing CI run times. This is distinct from job-level path filtering (see #7466) — this issue focuses on test-level conditional execution within jobs (e.g., selecting which Behave scenarios or Robot suites to run based on changed source modules). ## Current Behavior All tests (unit, integration, benchmarks) are executed on every push and pull request, regardless of which source files have changed. A change to a single module triggers the full test suite across all modules. ## Expected Behavior The CI pipeline should execute only the tests relevant to the files changed in a pull request: - Changes to a specific module should trigger only the tests for that module (and its dependents). - CI run time is measurably reduced for changes that only affect a subset of the codebase. - The correct set of tests is always run — no regressions are introduced by skipping unrelated tests. ## Acceptance Criteria - [ ] Investigate and implement a mechanism for conditional test execution based on the files changed in a pull request. - [ ] Ensure that the correct set of tests is run for each change. - [ ] Verify that the CI run time is reduced for changes that only affect a subset of the codebase. ## Supporting Information - Related issues: #7466 (CI job-level conditional execution via path filters), #7123 (investigation spike for conditional test execution), #7430 (parallelize CI jobs), Epic #1678 (CI Execution Time Optimization). - Forgejo Actions supports `paths` and `paths-ignore` filters on workflow triggers; test-level filtering may require a custom script or pytest/behave plugin. - Consider tools such as `pytest-testmon`, `behave` tag filtering, or a custom changed-files script to determine which test suites to invoke. ## Subtasks - [ ] Investigate available mechanisms for test-level conditional execution (e.g., changed-files detection, `pytest-testmon`, Behave tag filtering). - [ ] Design and document the mapping from changed source files to relevant test suites. - [ ] Implement the conditional test execution mechanism in the CI workflow (`.forgejo/workflows/ci.yml`). - [ ] Add inline comments to the workflow file explaining the conditional logic. - [ ] Verify that the full test suite still runs on changes to shared/core modules. - [ ] Measure and document CI run time improvement. ## Definition of Done - [ ] Conditional test execution is implemented and active in the CI pipeline. - [ ] The correct tests are selected for representative change sets (unit-tested via dry-run or CI log inspection). - [ ] CI run time is demonstrably reduced for single-module changes. - [ ] Workflow file is documented with inline comments. - [ ] All nox stages pass. - [ ] Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: new-issue-creator
HAL9000 added this to the v3.2.0 milestone 2026-04-12 03:47:01 +00:00
Author
Owner

Verified — CI improvement: conditional test execution. MoSCoW: Should-have. Priority: Medium.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — CI improvement: conditional test execution. MoSCoW: Should-have. Priority: Medium. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Author
Owner

Verified — CI improvement: conditional test execution. MoSCoW: Should-have. Priority: Medium.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — CI improvement: conditional test execution. MoSCoW: Should-have. Priority: Medium. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Author
Owner

Verified — CI improvement: conditional test execution. MoSCoW: Should-have. Priority: Medium.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — CI improvement: conditional test execution. MoSCoW: Should-have. Priority: Medium. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#7784
No description provided.