TEST-INFRA: [ci-pipeline-design] Investigate conditional test execution #7123

Open
opened 2026-04-10 07:57:13 +00:00 by HAL9000 · 1 comment
Owner

Metadata

  • Branch: chore/ci-conditional-test-execution
  • Commit Message: chore(ci): investigate conditional test execution strategy
  • Milestone: (none — backlog, see note below)
  • Parent Epic: #5407

Background and Context

The current CI pipeline runs the full test suite on every commit, regardless of which files were changed. For a project of this scale, this is inefficient — a one-line documentation fix triggers the same full test run as a core architectural change. Conditional (or "affected") test execution strategies selectively run only the tests that are relevant to the changed code, significantly reducing CI wall-clock time and resource consumption.

This task is to investigate the feasibility of implementing such a strategy in the current CI pipeline before committing to an implementation.

Expected Behavior

A feasibility study is produced that evaluates tools and techniques for conditional test execution in a Python/BDD/Robot Framework project, assesses integration complexity with the current CI pipeline (Forgejo Actions), and produces a documented recommendation (or a documented rationale for not proceeding).

Acceptance Criteria

  • At least two candidate tools or techniques for conditional test execution are evaluated (e.g., pytest-testmon, affected-tests, path-based job filtering in Forgejo Actions).
  • The evaluation covers compatibility with the project's BDD (Behave), Robot Framework, and ASV benchmark test suites.
  • Integration complexity with the current Forgejo Actions CI pipeline is assessed.
  • A proof-of-concept is attempted on a test branch to validate feasibility.
  • Findings and a recommendation (proceed / do not proceed, with rationale) are documented in a follow-up issue or ADR.

Subtasks

  • Research tools and techniques for conditional test execution in a Python/BDD/Robot Framework project (e.g., pytest-testmon, path-based filtering, change-set detection).
  • Evaluate compatibility of candidate tools with Behave, Robot Framework, and ASV benchmark suites.
  • Assess integration complexity with the current Forgejo Actions CI pipeline configuration.
  • Create a proof-of-concept implementation on a test branch.
  • Document findings and recommendations in a new follow-up issue (or ADR if an architectural decision is warranted).
  • Run nox (all default sessions) on the PoC branch; fix any errors introduced.

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A feasibility study for conditional test execution is completed and documented.
  • A follow-up issue (or ADR) is created with the findings and a clear recommendation.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%.

Backlog note: This issue was discovered during autonomous operation
on milestone v3.2.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.


Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: new-issue-creator

## Metadata - **Branch**: `chore/ci-conditional-test-execution` - **Commit Message**: `chore(ci): investigate conditional test execution strategy` - **Milestone**: *(none — backlog, see note below)* - **Parent Epic**: #5407 ## Background and Context The current CI pipeline runs the full test suite on every commit, regardless of which files were changed. For a project of this scale, this is inefficient — a one-line documentation fix triggers the same full test run as a core architectural change. Conditional (or "affected") test execution strategies selectively run only the tests that are relevant to the changed code, significantly reducing CI wall-clock time and resource consumption. This task is to investigate the feasibility of implementing such a strategy in the current CI pipeline before committing to an implementation. ## Expected Behavior A feasibility study is produced that evaluates tools and techniques for conditional test execution in a Python/BDD/Robot Framework project, assesses integration complexity with the current CI pipeline (Forgejo Actions), and produces a documented recommendation (or a documented rationale for not proceeding). ## Acceptance Criteria - [ ] At least two candidate tools or techniques for conditional test execution are evaluated (e.g., `pytest-testmon`, `affected-tests`, path-based job filtering in Forgejo Actions). - [ ] The evaluation covers compatibility with the project's BDD (Behave), Robot Framework, and ASV benchmark test suites. - [ ] Integration complexity with the current Forgejo Actions CI pipeline is assessed. - [ ] A proof-of-concept is attempted on a test branch to validate feasibility. - [ ] Findings and a recommendation (proceed / do not proceed, with rationale) are documented in a follow-up issue or ADR. ## Subtasks - [ ] Research tools and techniques for conditional test execution in a Python/BDD/Robot Framework project (e.g., `pytest-testmon`, path-based filtering, change-set detection). - [ ] Evaluate compatibility of candidate tools with Behave, Robot Framework, and ASV benchmark suites. - [ ] Assess integration complexity with the current Forgejo Actions CI pipeline configuration. - [ ] Create a proof-of-concept implementation on a test branch. - [ ] Document findings and recommendations in a new follow-up issue (or ADR if an architectural decision is warranted). - [ ] Run `nox` (all default sessions) on the PoC branch; fix any errors introduced. ## Definition of Done This issue is complete when: - [ ] All subtasks above are completed and checked off. - [ ] A feasibility study for conditional test execution is completed and documented. - [ ] A follow-up issue (or ADR) is created with the findings and a clear recommendation. - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.2.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: new-issue-creator
Author
Owner

Verified — CI improvement: investigate conditional test execution. MoSCoW: Could-have. Priority: Low.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — CI improvement: investigate conditional test execution. MoSCoW: Could-have. Priority: Low. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#7123
No description provided.