TEST-INFRA: [ci-pipeline-design] Conditional execution of E2E tests #1869

Open
opened 2026-04-03 00:02:37 +00:00 by freemo · 2 comments
Owner

Metadata

  • Branch: chore/ci-conditional-e2e-tests
  • Commit Message: chore(ci): add conditional execution for e2e_tests job based on path and label filters
  • Milestone: v3.2.0
  • Parent Epic: #1678

Background and Context

The e2e_tests job in the ci.yml workflow currently runs on every pull request, regardless of what files were changed. This is expensive and time-consuming when changes are unrelated to the E2E test suite (e.g., documentation updates, source-only changes). Conditional execution will reduce CI costs and improve developer feedback loop times.

Expected Behavior

The e2e_tests job should only execute when:

  • Files in the robot/e2e directory are changed, or
  • A specific label (e.g., run-e2e-tests) is added to the pull request.

Acceptance Criteria

  • The e2e_tests job does not run on every pull request unconditionally.
  • The e2e_tests job does run when files under robot/e2e/ are changed.
  • The e2e_tests job does run when the run-e2e-tests label is applied to the pull request.
  • All other CI pipeline jobs continue to pass normally.
  • The conditional logic is documented in a comment within ci.yml.

Subtasks

  • Audit the current e2e_tests job in .github/ci.yml (or equivalent) to understand its triggers and dependencies
  • Implement a path-filter step (e.g., using dorny/paths-filter or native GitHub/Forgejo on.paths) to detect changes under robot/e2e/
  • Add a label-based condition so the job also runs when the run-e2e-tests label is present on the PR
  • Combine both conditions with an OR so either trigger activates the job
  • Add the run-e2e-tests label to the repository label set if it does not already exist
  • Test the conditional logic by opening a PR with and without robot/e2e changes
  • Verify coverage >= 97% via nox -s coverage_report
  • Run nox (all default sessions), fix any errors

Definition of Done

  • The e2e_tests job is not triggered on PRs that do not touch robot/e2e/ and do not carry the run-e2e-tests label
  • The e2e_tests job is triggered when robot/e2e/ files are changed
  • The e2e_tests job is triggered when the run-e2e-tests label is applied
  • The CI pipeline passes end-to-end with the new conditional logic in place
  • All nox stages pass
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `chore/ci-conditional-e2e-tests` - **Commit Message**: `chore(ci): add conditional execution for e2e_tests job based on path and label filters` - **Milestone**: v3.2.0 - **Parent Epic**: #1678 ## Background and Context The `e2e_tests` job in the `ci.yml` workflow currently runs on every pull request, regardless of what files were changed. This is expensive and time-consuming when changes are unrelated to the E2E test suite (e.g., documentation updates, source-only changes). Conditional execution will reduce CI costs and improve developer feedback loop times. ## Expected Behavior The `e2e_tests` job should only execute when: - Files in the `robot/e2e` directory are changed, **or** - A specific label (e.g., `run-e2e-tests`) is added to the pull request. ## Acceptance Criteria - The `e2e_tests` job does **not** run on every pull request unconditionally. - The `e2e_tests` job **does** run when files under `robot/e2e/` are changed. - The `e2e_tests` job **does** run when the `run-e2e-tests` label is applied to the pull request. - All other CI pipeline jobs continue to pass normally. - The conditional logic is documented in a comment within `ci.yml`. ## Subtasks - [ ] Audit the current `e2e_tests` job in `.github/ci.yml` (or equivalent) to understand its triggers and dependencies - [ ] Implement a path-filter step (e.g., using `dorny/paths-filter` or native GitHub/Forgejo `on.paths`) to detect changes under `robot/e2e/` - [ ] Add a label-based condition so the job also runs when the `run-e2e-tests` label is present on the PR - [ ] Combine both conditions with an `OR` so either trigger activates the job - [ ] Add the `run-e2e-tests` label to the repository label set if it does not already exist - [ ] Test the conditional logic by opening a PR with and without `robot/e2e` changes - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done - [ ] The `e2e_tests` job is not triggered on PRs that do not touch `robot/e2e/` and do not carry the `run-e2e-tests` label - [ ] The `e2e_tests` job is triggered when `robot/e2e/` files are changed - [ ] The `e2e_tests` job is triggered when the `run-e2e-tests` label is applied - [ ] The CI pipeline passes end-to-end with the new conditional logic in place - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator
freemo added this to the v3.2.0 milestone 2026-04-03 00:02:50 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1869
No description provided.