TEST-INFRA: [ci-pipeline-design] Add conditional nox test execution via Behave tags and Robot Framework filters #1745

New issue

Open

opened 2026-04-02 23:40:34 +00:00 by freemo · 1 comment

freemo commented

2026-04-02 23:40:34 +00:00

Owner

Metadata

Branch: task/ci-conditional-nox-test-execution-tags
Commit Message: feat(ci): add conditional nox test execution via behave tags and robot filters
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The nox test sessions (unit_tests, integration_tests, slow_integration_tests) currently execute the entire test suite unconditionally. There is no mechanism for developers or CI jobs to run a targeted subset of tests — for example, only smoke tests, only fast tests, or only tests relevant to a specific module — without modifying the noxfile directly.

This gap has two consequences:

Local developer feedback is slow. A developer working on a small change must wait for the full suite to complete before getting feedback, even when only a handful of scenarios are relevant.
CI cannot leverage tag-based parallelism. Related issues (#1715, #1702) address CI-level job skipping, but even within a single job, all tests run regardless of relevance. Tag-based filtering within nox sessions would complement those optimizations.

The project uses Behave (BDD/Gherkin) for unit-level tests and Robot Framework for integration tests. Both frameworks support tag-based filtering natively (--tags in Behave, --include/--exclude in Robot Framework), but the nox sessions do not expose these options to callers.

Expected Behavior

Developers can pass Behave tag expressions to the unit_tests nox session:

nox -s unit_tests -- --tags=@smoke
nox -s unit_tests -- --tags=~@slow

Developers can pass Robot Framework include/exclude filters to the integration_tests and slow_integration_tests sessions:
```
nox -s integration_tests -- --include smoke
nox -s slow_integration_tests -- --exclude wip
```
A dedicated nox -s smoke_tests session exists that runs only @smoke-tagged Behave scenarios and smoke-tagged Robot tests, completing in under 60 seconds on a standard developer machine.
All existing nox sessions continue to work without arguments (full suite, no regression).
The CI workflow can invoke nox -s smoke_tests as a fast pre-flight check on PR commits (complementing #1715).

Acceptance Criteria

nox -s unit_tests -- --tags=<expr> passes the tag expression through to Behave and filters scenarios accordingly.
nox -s integration_tests -- --include <tag> and -- --exclude <tag> pass through to Robot Framework.
A smoke_tests nox session is defined that runs @smoke-tagged Behave scenarios and smoke-tagged Robot tests.
The smoke_tests session completes in under 60 seconds on a standard developer machine.
All existing nox sessions (unit_tests, integration_tests, slow_integration_tests) continue to run the full suite when invoked without extra arguments.
The noxfile changes are covered by updated features/ci_workflow_validation.feature scenarios.
Developer documentation (README or docs/) is updated to describe the new tag-based invocation patterns.

Supporting Information

Related: #1715 — Introduce Conditional Execution for Faster PR Checks (CI event-based)
Related: #1702 — Implement conditional execution to skip jobs based on file changes (CI path-based)
Parent Epic: #1678 — CI Execution Time Optimization
Behave tag expressions: https://behave.readthedocs.io/en/stable/tag_expressions.html
Robot Framework tag filtering: https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#tagging-test-cases

Subtasks

Audit existing Behave feature files for @smoke, @slow, @fast, and @wip tag usage; document findings
Audit existing Robot Framework test files for tag usage; document findings
Update noxfile.py: pass session.posargs through to Behave in unit_tests session
Update noxfile.py: pass session.posargs through to Robot Framework in integration_tests and slow_integration_tests sessions
Add smoke_tests nox session that runs @smoke-tagged Behave scenarios and smoke-tagged Robot tests
Tag a representative set of fast, self-contained Behave scenarios with @smoke
Tag a representative set of fast, self-contained Robot tests with smoke
Update features/ci_workflow_validation.feature to assert smoke_tests session and posargs passthrough are present
Update developer documentation with tag-based invocation examples
Tests (Behave): Add scenarios for conditional nox session invocation
Tests (Robot): Add integration test verifying smoke_tests session completes within time budget
Verify coverage >= 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
nox -s unit_tests -- --tags=@smoke and nox -s integration_tests -- --include smoke work correctly.
nox -s smoke_tests exists and completes in under 60 seconds.
All nox stages pass.
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-conditional-nox-test-execution-tags` - **Commit Message**: `feat(ci): add conditional nox test execution via behave tags and robot filters` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The nox test sessions (`unit_tests`, `integration_tests`, `slow_integration_tests`) currently execute the entire test suite unconditionally. There is no mechanism for developers or CI jobs to run a targeted subset of tests — for example, only smoke tests, only fast tests, or only tests relevant to a specific module — without modifying the noxfile directly. This gap has two consequences: 1. **Local developer feedback is slow.** A developer working on a small change must wait for the full suite to complete before getting feedback, even when only a handful of scenarios are relevant. 2. **CI cannot leverage tag-based parallelism.** Related issues (#1715, #1702) address CI-level job skipping, but even within a single job, all tests run regardless of relevance. Tag-based filtering within nox sessions would complement those optimizations. The project uses Behave (BDD/Gherkin) for unit-level tests and Robot Framework for integration tests. Both frameworks support tag-based filtering natively (`--tags` in Behave, `--include`/`--exclude` in Robot Framework), but the nox sessions do not expose these options to callers. ## Expected Behavior - Developers can pass Behave tag expressions to the `unit_tests` nox session: ``` nox -s unit_tests -- --tags=@smoke nox -s unit_tests -- --tags=~@slow ``` - Developers can pass Robot Framework include/exclude filters to the `integration_tests` and `slow_integration_tests` sessions: ``` nox -s integration_tests -- --include smoke nox -s slow_integration_tests -- --exclude wip ``` - A dedicated `nox -s smoke_tests` session exists that runs only `@smoke`-tagged Behave scenarios and `smoke`-tagged Robot tests, completing in under 60 seconds on a standard developer machine. - All existing nox sessions continue to work without arguments (full suite, no regression). - The CI workflow can invoke `nox -s smoke_tests` as a fast pre-flight check on PR commits (complementing #1715). ## Acceptance Criteria - [ ] `nox -s unit_tests -- --tags=<expr>` passes the tag expression through to Behave and filters scenarios accordingly. - [ ] `nox -s integration_tests -- --include <tag>` and `-- --exclude <tag>` pass through to Robot Framework. - [ ] A `smoke_tests` nox session is defined that runs `@smoke`-tagged Behave scenarios and `smoke`-tagged Robot tests. - [ ] The `smoke_tests` session completes in under 60 seconds on a standard developer machine. - [ ] All existing nox sessions (`unit_tests`, `integration_tests`, `slow_integration_tests`) continue to run the full suite when invoked without extra arguments. - [ ] The noxfile changes are covered by updated `features/ci_workflow_validation.feature` scenarios. - [ ] Developer documentation (README or `docs/`) is updated to describe the new tag-based invocation patterns. ## Supporting Information - Related: #1715 — Introduce Conditional Execution for Faster PR Checks (CI event-based) - Related: #1702 — Implement conditional execution to skip jobs based on file changes (CI path-based) - Parent Epic: #1678 — CI Execution Time Optimization - Behave tag expressions: https://behave.readthedocs.io/en/stable/tag_expressions.html - Robot Framework tag filtering: https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#tagging-test-cases ## Subtasks - [ ] Audit existing Behave feature files for `@smoke`, `@slow`, `@fast`, and `@wip` tag usage; document findings - [ ] Audit existing Robot Framework test files for tag usage; document findings - [ ] Update `noxfile.py`: pass `session.posargs` through to Behave in `unit_tests` session - [ ] Update `noxfile.py`: pass `session.posargs` through to Robot Framework in `integration_tests` and `slow_integration_tests` sessions - [ ] Add `smoke_tests` nox session that runs `@smoke`-tagged Behave scenarios and `smoke`-tagged Robot tests - [ ] Tag a representative set of fast, self-contained Behave scenarios with `@smoke` - [ ] Tag a representative set of fast, self-contained Robot tests with `smoke` - [ ] Update `features/ci_workflow_validation.feature` to assert `smoke_tests` session and posargs passthrough are present - [ ] Update developer documentation with tag-based invocation examples - [ ] Tests (Behave): Add scenarios for conditional nox session invocation - [ ] Tests (Robot): Add integration test verifying `smoke_tests` session completes within time budget - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - `nox -s unit_tests -- --tags=@smoke` and `nox -s integration_tests -- --include smoke` work correctly. - `nox -s smoke_tests` exists and completes in under 60 seconds. - All nox stages pass. - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-02 23:43:06 +00:00

freemo added a new dependency

2026-04-02 23:48:01 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing

freemo commented

2026-04-02 23:51:02 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo added the

MoSCoW

Could have

label

2026-04-02 23:51:02 +00:00

freemo referenced this issue

2026-04-02 23:51:34 +00:00

TEST-INFRA: [ci-pipeline-design] Implement conditional test execution based on code changes #1793

freemo referenced this issue

2026-04-02 23:56:06 +00:00

TEST-INFRA: [ci-pipeline-design] Implement conditional test execution #1831