TEST-INFRA: [ci-execution-time] `slow_integration_tests` session runs all tests, not just slow ones #2284

New issue

Open

opened 2026-04-03 12:49:56 +00:00 by freemo · 0 comments

freemo commented

2026-04-03 12:49:56 +00:00

Owner

Metadata

Branch: fix/ci-execution-time-slow-integration-tests-filter
Commit Message: fix(ci): restrict slow_integration_tests nox session to @slow-tagged tests only
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The slow_integration_tests session in noxfile.py is intended to run only the subset of Robot Framework integration tests that are tagged as slow — tests that are too time-consuming to run on every CI push but should still be exercised periodically (e.g., on scheduled runs or before releases).

Current Behavior

The slow_integration_tests nox session currently invokes robot with --exclude tdd_fixture but applies no --include filter to restrict execution to slow-tagged tests. As a result, it runs the entire integration test suite — identical in scope to the standard integration_tests session (which uses pabot for parallelism). This makes slow_integration_tests fully redundant: it duplicates the work of integration_tests without providing any distinct value, and it wastes CI runner time by re-executing tests that have already been covered.

Relevant session definition in noxfile.py:

def slow_integration_tests(session: nox.Session):
    """Run Robot Framework integration tests."""
    session.run(
        "robot",
        "--outputdir", "build/reports/robot",
        "--loglevel", "INFO",
        "--report", "report.html",
        "--log", "log.html",
        "--xunit", "xunit.xml",
        "--listener", tdd_listener,
        "--exclude", "tdd_fixture",
        "robot/",
        *session.posargs,
    )

Notice the absence of --include slow — the session runs all tests, not just those tagged slow.

Expected Behavior

The slow_integration_tests session should only execute Robot Framework tests that carry the slow tag. This is achieved by adding --include slow to the robot invocation, ensuring the session is meaningfully distinct from integration_tests and does not duplicate its work.

Acceptance Criteria

The slow_integration_tests session in noxfile.py passes --include slow to the robot command so that only tests tagged slow are executed.
The session docstring is updated to accurately describe its purpose (running only slow-tagged tests).
Running nox -s slow_integration_tests with no slow-tagged tests present exits cleanly (Robot Framework exits 0 with a "no tests matched" result, or the session is guarded appropriately).
The CI workflow (.forgejo/workflows/ci.yml) is reviewed to confirm slow_integration_tests is invoked in the correct context (e.g., scheduled runs or release gates) and not on every push alongside integration_tests.

Supporting Information

Parent Epic: #1678 — CI Execution Time Optimization
Robot Framework tag filtering docs: --include <tag> selects only tests with that tag; --exclude <tag> skips tests with that tag.
The integration_tests session (using pabot) already covers the full suite in parallel; slow_integration_tests should be a complementary, narrower session.

Subtasks

Add --include, slow arguments to the robot invocation in slow_integration_tests in noxfile.py
Update the session docstring to reflect its actual purpose
Review .forgejo/workflows/ci.yml to confirm the session is used in the correct CI context (not duplicating integration_tests on every push)
Add or update a Robot Framework feature/scenario that validates the slow_integration_tests session only runs slow-tagged tests
Verify coverage >= 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/ci-execution-time-slow-integration-tests-filter` - **Commit Message**: `fix(ci): restrict slow_integration_tests nox session to @slow-tagged tests only` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The `slow_integration_tests` session in `noxfile.py` is intended to run only the subset of Robot Framework integration tests that are tagged as `slow` — tests that are too time-consuming to run on every CI push but should still be exercised periodically (e.g., on scheduled runs or before releases). ## Current Behavior The `slow_integration_tests` nox session currently invokes `robot` with `--exclude tdd_fixture` but applies **no `--include` filter** to restrict execution to slow-tagged tests. As a result, it runs the **entire integration test suite** — identical in scope to the standard `integration_tests` session (which uses `pabot` for parallelism). This makes `slow_integration_tests` fully redundant: it duplicates the work of `integration_tests` without providing any distinct value, and it wastes CI runner time by re-executing tests that have already been covered. Relevant session definition in `noxfile.py`: ```python def slow_integration_tests(session: nox.Session): """Run Robot Framework integration tests.""" session.run( "robot", "--outputdir", "build/reports/robot", "--loglevel", "INFO", "--report", "report.html", "--log", "log.html", "--xunit", "xunit.xml", "--listener", tdd_listener, "--exclude", "tdd_fixture", "robot/", *session.posargs, ) ``` Notice the absence of `--include slow` — the session runs all tests, not just those tagged `slow`. ## Expected Behavior The `slow_integration_tests` session should only execute Robot Framework tests that carry the `slow` tag. This is achieved by adding `--include slow` to the `robot` invocation, ensuring the session is meaningfully distinct from `integration_tests` and does not duplicate its work. ## Acceptance Criteria - [ ] The `slow_integration_tests` session in `noxfile.py` passes `--include slow` to the `robot` command so that only tests tagged `slow` are executed. - [ ] The session docstring is updated to accurately describe its purpose (running only `slow`-tagged tests). - [ ] Running `nox -s slow_integration_tests` with no `slow`-tagged tests present exits cleanly (Robot Framework exits 0 with a "no tests matched" result, or the session is guarded appropriately). - [ ] The CI workflow (`.forgejo/workflows/ci.yml`) is reviewed to confirm `slow_integration_tests` is invoked in the correct context (e.g., scheduled runs or release gates) and not on every push alongside `integration_tests`. ## Supporting Information - Parent Epic: #1678 — CI Execution Time Optimization - Robot Framework tag filtering docs: `--include <tag>` selects only tests with that tag; `--exclude <tag>` skips tests with that tag. - The `integration_tests` session (using `pabot`) already covers the full suite in parallel; `slow_integration_tests` should be a complementary, narrower session. ## Subtasks - [ ] Add `--include`, `slow` arguments to the `robot` invocation in `slow_integration_tests` in `noxfile.py` - [ ] Update the session docstring to reflect its actual purpose - [ ] Review `.forgejo/workflows/ci.yml` to confirm the session is used in the correct CI context (not duplicating `integration_tests` on every push) - [ ] Add or update a Robot Framework feature/scenario that validates the `slow_integration_tests` session only runs `slow`-tagged tests - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-03 12:50:27 +00:00

freemo added a new dependency

2026-04-03 12:53:32 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing