TEST-INFRA: [ci-execution-time] Review and optimize slow integration tests #2217

New issue

Open

opened 2026-04-03 09:36:15 +00:00 by freemo · 1 comment

freemo commented

2026-04-03 09:36:15 +00:00

Owner

Metadata

Branch: chore/ci-execution-time-slow-integration-tests
Commit Message: chore(ci): add nightly slow integration test job and optimize slow tests
Milestone: v3.8.0
Parent Epic: #1678

Background and Context

The integration_tests nox session excludes tests tagged as slow. While this is a good practice to keep the main CI loop fast, we need to ensure these slow tests are run regularly and are as optimized as possible. Without a dedicated CI job, slow tests may go unexecuted for extended periods, allowing regressions to accumulate undetected. Additionally, slow tests that are never profiled may contain unnecessary bottlenecks that could be resolved without sacrificing coverage.

Expected Behavior

A dedicated nightly CI job runs the slow_integration_tests nox session on a schedule. Slow tests are profiled, documented, and where possible refactored to reduce execution time while maintaining full coverage.

Acceptance Criteria

A new CI job exists that runs the slow_integration_tests nox session.
The job is scheduled to run nightly.
The performance of the slow tests is analyzed and documented.
At least one slow test is optimized with a measurably reduced execution time.

Supporting Information

Related Epic: #1678 — CI Execution Time Optimization
The slow tag is used in the integration_tests nox session to exclude long-running tests from the default CI loop.

Subtasks

Create a new CI job that runs the slow_integration_tests nox session
Schedule this job to run nightly
Analyze the performance of the tests tagged as slow to identify optimization opportunities
Document the performance analysis findings
Refactor slow tests to improve their performance without reducing their coverage
Tests (Behave): Verify any refactored test logic still passes all scenarios
Tests (Robot): Confirm integration coverage is maintained after refactoring
Verify coverage >= 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A new CI job is created to run the slow_integration_tests nox session.
The job is scheduled to run nightly.
The performance of the slow tests is analyzed and documented.
At least one slow test is optimized, and its execution time is measurably reduced.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

## Metadata - **Branch**: `chore/ci-execution-time-slow-integration-tests` - **Commit Message**: `chore(ci): add nightly slow integration test job and optimize slow tests` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The `integration_tests` nox session excludes tests tagged as `slow`. While this is a good practice to keep the main CI loop fast, we need to ensure these slow tests are run regularly and are as optimized as possible. Without a dedicated CI job, slow tests may go unexecuted for extended periods, allowing regressions to accumulate undetected. Additionally, slow tests that are never profiled may contain unnecessary bottlenecks that could be resolved without sacrificing coverage. ## Expected Behavior A dedicated nightly CI job runs the `slow_integration_tests` nox session on a schedule. Slow tests are profiled, documented, and where possible refactored to reduce execution time while maintaining full coverage. ## Acceptance Criteria - A new CI job exists that runs the `slow_integration_tests` nox session. - The job is scheduled to run nightly. - The performance of the slow tests is analyzed and documented. - At least one slow test is optimized with a measurably reduced execution time. ## Supporting Information - Related Epic: #1678 — CI Execution Time Optimization - The `slow` tag is used in the `integration_tests` nox session to exclude long-running tests from the default CI loop. ## Subtasks - [ ] Create a new CI job that runs the `slow_integration_tests` nox session - [ ] Schedule this job to run nightly - [ ] Analyze the performance of the tests tagged as `slow` to identify optimization opportunities - [ ] Document the performance analysis findings - [ ] Refactor slow tests to improve their performance without reducing their coverage - [ ] Tests (Behave): Verify any refactored test logic still passes all scenarios - [ ] Tests (Robot): Confirm integration coverage is maintained after refactoring - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A new CI job is created to run the `slow_integration_tests` nox session. - The job is scheduled to run nightly. - The performance of the slow tests is analyzed and documented. - At least one slow test is optimized, and its execution time is measurably reduced. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure | Agent: ca-new-issue-creator

freemo added this to the v3.8.0 milestone

2026-04-03 09:36:21 +00:00

freemo commented

2026-04-03 09:37:10 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: Medium (confirmed)
Milestone: v3.8.0 (confirmed — CI infrastructure)
MoSCoW: Could Have — Nightly slow test job and optimization is a CI quality improvement. The slow tests are currently excluded from the main CI loop, which is correct. Adding a nightly job ensures they're still run regularly. Not blocking any deliverables.
Parent Epic: #1678 (confirmed correct)

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium (confirmed) - **Milestone**: v3.8.0 (confirmed — CI infrastructure) - **MoSCoW**: Could Have — Nightly slow test job and optimization is a CI quality improvement. The slow tests are currently excluded from the main CI loop, which is correct. Adding a nightly job ensures they're still run regularly. Not blocking any deliverables. - **Parent Epic**: #1678 (confirmed correct) --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo added the

MoSCoW

Could have

label

2026-04-03 09:37:12 +00:00

freemo added a new dependency

2026-04-03 09:37:58 +00:00

#1678 Epic: CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing