Run slow integration tests in CI on every PR #1901

Open
opened 2026-04-03 00:10:06 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: task/ci-run-slow-integration-tests
  • Commit Message: feat(ci): run slow integration tests in CI on every PR
  • Milestone: v3.8.0
  • Parent Epic: #1678

Background and Context

The CI pipeline currently skips or excludes slow integration tests, meaning regressions in long-running integration scenarios can go undetected until a developer runs the full suite locally. As the CleverAgents platform grows — spanning the plan lifecycle, actor runtime, resource registry, tool execution, ACMS, and server layers — the integration surface area is large enough that slow tests represent a critical quality gate that must run on every PR.

Current Behavior

Slow integration tests (those marked with a slow tag or equivalent) are not executed as part of the standard CI workflow. They are either excluded via a nox session filter or simply not invoked in .forgejo/workflows/ci.yml, meaning PRs can be merged without validating these scenarios.

Expected Behavior

Every PR triggers a CI job that executes the full integration test suite, including slow tests. The job must:

  • Run all integration tests regardless of execution time.
  • Report failures clearly so contributors can identify regressions before merge.
  • Respect the existing coverage threshold (≥ 97%).

Acceptance Criteria

  • .forgejo/workflows/ci.yml includes a job (or extends the existing integration_tests job) that runs slow integration tests without exclusion filters.
  • The nox session used by CI does not skip tests tagged slow (or equivalent marker).
  • The job runs on every PR and push to master.
  • CI fails if any slow integration test fails.
  • Coverage remains ≥ 97% after the change.

Supporting Information

  • Parent Epic: #1678 (CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing)
  • Related CI workflow: .forgejo/workflows/ci.yml
  • Task runner: nox
  • Testing framework: Behave (BDD/Gherkin) for unit/integration; Robot Framework for smoke/e2e

Subtasks

  • Audit current CI workflow to identify where slow integration tests are excluded or skipped
  • Update the integration_tests nox session (or add a slow_integration_tests session) to include slow tests
  • Update .forgejo/workflows/ci.yml to invoke the updated/new nox session on every PR
  • Add or update features/ci_workflow_validation.feature to assert slow integration tests are included in CI
  • Verify all nox stages pass: nox
  • Verify coverage ≥ 97%: nox -s coverage_report

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly (feat(ci): run slow integration tests in CI on every PR), followed by a blank line, then additional lines providing relevant implementation details.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly (task/ci-run-slow-integration-tests).
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
  • All nox stages pass.
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `task/ci-run-slow-integration-tests` - **Commit Message**: `feat(ci): run slow integration tests in CI on every PR` - **Milestone**: v3.8.0 - **Parent Epic**: #1678 ## Background and Context The CI pipeline currently skips or excludes slow integration tests, meaning regressions in long-running integration scenarios can go undetected until a developer runs the full suite locally. As the CleverAgents platform grows — spanning the plan lifecycle, actor runtime, resource registry, tool execution, ACMS, and server layers — the integration surface area is large enough that slow tests represent a critical quality gate that must run on every PR. ## Current Behavior Slow integration tests (those marked with a `slow` tag or equivalent) are not executed as part of the standard CI workflow. They are either excluded via a nox session filter or simply not invoked in `.forgejo/workflows/ci.yml`, meaning PRs can be merged without validating these scenarios. ## Expected Behavior Every PR triggers a CI job that executes the full integration test suite, including slow tests. The job must: - Run all integration tests regardless of execution time. - Report failures clearly so contributors can identify regressions before merge. - Respect the existing coverage threshold (≥ 97%). ## Acceptance Criteria - [ ] `.forgejo/workflows/ci.yml` includes a job (or extends the existing `integration_tests` job) that runs slow integration tests without exclusion filters. - [ ] The nox session used by CI does not skip tests tagged `slow` (or equivalent marker). - [ ] The job runs on every PR and push to `master`. - [ ] CI fails if any slow integration test fails. - [ ] Coverage remains ≥ 97% after the change. ## Supporting Information - Parent Epic: #1678 (CI Execution Time Optimization — Timeouts, Concurrency, and Coverage Artifact Sharing) - Related CI workflow: `.forgejo/workflows/ci.yml` - Task runner: `nox` - Testing framework: Behave (BDD/Gherkin) for unit/integration; Robot Framework for smoke/e2e ## Subtasks - [ ] Audit current CI workflow to identify where slow integration tests are excluded or skipped - [ ] Update the `integration_tests` nox session (or add a `slow_integration_tests` session) to include slow tests - [ ] Update `.forgejo/workflows/ci.yml` to invoke the updated/new nox session on every PR - [ ] Add or update `features/ci_workflow_validation.feature` to assert slow integration tests are included in CI - [ ] Verify all nox stages pass: `nox` - [ ] Verify coverage ≥ 97%: `nox -s coverage_report` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly (`feat(ci): run slow integration tests in CI on every PR`), followed by a blank line, then additional lines providing relevant implementation details. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly (`task/ci-run-slow-integration-tests`). - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.8.0 milestone 2026-04-03 00:10:13 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • MoSCoW: MoSCoW/Could Have — CI/test infrastructure improvement.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: MoSCoW/Could Have — CI/test infrastructure improvement. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1901
No description provided.