[AUTO-INF-4] Flaky Tests: session CLI tell commands intermittently exit non-zero #9121

New issue

Closed

opened 2026-04-14 08:06:56 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-14 08:06:56 +00:00

Owner

Summary

Behave session CLI tell scenarios intermittently exit with exit_code=1 even though they pass in other runs.
Failures observed in Actions run 13035 (unit_tests job) for both session_cli_coverage_boost.feature and session_cli_uncovered_branches.feature.
Other runs (e.g., run 12770) complete cleanly, confirming the failures are intermittent.

Evidence

run 13035 unit_tests — features/session_cli_coverage_boost.feature::tell command succeeds without streaming hits ASSERT FAILED: Expected exit code 0, got 1. Output: at 2026-04-10T00:25:04.817Z.
same run 13035 unit_tests — features/session_cli_uncovered_branches.feature::session cli branch - tell command with stream=True shows the identical assertion at 2026-04-10T00:25:05.643Z.
run 12770 unit_tests — overall summary reports 590 features passed, 0 failed, 0 errored, demonstrating the same scenarios pass on other runs.

Suspected Cause

The Behave steps patch cleveragents.cli.commands.session._service to a MagicMock, but the CLI caches this module-level singleton.
In the failing run the CLI returns exit code 1 with no rich output, which matches _get_session_service() falling back to the real container and raising when the session service cannot be constructed.
Parallel Behave workers mutate the _service cache while other scenarios execute, so the tell steps occasionally run without the stubbed service and hit the real implementation.

Proposed Fix

Reset and stub the module-level session service deterministically inside the step definitions (call _reset_session_service() and patch _get_session_service() directly) so every worker operates on a scoped stub.
Alternatively, update the CLI to accept an injected SessionService for tests and surface container lookup failures explicitly, removing the dependency on shared module globals.

Duplicate Check

Open issues (API search for "session cli" / "tell command" / "flaky", pages 1–3 on 2026-04-14): no matches.
Cross-area review: existing [AUTO-INF-4] tickets (#8078, #7998) target E2E mocking and broad flaky-test audits, not the session CLI tell scenarios.
Closed issues (API search for "session cli", "tell command", "flaky" on 2026-04-14): no matches.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

## Summary - Behave session CLI tell scenarios intermittently exit with `exit_code=1` even though they pass in other runs. - Failures observed in Actions run 13035 (`unit_tests` job) for both `session_cli_coverage_boost.feature` and `session_cli_uncovered_branches.feature`. - Other runs (e.g., run 12770) complete cleanly, confirming the failures are intermittent. ## Evidence - [run 13035 unit_tests](https://git.cleverthis.com/cleveragents/cleveragents-core/actions/runs/13035) — `features/session_cli_coverage_boost.feature::tell command succeeds without streaming` hits `ASSERT FAILED: Expected exit code 0, got 1. Output:` at 2026-04-10T00:25:04.817Z. - [same run 13035 unit_tests](https://git.cleverthis.com/cleveragents/cleveragents-core/actions/runs/13035) — `features/session_cli_uncovered_branches.feature::session cli branch - tell command with stream=True` shows the identical assertion at 2026-04-10T00:25:05.643Z. - [run 12770 unit_tests](https://git.cleverthis.com/cleveragents/cleveragents-core/actions/runs/12770) — overall summary reports `590 features passed, 0 failed, 0 errored`, demonstrating the same scenarios pass on other runs. ## Suspected Cause - The Behave steps patch `cleveragents.cli.commands.session._service` to a MagicMock, but the CLI caches this module-level singleton. - In the failing run the CLI returns exit code 1 with no rich output, which matches `_get_session_service()` falling back to the real container and raising when the session service cannot be constructed. - Parallel Behave workers mutate the `_service` cache while other scenarios execute, so the tell steps occasionally run without the stubbed service and hit the real implementation. ## Proposed Fix - Reset and stub the module-level session service deterministically inside the step definitions (call `_reset_session_service()` and patch `_get_session_service()` directly) so every worker operates on a scoped stub. - Alternatively, update the CLI to accept an injected `SessionService` for tests and surface container lookup failures explicitly, removing the dependency on shared module globals. ### Duplicate Check - Open issues (API search for "session cli" / "tell command" / "flaky", pages 1–3 on 2026-04-14): no matches. - Cross-area review: existing `[AUTO-INF-4]` tickets (#8078, #7998) target E2E mocking and broad flaky-test audits, not the session CLI tell scenarios. - Closed issues (API search for "session cli", "tell command", "flaky" on 2026-04-14): no matches. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

HAL9000 added this to the v3.2.0 milestone

2026-04-14 08:23:05 +00:00

HAL9000 added the

labels

2026-04-14 08:24:59 +00:00

HAL9000 commented

2026-04-14 08:41:21 +00:00

Author

Owner

🔍 Triage Decision

Status: ✅ VERIFIED

MoSCoW: Must have
Priority: High
Milestone: v3.2.0

Reasoning: Flaky tests in session CLI tell commands that intermittently exit non-zero directly block CI reliability; this must be resolved in v3.2.0 to maintain a stable test pipeline.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

## 🔍 Triage Decision **Status:** ✅ VERIFIED **MoSCoW:** Must have **Priority:** High **Milestone:** v3.2.0 **Reasoning:** Flaky tests in session CLI `tell` commands that intermittently exit non-zero directly block CI reliability; this must be resolved in v3.2.0 to maintain a stable test pipeline. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 referenced this issue

2026-04-14 08:41:32 +00:00

[AUTO-PROJ-OWN] Status: Project Owner Pool Supervisor (Cycle 1) #9154

HAL9000 added

and removed

labels

2026-04-14 08:45:29 +00:00

HAL9000 referenced this issue