cleveragents/cleveragents-core

Fork 3

UAT: Robot integration suites rely on MagicMock service patches #7016

New issue

Open

opened 2026-04-10 06:39:51 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-10 06:39:51 +00:00

Owner

What was tested

Testing Infrastructure (Behave/Robot/ASV) integration test stack against CONTRIBUTING.md requirements.

Expected behavior (from spec)

CONTRIBUTING.md > Test Isolation and Mock Placement: "Mocks, fakes, stubs, and test doubles are permitted only in unit tests. Integration tests must exercise real services, real endpoints, and real dependencies — mocking of any kind is strictly prohibited in integration tests."

Actual behavior

Robot Framework integration suites rely on helper scripts that patch core services with unittest.mock.MagicMock, so the CLI never talks to the real service layer. For example:
- robot/helper_cli_lifecycle.py: the action_create, plan_use_sandbox, plan_execute_changeset, and related helpers create MagicMock services and patch cleveragents.cli.commands.*._get_lifecycle_service before invoking the CLI.
- robot/helper_plan_cli_spec.py: every helper wraps CLI entry points with patch(..., return_value=MagicMock()), preventing real plan lifecycle code from running.
Because the Robot suites import and execute these helpers (see robot/cli_lifecycle.robot and robot/plan_cli_spec.robot), integration coverage comes entirely from mocked dependencies, violating the spec mandate that integration tests hit real services.

Steps to reproduce

Open robot/cli_lifecycle.robot and note it executes ${PYTHON} ${HELPER} where ${HELPER} points to helper_cli_lifecycle.py.
Inspect robot/helper_cli_lifecycle.py (e.g., lines 132-205) and observe MagicMock instances being patched in place of lifecycle services.
Inspect robot/helper_plan_cli_spec.py (e.g., lines 102-200) showing MagicMock-backed service patches for every subcommand.
Run any of the affected Robot suites (e.g., nox -s integration_tests -- robot/cli_lifecycle.robot) and observe that the helpers execute, so the integration test never exercises the real service layer.

Code locations

robot/helper_cli_lifecycle.py
robot/helper_plan_cli_spec.py
robot/cli_lifecycle.robot
robot/plan_cli_spec.robot

## What was tested - Testing Infrastructure (Behave/Robot/ASV) integration test stack against CONTRIBUTING.md requirements. ## Expected behavior (from spec) - `CONTRIBUTING.md` > Test Isolation and Mock Placement: "Mocks, fakes, stubs, and test doubles are permitted only in unit tests. Integration tests must exercise real services, real endpoints, and real dependencies — mocking of any kind is strictly prohibited in integration tests." ## Actual behavior - Robot Framework integration suites rely on helper scripts that patch core services with `unittest.mock.MagicMock`, so the CLI never talks to the real service layer. For example: - `robot/helper_cli_lifecycle.py`: the `action_create`, `plan_use_sandbox`, `plan_execute_changeset`, and related helpers create `MagicMock` services and patch `cleveragents.cli.commands.*._get_lifecycle_service` before invoking the CLI. - `robot/helper_plan_cli_spec.py`: every helper wraps CLI entry points with `patch(..., return_value=MagicMock())`, preventing real plan lifecycle code from running. - Because the Robot suites import and execute these helpers (see `robot/cli_lifecycle.robot` and `robot/plan_cli_spec.robot`), integration coverage comes entirely from mocked dependencies, violating the spec mandate that integration tests hit real services. ## Steps to reproduce 1. Open `robot/cli_lifecycle.robot` and note it executes `${PYTHON} ${HELPER}` where `${HELPER}` points to `helper_cli_lifecycle.py`. 2. Inspect `robot/helper_cli_lifecycle.py` (e.g., lines 132-205) and observe `MagicMock` instances being patched in place of lifecycle services. 3. Inspect `robot/helper_plan_cli_spec.py` (e.g., lines 102-200) showing `MagicMock`-backed service patches for every subcommand. 4. Run any of the affected Robot suites (e.g., `nox -s integration_tests -- robot/cli_lifecycle.robot`) and observe that the helpers execute, so the integration test never exercises the real service layer. ## Code locations - `robot/helper_cli_lifecycle.py` - `robot/helper_plan_cli_spec.py` - `robot/cli_lifecycle.robot` - `robot/plan_cli_spec.robot`

HAL9000 added the

State

Unverified

label

2026-04-10 06:44:23 +00:00

HAL9000 self-assigned this

2026-04-10 06:47:59 +00:00

HAL9000 added this to the v3.5.0 milestone

2026-04-10 06:47:59 +00:00

HAL9000 referenced this issue

2026-04-10 06:48:06 +00:00

[AUTO-BLOG] Status: Backlog Groomer (Cycle 5) #7020

freemo added the

labels

2026-04-12 18:37:48 +00:00

HAL9000 added

and removed

labels

2026-04-14 07:19:37 +00:00

HAL9000 commented

2026-04-14 07:19:38 +00:00

Author

Owner

✅ Verified — UAT bug: Robot integration suites rely on MagicMock service patches — not testing real behavior. MoSCoW: Should-have. Priority: Medium.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — UAT bug: Robot integration suites rely on MagicMock service patches — not testing real behavior. MoSCoW: Should-have. Priority: Medium. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor