UAT: `cli_lifecycle_e2e.robot` is named as an E2E integration test but uses mocked services throughout — provides no real integration coverage #4071

New issue

Open

opened 2026-04-06 09:55:44 +00:00 by freemo · 0 comments

freemo commented

2026-04-06 09:55:44 +00:00

Owner

Metadata

Branch: fix/cli-lifecycle-e2e-real-services
Commit Message: fix(robot): replace mocked services in cli_lifecycle_e2e with real CLI subprocess calls
Milestone: (none — backlog)
Parent Epic: (Integration Testing Epic)

Background and Context

robot/cli_lifecycle_e2e.robot is named with the _e2e suffix, clearly indicating it is intended to be an end-to-end integration test for the plan lifecycle CLI workflow. However, its helper (robot/helper_cli_lifecycle_e2e.py) uses unittest.mock.MagicMock and patch() to replace every service layer call, making it functionally equivalent to a unit test.

This violates CONTRIBUTING.md lines 568–570:

Integration tests must exercise real services, real endpoints, and real dependencies — mocking of any kind is strictly prohibited in integration tests.

What Was Tested

Code analysis of robot/cli_lifecycle_e2e.robot and robot/helper_cli_lifecycle_e2e.py.

Expected Behavior (from spec/CONTRIBUTING.md)

cli_lifecycle_e2e.robot should exercise the complete plan lifecycle (action create → plan use → plan execute → plan apply) using:

Real CLI subprocess invocations (via python -m cleveragents)
Real SQLite database (in a temporary workspace)
Real service layer (no mocked _get_lifecycle_service)
Real plan state transitions persisted to the database

The correct pattern is demonstrated in robot/helper_m1_e2e_verification.py which uses run_cli() from helper_e2e_common.py to invoke the real CLI with a real database.

Actual Behavior

robot/helper_cli_lifecycle_e2e.py (lines 13, 111–322) uses MagicMock and patch() to replace the entire service layer:

# robot/helper_cli_lifecycle_e2e.py (lines 13, 113-131)
from unittest.mock import MagicMock, patch

def action_create() -> None:
    """Verify action create from config file."""
    mock_service = MagicMock()
    mock_service.create_action.return_value = _mock_action()
    yaml_path = _write_yaml(_VALID_YAML)

    try:
        with patch(
            "cleveragents.cli.commands.action._get_lifecycle_service",
            return_value=mock_service,   # ← Real service replaced with mock
        ):
            result = runner.invoke(action_app, ["create", "--config", yaml_path])

The full_lifecycle function (lines 266–322) patches three separate service factories simultaneously:

with (
    patch("cleveragents.cli.commands.action._get_lifecycle_service", return_value=mock_service),
    patch("cleveragents.cli.commands.plan._get_lifecycle_service", return_value=mock_service),
    patch("cleveragents.cli.commands.plan._get_plan_executor", return_value=MagicMock()),
):

Every single test case in cli_lifecycle_e2e.robot uses mocked services. The test suite:

Never writes to a real database
Never exercises real plan state transitions
Never validates real persistence of actions or plans
Never exercises the real PlanLifecycleService or PlanExecutor
Cannot detect database schema mismatches or service layer regressions

Impact

The cli_lifecycle_e2e.robot suite is the primary integration test for the core plan lifecycle workflow (the most critical user workflow in the system). Because it uses mocks, it provides zero real integration coverage for:

Action creation and persistence
Plan creation from action
Plan phase transitions (Strategize → Execute → Apply)
Plan cancellation
Plan listing with namespace filtering

Steps to Reproduce

grep "MagicMock\|patch(" robot/helper_cli_lifecycle_e2e.py | wc -l
# Returns: 30+ lines of mock usage

Subtasks

Rewrite robot/helper_cli_lifecycle_e2e.py to use setup_workspace() + run_cli() from helper_e2e_common.py (same pattern as helper_m1_e2e_verification.py)
Remove all from unittest.mock import ... from helper_cli_lifecycle_e2e.py
Verify action create persists to real SQLite and is retrievable via action show
Verify plan use creates a real plan record in the database
Verify plan execute transitions the plan phase in the database
Verify plan apply transitions the plan to Apply phase in the database
Verify plan cancel marks the plan as cancelled in the database
Verify plan list --namespace filters by namespace using real database query
Run nox -s integration_tests to verify all tests pass

Definition of Done

robot/helper_cli_lifecycle_e2e.py contains zero mock imports
All test cases use real CLI subprocess invocations with real SQLite database
Plan state transitions are verified by querying the database after each CLI command
nox -s integration_tests passes
Coverage >= 97%

Supporting Information

Code location: robot/helper_cli_lifecycle_e2e.py — entire file
Robot test file: robot/cli_lifecycle_e2e.robot
Correct pattern: robot/helper_m1_e2e_verification.py — uses run_cli() with real subprocess and real database
Rule source: CONTRIBUTING.md lines 568–570

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Metadata - **Branch**: `fix/cli-lifecycle-e2e-real-services` - **Commit Message**: `fix(robot): replace mocked services in cli_lifecycle_e2e with real CLI subprocess calls` - **Milestone**: (none — backlog) - **Parent Epic**: (Integration Testing Epic) ## Background and Context `robot/cli_lifecycle_e2e.robot` is named with the `_e2e` suffix, clearly indicating it is intended to be an end-to-end integration test for the plan lifecycle CLI workflow. However, its helper (`robot/helper_cli_lifecycle_e2e.py`) uses `unittest.mock.MagicMock` and `patch()` to replace every service layer call, making it functionally equivalent to a unit test. This violates `CONTRIBUTING.md` lines 568–570: > Integration tests must exercise real services, real endpoints, and real dependencies — mocking of any kind is strictly prohibited in integration tests. ## What Was Tested Code analysis of `robot/cli_lifecycle_e2e.robot` and `robot/helper_cli_lifecycle_e2e.py`. ## Expected Behavior (from spec/CONTRIBUTING.md) `cli_lifecycle_e2e.robot` should exercise the complete plan lifecycle (action create → plan use → plan execute → plan apply) using: - Real CLI subprocess invocations (via `python -m cleveragents`) - Real SQLite database (in a temporary workspace) - Real service layer (no mocked `_get_lifecycle_service`) - Real plan state transitions persisted to the database The correct pattern is demonstrated in `robot/helper_m1_e2e_verification.py` which uses `run_cli()` from `helper_e2e_common.py` to invoke the real CLI with a real database. ## Actual Behavior `robot/helper_cli_lifecycle_e2e.py` (lines 13, 111–322) uses `MagicMock` and `patch()` to replace the entire service layer: ```python # robot/helper_cli_lifecycle_e2e.py (lines 13, 113-131) from unittest.mock import MagicMock, patch def action_create() -> None: """Verify action create from config file.""" mock_service = MagicMock() mock_service.create_action.return_value = _mock_action() yaml_path = _write_yaml(_VALID_YAML) try: with patch( "cleveragents.cli.commands.action._get_lifecycle_service", return_value=mock_service, # ← Real service replaced with mock ): result = runner.invoke(action_app, ["create", "--config", yaml_path]) ``` The `full_lifecycle` function (lines 266–322) patches three separate service factories simultaneously: ```python with ( patch("cleveragents.cli.commands.action._get_lifecycle_service", return_value=mock_service), patch("cleveragents.cli.commands.plan._get_lifecycle_service", return_value=mock_service), patch("cleveragents.cli.commands.plan._get_plan_executor", return_value=MagicMock()), ): ``` **Every single test case in `cli_lifecycle_e2e.robot` uses mocked services.** The test suite: - Never writes to a real database - Never exercises real plan state transitions - Never validates real persistence of actions or plans - Never exercises the real `PlanLifecycleService` or `PlanExecutor` - Cannot detect database schema mismatches or service layer regressions ## Impact The `cli_lifecycle_e2e.robot` suite is the primary integration test for the core plan lifecycle workflow (the most critical user workflow in the system). Because it uses mocks, it provides **zero real integration coverage** for: - Action creation and persistence - Plan creation from action - Plan phase transitions (Strategize → Execute → Apply) - Plan cancellation - Plan listing with namespace filtering ## Steps to Reproduce ```bash grep "MagicMock\|patch(" robot/helper_cli_lifecycle_e2e.py | wc -l # Returns: 30+ lines of mock usage ``` ## Subtasks - [ ] Rewrite `robot/helper_cli_lifecycle_e2e.py` to use `setup_workspace()` + `run_cli()` from `helper_e2e_common.py` (same pattern as `helper_m1_e2e_verification.py`) - [ ] Remove all `from unittest.mock import ...` from `helper_cli_lifecycle_e2e.py` - [ ] Verify `action create` persists to real SQLite and is retrievable via `action show` - [ ] Verify `plan use` creates a real plan record in the database - [ ] Verify `plan execute` transitions the plan phase in the database - [ ] Verify `plan apply` transitions the plan to Apply phase in the database - [ ] Verify `plan cancel` marks the plan as cancelled in the database - [ ] Verify `plan list --namespace` filters by namespace using real database query - [ ] Run `nox -s integration_tests` to verify all tests pass ## Definition of Done - [ ] `robot/helper_cli_lifecycle_e2e.py` contains zero mock imports - [ ] All test cases use real CLI subprocess invocations with real SQLite database - [ ] Plan state transitions are verified by querying the database after each CLI command - [ ] `nox -s integration_tests` passes - [ ] Coverage >= 97% ## Supporting Information - **Code location**: `robot/helper_cli_lifecycle_e2e.py` — entire file - **Robot test file**: `robot/cli_lifecycle_e2e.robot` - **Correct pattern**: `robot/helper_m1_e2e_verification.py` — uses `run_cli()` with real subprocess and real database - **Rule source**: `CONTRIBUTING.md` lines 568–570 --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo referenced this issue

2026-04-06 09:57:43 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 10:15:25 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 10:35:53 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775