UAT: cli_lifecycle_e2e integration tests use MagicMock throughout — not true end-to-end tests against real database/services #5521

New issue

Open

opened 2026-04-09 07:11:54 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-09 07:11:54 +00:00

Owner

Bug Report

Feature Area: E2E workflow specification tests — complete end-to-end tests that verify the full plan lifecycle

Severity: High — the "E2E" lifecycle tests are actually mock-based unit tests, not true integration tests

What Was Tested

Analyzed robot/cli_lifecycle_e2e.robot and robot/helper_cli_lifecycle_e2e.py to verify whether the lifecycle tests exercise real services.

Expected Behavior (from spec)

The v3.6.0 milestone requires E2E workflow specification tests that verify the full plan lifecycle. True E2E tests should:

Use a real SQLite database (not mocks)
Exercise the actual service layer (PlanLifecycleService, PlanApplyService)
Verify that data is actually persisted and retrieved correctly
Test the complete data flow from CLI → service → repository → database → response

Actual Behavior

All tests in cli_lifecycle_e2e.robot use MagicMock for every service call:

# plan_use() — uses mock
mock_service = MagicMock()
mock_service.get_action_by_name.return_value = _mock_action()
mock_service.use_action.return_value = _mock_plan()
with patch("cleveragents.cli.commands.plan._get_lifecycle_service", return_value=mock_service):
    result = runner.invoke(plan_app, ["use", "local/e2e-action", "proj-a"])

# plan_execute() — uses mock
mock_service = MagicMock()
mock_service.get_plan.return_value = _mock_plan(...)
mock_service.execute_plan.return_value = _mock_plan(...)
with patch("cleveragents.cli.commands.plan._get_lifecycle_service", return_value=mock_service):
    result = runner.invoke(plan_app, ["execute", _PLAN_ULID])

These tests verify that:

CLI argument parsing works
Output formatting is correct
The correct service methods are called with the right arguments

But they do NOT verify:

That the database schema is correct
That data is actually persisted
That the service layer correctly handles real database operations
That the full data flow works end-to-end

Contrast With Real Integration Tests

The robot/plan_diff_artifacts.robot tests (via helper_plan_diff_artifacts.py) DO use real services:

def _create_services():
    settings = Settings()
    lifecycle = PlanLifecycleService(settings=settings)
    store = InMemoryChangeSetStore()
    apply_svc = PlanApplyService(lifecycle_service=lifecycle, changeset_store=store)
    return lifecycle, store, apply_svc

This is the correct pattern for E2E tests. The lifecycle E2E tests should follow this pattern.

Code Location

robot/cli_lifecycle_e2e.robot — all test cases
robot/helper_cli_lifecycle_e2e.py — all helper functions use MagicMock
robot/helper_plan_diff_artifacts.py — correct pattern using real services

Impact

The "E2E" lifecycle tests provide false confidence — they pass even when the real service layer is broken
Database schema changes, migration failures, or service bugs would not be caught by these tests
The v3.6.0 milestone requirement for "complete end-to-end tests that verify the full plan lifecycle" is not met

Suggested Fix

Refactor helper_cli_lifecycle_e2e.py to use real services (similar to helper_plan_diff_artifacts.py), or add a separate helper_cli_lifecycle_e2e_real.py that exercises the full stack without mocks.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: E2E workflow specification tests — complete end-to-end tests that verify the full plan lifecycle **Severity**: High — the "E2E" lifecycle tests are actually mock-based unit tests, not true integration tests --- ## What Was Tested Analyzed `robot/cli_lifecycle_e2e.robot` and `robot/helper_cli_lifecycle_e2e.py` to verify whether the lifecycle tests exercise real services. ## Expected Behavior (from spec) The v3.6.0 milestone requires **E2E workflow specification tests** that verify the full plan lifecycle. True E2E tests should: - Use a real SQLite database (not mocks) - Exercise the actual service layer (PlanLifecycleService, PlanApplyService) - Verify that data is actually persisted and retrieved correctly - Test the complete data flow from CLI → service → repository → database → response ## Actual Behavior All tests in `cli_lifecycle_e2e.robot` use `MagicMock` for every service call: ```python # plan_use() — uses mock mock_service = MagicMock() mock_service.get_action_by_name.return_value = _mock_action() mock_service.use_action.return_value = _mock_plan() with patch("cleveragents.cli.commands.plan._get_lifecycle_service", return_value=mock_service): result = runner.invoke(plan_app, ["use", "local/e2e-action", "proj-a"]) # plan_execute() — uses mock mock_service = MagicMock() mock_service.get_plan.return_value = _mock_plan(...) mock_service.execute_plan.return_value = _mock_plan(...) with patch("cleveragents.cli.commands.plan._get_lifecycle_service", return_value=mock_service): result = runner.invoke(plan_app, ["execute", _PLAN_ULID]) ``` These tests verify that: - CLI argument parsing works - Output formatting is correct - The correct service methods are called with the right arguments But they do **NOT** verify: - That the database schema is correct - That data is actually persisted - That the service layer correctly handles real database operations - That the full data flow works end-to-end ## Contrast With Real Integration Tests The `robot/plan_diff_artifacts.robot` tests (via `helper_plan_diff_artifacts.py`) DO use real services: ```python def _create_services(): settings = Settings() lifecycle = PlanLifecycleService(settings=settings) store = InMemoryChangeSetStore() apply_svc = PlanApplyService(lifecycle_service=lifecycle, changeset_store=store) return lifecycle, store, apply_svc ``` This is the correct pattern for E2E tests. The lifecycle E2E tests should follow this pattern. ## Code Location - `robot/cli_lifecycle_e2e.robot` — all test cases - `robot/helper_cli_lifecycle_e2e.py` — all helper functions use `MagicMock` - `robot/helper_plan_diff_artifacts.py` — correct pattern using real services ## Impact - The "E2E" lifecycle tests provide false confidence — they pass even when the real service layer is broken - Database schema changes, migration failures, or service bugs would not be caught by these tests - The v3.6.0 milestone requirement for "complete end-to-end tests that verify the full plan lifecycle" is not met ## Suggested Fix Refactor `helper_cli_lifecycle_e2e.py` to use real services (similar to `helper_plan_diff_artifacts.py`), or add a separate `helper_cli_lifecycle_e2e_real.py` that exercises the full stack without mocks. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 07:12:28 +00:00

HAL9000 referenced this issue