UAT: 58 Robot Framework integration test helpers use unittest.mock — violates strict no-mock rule for integration tests #4069

New issue

Open

opened 2026-04-06 09:55:03 +00:00 by freemo · 0 comments

freemo commented

2026-04-06 09:55:03 +00:00

Owner

Metadata

Branch: fix/integration-test-mock-violations
Commit Message: fix(robot): remove unittest.mock usage from integration test helpers
Milestone: (none — backlog)
Parent Epic: (Integration Testing Epic)

Background and Context

CONTRIBUTING.md (lines 568–570) states explicitly:

Unit Tests Only: Mocks, fakes, stubs, and test doubles are permitted only in unit tests.
Integration tests must exercise real services, real endpoints, and real dependencies — mocking
of any kind is strictly prohibited in integration tests.

This rule is also repeated at line 1390:

ALL mocks, test doubles, and mock implementations must exist only in the features/ directory.

What Was Tested

Systematic review of all 259 helper Python files in the robot/ directory for use of unittest.mock, MagicMock, and patch().

Expected Behavior (from spec/CONTRIBUTING.md)

All Robot Framework integration tests in robot/ must exercise real services and real dependencies. No unittest.mock, MagicMock, patch(), or any other mocking mechanism is permitted in any file under robot/.

Actual Behavior

58 out of 259 helper files (22%) in robot/ use mocking, directly violating the integration test no-mock rule. The complete list of violating files:

robot/helper_a2a_facade_wiring.py          — uses MagicMock for service stubs
robot/helper_action_cli_spec.py            — uses MagicMock + patch() for service layer
robot/helper_actor_add_rich_output.py      — uses MagicMock + patch() for actor services
robot/helper_actor_cli_show.py             — uses MagicMock + patch() for actor registry
robot/helper_actor_run_signature.py        — uses MagicMock
robot/helper_apply_pipeline.py             — uses MagicMock for Plan objects
robot/helper_automation_profile_cli.py     — uses patch()
robot/helper_cli_consistency.py            — uses MagicMock + patch()
robot/helper_cli_extensions.py             — uses MagicMock + patch()
robot/helper_cli_formats.py                — uses MagicMock + patch() for service layer
robot/helper_cli_lifecycle.py              — uses MagicMock + patch()
robot/helper_cli_lifecycle_e2e.py          — uses MagicMock + patch() (named "e2e" but mocked)
robot/helper_cloud_resources.py            — uses MagicMock
robot/helper_config_cli.py                 — uses MagicMock + patch()
robot/helper_config_project_scope.py       — uses MagicMock + patch()
robot/helper_config_resolution.py          — uses MagicMock + patch()
robot/helper_correction_subtree_isolation.py — uses MagicMock
robot/helper_decision_di.py                — uses MagicMock
robot/helper_error_recovery.py             — uses MagicMock for Plan objects
robot/helper_event_bus.py                  — uses MagicMock
robot/helper_invariant_cli.py              — uses MagicMock + patch()
robot/helper_llm_trace.py                  — uses MagicMock
robot/helper_m1_sourcecode_smoke.py        — uses MagicMock + patch()
robot/helper_m3_decision_validation_smoke.py — uses MagicMock + patch()
robot/helper_m4_correction_subplan_smoke.py — uses MagicMock + patch()
robot/helper_m4_e2e_cli.py                 — uses MagicMock + patch() (named "e2e")
robot/helper_m4_e2e_cli_errors.py          — uses MagicMock + patch() (named "e2e")
robot/helper_m4_e2e_common.py              — uses MagicMock (shared mock factories)
robot/helper_m5_e2e_context.py             — uses MagicMock + patch() (named "e2e")
robot/helper_multi_project_subplan.py      — uses MagicMock
robot/helper_plan_cli_spec.py              — uses MagicMock + patch() for service layer
robot/helper_plan_correct_isolated_resolve.py — uses MagicMock
robot/helper_plan_correct_tree_wiring.py   — uses MagicMock
robot/helper_plan_execute_runtime.py       — (no mock import but uses InMemoryChangeSetStore)
robot/helper_plan_use_env_priority.py      — uses MagicMock + patch()
robot/helper_project_context_cli.py        — uses MagicMock + patch()
robot/helper_project_context_set_exec_env_priority.py — uses MagicMock + patch()
robot/helper_provider_registry.py          — uses MagicMock
robot/helper_repl_smoke.py                 — uses MagicMock
robot/helper_resource_handlers.py          — uses MagicMock
robot/helper_revert_re_execution.py        — uses MagicMock
robot/helper_sandbox_checkpoint.py         — uses MagicMock
robot/helper_sandbox_integration.py        — uses MagicMock
robot/helper_session_cli.py                — uses MagicMock (patches session_mod._service)
robot/helper_skill_actor_run.py            — uses MagicMock
robot/helper_subplan_spawn.py              — uses MagicMock
robot/helper_subplan_spawn_orchestration.py — uses MagicMock
robot/helper_tdd_actor_list_no_db_update.py — uses MagicMock
robot/helper_tdd_actor_list_validation.py  — uses MagicMock
robot/helper_tdd_init_yes_no_input.py      — uses MagicMock
robot/helper_tdd_invariant_persistence.py  — uses MagicMock
robot/helper_tdd_plan_correct_auto_resolve.py — uses MagicMock
robot/helper_tdd_plan_correct_plan_id.py   — uses MagicMock
robot/helper_tdd_subplan_spawn_orchestration.py — uses MagicMock
robot/helper_tdd_validation_required_flag.py — uses MagicMock
robot/helper_tool_cli.py                   — uses MagicMock + patch() for service layer
robot/helper_tool_env_preferences.py       — uses MagicMock
robot/helper_wf04_multi_project_dependency.py — uses MagicMock

Specific Example: helper_session_cli.py

# robot/helper_session_cli.py (lines 13, 71-75)
from unittest.mock import MagicMock

def _setup_service() -> MagicMock:
    """Create and install a mock service."""
    svc = MagicMock()
    session_mod._service = svc  # Directly patches module-level service
    return svc

This replaces the real SessionService with a mock, meaning the integration test never exercises the actual session persistence, database writes, or service logic.

Specific Example: helper_tool_cli.py

# robot/helper_tool_cli.py (lines 92-97)
svc = MagicMock()
with patch(
    "cleveragents.cli.commands.tool._get_tool_service",
    return_value=svc,
):
    result = runner.invoke(tool_app, ["list"])

This patches the service factory, meaning the CLI command never exercises real tool registry, database, or service layer.

Impact

These tests provide false confidence — they pass even when the real service layer, database persistence, or dependency injection is broken. The integration test suite is supposed to catch regressions in the real service stack, but mocked tests cannot detect:

Database schema mismatches
Service layer bugs
Dependency injection failures
Real data serialization/deserialization errors
Transaction boundary issues

Steps to Reproduce

grep -rl "from unittest.mock\|import MagicMock\|MagicMock()" robot/helper_*.py | wc -l
# Returns: 58

Subtasks

Audit all 58 violating helper files and categorize the mock usage
For each file, replace mocked services with real service instances backed by in-memory SQLite (as done correctly in helper_wf14_server_mode.py and helper_m1_e2e_verification.py)
Where real service instantiation requires complex setup, use the setup_workspace() + run_cli() pattern from helper_e2e_common.py
Remove all from unittest.mock import ... statements from robot/ directory
Ensure all affected robot test suites still pass after the refactor
Run nox -s integration_tests to verify

Definition of Done

Zero files in robot/ import from unittest.mock
All integration tests exercise real service instances
nox -s integration_tests passes
Coverage >= 97%

Supporting Information

Rule source: CONTRIBUTING.md lines 568–570 and 1390
Correct pattern: robot/helper_e2e_common.py setup_workspace() + run_cli() — uses real CLI subprocess with real SQLite database
Correct pattern: robot/helper_wf14_server_mode.py — uses PlanLifecycleService(settings=Settings()) with real in-memory service
Correct pattern: robot/helper_m1_e2e_verification.py — uses real CLI subprocess without mocking

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Metadata - **Branch**: `fix/integration-test-mock-violations` - **Commit Message**: `fix(robot): remove unittest.mock usage from integration test helpers` - **Milestone**: (none — backlog) - **Parent Epic**: (Integration Testing Epic) ## Background and Context `CONTRIBUTING.md` (lines 568–570) states explicitly: > **Unit Tests Only:** Mocks, fakes, stubs, and test doubles are permitted only in unit tests. > Integration tests must exercise real services, real endpoints, and real dependencies — mocking > of any kind is strictly prohibited in integration tests. This rule is also repeated at line 1390: > ALL mocks, test doubles, and mock implementations must exist only in the `features/` directory. ## What Was Tested Systematic review of all 259 helper Python files in the `robot/` directory for use of `unittest.mock`, `MagicMock`, and `patch()`. ## Expected Behavior (from spec/CONTRIBUTING.md) All Robot Framework integration tests in `robot/` must exercise **real services and real dependencies**. No `unittest.mock`, `MagicMock`, `patch()`, or any other mocking mechanism is permitted in any file under `robot/`. ## Actual Behavior **58 out of 259 helper files (22%) in `robot/` use mocking**, directly violating the integration test no-mock rule. The complete list of violating files: ``` robot/helper_a2a_facade_wiring.py — uses MagicMock for service stubs robot/helper_action_cli_spec.py — uses MagicMock + patch() for service layer robot/helper_actor_add_rich_output.py — uses MagicMock + patch() for actor services robot/helper_actor_cli_show.py — uses MagicMock + patch() for actor registry robot/helper_actor_run_signature.py — uses MagicMock robot/helper_apply_pipeline.py — uses MagicMock for Plan objects robot/helper_automation_profile_cli.py — uses patch() robot/helper_cli_consistency.py — uses MagicMock + patch() robot/helper_cli_extensions.py — uses MagicMock + patch() robot/helper_cli_formats.py — uses MagicMock + patch() for service layer robot/helper_cli_lifecycle.py — uses MagicMock + patch() robot/helper_cli_lifecycle_e2e.py — uses MagicMock + patch() (named "e2e" but mocked) robot/helper_cloud_resources.py — uses MagicMock robot/helper_config_cli.py — uses MagicMock + patch() robot/helper_config_project_scope.py — uses MagicMock + patch() robot/helper_config_resolution.py — uses MagicMock + patch() robot/helper_correction_subtree_isolation.py — uses MagicMock robot/helper_decision_di.py — uses MagicMock robot/helper_error_recovery.py — uses MagicMock for Plan objects robot/helper_event_bus.py — uses MagicMock robot/helper_invariant_cli.py — uses MagicMock + patch() robot/helper_llm_trace.py — uses MagicMock robot/helper_m1_sourcecode_smoke.py — uses MagicMock + patch() robot/helper_m3_decision_validation_smoke.py — uses MagicMock + patch() robot/helper_m4_correction_subplan_smoke.py — uses MagicMock + patch() robot/helper_m4_e2e_cli.py — uses MagicMock + patch() (named "e2e") robot/helper_m4_e2e_cli_errors.py — uses MagicMock + patch() (named "e2e") robot/helper_m4_e2e_common.py — uses MagicMock (shared mock factories) robot/helper_m5_e2e_context.py — uses MagicMock + patch() (named "e2e") robot/helper_multi_project_subplan.py — uses MagicMock robot/helper_plan_cli_spec.py — uses MagicMock + patch() for service layer robot/helper_plan_correct_isolated_resolve.py — uses MagicMock robot/helper_plan_correct_tree_wiring.py — uses MagicMock robot/helper_plan_execute_runtime.py — (no mock import but uses InMemoryChangeSetStore) robot/helper_plan_use_env_priority.py — uses MagicMock + patch() robot/helper_project_context_cli.py — uses MagicMock + patch() robot/helper_project_context_set_exec_env_priority.py — uses MagicMock + patch() robot/helper_provider_registry.py — uses MagicMock robot/helper_repl_smoke.py — uses MagicMock robot/helper_resource_handlers.py — uses MagicMock robot/helper_revert_re_execution.py — uses MagicMock robot/helper_sandbox_checkpoint.py — uses MagicMock robot/helper_sandbox_integration.py — uses MagicMock robot/helper_session_cli.py — uses MagicMock (patches session_mod._service) robot/helper_skill_actor_run.py — uses MagicMock robot/helper_subplan_spawn.py — uses MagicMock robot/helper_subplan_spawn_orchestration.py — uses MagicMock robot/helper_tdd_actor_list_no_db_update.py — uses MagicMock robot/helper_tdd_actor_list_validation.py — uses MagicMock robot/helper_tdd_init_yes_no_input.py — uses MagicMock robot/helper_tdd_invariant_persistence.py — uses MagicMock robot/helper_tdd_plan_correct_auto_resolve.py — uses MagicMock robot/helper_tdd_plan_correct_plan_id.py — uses MagicMock robot/helper_tdd_subplan_spawn_orchestration.py — uses MagicMock robot/helper_tdd_validation_required_flag.py — uses MagicMock robot/helper_tool_cli.py — uses MagicMock + patch() for service layer robot/helper_tool_env_preferences.py — uses MagicMock robot/helper_wf04_multi_project_dependency.py — uses MagicMock ``` ## Specific Example: helper_session_cli.py ```python # robot/helper_session_cli.py (lines 13, 71-75) from unittest.mock import MagicMock def _setup_service() -> MagicMock: """Create and install a mock service.""" svc = MagicMock() session_mod._service = svc # Directly patches module-level service return svc ``` This replaces the real `SessionService` with a mock, meaning the integration test never exercises the actual session persistence, database writes, or service logic. ## Specific Example: helper_tool_cli.py ```python # robot/helper_tool_cli.py (lines 92-97) svc = MagicMock() with patch( "cleveragents.cli.commands.tool._get_tool_service", return_value=svc, ): result = runner.invoke(tool_app, ["list"]) ``` This patches the service factory, meaning the CLI command never exercises real tool registry, database, or service layer. ## Impact These tests provide **false confidence** — they pass even when the real service layer, database persistence, or dependency injection is broken. The integration test suite is supposed to catch regressions in the real service stack, but mocked tests cannot detect: - Database schema mismatches - Service layer bugs - Dependency injection failures - Real data serialization/deserialization errors - Transaction boundary issues ## Steps to Reproduce ```bash grep -rl "from unittest.mock\|import MagicMock\|MagicMock()" robot/helper_*.py | wc -l # Returns: 58 ``` ## Subtasks - [ ] Audit all 58 violating helper files and categorize the mock usage - [ ] For each file, replace mocked services with real service instances backed by in-memory SQLite (as done correctly in `helper_wf14_server_mode.py` and `helper_m1_e2e_verification.py`) - [ ] Where real service instantiation requires complex setup, use the `setup_workspace()` + `run_cli()` pattern from `helper_e2e_common.py` - [ ] Remove all `from unittest.mock import ...` statements from `robot/` directory - [ ] Ensure all affected robot test suites still pass after the refactor - [ ] Run `nox -s integration_tests` to verify ## Definition of Done - [ ] Zero files in `robot/` import from `unittest.mock` - [ ] All integration tests exercise real service instances - [ ] `nox -s integration_tests` passes - [ ] Coverage >= 97% ## Supporting Information - **Rule source**: `CONTRIBUTING.md` lines 568–570 and 1390 - **Correct pattern**: `robot/helper_e2e_common.py` `setup_workspace()` + `run_cli()` — uses real CLI subprocess with real SQLite database - **Correct pattern**: `robot/helper_wf14_server_mode.py` — uses `PlanLifecycleService(settings=Settings())` with real in-memory service - **Correct pattern**: `robot/helper_m1_e2e_verification.py` — uses real CLI subprocess without mocking --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo referenced this issue

2026-04-06 09:57:43 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 10:15:25 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 10:35:53 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775