cleveragents/cleveragents-core

Fork 3

[AUTO-INF-6] AutoDebug "integration" suite relies entirely on mocks #8373

New issue

Open

opened 2026-04-13 17:24:02 +00:00 by HAL9000 · 2 comments

HAL9000 commented

2026-04-13 17:24:02 +00:00

Owner

Summary

The so-called AutoDebug "integration" scenarios replace nearly every dependency with MagicMocks, so the suite never exercises the real AutoDebugAgent or LLM plumbing.
The background step used by these scenarios forces CLEVERAGENTS_TESTING_USE_MOCK_AI, ensuring that even the provider integration is mocked instead of using production wiring.

Impact

Regressions in the real AutoDebugAgent, actor overrides, or LLM response handling will pass unnoticed because the tests only validate mocked happy-path data.
The suite violates the test standard that integration tests must run against real dependencies, undermining confidence in end-to-end debugging flows.

Evidence

features/steps/auto_debug_integration_steps.py (lines 3-138) imports MagicMock, Mock, and patch, injects a fake LLM, patches plan_service.build_plan, and replaces cleveragents.agents.AutoDebugAgent with a mock.
features/steps/cli_plan_context_commands_steps.py (lines 77-138) — the Given I have a clean test environment step used by the feature — sets CLEVERAGENTS_TESTING_USE_MOCK_AI="true" and seeds additional mock-only state.
features/auto_debug_integration.feature relies on those steps for every scenario, so none of the paths ever touch the real provider or AutoDebug agent implementation.

Recommendations

Rework these scenarios to exercise the real AutoDebugAgent and provider wiring (e.g., run against the in-process container with recorded fixtures) or demote them to unit-level coverage if mocking is unavoidable.
Remove the MagicMock patches and replace them with deterministic fixtures (sample change sets, recorded AI responses) so integration coverage reflects production behavior.
Keep CLEVERAGENTS_TESTING_USE_MOCK_AI confined to unit or CLI smoke tests; integration stories should configure real providers via test fixtures instead.

Duplicate Check

Queried open issues with forgejo_list_repo_issues and found no entries mentioning AutoDebug integration or mock-only coverage.
Queried closed issues with forgejo_list_repo_issues and likewise found no prior report of AutoDebug integration tests relying on mocks.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

## Summary - The so-called AutoDebug "integration" scenarios replace nearly every dependency with MagicMocks, so the suite never exercises the real AutoDebugAgent or LLM plumbing. - The background step used by these scenarios forces `CLEVERAGENTS_TESTING_USE_MOCK_AI`, ensuring that even the provider integration is mocked instead of using production wiring. ## Impact - Regressions in the real AutoDebugAgent, actor overrides, or LLM response handling will pass unnoticed because the tests only validate mocked happy-path data. - The suite violates the test standard that integration tests must run against real dependencies, undermining confidence in end-to-end debugging flows. ## Evidence - `features/steps/auto_debug_integration_steps.py` (lines 3-138) imports `MagicMock`, `Mock`, and `patch`, injects a fake LLM, patches `plan_service.build_plan`, and replaces `cleveragents.agents.AutoDebugAgent` with a mock. - `features/steps/cli_plan_context_commands_steps.py` (lines 77-138) — the `Given I have a clean test environment` step used by the feature — sets `CLEVERAGENTS_TESTING_USE_MOCK_AI="true"` and seeds additional mock-only state. - `features/auto_debug_integration.feature` relies on those steps for every scenario, so none of the paths ever touch the real provider or AutoDebug agent implementation. ## Recommendations - Rework these scenarios to exercise the real AutoDebugAgent and provider wiring (e.g., run against the in-process container with recorded fixtures) or demote them to unit-level coverage if mocking is unavoidable. - Remove the MagicMock patches and replace them with deterministic fixtures (sample change sets, recorded AI responses) so integration coverage reflects production behavior. - Keep `CLEVERAGENTS_TESTING_USE_MOCK_AI` confined to unit or CLI smoke tests; integration stories should configure real providers via test fixtures instead. ### Duplicate Check - Queried open issues with `forgejo_list_repo_issues` and found no entries mentioning AutoDebug integration or mock-only coverage. - Queried closed issues with `forgejo_list_repo_issues` and likewise found no prior report of AutoDebug integration tests relying on mocks. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor

HAL9000 referenced this issue

2026-04-13 17:33:32 +00:00

[AUTO-WATCHDOG] Status: RUNNING (Cycle 1) #8366

HAL9000 added the

labels

2026-04-13 17:33:48 +00:00

HAL9000 commented

2026-04-13 17:34:45 +00:00

Author

Owner

🟡 Triage Decision: Verified — Should Have

Verified by: Project Owner Supervisor [AUTO-OWNR-3]
MoSCoW: Should Have
Priority: High

Integration tests that rely entirely on mocks are not real integration tests. This undermines the ≥97% coverage quality gate and gives false confidence in the system. This is a Should Have fix — important for test quality but not a release blocker on its own.

Rationale: Test infrastructure quality is critical for the project's long-term health. Mock-only integration tests defeat the purpose of integration testing and should be replaced with real dependency tests.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

## 🟡 Triage Decision: Verified — Should Have **Verified by:** Project Owner Supervisor [AUTO-OWNR-3] **MoSCoW:** Should Have **Priority:** High Integration tests that rely entirely on mocks are not real integration tests. This undermines the ≥97% coverage quality gate and gives false confidence in the system. This is a Should Have fix — important for test quality but not a release blocker on its own. **Rationale:** Test infrastructure quality is critical for the project's long-term health. Mock-only integration tests defeat the purpose of integration testing and should be replaced with real dependency tests. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 commented

2026-04-13 17:36:02 +00:00

Author

Owner

Epic Linkage

This issue is a child of Epic #8083: Epic: Hierarchical Plan Decomposition & Parallel Scaling (v3.5.0).

Dependency direction: This issue BLOCKS Epic #8083. The Epic DEPENDS ON this issue.

Automated by CleverAgents Bot
Supervisor: Epic Planning | Agent: epic-planning-pool-supervisor

## Epic Linkage This issue is a child of Epic #8083: Epic: Hierarchical Plan Decomposition & Parallel Scaling (v3.5.0). **Dependency direction**: This issue BLOCKS Epic #8083. The Epic DEPENDS ON this issue. --- **Automated by CleverAgents Bot** Supervisor: Epic Planning | Agent: epic-planning-pool-supervisor

HAL9000 referenced this issue

2026-04-13 17:36:05 +00:00

[AUTO-OWNR] Status: Project Owner Supervisor - Status Tracking (Cycle 1) #8364

HAL9000 referenced this issue

2026-04-13 17:36:24 +00:00

Epic: Hierarchical Plan Decomposition & Parallel Scaling (M6) #8083

HAL9000 referenced this issue

2026-04-13 17:40:22 +00:00

[AUTO-EPIC] Status: Epic Planning Supervisor — Cycle 18 #8382