TDD: Write failing test demonstrating E2E mock-only coverage (bug #658) #697

Closed
opened 2026-03-11 18:32:39 +00:00 by freemo · 1 comment
Owner

Summary

TDD counterpart for bug #658. Write a failing test that demonstrates M1-M6 E2E verification suites are using mocked services instead of exercising real system behavior.

Acceptance Criteria

  • Write a test that asserts at least one E2E suite test case exercises a real (non-mocked) code path
  • Test MUST fail with current implementation (validates the bug exists)
  • Test tagged with @tdd_expected_fail per CONTRIBUTING.md TDD workflow
  • Test tagged with @llm or @mocked per integration test tag conventions

Depends On

  • #684 — integration test tagging system must exist first

Blocks

  • #658 — this TDD test must pass before #658 can be closed

References

  • CONTRIBUTING.md: TDD Workflow for Bug Fixes
  • CONTRIBUTING.md: Integration Test Tags
## Summary TDD counterpart for bug #658. Write a failing test that demonstrates M1-M6 E2E verification suites are using mocked services instead of exercising real system behavior. ## Acceptance Criteria - [ ] Write a test that asserts at least one E2E suite test case exercises a real (non-mocked) code path - [ ] Test MUST fail with current implementation (validates the bug exists) - [ ] Test tagged with `@tdd_expected_fail` per CONTRIBUTING.md TDD workflow - [ ] Test tagged with `@llm` or `@mocked` per integration test tag conventions ## Depends On - #684 — integration test tagging system must exist first ## Blocks - #658 — this TDD test must pass before #658 can be closed ## References - CONTRIBUTING.md: TDD Workflow for Bug Fixes - CONTRIBUTING.md: Integration Test Tags
freemo added this to the v3.5.0 milestone 2026-03-11 18:32:47 +00:00
Member

PR #738 implements this TDD test.

The tests use AST analysis to inspect robot/helper_m*_e2e_verification.py and verify that all 21 CLI-facing test functions use unittest.mock.patch + CliRunner instead of real subprocess invocations. Three scenarios confirm:

  1. 0/21 CLI-facing tests use subprocess.run (all use CliRunner)
  2. 0/21 CLI-facing tests exercise the real service layer (all mock via patch)
  3. 5/5 suites with CLI tests (M1-M4, M6) have 100% mocked tests

All tagged @tdd_expected_fail @tdd_bug @tdd_bug_658 per CONTRIBUTING.md workflow. Quality gates pass: lint, typecheck (0 errors), unit_tests (375 features, 0 failures), coverage (98%), security_scan.

PR #738 implements this TDD test. The tests use AST analysis to inspect `robot/helper_m*_e2e_verification.py` and verify that all 21 CLI-facing test functions use `unittest.mock.patch` + `CliRunner` instead of real subprocess invocations. Three scenarios confirm: 1. 0/21 CLI-facing tests use `subprocess.run` (all use `CliRunner`) 2. 0/21 CLI-facing tests exercise the real service layer (all mock via `patch`) 3. 5/5 suites with CLI tests (M1-M4, M6) have 100% mocked tests All tagged `@tdd_expected_fail @tdd_bug @tdd_bug_658` per CONTRIBUTING.md workflow. Quality gates pass: lint, typecheck (0 errors), unit_tests (375 features, 0 failures), coverage (98%), security_scan.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#697
No description provided.