task(testing): Tag all existing integration tests with @mocked/@llm annotations #686

Closed
opened 2026-03-11 18:11:03 +00:00 by freemo · 1 comment
Owner

Background and Context

Once the @mocked/@llm tagging system is implemented, all ~370 Behave feature files and ~235 Robot Framework test files need to be audited and tagged appropriately. This is a large but straightforward task.

Scope Assessment

Based on the current codebase analysis:

  • 370 feature files — nearly all use MockAIProvider implicitly (injected globally)
  • 215 step definition files use unittest.mock directly (39% of total)
  • 37 robot helpers use unittest.mock directly (20% of total)
  • 9 centralized mock files in features/mocks/

Tagging Strategy

  1. Most existing tests → @mocked_llm: Since MockAIProvider is injected globally, every test that generates changes via the AI provider is implicitly using a mock LLM
  2. Tests using unittest.mock for non-LLM purposes → @mocked: e.g., MockDevcontainerRunner, MockMCPTransport, MockLspTransport
  3. Tests that should use real LLM → @llm: Identify candidates for conversion (E2E verification suites, key integration flows)
  4. The single @mock_only feature → keep as-is, plus add @mocked and @mocked_llm

Acceptance Criteria

  • Every Behave feature file has at least one of: @mocked_llm, @mocked, or @llm
  • Every Robot Framework test suite has appropriate tags
  • Tags accurately reflect actual mock usage (no false labels)
  • @mocked is used when non-LLM mocks are in play
  • @mocked_llm is used when the LLM provider is mocked
  • A candidate list of tests for @llm conversion is documented
  • Full test suite passes with all tags applied

Metadata

  • Commit message: task(testing): tag all integration tests with @mocked/@llm annotations
  • Branch name: feature/m3-test-tagging-audit

Subtasks

  • Audit all 370 Behave feature files for mock usage
  • Audit all 235 Robot Framework test files
  • Apply @mocked_llm to tests using MockAIProvider
  • Apply @mocked to tests using other mocks
  • Identify and document candidate tests for @llm conversion
  • Verify full test suite passes

Definition of Done

  • All integration tests properly tagged
  • No untagged tests remain
  • Full CI suite passes
  • Candidate list for real-LLM conversion reviewed by team
## Background and Context Once the `@mocked/@llm` tagging system is implemented, all ~370 Behave feature files and ~235 Robot Framework test files need to be audited and tagged appropriately. This is a large but straightforward task. ### Scope Assessment Based on the current codebase analysis: - **370 feature files** — nearly all use `MockAIProvider` implicitly (injected globally) - **215 step definition files** use `unittest.mock` directly (39% of total) - **37 robot helpers** use `unittest.mock` directly (20% of total) - **9 centralized mock files** in `features/mocks/` ### Tagging Strategy 1. **Most existing tests → `@mocked_llm`**: Since `MockAIProvider` is injected globally, every test that generates changes via the AI provider is implicitly using a mock LLM 2. **Tests using `unittest.mock` for non-LLM purposes → `@mocked`**: e.g., `MockDevcontainerRunner`, `MockMCPTransport`, `MockLspTransport` 3. **Tests that should use real LLM → `@llm`**: Identify candidates for conversion (E2E verification suites, key integration flows) 4. **The single `@mock_only` feature → keep as-is**, plus add `@mocked` and `@mocked_llm` ## Acceptance Criteria - [ ] Every Behave feature file has at least one of: `@mocked_llm`, `@mocked`, or `@llm` - [ ] Every Robot Framework test suite has appropriate tags - [ ] Tags accurately reflect actual mock usage (no false labels) - [ ] `@mocked` is used when non-LLM mocks are in play - [ ] `@mocked_llm` is used when the LLM provider is mocked - [ ] A candidate list of tests for `@llm` conversion is documented - [ ] Full test suite passes with all tags applied ## Metadata - **Commit message**: `task(testing): tag all integration tests with @mocked/@llm annotations` - **Branch name**: `feature/m3-test-tagging-audit` ## Subtasks - [ ] Audit all 370 Behave feature files for mock usage - [ ] Audit all 235 Robot Framework test files - [ ] Apply `@mocked_llm` to tests using `MockAIProvider` - [ ] Apply `@mocked` to tests using other mocks - [ ] Identify and document candidate tests for `@llm` conversion - [ ] Verify full test suite passes ## Definition of Done - All integration tests properly tagged - No untagged tests remain - Full CI suite passes - Candidate list for real-LLM conversion reviewed by team
freemo added this to the v3.2.0 milestone 2026-03-11 18:13:19 +00:00
Author
Owner

Closing as superseded. The tagging approach has been abandoned. Instead, all mocking is now prohibited in integration tests. No tags needed — the rule is absolute: unit tests may mock, integration tests may not. Replaced by new issues covering complete mock removal from integration tests.

**Closing as superseded.** The tagging approach has been abandoned. Instead, all mocking is now prohibited in integration tests. No tags needed — the rule is absolute: unit tests may mock, integration tests may not. Replaced by new issues covering complete mock removal from integration tests.
freemo 2026-03-11 19:00:34 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#686
No description provided.