test(integration): implement mocked LLM provider fixture and shared integration test infrastructure #8989

Open
opened 2026-04-14 04:35:30 +00:00 by HAL9000 · 1 comment
Owner

Background and Context

Epic #401 requires milestone acceptance and workflow example integration tests to use integration-appropriate mocking (e.g., mocked LLM providers). Before individual test suites can be written, a shared mocked LLM provider fixture and integration test infrastructure must exist. Without this foundation, every integration test suite would need to independently implement LLM mocking, leading to duplication, inconsistency, and maintenance burden. This issue establishes the shared infrastructure that all other Epic #401 child issues depend on.

Expected Behavior

When this issue is complete:

  • A mocked LLM provider fixture exists that intercepts LLM API calls and returns deterministic, configurable responses
  • A shared Robot Framework keywords library (tests/integration/resources/integration_keywords.resource) provides reusable keywords for CLI invocation with the mocked LLM
  • The mocked LLM provider is configurable per test case (response content, latency simulation, error injection)
  • The fixture integrates cleanly with the nox -s integration_tests session
  • Documentation explains how to use the mocked LLM fixture in new integration test suites

Acceptance Criteria

  • A mocked LLM provider fixture exists (e.g., tests/integration/fixtures/mock_llm_provider.py) that intercepts Anthropic/OpenAI API calls
  • The fixture supports configurable deterministic responses (response text, token counts, finish reason)
  • The fixture supports error injection (rate limit errors, API errors, timeout simulation)
  • A shared Robot Framework keywords library (tests/integration/resources/integration_keywords.resource) exists with keywords for CLI invocation using the mocked LLM
  • The mocked LLM fixture integrates with nox -s integration_tests without requiring real API keys
  • A tests/integration/README.md documents how to use the mocked LLM fixture and shared keywords
  • At least one smoke integration test using the mocked LLM fixture passes under nox -s integration_tests

Subtasks

  • Design and implement tests/integration/fixtures/mock_llm_provider.py with configurable response support
  • Add error injection support (rate limit, API error, timeout) to the mock LLM provider
  • Create tests/integration/resources/integration_keywords.resource with CLI invocation keywords
  • Add Setup Mock LLM Provider and Teardown Mock LLM Provider Robot Framework keywords
  • Write a smoke integration test (tests/integration/smoke/smoke.robot) using the mocked LLM
  • Run smoke test under nox -s integration_tests and verify it passes
  • Write tests/integration/README.md documenting the fixture and keywords
  • Verify coverage >= 97% via nox -s coverage_report

Definition of Done

  • All acceptance criteria met
  • Tests written and passing (coverage >= 97%)
  • Code reviewed and approved
  • Documentation updated if needed
  • No regressions introduced

Metadata

  • Commit message: test(integration): implement mocked LLM provider fixture and shared integration test infrastructure
  • Branch name: test/integration-mocked-llm-fixture-infrastructure

Automated by CleverAgents Bot
Supervisor: Epic Planning Pool | Agent: epic-planning-pool-supervisor

## Background and Context Epic #401 requires milestone acceptance and workflow example integration tests to use integration-appropriate mocking (e.g., mocked LLM providers). Before individual test suites can be written, a shared mocked LLM provider fixture and integration test infrastructure must exist. Without this foundation, every integration test suite would need to independently implement LLM mocking, leading to duplication, inconsistency, and maintenance burden. This issue establishes the shared infrastructure that all other Epic #401 child issues depend on. ## Expected Behavior When this issue is complete: - A mocked LLM provider fixture exists that intercepts LLM API calls and returns deterministic, configurable responses - A shared Robot Framework keywords library (`tests/integration/resources/integration_keywords.resource`) provides reusable keywords for CLI invocation with the mocked LLM - The mocked LLM provider is configurable per test case (response content, latency simulation, error injection) - The fixture integrates cleanly with the `nox -s integration_tests` session - Documentation explains how to use the mocked LLM fixture in new integration test suites ## Acceptance Criteria - [ ] A mocked LLM provider fixture exists (e.g., `tests/integration/fixtures/mock_llm_provider.py`) that intercepts Anthropic/OpenAI API calls - [ ] The fixture supports configurable deterministic responses (response text, token counts, finish reason) - [ ] The fixture supports error injection (rate limit errors, API errors, timeout simulation) - [ ] A shared Robot Framework keywords library (`tests/integration/resources/integration_keywords.resource`) exists with keywords for CLI invocation using the mocked LLM - [ ] The mocked LLM fixture integrates with `nox -s integration_tests` without requiring real API keys - [ ] A `tests/integration/README.md` documents how to use the mocked LLM fixture and shared keywords - [ ] At least one smoke integration test using the mocked LLM fixture passes under `nox -s integration_tests` ## Subtasks - [ ] Design and implement `tests/integration/fixtures/mock_llm_provider.py` with configurable response support - [ ] Add error injection support (rate limit, API error, timeout) to the mock LLM provider - [ ] Create `tests/integration/resources/integration_keywords.resource` with CLI invocation keywords - [ ] Add `Setup Mock LLM Provider` and `Teardown Mock LLM Provider` Robot Framework keywords - [ ] Write a smoke integration test (`tests/integration/smoke/smoke.robot`) using the mocked LLM - [ ] Run smoke test under `nox -s integration_tests` and verify it passes - [ ] Write `tests/integration/README.md` documenting the fixture and keywords - [ ] Verify coverage >= 97% via `nox -s coverage_report` ## Definition of Done - [ ] All acceptance criteria met - [ ] Tests written and passing (coverage >= 97%) - [ ] Code reviewed and approved - [ ] Documentation updated if needed - [ ] No regressions introduced ## Metadata - **Commit message:** `test(integration): implement mocked LLM provider fixture and shared integration test infrastructure` - **Branch name:** `test/integration-mocked-llm-fixture-infrastructure` --- **Automated by CleverAgents Bot** Supervisor: Epic Planning Pool | Agent: epic-planning-pool-supervisor
HAL9000 added this to the v3.2.0 milestone 2026-04-14 04:45:27 +00:00
Author
Owner

Verified — Mocked LLM provider fixture is foundational test infrastructure for v3.2.0. MoSCoW: Must-have. Priority: Medium.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — Mocked LLM provider fixture is foundational test infrastructure for v3.2.0. MoSCoW: Must-have. Priority: Medium. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#8989
No description provided.