UAT: `ProviderRegistry.create_llm('mock')` always raises `ValueError: Unsupported provider type: mock` — mock LLM creation is broken despite `MOCK` being a registered `ProviderType` #4118

New issue

Open

opened 2026-04-06 10:28:25 +00:00 by freemo · 1 comment

freemo commented

2026-04-06 10:28:25 +00:00

Owner

Metadata

Branch: fix/providers-mock-llm-creation
Commit Message: fix(providers): support mock provider in _create_provider_llm for testing
Milestone: (none — backlog)
Parent Epic: #3365 (Epic: Additional LLM Provider Integrations — Cohere, Groq, Together AI, and Provider Abstraction)

Backlog note: This issue was discovered during autonomous UAT testing
on the LLM Provider Integration feature area. It does not block milestone
completion and has been placed in the backlog for human review and future
milestone assignment.

Bug Report

What was tested: ProviderRegistry.create_llm() with mock provider type

Expected behavior:

ProviderType.MOCK is a registered provider type in ProviderRegistry. The create_llm() method correctly skips the API key check for mock (since MOCK is not in PROVIDER_KEY_ATTRS). Therefore, calling create_llm("mock") should succeed and return a mock LLM instance suitable for testing.

The existing Behave scenario confirms this expectation:

Scenario: Create LLM for mock provider skips API key requirement
    Given I have a ProviderRegistry with no API keys
    When I call create_llm for provider "mock" without specifying a model
    Then the stubbed LLM factory should be called for ProviderType.MOCK with model "mock-gpt"

Actual behavior (from code analysis):

_create_provider_llm() in src/cleveragents/providers/registry.py handles all provider types EXCEPT ProviderType.MOCK. The method falls through all if branches and reaches:

raise ValueError(f"Unsupported provider type: {provider_type}")

This means create_llm("mock") always raises ValueError: Unsupported provider type: mock, even though:

ProviderType.MOCK is a valid enum value
create_llm() correctly skips the API key check for mock
The Behave test only passes because it stubs _create_provider_llm to avoid the ValueError

Code location:

src/cleveragents/providers/registry.py — _create_provider_llm() method (lines 460-552): no ProviderType.MOCK case
src/cleveragents/providers/registry.py — create_llm() method: correctly skips API key check for mock (MOCK not in PROVIDER_KEY_ATTRS)

Steps to reproduce:

from cleveragents.providers.registry import ProviderRegistry
registry = ProviderRegistry()
llm = registry.create_llm("mock")  # Raises ValueError: Unsupported provider type: mock

Severity: Medium — mock LLM creation is broken. Tests that rely on create_llm("mock") without stubbing _create_provider_llm will fail. This also means the MOCK provider type is effectively unusable via the create_llm API.

Subtasks

Write failing Behave scenario that calls create_llm("mock") without stubbing _create_provider_llm
Add ProviderType.MOCK case to _create_provider_llm() that returns a FakeListLLM or similar test LLM
Ensure mock LLM creation only works when mock_providers=True in settings (to prevent accidental use in production)
Verify nox -e typecheck passes
Run nox (all default sessions)

Definition of Done

ProviderRegistry.create_llm("mock") succeeds without raising ValueError
Mock LLM creation is guarded to prevent production use
All existing provider tests continue to pass
New Behave scenario covers mock LLM creation without stubbing
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/providers-mock-llm-creation` - **Commit Message**: `fix(providers): support mock provider in _create_provider_llm for testing` - **Milestone**: *(none — backlog)* - **Parent Epic**: #3365 (Epic: Additional LLM Provider Integrations — Cohere, Groq, Together AI, and Provider Abstraction) > **Backlog note:** This issue was discovered during autonomous UAT testing > on the LLM Provider Integration feature area. It does not block milestone > completion and has been placed in the backlog for human review and future > milestone assignment. ## Bug Report **What was tested:** `ProviderRegistry.create_llm()` with mock provider type **Expected behavior:** `ProviderType.MOCK` is a registered provider type in `ProviderRegistry`. The `create_llm()` method correctly skips the API key check for mock (since `MOCK` is not in `PROVIDER_KEY_ATTRS`). Therefore, calling `create_llm("mock")` should succeed and return a mock LLM instance suitable for testing. The existing Behave scenario confirms this expectation: ``` Scenario: Create LLM for mock provider skips API key requirement Given I have a ProviderRegistry with no API keys When I call create_llm for provider "mock" without specifying a model Then the stubbed LLM factory should be called for ProviderType.MOCK with model "mock-gpt" ``` **Actual behavior (from code analysis):** `_create_provider_llm()` in `src/cleveragents/providers/registry.py` handles all provider types EXCEPT `ProviderType.MOCK`. The method falls through all `if` branches and reaches: ```python raise ValueError(f"Unsupported provider type: {provider_type}") ``` This means `create_llm("mock")` always raises `ValueError: Unsupported provider type: mock`, even though: 1. `ProviderType.MOCK` is a valid enum value 2. `create_llm()` correctly skips the API key check for mock 3. The Behave test only passes because it stubs `_create_provider_llm` to avoid the ValueError **Code location:** - `src/cleveragents/providers/registry.py` — `_create_provider_llm()` method (lines 460-552): no `ProviderType.MOCK` case - `src/cleveragents/providers/registry.py` — `create_llm()` method: correctly skips API key check for mock (MOCK not in `PROVIDER_KEY_ATTRS`) **Steps to reproduce:** ```python from cleveragents.providers.registry import ProviderRegistry registry = ProviderRegistry() llm = registry.create_llm("mock") # Raises ValueError: Unsupported provider type: mock ``` **Severity:** Medium — mock LLM creation is broken. Tests that rely on `create_llm("mock")` without stubbing `_create_provider_llm` will fail. This also means the `MOCK` provider type is effectively unusable via the `create_llm` API. ## Subtasks - [ ] Write failing Behave scenario that calls `create_llm("mock")` without stubbing `_create_provider_llm` - [ ] Add `ProviderType.MOCK` case to `_create_provider_llm()` that returns a `FakeListLLM` or similar test LLM - [ ] Ensure mock LLM creation only works when `mock_providers=True` in settings (to prevent accidental use in production) - [ ] Verify `nox -e typecheck` passes - [ ] Run `nox` (all default sessions) ## Definition of Done - [ ] `ProviderRegistry.create_llm("mock")` succeeds without raising `ValueError` - [ ] Mock LLM creation is guarded to prevent production use - [ ] All existing provider tests continue to pass - [ ] New Behave scenario covers mock LLM creation without stubbing - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added a new dependency

2026-04-06 10:28:32 +00:00

#3365 Epic: Additional LLM Provider Integrations — Cohere, Groq, Together AI, and Provider Abstraction

freemo referenced this issue

2026-04-06 10:32:27 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 10:35:53 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo added this to the v3.2.0 milestone

2026-04-06 18:06:55 +00:00

freemo commented

2026-04-06 18:50:42 +00:00

Author

Owner

Milestone Triage Decision: Moved to Backlog

This CLI enhancement issue has been moved out of v3.3.0 during aggressive milestone triage. While useful for user experience, it does not relate to the core focus of Corrections + Subplans + Checkpoints.

Reasoning:

v3.3.0 focus: Essential corrections, subplan management, and checkpoint functionality
This issue: CLI validation enhancement - user experience improvement
Impact: UX enhancement, not core corrections/subplans/checkpoints functionality

Will be addressed in a future milestone focused on CLI polish and user experience enhancements.

**Milestone Triage Decision: Moved to Backlog** This CLI enhancement issue has been moved out of v3.3.0 during aggressive milestone triage. While useful for user experience, it does not relate to the core focus of Corrections + Subplans + Checkpoints. **Reasoning:** - v3.3.0 focus: Essential corrections, subplan management, and checkpoint functionality - This issue: CLI validation enhancement - user experience improvement - Impact: UX enhancement, not core corrections/subplans/checkpoints functionality Will be addressed in a future milestone focused on CLI polish and user experience enhancements.

freemo removed this from the v3.2.0 milestone

2026-04-06 20:42:57 +00:00