Tests (Robot): Add missing Robot Framework integration test for the providers module #2799

Open
opened 2026-04-04 20:02:58 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: test/providers-robot-integration
  • Commit Message: test(providers): add Robot Framework integration test suite for providers module
  • Milestone: v3.7.0
  • Parent Epic: #933

Background and Context

The src/cleveragents/providers/ module is the AI model provider layer — responsible for provider discovery, cost tracking, budget enforcement, fallback selection, and LLM adapter implementations. Per the project specification and CONTRIBUTING.md's Multi-Level Testing Mandate, every module must have tests at all required levels: Behave BDD unit tests, Robot Framework integration tests, and ASV performance benchmarks.

A scan of /app/robot/ reveals that while several narrow provider-related Robot files exist (google_provider.robot, openai_provider.robot, provider_detection_smoke.robot, provider_registry.robot, cost_controls.robot), no .robot file provides comprehensive integration coverage of the full providers module public API. In particular, the following are entirely absent from the Robot Framework integration test suite:

  • providers/llm/anthropic_provider.pyAnthropicChatProvider (no dedicated Robot test)
  • providers/llm/openrouter_provider.pyOpenRouterChatProvider (no dedicated Robot test)
  • providers/llm/langchain_chat_provider.pyLangChainChatProvider base class (no dedicated Robot test)
  • End-to-end provider selection → LLM invocation → cost tracking pipeline (no unified integration test)

The providers module exports the following public API (from __init__.py):

  • cost_table.pyCostEntry, ProviderCostTable
  • cost_tracker.pyBudgetCheckResult, BudgetStatus, CostTracker
  • fallback_selector.pyFallbackResult, FallbackSelector
  • registry.pyProviderRegistry, get_provider_registry, reset_provider_registry, resolve_provider_by_name
  • llm/anthropic_provider.pyAnthropicChatProvider
  • llm/openrouter_provider.pyOpenRouterChatProvider
  • llm/langchain_chat_provider.pyLangChainChatProvider
  • llm/openai_provider.pyOpenAIChatProvider
  • llm/google_provider.pyGoogleChatProvider

The absence of a unified integration test means there is no end-to-end validation of the provider selection → cost tracking → fallback pipeline under realistic conditions.

Subtasks

  • Create robot/providers_integration.robot with Robot Framework test cases covering the full providers module public API
  • Add integration test scenarios for AnthropicChatProvider — instantiation, generate_changes, streaming, retry behaviour
  • Add integration test scenarios for OpenRouterChatProvider — instantiation, base URL configuration, model resolution
  • Add integration test scenarios for LangChainChatProvider — base class lifecycle (start, stop, generate_changes, generate_changes_stream, get_execution_history)
  • Add integration test scenarios for FallbackSelector — capability filtering, budget-gated fallback chain, select() with require_tool_calls and require_streaming
  • Add integration test scenarios for ProviderRegistryget_default_provider, get_all_providers, resolve_provider_by_name, reset_provider_registry
  • Add integration test scenarios for the end-to-end pipeline: provider selection → CostTracker.record_usageBudgetCheckResult enforcement
  • Create robot/helper_providers_integration.py with the corresponding Python keyword library
  • Ensure all test cases use real services — no mocking permitted in integration tests (per CONTRIBUTING.md)
  • Run nox -e integration_tests locally and confirm all new test cases pass
  • Verify coverage >= 97% via nox -e coverage_report
  • Run nox (all default sessions) and fix any errors

Definition of Done

  • A robot/providers_integration.robot file exists and covers all public classes and functions listed above
  • AnthropicChatProvider, OpenRouterChatProvider, and LangChainChatProvider are exercised at the integration level
  • The end-to-end provider selection → cost tracking → fallback pipeline is validated
  • All Robot Framework scenarios pass under nox -e integration_tests with no failures or skips
  • No mock objects or test doubles are used in the Robot test file
  • nox -e lint passes with no new violations
  • nox -e typecheck passes with no new type errors
  • All nox stages pass
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: Unknown | Agent: ca-new-issue-creator

## Metadata - **Branch**: `test/providers-robot-integration` - **Commit Message**: `test(providers): add Robot Framework integration test suite for providers module` - **Milestone**: v3.7.0 - **Parent Epic**: #933 ## Background and Context The `src/cleveragents/providers/` module is the AI model provider layer — responsible for provider discovery, cost tracking, budget enforcement, fallback selection, and LLM adapter implementations. Per the project specification and `CONTRIBUTING.md`'s Multi-Level Testing Mandate, every module must have tests at all required levels: Behave BDD unit tests, Robot Framework integration tests, and ASV performance benchmarks. A scan of `/app/robot/` reveals that while several narrow provider-related Robot files exist (`google_provider.robot`, `openai_provider.robot`, `provider_detection_smoke.robot`, `provider_registry.robot`, `cost_controls.robot`), **no `.robot` file provides comprehensive integration coverage of the full `providers` module public API**. In particular, the following are entirely absent from the Robot Framework integration test suite: - `providers/llm/anthropic_provider.py` — `AnthropicChatProvider` (no dedicated Robot test) - `providers/llm/openrouter_provider.py` — `OpenRouterChatProvider` (no dedicated Robot test) - `providers/llm/langchain_chat_provider.py` — `LangChainChatProvider` base class (no dedicated Robot test) - End-to-end provider selection → LLM invocation → cost tracking pipeline (no unified integration test) The `providers` module exports the following public API (from `__init__.py`): - `cost_table.py` — `CostEntry`, `ProviderCostTable` - `cost_tracker.py` — `BudgetCheckResult`, `BudgetStatus`, `CostTracker` - `fallback_selector.py` — `FallbackResult`, `FallbackSelector` - `registry.py` — `ProviderRegistry`, `get_provider_registry`, `reset_provider_registry`, `resolve_provider_by_name` - `llm/anthropic_provider.py` — `AnthropicChatProvider` - `llm/openrouter_provider.py` — `OpenRouterChatProvider` - `llm/langchain_chat_provider.py` — `LangChainChatProvider` - `llm/openai_provider.py` — `OpenAIChatProvider` - `llm/google_provider.py` — `GoogleChatProvider` The absence of a unified integration test means there is no end-to-end validation of the provider selection → cost tracking → fallback pipeline under realistic conditions. ## Subtasks - [ ] Create `robot/providers_integration.robot` with Robot Framework test cases covering the full `providers` module public API - [ ] Add integration test scenarios for `AnthropicChatProvider` — instantiation, `generate_changes`, streaming, retry behaviour - [ ] Add integration test scenarios for `OpenRouterChatProvider` — instantiation, base URL configuration, model resolution - [ ] Add integration test scenarios for `LangChainChatProvider` — base class lifecycle (`start`, `stop`, `generate_changes`, `generate_changes_stream`, `get_execution_history`) - [ ] Add integration test scenarios for `FallbackSelector` — capability filtering, budget-gated fallback chain, `select()` with `require_tool_calls` and `require_streaming` - [ ] Add integration test scenarios for `ProviderRegistry` — `get_default_provider`, `get_all_providers`, `resolve_provider_by_name`, `reset_provider_registry` - [ ] Add integration test scenarios for the end-to-end pipeline: provider selection → `CostTracker.record_usage` → `BudgetCheckResult` enforcement - [ ] Create `robot/helper_providers_integration.py` with the corresponding Python keyword library - [ ] Ensure all test cases use real services — no mocking permitted in integration tests (per CONTRIBUTING.md) - [ ] Run `nox -e integration_tests` locally and confirm all new test cases pass - [ ] Verify coverage >= 97% via `nox -e coverage_report` - [ ] Run `nox` (all default sessions) and fix any errors ## Definition of Done - [ ] A `robot/providers_integration.robot` file exists and covers all public classes and functions listed above - [ ] `AnthropicChatProvider`, `OpenRouterChatProvider`, and `LangChainChatProvider` are exercised at the integration level - [ ] The end-to-end provider selection → cost tracking → fallback pipeline is validated - [ ] All Robot Framework scenarios pass under `nox -e integration_tests` with no failures or skips - [ ] No mock objects or test doubles are used in the Robot test file - [ ] `nox -e lint` passes with no new violations - [ ] `nox -e typecheck` passes with no new type errors - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: Unknown | Agent: ca-new-issue-creator
freemo added this to the v3.7.0 milestone 2026-04-04 20:03:07 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified | MoSCoW: Should Have — Missing Robot Framework integration test for the providers module.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified | **MoSCoW**: Should Have — Missing Robot Framework integration test for the providers module. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#2799
No description provided.