freemo
f44060e857
CI / lint (pull_request) Successful in 20s
CI / typecheck (pull_request) Successful in 51s
CI / quality (pull_request) Successful in 34s
CI / security (pull_request) Successful in 1m4s
CI / build (pull_request) Successful in 18s
CI / helm (pull_request) Successful in 22s
CI / unit_tests (pull_request) Successful in 6m54s
CI / e2e_tests (pull_request) Successful in 17m37s
CI / integration_tests (pull_request) Successful in 23m11s
CI / docker (pull_request) Successful in 1m32s
CI / coverage (pull_request) Successful in 10m54s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 56m53s
Previously, ProviderRegistry.create_ai_provider() dispatched to dedicated provider classes only for Google and OpenRouter. For OpenAI and Anthropic, execution fell through to the generic LangChainChatProvider factory, leaving OpenAIChatProvider and AnthropicChatProvider as dead code in production. This commit: - Adds ProviderType.OPENAI dispatch branch in create_ai_provider() to instantiate OpenAIChatProvider with the configured API key and model - Adds ProviderType.ANTHROPIC dispatch branch in create_ai_provider() to instantiate AnthropicChatProvider with the configured API key and model - Both branches raise ValueError with the missing env var name when the API key is not configured, consistent with Google and OpenRouter branches - Updates src/cleveragents/providers/llm/__init__.py to export all four dedicated provider classes (OpenAIChatProvider, AnthropicChatProvider, GoogleChatProvider, OpenRouterChatProvider) - Updates provider_registry_coverage.feature to assert that create_ai_provider returns OpenAIChatProvider for openai and AnthropicChatProvider for anthropic - Adds new scenarios for Anthropic dispatch and missing-key error paths for both OpenAI and Anthropic - Updates the 'AI provider exposes working llm factory' scenario to use groq (which still goes through the generic LangChainChatProvider path) since OpenAI now uses the dedicated class - Adds step definitions for OpenAIChatProvider and AnthropicChatProvider isinstance assertions Closes #3427
469 lines
21 KiB
Gherkin
469 lines
21 KiB
Gherkin
Feature: Provider Registry Coverage
|
|
As a developer
|
|
I want to ensure the ProviderRegistry class is thoroughly tested
|
|
So that provider discovery and selection work correctly
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create ProviderRegistry with default settings
|
|
Given I import the ProviderRegistry class
|
|
When I create a ProviderRegistry with default settings
|
|
Then the registry should be initialized
|
|
And the registry should have provider information
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderType enum values
|
|
Given I import the ProviderType enum
|
|
Then ProviderType should have OPENAI value "openai"
|
|
And ProviderType should have ANTHROPIC value "anthropic"
|
|
And ProviderType should have GOOGLE value "google"
|
|
And ProviderType should have AZURE value "azure"
|
|
And ProviderType should have OPENROUTER value "openrouter"
|
|
And ProviderType should have GEMINI value "gemini"
|
|
And ProviderType should have COHERE value "cohere"
|
|
And ProviderType should have GROQ value "groq"
|
|
And ProviderType should have TOGETHER value "together"
|
|
And ProviderType should have MOCK value "mock"
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderCapabilities with all fields
|
|
Given I import the ProviderCapabilities class
|
|
When I create a ProviderCapabilities with all fields
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 128000
|
|
And supports_json_mode should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderCapabilities with default values
|
|
Given I import the ProviderCapabilities class
|
|
When I create a ProviderCapabilities with default values
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be False
|
|
And supports_vision should be False
|
|
And max_context_length should be 4096
|
|
And supports_json_mode should be False
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderInfo dataclass
|
|
Given I import the ProviderInfo class
|
|
And I import the ProviderType enum
|
|
When I create a ProviderInfo for openai
|
|
Then the provider_type should be ProviderType.OPENAI
|
|
And the provider info name should be "Openai"
|
|
And the is_configured should be False by default
|
|
|
|
@unit @providers @registry
|
|
Scenario: Optional string helper coerces non-string values
|
|
Given I import the optional string coercion helper
|
|
When I coerce optional string with integer 42
|
|
Then the coerced optional string result should be "42"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get all providers from registry
|
|
Given I have a ProviderRegistry instance
|
|
When I call get_all_providers
|
|
Then the provider registry result should be a list of ProviderInfo
|
|
And the provider registry result should include core provider types
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get configured providers when none configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I call get_configured_providers
|
|
Then the provider registry result should be an empty list
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get configured providers with OpenAI key
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
When I call get_configured_providers
|
|
Then the provider registry result should contain OpenAI provider
|
|
And the provider registry OpenAI entry should be configured
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info by ProviderType
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for ProviderType.OPENAI
|
|
Then the provider registry result should be a ProviderInfo instance
|
|
And the provider_type should be ProviderType.OPENAI
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info by string
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for string "openai"
|
|
Then the provider registry result should be a ProviderInfo instance
|
|
And the provider_type should be ProviderType.OPENAI
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info by uppercase string
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for string "ANTHROPIC"
|
|
Then the provider registry result should be a ProviderInfo instance
|
|
And the provider_type should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info for invalid provider
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for string "invalid_provider"
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Check if provider is configured - not configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I check if openai is configured
|
|
Then the provider registry result should be False
|
|
|
|
@unit @providers @registry
|
|
Scenario: Check if provider is configured - configured
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
When I check if openai is configured
|
|
Then the provider registry result should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type with env var
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is set to "openai"
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.OPENAI
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type fallback order
|
|
Given I have a ProviderRegistry with Anthropic API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type when none configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type ignores invalid environment override
|
|
Given I have a ProviderRegistry with Anthropic API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is set to "not_a_real_provider"
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type skips unconfigured provider override
|
|
Given I have a ProviderRegistry with Anthropic API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is set to "openai"
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type honors settings default when valid
|
|
Given I have a ProviderRegistry with "anthropic" API key set and settings default provider "anthropic"
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type ignores invalid settings default
|
|
Given I have a ProviderRegistry with "anthropic" API key set and settings default provider "not_a_real_provider"
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model with env var
|
|
Given I have a ProviderRegistry instance
|
|
And CLEVERAGENTS_DEFAULT_MODEL is set to "gpt-4-turbo"
|
|
When I call get_default_model
|
|
Then the provider registry result should be "gpt-4-turbo"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model from settings configuration
|
|
Given I have a ProviderRegistry with default model "custom-model" configured
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I call get_default_model
|
|
Then the provider registry result should be "custom-model"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model for provider
|
|
Given I have a ProviderRegistry instance
|
|
When I call get_default_model for ProviderType.OPENAI
|
|
Then the provider registry result should be "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model for provider by string
|
|
Given I have a ProviderRegistry instance
|
|
When I call get_default_model for string "anthropic"
|
|
Then the provider registry result should be "claude-sonnet-4-20250514"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model for invalid provider
|
|
Given I have a ProviderRegistry instance
|
|
When I call get_default_model for string "invalid"
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model when no providers are configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I call get_default_model
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default capabilities for OpenAI
|
|
Given I have a ProviderRegistry instance
|
|
When I get capabilities for ProviderType.OPENAI
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 128000
|
|
And supports_json_mode should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default capabilities for Anthropic
|
|
Given I have a ProviderRegistry instance
|
|
When I get capabilities for ProviderType.ANTHROPIC
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 200000
|
|
And supports_json_mode should be False
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default capabilities for Google/Gemini
|
|
Given I have a ProviderRegistry instance
|
|
When I get capabilities for ProviderType.GOOGLE
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 1000000
|
|
And supports_json_mode should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: Global registry singleton
|
|
Given I import the get_provider_registry function
|
|
When I call get_provider_registry twice
|
|
Then both provider registry calls should return the same instance
|
|
|
|
@unit @providers @registry
|
|
Scenario: Reset global registry
|
|
Given I import the reset_provider_registry function
|
|
When I get the global registry and then reset it
|
|
Then resetting the provider registry should return a new instance
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM raises error when no provider configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an LLM without specifying a provider
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention configuring API keys
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM raises error for invalid provider string
|
|
Given I have a ProviderRegistry instance
|
|
When I try to create an LLM for provider "not_a_provider"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention unknown provider type
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM raises error when provider API key is missing
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an LLM for provider "openai"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention configuring API keys
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM delegates to provider factory for configured string input
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm for provider "openai" and model "gpt-mini"
|
|
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-mini"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM without provider uses fallback default
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm without specifying a provider
|
|
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM accepts ProviderType argument
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm for ProviderType.OPENAI
|
|
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM for mock provider skips API key requirement
|
|
Given I have a ProviderRegistry with no API keys
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm for provider "mock" without specifying a model
|
|
Then the stubbed LLM factory should be called for ProviderType.MOCK with model "mock-gpt"
|
|
|
|
@unit @providers @registry
|
|
Scenario: _create_provider_llm builds stubbed OpenAI client
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And langchain OpenAI client is stubbed
|
|
When I call _create_provider_llm for provider "openai" with model "custom-openai"
|
|
Then the stubbed ChatOpenAI should be created with model "custom-openai"
|
|
|
|
@unit @providers @registry
|
|
Scenario Outline: _create_provider_llm builds <ProviderName> client
|
|
Given I have a ProviderRegistry with "<provider_key>" API key set
|
|
And the "<ProviderName>" LangChain client is stubbed
|
|
When I call _create_provider_llm for provider "<provider_key>" with model "<model>"
|
|
Then the stubbed <ProviderName> client should be created with argument "model" value "<model>"
|
|
|
|
Examples:
|
|
| ProviderName | provider_key | model |
|
|
| Anthropic | anthropic | claude-mini-cover |
|
|
| Google | google | gemini-cover |
|
|
| Gemini | gemini | gemini-pro-cover |
|
|
| Groq | groq | llama-cover |
|
|
| Together | together | llama-together-cover |
|
|
| Cohere | cohere | command-cover |
|
|
|
|
@unit @providers @registry
|
|
Scenario: _create_provider_llm builds Azure client with deployment override
|
|
Given I have a ProviderRegistry with "azure" API key set
|
|
And the "Azure" LangChain client is stubbed
|
|
When I call _create_provider_llm for provider "azure" with model "azure-cover"
|
|
Then the stubbed Azure client should be created with argument "deployment_name" value "azure-cover"
|
|
|
|
@unit @providers @registry
|
|
Scenario: _create_provider_llm raises when Azure endpoint is missing
|
|
Given I have a ProviderRegistry with "azure" API key set
|
|
And the Azure endpoint setting is cleared
|
|
When I try to call _create_provider_llm for provider "azure" with model "azure-cover"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention missing Azure endpoint
|
|
|
|
|
|
@unit @providers @registry
|
|
Scenario: _create_provider_llm raises error for unsupported provider
|
|
Given I have a ProviderRegistry instance
|
|
When I try to call _create_provider_llm for provider "mock" with model "mock-gpt"
|
|
Then a provider registry ValueError should be raised
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error when no provider configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an AI provider without specifying a provider
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention configuring API keys
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error for invalid provider string
|
|
Given I have a ProviderRegistry instance
|
|
When I try to create an AI provider for provider "not_a_provider"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention unknown provider type
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error when Google API key is missing
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an AI provider for provider "google"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention missing "GOOGLE_API_KEY" environment variable
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider succeeds for configured provider string
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "openai"
|
|
Then the provider registry AI provider should be an OpenAIChatProvider
|
|
And the provider registry AI provider name should be "openai"
|
|
And the provider registry AI provider model_id should be "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider returns Anthropic adapter
|
|
Given I have a ProviderRegistry with Anthropic API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "anthropic"
|
|
Then the provider registry AI provider should be an AnthropicChatProvider
|
|
And the provider registry AI provider name should be "anthropic"
|
|
And the provider registry AI provider model_id should be "claude-sonnet-4-20250514"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error when OpenAI API key is missing
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an AI provider for provider "openai"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention missing "OPENAI_API_KEY" environment variable
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error when Anthropic API key is missing
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an AI provider for provider "anthropic"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention missing "ANTHROPIC_API_KEY" environment variable
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider returns Google adapter
|
|
Given I have a ProviderRegistry with "google" API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "google"
|
|
Then the provider registry AI provider should be a GoogleChatProvider
|
|
And the provider registry AI provider name should be "google"
|
|
And the provider registry AI provider model_id should be "gemini-2.0-flash"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider returns OpenRouter adapter
|
|
Given I have a ProviderRegistry with "openrouter" API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "openrouter"
|
|
Then the provider registry AI provider should be an OpenRouterChatProvider
|
|
And the provider registry AI provider name should be "openrouter"
|
|
And the provider registry AI provider model_id should be "anthropic/claude-sonnet-4-20250514"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider without specifying provider uses fallback
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider without specifying a provider
|
|
Then the provider registry AI provider should be an OpenAIChatProvider
|
|
And the provider registry AI provider name should be "openai"
|
|
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider honors explicit model overrides
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider specifying provider "openai" and model "custom-model"
|
|
Then the provider registry AI provider model_id should be "custom-model"
|
|
|
|
@unit @providers @registry
|
|
Scenario: AI provider exposes working llm factory
|
|
Given I have a ProviderRegistry with "groq" API key set
|
|
And the provider registry LLM factory is stubbed
|
|
When I create an AI provider for provider "groq"
|
|
And I call the AI provider llm factory with model "llama-mini-cover"
|
|
Then the stubbed LLM factory should be called for ProviderType.GROQ with model "llama-mini-cover"
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderCapabilities is immutable
|
|
Given I import the ProviderCapabilities class
|
|
When I create a ProviderCapabilities instance
|
|
Then the instance should be frozen
|
|
And attempting to modify it should raise an error
|
|
|
|
@unit @providers @registry
|
|
Scenario: Provider fallback order
|
|
Given I have the ProviderRegistry class
|
|
When I check the FALLBACK_ORDER
|
|
Then OpenAI should be first in the order
|
|
And Anthropic should be second in the order
|
|
And Google should be third in the order
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default models for all providers
|
|
Given I have the ProviderRegistry class
|
|
When I check the DEFAULT_MODELS
|
|
Then OpenAI default should be "gpt-4o"
|
|
And Anthropic default should be "claude-sonnet-4-20250514"
|
|
And Google default should be "gemini-2.0-flash"
|
|
And Groq default should be "llama-3.1-70b-versatile"
|