Feature: Provider Registry Coverage As a developer I want to ensure the ProviderRegistry class is thoroughly tested So that provider discovery and selection work correctly @unit @providers @registry Scenario: Create ProviderRegistry with default settings Given I import the ProviderRegistry class When I create a ProviderRegistry with default settings Then the registry should be initialized And the registry should have provider information @unit @providers @registry Scenario: ProviderType enum values Given I import the ProviderType enum Then ProviderType should have OPENAI value "openai" And ProviderType should have ANTHROPIC value "anthropic" And ProviderType should have GOOGLE value "google" And ProviderType should have AZURE value "azure" And ProviderType should have OPENROUTER value "openrouter" And ProviderType should have GEMINI value "gemini" And ProviderType should have COHERE value "cohere" And ProviderType should have GROQ value "groq" And ProviderType should have TOGETHER value "together" And ProviderType should have MOCK value "mock" @unit @providers @registry Scenario: ProviderCapabilities with all fields Given I import the ProviderCapabilities class When I create a ProviderCapabilities with all fields Then supports_streaming should be True And supports_tool_calls should be True And supports_vision should be True And max_context_length should be 128000 And supports_json_mode should be True @unit @providers @registry Scenario: ProviderCapabilities with default values Given I import the ProviderCapabilities class When I create a ProviderCapabilities with default values Then supports_streaming should be True And supports_tool_calls should be False And supports_vision should be False And max_context_length should be 4096 And supports_json_mode should be False @unit @providers @registry Scenario: ProviderInfo dataclass Given I import the ProviderInfo class And I import the ProviderType enum When I create a ProviderInfo for openai Then the provider_type should be ProviderType.OPENAI And the provider info name should be "Openai" And the is_configured should be False by default @unit @providers @registry Scenario: Get configured providers when none configured and mock not allowed Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set And I have a ProviderRegistry with no API keys When I call get_configured_providers Then the provider registry result should be an empty list @unit @providers @registry Scenario: Get configured providers when none configured Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set And I have a ProviderRegistry with no API keys When I call get_configured_providers Then the provider registry result should contain only the MOCK provider @unit @providers @registry Scenario: Get configured providers with OpenAI key Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set And I have a ProviderRegistry with OpenAI API key set When I call get_configured_providers Then the provider registry result should contain OpenAI provider And the provider registry result should contain the MOCK provider And the provider registry OpenAI entry should be configured @unit @providers @registry Scenario: Get provider info by ProviderType Given I have a ProviderRegistry instance When I get provider info for ProviderType.OPENAI Then the provider registry result should be a ProviderInfo instance And the provider_type should be ProviderType.OPENAI @unit @providers @registry Scenario: Get provider info by string Given I have a ProviderRegistry instance When I get provider info for string "openai" Then the provider registry result should be a ProviderInfo instance And the provider_type should be ProviderType.OPENAI @unit @providers @registry Scenario: Get provider info by uppercase string Given I have a ProviderRegistry instance When I get provider info for string "ANTHROPIC" Then the provider registry result should be a ProviderInfo instance And the provider_type should be ProviderType.ANTHROPIC @unit @providers @registry Scenario: Get provider info for invalid provider Given I have a ProviderRegistry instance When I get provider info for string "invalid_provider" Then the provider registry result should be None @unit @providers @registry Scenario: Check if provider is configured - not configured Given I have a ProviderRegistry with no API keys When I check if openai is configured Then the provider registry result should be False @unit @providers @registry Scenario: Check if provider is configured - configured Given I have a ProviderRegistry with OpenAI API key set When I check if openai is configured Then the provider registry result should be True @unit @providers @registry Scenario: Get default provider type with env var Given I have a ProviderRegistry with OpenAI API key set And CLEVERAGENTS_DEFAULT_PROVIDER is set to "openai" When I call get_default_provider_type Then the provider registry result should be ProviderType.OPENAI @unit @providers @registry Scenario: Get default provider type fallback order Given I have a ProviderRegistry with Anthropic API key set And CLEVERAGENTS_DEFAULT_PROVIDER is not set When I call get_default_provider_type Then the provider registry result should be ProviderType.ANTHROPIC @unit @providers @registry Scenario: Get default provider type when none configured Given I have a ProviderRegistry with no API keys When I call get_default_provider_type Then the provider registry result should be None @unit @providers @registry Scenario: Get default provider type ignores invalid environment override Given I have a ProviderRegistry with Anthropic API key set And CLEVERAGENTS_DEFAULT_PROVIDER is set to "not_a_real_provider" When I call get_default_provider_type Then the provider registry result should be ProviderType.ANTHROPIC @unit @providers @registry Scenario: Get default provider type skips unconfigured provider override Given I have a ProviderRegistry with Anthropic API key set And CLEVERAGENTS_DEFAULT_PROVIDER is set to "openai" When I call get_default_provider_type Then the provider registry result should be ProviderType.ANTHROPIC @unit @providers @registry Scenario: Get default provider type honors settings default when valid Given I have a ProviderRegistry with "anthropic" API key set and settings default provider "anthropic" And CLEVERAGENTS_DEFAULT_PROVIDER is not set When I call get_default_provider_type Then the provider registry result should be ProviderType.ANTHROPIC @unit @providers @registry Scenario: Get default provider type ignores invalid settings default Given I have a ProviderRegistry with "anthropic" API key set and settings default provider "not_a_real_provider" And CLEVERAGENTS_DEFAULT_PROVIDER is not set When I call get_default_provider_type Then the provider registry result should be ProviderType.ANTHROPIC @unit @providers @registry Scenario: Get default model with env var Given I have a ProviderRegistry instance And CLEVERAGENTS_DEFAULT_MODEL is set to "gpt-4-turbo" When I call get_default_model Then the provider registry result should be "gpt-4-turbo" @unit @providers @registry Scenario: Get default model from settings configuration Given I have a ProviderRegistry with default model "custom-model" configured And CLEVERAGENTS_DEFAULT_MODEL is not set When I call get_default_model Then the provider registry result should be "custom-model" @unit @providers @registry Scenario: Get default model for provider Given I have a ProviderRegistry instance When I call get_default_model for ProviderType.OPENAI Then the provider registry result should be "gpt-4o" @unit @providers @registry Scenario: Get default model for provider by string Given I have a ProviderRegistry instance When I call get_default_model for string "anthropic" Then the provider registry result should be "claude-sonnet-4-20250514" @unit @providers @registry Scenario: Get default model for invalid provider Given I have a ProviderRegistry instance When I call get_default_model for string "invalid" Then the provider registry result should be None @unit @providers @registry Scenario: Get default model when no providers are configured Given I have a ProviderRegistry with no API keys And CLEVERAGENTS_DEFAULT_MODEL is not set When I call get_default_model Then the provider registry result should be None @unit @providers @registry Scenario: Default capabilities for OpenAI Given I have a ProviderRegistry instance When I get capabilities for ProviderType.OPENAI Then supports_streaming should be True And supports_tool_calls should be True And supports_vision should be True And max_context_length should be 128000 And supports_json_mode should be True @unit @providers @registry Scenario: Default capabilities for Anthropic Given I have a ProviderRegistry instance When I get capabilities for ProviderType.ANTHROPIC Then supports_streaming should be True And supports_tool_calls should be True And supports_vision should be True And max_context_length should be 200000 And supports_json_mode should be False @unit @providers @registry Scenario: Default capabilities for Google/Gemini Given I have a ProviderRegistry instance When I get capabilities for ProviderType.GOOGLE Then supports_streaming should be True And supports_tool_calls should be True And supports_vision should be True And max_context_length should be 1000000 And supports_json_mode should be True @unit @providers @registry Scenario: Global registry singleton Given I import the get_provider_registry function When I call get_provider_registry twice Then both provider registry calls should return the same instance @unit @providers @registry Scenario: Reset global registry Given I import the reset_provider_registry function When I get the global registry and then reset it Then resetting the provider registry should return a new instance @unit @providers @registry Scenario: Create LLM raises error when no provider configured Given I have a ProviderRegistry with no API keys When I try to create an LLM without specifying a provider Then a provider registry ValueError should be raised And the provider registry error should mention configuring API keys @unit @providers @registry Scenario: Create LLM raises error for invalid provider string Given I have a ProviderRegistry instance When I try to create an LLM for provider "not_a_provider" Then a provider registry ValueError should be raised And the provider registry error should mention unknown provider type @unit @providers @registry Scenario: Create LLM raises error when provider API key is missing Given I have a ProviderRegistry with no API keys When I try to create an LLM for provider "openai" Then a provider registry ValueError should be raised And the provider registry error should mention configuring API keys @unit @providers @registry Scenario: Create LLM delegates to provider factory for configured string input Given I have a ProviderRegistry with OpenAI API key set And the provider registry LLM factory is stubbed When I call create_llm for provider "openai" and model "gpt-mini" Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-mini" @unit @providers @registry Scenario: Create LLM without provider uses fallback default Given I have a ProviderRegistry with OpenAI API key set And CLEVERAGENTS_DEFAULT_PROVIDER is not set And CLEVERAGENTS_DEFAULT_MODEL is not set And the provider registry LLM factory is stubbed When I call create_llm without specifying a provider Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-4o" @unit @providers @registry Scenario: Create LLM accepts ProviderType argument Given I have a ProviderRegistry with OpenAI API key set And CLEVERAGENTS_DEFAULT_PROVIDER is not set And CLEVERAGENTS_DEFAULT_MODEL is not set And the provider registry LLM factory is stubbed When I call create_llm for ProviderType.OPENAI Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-4o" @unit @providers @registry Scenario: Create LLM for mock provider skips API key requirement Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set And I have a ProviderRegistry with no API keys And CLEVERAGENTS_DEFAULT_PROVIDER is not set And CLEVERAGENTS_DEFAULT_MODEL is not set And the provider registry LLM factory is stubbed When I call create_llm for provider "mock" without specifying a model Then the stubbed LLM factory should be called for ProviderType.MOCK with model "mock-gpt" @unit @providers @registry Scenario: Create LLM for mock provider is rejected without env var Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set And I have a ProviderRegistry with no API keys When I try to create an LLM for provider "mock" Then a provider registry ValueError should be raised And the provider registry error should mention "not allowed" @unit @providers @registry Scenario: Create LLM for mock provider returns FakeListLLM end-to-end Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set And I have a ProviderRegistry with no API keys When I call create_llm for provider "mock" without specifying a model Then the created LLM should be a FakeListLLM instance And the FakeListLLM produces response "mock response" on invoke And the FakeListLLM produces response "mock response" on invoke @unit @providers @registry Scenario: _create_provider_instance builds stubbed OpenAI client Given I have a ProviderRegistry with OpenAI API key set And langchain OpenAI client is stubbed When I call _create_provider_instance for provider "openai" with model "custom-openai" Then the stubbed ChatOpenAI should be created with model "custom-openai" @unit @providers @registry Scenario Outline: _create_provider_instance builds client Given I have a ProviderRegistry with "" API key set And the "" LangChain client is stubbed When I call _create_provider_instance for provider "" with model "" Then the stubbed client should be created with argument "model" value "" Examples: | ProviderName | provider_key | model | | Anthropic | anthropic | claude-mini-cover | | Google | google | gemini-cover | | Gemini | gemini | gemini-pro-cover | | Groq | groq | llama-cover | | Together | together | llama-together-cover | | Cohere | cohere | command-cover | @unit @providers @registry Scenario: _create_provider_instance handles MOCK provider Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set And I have a ProviderRegistry with no API keys And the "mock" LangChain client is stubbed When I call _create_provider_instance for provider "MOCK" with model "mock-gpt" Then the stubbed mock client should be instantiated @unit @providers @registry Scenario: Create AI provider raises error when no provider configured Given I have a ProviderRegistry with no API keys When I try to create an AI provider without specifying a provider Then a provider registry ValueError should be raised And the provider registry error should mention configuring API keys @unit @providers @registry Scenario: Create AI provider raises error for invalid provider string Given I have a ProviderRegistry instance When I try to create an AI provider for provider "not_a_provider" Then a provider registry ValueError should be raised And the provider registry error should mention unknown provider type @unit @providers @registry Scenario: Create AI provider raises error when Google API key is missing Given I have a ProviderRegistry with no API keys When I try to create an AI provider for provider "google" Then a provider registry ValueError should be raised And the provider registry error should mention missing "GOOGLE_API_KEY" environment variable @unit @providers @registry Scenario: Create AI provider succeeds for MOCK provider without API key Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set And I have a ProviderRegistry with no API keys And CLEVERAGENTS_DEFAULT_MODEL is not set When I create an AI provider for provider "mock" Then the provider registry AI provider should be a LangChainChatProvider And the provider registry AI provider name should be "mock" And the provider registry AI provider model_id should be "mock-gpt" @unit @providers @registry Scenario: Create AI provider for MOCK is rejected without env var Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set And I have a ProviderRegistry with no API keys When I try to create an AI provider for provider "mock" Then a provider registry ValueError should be raised And the provider registry error should mention "not allowed" @unit @providers @registry Scenario: Create AI provider succeeds for configured provider string Given I have a ProviderRegistry with OpenAI API key set And CLEVERAGENTS_DEFAULT_MODEL is not set When I create an AI provider for provider "openai" Then the provider registry AI provider should be a LangChainChatProvider And the provider registry AI provider name should be "openai" And the provider registry AI provider model_id should be "gpt-4o" @unit @providers @registry Scenario: Create AI provider returns Anthropic adapter Given I have a ProviderRegistry with Anthropic API key set And CLEVERAGENTS_DEFAULT_MODEL is not set When I create an AI provider for provider "anthropic" Then the provider registry AI provider should be a LangChainChatProvider And the provider registry AI provider name should be "anthropic" And the provider registry AI provider model_id should be "claude-sonnet-4-20250514" @unit @providers @registry Scenario: Create AI provider raises error when OpenAI API key is missing Given I have a ProviderRegistry with no API keys When I try to create an AI provider for provider "openai" Then a provider registry ValueError should be raised And the provider registry error should mention missing "OPENAI_API_KEY" environment variable @unit @providers @registry Scenario: Create AI provider raises error when Anthropic API key is missing Given I have a ProviderRegistry with no API keys When I try to create an AI provider for provider "anthropic" Then a provider registry ValueError should be raised And the provider registry error should mention missing "ANTHROPIC_API_KEY" environment variable @unit @providers @registry Scenario: Create AI provider returns Google adapter Given I have a ProviderRegistry with "google" API key set And CLEVERAGENTS_DEFAULT_MODEL is not set When I create an AI provider for provider "google" Then the provider registry AI provider should be a LangChainChatProvider And the provider registry AI provider name should be "google" And the provider registry AI provider model_id should be "gemini-2.0-flash" @unit @providers @registry Scenario: Create AI provider returns OpenRouter adapter Given I have a ProviderRegistry with "openrouter" API key set And CLEVERAGENTS_DEFAULT_MODEL is not set When I create an AI provider for provider "openrouter" Then the provider registry AI provider should be a LangChainChatProvider And the provider registry AI provider name should be "openrouter" And the provider registry AI provider model_id should be "anthropic/claude-sonnet-4-20250514" @unit @providers @registry Scenario: Create AI provider without specifying provider uses fallback Given I have a ProviderRegistry with OpenAI API key set And CLEVERAGENTS_DEFAULT_PROVIDER is not set And CLEVERAGENTS_DEFAULT_MODEL is not set When I create an AI provider without specifying a provider Then the provider registry AI provider should be a LangChainChatProvider And the provider registry AI provider name should be "openai" @unit @providers @registry Scenario: Create AI provider honors explicit model overrides Given I have a ProviderRegistry with OpenAI API key set And CLEVERAGENTS_DEFAULT_MODEL is not set When I create an AI provider specifying provider "openai" and model "custom-model" Then the provider registry AI provider model_id should be "custom-model" @unit @providers @registry Scenario: AI provider exposes working llm factory Given I have a ProviderRegistry with "groq" API key set And the provider registry LLM factory is stubbed When I create an AI provider for provider "groq" And I call the AI provider llm factory with model "llama-mini-cover" Then the stubbed LLM factory should be called for ProviderType.GROQ with model "llama-mini-cover" @unit @providers @registry Scenario: _create_provider_instance handles MOCK provider and response list Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set And I have a ProviderRegistry with no API keys And the "mock" LangChain client is stubbed When I call _create_provider_instance for provider "MOCK" with model "mock-gpt" Then the stubbed mock client should be instantiated And the stubbed mock client responses should be ["mock response"] * 10 @unit @providers @registry Scenario: API key is forwarded through create_ai_provider closure Given I have a ProviderRegistry with OpenAI API key set And the provider registry LLM factory is stubbed When I create an AI provider for provider "openai" And I call the AI provider llm factory with model "gpt-mini" Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-mini" And the stubbed LLM factory should be called with api_key present @unit @providers @registry Scenario: resolve_provider_by_name rejects mock without env var Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set When I call resolve_provider_by_name with "mock" Then a provider registry ValueError should be raised And the provider registry error should mention "not configured" @unit @providers @registry Scenario: resolve_provider_by_name resolves mock with env var Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set And I have a ProviderRegistry with no API keys When I call resolve_provider_by_name with "mock" Then the resolved provider should be a LangChainChatProvider @unit @providers @registry Scenario: ProviderCapabilities is immutable Given I import the ProviderCapabilities class When I create a ProviderCapabilities instance Then the instance should be frozen And attempting to modify it should raise an error @unit @providers @registry Scenario: Provider fallback order Given I have the ProviderRegistry class When I check the FALLBACK_ORDER Then OpenAI should be first in the order And Anthropic should be second in the order And Google should be third in the order @unit @providers @registry Scenario: Default models for all providers Given I have the ProviderRegistry class When I check the DEFAULT_MODELS Then OpenAI default should be "gpt-4o" And Anthropic default should be "claude-sonnet-4-20250514" And Google default should be "gemini-2.0-flash" And Groq default should be "llama-3.1-70b-versatile"