446 lines
20 KiB
Gherkin
446 lines
20 KiB
Gherkin
Feature: Provider Registry Coverage
|
|
As a developer
|
|
I want to ensure the ProviderRegistry class is thoroughly tested
|
|
So that provider discovery and selection work correctly
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create ProviderRegistry with default settings
|
|
Given I import the ProviderRegistry class
|
|
When I create a ProviderRegistry with default settings
|
|
Then the registry should be initialized
|
|
And the registry should have provider information
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderType enum values
|
|
Given I import the ProviderType enum
|
|
Then ProviderType should have OPENAI value "openai"
|
|
And ProviderType should have ANTHROPIC value "anthropic"
|
|
And ProviderType should have GOOGLE value "google"
|
|
And ProviderType should have AZURE value "azure"
|
|
And ProviderType should have OPENROUTER value "openrouter"
|
|
And ProviderType should have GEMINI value "gemini"
|
|
And ProviderType should have COHERE value "cohere"
|
|
And ProviderType should have GROQ value "groq"
|
|
And ProviderType should have TOGETHER value "together"
|
|
And ProviderType should have MOCK value "mock"
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderCapabilities with all fields
|
|
Given I import the ProviderCapabilities class
|
|
When I create a ProviderCapabilities with all fields
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 128000
|
|
And supports_json_mode should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderCapabilities with default values
|
|
Given I import the ProviderCapabilities class
|
|
When I create a ProviderCapabilities with default values
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be False
|
|
And supports_vision should be False
|
|
And max_context_length should be 4096
|
|
And supports_json_mode should be False
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderInfo dataclass
|
|
Given I import the ProviderInfo class
|
|
And I import the ProviderType enum
|
|
When I create a ProviderInfo for openai
|
|
Then the provider_type should be ProviderType.OPENAI
|
|
And the provider info name should be "Openai"
|
|
And the is_configured should be False by default
|
|
|
|
@unit @providers @registry
|
|
Scenario: Optional string helper coerces non-string values
|
|
Given I import the optional string coercion helper
|
|
When I coerce optional string with integer 42
|
|
Then the coerced optional string result should be "42"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get all providers from registry
|
|
Given I have a ProviderRegistry instance
|
|
When I call get_all_providers
|
|
Then the provider registry result should be a list of ProviderInfo
|
|
And the provider registry result should include core provider types
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get configured providers when none configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I call get_configured_providers
|
|
Then the provider registry result should be an empty list
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get configured providers with OpenAI key
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
When I call get_configured_providers
|
|
Then the provider registry result should contain OpenAI provider
|
|
And the provider registry OpenAI entry should be configured
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info by ProviderType
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for ProviderType.OPENAI
|
|
Then the provider registry result should be a ProviderInfo instance
|
|
And the provider_type should be ProviderType.OPENAI
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info by string
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for string "openai"
|
|
Then the provider registry result should be a ProviderInfo instance
|
|
And the provider_type should be ProviderType.OPENAI
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info by uppercase string
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for string "ANTHROPIC"
|
|
Then the provider registry result should be a ProviderInfo instance
|
|
And the provider_type should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info for invalid provider
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for string "invalid_provider"
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Check if provider is configured - not configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I check if openai is configured
|
|
Then the provider registry result should be False
|
|
|
|
@unit @providers @registry
|
|
Scenario: Check if provider is configured - configured
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
When I check if openai is configured
|
|
Then the provider registry result should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type with env var
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is set to "openai"
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.OPENAI
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type fallback order
|
|
Given I have a ProviderRegistry with Anthropic API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type when none configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type ignores invalid environment override
|
|
Given I have a ProviderRegistry with Anthropic API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is set to "not_a_real_provider"
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type skips unconfigured provider override
|
|
Given I have a ProviderRegistry with Anthropic API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is set to "openai"
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type honors settings default when valid
|
|
Given I have a ProviderRegistry with "anthropic" API key set and settings default provider "anthropic"
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type ignores invalid settings default
|
|
Given I have a ProviderRegistry with "anthropic" API key set and settings default provider "not_a_real_provider"
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model with env var
|
|
Given I have a ProviderRegistry instance
|
|
And CLEVERAGENTS_DEFAULT_MODEL is set to "gpt-4-turbo"
|
|
When I call get_default_model
|
|
Then the provider registry result should be "gpt-4-turbo"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model from settings configuration
|
|
Given I have a ProviderRegistry with default model "custom-model" configured
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I call get_default_model
|
|
Then the provider registry result should be "custom-model"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model for provider
|
|
Given I have a ProviderRegistry instance
|
|
When I call get_default_model for ProviderType.OPENAI
|
|
Then the provider registry result should be "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model for provider by string
|
|
Given I have a ProviderRegistry instance
|
|
When I call get_default_model for string "anthropic"
|
|
Then the provider registry result should be "claude-sonnet-4-20250514"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model for invalid provider
|
|
Given I have a ProviderRegistry instance
|
|
When I call get_default_model for string "invalid"
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model when no providers are configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I call get_default_model
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default capabilities for OpenAI
|
|
Given I have a ProviderRegistry instance
|
|
When I get capabilities for ProviderType.OPENAI
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 128000
|
|
And supports_json_mode should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default capabilities for Anthropic
|
|
Given I have a ProviderRegistry instance
|
|
When I get capabilities for ProviderType.ANTHROPIC
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 200000
|
|
And supports_json_mode should be False
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default capabilities for Google/Gemini
|
|
Given I have a ProviderRegistry instance
|
|
When I get capabilities for ProviderType.GOOGLE
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 1000000
|
|
And supports_json_mode should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: Global registry singleton
|
|
Given I import the get_provider_registry function
|
|
When I call get_provider_registry twice
|
|
Then both provider registry calls should return the same instance
|
|
|
|
@unit @providers @registry
|
|
Scenario: Reset global registry
|
|
Given I import the reset_provider_registry function
|
|
When I get the global registry and then reset it
|
|
Then resetting the provider registry should return a new instance
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM raises error when no provider configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an LLM without specifying a provider
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention configuring API keys
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM raises error for invalid provider string
|
|
Given I have a ProviderRegistry instance
|
|
When I try to create an LLM for provider "not_a_provider"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention unknown provider type
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM raises error when provider API key is missing
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an LLM for provider "openai"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention configuring API keys
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM delegates to provider factory for configured string input
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm for provider "openai" and model "gpt-mini"
|
|
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-mini"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM without provider uses fallback default
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm without specifying a provider
|
|
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM accepts ProviderType argument
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm for ProviderType.OPENAI
|
|
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM for mock provider skips API key requirement
|
|
Given I have a ProviderRegistry with no API keys
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm for provider "mock" without specifying a model
|
|
Then the stubbed LLM factory should be called for ProviderType.MOCK with model "mock-gpt"
|
|
|
|
@unit @providers @registry
|
|
Scenario: _create_provider_llm builds stubbed OpenAI client
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And langchain OpenAI client is stubbed
|
|
When I call _create_provider_llm for provider "openai" with model "custom-openai"
|
|
Then the stubbed ChatOpenAI should be created with model "custom-openai"
|
|
|
|
@unit @providers @registry
|
|
Scenario Outline: _create_provider_llm builds <ProviderName> client
|
|
Given I have a ProviderRegistry with "<provider_key>" API key set
|
|
And the "<ProviderName>" LangChain client is stubbed
|
|
When I call _create_provider_llm for provider "<provider_key>" with model "<model>"
|
|
Then the stubbed <ProviderName> client should be created with argument "model" value "<model>"
|
|
|
|
Examples:
|
|
| ProviderName | provider_key | model |
|
|
| Anthropic | anthropic | claude-mini-cover |
|
|
| Google | google | gemini-cover |
|
|
| Gemini | gemini | gemini-pro-cover |
|
|
| Groq | groq | llama-cover |
|
|
| Together | together | llama-together-cover |
|
|
| Cohere | cohere | command-cover |
|
|
|
|
@unit @providers @registry
|
|
Scenario: _create_provider_llm builds Azure client with deployment override
|
|
Given I have a ProviderRegistry with "azure" API key set
|
|
And the "Azure" LangChain client is stubbed
|
|
When I call _create_provider_llm for provider "azure" with model "azure-cover"
|
|
Then the stubbed Azure client should be created with argument "deployment_name" value "azure-cover"
|
|
|
|
@unit @providers @registry
|
|
Scenario: _create_provider_llm raises when Azure endpoint is missing
|
|
Given I have a ProviderRegistry with "azure" API key set
|
|
And the Azure endpoint setting is cleared
|
|
When I try to call _create_provider_llm for provider "azure" with model "azure-cover"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention missing Azure endpoint
|
|
|
|
|
|
@unit @providers @registry
|
|
Scenario: _create_provider_llm raises error for unsupported provider
|
|
Given I have a ProviderRegistry instance
|
|
When I try to call _create_provider_llm for provider "mock" with model "mock-gpt"
|
|
Then a provider registry ValueError should be raised
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error when no provider configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an AI provider without specifying a provider
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention configuring API keys
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error for invalid provider string
|
|
Given I have a ProviderRegistry instance
|
|
When I try to create an AI provider for provider "not_a_provider"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention unknown provider type
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error when Google API key is missing
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an AI provider for provider "google"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention missing "GOOGLE_API_KEY" environment variable
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider succeeds for configured provider string
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "openai"
|
|
Then the provider registry AI provider should be a LangChainChatProvider
|
|
And the provider registry AI provider name should be "openai"
|
|
And the provider registry AI provider model_id should be "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider returns Google adapter
|
|
Given I have a ProviderRegistry with "google" API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "google"
|
|
Then the provider registry AI provider should be a GoogleChatProvider
|
|
And the provider registry AI provider name should be "google"
|
|
And the provider registry AI provider model_id should be "gemini-2.0-flash"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider returns OpenRouter adapter
|
|
Given I have a ProviderRegistry with "openrouter" API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "openrouter"
|
|
Then the provider registry AI provider should be an OpenRouterChatProvider
|
|
And the provider registry AI provider name should be "openrouter"
|
|
And the provider registry AI provider model_id should be "anthropic/claude-sonnet-4-20250514"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider without specifying provider uses fallback
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider without specifying a provider
|
|
Then the provider registry AI provider should be a LangChainChatProvider
|
|
And the provider registry AI provider name should be "openai"
|
|
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider honors explicit model overrides
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider specifying provider "openai" and model "custom-model"
|
|
Then the provider registry AI provider model_id should be "custom-model"
|
|
|
|
@unit @providers @registry
|
|
Scenario: AI provider exposes working llm factory
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And the provider registry LLM factory is stubbed
|
|
When I create an AI provider for provider "openai"
|
|
And I call the AI provider llm factory with model "gpt-mini-cover"
|
|
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-mini-cover"
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderCapabilities is immutable
|
|
Given I import the ProviderCapabilities class
|
|
When I create a ProviderCapabilities instance
|
|
Then the instance should be frozen
|
|
And attempting to modify it should raise an error
|
|
|
|
@unit @providers @registry
|
|
Scenario: Provider fallback order
|
|
Given I have the ProviderRegistry class
|
|
When I check the FALLBACK_ORDER
|
|
Then OpenAI should be first in the order
|
|
And Anthropic should be second in the order
|
|
And Google should be third in the order
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default models for all providers
|
|
Given I have the ProviderRegistry class
|
|
When I check the DEFAULT_MODELS
|
|
Then OpenAI default should be "gpt-4o"
|
|
And Anthropic default should be "claude-sonnet-4-20250514"
|
|
And Google default should be "gemini-2.0-flash"
|
|
And Groq default should be "llama-3.1-70b-versatile"
|