hurui200320
7164b04019
CI / lint (push) Successful in 1m1s
CI / build (push) Successful in 53s
CI / quality (push) Successful in 1m23s
CI / typecheck (push) Successful in 1m24s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 1m42s
CI / push-validation (push) Successful in 21s
CI / helm (push) Successful in 25s
CI / e2e_tests (push) Successful in 4m6s
CI / integration_tests (push) Successful in 4m12s
CI / unit_tests (push) Successful in 6m39s
CI / docker (push) Successful in 1m44s
CI / coverage (push) Successful in 10m51s
CI / status-check (push) Successful in 2s
CI / benchmark-publish (push) Successful in 1h17m10s
CI / benchmark-publish (pull_request) Has been skipped
CI / helm (pull_request) Successful in 33s
CI / build (pull_request) Successful in 50s
CI / lint (pull_request) Successful in 1m7s
CI / quality (pull_request) Successful in 1m9s
CI / typecheck (pull_request) Successful in 1m20s
CI / security (pull_request) Successful in 1m25s
CI / push-validation (pull_request) Successful in 19s
CI / integration_tests (pull_request) Successful in 3m20s
CI / e2e_tests (pull_request) Failing after 5m3s
CI / unit_tests (pull_request) Successful in 6m40s
CI / docker (pull_request) Successful in 1m30s
CI / coverage (pull_request) Successful in 11m9s
CI / status-check (pull_request) Failing after 3s
CI / benchmark-regression (pull_request) Failing after 1m11s
Extract _get_api_key() and _create_provider_instance() as a single internal factory that handles all provider types, including MOCK. Both create_llm() and create_ai_provider() now delegate to _create_provider_instance(), eliminating the duplicated if/elif provider-switching chains and the divergence between the two code paths. Adding a new provider now requires touching exactly one place. Key changes: - New _get_api_key(provider_type) consolidates API key lookup and validation that was previously duplicated across create_llm(), _create_provider_llm(), and every branch of create_ai_provider(). - New _create_provider_instance() is the unified raw-LLM factory, replacing _create_provider_llm() and the specialised adapter constructors in create_ai_provider(). MOCK is now handled here via FakeListLLM. - create_ai_provider() wraps all providers uniformly in LangChainChatProvider instead of using specialised adapters (OpenAIChatProvider, AnthropicChatProvider, GoogleChatProvider, OpenRouterChatProvider) for the four primary providers. - create_llm() drops its own API key validation block; validation now happens inside _create_provider_instance() via _get_api_key(). - All BDD scenarios updated: _create_provider_llm references become _create_provider_instance, and provider-type assertions on create_ai_provider results are updated to LangChainChatProvider. Review fixes (Cycle 2): - Add _coerce_env_bool() for consistent boolean env-var coercion in CLEVERAGENTS_ALLOW_MOCK_PROVIDER checks. - Introduce _ApiKeyMissing sentinel to distinguish 'not provided' from None in provider configuration. - Set MOCK supports_streaming=False in DEFAULT_CAPABILITIES to align with FakeListLLM semantics. - Make MOCK is_configured conditional on CLEVERAGENTS_ALLOW_MOCK_PROVIDER. - Use FakeListLLM(responses=['mock response'] * 10) to prevent IndexError in multi-step workflows. - Strip max_retries from kwargs before passing to LangChain constructors. - Extract _resolve_provider_type() helper shared by create_llm() and create_ai_provider() to reduce duplication and improve readability. - Ensure __api_key_sentinel flows through factory_kwargs even when api_key is None, avoiding double _get_api_key() calls. - Add CHANGELOG env-var documentation and providers.md env-var table. - Resolve template DB lock issues by persisting in-memory SQLite via VACUUM INTO instead of direct file creation on tmpfs. ISSUES CLOSED: #10949
520 lines
24 KiB
Gherkin
520 lines
24 KiB
Gherkin
Feature: Provider Registry Coverage
|
|
As a developer
|
|
I want to ensure the ProviderRegistry class is thoroughly tested
|
|
So that provider discovery and selection work correctly
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create ProviderRegistry with default settings
|
|
Given I import the ProviderRegistry class
|
|
When I create a ProviderRegistry with default settings
|
|
Then the registry should be initialized
|
|
And the registry should have provider information
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderType enum values
|
|
Given I import the ProviderType enum
|
|
Then ProviderType should have OPENAI value "openai"
|
|
And ProviderType should have ANTHROPIC value "anthropic"
|
|
And ProviderType should have GOOGLE value "google"
|
|
And ProviderType should have AZURE value "azure"
|
|
And ProviderType should have OPENROUTER value "openrouter"
|
|
And ProviderType should have GEMINI value "gemini"
|
|
And ProviderType should have COHERE value "cohere"
|
|
And ProviderType should have GROQ value "groq"
|
|
And ProviderType should have TOGETHER value "together"
|
|
And ProviderType should have MOCK value "mock"
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderCapabilities with all fields
|
|
Given I import the ProviderCapabilities class
|
|
When I create a ProviderCapabilities with all fields
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 128000
|
|
And supports_json_mode should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderCapabilities with default values
|
|
Given I import the ProviderCapabilities class
|
|
When I create a ProviderCapabilities with default values
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be False
|
|
And supports_vision should be False
|
|
And max_context_length should be 4096
|
|
And supports_json_mode should be False
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderInfo dataclass
|
|
Given I import the ProviderInfo class
|
|
And I import the ProviderType enum
|
|
When I create a ProviderInfo for openai
|
|
Then the provider_type should be ProviderType.OPENAI
|
|
And the provider info name should be "Openai"
|
|
And the is_configured should be False by default
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get configured providers when none configured and mock not allowed
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set
|
|
And I have a ProviderRegistry with no API keys
|
|
When I call get_configured_providers
|
|
Then the provider registry result should be an empty list
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get configured providers when none configured
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
|
|
And I have a ProviderRegistry with no API keys
|
|
When I call get_configured_providers
|
|
Then the provider registry result should contain only the MOCK provider
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get configured providers with OpenAI key
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
|
|
And I have a ProviderRegistry with OpenAI API key set
|
|
When I call get_configured_providers
|
|
Then the provider registry result should contain OpenAI provider
|
|
And the provider registry result should contain the MOCK provider
|
|
And the provider registry OpenAI entry should be configured
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info by ProviderType
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for ProviderType.OPENAI
|
|
Then the provider registry result should be a ProviderInfo instance
|
|
And the provider_type should be ProviderType.OPENAI
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info by string
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for string "openai"
|
|
Then the provider registry result should be a ProviderInfo instance
|
|
And the provider_type should be ProviderType.OPENAI
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info by uppercase string
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for string "ANTHROPIC"
|
|
Then the provider registry result should be a ProviderInfo instance
|
|
And the provider_type should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get provider info for invalid provider
|
|
Given I have a ProviderRegistry instance
|
|
When I get provider info for string "invalid_provider"
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Check if provider is configured - not configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I check if openai is configured
|
|
Then the provider registry result should be False
|
|
|
|
@unit @providers @registry
|
|
Scenario: Check if provider is configured - configured
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
When I check if openai is configured
|
|
Then the provider registry result should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type with env var
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is set to "openai"
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.OPENAI
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type fallback order
|
|
Given I have a ProviderRegistry with Anthropic API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type when none configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type ignores invalid environment override
|
|
Given I have a ProviderRegistry with Anthropic API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is set to "not_a_real_provider"
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type skips unconfigured provider override
|
|
Given I have a ProviderRegistry with Anthropic API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is set to "openai"
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type honors settings default when valid
|
|
Given I have a ProviderRegistry with "anthropic" API key set and settings default provider "anthropic"
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default provider type ignores invalid settings default
|
|
Given I have a ProviderRegistry with "anthropic" API key set and settings default provider "not_a_real_provider"
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
When I call get_default_provider_type
|
|
Then the provider registry result should be ProviderType.ANTHROPIC
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model with env var
|
|
Given I have a ProviderRegistry instance
|
|
And CLEVERAGENTS_DEFAULT_MODEL is set to "gpt-4-turbo"
|
|
When I call get_default_model
|
|
Then the provider registry result should be "gpt-4-turbo"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model from settings configuration
|
|
Given I have a ProviderRegistry with default model "custom-model" configured
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I call get_default_model
|
|
Then the provider registry result should be "custom-model"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model for provider
|
|
Given I have a ProviderRegistry instance
|
|
When I call get_default_model for ProviderType.OPENAI
|
|
Then the provider registry result should be "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model for provider by string
|
|
Given I have a ProviderRegistry instance
|
|
When I call get_default_model for string "anthropic"
|
|
Then the provider registry result should be "claude-sonnet-4-20250514"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model for invalid provider
|
|
Given I have a ProviderRegistry instance
|
|
When I call get_default_model for string "invalid"
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Get default model when no providers are configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I call get_default_model
|
|
Then the provider registry result should be None
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default capabilities for OpenAI
|
|
Given I have a ProviderRegistry instance
|
|
When I get capabilities for ProviderType.OPENAI
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 128000
|
|
And supports_json_mode should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default capabilities for Anthropic
|
|
Given I have a ProviderRegistry instance
|
|
When I get capabilities for ProviderType.ANTHROPIC
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 200000
|
|
And supports_json_mode should be False
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default capabilities for Google/Gemini
|
|
Given I have a ProviderRegistry instance
|
|
When I get capabilities for ProviderType.GOOGLE
|
|
Then supports_streaming should be True
|
|
And supports_tool_calls should be True
|
|
And supports_vision should be True
|
|
And max_context_length should be 1000000
|
|
And supports_json_mode should be True
|
|
|
|
@unit @providers @registry
|
|
Scenario: Global registry singleton
|
|
Given I import the get_provider_registry function
|
|
When I call get_provider_registry twice
|
|
Then both provider registry calls should return the same instance
|
|
|
|
@unit @providers @registry
|
|
Scenario: Reset global registry
|
|
Given I import the reset_provider_registry function
|
|
When I get the global registry and then reset it
|
|
Then resetting the provider registry should return a new instance
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM raises error when no provider configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an LLM without specifying a provider
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention configuring API keys
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM raises error for invalid provider string
|
|
Given I have a ProviderRegistry instance
|
|
When I try to create an LLM for provider "not_a_provider"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention unknown provider type
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM raises error when provider API key is missing
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an LLM for provider "openai"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention configuring API keys
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM delegates to provider factory for configured string input
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm for provider "openai" and model "gpt-mini"
|
|
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-mini"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM without provider uses fallback default
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm without specifying a provider
|
|
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM accepts ProviderType argument
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm for ProviderType.OPENAI
|
|
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM for mock provider skips API key requirement
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
|
|
And I have a ProviderRegistry with no API keys
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
And the provider registry LLM factory is stubbed
|
|
When I call create_llm for provider "mock" without specifying a model
|
|
Then the stubbed LLM factory should be called for ProviderType.MOCK with model "mock-gpt"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM for mock provider is rejected without env var
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set
|
|
And I have a ProviderRegistry with no API keys
|
|
When I try to create an LLM for provider "mock"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention "not allowed"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create LLM for mock provider returns FakeListLLM end-to-end
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
|
|
And I have a ProviderRegistry with no API keys
|
|
When I call create_llm for provider "mock" without specifying a model
|
|
Then the created LLM should be a FakeListLLM instance
|
|
And the FakeListLLM produces response "mock response" on invoke
|
|
And the FakeListLLM produces response "mock response" on invoke
|
|
|
|
@unit @providers @registry
|
|
Scenario: _create_provider_instance builds stubbed OpenAI client
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And langchain OpenAI client is stubbed
|
|
When I call _create_provider_instance for provider "openai" with model "custom-openai"
|
|
Then the stubbed ChatOpenAI should be created with model "custom-openai"
|
|
|
|
@unit @providers @registry
|
|
Scenario Outline: _create_provider_instance builds <ProviderName> client
|
|
Given I have a ProviderRegistry with "<provider_key>" API key set
|
|
And the "<ProviderName>" LangChain client is stubbed
|
|
When I call _create_provider_instance for provider "<provider_key>" with model "<model>"
|
|
Then the stubbed <ProviderName> client should be created with argument "model" value "<model>"
|
|
|
|
Examples:
|
|
| ProviderName | provider_key | model |
|
|
| Anthropic | anthropic | claude-mini-cover |
|
|
| Google | google | gemini-cover |
|
|
| Gemini | gemini | gemini-pro-cover |
|
|
| Groq | groq | llama-cover |
|
|
| Together | together | llama-together-cover |
|
|
| Cohere | cohere | command-cover |
|
|
|
|
@unit @providers @registry
|
|
Scenario: _create_provider_instance handles MOCK provider
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
|
|
And I have a ProviderRegistry with no API keys
|
|
And the "mock" LangChain client is stubbed
|
|
When I call _create_provider_instance for provider "MOCK" with model "mock-gpt"
|
|
Then the stubbed mock client should be instantiated
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error when no provider configured
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an AI provider without specifying a provider
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention configuring API keys
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error for invalid provider string
|
|
Given I have a ProviderRegistry instance
|
|
When I try to create an AI provider for provider "not_a_provider"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention unknown provider type
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error when Google API key is missing
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an AI provider for provider "google"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention missing "GOOGLE_API_KEY" environment variable
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider succeeds for MOCK provider without API key
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
|
|
And I have a ProviderRegistry with no API keys
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "mock"
|
|
Then the provider registry AI provider should be a LangChainChatProvider
|
|
And the provider registry AI provider name should be "mock"
|
|
And the provider registry AI provider model_id should be "mock-gpt"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider for MOCK is rejected without env var
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set
|
|
And I have a ProviderRegistry with no API keys
|
|
When I try to create an AI provider for provider "mock"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention "not allowed"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider succeeds for configured provider string
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "openai"
|
|
Then the provider registry AI provider should be a LangChainChatProvider
|
|
And the provider registry AI provider name should be "openai"
|
|
And the provider registry AI provider model_id should be "gpt-4o"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider returns Anthropic adapter
|
|
Given I have a ProviderRegistry with Anthropic API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "anthropic"
|
|
Then the provider registry AI provider should be a LangChainChatProvider
|
|
And the provider registry AI provider name should be "anthropic"
|
|
And the provider registry AI provider model_id should be "claude-sonnet-4-20250514"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error when OpenAI API key is missing
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an AI provider for provider "openai"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention missing "OPENAI_API_KEY" environment variable
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider raises error when Anthropic API key is missing
|
|
Given I have a ProviderRegistry with no API keys
|
|
When I try to create an AI provider for provider "anthropic"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention missing "ANTHROPIC_API_KEY" environment variable
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider returns Google adapter
|
|
Given I have a ProviderRegistry with "google" API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "google"
|
|
Then the provider registry AI provider should be a LangChainChatProvider
|
|
And the provider registry AI provider name should be "google"
|
|
And the provider registry AI provider model_id should be "gemini-2.0-flash"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider returns OpenRouter adapter
|
|
Given I have a ProviderRegistry with "openrouter" API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider for provider "openrouter"
|
|
Then the provider registry AI provider should be a LangChainChatProvider
|
|
And the provider registry AI provider name should be "openrouter"
|
|
And the provider registry AI provider model_id should be "anthropic/claude-sonnet-4-20250514"
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider without specifying provider uses fallback
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_PROVIDER is not set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider without specifying a provider
|
|
Then the provider registry AI provider should be a LangChainChatProvider
|
|
And the provider registry AI provider name should be "openai"
|
|
|
|
|
|
@unit @providers @registry
|
|
Scenario: Create AI provider honors explicit model overrides
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And CLEVERAGENTS_DEFAULT_MODEL is not set
|
|
When I create an AI provider specifying provider "openai" and model "custom-model"
|
|
Then the provider registry AI provider model_id should be "custom-model"
|
|
|
|
@unit @providers @registry
|
|
Scenario: AI provider exposes working llm factory
|
|
Given I have a ProviderRegistry with "groq" API key set
|
|
And the provider registry LLM factory is stubbed
|
|
When I create an AI provider for provider "groq"
|
|
And I call the AI provider llm factory with model "llama-mini-cover"
|
|
Then the stubbed LLM factory should be called for ProviderType.GROQ with model "llama-mini-cover"
|
|
|
|
@unit @providers @registry
|
|
Scenario: _create_provider_instance handles MOCK provider and response list
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
|
|
And I have a ProviderRegistry with no API keys
|
|
And the "mock" LangChain client is stubbed
|
|
When I call _create_provider_instance for provider "MOCK" with model "mock-gpt"
|
|
Then the stubbed mock client should be instantiated
|
|
And the stubbed mock client responses should be ["mock response"] * 10
|
|
|
|
@unit @providers @registry
|
|
Scenario: API key is forwarded through create_ai_provider closure
|
|
Given I have a ProviderRegistry with OpenAI API key set
|
|
And the provider registry LLM factory is stubbed
|
|
When I create an AI provider for provider "openai"
|
|
And I call the AI provider llm factory with model "gpt-mini"
|
|
Then the stubbed LLM factory should be called for ProviderType.OPENAI with model "gpt-mini"
|
|
And the stubbed LLM factory should be called with api_key present
|
|
|
|
@unit @providers @registry
|
|
Scenario: resolve_provider_by_name rejects mock without env var
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is not set
|
|
When I call resolve_provider_by_name with "mock"
|
|
Then a provider registry ValueError should be raised
|
|
And the provider registry error should mention "not configured"
|
|
|
|
@unit @providers @registry
|
|
Scenario: resolve_provider_by_name resolves mock with env var
|
|
Given CLEVERAGENTS_ALLOW_MOCK_PROVIDER is set
|
|
And I have a ProviderRegistry with no API keys
|
|
When I call resolve_provider_by_name with "mock"
|
|
Then the resolved provider should be a LangChainChatProvider
|
|
|
|
@unit @providers @registry
|
|
Scenario: ProviderCapabilities is immutable
|
|
Given I import the ProviderCapabilities class
|
|
When I create a ProviderCapabilities instance
|
|
Then the instance should be frozen
|
|
And attempting to modify it should raise an error
|
|
|
|
@unit @providers @registry
|
|
Scenario: Provider fallback order
|
|
Given I have the ProviderRegistry class
|
|
When I check the FALLBACK_ORDER
|
|
Then OpenAI should be first in the order
|
|
And Anthropic should be second in the order
|
|
And Google should be third in the order
|
|
|
|
@unit @providers @registry
|
|
Scenario: Default models for all providers
|
|
Given I have the ProviderRegistry class
|
|
When I check the DEFAULT_MODELS
|
|
Then OpenAI default should be "gpt-4o"
|
|
And Anthropic default should be "claude-sonnet-4-20250514"
|
|
And Google default should be "gemini-2.0-flash"
|
|
And Groq default should be "llama-3.1-70b-versatile"
|