HAL9000
f808abff86
Fix JSON syntax errors in .devcontainer/devcontainer.json (removed
invalid JS-style // comments) and .devcontainer/opencode.json (removed
90+ trailing commas). Apply auto-fixes for end-of-file and trailing
whitespace issues across 100+ files. Fix SIM105 ruff violations in
benchmarks/core_circuit_breaker_bench.py (use contextlib.suppress).
Note: The security fix from issue #7478 (validate_path startswith bypass)
was already delivered to master in commit e18ac5f2. This PR as currently
structured is non-atomic (35 commits across 10+ issues) and needs
significant restructure before merge. This commit only addresses the
CI/pre-commit failures.
ISSUES CLOSED: #7478
743 lines
32 KiB
Gherkin
743 lines
32 KiB
Gherkin
Feature: Consolidated Ai Models Providers
|
|
Combined scenarios from: aimodelscredentials_coverage, aimodelserrors_coverage, aimodelsproviders_coverage, anthropic_provider, google_provider, langsmith_config, openrouter_provider
|
|
|
|
# ============================================================
|
|
# Originally from: aimodelscredentials_coverage.feature
|
|
# Feature: AI Models Credentials Domain Model Coverage
|
|
# ============================================================
|
|
|
|
Scenario: Import and instantiate ModelProviderOption with all fields
|
|
Given the cleveragents package is available
|
|
When I import ModelProviderOption from aimodelscredentials
|
|
Then the ModelProviderOption class should be available
|
|
And I can create a ModelProviderOption with publishers, config, and priority
|
|
|
|
|
|
Scenario: Create ModelProviderOption with minimal required fields
|
|
Given the cleveragents package is available
|
|
When I create a ModelProviderOption with only priority set to 1
|
|
Then the ModelProviderOption should have priority 1
|
|
And the publishers dictionary should be empty
|
|
And the config should be None
|
|
|
|
|
|
Scenario: Create ModelProviderOption with publishers dictionary
|
|
Given the cleveragents package is available
|
|
When I create a ModelProviderOption with complex publishers structure
|
|
Then the publishers field should contain the expected dictionary structure
|
|
And the ModelPublisher enum values should be properly handled
|
|
|
|
|
|
Scenario: Create ModelProviderOption with ModelProviderConfigSchema
|
|
Given the cleveragents package is available
|
|
When I create a ModelProviderOption with a config object
|
|
Then the config field should contain the ModelProviderConfigSchema instance
|
|
And the config should be properly validated
|
|
|
|
|
|
Scenario: Validate ModelProviderOption strips whitespace
|
|
Given the cleveragents package is available
|
|
When I create a ModelProviderOption with strings containing whitespace
|
|
Then the string fields should have whitespace stripped
|
|
|
|
|
|
Scenario: Validate ModelProviderOption assignment validation
|
|
Given the cleveragents package is available
|
|
When I create a ModelProviderOption instance
|
|
And I update its fields with new values
|
|
Then the assignment validation should be triggered
|
|
And the values should be properly validated
|
|
|
|
|
|
Scenario: Validate ModelProviderOption populate by name
|
|
Given the cleveragents package is available
|
|
When I create a ModelProviderOption using field aliases
|
|
Then the fields should be populated correctly by name
|
|
|
|
|
|
Scenario: Validate ModelProviderOption use enum values
|
|
Given the cleveragents package is available
|
|
When I create a ModelProviderOption with ModelPublisher enums
|
|
Then the enum values should be used in serialization
|
|
And the model should handle enum values properly
|
|
|
|
|
|
Scenario: Test ModelProviderOption field defaults
|
|
Given the cleveragents package is available
|
|
When I create a ModelProviderOption with default fields
|
|
Then publishers should default to an empty dictionary
|
|
And config should default to None
|
|
|
|
|
|
Scenario: Test ModelProviderOption field validation with invalid data
|
|
Given the cleveragents package is available
|
|
When I try to create a ModelProviderOption with invalid priority
|
|
Then a ModelProviderOption validation error should be raised
|
|
And the error should indicate the priority field issue
|
|
|
|
|
|
Scenario: Test ModelProviderOption model config settings
|
|
Given the cleveragents package is available
|
|
When I examine the ModelProviderOption model_config
|
|
Then the ModelProviderOption str_strip_whitespace should be True
|
|
And the ModelProviderOption validate_assignment should be True
|
|
And the ModelProviderOption arbitrary_types_allowed should be False
|
|
And the ModelProviderOption populate_by_name should be True
|
|
And the ModelProviderOption use_enum_values should be True
|
|
|
|
|
|
Scenario: Test ModelProviderOption with nested publisher structures
|
|
Given the cleveragents package is available
|
|
When I create a ModelProviderOption with nested publisher dictionaries
|
|
Then the nested structure should be properly maintained
|
|
And the ModelPublisher boolean values should work correctly
|
|
|
|
|
|
Scenario: Test ModelProviderOption serialization and deserialization
|
|
Given the cleveragents package is available
|
|
When I serialize a ModelProviderOption to dict
|
|
And I deserialize it back to ModelProviderOption
|
|
Then the round-trip should preserve all data
|
|
|
|
|
|
Scenario: Test ModelProviderOption with None config field
|
|
Given the cleveragents package is available
|
|
When I explicitly set config to None in ModelProviderOption
|
|
Then the config field should accept None value
|
|
And the model should be valid
|
|
|
|
|
|
Scenario: Test ModelProviderOption priority field requirement
|
|
Given the cleveragents package is available
|
|
When I try to create a ModelProviderOption without priority
|
|
Then a ModelProviderOption validation error should be raised for missing required field
|
|
|
|
# ============================================================
|
|
# Originally from: aimodelserrors_coverage.feature
|
|
# Feature: AI Models Errors Coverage
|
|
# ============================================================
|
|
|
|
Scenario: Create ModelError with overloaded error type
|
|
Given I import the ModelError class
|
|
When I create a ModelError with kind "OVERLOADED"
|
|
And I set retriable to true
|
|
And I set retryafterseconds to 30
|
|
Then the ModelError instance should be created successfully
|
|
And the kind should be "ErrOverloaded"
|
|
And the retriable flag should be true
|
|
And the retry after seconds should be 30
|
|
|
|
|
|
Scenario: Create ModelError with context too long error
|
|
Given I import the ModelError class
|
|
When I create a ModelError with kind "CONTEXT_TOO_LONG"
|
|
And I set retriable to false
|
|
And I set retryafterseconds to 0
|
|
Then the ModelError instance should be created successfully
|
|
And the kind should be "ErrContextTooLong"
|
|
And the retriable flag should be false
|
|
And the retry after seconds should be 0
|
|
|
|
|
|
Scenario: Create ModelError with rate limited error
|
|
Given I import the ModelError class
|
|
When I create a ModelError with kind "RATE_LIMITED"
|
|
And I set retriable to true
|
|
And I set retryafterseconds to 60
|
|
Then the ModelError instance should be created successfully
|
|
And the kind should be "ErrRateLimited"
|
|
And the retriable flag should be true
|
|
And the retry after seconds should be 60
|
|
|
|
|
|
Scenario: Create ModelError with subscription quota exhausted
|
|
Given I import the ModelError class
|
|
When I create a ModelError with kind "SUBSCRIPTION_QUOTA_EXHAUSTED"
|
|
And I set retriable to false
|
|
And I set retryafterseconds to 3600
|
|
Then the ModelError instance should be created successfully
|
|
And the kind should be "ErrSubscriptionQuotaExhausted"
|
|
And the retriable flag should be false
|
|
And the retry after seconds should be 3600
|
|
|
|
|
|
Scenario: Create ModelError with other error type
|
|
Given I import the ModelError class
|
|
When I create a ModelError with kind "OTHER"
|
|
And I set retriable to true
|
|
And I set retryafterseconds to 10
|
|
Then the ModelError instance should be created successfully
|
|
And the kind should be "ErrOther"
|
|
And the retriable flag should be true
|
|
And the retry after seconds should be 10
|
|
|
|
|
|
Scenario: Create ModelError with cache support error
|
|
Given I import the ModelError class
|
|
When I create a ModelError with kind "CACHE_SUPPORT"
|
|
And I set retriable to false
|
|
And I set retryafterseconds to 0
|
|
Then the ModelError instance should be created successfully
|
|
And the kind should be "ErrCacheSupport"
|
|
And the retriable flag should be false
|
|
And the retry after seconds should be 0
|
|
|
|
|
|
Scenario: Validate ModelError configuration
|
|
Given I import the ModelError class
|
|
When I create a ModelError with valid data
|
|
Then the model configuration should have str_strip_whitespace as true
|
|
And the model configuration should have validate_assignment as true
|
|
And the model configuration should have arbitrary_types_allowed as false
|
|
And the model configuration should have populate_by_name as true
|
|
And the model configuration should have use_enum_values as true
|
|
|
|
|
|
Scenario: Create FallbackResult with error fallback type
|
|
Given I import the FallbackResult class
|
|
When I create a FallbackResult with fallback type "ERROR"
|
|
And I set isfallback to true
|
|
And I set modelroleconfig to None
|
|
And I set basemodelconfig to None
|
|
Then the FallbackResult instance should be created successfully
|
|
And the fallback type should be "FallbackTypeError"
|
|
And the is fallback flag should be true
|
|
And the model role config should be None
|
|
And the base model config should be None
|
|
|
|
|
|
Scenario: Create FallbackResult with context fallback type
|
|
Given I import the FallbackResult class
|
|
When I create a FallbackResult with fallback type "CONTEXT"
|
|
And I set isfallback to true
|
|
And I set modelroleconfig to None
|
|
And I set basemodelconfig to None
|
|
Then the FallbackResult instance should be created successfully
|
|
And the fallback type should be "FallbackTypeContext"
|
|
And the is fallback flag should be true
|
|
|
|
|
|
Scenario: Create FallbackResult with provider fallback type
|
|
Given I import the FallbackResult class
|
|
When I create a FallbackResult with fallback type "PROVIDER"
|
|
And I set isfallback to false
|
|
And I set modelroleconfig to None
|
|
And I set basemodelconfig to None
|
|
Then the FallbackResult instance should be created successfully
|
|
And the fallback type should be "FallbackTypeProvider"
|
|
And the is fallback flag should be false
|
|
|
|
|
|
Scenario: Create FallbackResult with model role config
|
|
Given I import the FallbackResult class
|
|
And I have a valid ModelRoleConfig instance
|
|
When I create a FallbackResult with fallback type "ERROR"
|
|
And I set isfallback to true
|
|
And I set modelroleconfig to the valid instance
|
|
And I set basemodelconfig to None
|
|
Then the FallbackResult instance should be created successfully
|
|
And the model role config should not be None
|
|
And the model role config role should be "assistant"
|
|
|
|
|
|
Scenario: Create FallbackResult with base model config
|
|
Given I import the FallbackResult class
|
|
And I have a valid BaseModelConfig instance
|
|
When I create a FallbackResult with fallback type "CONTEXT"
|
|
And I set isfallback to true
|
|
And I set modelroleconfig to None
|
|
And I set basemodelconfig to the valid instance
|
|
Then the FallbackResult instance should be created successfully
|
|
And the base model config should not be None
|
|
And the base model config model_tag should be "gpt-4"
|
|
|
|
|
|
Scenario: Create FallbackResult with both configs
|
|
Given I import the FallbackResult class
|
|
And I have a valid ModelRoleConfig instance
|
|
And I have a valid BaseModelConfig instance
|
|
When I create a FallbackResult with fallback type "PROVIDER"
|
|
And I set isfallback to false
|
|
And I set modelroleconfig to the valid instance
|
|
And I set basemodelconfig to the valid instance
|
|
Then the FallbackResult instance should be created successfully
|
|
And the model role config should not be None
|
|
And the base model config should not be None
|
|
|
|
|
|
Scenario: Validate FallbackResult configuration
|
|
Given I import the FallbackResult class
|
|
When I create a FallbackResult with valid data
|
|
Then the model configuration should have str_strip_whitespace as true
|
|
And the model configuration should have validate_assignment as true
|
|
And the model configuration should have arbitrary_types_allowed as false
|
|
And the model configuration should have populate_by_name as true
|
|
And the model configuration should have use_enum_values as true
|
|
|
|
|
|
Scenario: ModelError with invalid kind raises validation error
|
|
Given I import the ModelError class
|
|
When I try to create a ModelError with invalid kind "INVALID_KIND"
|
|
Then a validation error should be raised
|
|
And the error should mention "Input should be"
|
|
|
|
|
|
Scenario: FallbackResult with invalid fallback type raises error
|
|
Given I import the FallbackResult class
|
|
When I try to create a FallbackResult with invalid fallback type "INVALID_TYPE"
|
|
Then a validation error should be raised
|
|
And the error should mention "Input should be"
|
|
|
|
|
|
Scenario: Test ModelError field aliases
|
|
Given I import the ModelError class
|
|
When I create a ModelError using field aliases
|
|
Then the ModelError should accept the aliased fields
|
|
And the values should be properly mapped
|
|
|
|
|
|
Scenario: Test FallbackResult field aliases
|
|
Given I import the FallbackResult class
|
|
When I create a FallbackResult using field aliases
|
|
Then the FallbackResult should accept the aliased fields
|
|
And the values should be properly mapped
|
|
|
|
|
|
Scenario: ModelError strips whitespace from string fields
|
|
Given I import the ModelError class
|
|
When I create a ModelError with whitespace in enum values
|
|
Then the whitespace should be stripped from strings
|
|
And the model should be created successfully
|
|
|
|
|
|
Scenario: FallbackResult strips whitespace from string fields
|
|
Given I import the FallbackResult class
|
|
When I create a FallbackResult with whitespace in enum values
|
|
Then the whitespace should be stripped from strings
|
|
And the model should be created successfully
|
|
|
|
# ============================================================
|
|
# Originally from: aimodelsproviders_coverage.feature
|
|
# Feature: AI Models Providers Coverage
|
|
# ============================================================
|
|
|
|
@unit @models
|
|
Scenario: Create ModelProviderExtraAuthVars with all fields
|
|
Given I import the ModelProviderExtraAuthVars class
|
|
When I create a ModelProviderExtraAuthVars with var "AWS_REGION"
|
|
And I set maybeJSONFilePath to true
|
|
And I set required to true
|
|
And I set default to "us-east-1"
|
|
Then the var field should equal "AWS_REGION"
|
|
And the maybe_j_s_o_n_file_path field should be true
|
|
And the required field should be true
|
|
And the default field should equal "us-east-1"
|
|
|
|
|
|
@unit @models
|
|
Scenario: Create ModelProviderExtraAuthVars with minimal fields
|
|
Given I import the ModelProviderExtraAuthVars class
|
|
When I create a ModelProviderExtraAuthVars with only var "API_KEY"
|
|
Then the var field should equal "API_KEY"
|
|
And the maybe_j_s_o_n_file_path field should be None
|
|
And the required field should be None
|
|
And the default field should be None
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderExtraAuthVars field validation
|
|
Given I import the ModelProviderExtraAuthVars class
|
|
When I create a ModelProviderExtraAuthVars with var " TRIMMED_VAR "
|
|
Then the var field should equal "TRIMMED_VAR"
|
|
And the model should strip whitespace from string fields
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderExtraAuthVars alias support
|
|
Given I import the ModelProviderExtraAuthVars class
|
|
When I create a ModelProviderExtraAuthVars using alias "maybeJSONFilePath"
|
|
Then the field should be accessible as maybe_j_s_o_n_file_path
|
|
And the model should populate by name
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderExtraAuthVars dict export
|
|
Given I have a ModelProviderExtraAuthVars instance
|
|
When I export it to dict with aliases
|
|
Then the dict should contain "maybeJSONFilePath" key
|
|
And the dict should not contain "maybe_j_s_o_n_file_path" key
|
|
|
|
|
|
@unit @models
|
|
Scenario: Create ModelProviderConfigSchema with all fields
|
|
Given I import the ModelProviderConfigSchema class
|
|
When I create a ModelProviderConfigSchema with provider "ModelProviderOpenAI"
|
|
And I set base_url to "https://api.openai.com"
|
|
And I set custom_provider to "custom-gpt"
|
|
And I set has_a_w_s_auth to true
|
|
And I set has_claude_max_auth to false
|
|
And I set skip_auth to false
|
|
And I set local_only to false
|
|
And I set api_key_env_var to "OPENAI_API_KEY"
|
|
And I add extra_auth_vars list
|
|
Then the provider field should equal "ModelProviderOpenAI"
|
|
And the base_url field should equal "https://api.openai.com"
|
|
And the custom_provider field should equal "custom-gpt"
|
|
And the has_a_w_s_auth field should be true
|
|
And the has_claude_max_auth field should be false
|
|
And the skip_auth field should be false
|
|
And the local_only field should be false
|
|
And the api_key_env_var field should equal "OPENAI_API_KEY"
|
|
And the extra_auth_vars should be a list
|
|
|
|
|
|
@unit @models
|
|
Scenario: Create ModelProviderConfigSchema with minimal fields
|
|
Given I import the ModelProviderConfigSchema class
|
|
When I create a ModelProviderConfigSchema with only required fields
|
|
Then the provider field should be set
|
|
And the base_url field should be set
|
|
And all optional fields should be None
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema with ModelProvider enum
|
|
Given I import the ModelProviderConfigSchema class
|
|
And I import the ModelProvider enum for provider config
|
|
When I create a ModelProviderConfigSchema with ModelProvider.ANTHROPIC
|
|
Then the provider field should equal "ModelProviderAnthropic"
|
|
And the model should use enum values
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema field validation
|
|
Given I import the ModelProviderConfigSchema class
|
|
When I create a ModelProviderConfigSchema with whitespace in fields
|
|
Then all string fields should have whitespace stripped
|
|
And the model should validate assignment
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema alias support
|
|
Given I import the ModelProviderConfigSchema class
|
|
When I create a ModelProviderConfigSchema using aliases
|
|
Then "baseUrl" should map to base_url
|
|
And "customProvider" should map to custom_provider
|
|
And "hasAWSAuth" should map to has_a_w_s_auth
|
|
And "hasClaudeMaxAuth" should map to has_claude_max_auth
|
|
And "skipAuth" should map to skip_auth
|
|
And "localOnly" should map to local_only
|
|
And "apiKeyEnvVar" should map to api_key_env_var
|
|
And "extraAuthVars" should map to extra_auth_vars
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema dict export with aliases
|
|
Given I have a ModelProviderConfigSchema instance
|
|
When I export it to dict with aliases
|
|
Then the dict should use camelCase keys
|
|
And "baseUrl" should be in the dict
|
|
And "hasAWSAuth" should be in the dict
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema with empty extra_auth_vars
|
|
Given I import the ModelProviderConfigSchema class
|
|
When I create a ModelProviderConfigSchema with empty extra_auth_vars list
|
|
Then the extra_auth_vars should be an empty list
|
|
And the model should accept empty lists
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema with multiple extra_auth_vars
|
|
Given I import the ModelProviderConfigSchema class
|
|
When I create a ModelProviderConfigSchema with multiple extra_auth_vars
|
|
Then each extra_auth_var should be a ModelProviderExtraAuthVars instance
|
|
And the list should maintain order
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema JSON serialization
|
|
Given I have a ModelProviderConfigSchema instance
|
|
When I serialize it to JSON
|
|
Then the JSON should be valid
|
|
And it should contain all set fields
|
|
And it should use aliases in the output
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema JSON deserialization
|
|
Given I have a JSON string with provider config
|
|
When I deserialize it to ModelProviderConfigSchema
|
|
Then the object should be correctly populated
|
|
And aliases should be resolved to field names
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderExtraAuthVars model config validation
|
|
Given I import the ModelProviderExtraAuthVars class
|
|
When I check the model configuration
|
|
Then str_strip_whitespace should be True
|
|
And validate_assignment should be True
|
|
And arbitrary_types_allowed should be False
|
|
And populate_by_name should be True
|
|
And use_enum_values should be True
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema model config validation
|
|
Given I import the ModelProviderConfigSchema class
|
|
When I check the model configuration
|
|
Then str_strip_whitespace should be True
|
|
And validate_assignment should be True
|
|
And arbitrary_types_allowed should be False
|
|
And populate_by_name should be True
|
|
And use_enum_values should be True
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderExtraAuthVars field update
|
|
Given I have a ModelProviderExtraAuthVars instance
|
|
When I update the var field to "NEW_VAR"
|
|
Then the var field should equal "NEW_VAR"
|
|
And the update should be validated
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema field update
|
|
Given I have a ModelProviderConfigSchema instance
|
|
When I update the base_url field to "https://new-api.com"
|
|
Then the base_url field should equal "https://new-api.com"
|
|
And the update should be validated
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderExtraAuthVars copy with update
|
|
Given I have a ModelProviderExtraAuthVars instance
|
|
When I create a copy with updated fields
|
|
Then the copy should have new values
|
|
And the original should remain unchanged
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema copy with update
|
|
Given I have a ModelProviderConfigSchema instance
|
|
When I create a copy with updated fields
|
|
Then the copy should have new values
|
|
And the original should remain unchanged
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderExtraAuthVars equality comparison
|
|
Given I have two ModelProviderExtraAuthVars instances
|
|
When I compare them for equality
|
|
Then identical instances should be equal
|
|
And different instances should not be equal
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema equality comparison
|
|
Given I have two ModelProviderConfigSchema instances
|
|
When I compare them for equality
|
|
Then identical instances should be equal
|
|
And different instances should not be equal
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderExtraAuthVars are not hashable
|
|
Given I have ModelProviderExtraAuthVars instances
|
|
When I try to use them as dictionary keys
|
|
Then the instances should not work as dictionary keys
|
|
And a TypeError should be raised when hashing
|
|
|
|
|
|
@unit @models
|
|
Scenario: ModelProviderConfigSchema validation error
|
|
Given I import the ModelProviderConfigSchema class
|
|
When I try to create an instance with missing required fields
|
|
Then a validation error should be raised for missing fields
|
|
And the error should indicate missing fields
|
|
|
|
# ============================================================
|
|
# Originally from: anthropic_provider.feature
|
|
# Feature: Anthropic chat provider coverage
|
|
# ============================================================
|
|
|
|
@unit @providers @anthropic
|
|
Scenario: Anthropic provider rejects missing API key
|
|
Given I have sample provider domain inputs
|
|
When I attempt to create an Anthropic chat provider without an API key
|
|
Then the Anthropic provider creation should fail with "Anthropic API key is required"
|
|
|
|
|
|
@unit @providers @anthropic
|
|
Scenario: Anthropic provider forwards keyword overrides to ChatAnthropic
|
|
Given I have sample provider domain inputs
|
|
When I create an Anthropic chat provider with API key "sk-anthropic-unit" and model "claude-3-5-sonnet-20241022" with custom overrides
|
|
And I request plan generation from the Anthropic provider
|
|
Then the Anthropic provider should construct ChatAnthropic with api key "sk-anthropic-unit" and model "claude-3-5-sonnet-20241022"
|
|
And the Anthropic provider should apply the custom overrides to ChatAnthropic
|
|
And the Anthropic provider metadata should report name "anthropic" and model "claude-3-5-sonnet-20241022"
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: google_provider.feature
|
|
# Feature: Google provider adapter coverage
|
|
# ============================================================
|
|
|
|
@unit @providers @google
|
|
Scenario: Google provider instantiates ChatGoogleGenerativeAI with provided credentials
|
|
Given I have sample provider domain inputs
|
|
When I create a Google chat provider with API key "sk-google-unit" and model "gemini-2.0-flash"
|
|
And I request plan generation from the Google provider
|
|
Then the Google provider should construct ChatGoogleGenerativeAI with api key "sk-google-unit" and model "gemini-2.0-flash"
|
|
And the Google provider metadata should report name "google" and model "gemini-2.0-flash"
|
|
|
|
|
|
@unit @providers @google
|
|
Scenario: Google provider forwards extra kwargs to ChatGoogleGenerativeAI
|
|
Given I have sample provider domain inputs
|
|
And I set the Google provider extra kwargs "temperature=0.25,max_output_tokens=2048"
|
|
When I create a Google chat provider with API key "sk-google-unit" and model "gemini-2.0-pro"
|
|
And I request plan generation from the Google provider
|
|
Then the Google provider should construct ChatGoogleGenerativeAI with api key "sk-google-unit" and model "gemini-2.0-pro"
|
|
And the Google provider should include kwargs "temperature=0.25,max_output_tokens=2048" in the ChatGoogleGenerativeAI call
|
|
And the Google provider metadata should report name "google" and model "gemini-2.0-pro"
|
|
|
|
|
|
@unit @providers @google
|
|
Scenario: Google provider returns generated changes with token count
|
|
Given I have sample provider domain inputs
|
|
And the plan generation graph returns a generated change for "app/google_provider.py"
|
|
And the Google provider token estimator returns 256 tokens
|
|
When I create a Google chat provider with API key "sk-google-unit" and model "gemini-2.0-flash"
|
|
And I request plan generation from the Google provider
|
|
Then the Google provider response should include 1 generated change for "app/google_provider.py"
|
|
And the Google provider response token count should equal 256
|
|
|
|
|
|
@unit @providers @google
|
|
Scenario: Google provider rejects missing API key
|
|
Given I have sample provider domain inputs
|
|
When I attempt to create a Google chat provider without an API key
|
|
Then the Google provider creation should fail with error "Google API key is required"
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: langsmith_config.feature
|
|
# Feature: LangSmith configuration detection
|
|
# ============================================================
|
|
|
|
Scenario: LangSmith tracing disabled by default
|
|
Given LangSmith environment is clean
|
|
When I load CleverAgents settings for LangSmith
|
|
Then LangSmith tracing should be disabled
|
|
|
|
|
|
Scenario: LangSmith tracing enabled via LangChain env vars
|
|
Given LangSmith environment is clean
|
|
Given I set environment variable "LANGCHAIN_TRACING_V2" to "true"
|
|
And I set environment variable "LANGCHAIN_API_KEY" to "demo-key"
|
|
And I set environment variable "LANGCHAIN_PROJECT" to "langchain-env-project"
|
|
When I load CleverAgents settings for LangSmith
|
|
Then LangSmith tracing should be enabled
|
|
|
|
|
|
Scenario: Building LangSmith config metadata
|
|
Given LangSmith environment is clean
|
|
Given I set environment variable "CLEVERAGENTS_LANGSMITH_ENABLED" to "true"
|
|
And I set environment variable "CLEVERAGENTS_LANGSMITH_PROJECT" to "cleveragents-core"
|
|
And I set environment variable "CLEVERAGENTS_LANGSMITH_API_KEY" to "demo-key"
|
|
When I load CleverAgents settings for LangSmith
|
|
And I build a LangSmith config with run name "demo-run"
|
|
Then the LangSmith config should include tag "context-analysis"
|
|
And the LangSmith config should include metadata key "langsmith_project"
|
|
|
|
|
|
Scenario: LangSmith validation fails without API key
|
|
Given LangSmith environment is clean
|
|
Given I set environment variable "CLEVERAGENTS_LANGSMITH_ENABLED" to "true"
|
|
When I load CleverAgents settings for LangSmith
|
|
Then LangSmith tracing should be disabled
|
|
And the LangSmith validation errors should mention "API key"
|
|
|
|
|
|
Scenario: LangSmith detects LangChain variables
|
|
Given LangSmith environment is clean
|
|
Given I set environment variable "LANGCHAIN_TRACING_V2" to "true"
|
|
And I set environment variable "LANGCHAIN_API_KEY" to "demo-key"
|
|
And I set environment variable "LANGCHAIN_PROJECT" to "observability-demo"
|
|
When I load CleverAgents settings for LangSmith
|
|
And I build a LangSmith config with run name "auto-detected"
|
|
Then LangSmith tracing should be enabled
|
|
And the LangSmith config should include metadata key "langsmith_project"
|
|
And the LangSmith config should include metadata value "observability-demo" for key "langsmith_project"
|
|
|
|
|
|
Scenario: LangSmith synchronization sets LangChain env vars
|
|
Given LangSmith environment is clean
|
|
Given I set environment variable "CLEVERAGENTS_LANGSMITH_ENABLED" to "true"
|
|
And I set environment variable "CLEVERAGENTS_LANGSMITH_API_KEY" to "sync-key"
|
|
And I set environment variable "CLEVERAGENTS_LANGSMITH_PROJECT" to "sync-project"
|
|
When I load CleverAgents settings for LangSmith
|
|
Then the environment variable "LANGCHAIN_TRACING_V2" should equal "true"
|
|
And the environment variable "LANGCHAIN_PROJECT" should equal "sync-project"
|
|
And the environment variable "LANGCHAIN_API_KEY" should equal "sync-key"
|
|
|
|
|
|
Scenario: LangSmith config includes user metadata
|
|
Given LangSmith environment is clean
|
|
Given I set environment variable "CLEVERAGENTS_LANGSMITH_ENABLED" to "true"
|
|
And I set environment variable "CLEVERAGENTS_LANGSMITH_API_KEY" to "user-key"
|
|
And I set environment variable "CLEVERAGENTS_LANGSMITH_PROJECT" to "user-project"
|
|
And I set environment variable "CLEVERAGENTS_LANGSMITH_USER_ID" to "user-123"
|
|
When I load CleverAgents settings for LangSmith
|
|
And I build a LangSmith config with run name "user-run"
|
|
Then the LangSmith config should include metadata key "user_id"
|
|
And the LangSmith config should include metadata value "user-123" for key "user_id"
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: openrouter_provider.feature
|
|
# Feature: OpenRouter provider adapter coverage
|
|
# ============================================================
|
|
|
|
@unit @providers @openrouter
|
|
Scenario: OpenRouter provider instantiates ChatOpenAI with provided credentials
|
|
Given I have sample provider domain inputs
|
|
When I create an OpenRouter chat provider with API key "sk-openrouter-unit" and model "anthropic/claude-sonnet-4-20250514"
|
|
And I request plan generation from the OpenRouter provider
|
|
Then the OpenRouter provider should construct ChatOpenAI with api key "sk-openrouter-unit", model "anthropic/claude-sonnet-4-20250514", and base url "https://openrouter.ai/api/v1"
|
|
And the OpenRouter provider metadata should report name "openrouter" and model "anthropic/claude-sonnet-4-20250514"
|
|
|
|
|
|
@unit @providers @openrouter
|
|
Scenario: OpenRouter provider forwards extra kwargs and reports token usage
|
|
Given I have sample provider domain inputs
|
|
And the plan generation graph returns a generated change for "app/openrouter_provider.py"
|
|
And the OpenRouter provider token estimator returns 512 tokens
|
|
And I set the OpenRouter provider extra kwargs "temperature=0.15,max_tokens=2048"
|
|
When I create an OpenRouter chat provider with API key "sk-openrouter-unit" and model "anthropic/claude-sonnet-4-20250514"
|
|
And I request plan generation from the OpenRouter provider
|
|
Then the OpenRouter provider should construct ChatOpenAI with api key "sk-openrouter-unit", model "anthropic/claude-sonnet-4-20250514", and base url "https://openrouter.ai/api/v1"
|
|
And the OpenRouter provider should include kwargs "temperature=0.15,max_tokens=2048" in the ChatOpenAI call
|
|
And the OpenRouter provider response should include 1 generated change for "app/openrouter_provider.py"
|
|
And the OpenRouter provider response token count should equal 512
|
|
|
|
|
|
@unit @providers @openrouter
|
|
Scenario: OpenRouter provider attaches organization headers when provided
|
|
Given I have sample provider domain inputs
|
|
And I set the OpenRouter provider default headers "X-Trace-ID=trace-openrouter"
|
|
And I set the OpenRouter provider organization "cleveragents.dev"
|
|
When I create an OpenRouter chat provider with API key "sk-openrouter-unit" and model "anthropic/claude-sonnet-4-20250514"
|
|
And I request plan generation from the OpenRouter provider
|
|
Then the OpenRouter provider should include headers "X-Trace-ID=trace-openrouter,HTTP-Referer=cleveragents.dev,X-Title=cleveragents.dev" in the ChatOpenAI call
|
|
|
|
|
|
@unit @providers @openrouter
|
|
Scenario: OpenRouter provider rejects missing API key
|
|
Given I have sample provider domain inputs
|
|
When I attempt to create an OpenRouter chat provider without an API key
|
|
Then the OpenRouter provider creation should fail with error "OpenRouter API key is required"
|