freemo
a074b4846f
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 15s
CI / build (pull_request) Successful in 19s
CI / quality (pull_request) Successful in 28s
CI / security (pull_request) Successful in 50s
CI / typecheck (pull_request) Successful in 56s
CI / integration_tests (pull_request) Successful in 4m51s
CI / unit_tests (pull_request) Successful in 19m29s
CI / docker (pull_request) Successful in 39s
CI / benchmark-regression (pull_request) Successful in 26m10s
CI / coverage (pull_request) Successful in 47m42s
CI / lint (push) Successful in 13s
CI / quality (push) Successful in 17s
CI / build (push) Successful in 23s
CI / typecheck (push) Successful in 30s
CI / security (push) Successful in 30s
CI / benchmark-regression (push) Has been skipped
CI / integration_tests (push) Successful in 2m52s
CI / unit_tests (push) Successful in 10m8s
CI / docker (push) Successful in 1m19s
CI / benchmark-publish (push) Successful in 11m54s
CI / coverage (push) Failing after 39m51s
Remove FakeListLLM as a silent fallback in agent graph constructors (plan_generation.py, context_analysis.py, auto_debug.py). All three now raise ValueError when llm=None, making missing-provider errors explicit. Add Settings.mock_providers flag and validate_provider_availability() method. Update container.get_ai_provider() to check Settings.mock_providers first, with env-var fallback for backward compatibility. Add resolve_provider_by_name() helper to the provider registry and export it from cleveragents.providers. Add structlog trace logging to ProviderRegistry.get_default_provider_type() to record selection reasoning. Update all existing behave step files, robot tests, and benchmarks that relied on the implicit FakeListLLM default to pass an explicit LLM instance instead. Add new BDD tests (features/provider_fixes.feature with 17 scenarios), Robot Framework integration tests (robot/provider_detection_smoke.robot), and ASV benchmarks (benchmarks/provider_selection_bench.py). ISSUES CLOSED: #323
106 lines
4.8 KiB
Gherkin
106 lines
4.8 KiB
Gherkin
Feature: Provider configuration fixes remove FakeListLLM defaults
|
|
As a developer
|
|
I want providers to require explicit LLM configuration
|
|
So that FakeListLLM is never silently used as a production fallback
|
|
|
|
@phase1
|
|
Scenario: Provider auto-detection selects first configured provider
|
|
Given a provider-fix settings instance with openai configured
|
|
When I create a provider-fix registry from those settings
|
|
Then the provider-fix default provider type should be "openai"
|
|
|
|
@phase1
|
|
Scenario: No providers configured and mock_providers off raises error
|
|
Given a provider-fix settings instance with no providers configured
|
|
And provider-fix mock_providers is disabled
|
|
When I call provider-fix validate_provider_availability
|
|
Then a provider-fix ValueError should be raised containing "No AI providers configured"
|
|
|
|
@phase1
|
|
Scenario: mock_providers flag enables mock provider
|
|
Given a provider-fix settings instance with mock_providers enabled
|
|
When I build the provider-fix AI provider from the container helper
|
|
Then a provider-fix mock AI provider should be returned
|
|
|
|
@phase1
|
|
Scenario: FakeListLLM is not used as default fallback in PlanGenerationGraph
|
|
When I attempt to create a provider-fix PlanGenerationGraph without an LLM
|
|
Then a provider-fix ValueError should be raised containing "No LLM provider configured"
|
|
|
|
@phase1
|
|
Scenario: FakeListLLM is not used as default fallback in ContextAnalysisAgent
|
|
When I attempt to create a provider-fix ContextAnalysisAgent without an LLM
|
|
Then a provider-fix ValueError should be raised containing "No LLM provider configured"
|
|
|
|
@phase1
|
|
Scenario: FakeListLLM is not used as default fallback in AutoDebugAgent
|
|
When I attempt to create a provider-fix AutoDebugAgent without an LLM
|
|
Then a provider-fix ValueError should be raised containing "No LLM provider configured"
|
|
|
|
@phase1
|
|
Scenario: Provider selection trace logging outputs chosen provider
|
|
Given a provider-fix settings instance with openai configured
|
|
When I create a provider-fix registry and request the default provider
|
|
Then the provider-fix selection should be logged
|
|
|
|
@phase1
|
|
Scenario: resolve_provider_by_name returns provider for configured name
|
|
Given a provider-fix settings instance with openai configured
|
|
When I provider-fix resolve provider by name "openai"
|
|
Then I should receive a provider-fix AI provider instance
|
|
|
|
@phase1
|
|
Scenario: resolve_provider_by_name raises error for unconfigured name
|
|
Given a provider-fix settings instance with no providers configured
|
|
When I provider-fix resolve provider by name "openai"
|
|
Then a provider-fix ValueError should be raised containing "not configured"
|
|
|
|
@phase1
|
|
Scenario: resolve_provider_by_name raises error for unknown name
|
|
Given a provider-fix settings instance with no providers configured
|
|
When I provider-fix resolve provider by name "nonexistent_provider"
|
|
Then a provider-fix ValueError should be raised containing "Unknown provider"
|
|
|
|
@phase1
|
|
Scenario: validate_provider_availability passes with provider configured
|
|
Given a provider-fix settings instance with openai configured
|
|
And provider-fix mock_providers is disabled on that instance
|
|
When I call provider-fix validate_provider_availability
|
|
Then no provider-fix error should be raised
|
|
|
|
@phase1
|
|
Scenario: mock_providers warns outside test mode
|
|
Given a provider-fix settings instance with mock_providers enabled
|
|
And the provider-fix environment is set to "production"
|
|
When I call provider-fix validate_provider_availability
|
|
Then a provider-fix warning should be logged about mock_providers
|
|
|
|
@phase1
|
|
Scenario: Settings mock_providers field defaults to False
|
|
Given I create a provider-fix fresh settings instance
|
|
Then provider-fix mock_providers should be False by default
|
|
|
|
@phase1
|
|
Scenario: Container get_ai_provider respects mock_providers setting
|
|
Given a provider-fix settings instance with mock_providers enabled
|
|
When I call provider-fix get_ai_provider with those settings
|
|
Then a provider-fix mock provider or None should be returned
|
|
|
|
@phase1
|
|
Scenario: PlanGenerationGraph works with explicit LLM
|
|
Given I have a provider-fix FakeListLLM instance for testing
|
|
When I create a provider-fix PlanGenerationGraph with that LLM
|
|
Then the provider-fix graph should be created successfully
|
|
|
|
@phase1
|
|
Scenario: ContextAnalysisAgent works with explicit LLM
|
|
Given I have a provider-fix FakeListLLM instance for testing
|
|
When I create a provider-fix ContextAnalysisAgent with that LLM
|
|
Then the provider-fix agent should be created successfully
|
|
|
|
@phase1
|
|
Scenario: AutoDebugAgent works with explicit LLM
|
|
Given I have a provider-fix FakeListLLM instance for testing
|
|
When I create a provider-fix AutoDebugAgent with that LLM
|
|
Then the provider-fix agent should be created successfully
|