HAL9000
1b12af2765
CI / lint (pull_request) Successful in 40s
CI / helm (pull_request) Successful in 35s
CI / build (pull_request) Successful in 56s
CI / quality (pull_request) Successful in 1m29s
CI / typecheck (pull_request) Successful in 1m36s
CI / security (pull_request) Successful in 1m35s
CI / push-validation (pull_request) Successful in 26s
CI / unit_tests (pull_request) Successful in 6m2s
CI / docker (pull_request) Successful in 1m44s
CI / integration_tests (pull_request) Successful in 9m56s
CI / coverage (pull_request) Successful in 12m16s
CI / status-check (pull_request) Successful in 17s
Imported CostMetadata into fallback_selector.py to access per-plan budget data. Extended FallbackSelector.__init__ with cost_metadata: CostMetadata | None = None and stored it in self._cost_metadata. Implemented per-plan budget check in FallbackSelector.select() immediately after the daily budget validation to enforce per-plan limits during selection. Added two new TDD scenarios to features/cost_controls.feature to exercise per-plan budget behavior, tagged @tdd_issue @tdd_issue_10471. Added new test steps at features/steps/tdd_fallback_plan_budget_steps.py to support the new scenarios. ISSUES CLOSED: #10485
491 lines
19 KiB
Gherkin
491 lines
19 KiB
Gherkin
Feature: Cost controls and provider fallback
|
|
As a developer using CleverAgents
|
|
I want cost tracking, budget enforcement, and provider fallback
|
|
So that I can control AI spending and ensure service continuity
|
|
|
|
# ---- CostEntry ----
|
|
|
|
@unit @cost
|
|
Scenario: CostEntry validates non-negative input cost
|
|
Given I import the CostEntry class
|
|
When I create a CostEntry with negative input cost
|
|
Then a ValueError should be raised for CostEntry
|
|
|
|
@unit @cost
|
|
Scenario: CostEntry validates non-negative output cost
|
|
Given I import the CostEntry class
|
|
When I create a CostEntry with negative output cost
|
|
Then a ValueError should be raised for CostEntry
|
|
|
|
@unit @cost
|
|
Scenario: CostEntry estimates cost correctly
|
|
Given I import the CostEntry class
|
|
When I create a CostEntry with input cost 0.00001 and output cost 0.00003
|
|
And I estimate cost for 1000 input tokens and 500 output tokens
|
|
Then the estimated cost should be 0.025
|
|
|
|
@unit @cost
|
|
Scenario: CostEntry rejects negative token counts
|
|
Given I import the CostEntry class
|
|
When I create a CostEntry with input cost 0.00001 and output cost 0.00003
|
|
Then estimating cost with negative input tokens raises ValueError
|
|
And estimating cost with negative output tokens raises ValueError
|
|
|
|
# ---- ProviderCostTable ----
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable returns default cost for unknown model
|
|
Given I create a ProviderCostTable with defaults
|
|
When I look up cost for provider "openai" model "unknown-model-xyz"
|
|
Then the cost entry should be the default cost
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable returns known cost for gpt-4o
|
|
Given I create a ProviderCostTable with defaults
|
|
When I look up cost for provider "openai" model "gpt-4o"
|
|
Then the cost entry input cost should be 0.0000025
|
|
And the cost entry output cost should be 0.00001
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable with custom entries overrides defaults
|
|
Given I create a ProviderCostTable with custom entry for "openai" "gpt-4o"
|
|
When I look up cost for provider "openai" model "gpt-4o"
|
|
Then the cost entry input cost should be 0.001
|
|
And the cost entry output cost should be 0.002
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable validates empty provider
|
|
Given I create a ProviderCostTable with defaults
|
|
Then looking up cost for empty provider raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable validates empty model
|
|
Given I create a ProviderCostTable with defaults
|
|
Then looking up cost for empty model raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable lists providers and models
|
|
Given I create a ProviderCostTable with defaults
|
|
When I list providers from the cost table
|
|
Then the provider list should include "openai"
|
|
And the provider list should include "anthropic"
|
|
When I list models for provider "openai"
|
|
Then the model list should include "gpt-4o"
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable rejects non-dict custom entries
|
|
When I create a ProviderCostTable with non-dict custom entries
|
|
Then a TypeError should be raised for ProviderCostTable
|
|
|
|
# ---- CostMetadata ----
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata records usage correctly
|
|
Given I create a fresh CostMetadata
|
|
When I record usage of 100 input tokens and 50 output tokens at cost 0.01 for provider "openai"
|
|
Then total tokens should be 150
|
|
And total cost should be 0.01
|
|
And provider costs for "openai" should be 0.01
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata accumulates multiple usages
|
|
Given I create a fresh CostMetadata
|
|
When I record usage of 100 input tokens and 50 output tokens at cost 0.01 for provider "openai"
|
|
And I record usage of 200 input tokens and 100 output tokens at cost 0.02 for provider "anthropic"
|
|
Then total tokens should be 450
|
|
And total cost should be 0.03
|
|
And provider costs for "openai" should be 0.01
|
|
And provider costs for "anthropic" should be 0.02
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata rejects negative input tokens
|
|
Given I create a fresh CostMetadata
|
|
Then recording usage with negative input tokens raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata rejects negative cost
|
|
Given I create a fresh CostMetadata
|
|
Then recording usage with negative cost raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata rejects empty provider
|
|
Given I create a fresh CostMetadata
|
|
Then recording usage with empty provider raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata display dict includes all fields
|
|
Given I create a fresh CostMetadata
|
|
When I record usage of 100 input tokens and 50 output tokens at cost 0.01 for provider "openai"
|
|
And I get the display dict from cost metadata
|
|
Then the display dict should have key "total_tokens"
|
|
And the display dict should have key "total_cost_usd"
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata display dict includes budget remaining when set
|
|
Given I create a fresh CostMetadata
|
|
When I set budget remaining to 5.0
|
|
And I get the display dict from cost metadata
|
|
Then the display dict should have key "budget_remaining_usd"
|
|
|
|
# ---- BudgetExhaustionEvent ----
|
|
|
|
@unit @cost
|
|
Scenario: BudgetExhaustionEvent validates budget type
|
|
Given I import the BudgetExhaustionEvent class
|
|
When I create a BudgetExhaustionEvent with invalid budget type
|
|
Then a validation error should be raised for BudgetExhaustionEvent
|
|
|
|
@unit @cost
|
|
Scenario: BudgetExhaustionEvent accepts plan budget type
|
|
Given I import the BudgetExhaustionEvent class
|
|
When I create a BudgetExhaustionEvent with budget type "plan"
|
|
Then the event should be created successfully
|
|
|
|
@unit @cost
|
|
Scenario: BudgetExhaustionEvent accepts daily budget type
|
|
Given I import the BudgetExhaustionEvent class
|
|
When I create a BudgetExhaustionEvent with budget type "daily"
|
|
Then the event should be created successfully
|
|
|
|
# ---- CostTracker ----
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker with no budget limits
|
|
Given I create a CostTracker with no limits
|
|
When I record tracked usage of 1000 input and 500 output for "openai" "gpt-4o"
|
|
Then the budget check should be under budget
|
|
And the cost metadata total tokens should be 1500
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker warns at 90% of plan budget
|
|
Given I create a CostTracker with plan budget 0.001
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 500 input and 500 output for "openai" "gpt-4o"
|
|
Then the budget check should be warning or exceeded
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker blocks at 100% of plan budget
|
|
Given I create a CostTracker with plan budget 0.0000001
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 10000 input and 5000 output for "openai" "gpt-4o"
|
|
Then the budget check should be exceeded
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker blocks at 100% of daily budget
|
|
Given I create a CostTracker with daily budget 0.0000001
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 10000 input and 5000 output for "openai" "gpt-4o"
|
|
Then the budget check should be exceeded
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker persists exhaustion events
|
|
Given I create a CostTracker with plan budget 0.0000001
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 10000 input and 5000 output for "openai" "gpt-4o"
|
|
Then the cost metadata should have budget exhaustion events
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates negative plan budget
|
|
When I create a CostTracker with negative plan budget
|
|
Then a ValueError should be raised for CostTracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates negative daily budget
|
|
When I create a CostTracker with negative daily budget
|
|
Then a ValueError should be raised for CostTracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker estimate_cost delegates to cost table
|
|
Given I create a CostTracker with no limits
|
|
When I estimate cost for "openai" "gpt-4o" with 1000 input and 500 output
|
|
Then the estimated tracker cost should be positive
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates empty provider in estimate
|
|
Given I create a CostTracker with no limits
|
|
Then estimating cost with empty provider raises ValueError on tracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates empty model in estimate
|
|
Given I create a CostTracker with no limits
|
|
Then estimating cost with empty model raises ValueError on tracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates negative tokens in record
|
|
Given I create a CostTracker with no limits
|
|
And I create a fresh CostMetadata
|
|
Then recording tracked usage with negative input raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker check_plan_budget with no limit returns under budget
|
|
Given I create a CostTracker with no limits
|
|
And I create a fresh CostMetadata
|
|
When I check plan budget
|
|
Then the plan budget check should be under budget
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker check_daily_budget with no limit returns under budget
|
|
Given I create a CostTracker with no limits
|
|
When I check daily budget
|
|
Then the daily budget check should be under budget
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker get_daily_spend returns zero initially
|
|
Given I create a CostTracker with no limits
|
|
Then the daily spend should be 0.0
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker budget properties are accessible
|
|
Given I create a CostTracker with plan budget 10.0 and daily budget 50.0
|
|
Then the tracker plan budget should be 10.0
|
|
And the tracker daily budget should be 50.0
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker get_cost_entry validates empty provider
|
|
Given I create a CostTracker with no limits
|
|
Then getting cost entry with empty provider raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker get_cost_entry validates empty model
|
|
Given I create a CostTracker with no limits
|
|
Then getting cost entry with empty model raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates empty provider in record_usage
|
|
Given I create a CostTracker with no limits
|
|
And I create a fresh CostMetadata
|
|
Then recording tracked usage with empty provider raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates empty model in record_usage
|
|
Given I create a CostTracker with no limits
|
|
And I create a fresh CostMetadata
|
|
Then recording tracked usage with empty model raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker with zero plan budget returns exceeded
|
|
Given I create a CostTracker with plan budget 0.0
|
|
And I create a fresh CostMetadata
|
|
When I check plan budget
|
|
Then the plan budget check should be exceeded
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker with zero daily budget returns exceeded
|
|
Given I create a CostTracker with daily budget 0.0
|
|
When I check daily budget
|
|
Then the daily budget check should be exceeded
|
|
|
|
# ---- CostTracker concurrency ----
|
|
|
|
@unit @cost @concurrency
|
|
Scenario: CostTracker record_usage is thread-safe under concurrent calls
|
|
Given I create a CostTracker with no limits
|
|
When 20 threads concurrently record usage of 100 input and 50 output for "openai" "gpt-4o"
|
|
Then the daily spend should equal the sum of all concurrent costs
|
|
|
|
# ---- FallbackSelector ----
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector with no configured providers returns None
|
|
Given I create a ProviderRegistry with no API keys for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider
|
|
Then the fallback result provider should be None
|
|
And the fallback result should have skipped entries
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips providers missing tool calls
|
|
Given I create a ProviderRegistry with mock provider for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider requiring tool calls
|
|
Then the fallback result should skip mock for missing tool calls
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector selects first capable provider
|
|
Given I create a ProviderRegistry with openai key for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider requiring tool calls
|
|
Then the fallback result provider type should be "openai"
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector uses custom fallback order
|
|
Given I create a ProviderRegistry with openai key for fallback
|
|
When I create a FallbackSelector with custom order "openai"
|
|
And I select a fallback provider
|
|
Then the fallback result provider type should be "openai"
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector rejects None registry
|
|
When I create a FallbackSelector with None registry
|
|
Then a TypeError should be raised for FallbackSelector
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector rejects empty string in fallback list
|
|
Given I create a ProviderRegistry with no API keys for fallback
|
|
When I create a FallbackSelector with empty string in fallback list
|
|
Then a ValueError should be raised for FallbackSelector
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips provider when daily budget exceeded
|
|
Given I create a ProviderRegistry with openai key for fallback
|
|
And I create a CostTracker with daily budget 0.0
|
|
When I create a FallbackSelector with cost tracker
|
|
And I select a fallback provider
|
|
Then the fallback result provider should be None
|
|
|
|
# ---- Settings config keys ----
|
|
|
|
@unit @cost
|
|
Scenario: Settings budget_per_plan defaults to None
|
|
When I load cost control settings with defaults
|
|
Then budget_per_plan should be None
|
|
|
|
@unit @cost
|
|
Scenario: Settings budget_per_day defaults to None
|
|
When I load cost control settings with defaults
|
|
Then budget_per_day should be None
|
|
|
|
@unit @cost
|
|
Scenario: Settings fallback_providers defaults to empty list
|
|
When I load cost control settings with defaults
|
|
Then fallback_providers should be an empty list
|
|
|
|
@unit @cost
|
|
Scenario: Settings budget_per_plan accepts positive value
|
|
When I load cost control settings with budget_per_plan 10.0
|
|
Then budget_per_plan should be 10.0
|
|
|
|
@unit @cost
|
|
Scenario: Settings budget_per_day accepts positive value
|
|
When I load cost control settings with budget_per_day 50.0
|
|
Then budget_per_day should be 50.0
|
|
|
|
# ---- Plan model cost_metadata ----
|
|
|
|
@unit @cost
|
|
Scenario: Plan model includes cost_metadata field
|
|
Given I create a minimal Plan with cost metadata
|
|
Then the plan should have cost_metadata set
|
|
|
|
@unit @cost
|
|
Scenario: Plan as_cli_dict includes cost when present
|
|
Given I create a minimal Plan with cost metadata
|
|
When I get the CLI dict from the plan
|
|
Then the CLI dict should have a "cost" key
|
|
|
|
@unit @cost
|
|
Scenario: Plan as_cli_dict omits cost when None
|
|
Given I create a minimal Plan without cost metadata
|
|
When I get the CLI dict from the plan
|
|
Then the CLI dict should not have a "cost" key
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata display dict with exhaustion events
|
|
Given I create a fresh CostMetadata
|
|
When I add a budget exhaustion event of type "plan"
|
|
And I get the display dict from cost metadata
|
|
Then the display dict should have key "budget_exhaustion_events"
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata records negative output tokens rejection
|
|
Given I create a fresh CostMetadata
|
|
Then recording usage with negative output tokens raises ValueError
|
|
|
|
# ---- Additional coverage scenarios ----
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable custom entries for brand new provider
|
|
Given I create a ProviderCostTable with custom entry for new provider "myprovider" model "mymodel"
|
|
When I look up cost for provider "myprovider" model "mymodel"
|
|
Then the cost entry input cost should be 0.005
|
|
And the cost entry output cost should be 0.01
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable list_models with empty provider raises ValueError
|
|
Given I create a ProviderCostTable with defaults
|
|
Then listing models for empty provider raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker estimate_cost rejects negative input tokens
|
|
Given I create a CostTracker with no limits
|
|
Then estimating cost with negative input tokens raises ValueError on tracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker estimate_cost rejects negative output tokens
|
|
Given I create a CostTracker with no limits
|
|
Then estimating cost with negative output tokens raises ValueError on tracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker record_usage rejects negative output tokens
|
|
Given I create a CostTracker with no limits
|
|
And I create a fresh CostMetadata
|
|
Then recording tracked usage with negative output raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker plan budget warning at 90 percent
|
|
Given I create a CostTracker with plan budget 0.10
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 3600 input and 8500 output for "openai" "gpt-4o"
|
|
Then the budget check should be warning
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker daily budget warning at 90 percent
|
|
Given I create a CostTracker with daily budget 0.10
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 3600 input and 8500 output for "openai" "gpt-4o"
|
|
Then the budget check should be warning
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker get_cost_entry returns valid entry
|
|
Given I create a CostTracker with no limits
|
|
When I get cost entry for "openai" "gpt-4o"
|
|
Then the returned cost entry should have positive input cost
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips unknown provider types
|
|
Given I create a ProviderRegistry with no API keys for fallback
|
|
When I create a FallbackSelector with custom order "nonexistent_provider"
|
|
And I select a fallback provider
|
|
Then the fallback result provider should be None
|
|
And the fallback skipped should mention "unknown provider type"
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips providers missing streaming
|
|
Given I create a ProviderRegistry with mock provider for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider requiring streaming
|
|
Then the fallback result should skip for missing streaming
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips providers missing vision
|
|
Given I create a ProviderRegistry with mock provider for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider requiring vision
|
|
Then the fallback result should skip for missing vision
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips providers missing json mode
|
|
Given I create a ProviderRegistry with mock provider for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider requiring json mode
|
|
Then the fallback result should skip for missing json mode
|
|
|
|
@unit @cost @tdd_issue @tdd_issue_10471
|
|
Scenario: FallbackSelector skips provider when per-plan budget exceeded
|
|
Given I create a ProviderRegistry with openai key for fallback
|
|
And I create a CostTracker with plan budget 0.0
|
|
And I create a CostMetadata with total cost 1.0
|
|
When I create a FallbackSelector with cost tracker and cost metadata
|
|
And I select a fallback provider
|
|
Then the fallback result provider should be None
|
|
And the fallback skipped should mention "plan budget exceeded"
|
|
|
|
@unit @cost @tdd_issue @tdd_issue_10471
|
|
Scenario: FallbackSelector selects provider when per-plan budget not exceeded
|
|
Given I create a ProviderRegistry with openai key for fallback
|
|
And I create a CostTracker with plan budget 10.0
|
|
And I create a CostMetadata with total cost 1.0
|
|
When I create a FallbackSelector with cost tracker and cost metadata
|
|
And I select a fallback provider
|
|
Then the fallback result provider type should be "openai"
|