Feature: Cost controls and provider fallback As a developer using CleverAgents I want cost tracking, budget enforcement, and provider fallback So that I can control AI spending and ensure service continuity # ---- CostEntry ---- @unit @cost Scenario: CostEntry validates non-negative input cost Given I import the CostEntry class When I create a CostEntry with negative input cost Then a ValueError should be raised for CostEntry @unit @cost Scenario: CostEntry validates non-negative output cost Given I import the CostEntry class When I create a CostEntry with negative output cost Then a ValueError should be raised for CostEntry @unit @cost Scenario: CostEntry estimates cost correctly Given I import the CostEntry class When I create a CostEntry with input cost 0.00001 and output cost 0.00003 And I estimate cost for 1000 input tokens and 500 output tokens Then the estimated cost should be 0.025 @unit @cost Scenario: CostEntry rejects negative token counts Given I import the CostEntry class When I create a CostEntry with input cost 0.00001 and output cost 0.00003 Then estimating cost with negative input tokens raises ValueError And estimating cost with negative output tokens raises ValueError # ---- ProviderCostTable ---- @unit @cost Scenario: ProviderCostTable returns default cost for unknown model Given I create a ProviderCostTable with defaults When I look up cost for provider "openai" model "unknown-model-xyz" Then the cost entry should be the default cost @unit @cost Scenario: ProviderCostTable returns known cost for gpt-4o Given I create a ProviderCostTable with defaults When I look up cost for provider "openai" model "gpt-4o" Then the cost entry input cost should be 0.0000025 And the cost entry output cost should be 0.00001 @unit @cost Scenario: ProviderCostTable with custom entries overrides defaults Given I create a ProviderCostTable with custom entry for "openai" "gpt-4o" When I look up cost for provider "openai" model "gpt-4o" Then the cost entry input cost should be 0.001 And the cost entry output cost should be 0.002 @unit @cost Scenario: ProviderCostTable validates empty provider Given I create a ProviderCostTable with defaults Then looking up cost for empty provider raises ValueError @unit @cost Scenario: ProviderCostTable validates empty model Given I create a ProviderCostTable with defaults Then looking up cost for empty model raises ValueError @unit @cost Scenario: ProviderCostTable lists providers and models Given I create a ProviderCostTable with defaults When I list providers from the cost table Then the provider list should include "openai" And the provider list should include "anthropic" When I list models for provider "openai" Then the model list should include "gpt-4o" @unit @cost Scenario: ProviderCostTable rejects non-dict custom entries When I create a ProviderCostTable with non-dict custom entries Then a TypeError should be raised for ProviderCostTable # ---- CostMetadata ---- @unit @cost Scenario: CostMetadata records usage correctly Given I create a fresh CostMetadata When I record usage of 100 input tokens and 50 output tokens at cost 0.01 for provider "openai" Then total tokens should be 150 And total cost should be 0.01 And provider costs for "openai" should be 0.01 @unit @cost Scenario: CostMetadata accumulates multiple usages Given I create a fresh CostMetadata When I record usage of 100 input tokens and 50 output tokens at cost 0.01 for provider "openai" And I record usage of 200 input tokens and 100 output tokens at cost 0.02 for provider "anthropic" Then total tokens should be 450 And total cost should be 0.03 And provider costs for "openai" should be 0.01 And provider costs for "anthropic" should be 0.02 @unit @cost Scenario: CostMetadata rejects negative input tokens Given I create a fresh CostMetadata Then recording usage with negative input tokens raises ValueError @unit @cost Scenario: CostMetadata rejects negative cost Given I create a fresh CostMetadata Then recording usage with negative cost raises ValueError @unit @cost Scenario: CostMetadata rejects empty provider Given I create a fresh CostMetadata Then recording usage with empty provider raises ValueError @unit @cost Scenario: CostMetadata display dict includes all fields Given I create a fresh CostMetadata When I record usage of 100 input tokens and 50 output tokens at cost 0.01 for provider "openai" And I get the display dict from cost metadata Then the display dict should have key "total_tokens" And the display dict should have key "total_cost_usd" @unit @cost Scenario: CostMetadata display dict includes budget remaining when set Given I create a fresh CostMetadata When I set budget remaining to 5.0 And I get the display dict from cost metadata Then the display dict should have key "budget_remaining_usd" # ---- BudgetExhaustionEvent ---- @unit @cost Scenario: BudgetExhaustionEvent validates budget type Given I import the BudgetExhaustionEvent class When I create a BudgetExhaustionEvent with invalid budget type Then a validation error should be raised for BudgetExhaustionEvent @unit @cost Scenario: BudgetExhaustionEvent accepts plan budget type Given I import the BudgetExhaustionEvent class When I create a BudgetExhaustionEvent with budget type "plan" Then the event should be created successfully @unit @cost Scenario: BudgetExhaustionEvent accepts daily budget type Given I import the BudgetExhaustionEvent class When I create a BudgetExhaustionEvent with budget type "daily" Then the event should be created successfully # ---- CostTracker ---- @unit @cost Scenario: CostTracker with no budget limits Given I create a CostTracker with no limits When I record tracked usage of 1000 input and 500 output for "openai" "gpt-4o" Then the budget check should be under budget And the cost metadata total tokens should be 1500 @unit @cost Scenario: CostTracker warns at 90% of plan budget Given I create a CostTracker with plan budget 0.001 And I create a fresh CostMetadata When I record tracked usage of 500 input and 500 output for "openai" "gpt-4o" Then the budget check should be warning or exceeded @unit @cost Scenario: CostTracker blocks at 100% of plan budget Given I create a CostTracker with plan budget 0.0000001 And I create a fresh CostMetadata When I record tracked usage of 10000 input and 5000 output for "openai" "gpt-4o" Then the budget check should be exceeded @unit @cost Scenario: CostTracker blocks at 100% of daily budget Given I create a CostTracker with daily budget 0.0000001 And I create a fresh CostMetadata When I record tracked usage of 10000 input and 5000 output for "openai" "gpt-4o" Then the budget check should be exceeded @unit @cost Scenario: CostTracker persists exhaustion events Given I create a CostTracker with plan budget 0.0000001 And I create a fresh CostMetadata When I record tracked usage of 10000 input and 5000 output for "openai" "gpt-4o" Then the cost metadata should have budget exhaustion events @unit @cost Scenario: CostTracker validates negative plan budget When I create a CostTracker with negative plan budget Then a ValueError should be raised for CostTracker @unit @cost Scenario: CostTracker validates negative daily budget When I create a CostTracker with negative daily budget Then a ValueError should be raised for CostTracker @unit @cost Scenario: CostTracker estimate_cost delegates to cost table Given I create a CostTracker with no limits When I estimate cost for "openai" "gpt-4o" with 1000 input and 500 output Then the estimated tracker cost should be positive @unit @cost Scenario: CostTracker validates empty provider in estimate Given I create a CostTracker with no limits Then estimating cost with empty provider raises ValueError on tracker @unit @cost Scenario: CostTracker validates empty model in estimate Given I create a CostTracker with no limits Then estimating cost with empty model raises ValueError on tracker @unit @cost Scenario: CostTracker validates negative tokens in record Given I create a CostTracker with no limits And I create a fresh CostMetadata Then recording tracked usage with negative input raises ValueError @unit @cost Scenario: CostTracker check_plan_budget with no limit returns under budget Given I create a CostTracker with no limits And I create a fresh CostMetadata When I check plan budget Then the plan budget check should be under budget @unit @cost Scenario: CostTracker check_daily_budget with no limit returns under budget Given I create a CostTracker with no limits When I check daily budget Then the daily budget check should be under budget @unit @cost Scenario: CostTracker get_daily_spend returns zero initially Given I create a CostTracker with no limits Then the daily spend should be 0.0 @unit @cost Scenario: CostTracker budget properties are accessible Given I create a CostTracker with plan budget 10.0 and daily budget 50.0 Then the tracker plan budget should be 10.0 And the tracker daily budget should be 50.0 @unit @cost Scenario: CostTracker get_cost_entry validates empty provider Given I create a CostTracker with no limits Then getting cost entry with empty provider raises ValueError @unit @cost Scenario: CostTracker get_cost_entry validates empty model Given I create a CostTracker with no limits Then getting cost entry with empty model raises ValueError @unit @cost Scenario: CostTracker validates empty provider in record_usage Given I create a CostTracker with no limits And I create a fresh CostMetadata Then recording tracked usage with empty provider raises ValueError @unit @cost Scenario: CostTracker validates empty model in record_usage Given I create a CostTracker with no limits And I create a fresh CostMetadata Then recording tracked usage with empty model raises ValueError @unit @cost Scenario: CostTracker with zero plan budget returns exceeded Given I create a CostTracker with plan budget 0.0 And I create a fresh CostMetadata When I check plan budget Then the plan budget check should be exceeded @unit @cost Scenario: CostTracker with zero daily budget returns exceeded Given I create a CostTracker with daily budget 0.0 When I check daily budget Then the daily budget check should be exceeded # ---- CostTracker concurrency ---- @unit @cost @concurrency Scenario: CostTracker record_usage is thread-safe under concurrent calls Given I create a CostTracker with no limits When 20 threads concurrently record usage of 100 input and 50 output for "openai" "gpt-4o" Then the daily spend should equal the sum of all concurrent costs # ---- FallbackSelector ---- @unit @cost Scenario: FallbackSelector with no configured providers returns None Given I create a ProviderRegistry with no API keys for fallback When I create a FallbackSelector with the registry And I select a fallback provider Then the fallback result provider should be None And the fallback result should have skipped entries @unit @cost Scenario: FallbackSelector skips providers missing tool calls Given I create a ProviderRegistry with mock provider for fallback When I create a FallbackSelector with the registry And I select a fallback provider requiring tool calls Then the fallback result should skip mock for missing tool calls @unit @cost Scenario: FallbackSelector selects first capable provider Given I create a ProviderRegistry with openai key for fallback When I create a FallbackSelector with the registry And I select a fallback provider requiring tool calls Then the fallback result provider type should be "openai" @unit @cost Scenario: FallbackSelector uses custom fallback order Given I create a ProviderRegistry with openai key for fallback When I create a FallbackSelector with custom order "openai" And I select a fallback provider Then the fallback result provider type should be "openai" @unit @cost Scenario: FallbackSelector rejects None registry When I create a FallbackSelector with None registry Then a TypeError should be raised for FallbackSelector @unit @cost Scenario: FallbackSelector rejects empty string in fallback list Given I create a ProviderRegistry with no API keys for fallback When I create a FallbackSelector with empty string in fallback list Then a ValueError should be raised for FallbackSelector @unit @cost Scenario: FallbackSelector skips provider when daily budget exceeded Given I create a ProviderRegistry with openai key for fallback And I create a CostTracker with daily budget 0.0 When I create a FallbackSelector with cost tracker And I select a fallback provider Then the fallback result provider should be None # ---- Settings config keys ---- @unit @cost Scenario: Settings budget_per_plan defaults to None When I load cost control settings with defaults Then budget_per_plan should be None @unit @cost Scenario: Settings budget_per_day defaults to None When I load cost control settings with defaults Then budget_per_day should be None @unit @cost Scenario: Settings fallback_providers defaults to empty list When I load cost control settings with defaults Then fallback_providers should be an empty list @unit @cost Scenario: Settings budget_per_plan accepts positive value When I load cost control settings with budget_per_plan 10.0 Then budget_per_plan should be 10.0 @unit @cost Scenario: Settings budget_per_day accepts positive value When I load cost control settings with budget_per_day 50.0 Then budget_per_day should be 50.0 # ---- Plan model cost_metadata ---- @unit @cost Scenario: Plan model includes cost_metadata field Given I create a minimal Plan with cost metadata Then the plan should have cost_metadata set @unit @cost Scenario: Plan as_cli_dict includes cost when present Given I create a minimal Plan with cost metadata When I get the CLI dict from the plan Then the CLI dict should have a "cost" key @unit @cost Scenario: Plan as_cli_dict omits cost when None Given I create a minimal Plan without cost metadata When I get the CLI dict from the plan Then the CLI dict should not have a "cost" key @unit @cost Scenario: CostMetadata display dict with exhaustion events Given I create a fresh CostMetadata When I add a budget exhaustion event of type "plan" And I get the display dict from cost metadata Then the display dict should have key "budget_exhaustion_events" @unit @cost Scenario: CostMetadata records negative output tokens rejection Given I create a fresh CostMetadata Then recording usage with negative output tokens raises ValueError # ---- Additional coverage scenarios ---- @unit @cost Scenario: ProviderCostTable custom entries for brand new provider Given I create a ProviderCostTable with custom entry for new provider "myprovider" model "mymodel" When I look up cost for provider "myprovider" model "mymodel" Then the cost entry input cost should be 0.005 And the cost entry output cost should be 0.01 @unit @cost Scenario: ProviderCostTable list_models with empty provider raises ValueError Given I create a ProviderCostTable with defaults Then listing models for empty provider raises ValueError @unit @cost Scenario: CostTracker estimate_cost rejects negative input tokens Given I create a CostTracker with no limits Then estimating cost with negative input tokens raises ValueError on tracker @unit @cost Scenario: CostTracker estimate_cost rejects negative output tokens Given I create a CostTracker with no limits Then estimating cost with negative output tokens raises ValueError on tracker @unit @cost Scenario: CostTracker record_usage rejects negative output tokens Given I create a CostTracker with no limits And I create a fresh CostMetadata Then recording tracked usage with negative output raises ValueError @unit @cost Scenario: CostTracker plan budget warning at 90 percent Given I create a CostTracker with plan budget 0.10 And I create a fresh CostMetadata When I record tracked usage of 3600 input and 8500 output for "openai" "gpt-4o" Then the budget check should be warning @unit @cost Scenario: CostTracker daily budget warning at 90 percent Given I create a CostTracker with daily budget 0.10 And I create a fresh CostMetadata When I record tracked usage of 3600 input and 8500 output for "openai" "gpt-4o" Then the budget check should be warning @unit @cost Scenario: CostTracker get_cost_entry returns valid entry Given I create a CostTracker with no limits When I get cost entry for "openai" "gpt-4o" Then the returned cost entry should have positive input cost @unit @cost Scenario: FallbackSelector skips unknown provider types Given I create a ProviderRegistry with no API keys for fallback When I create a FallbackSelector with custom order "nonexistent_provider" And I select a fallback provider Then the fallback result provider should be None And the fallback skipped should mention "unknown provider type" @unit @cost Scenario: FallbackSelector skips providers missing streaming Given I create a ProviderRegistry with mock provider for fallback When I create a FallbackSelector with the registry And I select a fallback provider requiring streaming Then the fallback result should skip for missing streaming @unit @cost Scenario: FallbackSelector skips providers missing vision Given I create a ProviderRegistry with mock provider for fallback When I create a FallbackSelector with the registry And I select a fallback provider requiring vision Then the fallback result should skip for missing vision @unit @cost Scenario: FallbackSelector skips providers missing json mode Given I create a ProviderRegistry with mock provider for fallback When I create a FallbackSelector with the registry And I select a fallback provider requiring json mode Then the fallback result should skip for missing json mode @unit @cost @tdd_issue @tdd_issue_10471 Scenario: FallbackSelector skips provider when per-plan budget exceeded Given I create a ProviderRegistry with openai key for fallback And I create a CostTracker with plan budget 0.0 And I create a CostMetadata with total cost 1.0 When I create a FallbackSelector with cost tracker and cost metadata And I select a fallback provider Then the fallback result provider should be None And the fallback skipped should mention "plan budget exceeded" @unit @cost @tdd_issue @tdd_issue_10471 Scenario: FallbackSelector selects provider when per-plan budget not exceeded Given I create a ProviderRegistry with openai key for fallback And I create a CostTracker with plan budget 10.0 And I create a CostMetadata with total cost 1.0 When I create a FallbackSelector with cost tracker and cost metadata And I select a fallback provider Then the fallback result provider type should be "openai"