freemo
134dce896a
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 17s
CI / lint (pull_request) Successful in 18s
CI / quality (pull_request) Successful in 20s
CI / security (pull_request) Successful in 34s
CI / typecheck (pull_request) Successful in 37s
CI / integration_tests (pull_request) Successful in 3m46s
CI / unit_tests (pull_request) Successful in 8m32s
CI / docker (pull_request) Successful in 39s
CI / benchmark-regression (pull_request) Successful in 19m26s
CI / coverage (pull_request) Successful in 31m23s
CI / lint (push) Successful in 13s
CI / build (push) Successful in 15s
CI / quality (push) Successful in 16s
CI / typecheck (push) Successful in 30s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 33s
CI / integration_tests (push) Successful in 3m44s
CI / unit_tests (push) Successful in 8m53s
CI / docker (push) Successful in 1m1s
CI / benchmark-publish (push) Successful in 10m35s
CI / coverage (push) Successful in 37m55s
Add token/cost tracking, budget enforcement (per-plan and per-day), provider fallback selection with capability filtering, and cost metadata for plan models. New modules: - providers/cost_table.py: CostEntry + ProviderCostTable with default pricing for OpenAI, Anthropic, Google, Groq, Together, Cohere, Mock - providers/cost_tracker.py: BudgetStatus enum, BudgetCheckResult, CostTracker with warn-at-90%/block-at-100% enforcement - providers/fallback_selector.py: FallbackSelector with capability filtering (tool_calls, streaming, vision, json_mode) and budget - domain/models/core/cost_metadata.py: CostMetadata + BudgetExhaustionEvent Modified: - config/settings.py: budget_per_plan, budget_per_day, fallback_providers - domain/models/core/plan.py: cost_metadata field + CLI dict surfacing - providers/__init__.py: public exports for new classes Testing: - 71 Behave scenarios in features/cost_controls.feature (all pass) - 6 Robot Framework integration tests (all pass) - ASV benchmarks in benchmarks/cost_controls_bench.py - nox lint, typecheck, format, docs, build, dead_code, security_scan pass Closes #324
464 lines
18 KiB
Gherkin
464 lines
18 KiB
Gherkin
Feature: Cost controls and provider fallback
|
|
As a developer using CleverAgents
|
|
I want cost tracking, budget enforcement, and provider fallback
|
|
So that I can control AI spending and ensure service continuity
|
|
|
|
# ---- CostEntry ----
|
|
|
|
@unit @cost
|
|
Scenario: CostEntry validates non-negative input cost
|
|
Given I import the CostEntry class
|
|
When I create a CostEntry with negative input cost
|
|
Then a ValueError should be raised for CostEntry
|
|
|
|
@unit @cost
|
|
Scenario: CostEntry validates non-negative output cost
|
|
Given I import the CostEntry class
|
|
When I create a CostEntry with negative output cost
|
|
Then a ValueError should be raised for CostEntry
|
|
|
|
@unit @cost
|
|
Scenario: CostEntry estimates cost correctly
|
|
Given I import the CostEntry class
|
|
When I create a CostEntry with input cost 0.00001 and output cost 0.00003
|
|
And I estimate cost for 1000 input tokens and 500 output tokens
|
|
Then the estimated cost should be 0.025
|
|
|
|
@unit @cost
|
|
Scenario: CostEntry rejects negative token counts
|
|
Given I import the CostEntry class
|
|
When I create a CostEntry with input cost 0.00001 and output cost 0.00003
|
|
Then estimating cost with negative input tokens raises ValueError
|
|
And estimating cost with negative output tokens raises ValueError
|
|
|
|
# ---- ProviderCostTable ----
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable returns default cost for unknown model
|
|
Given I create a ProviderCostTable with defaults
|
|
When I look up cost for provider "openai" model "unknown-model-xyz"
|
|
Then the cost entry should be the default cost
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable returns known cost for gpt-4o
|
|
Given I create a ProviderCostTable with defaults
|
|
When I look up cost for provider "openai" model "gpt-4o"
|
|
Then the cost entry input cost should be 0.0000025
|
|
And the cost entry output cost should be 0.00001
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable with custom entries overrides defaults
|
|
Given I create a ProviderCostTable with custom entry for "openai" "gpt-4o"
|
|
When I look up cost for provider "openai" model "gpt-4o"
|
|
Then the cost entry input cost should be 0.001
|
|
And the cost entry output cost should be 0.002
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable validates empty provider
|
|
Given I create a ProviderCostTable with defaults
|
|
Then looking up cost for empty provider raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable validates empty model
|
|
Given I create a ProviderCostTable with defaults
|
|
Then looking up cost for empty model raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable lists providers and models
|
|
Given I create a ProviderCostTable with defaults
|
|
When I list providers from the cost table
|
|
Then the provider list should include "openai"
|
|
And the provider list should include "anthropic"
|
|
When I list models for provider "openai"
|
|
Then the model list should include "gpt-4o"
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable rejects non-dict custom entries
|
|
When I create a ProviderCostTable with non-dict custom entries
|
|
Then a TypeError should be raised for ProviderCostTable
|
|
|
|
# ---- CostMetadata ----
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata records usage correctly
|
|
Given I create a fresh CostMetadata
|
|
When I record usage of 100 input tokens and 50 output tokens at cost 0.01 for provider "openai"
|
|
Then total tokens should be 150
|
|
And total cost should be 0.01
|
|
And provider costs for "openai" should be 0.01
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata accumulates multiple usages
|
|
Given I create a fresh CostMetadata
|
|
When I record usage of 100 input tokens and 50 output tokens at cost 0.01 for provider "openai"
|
|
And I record usage of 200 input tokens and 100 output tokens at cost 0.02 for provider "anthropic"
|
|
Then total tokens should be 450
|
|
And total cost should be 0.03
|
|
And provider costs for "openai" should be 0.01
|
|
And provider costs for "anthropic" should be 0.02
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata rejects negative input tokens
|
|
Given I create a fresh CostMetadata
|
|
Then recording usage with negative input tokens raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata rejects negative cost
|
|
Given I create a fresh CostMetadata
|
|
Then recording usage with negative cost raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata rejects empty provider
|
|
Given I create a fresh CostMetadata
|
|
Then recording usage with empty provider raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata display dict includes all fields
|
|
Given I create a fresh CostMetadata
|
|
When I record usage of 100 input tokens and 50 output tokens at cost 0.01 for provider "openai"
|
|
And I get the display dict from cost metadata
|
|
Then the display dict should have key "total_tokens"
|
|
And the display dict should have key "total_cost_usd"
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata display dict includes budget remaining when set
|
|
Given I create a fresh CostMetadata
|
|
When I set budget remaining to 5.0
|
|
And I get the display dict from cost metadata
|
|
Then the display dict should have key "budget_remaining_usd"
|
|
|
|
# ---- BudgetExhaustionEvent ----
|
|
|
|
@unit @cost
|
|
Scenario: BudgetExhaustionEvent validates budget type
|
|
Given I import the BudgetExhaustionEvent class
|
|
When I create a BudgetExhaustionEvent with invalid budget type
|
|
Then a validation error should be raised for BudgetExhaustionEvent
|
|
|
|
@unit @cost
|
|
Scenario: BudgetExhaustionEvent accepts plan budget type
|
|
Given I import the BudgetExhaustionEvent class
|
|
When I create a BudgetExhaustionEvent with budget type "plan"
|
|
Then the event should be created successfully
|
|
|
|
@unit @cost
|
|
Scenario: BudgetExhaustionEvent accepts daily budget type
|
|
Given I import the BudgetExhaustionEvent class
|
|
When I create a BudgetExhaustionEvent with budget type "daily"
|
|
Then the event should be created successfully
|
|
|
|
# ---- CostTracker ----
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker with no budget limits
|
|
Given I create a CostTracker with no limits
|
|
When I record tracked usage of 1000 input and 500 output for "openai" "gpt-4o"
|
|
Then the budget check should be under budget
|
|
And the cost metadata total tokens should be 1500
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker warns at 90% of plan budget
|
|
Given I create a CostTracker with plan budget 0.001
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 500 input and 500 output for "openai" "gpt-4o"
|
|
Then the budget check should be warning or exceeded
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker blocks at 100% of plan budget
|
|
Given I create a CostTracker with plan budget 0.0000001
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 10000 input and 5000 output for "openai" "gpt-4o"
|
|
Then the budget check should be exceeded
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker blocks at 100% of daily budget
|
|
Given I create a CostTracker with daily budget 0.0000001
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 10000 input and 5000 output for "openai" "gpt-4o"
|
|
Then the budget check should be exceeded
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker persists exhaustion events
|
|
Given I create a CostTracker with plan budget 0.0000001
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 10000 input and 5000 output for "openai" "gpt-4o"
|
|
Then the cost metadata should have budget exhaustion events
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates negative plan budget
|
|
When I create a CostTracker with negative plan budget
|
|
Then a ValueError should be raised for CostTracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates negative daily budget
|
|
When I create a CostTracker with negative daily budget
|
|
Then a ValueError should be raised for CostTracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker estimate_cost delegates to cost table
|
|
Given I create a CostTracker with no limits
|
|
When I estimate cost for "openai" "gpt-4o" with 1000 input and 500 output
|
|
Then the estimated tracker cost should be positive
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates empty provider in estimate
|
|
Given I create a CostTracker with no limits
|
|
Then estimating cost with empty provider raises ValueError on tracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates empty model in estimate
|
|
Given I create a CostTracker with no limits
|
|
Then estimating cost with empty model raises ValueError on tracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates negative tokens in record
|
|
Given I create a CostTracker with no limits
|
|
And I create a fresh CostMetadata
|
|
Then recording tracked usage with negative input raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker check_plan_budget with no limit returns under budget
|
|
Given I create a CostTracker with no limits
|
|
And I create a fresh CostMetadata
|
|
When I check plan budget
|
|
Then the plan budget check should be under budget
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker check_daily_budget with no limit returns under budget
|
|
Given I create a CostTracker with no limits
|
|
When I check daily budget
|
|
Then the daily budget check should be under budget
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker get_daily_spend returns zero initially
|
|
Given I create a CostTracker with no limits
|
|
Then the daily spend should be 0.0
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker budget properties are accessible
|
|
Given I create a CostTracker with plan budget 10.0 and daily budget 50.0
|
|
Then the tracker plan budget should be 10.0
|
|
And the tracker daily budget should be 50.0
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker get_cost_entry validates empty provider
|
|
Given I create a CostTracker with no limits
|
|
Then getting cost entry with empty provider raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker get_cost_entry validates empty model
|
|
Given I create a CostTracker with no limits
|
|
Then getting cost entry with empty model raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates empty provider in record_usage
|
|
Given I create a CostTracker with no limits
|
|
And I create a fresh CostMetadata
|
|
Then recording tracked usage with empty provider raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker validates empty model in record_usage
|
|
Given I create a CostTracker with no limits
|
|
And I create a fresh CostMetadata
|
|
Then recording tracked usage with empty model raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker with zero plan budget returns exceeded
|
|
Given I create a CostTracker with plan budget 0.0
|
|
And I create a fresh CostMetadata
|
|
When I check plan budget
|
|
Then the plan budget check should be exceeded
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker with zero daily budget returns exceeded
|
|
Given I create a CostTracker with daily budget 0.0
|
|
When I check daily budget
|
|
Then the daily budget check should be exceeded
|
|
|
|
# ---- FallbackSelector ----
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector with no configured providers returns None
|
|
Given I create a ProviderRegistry with no API keys for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider
|
|
Then the fallback result provider should be None
|
|
And the fallback result should have skipped entries
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips providers missing tool calls
|
|
Given I create a ProviderRegistry with mock provider for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider requiring tool calls
|
|
Then the fallback result should skip mock for missing tool calls
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector selects first capable provider
|
|
Given I create a ProviderRegistry with openai key for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider requiring tool calls
|
|
Then the fallback result provider type should be "openai"
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector uses custom fallback order
|
|
Given I create a ProviderRegistry with openai key for fallback
|
|
When I create a FallbackSelector with custom order "openai"
|
|
And I select a fallback provider
|
|
Then the fallback result provider type should be "openai"
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector rejects None registry
|
|
When I create a FallbackSelector with None registry
|
|
Then a TypeError should be raised for FallbackSelector
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector rejects empty string in fallback list
|
|
Given I create a ProviderRegistry with no API keys for fallback
|
|
When I create a FallbackSelector with empty string in fallback list
|
|
Then a ValueError should be raised for FallbackSelector
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips provider when daily budget exceeded
|
|
Given I create a ProviderRegistry with openai key for fallback
|
|
And I create a CostTracker with daily budget 0.0
|
|
When I create a FallbackSelector with cost tracker
|
|
And I select a fallback provider
|
|
Then the fallback result provider should be None
|
|
|
|
# ---- Settings config keys ----
|
|
|
|
@unit @cost
|
|
Scenario: Settings budget_per_plan defaults to None
|
|
When I load cost control settings with defaults
|
|
Then budget_per_plan should be None
|
|
|
|
@unit @cost
|
|
Scenario: Settings budget_per_day defaults to None
|
|
When I load cost control settings with defaults
|
|
Then budget_per_day should be None
|
|
|
|
@unit @cost
|
|
Scenario: Settings fallback_providers defaults to empty list
|
|
When I load cost control settings with defaults
|
|
Then fallback_providers should be an empty list
|
|
|
|
@unit @cost
|
|
Scenario: Settings budget_per_plan accepts positive value
|
|
When I load cost control settings with budget_per_plan 10.0
|
|
Then budget_per_plan should be 10.0
|
|
|
|
@unit @cost
|
|
Scenario: Settings budget_per_day accepts positive value
|
|
When I load cost control settings with budget_per_day 50.0
|
|
Then budget_per_day should be 50.0
|
|
|
|
# ---- Plan model cost_metadata ----
|
|
|
|
@unit @cost
|
|
Scenario: Plan model includes cost_metadata field
|
|
Given I create a minimal Plan with cost metadata
|
|
Then the plan should have cost_metadata set
|
|
|
|
@unit @cost
|
|
Scenario: Plan as_cli_dict includes cost when present
|
|
Given I create a minimal Plan with cost metadata
|
|
When I get the CLI dict from the plan
|
|
Then the CLI dict should have a "cost" key
|
|
|
|
@unit @cost
|
|
Scenario: Plan as_cli_dict omits cost when None
|
|
Given I create a minimal Plan without cost metadata
|
|
When I get the CLI dict from the plan
|
|
Then the CLI dict should not have a "cost" key
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata display dict with exhaustion events
|
|
Given I create a fresh CostMetadata
|
|
When I add a budget exhaustion event of type "plan"
|
|
And I get the display dict from cost metadata
|
|
Then the display dict should have key "budget_exhaustion_events"
|
|
|
|
@unit @cost
|
|
Scenario: CostMetadata records negative output tokens rejection
|
|
Given I create a fresh CostMetadata
|
|
Then recording usage with negative output tokens raises ValueError
|
|
|
|
# ---- Additional coverage scenarios ----
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable custom entries for brand new provider
|
|
Given I create a ProviderCostTable with custom entry for new provider "myprovider" model "mymodel"
|
|
When I look up cost for provider "myprovider" model "mymodel"
|
|
Then the cost entry input cost should be 0.005
|
|
And the cost entry output cost should be 0.01
|
|
|
|
@unit @cost
|
|
Scenario: ProviderCostTable list_models with empty provider raises ValueError
|
|
Given I create a ProviderCostTable with defaults
|
|
Then listing models for empty provider raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker estimate_cost rejects negative input tokens
|
|
Given I create a CostTracker with no limits
|
|
Then estimating cost with negative input tokens raises ValueError on tracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker estimate_cost rejects negative output tokens
|
|
Given I create a CostTracker with no limits
|
|
Then estimating cost with negative output tokens raises ValueError on tracker
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker record_usage rejects negative output tokens
|
|
Given I create a CostTracker with no limits
|
|
And I create a fresh CostMetadata
|
|
Then recording tracked usage with negative output raises ValueError
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker plan budget warning at 90 percent
|
|
Given I create a CostTracker with plan budget 0.10
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 3600 input and 8500 output for "openai" "gpt-4o"
|
|
Then the budget check should be warning
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker daily budget warning at 90 percent
|
|
Given I create a CostTracker with daily budget 0.10
|
|
And I create a fresh CostMetadata
|
|
When I record tracked usage of 3600 input and 8500 output for "openai" "gpt-4o"
|
|
Then the budget check should be warning
|
|
|
|
@unit @cost
|
|
Scenario: CostTracker get_cost_entry returns valid entry
|
|
Given I create a CostTracker with no limits
|
|
When I get cost entry for "openai" "gpt-4o"
|
|
Then the returned cost entry should have positive input cost
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips unknown provider types
|
|
Given I create a ProviderRegistry with no API keys for fallback
|
|
When I create a FallbackSelector with custom order "nonexistent_provider"
|
|
And I select a fallback provider
|
|
Then the fallback result provider should be None
|
|
And the fallback skipped should mention "unknown provider type"
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips providers missing streaming
|
|
Given I create a ProviderRegistry with mock provider for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider requiring streaming
|
|
Then the fallback result should skip for missing streaming
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips providers missing vision
|
|
Given I create a ProviderRegistry with mock provider for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider requiring vision
|
|
Then the fallback result should skip for missing vision
|
|
|
|
@unit @cost
|
|
Scenario: FallbackSelector skips providers missing json mode
|
|
Given I create a ProviderRegistry with mock provider for fallback
|
|
When I create a FallbackSelector with the registry
|
|
And I select a fallback provider requiring json mode
|
|
Then the fallback result should skip for missing json mode
|