freemo
134dce896a
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 17s
CI / lint (pull_request) Successful in 18s
CI / quality (pull_request) Successful in 20s
CI / security (pull_request) Successful in 34s
CI / typecheck (pull_request) Successful in 37s
CI / integration_tests (pull_request) Successful in 3m46s
CI / unit_tests (pull_request) Successful in 8m32s
CI / docker (pull_request) Successful in 39s
CI / benchmark-regression (pull_request) Successful in 19m26s
CI / coverage (pull_request) Successful in 31m23s
CI / lint (push) Successful in 13s
CI / build (push) Successful in 15s
CI / quality (push) Successful in 16s
CI / typecheck (push) Successful in 30s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 33s
CI / integration_tests (push) Successful in 3m44s
CI / unit_tests (push) Successful in 8m53s
CI / docker (push) Successful in 1m1s
CI / benchmark-publish (push) Successful in 10m35s
CI / coverage (push) Successful in 37m55s
Add token/cost tracking, budget enforcement (per-plan and per-day), provider fallback selection with capability filtering, and cost metadata for plan models. New modules: - providers/cost_table.py: CostEntry + ProviderCostTable with default pricing for OpenAI, Anthropic, Google, Groq, Together, Cohere, Mock - providers/cost_tracker.py: BudgetStatus enum, BudgetCheckResult, CostTracker with warn-at-90%/block-at-100% enforcement - providers/fallback_selector.py: FallbackSelector with capability filtering (tool_calls, streaming, vision, json_mode) and budget - domain/models/core/cost_metadata.py: CostMetadata + BudgetExhaustionEvent Modified: - config/settings.py: budget_per_plan, budget_per_day, fallback_providers - domain/models/core/plan.py: cost_metadata field + CLI dict surfacing - providers/__init__.py: public exports for new classes Testing: - 71 Behave scenarios in features/cost_controls.feature (all pass) - 6 Robot Framework integration tests (all pass) - ASV benchmarks in benchmarks/cost_controls_bench.py - nox lint, typecheck, format, docs, build, dead_code, security_scan pass Closes #324
3.1 KiB
3.1 KiB
Cost Controls
CleverAgents provides cost tracking, budget enforcement, and provider fallback controls for AI provider usage.
Configuration Keys
| Key | Env Variable | Type | Default | Description |
|---|---|---|---|---|
budget_per_plan |
CLEVERAGENTS_BUDGET_PER_PLAN |
float | None |
None |
Max USD spend per plan execution. None = unlimited. |
budget_per_day |
CLEVERAGENTS_BUDGET_PER_DAY |
float | None |
None |
Max USD spend per calendar day. None = unlimited. |
fallback_providers |
CLEVERAGENTS_FALLBACK_PROVIDERS |
list[str] |
[] |
Ordered provider fallback list. Empty uses default order. |
Budget Thresholds
| Threshold | Value | Behavior |
|---|---|---|
| Warning | 90% of limit | Log warning, continue execution |
| Block | 100% of limit | Block further provider calls, persist exhaustion event |
Default Fallback Order
When fallback_providers is empty, the default order is:
- OpenAI
- Anthropic
- Azure
- OpenRouter
- Groq
- Together
- Cohere
Providers are skipped if they:
- Are not configured (no API key)
- Lack required capabilities (tool calling, streaming, vision, JSON mode)
- Would exceed the daily budget
Cost Tracking Fields
Plan execution metadata includes:
| Field | Type | Description |
|---|---|---|
total_tokens |
int |
Total tokens consumed (input + output) |
input_tokens |
int |
Input/prompt tokens |
output_tokens |
int |
Output/completion tokens |
total_cost |
float |
Estimated cost in USD |
budget_remaining |
float | None |
Remaining plan budget (None if unlimited) |
provider_costs |
dict[str, float] |
Per-provider cost breakdown |
budget_exhaustion_events |
list |
Audit log of budget limit events |
Viewing Costs
Use agents plan status to see cost data for a plan:
agents plan status <plan_id>
Cost data appears under the cost key in the output.
Per-Provider Cost Table
Default token cost estimates are provided for offline reporting. Models not in the table use a conservative default estimate.
Supported Providers
- OpenAI: gpt-4o, gpt-4o-mini, gpt-4-turbo
- Anthropic: claude-sonnet-4-20250514, claude-3-5-haiku-20241022, claude-opus-4-20250514
- Google: gemini-2.0-flash, gemini-1.5-pro
- Groq: llama-3.1-70b-versatile
- Together: meta-llama/Llama-3.1-70B-Instruct-Turbo
- Cohere: command-r-plus
- Mock: mock-gpt (zero cost)
Budget Exhaustion Events
When a budget limit is reached, an exhaustion event is persisted to the plan's cost metadata with:
- Timestamp
- Budget type (
planordaily) - Limit amount
- Used amount
- Provider and model that triggered it
These events are retained for auditability and appear in plan status
output.
Examples
Set a per-plan budget
export CLEVERAGENTS_BUDGET_PER_PLAN=5.00
Set a daily budget
export CLEVERAGENTS_BUDGET_PER_DAY=50.00
Configure fallback providers
export CLEVERAGENTS_FALLBACK_PROVIDERS='["anthropic", "openai", "groq"]'