freemo
134dce896a
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 17s
CI / lint (pull_request) Successful in 18s
CI / quality (pull_request) Successful in 20s
CI / security (pull_request) Successful in 34s
CI / typecheck (pull_request) Successful in 37s
CI / integration_tests (pull_request) Successful in 3m46s
CI / unit_tests (pull_request) Successful in 8m32s
CI / docker (pull_request) Successful in 39s
CI / benchmark-regression (pull_request) Successful in 19m26s
CI / coverage (pull_request) Successful in 31m23s
CI / lint (push) Successful in 13s
CI / build (push) Successful in 15s
CI / quality (push) Successful in 16s
CI / typecheck (push) Successful in 30s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 33s
CI / integration_tests (push) Successful in 3m44s
CI / unit_tests (push) Successful in 8m53s
CI / docker (push) Successful in 1m1s
CI / benchmark-publish (push) Successful in 10m35s
CI / coverage (push) Successful in 37m55s
Add token/cost tracking, budget enforcement (per-plan and per-day), provider fallback selection with capability filtering, and cost metadata for plan models. New modules: - providers/cost_table.py: CostEntry + ProviderCostTable with default pricing for OpenAI, Anthropic, Google, Groq, Together, Cohere, Mock - providers/cost_tracker.py: BudgetStatus enum, BudgetCheckResult, CostTracker with warn-at-90%/block-at-100% enforcement - providers/fallback_selector.py: FallbackSelector with capability filtering (tool_calls, streaming, vision, json_mode) and budget - domain/models/core/cost_metadata.py: CostMetadata + BudgetExhaustionEvent Modified: - config/settings.py: budget_per_plan, budget_per_day, fallback_providers - domain/models/core/plan.py: cost_metadata field + CLI dict surfacing - providers/__init__.py: public exports for new classes Testing: - 71 Behave scenarios in features/cost_controls.feature (all pass) - 6 Robot Framework integration tests (all pass) - ASV benchmarks in benchmarks/cost_controls_bench.py - nox lint, typecheck, format, docs, build, dead_code, security_scan pass Closes #324
112 lines
3.1 KiB
Markdown
112 lines
3.1 KiB
Markdown
# Cost Controls
|
|
|
|
CleverAgents provides cost tracking, budget enforcement, and provider
|
|
fallback controls for AI provider usage.
|
|
|
|
## Configuration Keys
|
|
|
|
| Key | Env Variable | Type | Default | Description |
|
|
|-----|-------------|------|---------|-------------|
|
|
| `budget_per_plan` | `CLEVERAGENTS_BUDGET_PER_PLAN` | `float \| None` | `None` | Max USD spend per plan execution. `None` = unlimited. |
|
|
| `budget_per_day` | `CLEVERAGENTS_BUDGET_PER_DAY` | `float \| None` | `None` | Max USD spend per calendar day. `None` = unlimited. |
|
|
| `fallback_providers` | `CLEVERAGENTS_FALLBACK_PROVIDERS` | `list[str]` | `[]` | Ordered provider fallback list. Empty uses default order. |
|
|
|
|
## Budget Thresholds
|
|
|
|
| Threshold | Value | Behavior |
|
|
|-----------|-------|----------|
|
|
| Warning | 90% of limit | Log warning, continue execution |
|
|
| Block | 100% of limit | Block further provider calls, persist exhaustion event |
|
|
|
|
## Default Fallback Order
|
|
|
|
When `fallback_providers` is empty, the default order is:
|
|
|
|
1. OpenAI
|
|
2. Anthropic
|
|
3. Google
|
|
4. Azure
|
|
5. OpenRouter
|
|
6. Groq
|
|
7. Together
|
|
8. Cohere
|
|
|
|
Providers are skipped if they:
|
|
|
|
- Are not configured (no API key)
|
|
- Lack required capabilities (tool calling, streaming, vision, JSON mode)
|
|
- Would exceed the daily budget
|
|
|
|
## Cost Tracking Fields
|
|
|
|
Plan execution metadata includes:
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `total_tokens` | `int` | Total tokens consumed (input + output) |
|
|
| `input_tokens` | `int` | Input/prompt tokens |
|
|
| `output_tokens` | `int` | Output/completion tokens |
|
|
| `total_cost` | `float` | Estimated cost in USD |
|
|
| `budget_remaining` | `float \| None` | Remaining plan budget (None if unlimited) |
|
|
| `provider_costs` | `dict[str, float]` | Per-provider cost breakdown |
|
|
| `budget_exhaustion_events` | `list` | Audit log of budget limit events |
|
|
|
|
## Viewing Costs
|
|
|
|
Use `agents plan status` to see cost data for a plan:
|
|
|
|
```
|
|
agents plan status <plan_id>
|
|
```
|
|
|
|
Cost data appears under the `cost` key in the output.
|
|
|
|
## Per-Provider Cost Table
|
|
|
|
Default token cost estimates are provided for offline reporting.
|
|
Models not in the table use a conservative default estimate.
|
|
|
|
### Supported Providers
|
|
|
|
- **OpenAI**: gpt-4o, gpt-4o-mini, gpt-4-turbo
|
|
- **Anthropic**: claude-sonnet-4-20250514, claude-3-5-haiku-20241022, claude-opus-4-20250514
|
|
- **Google**: gemini-2.0-flash, gemini-1.5-pro
|
|
- **Groq**: llama-3.1-70b-versatile
|
|
- **Together**: meta-llama/Llama-3.1-70B-Instruct-Turbo
|
|
- **Cohere**: command-r-plus
|
|
- **Mock**: mock-gpt (zero cost)
|
|
|
|
## Budget Exhaustion Events
|
|
|
|
When a budget limit is reached, an exhaustion event is persisted to the
|
|
plan's cost metadata with:
|
|
|
|
- Timestamp
|
|
- Budget type (`plan` or `daily`)
|
|
- Limit amount
|
|
- Used amount
|
|
- Provider and model that triggered it
|
|
|
|
These events are retained for auditability and appear in `plan status`
|
|
output.
|
|
|
|
## Examples
|
|
|
|
### Set a per-plan budget
|
|
|
|
```bash
|
|
export CLEVERAGENTS_BUDGET_PER_PLAN=5.00
|
|
```
|
|
|
|
### Set a daily budget
|
|
|
|
```bash
|
|
export CLEVERAGENTS_BUDGET_PER_DAY=50.00
|
|
```
|
|
|
|
### Configure fallback providers
|
|
|
|
```bash
|
|
export CLEVERAGENTS_FALLBACK_PROVIDERS='["anthropic", "openai", "groq"]'
|
|
```
|