HAL9000
4699d29057
CI / push-validation (pull_request) Successful in 29s
CI / lint (pull_request) Successful in 40s
CI / helm (pull_request) Successful in 44s
CI / quality (pull_request) Successful in 47s
CI / build (pull_request) Successful in 48s
CI / typecheck (pull_request) Successful in 1m11s
CI / security (pull_request) Successful in 1m13s
CI / unit_tests (pull_request) Failing after 5m29s
CI / coverage (pull_request) Has been skipped
CI / docker (pull_request) Has been skipped
CI / integration_tests (pull_request) Successful in 9m43s
CI / status-check (pull_request) Failing after 3s
- Remove triple blank line in steps file (ruff format E303)
- Collapse multiline if conditions in cost_tracking.py (ruff format)
- Add BudgetExceededError to __all__ in exceptions.py
- Fix step_check_budget to pass context.org_accumulator instead of None
- Fix Three-tier scenario amounts so org budget fails first (session $200, org $100)
- Disambiguate @when step using {cost:f} type specifier to avoid AmbiguousStep
- Rename duplicate @then step to avoid conflict with lsp_registry_steps.py
- Add missing column header rows to Gherkin tables in feature file
ISSUES CLOSED: #8609
84 lines
2.9 KiB
Gherkin
84 lines
2.9 KiB
Gherkin
Feature: Cost and Session Budget Tracking and Enforcement
|
|
|
|
Background:
|
|
Given a cost tracking service is initialized
|
|
And a session cost budget of $10.00 is configured
|
|
And a cost metadata tracker is created
|
|
|
|
Scenario: Track cost usage after LLM call
|
|
When an LLM call costs $2.50 with 100 input tokens and 50 output tokens
|
|
Then the cost metadata should show:
|
|
| field | value |
|
|
| total_cost | 2.50 |
|
|
| total_tokens | 150 |
|
|
| input_tokens | 100 |
|
|
| output_tokens | 50 |
|
|
And the session budget should show $7.50 remaining
|
|
|
|
Scenario: Enforce budget limit when exceeded
|
|
Given the session budget is $5.00
|
|
When an LLM call costs $3.00
|
|
And another LLM call costs $2.50
|
|
Then the second call should raise BudgetExceededError
|
|
And the error should indicate plan budget exceeded
|
|
|
|
Scenario: Track provider costs separately
|
|
When an LLM call from OpenAI costs $1.50
|
|
And an LLM call from Anthropic costs $2.00
|
|
Then the cost metadata should show provider breakdown:
|
|
| provider | cost |
|
|
| openai | 1.50 |
|
|
| anthropic | 2.00 |
|
|
|
|
Scenario: Query budget status
|
|
Given the session budget is $10.00
|
|
When $3.50 has been spent
|
|
Then budget status should show:
|
|
| field | value |
|
|
| limit | 10.00 |
|
|
| used | 3.50 |
|
|
| remaining | 6.50 |
|
|
| utilization | 0.35 |
|
|
|
|
Scenario: Warn at 90% budget utilization
|
|
Given the session budget is $10.00
|
|
When $9.00 has been spent
|
|
And a check for $0.50 cost is performed
|
|
Then the budget check should return warning=true
|
|
And the check should still be allowed
|
|
|
|
Scenario: Record budget exhaustion events
|
|
Given the session budget is $5.00
|
|
When an LLM call costs $5.50 from OpenAI using gpt-4o
|
|
Then a BudgetExceededError should be raised
|
|
And the cost metadata should record an exhaustion event with:
|
|
| field | value |
|
|
| budget_type | session |
|
|
| limit | 5.00 |
|
|
| used | 5.50 |
|
|
| provider | openai |
|
|
| model | gpt-4o |
|
|
|
|
Scenario: Handle unlimited budgets
|
|
Given no session budget is configured
|
|
When multiple LLM calls totaling $100.00 are made
|
|
Then all calls should succeed
|
|
And the cost metadata should track the total cost
|
|
|
|
Scenario: Three-tier budget hierarchy
|
|
Given a session budget of $200.00
|
|
And an organization budget of $100.00
|
|
When $10.00 has been spent at session level
|
|
And $82.00 has been spent at organization level
|
|
And a check for $15.00 cost is performed
|
|
Then the check should fail due to organization budget
|
|
And the exceeded_level should be "org"
|
|
|
|
Scenario: Budget check with negative cost
|
|
When a budget check is performed with a negative cost
|
|
Then a ValueError should be raised for cost validation
|
|
|
|
Scenario: Record cost with invalid tokens
|
|
When recording cost with negative input tokens
|
|
Then a ValueError should be raised for cost validation
|