HAL9000
aee4e25853
CI / build (pull_request) Successful in 44s
CI / lint (pull_request) Successful in 52s
CI / quality (pull_request) Successful in 1m5s
CI / typecheck (pull_request) Successful in 1m19s
CI / helm (pull_request) Successful in 33s
CI / security (pull_request) Successful in 1m28s
CI / push-validation (pull_request) Successful in 26s
CI / unit_tests (pull_request) Successful in 4m42s
CI / integration_tests (pull_request) Successful in 9m38s
CI / docker (pull_request) Successful in 2m10s
CI / coverage (pull_request) Successful in 10m42s
CI / status-check (pull_request) Successful in 4s
- Convert CostRecord from @dataclass to Pydantic BaseModel (architecture policy) - Replace declarative_base() + # type: ignore[misc] with DeclarativeBase subclass - Use Mapped/mapped_column for proper SQLAlchemy 2.0 typed columns - Remove all cast() calls — types now flow from Mapped[T] annotations - Change timestamp column from String to DateTime; use datetime.now(UTC) - Fix wrong expected totals in cost_tracker_service.feature (0.00035 → 0.00075) - Fix step_check_entries_ids AttributeError: read plan_entries when session_entries absent - Apply ruff format to both changed files ISSUES CLOSED: #5248
94 lines
4.6 KiB
Gherkin
94 lines
4.6 KiB
Gherkin
Feature: CostTracker service for per-session and per-plan spending tracking
|
|
As a system administrator
|
|
I want to track LLM API spending per session and per plan
|
|
So that I can monitor costs and enforce budget limits
|
|
|
|
Background:
|
|
Given a temporary data directory for cost tracking
|
|
And a CostTracker instance with default model pricing
|
|
|
|
Scenario: Record usage for a single LLM call
|
|
When I record usage for session "sess-001" and plan "plan-001"
|
|
| tokens_in | tokens_out | model |
|
|
| 100 | 50 | gpt-3.5-turbo |
|
|
Then the recorded cost should be approximately 0.000125 USD
|
|
And the cost record should have session_id "sess-001"
|
|
And the cost record should have plan_id "plan-001"
|
|
|
|
Scenario: Calculate cost for different models
|
|
When I record usage for session "sess-002" and plan "plan-002"
|
|
| tokens_in | tokens_out | model |
|
|
| 1000 | 1000 | gpt-4 |
|
|
Then the recorded cost should be approximately 0.09 USD
|
|
When I record usage for session "sess-002" and plan "plan-002"
|
|
| tokens_in | tokens_out | model |
|
|
| 1000 | 1000 | gpt-3.5-turbo |
|
|
Then the recorded cost should be approximately 0.002 USD
|
|
|
|
Scenario: Get total session cost
|
|
When I record multiple usages for session "sess-003"
|
|
| plan_id | tokens_in | tokens_out | model |
|
|
| plan-001 | 100 | 50 | gpt-3.5-turbo |
|
|
| plan-002 | 200 | 100 | gpt-3.5-turbo |
|
|
| plan-003 | 300 | 150 | gpt-3.5-turbo |
|
|
Then the total session cost for "sess-003" should be approximately 0.00075 USD
|
|
|
|
Scenario: Get total plan cost
|
|
When I record multiple usages for plan "plan-004"
|
|
| session_id | tokens_in | tokens_out | model |
|
|
| sess-001 | 100 | 50 | gpt-3.5-turbo |
|
|
| sess-002 | 200 | 100 | gpt-3.5-turbo |
|
|
| sess-003 | 300 | 150 | gpt-3.5-turbo |
|
|
Then the total plan cost for "plan-004" should be approximately 0.00075 USD
|
|
|
|
Scenario: Cost persistence across tracker instances
|
|
When I record usage for session "sess-004" and plan "plan-005"
|
|
| tokens_in | tokens_out | model |
|
|
| 500 | 250 | gpt-3.5-turbo |
|
|
And I create a new CostTracker instance with the same data directory
|
|
Then the total session cost for "sess-004" should be approximately 0.000625 USD
|
|
And the total plan cost for "plan-005" should be approximately 0.000625 USD
|
|
|
|
Scenario: Get session entries
|
|
When I record multiple usages for session "sess-005"
|
|
| plan_id | tokens_in | tokens_out | model |
|
|
| plan-001 | 100 | 50 | gpt-3.5-turbo |
|
|
| plan-002 | 200 | 100 | gpt-3.5-turbo |
|
|
Then I should get 2 entries for session "sess-005"
|
|
And each entry should have the correct session_id and plan_id
|
|
|
|
Scenario: Get plan entries
|
|
When I record multiple usages for plan "plan-006"
|
|
| session_id | tokens_in | tokens_out | model |
|
|
| sess-001 | 100 | 50 | gpt-3.5-turbo |
|
|
| sess-002 | 200 | 100 | gpt-3.5-turbo |
|
|
Then I should get 2 entries for plan "plan-006"
|
|
And each entry should have the correct session_id and plan_id
|
|
|
|
Scenario: Custom model pricing
|
|
Given a CostTracker instance with custom model pricing
|
|
| model | input_price | output_price |
|
|
| custom-llm | 0.001 | 0.002 |
|
|
When I record usage for session "sess-006" and plan "plan-007"
|
|
| tokens_in | tokens_out | model |
|
|
| 1000 | 1000 | custom-llm |
|
|
Then the recorded cost should be approximately 0.003 USD
|
|
|
|
Scenario: Zero cost for unknown model
|
|
When I record usage for session "sess-007" and plan "plan-008"
|
|
| tokens_in | tokens_out | model |
|
|
| 1000 | 1000 | unknown-llm |
|
|
Then the recorded cost should be approximately 0.0 USD
|
|
|
|
Scenario: Multiple sessions and plans isolation
|
|
When I record usage for session "sess-008" and plan "plan-009"
|
|
| tokens_in | tokens_out | model |
|
|
| 100 | 50 | gpt-3.5-turbo |
|
|
And I record usage for session "sess-009" and plan "plan-010"
|
|
| tokens_in | tokens_out | model |
|
|
| 200 | 100 | gpt-3.5-turbo |
|
|
Then the total session cost for "sess-008" should be approximately 0.000125 USD
|
|
And the total session cost for "sess-009" should be approximately 0.00025 USD
|
|
And the total plan cost for "plan-009" should be approximately 0.000125 USD
|
|
And the total plan cost for "plan-010" should be approximately 0.00025 USD
|