freemo
134dce896a
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 17s
CI / lint (pull_request) Successful in 18s
CI / quality (pull_request) Successful in 20s
CI / security (pull_request) Successful in 34s
CI / typecheck (pull_request) Successful in 37s
CI / integration_tests (pull_request) Successful in 3m46s
CI / unit_tests (pull_request) Successful in 8m32s
CI / docker (pull_request) Successful in 39s
CI / benchmark-regression (pull_request) Successful in 19m26s
CI / coverage (pull_request) Successful in 31m23s
CI / lint (push) Successful in 13s
CI / build (push) Successful in 15s
CI / quality (push) Successful in 16s
CI / typecheck (push) Successful in 30s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 33s
CI / integration_tests (push) Successful in 3m44s
CI / unit_tests (push) Successful in 8m53s
CI / docker (push) Successful in 1m1s
CI / benchmark-publish (push) Successful in 10m35s
CI / coverage (push) Successful in 37m55s
Add token/cost tracking, budget enforcement (per-plan and per-day), provider fallback selection with capability filtering, and cost metadata for plan models. New modules: - providers/cost_table.py: CostEntry + ProviderCostTable with default pricing for OpenAI, Anthropic, Google, Groq, Together, Cohere, Mock - providers/cost_tracker.py: BudgetStatus enum, BudgetCheckResult, CostTracker with warn-at-90%/block-at-100% enforcement - providers/fallback_selector.py: FallbackSelector with capability filtering (tool_calls, streaming, vision, json_mode) and budget - domain/models/core/cost_metadata.py: CostMetadata + BudgetExhaustionEvent Modified: - config/settings.py: budget_per_plan, budget_per_day, fallback_providers - domain/models/core/plan.py: cost_metadata field + CLI dict surfacing - providers/__init__.py: public exports for new classes Testing: - 71 Behave scenarios in features/cost_controls.feature (all pass) - 6 Robot Framework integration tests (all pass) - ASV benchmarks in benchmarks/cost_controls_bench.py - nox lint, typecheck, format, docs, build, dead_code, security_scan pass Closes #324
72 lines
2.9 KiB
Plaintext
72 lines
2.9 KiB
Plaintext
*** Settings ***
|
|
Resource ${CURDIR}/common.resource
|
|
Library OperatingSystem
|
|
Library Process
|
|
|
|
*** Variables ***
|
|
${PYTHON} python
|
|
${SRC_DIR} ${CURDIR}/..
|
|
|
|
*** Test Cases ***
|
|
Cost Entry Creation And Estimation
|
|
[Documentation] Verify CostEntry creation and cost estimation
|
|
[Tags] cost_controls provider model
|
|
${result}= Run Process ${PYTHON} robot/helper_cost_controls.py cost-entry
|
|
... cwd=${SRC_DIR}
|
|
Log Process Failure ${result}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} cost-entry-ok
|
|
|
|
Provider Cost Table Lookup
|
|
[Documentation] Verify ProviderCostTable model lookup
|
|
[Tags] cost_controls provider model
|
|
${result}= Run Process ${PYTHON} robot/helper_cost_controls.py cost-table
|
|
... cwd=${SRC_DIR}
|
|
Log Process Failure ${result}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} cost-table-ok
|
|
|
|
Cost Tracker Budget Enforcement
|
|
[Documentation] Verify CostTracker budget enforcement
|
|
[Tags] cost_controls provider budget
|
|
${result}= Run Process ${PYTHON} robot/helper_cost_controls.py cost-tracker
|
|
... cwd=${SRC_DIR}
|
|
Log Process Failure ${result}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} cost-tracker-budget-ok
|
|
|
|
Fallback Selector No Providers
|
|
[Documentation] Verify FallbackSelector with no configured providers
|
|
[Tags] cost_controls provider fallback
|
|
${result}= Run Process ${PYTHON} robot/helper_cost_controls.py fallback-selector
|
|
... cwd=${SRC_DIR}
|
|
Log Process Failure ${result}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} fallback-selector-ok
|
|
|
|
Settings Cost Control Defaults
|
|
[Documentation] Verify Settings cost control config defaults
|
|
[Tags] cost_controls config settings
|
|
${result}= Run Process ${PYTHON} robot/helper_cost_controls.py settings-defaults
|
|
... cwd=${SRC_DIR}
|
|
Log Process Failure ${result}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} settings-defaults-ok
|
|
|
|
Cost Metadata Display Dict
|
|
[Documentation] Verify CostMetadata display dictionary
|
|
[Tags] cost_controls provider model
|
|
${result}= Run Process ${PYTHON} robot/helper_cost_controls.py cost-metadata
|
|
... cwd=${SRC_DIR}
|
|
Log Process Failure ${result}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} cost-metadata-display-ok
|
|
|
|
*** Keywords ***
|
|
Log Process Failure
|
|
[Arguments] ${result}
|
|
Run Keyword If ${result.rc} == 0 Return From Keyword
|
|
Log To Console Process failed with rc=${result.rc}
|
|
Log To Console STDOUT:${\n}${result.stdout}
|
|
Log To Console STDERR:${\n}${result.stderr}
|