UAT: ProviderCostTable missing cost entries for Azure, OpenRouter, and Gemini providers #6087

New issue

Open

opened 2026-04-09 14:30:24 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-09 14:30:24 +00:00

Owner

Bug Report

Summary

ProviderCostTable in src/cleveragents/providers/cost_table.py has cost entries for OpenAI, Anthropic, Google, Groq, Together, Cohere, and Mock providers, but is missing entries for Azure, OpenRouter, and Gemini — three of the ten supported provider types. These providers will always fall back to the DEFAULT_COST estimate, which may be significantly inaccurate.

What Was Tested

Inspected src/cleveragents/providers/cost_table.py — _DEFAULT_ENTRIES class variable
Cross-referenced with ProviderType enum in registry.py

Expected Behavior

All supported providers should have at least one model cost entry in ProviderCostTable to enable accurate cost estimation and budget enforcement.

Actual Behavior

Providers WITH cost entries:

openai — gpt-4o, gpt-4o-mini, gpt-4-turbo
anthropic — claude-sonnet-4, claude-3-5-haiku, claude-opus-4
google — gemini-2.0-flash, gemini-1.5-pro
groq — llama-3.1-70b-versatile
together — meta-llama/Llama-3.1-70B-Instruct-Turbo
cohere — command-r-plus
mock — mock-gpt

Providers WITHOUT cost entries (fall back to DEFAULT_COST):

azure — MISSING (Azure OpenAI pricing varies by deployment)
openrouter — MISSING (OpenRouter is a proxy; pricing varies by model)
gemini — MISSING (Gemini API has separate pricing from Google AI Studio)

Impact

When the cost tracker estimates usage for Azure, OpenRouter, or Gemini providers, it uses DEFAULT_COST:

DEFAULT_COST: ClassVar[CostEntry] = CostEntry(
    input_cost_per_token=0.000005,   # $5/M tokens
    output_cost_per_token=0.000015,  # $15/M tokens
)

This default is based on GPT-4 pricing and may be:

Too high for Gemini (which is much cheaper: $0.10/M input tokens for gemini-2.0-flash)
Inaccurate for Azure (pricing depends on deployment region and model)
Inaccurate for OpenRouter (pricing depends on the underlying model being routed to)

This causes incorrect budget enforcement — users may hit budget limits prematurely or exceed budgets unexpectedly.

Code Location

src/cleveragents/providers/cost_table.py — _DEFAULT_ENTRIES class variable

Fix

Add cost entries for the missing providers:

"gemini": {
    "gemini-2.0-flash": CostEntry(
        input_cost_per_token=0.0000001,   # $0.10/M tokens
        output_cost_per_token=0.0000004,  # $0.40/M tokens
    ),
    "gemini-1.5-pro": CostEntry(
        input_cost_per_token=0.00000125,
        output_cost_per_token=0.000005,
    ),
},
"azure": {
    # Azure pricing mirrors OpenAI pricing
    "gpt-4o": CostEntry(
        input_cost_per_token=0.0000025,
        output_cost_per_token=0.00001,
    ),
},
"openrouter": {
    # OpenRouter passes through model pricing; use a conservative estimate
    # for the default model (anthropic/claude-sonnet-4)
    "anthropic/claude-sonnet-4-20250514": CostEntry(
        input_cost_per_token=0.000003,
        output_cost_per_token=0.000015,
    ),
},

Note: OpenRouter's pricing is model-dependent and dynamic. A better long-term solution would be to query OpenRouter's pricing API at runtime, but static defaults for common models are a reasonable starting point.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report ### Summary `ProviderCostTable` in `src/cleveragents/providers/cost_table.py` has cost entries for OpenAI, Anthropic, Google, Groq, Together, Cohere, and Mock providers, but is **missing entries for Azure, OpenRouter, and Gemini** — three of the ten supported provider types. These providers will always fall back to the `DEFAULT_COST` estimate, which may be significantly inaccurate. ### What Was Tested - Inspected `src/cleveragents/providers/cost_table.py` — `_DEFAULT_ENTRIES` class variable - Cross-referenced with `ProviderType` enum in `registry.py` ### Expected Behavior All supported providers should have at least one model cost entry in `ProviderCostTable` to enable accurate cost estimation and budget enforcement. ### Actual Behavior **Providers WITH cost entries:** - `openai` — gpt-4o, gpt-4o-mini, gpt-4-turbo - `anthropic` — claude-sonnet-4, claude-3-5-haiku, claude-opus-4 - `google` — gemini-2.0-flash, gemini-1.5-pro - `groq` — llama-3.1-70b-versatile - `together` — meta-llama/Llama-3.1-70B-Instruct-Turbo - `cohere` — command-r-plus - `mock` — mock-gpt **Providers WITHOUT cost entries (fall back to DEFAULT_COST):** - `azure` — **MISSING** (Azure OpenAI pricing varies by deployment) - `openrouter` — **MISSING** (OpenRouter is a proxy; pricing varies by model) - `gemini` — **MISSING** (Gemini API has separate pricing from Google AI Studio) ### Impact When the cost tracker estimates usage for Azure, OpenRouter, or Gemini providers, it uses `DEFAULT_COST`: ```python DEFAULT_COST: ClassVar[CostEntry] = CostEntry( input_cost_per_token=0.000005, # $5/M tokens output_cost_per_token=0.000015, # $15/M tokens ) ``` This default is based on GPT-4 pricing and may be: - **Too high** for Gemini (which is much cheaper: $0.10/M input tokens for gemini-2.0-flash) - **Inaccurate** for Azure (pricing depends on deployment region and model) - **Inaccurate** for OpenRouter (pricing depends on the underlying model being routed to) This causes incorrect budget enforcement — users may hit budget limits prematurely or exceed budgets unexpectedly. ### Code Location `src/cleveragents/providers/cost_table.py` — `_DEFAULT_ENTRIES` class variable ### Fix Add cost entries for the missing providers: ```python "gemini": { "gemini-2.0-flash": CostEntry( input_cost_per_token=0.0000001, # $0.10/M tokens output_cost_per_token=0.0000004, # $0.40/M tokens ), "gemini-1.5-pro": CostEntry( input_cost_per_token=0.00000125, output_cost_per_token=0.000005, ), }, "azure": { # Azure pricing mirrors OpenAI pricing "gpt-4o": CostEntry( input_cost_per_token=0.0000025, output_cost_per_token=0.00001, ), }, "openrouter": { # OpenRouter passes through model pricing; use a conservative estimate # for the default model (anthropic/claude-sonnet-4) "anthropic/claude-sonnet-4-20250514": CostEntry( input_cost_per_token=0.000003, output_cost_per_token=0.000015, ), }, ``` Note: OpenRouter's pricing is model-dependent and dynamic. A better long-term solution would be to query OpenRouter's pricing API at runtime, but static defaults for common models are a reasonable starting point. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 14:34:59 +00:00

HAL9000 added this to the v3.2.0 milestone

2026-04-09 21:19:18 +00:00

HAL9000 added