cleveragents/cleveragents-core

Fork 3

UAT: ProviderCostTable missing cost entries for OpenRouter provider #5508

New issue

Open

opened 2026-04-09 07:07:27 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-09 07:07:27 +00:00

Owner

Bug Report

Feature Area: Token Counting and Cost Tracking Per Provider
Severity: Low (cost estimation falls back to generic default)
Found by: UAT Testing (uat-pool-1, worker: additional-llm-backends)

What Was Tested

The ProviderCostTable in cost_table.py for completeness of cost entries across all supported providers.

Expected Behavior

All supported providers should have cost entries in ProviderCostTable._DEFAULT_ENTRIES so that cost estimation is accurate for budget enforcement.

Actual Behavior

ProviderCostTable._DEFAULT_ENTRIES contains entries for:

openai ✓ (gpt-4o, gpt-4o-mini, gpt-4-turbo)
anthropic ✓ (claude-sonnet-4, claude-3-5-haiku, claude-opus-4)
google ✓ (gemini-2.0-flash, gemini-1.5-pro)
groq ✓ (llama-3.1-70b-versatile)
together ✓ (Llama-3.1-70B-Instruct-Turbo)
cohere ✓ (command-r-plus)
mock ✓

Missing:

openrouter ✗ — No cost entries despite being a supported provider type with its own OpenRouterChatProvider class

Additionally:

azure ✗ — No cost entries (Azure OpenAI pricing differs from direct OpenAI)
gemini ✗ — The gemini provider type (separate from google) has no entries; it would need to use the google entries

Code Location

src/cleveragents/providers/cost_table.py, ProviderCostTable._DEFAULT_ENTRIES

Impact

When using OpenRouter:

CostTracker.estimate_cost("openrouter", "anthropic/claude-sonnet-4-20250514", ...) falls back to DEFAULT_COST (0.00005 input, 0.00015 output per token)
The default is a conservative overestimate — OpenRouter models often have different pricing than the defaults
Budget enforcement may trigger prematurely or inaccurately for OpenRouter users

Suggested Fix

Add OpenRouter cost entries to _DEFAULT_ENTRIES:

"openrouter": {
    "anthropic/claude-sonnet-4-20250514": CostEntry(
        input_cost_per_token=0.000003,   # OpenRouter pricing for Claude Sonnet
        output_cost_per_token=0.000015,
    ),
    "openai/gpt-4o": CostEntry(
        input_cost_per_token=0.0000025,
        output_cost_per_token=0.00001,
    ),
    # Add other popular OpenRouter models
},

Note: OpenRouter pricing varies by model and may change. Consider adding a note that these are approximate estimates.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: Token Counting and Cost Tracking Per Provider **Severity**: Low (cost estimation falls back to generic default) **Found by**: UAT Testing (uat-pool-1, worker: additional-llm-backends) --- ## What Was Tested The `ProviderCostTable` in `cost_table.py` for completeness of cost entries across all supported providers. ## Expected Behavior All supported providers should have cost entries in `ProviderCostTable._DEFAULT_ENTRIES` so that cost estimation is accurate for budget enforcement. ## Actual Behavior `ProviderCostTable._DEFAULT_ENTRIES` contains entries for: - `openai` ✓ (gpt-4o, gpt-4o-mini, gpt-4-turbo) - `anthropic` ✓ (claude-sonnet-4, claude-3-5-haiku, claude-opus-4) - `google` ✓ (gemini-2.0-flash, gemini-1.5-pro) - `groq` ✓ (llama-3.1-70b-versatile) - `together` ✓ (Llama-3.1-70B-Instruct-Turbo) - `cohere` ✓ (command-r-plus) - `mock` ✓ **Missing**: - `openrouter` ✗ — No cost entries despite being a supported provider type with its own `OpenRouterChatProvider` class Additionally: - `azure` ✗ — No cost entries (Azure OpenAI pricing differs from direct OpenAI) - `gemini` ✗ — The `gemini` provider type (separate from `google`) has no entries; it would need to use the `google` entries ## Code Location `src/cleveragents/providers/cost_table.py`, `ProviderCostTable._DEFAULT_ENTRIES` ## Impact When using OpenRouter: 1. `CostTracker.estimate_cost("openrouter", "anthropic/claude-sonnet-4-20250514", ...)` falls back to `DEFAULT_COST` (0.00005 input, 0.00015 output per token) 2. The default is a conservative overestimate — OpenRouter models often have different pricing than the defaults 3. Budget enforcement may trigger prematurely or inaccurately for OpenRouter users ## Suggested Fix Add OpenRouter cost entries to `_DEFAULT_ENTRIES`: ```python "openrouter": { "anthropic/claude-sonnet-4-20250514": CostEntry( input_cost_per_token=0.000003, # OpenRouter pricing for Claude Sonnet output_cost_per_token=0.000015, ), "openai/gpt-4o": CostEntry( input_cost_per_token=0.0000025, output_cost_per_token=0.00001, ), # Add other popular OpenRouter models }, ``` Note: OpenRouter pricing varies by model and may change. Consider adding a note that these are approximate estimates. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 07:08:37 +00:00

HAL9000 added this to the v3.2.0 milestone

2026-04-09 07:12:12 +00:00

HAL9000 commented

2026-04-09 07:13:38 +00:00

Author

Owner

Label compliance fix applied:

Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: backlog-groomer

Label compliance fix applied: - Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: backlog-groomer

HAL9000 added

and removed

labels

2026-04-09 07:17:57 +00:00

HAL9000 referenced this issue

2026-04-09 07:18:38 +00:00