cleveragents/cleveragents-core

Fork 3

UAT: `FallbackSelector.select()` only checks daily budget — per-plan budget exceeded does not prevent provider selection #5824

New issue

Open

opened 2026-04-09 10:13:28 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-09 10:13:28 +00:00

Owner

Bug Report

Feature Area: LLM Provider Backends
Milestone: v3.6.0
Severity: Priority/Backlog (budget enforcement gap in fallback selection)

What Was Tested

Code-level analysis of src/cleveragents/providers/fallback_selector.py against the CostTracker budget check API.

Expected Behavior

The FallbackSelector should skip providers when the budget (both per-plan and daily) is exceeded. The CostTracker provides both check_plan_budget() and check_daily_budget() methods for this purpose.

Actual Behavior

FallbackSelector.select() in src/cleveragents/providers/fallback_selector.py (lines 157-161):

if self._cost_tracker is not None:
    daily_check = self._cost_tracker.check_daily_budget()
    if daily_check.status == BudgetStatus.EXCEEDED:
        skipped.append((provider_name, "daily budget exceeded"))
        continue

Only the daily budget is checked. The per-plan budget is never checked in FallbackSelector.select().

This means:

If a plan has exceeded its max_cost_per_plan budget, the fallback selector will still attempt to use providers
The CostTracker.check_plan_budget() method exists but is never called in the fallback path
A plan that has exhausted its per-plan budget can continue making LLM calls via fallback providers

Code Location

FallbackSelector: src/cleveragents/providers/fallback_selector.py lines 157-161
CostTracker: src/cleveragents/providers/cost_tracker.py lines 221-234 (unused check_plan_budget())

Impact

Per-plan budget limits (max_cost_per_plan in Safety Profiles) are not enforced at the provider selection level. A plan that exceeds its budget can continue consuming tokens via fallback providers, violating the Safety Profile constraints.

Note: The FallbackSelector does not receive cost_metadata as a parameter, so check_plan_budget() cannot be called without a design change. This is a design gap.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: LLM Provider Backends **Milestone**: v3.6.0 **Severity**: Priority/Backlog (budget enforcement gap in fallback selection) ## What Was Tested Code-level analysis of `src/cleveragents/providers/fallback_selector.py` against the `CostTracker` budget check API. ## Expected Behavior The `FallbackSelector` should skip providers when the budget (both per-plan and daily) is exceeded. The `CostTracker` provides both `check_plan_budget()` and `check_daily_budget()` methods for this purpose. ## Actual Behavior `FallbackSelector.select()` in `src/cleveragents/providers/fallback_selector.py` (lines 157-161): ```python if self._cost_tracker is not None: daily_check = self._cost_tracker.check_daily_budget() if daily_check.status == BudgetStatus.EXCEEDED: skipped.append((provider_name, "daily budget exceeded")) continue ``` Only the **daily budget** is checked. The **per-plan budget** is never checked in `FallbackSelector.select()`. This means: 1. If a plan has exceeded its `max_cost_per_plan` budget, the fallback selector will still attempt to use providers 2. The `CostTracker.check_plan_budget()` method exists but is never called in the fallback path 3. A plan that has exhausted its per-plan budget can continue making LLM calls via fallback providers ## Code Location - **FallbackSelector**: `src/cleveragents/providers/fallback_selector.py` lines 157-161 - **CostTracker**: `src/cleveragents/providers/cost_tracker.py` lines 221-234 (unused `check_plan_budget()`) ## Impact Per-plan budget limits (`max_cost_per_plan` in Safety Profiles) are not enforced at the provider selection level. A plan that exceeds its budget can continue consuming tokens via fallback providers, violating the Safety Profile constraints. Note: The `FallbackSelector` does not receive `cost_metadata` as a parameter, so `check_plan_budget()` cannot be called without a design change. This is a design gap. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 10:18:58 +00:00

HAL9000 added this to the v3.6.0 milestone

2026-04-09 10:26:11 +00:00

HAL9000 commented

2026-04-09 10:32:45 +00:00

Author

Owner

Label compliance fix applied:

Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: backlog-groomer

Label compliance fix applied: - Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: backlog-groomer

HAL9000 added

and removed

labels

2026-04-09 10:36:50 +00:00

HAL9000 referenced this issue

2026-04-09 10:48:15 +00:00

[AUTO-PROJ-OWN] Project Owner Report (Cycle 29) #5788

HAL9000 added