cleveragents/cleveragents-core

Fork 3

Bug: `FallbackSelector.select()` only checks daily budget, ignoring per-plan budget limits #10485

New issue

Open

opened 2026-04-18 10:06:52 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-18 10:06:52 +00:00

Owner

Metadata

Commit message (first line): fix(providers): enforce per-plan budget in FallbackSelector.select()
Branch: bugfix/mN-fallback-plan-budget
Blocked by: #10471 (TDD issue — must be merged first)

Background and Context

FallbackSelector.select() in src/cleveragents/providers/fallback_selector.py evaluates provider candidates against the daily budget via cost_tracker.check_daily_budget(), but it never checks the per-plan budget. As a result, a plan that has exhausted its plan-level cost allocation can still select a provider and continue incurring charges, violating the specification requirement that "rate limiting must be respected" and that budget enforcement must be comprehensive.

Exact Code Evidence

File: src/cleveragents/providers/fallback_selector.py, approximately lines 110–145

if self._cost_tracker is not None:
    daily_check = self._cost_tracker.check_daily_budget()
    if daily_check.status == BudgetStatus.EXCEEDED:
        skipped.append((provider_name, "daily budget exceeded"))
        continue
    # ← NO per-plan budget check here

The CostTracker exposes both check_daily_budget() and (presumably) a per-plan budget check method, but FallbackSelector.select() only calls the daily one. A plan whose plan-level budget is exhausted will still have providers selected for it.

Impact

Severity: HIGH
Affected module: cleveragents.providers.fallback_selector
Class: FallbackSelector
Method: select
Reproducible in: Any scenario where a plan's per-plan budget is exhausted but the daily budget has not been reached
Consequence: Plan-level budget limits are not enforced during provider fallback selection; plans can exceed their allocated cost budget

Specification Violations

"Rate limiting must be respected" — per-plan budget is a form of rate/cost limiting that must be enforced
"Provider fallback must be implemented when primary provider fails" — the fallback mechanism must also respect all budget constraints, not just daily ones

Proposed Fix

Add a per-plan budget check alongside the existing daily budget check:

if self._cost_tracker is not None:
    daily_check = self._cost_tracker.check_daily_budget()
    if daily_check.status == BudgetStatus.EXCEEDED:
        skipped.append((provider_name, "daily budget exceeded"))
        continue
    
    # Also check per-plan budget
    if self._plan_id is not None:
        plan_check = self._cost_tracker.check_plan_budget(self._plan_id)
        if plan_check.status == BudgetStatus.EXCEEDED:
            skipped.append((provider_name, "plan budget exceeded"))
            continue

Expected Behavior

FallbackSelector.select() should check both the daily budget and the per-plan budget before returning a provider. If the per-plan budget is exhausted, select() should return no provider and indicate the reason as "plan budget exceeded".

Acceptance Criteria

FallbackSelector.select() checks both daily AND per-plan budgets before selecting a provider
Providers are skipped when either the daily budget or the per-plan budget is exhausted
The skip reason correctly indicates "plan budget exceeded" when the per-plan budget is the limiting factor
@tdd_issue_10471 scenario passes without @tdd_expected_fail
All nox sessions pass green

Subtasks

Identify the per-plan budget check method on CostTracker (or add one if missing)
Add per-plan budget check in FallbackSelector.select() after the daily budget check
Ensure FallbackSelector receives or can access the current plan ID
Remove @tdd_expected_fail from @tdd_issue_10471 scenario
Verify nox -s unit_tests passes
Verify nox -s typecheck passes (Pyright strict)
Verify nox -s coverage_report ≥ 97%

Definition of Done

FallbackSelector.select() checks both daily AND per-plan budgets
Providers are skipped when either budget is exhausted
@tdd_issue_10471 scenario passes without @tdd_expected_fail
All nox sessions green
PR closes this issue with "Closes #<this_issue_number>"

Automated by CleverAgents Bot
Agent: new-issue-creator

Automated by CleverAgents Bot
Supervisor: Bug Hunt Pool | Agent: bug-hunt-pool-supervisor [AUTO-BUG-8]

## Metadata **Commit message (first line):** `fix(providers): enforce per-plan budget in FallbackSelector.select()` **Branch:** `bugfix/mN-fallback-plan-budget` **Blocked by:** #10471 (TDD issue — must be merged first) ## Background and Context `FallbackSelector.select()` in `src/cleveragents/providers/fallback_selector.py` evaluates provider candidates against the daily budget via `cost_tracker.check_daily_budget()`, but it never checks the per-plan budget. As a result, a plan that has exhausted its plan-level cost allocation can still select a provider and continue incurring charges, violating the specification requirement that "rate limiting must be respected" and that budget enforcement must be comprehensive. ## Exact Code Evidence **File:** `src/cleveragents/providers/fallback_selector.py`, approximately lines 110–145 ```python if self._cost_tracker is not None: daily_check = self._cost_tracker.check_daily_budget() if daily_check.status == BudgetStatus.EXCEEDED: skipped.append((provider_name, "daily budget exceeded")) continue # ← NO per-plan budget check here ``` The `CostTracker` exposes both `check_daily_budget()` and (presumably) a per-plan budget check method, but `FallbackSelector.select()` only calls the daily one. A plan whose plan-level budget is exhausted will still have providers selected for it. ## Impact - **Severity:** HIGH - **Affected module:** `cleveragents.providers.fallback_selector` - **Class:** `FallbackSelector` - **Method:** `select` - **Reproducible in:** Any scenario where a plan's per-plan budget is exhausted but the daily budget has not been reached - **Consequence:** Plan-level budget limits are not enforced during provider fallback selection; plans can exceed their allocated cost budget ## Specification Violations 1. **"Rate limiting must be respected"** — per-plan budget is a form of rate/cost limiting that must be enforced 2. **"Provider fallback must be implemented when primary provider fails"** — the fallback mechanism must also respect all budget constraints, not just daily ones ## Proposed Fix Add a per-plan budget check alongside the existing daily budget check: ```python if self._cost_tracker is not None: daily_check = self._cost_tracker.check_daily_budget() if daily_check.status == BudgetStatus.EXCEEDED: skipped.append((provider_name, "daily budget exceeded")) continue # Also check per-plan budget if self._plan_id is not None: plan_check = self._cost_tracker.check_plan_budget(self._plan_id) if plan_check.status == BudgetStatus.EXCEEDED: skipped.append((provider_name, "plan budget exceeded")) continue ``` ## Expected Behavior `FallbackSelector.select()` should check both the daily budget **and** the per-plan budget before returning a provider. If the per-plan budget is exhausted, `select()` should return no provider and indicate the reason as "plan budget exceeded". ## Acceptance Criteria - `FallbackSelector.select()` checks both daily AND per-plan budgets before selecting a provider - Providers are skipped when either the daily budget or the per-plan budget is exhausted - The skip reason correctly indicates "plan budget exceeded" when the per-plan budget is the limiting factor - `@tdd_issue_10471` scenario passes without `@tdd_expected_fail` - All nox sessions pass green ## Subtasks - [ ] Identify the per-plan budget check method on `CostTracker` (or add one if missing) - [ ] Add per-plan budget check in `FallbackSelector.select()` after the daily budget check - [ ] Ensure `FallbackSelector` receives or can access the current plan ID - [ ] Remove `@tdd_expected_fail` from `@tdd_issue_10471` scenario - [ ] Verify `nox -s unit_tests` passes - [ ] Verify `nox -s typecheck` passes (Pyright strict) - [ ] Verify `nox -s coverage_report` ≥ 97% ## Definition of Done - [ ] `FallbackSelector.select()` checks both daily AND per-plan budgets - [ ] Providers are skipped when either budget is exhausted - [ ] `@tdd_issue_10471` scenario passes without `@tdd_expected_fail` - [ ] All nox sessions green - [ ] PR closes this issue with "Closes #<this_issue_number>" --- **Automated by CleverAgents Bot** Agent: new-issue-creator --- Automated by CleverAgents Bot Supervisor: Bug Hunt Pool | Agent: bug-hunt-pool-supervisor [AUTO-BUG-8]

HAL9000 added the

labels

2026-04-18 10:08:05 +00:00

HAL9000 referenced this issue

2026-04-19 04:31:57 +00:00

[AUTO-IMP-POOL] Status: Cycle 1 - Full Pool Active #10685

HAL9000 referenced this issue from a commit

2026-04-19 10:58:24 +00:00

fix(providers): enforce per-plan budget in FallbackSelector.select()

HAL9000 referenced this issue from a pull request that will close it,

2026-04-19 11:13:14 +00:00

fix(providers): enforce per-plan budget in FallbackSelector.select() #10747

HAL9000 added

and removed

labels

2026-04-19 11:15:59 +00:00

HAL9000 commented

2026-04-19 11:19:19 +00:00

Author

Owner

Implementation Attempt - Tier 1: Haiku - Success

Implemented per-plan budget enforcement in FallbackSelector.select().
Added cost_metadata parameter to FallbackSelector, per-plan budget check in select(), 2 new TDD scenarios tagged @tdd_issue @tdd_issue_10471, and new step definitions.
All quality gates pass: lint, typecheck, unit_tests (74 scenarios pass including 2 new TDD scenarios).
PR: #10747 - #10747

Automated by CleverAgents Bot
Supervisor: Implementation Pool | Agent: implementation-worker

**Implementation Attempt** - Tier 1: Haiku - Success Implemented per-plan budget enforcement in FallbackSelector.select(). Added cost_metadata parameter to FallbackSelector, per-plan budget check in select(), 2 new TDD scenarios tagged @tdd_issue @tdd_issue_10471, and new step definitions. All quality gates pass: lint, typecheck, unit_tests (74 scenarios pass including 2 new TDD scenarios). PR: #10747 - https://git.cleverthis.com/cleveragents/cleveragents-core/pulls/10747 --- **Automated by CleverAgents Bot** Supervisor: Implementation Pool | Agent: implementation-worker