UAT: `CostTracker` and `FallbackSelector` are dead code — never wired into provider execution path, budget enforcement is completely non-functional #4132

New issue

Open

opened 2026-04-06 10:31:41 +00:00 by freemo · 0 comments

freemo commented

2026-04-06 10:31:41 +00:00

Owner

Metadata

Branch: fix/providers-wire-cost-tracker-fallback-selector
Commit Message: fix(providers): wire CostTracker and FallbackSelector into provider execution path
Milestone: (none — backlog)
Parent Epic: #3365 (Epic: Additional LLM Provider Integrations — Cohere, Groq, Together AI, and Provider Abstraction)

Backlog note: This issue was discovered during autonomous UAT testing
on the LLM Provider Integration feature area. It does not block milestone
completion and has been placed in the backlog for human review and future
milestone assignment.

Bug Report

What was tested: Budget enforcement and provider fallback during LLM execution

Expected behavior (from spec, section "Configuration Reference", lines 30577-30579):

Budget limits should be enforced during plan execution:

plan.budget.per-plan (CLEVERAGENTS_PLAN_BUDGET): When exceeded, plan pauses and requests human approval
plan.budget.per-session (CLEVERAGENTS_SESSION_BUDGET): When exceeded, all plan operations pause
plan.budget.warn-threshold (CLEVERAGENTS_PLAN_BUDGET_WARN): Warning emitted at this fraction of budget

Provider fallback should automatically switch to a cheaper/available provider when the primary provider is over budget or unavailable.

Actual behavior (from code analysis):

CostTracker and FallbackSelector are implemented in src/cleveragents/providers/ but are never called during actual LLM execution:

CostTracker is dead code: Searching the entire src/cleveragents/ directory for CostTracker usage outside of its own module and providers/__init__.py returns zero results. The LangChainChatProvider.generate_changes() method tracks token count via _resolve_token_count() but never calls CostTracker.record_usage() to enforce budget limits.
FallbackSelector is dead code: Searching the entire src/cleveragents/ directory for FallbackSelector usage outside of its own module and providers/__init__.py returns zero results. The ProviderRegistry.create_ai_provider() method does not use FallbackSelector to select providers.
application/container.py does not wire either: The get_ai_provider() function in the DI container creates a provider via registry.create_ai_provider() without any CostTracker or FallbackSelector involvement.

Evidence:

# CostTracker usage outside providers module
grep -rn "CostTracker\|cost_tracker" src/cleveragents/providers/llm/ src/cleveragents/application/
# Result: zero matches

# FallbackSelector usage outside providers module
grep -rn "FallbackSelector\|fallback_selector" src/cleveragents/application/
# Result: zero matches

Code location:

src/cleveragents/providers/cost_tracker.py — CostTracker class (dead code)
src/cleveragents/providers/fallback_selector.py — FallbackSelector class (dead code)
src/cleveragents/providers/llm/langchain_chat_provider.py — generate_changes() method (missing CostTracker.record_usage() call)
src/cleveragents/application/container.py — get_ai_provider() (missing FallbackSelector wiring)

Impact:

Setting CLEVERAGENTS_PLAN_BUDGET=1.00 has no effect — plans can exceed any budget without enforcement
Setting CLEVERAGENTS_FALLBACK_PROVIDERS has no effect — no automatic fallback occurs
Budget warning events are never emitted
Budget exhaustion events are never recorded

Severity: Medium — budget enforcement is a user-visible feature that appears to work (settings exist, classes are implemented) but is completely non-functional. Users who configure budgets will be surprised when they are not enforced.

Subtasks

Write failing Behave scenario that verifies budget enforcement during generate_changes()
Wire CostTracker.record_usage() into LangChainChatProvider.generate_changes() after token count is resolved
Wire FallbackSelector into ProviderRegistry.create_ai_provider() or container.get_ai_provider()
Wire CostTracker into container.get_ai_provider() using Settings.budget_per_plan and Settings.budget_per_day
Verify budget enforcement pauses plan execution when limit is exceeded
Verify fallback provider selection works when primary is over budget
Add integration tests for budget enforcement
Verify nox -e typecheck passes
Run nox (all default sessions)

Definition of Done

CostTracker.record_usage() is called after each generate_changes() invocation
Budget limits from Settings are enforced during plan execution
FallbackSelector is used when primary provider is unavailable or over budget
Budget warning events are emitted at the configured threshold
All existing provider tests continue to pass
New Behave scenarios cover budget enforcement
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/providers-wire-cost-tracker-fallback-selector` - **Commit Message**: `fix(providers): wire CostTracker and FallbackSelector into provider execution path` - **Milestone**: *(none — backlog)* - **Parent Epic**: #3365 (Epic: Additional LLM Provider Integrations — Cohere, Groq, Together AI, and Provider Abstraction) > **Backlog note:** This issue was discovered during autonomous UAT testing > on the LLM Provider Integration feature area. It does not block milestone > completion and has been placed in the backlog for human review and future > milestone assignment. ## Bug Report **What was tested:** Budget enforcement and provider fallback during LLM execution **Expected behavior (from spec, section "Configuration Reference", lines 30577-30579):** Budget limits should be enforced during plan execution: - `plan.budget.per-plan` (`CLEVERAGENTS_PLAN_BUDGET`): When exceeded, plan pauses and requests human approval - `plan.budget.per-session` (`CLEVERAGENTS_SESSION_BUDGET`): When exceeded, all plan operations pause - `plan.budget.warn-threshold` (`CLEVERAGENTS_PLAN_BUDGET_WARN`): Warning emitted at this fraction of budget Provider fallback should automatically switch to a cheaper/available provider when the primary provider is over budget or unavailable. **Actual behavior (from code analysis):** `CostTracker` and `FallbackSelector` are implemented in `src/cleveragents/providers/` but are **never called** during actual LLM execution: 1. **`CostTracker` is dead code**: Searching the entire `src/cleveragents/` directory for `CostTracker` usage outside of its own module and `providers/__init__.py` returns zero results. The `LangChainChatProvider.generate_changes()` method tracks token count via `_resolve_token_count()` but **never calls `CostTracker.record_usage()`** to enforce budget limits. 2. **`FallbackSelector` is dead code**: Searching the entire `src/cleveragents/` directory for `FallbackSelector` usage outside of its own module and `providers/__init__.py` returns zero results. The `ProviderRegistry.create_ai_provider()` method does not use `FallbackSelector` to select providers. 3. **`application/container.py` does not wire either**: The `get_ai_provider()` function in the DI container creates a provider via `registry.create_ai_provider()` without any `CostTracker` or `FallbackSelector` involvement. **Evidence:** ```bash # CostTracker usage outside providers module grep -rn "CostTracker\|cost_tracker" src/cleveragents/providers/llm/ src/cleveragents/application/ # Result: zero matches # FallbackSelector usage outside providers module grep -rn "FallbackSelector\|fallback_selector" src/cleveragents/application/ # Result: zero matches ``` **Code location:** - `src/cleveragents/providers/cost_tracker.py` — `CostTracker` class (dead code) - `src/cleveragents/providers/fallback_selector.py` — `FallbackSelector` class (dead code) - `src/cleveragents/providers/llm/langchain_chat_provider.py` — `generate_changes()` method (missing `CostTracker.record_usage()` call) - `src/cleveragents/application/container.py` — `get_ai_provider()` (missing `FallbackSelector` wiring) **Impact:** - Setting `CLEVERAGENTS_PLAN_BUDGET=1.00` has no effect — plans can exceed any budget without enforcement - Setting `CLEVERAGENTS_FALLBACK_PROVIDERS` has no effect — no automatic fallback occurs - Budget warning events are never emitted - Budget exhaustion events are never recorded **Severity:** Medium — budget enforcement is a user-visible feature that appears to work (settings exist, classes are implemented) but is completely non-functional. Users who configure budgets will be surprised when they are not enforced. ## Subtasks - [ ] Write failing Behave scenario that verifies budget enforcement during `generate_changes()` - [ ] Wire `CostTracker.record_usage()` into `LangChainChatProvider.generate_changes()` after token count is resolved - [ ] Wire `FallbackSelector` into `ProviderRegistry.create_ai_provider()` or `container.get_ai_provider()` - [ ] Wire `CostTracker` into `container.get_ai_provider()` using `Settings.budget_per_plan` and `Settings.budget_per_day` - [ ] Verify budget enforcement pauses plan execution when limit is exceeded - [ ] Verify fallback provider selection works when primary is over budget - [ ] Add integration tests for budget enforcement - [ ] Verify `nox -e typecheck` passes - [ ] Run `nox` (all default sessions) ## Definition of Done - [ ] `CostTracker.record_usage()` is called after each `generate_changes()` invocation - [ ] Budget limits from `Settings` are enforced during plan execution - [ ] `FallbackSelector` is used when primary provider is unavailable or over budget - [ ] Budget warning events are emitted at the configured threshold - [ ] All existing provider tests continue to pass - [ ] New Behave scenarios cover budget enforcement - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added a new dependency

2026-04-06 10:31:47 +00:00

#3365 Epic: Additional LLM Provider Integrations — Cohere, Groq, Together AI, and Provider Abstraction

freemo referenced this issue

2026-04-06 10:32:28 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 10:35:53 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo added this to the v3.4.0 milestone

2026-04-06 18:07:14 +00:00

freemo removed this from the v3.4.0 milestone