bug(agents): LLM token consumption undercounted — pruning model calls untracked, earlier loop rounds overwritten #65

New issue

Closed

opened 2026-06-30 11:21:57 +00:00 by CoreRasurae · 0 comments

CoreRasurae commented

2026-06-30 11:21:57 +00:00

Member

Metadata

Commit Message: fix(actors): llm token consumption undercounted
Branch: bugfix/m2-lllm-token-consumption-undercounted

Summary

Three related defects in LLMAgent.process_message() cause actual LLM token consumption to be undercounted in ActorResult / NodeUsage:

Pruning model LLM calls are never tracked. _run_pruning_pass() creates a separate build_chat_model() instance and calls ainvoke(), but never extracts usage_metadata from the response. The provider charges for these tokens; billing reports do not reflect them.
Earlier tool-loop rounds' output tokens are overwritten. The tool-call loop reuses the response variable across iterations. Only the final round's usage_metadata is captured at line 1252. Each earlier round's completion tokens are lost.
ADR D-6 actual-usage-based estimates unimplemented. _estimate_token_count() does not reference _last_token_usage to refine estimates from prior rounds' actual consumption — it relies solely on get_num_tokens() or a len//4 heuristic.

Root Cause

Gap 1: `_run_pruning_pass` (llm.py:468–536)

# Line 484–527
prune_model = build_chat_model(...)          # separate model instance
prune_response = await prune_model.ainvoke(prune_messages)  # tokens consumed
return self._parse_pruning_response(str(prune_response.content))  # usage_metadata discarded!

The response object carries usage_metadata.input_tokens + usage_metadata.output_tokens, but these are never read. Meanwhile _last_token_usage (line 1252) is set only from the main model's final response after the tool-loop exits.

Gap 2: Loop variable reuse (llm.py:759–1051)

for _tool_round in range(_TOOL_MAX_ROUNDS):
    # ...
    response = await self.chat_model.ainvoke(messages, **invoke_kwargs)  # ← overwritten each round
    # ... break when no tool_calls
# Line 1171–1253: only response from the FINAL ainvoke() is captured

For a 5-round loop, ~80% of completion token counts are missing from billing.

Gap 3: `_estimate_token_count` (llm.py:371–386)

Does not use self._last_token_usage — only get_num_tokens() or len//4 heuristic. The ADR D-6 intent to "use this actual consumption data to improve estimation accuracy" was not wired up.

Affected Code

File	Lines	Role
`src/cleveractors/agents/llm.py`	468–536	`_run_pruning_pass()` — drops `usage_metadata`
`src/cleveractors/agents/llm.py`	759–1051	Tool-loop — `response` overwritten each round
`src/cleveractors/agents/llm.py`	1171–1253	Token extraction — only last response read
`src/cleveractors/agents/llm.py`	371–386	`_estimate_token_count()` — ignores prior actuals

Important thing to take into account

Different models, different costs: Note that the model used for pruning, may or may not be the same model employed in the main agent so costs can be different (see pruning_model configuration property), and this needs to be accounted for in the token usage, or cost calculation. If this is not feasible, inform in implementation report, in which case a proper approach will need to be devised by the sofware architect.

Temporary Limitations (implemented in PR #66)

1. `pruning_model` must match `model` (deviation from ADR-2031 §4.4.8)

A ConfigurationError is raised at agent creation time if pruning_model is set to a value different from model. This is because token tracking across separate LangChain model instances (main model vs pruning model) has not yet been plumbed through NodeUsage. The NodeUsage type currently holds a single (prompt_tokens, completion_tokens) tuple per node; there is no mechanism to attribute tokens to the pruning model vs the main model.

To lift this limitation, NodeUsage (in src/cleveractors/result.py) would need:

A new field, e.g. pruner_prompt_tokens / pruner_completion_tokens, or a more general per_model: dict[str, tuple[int, int]] breakdown.
The outer router (runtime_dispatch.py) would need to sum these fields together for total cost, or expose them separately for per-model billing.

Until this is done, any agent that attempts to configure a separate pruning_model will fail fast on construction.

2. Pruning token usage metadata is enforced (deviation from ADR-2031 §4.4.8)

If the pruning model's response lacks usage_metadata (and has no response_metadata["token_usage"] fallback), a RuntimeError is raised immediately — the old silent fallback to (0, 0) is removed. This ensures the caller is always alerted when billing data would be incomplete.

The error propagates through process_message()'s outer except Exception handler and is re-raised as ExecutionError("LLM processing failed: ...") with the original message preserved.

Impact

Agents must use models that report usage_metadata (Anthropic, OpenAI, and most LangChain-compatible providers do). Models that do not return usage info cannot be used with pruning enabled.
The pruning_model field is effectively disabled until cross-model token tracking is implemented in NodeUsage.

Impact

Billing underreporting: pruning-model tokens are entirely invisible; earlier rounds' completion tokens are missing. A 4-prune, 5-round execution could underreport by 50–80%.
Cost-limit evasion: max_cost_usd enforcement in runtime_dispatch.py uses the same undercounted NodeUsage data, so the budget guardrail is weaker than intended.
Operational blind spot: log events (budget_warning, budget_exhausted) and token-budget estimation both underestimate true consumption.

No Existing Coverage

No BDD scenarios verify that pruning-call token counts appear in ActorResult / NodeUsage.
No unit tests exercise multi-round accumulation across the tool loop.
features/llm_agent_tool_calling.feature covers pruning mechanics (schema augmentation, marker parsing, error fallback) but omits token-tracking assertions.

Suggested Fix Direction

In _run_pruning_pass, capture usage_metadata (or response_metadata.token_usage) from the pruning model's response and return it alongside the parsed content.
Accumulate token counts across all tool-loop rounds (summing each ainvoke() response's metadata) instead of overwriting.
Aggregate pruning-pass token counts into the accumulated total.
Optionally, have _estimate_token_count() blend prior _last_token_usage actuals with the heuristic for better budget warnings.

Discovered during code review of src/cleveractors/agents/llm.py against ADR-2031 and ADR-2027.

## Metadata Commit Message: fix(actors): llm token consumption undercounted Branch: bugfix/m2-lllm-token-consumption-undercounted ## Summary Three related defects in `LLMAgent.process_message()` cause actual LLM token consumption to be undercounted in `ActorResult` / `NodeUsage`: 1. **Pruning model LLM calls are never tracked.** `_run_pruning_pass()` creates a separate `build_chat_model()` instance and calls `ainvoke()`, but never extracts `usage_metadata` from the response. The provider charges for these tokens; billing reports do not reflect them. 2. **Earlier tool-loop rounds' output tokens are overwritten.** The tool-call loop reuses the `response` variable across iterations. Only the final round's `usage_metadata` is captured at line 1252. Each earlier round's completion tokens are lost. 3. **ADR D-6 actual-usage-based estimates unimplemented.** `_estimate_token_count()` does not reference `_last_token_usage` to refine estimates from prior rounds' actual consumption — it relies solely on `get_num_tokens()` or a `len//4` heuristic. ## Root Cause ### Gap 1: `_run_pruning_pass` (llm.py:468–536) ```python # Line 484–527 prune_model = build_chat_model(...) # separate model instance prune_response = await prune_model.ainvoke(prune_messages) # tokens consumed return self._parse_pruning_response(str(prune_response.content)) # usage_metadata discarded! ``` The response object carries `usage_metadata.input_tokens` + `usage_metadata.output_tokens`, but these are never read. Meanwhile `_last_token_usage` (line 1252) is set only from the main model's final `response` after the tool-loop exits. ### Gap 2: Loop variable reuse (llm.py:759–1051) ```python for _tool_round in range(_TOOL_MAX_ROUNDS): # ... response = await self.chat_model.ainvoke(messages, **invoke_kwargs) # ← overwritten each round # ... break when no tool_calls # Line 1171–1253: only response from the FINAL ainvoke() is captured ``` For a 5-round loop, ~80% of completion token counts are missing from billing. ### Gap 3: `_estimate_token_count` (llm.py:371–386) Does not use `self._last_token_usage` — only `get_num_tokens()` or `len//4` heuristic. The ADR D-6 intent to "use this actual consumption data to improve estimation accuracy" was not wired up. ## Affected Code | File | Lines | Role | |------|-------|------| | `src/cleveractors/agents/llm.py` | 468–536 | `_run_pruning_pass()` — drops `usage_metadata` | | `src/cleveractors/agents/llm.py` | 759–1051 | Tool-loop — `response` overwritten each round | | `src/cleveractors/agents/llm.py` | 1171–1253 | Token extraction — only last response read | | `src/cleveractors/agents/llm.py` | 371–386 | `_estimate_token_count()` — ignores prior actuals | ## Important thing to take into account - **Different models, different costs**: Note that the model used for pruning, may or may not be the same model employed in the main agent so costs can be different (see `pruning_model` configuration property), and this needs to be accounted for in the token usage, or cost calculation. If this is not feasible, inform in implementation report, in which case a proper approach will need to be devised by the sofware architect. ## Temporary Limitations (implemented in PR #66) ### 1. `pruning_model` must match `model` (deviation from ADR-2031 §4.4.8) A `ConfigurationError` is raised at agent creation time if `pruning_model` is set to a value different from `model`. This is because token tracking across separate LangChain model instances (main model vs pruning model) has not yet been plumbed through `NodeUsage`. The `NodeUsage` type currently holds a single `(prompt_tokens, completion_tokens)` tuple per node; there is no mechanism to attribute tokens to the pruning model vs the main model. **To lift this limitation**, `NodeUsage` (in `src/cleveractors/result.py`) would need: - A new field, e.g. `pruner_prompt_tokens` / `pruner_completion_tokens`, or a more general `per_model: dict[str, tuple[int, int]]` breakdown. - The outer router (`runtime_dispatch.py`) would need to sum these fields together for total cost, or expose them separately for per-model billing. Until this is done, any agent that attempts to configure a separate `pruning_model` will fail fast on construction. ### 2. Pruning token usage metadata is enforced (deviation from ADR-2031 §4.4.8) If the pruning model's response lacks `usage_metadata` (and has no `response_metadata["token_usage"]` fallback), a `RuntimeError` is raised immediately — the old silent fallback to `(0, 0)` is removed. This ensures the caller is always alerted when billing data would be incomplete. The error propagates through `process_message()`'s outer `except Exception` handler and is re-raised as `ExecutionError("LLM processing failed: ...")` with the original message preserved. ### Impact - Agents must use models that report `usage_metadata` (Anthropic, OpenAI, and most LangChain-compatible providers do). Models that do not return usage info cannot be used with pruning enabled. - The `pruning_model` field is effectively disabled until cross-model token tracking is implemented in `NodeUsage`. ## Impact - **Billing underreporting**: pruning-model tokens are entirely invisible; earlier rounds' completion tokens are missing. A 4-prune, 5-round execution could underreport by 50–80%. - **Cost-limit evasion**: `max_cost_usd` enforcement in `runtime_dispatch.py` uses the same undercounted `NodeUsage` data, so the budget guardrail is weaker than intended. - **Operational blind spot**: log events (`budget_warning`, `budget_exhausted`) and token-budget estimation both underestimate true consumption. ## No Existing Coverage - No BDD scenarios verify that pruning-call token counts appear in `ActorResult` / `NodeUsage`. - No unit tests exercise multi-round accumulation across the tool loop. - `features/llm_agent_tool_calling.feature` covers pruning mechanics (schema augmentation, marker parsing, error fallback) but omits token-tracking assertions. ## Suggested Fix Direction 1. In `_run_pruning_pass`, capture `usage_metadata` (or `response_metadata.token_usage`) from the pruning model's response and return it alongside the parsed content. 2. Accumulate token counts across all tool-loop rounds (summing each `ainvoke()` response's metadata) instead of overwriting. 3. Aggregate pruning-pass token counts into the accumulated total. 4. Optionally, have `_estimate_token_count()` blend prior `_last_token_usage` actuals with the heuristic for better budget warnings. --- _Discovered during code review of `src/cleveractors/agents/llm.py` against ADR-2031 and ADR-2027._

CoreRasurae added the

labels

2026-06-30 13:02:01 +00:00

CoreRasurae referenced this issue from a commit

2026-06-30 14:20:20 +00:00

fix(actors): llm token consumption undercounted

CoreRasurae referenced this issue from a pull request that will close it,

2026-06-30 14:20:42 +00:00