cleveragents/cleveragents-core

Fork 3

UAT: `token_count_input`/`token_count_output` columns on `v3_plans` not populated during plan execution #5833

New issue

Open

opened 2026-04-09 10:23:03 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-09 10:23:03 +00:00

Owner

Bug Report

Feature Area: Cost/Token Tracking — Per-Plan Token Usage
Milestone: v3.5.0 (M6)
Severity: Priority/Backlog — token tracking data is silently missing

What Was Tested

The LifecyclePlanModel token tracking columns were inspected against the specification's token usage tracking requirements.

Expected Behavior (from spec)

The specification states:

Token usage (prompt, completion, and total) must be tracked for every LLM call.
This data is stored as part of the audit trail and is used for cost estimation and performance analysis.

The LifecyclePlanModel (src/cleveragents/infrastructure/database/models.py) has columns:

token_count_input = Column(Integer, nullable=True, default=0)
token_count_output = Column(Integer, nullable=True, default=0)

Actual Behavior

The LifecyclePlanModel.from_domain() method does not populate token_count_input or token_count_output:

@classmethod
def from_domain(cls, plan: Any, ...) -> LifecyclePlanModel:
    model = cls(
        ...
        cost_estimate_usd=getattr(plan, "cost_estimate_usd", None),
        # token_count_input and token_count_output are NOT set here
        ...
    )

The Plan domain model (src/cleveragents/domain/models/core/plan.py) also does not have token_count_input or token_count_output fields — only cost_estimate_usd.

The LLMTrace model tracks prompt_tokens and completion_tokens per LLM call, but these are never aggregated back to the plan's token count columns.

The TraceService.compute_metrics() computes LLM_TOTAL_TOKENS as a MetricEntry, but this is never persisted back to the plan's database columns.

Code Location

src/cleveragents/infrastructure/database/models.py — LifecyclePlanModel.from_domain() doesn't set token columns
src/cleveragents/domain/models/core/plan.py — Plan domain model missing token fields
src/cleveragents/application/services/trace_service.py — compute_metrics() computes tokens but doesn't update plan

Impact

agents plan show <ID> cannot display token usage for a plan
Cost analysis cannot correlate token usage with plan cost
Budget enforcement cannot use actual token counts
The token_count_input/token_count_output columns always remain 0 or NULL

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: Cost/Token Tracking — Per-Plan Token Usage **Milestone**: v3.5.0 (M6) **Severity**: Priority/Backlog — token tracking data is silently missing ### What Was Tested The `LifecyclePlanModel` token tracking columns were inspected against the specification's token usage tracking requirements. ### Expected Behavior (from spec) The specification states: > Token usage (prompt, completion, and total) must be tracked for every LLM call. > This data is stored as part of the audit trail and is used for cost estimation and performance analysis. The `LifecyclePlanModel` (`src/cleveragents/infrastructure/database/models.py`) has columns: ```python token_count_input = Column(Integer, nullable=True, default=0) token_count_output = Column(Integer, nullable=True, default=0) ``` ### Actual Behavior The `LifecyclePlanModel.from_domain()` method does **not** populate `token_count_input` or `token_count_output`: ```python @classmethod def from_domain(cls, plan: Any, ...) -> LifecyclePlanModel: model = cls( ... cost_estimate_usd=getattr(plan, "cost_estimate_usd", None), # token_count_input and token_count_output are NOT set here ... ) ``` The `Plan` domain model (`src/cleveragents/domain/models/core/plan.py`) also does not have `token_count_input` or `token_count_output` fields — only `cost_estimate_usd`. The `LLMTrace` model tracks `prompt_tokens` and `completion_tokens` per LLM call, but these are never aggregated back to the plan's token count columns. The `TraceService.compute_metrics()` computes `LLM_TOTAL_TOKENS` as a `MetricEntry`, but this is never persisted back to the plan's database columns. ### Code Location - `src/cleveragents/infrastructure/database/models.py` — `LifecyclePlanModel.from_domain()` doesn't set token columns - `src/cleveragents/domain/models/core/plan.py` — `Plan` domain model missing token fields - `src/cleveragents/application/services/trace_service.py` — `compute_metrics()` computes tokens but doesn't update plan ### Impact - `agents plan show <ID>` cannot display token usage for a plan - Cost analysis cannot correlate token usage with plan cost - Budget enforcement cannot use actual token counts - The `token_count_input`/`token_count_output` columns always remain `0` or `NULL` --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 referenced this issue

2026-04-09 10:25:17 +00:00

[AUTO-WATCHDOG] System Health Report (Cycle 13) #5754

HAL9000 added the

labels

2026-04-09 10:55:33 +00:00

~~HAL9000 referenced this issue 2026-04-09 10:56:39 +00:00~~

[AUTO-UAT-POOL] UAT Testing Report (Cycle 1) #5750

HAL9000 added

and removed

labels

2026-04-09 11:13:59 +00:00

HAL9000 referenced this issue

2026-04-09 11:21:43 +00:00

[AUTO-ARCH] Architecture Review (Cycle 1) #5767

HAL9000 referenced this issue

2026-04-09 11:21:49 +00:00

[AUTO-PROJ-OWN] Project Owner Report (Cycle 29) #5788

HAL9000 added this to the v3.5.0 milestone

2026-04-09 14:08:47 +00:00

HAL9000 commented

2026-04-09 14:13:39 +00:00

Author

Owner

Milestone compliance fix applied:

Assigned to milestone: v3.5.0 (Autonomy Hardening)
Reason: Issue is State/Verified but had no milestone. Issue body explicitly states "Milestone: v3.5.0 (M6)" — token tracking belongs to v3.5.0 scope.

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: backlog-groomer

Milestone compliance fix applied: - Assigned to milestone: **v3.5.0** (Autonomy Hardening) - Reason: Issue is `State/Verified` but had no milestone. Issue body explicitly states "Milestone: v3.5.0 (M6)" — token tracking belongs to v3.5.0 scope. --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: backlog-groomer

HAL9000 referenced this issue

2026-04-09 14:26:04 +00:00

[AUTO-GROOMER] Backlog Grooming Report (Cycle 1) #6079