UAT: `v3_plans` table has spec-required columns `token_count_input`, `token_count_output`, and `cost_actual_usd` that are never populated — dead columns causing data loss #4504

New issue

Open

opened 2026-04-08 13:54:33 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-08 13:54:33 +00:00

Owner

Bug Report

Tested by: UAT worker instance uat-worker-storage-persistence-001
Feature area: Storage and Persistence Layer
Severity: Medium — spec-required data never persisted, data completeness gap

Summary

The v3_plans table has three columns that are defined in both the spec DDL and the ORM model (LifecyclePlanModel) but are never populated by from_domain() or any service layer code:

token_count_input — always NULL/0
token_count_output — always NULL/0
cost_actual_usd — always NULL

These columns exist in the spec DDL and ORM model but are dead code — they're never written to, and to_domain() doesn't read them back either.

Evidence

Spec DDL (docs/specification.md, line 45729):

token_count_input INTEGER DEFAULT 0,
token_count_output INTEGER DEFAULT 0,
cost_actual_usd REAL,

ORM model (src/cleveragents/infrastructure/database/models.py, lines 695-698):

cost_actual_usd = Column(Float, nullable=True)
token_count_input = Column(Integer, nullable=True, default=0)
token_count_output = Column(Integer, nullable=True, default=0)

from_domain() does NOT populate these fields — searching the LifecyclePlanModel.from_domain() method shows no assignment to token_count_input, token_count_output, or cost_actual_usd.

to_domain() does NOT read these fields — the Plan domain model has no token_count_input, token_count_output, or cost_actual_usd fields at the top level (token tracking is done via CostMetadata in cost_metadata).

Root Cause

The Plan domain model tracks token usage via CostMetadata (a nested Pydantic model), but the ORM model has flat columns for these values that were never wired up to the domain model. The cost_estimate_usd column IS populated (from plan.cost_estimate_usd), but cost_actual_usd is not.

Expected Behavior (per spec)

Per docs/specification.md §Storage and Persistence:

The relational database follows a normalized design with foreign key constraints enforcing referential integrity.

The spec DDL explicitly includes token_count_input, token_count_output, and cost_actual_usd as required columns. These should be populated from the domain model's CostMetadata when persisting plans.

Impact

Data loss: Token usage and actual cost data is tracked in memory via CostMetadata but never persisted to the database
Analytics gap: agents plan list and plan history queries cannot show actual token usage or costs
Billing/audit gap: The spec requires cost tracking for compliance; cost_actual_usd is always NULL

Fix Required

Wire token_count_input and token_count_output from plan.cost_metadata (if available) in LifecyclePlanModel.from_domain()
Wire cost_actual_usd from plan.cost_metadata.total_cost_usd (if available) in LifecyclePlanModel.from_domain()
Read these values back in to_domain() and populate the domain model's cost_metadata accordingly

Metadata

Commit Message: fix(db): wire token_count and cost_actual_usd columns in LifecyclePlanModel
Branch Name: fix/plan-token-cost-persistence

Subtasks

Populate token_count_input and token_count_output from CostMetadata in from_domain()
Populate cost_actual_usd from CostMetadata.total_cost_usd in from_domain()
Read these values back in to_domain() and populate CostMetadata
Add Behave scenario verifying token counts are persisted and retrieved correctly

Definition of Done

token_count_input and token_count_output are populated when a plan with token usage is saved
cost_actual_usd is populated when a plan with actual cost is saved
Round-trip persistence test passes (save → load → verify values match)
All existing tests pass

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Tested by:** UAT worker instance `uat-worker-storage-persistence-001` **Feature area:** Storage and Persistence Layer **Severity:** Medium — spec-required data never persisted, data completeness gap --- ## Summary The `v3_plans` table has three columns that are defined in both the spec DDL and the ORM model (`LifecyclePlanModel`) but are **never populated** by `from_domain()` or any service layer code: 1. `token_count_input` — always NULL/0 2. `token_count_output` — always NULL/0 3. `cost_actual_usd` — always NULL These columns exist in the spec DDL and ORM model but are dead code — they're never written to, and `to_domain()` doesn't read them back either. --- ## Evidence **Spec DDL** (`docs/specification.md`, line 45729): ```sql token_count_input INTEGER DEFAULT 0, token_count_output INTEGER DEFAULT 0, cost_actual_usd REAL, ``` **ORM model** (`src/cleveragents/infrastructure/database/models.py`, lines 695-698): ```python cost_actual_usd = Column(Float, nullable=True) token_count_input = Column(Integer, nullable=True, default=0) token_count_output = Column(Integer, nullable=True, default=0) ``` **`from_domain()` does NOT populate these fields** — searching the `LifecyclePlanModel.from_domain()` method shows no assignment to `token_count_input`, `token_count_output`, or `cost_actual_usd`. **`to_domain()` does NOT read these fields** — the `Plan` domain model has no `token_count_input`, `token_count_output`, or `cost_actual_usd` fields at the top level (token tracking is done via `CostMetadata` in `cost_metadata`). --- ## Root Cause The `Plan` domain model tracks token usage via `CostMetadata` (a nested Pydantic model), but the ORM model has flat columns for these values that were never wired up to the domain model. The `cost_estimate_usd` column IS populated (from `plan.cost_estimate_usd`), but `cost_actual_usd` is not. --- ## Expected Behavior (per spec) Per `docs/specification.md` §Storage and Persistence: > The relational database follows a normalized design with foreign key constraints enforcing referential integrity. The spec DDL explicitly includes `token_count_input`, `token_count_output`, and `cost_actual_usd` as required columns. These should be populated from the domain model's `CostMetadata` when persisting plans. --- ## Impact 1. **Data loss**: Token usage and actual cost data is tracked in memory via `CostMetadata` but never persisted to the database 2. **Analytics gap**: `agents plan list` and plan history queries cannot show actual token usage or costs 3. **Billing/audit gap**: The spec requires cost tracking for compliance; `cost_actual_usd` is always NULL --- ## Fix Required 1. Wire `token_count_input` and `token_count_output` from `plan.cost_metadata` (if available) in `LifecyclePlanModel.from_domain()` 2. Wire `cost_actual_usd` from `plan.cost_metadata.total_cost_usd` (if available) in `LifecyclePlanModel.from_domain()` 3. Read these values back in `to_domain()` and populate the domain model's `cost_metadata` accordingly --- ## Metadata **Commit Message:** `fix(db): wire token_count and cost_actual_usd columns in LifecyclePlanModel` **Branch Name:** `fix/plan-token-cost-persistence` ### Subtasks - [ ] Populate `token_count_input` and `token_count_output` from `CostMetadata` in `from_domain()` - [ ] Populate `cost_actual_usd` from `CostMetadata.total_cost_usd` in `from_domain()` - [ ] Read these values back in `to_domain()` and populate `CostMetadata` - [ ] Add Behave scenario verifying token counts are persisted and retrieved correctly ### Definition of Done - [ ] `token_count_input` and `token_count_output` are populated when a plan with token usage is saved - [ ] `cost_actual_usd` is populated when a plan with actual cost is saved - [ ] Round-trip persistence test passes (save → load → verify values match) - [ ] All existing tests pass --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added this to the v3.5.0 milestone

2026-04-08 17:42:06 +00:00

HAL9000 added the