UAT: `LLMTrace` model missing `total_tokens`, `temperature`, and `context_refs` fields required by spec #3937

New issue

Closed

opened 2026-04-06 07:38:19 +00:00 by freemo · 1 comment

freemo commented

2026-04-06 07:38:19 +00:00

Owner

Metadata

Branch: fix/llm-trace-missing-fields-total-tokens-temperature-context-refs
Commit Message: fix(observability): add missing total_tokens, temperature, and context_refs fields to LLMTrace model
Milestone: (none — backlog)
Parent Epic: #945

Summary

The LLMTrace domain model in src/cleveragents/domain/models/observability/llm_trace.py is missing three fields defined in the specification: total_tokens, temperature, and context_refs. These fields are required for complete LLM call observability.

Expected Behavior (from spec)

Per docs/specification.md §LLM Call Tracing (line ~45982):

class LLMTrace(BaseModel):
    trace_id: str                     # ULID
    plan_id: str
    decision_id: str | None
    actor_name: str
    provider: str
    model: str
    prompt_tokens: int
    completion_tokens: int
    total_tokens: int          # ← MISSING in implementation
    cost_usd: float
    latency_ms: int
    temperature: float         # ← MISSING in implementation
    tool_calls: list[str]
    context_hash: str
    context_refs: list[str]    # ← MISSING in implementation (resource IDs referenced in context)
    streaming: bool
    retry_count: int
    error: str | None
    timestamp: datetime

Actual Behavior

The current LLMTrace model (src/cleveragents/domain/models/observability/llm_trace.py) is missing:

total_tokens: int — total token count (prompt + completion)
temperature: float — LLM temperature setting used for the call
context_refs: list[str] — resource IDs referenced in the context window

Additionally, the spec defines actor_name: str but the implementation uses actor: str (field name mismatch).

Root Cause

The LLMTrace model was implemented with a subset of the spec-required fields. Three fields were omitted and one field has a different name than specified.

Impact

total_tokens is missing: token usage reporting is incomplete (must be computed from prompt + completion)
temperature is missing: cannot correlate LLM behavior with temperature settings
context_refs is missing: cannot trace which resources were referenced in each LLM call
actor_name vs actor naming inconsistency with spec

Code Location

src/cleveragents/domain/models/observability/llm_trace.py — LLMTrace class

Subtasks

Add total_tokens: int = Field(default=0, ge=0, ...) to LLMTrace (or compute from prompt + completion)
Add temperature: float = Field(default=0.0, ge=0.0, le=2.0, ...) to LLMTrace
Add context_refs: list[str] = Field(default_factory=list, ...) to LLMTrace
Evaluate whether to rename actor to actor_name for spec alignment (breaking change)
Update TraceService.record_trace() to accept and persist new fields
Update LangSmith forwarding to include new fields
Update unit tests (Behave) to cover new fields
Update integration tests (Robot Framework) to cover new fields
Verify coverage ≥ 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

LLMTrace has total_tokens, temperature, and context_refs fields matching the spec
actor_name field alignment evaluated and resolved
TraceService persists all new fields
Unit tests (Behave) cover new fields
Integration tests (Robot Framework) cover new fields
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done
All nox stages pass
Coverage >= 97%

Backlog note: This issue was discovered during autonomous operation
on milestone v3.6.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/llm-trace-missing-fields-total-tokens-temperature-context-refs` - **Commit Message**: `fix(observability): add missing total_tokens, temperature, and context_refs fields to LLMTrace model` - **Milestone**: *(none — backlog)* - **Parent Epic**: #945 ## Summary The `LLMTrace` domain model in `src/cleveragents/domain/models/observability/llm_trace.py` is missing three fields defined in the specification: `total_tokens`, `temperature`, and `context_refs`. These fields are required for complete LLM call observability. ## Expected Behavior (from spec) Per `docs/specification.md` §LLM Call Tracing (line ~45982): ```python class LLMTrace(BaseModel): trace_id: str # ULID plan_id: str decision_id: str | None actor_name: str provider: str model: str prompt_tokens: int completion_tokens: int total_tokens: int # ← MISSING in implementation cost_usd: float latency_ms: int temperature: float # ← MISSING in implementation tool_calls: list[str] context_hash: str context_refs: list[str] # ← MISSING in implementation (resource IDs referenced in context) streaming: bool retry_count: int error: str | None timestamp: datetime ``` ## Actual Behavior The current `LLMTrace` model (`src/cleveragents/domain/models/observability/llm_trace.py`) is missing: 1. `total_tokens: int` — total token count (prompt + completion) 2. `temperature: float` — LLM temperature setting used for the call 3. `context_refs: list[str]` — resource IDs referenced in the context window Additionally, the spec defines `actor_name: str` but the implementation uses `actor: str` (field name mismatch). ## Root Cause The `LLMTrace` model was implemented with a subset of the spec-required fields. Three fields were omitted and one field has a different name than specified. ## Impact - `total_tokens` is missing: token usage reporting is incomplete (must be computed from prompt + completion) - `temperature` is missing: cannot correlate LLM behavior with temperature settings - `context_refs` is missing: cannot trace which resources were referenced in each LLM call - `actor_name` vs `actor` naming inconsistency with spec ## Code Location `src/cleveragents/domain/models/observability/llm_trace.py` — `LLMTrace` class ## Subtasks - [ ] Add `total_tokens: int = Field(default=0, ge=0, ...)` to `LLMTrace` (or compute from prompt + completion) - [ ] Add `temperature: float = Field(default=0.0, ge=0.0, le=2.0, ...)` to `LLMTrace` - [ ] Add `context_refs: list[str] = Field(default_factory=list, ...)` to `LLMTrace` - [ ] Evaluate whether to rename `actor` to `actor_name` for spec alignment (breaking change) - [ ] Update `TraceService.record_trace()` to accept and persist new fields - [ ] Update LangSmith forwarding to include new fields - [ ] Update unit tests (Behave) to cover new fields - [ ] Update integration tests (Robot Framework) to cover new fields - [ ] Verify coverage ≥ 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - [ ] `LLMTrace` has `total_tokens`, `temperature`, and `context_refs` fields matching the spec - [ ] `actor_name` field alignment evaluated and resolved - [ ] `TraceService` persists all new fields - [ ] Unit tests (Behave) cover new fields - [ ] Integration tests (Robot Framework) cover new fields - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done - [ ] All nox stages pass - [ ] Coverage >= 97% > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.6.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added a new dependency

2026-04-06 07:38:31 +00:00

#945 Legendary: Specification Alignment — Close All Gaps Between Implementation and Specification

freemo referenced this issue

2026-04-06 07:39:13 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 07:49:16 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo commented

2026-04-06 09:58:05 +00:00

Author

Owner

Duplicate of #3765

This issue covers the same problem as #3765: LLMTrace domain model missing total_tokens, temperature, and context_refs fields (and actor vs actor_name naming inconsistency). Issue #3765 was filed earlier (2026-04-05T22:33:41Z vs this issue at 2026-04-06T07:38:19Z) and contains an equivalent analysis. Closing as a duplicate — please track the fix in #3765.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

**Duplicate of #3765** This issue covers the same problem as #3765: `LLMTrace` domain model missing `total_tokens`, `temperature`, and `context_refs` fields (and `actor` vs `actor_name` naming inconsistency). Issue #3765 was filed earlier (2026-04-05T22:33:41Z vs this issue at 2026-04-06T07:38:19Z) and contains an equivalent analysis. Closing as a duplicate — please track the fix in #3765. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo closed this issue

2026-04-06 09:58:07 +00:00

freemo referenced this issue

2026-04-06 09:59:07 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775