feat(observability): add LLMTrace model and operational metrics #500

Closed
opened 2026-03-02 04:27:29 +00:00 by freemo · 0 comments
Owner

Metadata

  • Commit Message: feat(observability): add LLMTrace model and operational metrics
  • Branch: feature/m6-llm-trace-metrics

Background

The specification (Section 18: Observability & Tracing) defines an LLMTrace model that captures per-call telemetry (provider, model, tokens, cost, latency, tool calls, context hash) and 14 operational metrics (plan duration, cost, decisions, actor invocations, tool error rates, context build times, etc.). This data powers cost dashboards, performance analysis, and LangSmith integration. Currently, the CostTracker handles budget enforcement but does not persist per-call traces or expose operational metrics.

Acceptance Criteria

  • Define LLMTrace Pydantic v2 model with fields: trace_id, plan_id, decision_id, actor, provider, model, prompt_tokens, completion_tokens, cost_usd, latency_ms, tool_calls, context_hash, streaming, retry_count, error.
  • Add llm_traces database table and repository.
  • Define 14 operational metric keys matching the specification.
  • Add metric collection hooks in plan lifecycle, actor invocation, and tool execution paths.
  • Emit LLMTrace entries from actor execution and wire to persistence.
  • Optional LangSmith forwarding when LANGCHAIN_TRACING_V2 env var is set.

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.

Subtasks

  • Define LLMTrace Pydantic v2 model.
  • Add llm_traces table and LLMTraceRepository.
  • Define 14 operational metric keys.
  • Add metric collection hooks in plan/actor/tool paths.
  • Wire LLMTrace persistence to actor execution.
  • Add optional LangSmith trace forwarding.
  • Add docs/reference/observability.md with trace schema and metric definitions.
  • Tests (Behave): Add features/llm_trace.feature for trace persistence and query.
  • Tests (Robot): Add robot/llm_trace.robot smoke tests.
  • Tests (ASV): Add benchmarks/llm_trace_bench.py for trace write throughput.
  • Verify coverage >=97% via nox -s coverage_report.
  • Run nox (all default sessions), fix any errors.
## Metadata - **Commit Message**: `feat(observability): add LLMTrace model and operational metrics` - **Branch**: `feature/m6-llm-trace-metrics` ## Background The specification (Section 18: Observability & Tracing) defines an `LLMTrace` model that captures per-call telemetry (provider, model, tokens, cost, latency, tool calls, context hash) and 14 operational metrics (plan duration, cost, decisions, actor invocations, tool error rates, context build times, etc.). This data powers cost dashboards, performance analysis, and LangSmith integration. Currently, the `CostTracker` handles budget enforcement but does not persist per-call traces or expose operational metrics. ## Acceptance Criteria - [ ] Define `LLMTrace` Pydantic v2 model with fields: `trace_id`, `plan_id`, `decision_id`, `actor`, `provider`, `model`, `prompt_tokens`, `completion_tokens`, `cost_usd`, `latency_ms`, `tool_calls`, `context_hash`, `streaming`, `retry_count`, `error`. - [ ] Add `llm_traces` database table and repository. - [ ] Define 14 operational metric keys matching the specification. - [ ] Add metric collection hooks in plan lifecycle, actor invocation, and tool execution paths. - [ ] Emit `LLMTrace` entries from actor execution and wire to persistence. - [ ] Optional LangSmith forwarding when `LANGCHAIN_TRACING_V2` env var is set. ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. ## Subtasks - [ ] Define `LLMTrace` Pydantic v2 model. - [ ] Add `llm_traces` table and `LLMTraceRepository`. - [ ] Define 14 operational metric keys. - [ ] Add metric collection hooks in plan/actor/tool paths. - [ ] Wire `LLMTrace` persistence to actor execution. - [ ] Add optional LangSmith trace forwarding. - [ ] Add `docs/reference/observability.md` with trace schema and metric definitions. - [ ] Tests (Behave): Add `features/llm_trace.feature` for trace persistence and query. - [ ] Tests (Robot): Add `robot/llm_trace.robot` smoke tests. - [ ] Tests (ASV): Add `benchmarks/llm_trace_bench.py` for trace write throughput. - [ ] Verify coverage >=97% via `nox -s coverage_report`. - [ ] Run `nox` (all default sessions), fix any errors.
freemo added this to the v3.5.0 milestone 2026-03-02 04:27:52 +00:00
freemo self-assigned this 2026-03-02 16:26:07 +00:00
freemo 2026-03-03 22:12:14 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#500
No description provided.