freemo
3f14cbbf7e
CI / lint (pull_request) Successful in 40s
CI / security (pull_request) Successful in 32s
CI / quality (pull_request) Successful in 42s
CI / typecheck (pull_request) Successful in 1m1s
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 14s
CI / unit_tests (pull_request) Successful in 3m58s
CI / integration_tests (pull_request) Successful in 4m34s
CI / docker (pull_request) Successful in 1m6s
CI / coverage (pull_request) Successful in 6m21s
CI / quality (push) Successful in 16s
CI / lint (push) Successful in 21s
CI / security (push) Successful in 32s
CI / typecheck (push) Successful in 35s
CI / benchmark-regression (push) Has been skipped
CI / build (push) Successful in 39s
CI / unit_tests (push) Successful in 2m38s
CI / integration_tests (push) Successful in 3m2s
CI / docker (push) Successful in 40s
CI / coverage (push) Successful in 4m9s
CI / benchmark-publish (push) Successful in 14m42s
CI / benchmark-regression (pull_request) Successful in 26m29s
Add LLMTrace Pydantic v2 domain model with all required fields (trace_id, plan_id, decision_id, actor, provider, model, prompt_tokens, completion_tokens, cost_usd, latency_ms, tool_calls, context_hash, streaming, retry_count, error). Define 14 OperationalMetricKey values (PLAN_DURATION_MS, PLAN_TOTAL_COST_USD, PLAN_DECISION_COUNT, ACTOR_INVOCATION_COUNT, ACTOR_LATENCY_MS, TOOL_INVOCATION_COUNT, TOOL_ERROR_RATE, CONTEXT_BUILD_TIME_MS, CONTEXT_TOKEN_COUNT, LLM_CALL_COUNT, LLM_TOTAL_TOKENS, LLM_TOTAL_COST_USD, LLM_AVG_LATENCY_MS, SUBPLAN_COUNT) with MetricEntry model and MetricCollector. Add llm_traces database table (LLMTraceModel) with LLMTraceRepository for persistence. TraceService provides recording, querying, metric computation, plan lifecycle hooks, and optional LangSmith forwarding when LANGCHAIN_TRACING_V2=true. Wired into DI container as trace_service. Includes 28 Behave BDD scenarios, 6 Robot Framework smoke tests, 3 ASV benchmark suites, and reference documentation. Fix pre-existing cli_core server_mode test flake by mocking resolve_server_mode in test steps to avoid stale config file interference. Closes #500
90 lines
4.6 KiB
Markdown
90 lines
4.6 KiB
Markdown
# Observability — LLM Trace & Operational Metrics
|
|
|
|
## Overview
|
|
|
|
The observability subsystem records telemetry for every LLM provider
|
|
call and exposes aggregated operational metrics per plan. Trace data
|
|
drives cost analysis, latency profiling, and optional forwarding to
|
|
LangSmith for external analysis.
|
|
|
|
## LLMTrace Model
|
|
|
|
`cleveragents.domain.models.observability.llm_trace.LLMTrace`
|
|
|
|
Each trace captures one LLM invocation:
|
|
|
|
| Field | Type | Description |
|
|
|:-------------------|:------------------|:-----------------------------------------|
|
|
| `trace_id` | `str` (ULID) | Unique identifier for the trace |
|
|
| `plan_id` | `str` (ULID) | Plan that owns this trace |
|
|
| `decision_id` | `str \| None` | Decision that triggered the call |
|
|
| `actor` | `str` | Actor name |
|
|
| `provider` | `str` | Provider identifier (e.g. `openai`) |
|
|
| `model` | `str` | Model name (e.g. `gpt-4o`) |
|
|
| `prompt_tokens` | `int` | Prompt token count |
|
|
| `completion_tokens`| `int` | Completion token count |
|
|
| `cost_usd` | `float` | Estimated USD cost |
|
|
| `latency_ms` | `float` | Wall-clock latency (ms) |
|
|
| `tool_calls` | `list[dict]` | Tool call descriptors from the model |
|
|
| `context_hash` | `str \| None` | SHA-256 of the context window |
|
|
| `streaming` | `bool` | Whether streaming was used |
|
|
| `retry_count` | `int` | Retries before success |
|
|
| `error` | `str \| None` | Error message on failure |
|
|
| `timestamp` | `datetime` | When the trace was recorded |
|
|
|
|
The model is **frozen** (immutable) once created.
|
|
|
|
## Operational Metrics
|
|
|
|
`cleveragents.domain.models.observability.metrics.OperationalMetricKey`
|
|
|
|
14 metric keys grouped by subsystem:
|
|
|
|
| Key | Subsystem | Description |
|
|
|:---------------------------|:----------|:-----------------------------------|
|
|
| `PLAN_DURATION_MS` | Plan | Total plan execution time |
|
|
| `PLAN_TOTAL_COST_USD` | Plan | Aggregate cost across all LLM calls|
|
|
| `PLAN_DECISION_COUNT` | Plan | Number of decisions made |
|
|
| `SUBPLAN_COUNT` | Plan | Number of subplans spawned |
|
|
| `ACTOR_INVOCATION_COUNT` | Actor | Actor call count |
|
|
| `ACTOR_LATENCY_MS` | Actor | Actor invocation latency |
|
|
| `TOOL_INVOCATION_COUNT` | Tool | Tool execution count |
|
|
| `TOOL_ERROR_RATE` | Tool | Tool error occurrences |
|
|
| `CONTEXT_BUILD_TIME_MS` | Context | Context assembly time |
|
|
| `CONTEXT_TOKEN_COUNT` | Context | Context token count |
|
|
| `LLM_CALL_COUNT` | LLM | Total LLM calls |
|
|
| `LLM_TOTAL_TOKENS` | LLM | Sum of all tokens |
|
|
| `LLM_TOTAL_COST_USD` | LLM | Sum of all costs |
|
|
| `LLM_AVG_LATENCY_MS` | LLM | Average call latency |
|
|
|
|
## TraceService
|
|
|
|
`cleveragents.application.services.trace_service.TraceService`
|
|
|
|
Registered in the DI container as `trace_service`.
|
|
|
|
### Methods
|
|
|
|
| Method | Description |
|
|
|:------------------------|:------------------------------------------------|
|
|
| `record_trace(trace)` | Persist trace and optionally forward to LangSmith|
|
|
| `get_traces(plan_id)` | List traces for a plan |
|
|
| `get_trace(trace_id)` | Retrieve a single trace |
|
|
| `compute_metrics(plan_id)` | Compute LLM-level metrics from traces |
|
|
| `on_plan_start(plan_id)` | Lifecycle hook: plan start |
|
|
| `on_actor_invocation(...)` | Lifecycle hook: actor invocation |
|
|
| `on_tool_execution(...)` | Lifecycle hook: tool execution |
|
|
|
|
### LangSmith Forwarding
|
|
|
|
When `LANGCHAIN_TRACING_V2=true` is set in the environment,
|
|
`record_trace` automatically forwards each trace to LangSmith
|
|
via the `langsmith` SDK. Forwarding is best-effort: failures
|
|
are logged but do not raise exceptions.
|
|
|
|
## Database
|
|
|
|
Table `llm_traces` with indexes on `plan_id`, `decision_id`,
|
|
`actor`, and `provider`. Repository:
|
|
`cleveragents.infrastructure.database.llm_trace_repository.LLMTraceRepository`.
|