Commit Graph

1 Commits

Author SHA1 Message Date
freemo 3f14cbbf7e feat(observability): add LLMTrace model and operational metrics
CI / lint (pull_request) Successful in 40s
CI / security (pull_request) Successful in 32s
CI / quality (pull_request) Successful in 42s
CI / typecheck (pull_request) Successful in 1m1s
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 14s
CI / unit_tests (pull_request) Successful in 3m58s
CI / integration_tests (pull_request) Successful in 4m34s
CI / docker (pull_request) Successful in 1m6s
CI / coverage (pull_request) Successful in 6m21s
CI / quality (push) Successful in 16s
CI / lint (push) Successful in 21s
CI / security (push) Successful in 32s
CI / typecheck (push) Successful in 35s
CI / benchmark-regression (push) Has been skipped
CI / build (push) Successful in 39s
CI / unit_tests (push) Successful in 2m38s
CI / integration_tests (push) Successful in 3m2s
CI / docker (push) Successful in 40s
CI / coverage (push) Successful in 4m9s
CI / benchmark-publish (push) Successful in 14m42s
CI / benchmark-regression (pull_request) Successful in 26m29s
Add LLMTrace Pydantic v2 domain model with all required fields (trace_id,
plan_id, decision_id, actor, provider, model, prompt_tokens, completion_tokens,
cost_usd, latency_ms, tool_calls, context_hash, streaming, retry_count, error).

Define 14 OperationalMetricKey values (PLAN_DURATION_MS, PLAN_TOTAL_COST_USD,
PLAN_DECISION_COUNT, ACTOR_INVOCATION_COUNT, ACTOR_LATENCY_MS,
TOOL_INVOCATION_COUNT, TOOL_ERROR_RATE, CONTEXT_BUILD_TIME_MS,
CONTEXT_TOKEN_COUNT, LLM_CALL_COUNT, LLM_TOTAL_TOKENS, LLM_TOTAL_COST_USD,
LLM_AVG_LATENCY_MS, SUBPLAN_COUNT) with MetricEntry model and MetricCollector.

Add llm_traces database table (LLMTraceModel) with LLMTraceRepository for
persistence. TraceService provides recording, querying, metric computation,
plan lifecycle hooks, and optional LangSmith forwarding when
LANGCHAIN_TRACING_V2=true.

Wired into DI container as trace_service. Includes 28 Behave BDD scenarios,
6 Robot Framework smoke tests, 3 ASV benchmark suites, and reference
documentation.

Fix pre-existing cli_core server_mode test flake by mocking
resolve_server_mode in test steps to avoid stale config file interference.

Closes #500
2026-03-03 17:04:17 -05:00