forked from cleveragents/cleveragents-core
freemo
3f14cbbf7e
Add LLMTrace Pydantic v2 domain model with all required fields (trace_id, plan_id, decision_id, actor, provider, model, prompt_tokens, completion_tokens, cost_usd, latency_ms, tool_calls, context_hash, streaming, retry_count, error). Define 14 OperationalMetricKey values (PLAN_DURATION_MS, PLAN_TOTAL_COST_USD, PLAN_DECISION_COUNT, ACTOR_INVOCATION_COUNT, ACTOR_LATENCY_MS, TOOL_INVOCATION_COUNT, TOOL_ERROR_RATE, CONTEXT_BUILD_TIME_MS, CONTEXT_TOKEN_COUNT, LLM_CALL_COUNT, LLM_TOTAL_TOKENS, LLM_TOTAL_COST_USD, LLM_AVG_LATENCY_MS, SUBPLAN_COUNT) with MetricEntry model and MetricCollector. Add llm_traces database table (LLMTraceModel) with LLMTraceRepository for persistence. TraceService provides recording, querying, metric computation, plan lifecycle hooks, and optional LangSmith forwarding when LANGCHAIN_TRACING_V2=true. Wired into DI container as trace_service. Includes 28 Behave BDD scenarios, 6 Robot Framework smoke tests, 3 ASV benchmark suites, and reference documentation. Fix pre-existing cli_core server_mode test flake by mocking resolve_server_mode in test steps to avoid stale config file interference. Closes #500
48 lines
2.1 KiB
Plaintext
48 lines
2.1 KiB
Plaintext
*** Settings ***
|
|
Documentation Smoke tests for LLM trace observability
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_llm_trace.py
|
|
|
|
*** Test Cases ***
|
|
Record And Retrieve LLM Trace
|
|
[Documentation] Record a trace and verify retrieval by ID
|
|
${result}= Run Process ${PYTHON} ${HELPER} record-and-retrieve cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-record-ok
|
|
|
|
List Traces By Plan
|
|
[Documentation] Record multiple traces and list by plan ID
|
|
${result}= Run Process ${PYTHON} ${HELPER} list-by-plan cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-list-plan-ok
|
|
|
|
Compute Metrics For Plan
|
|
[Documentation] Compute operational metrics from recorded traces
|
|
${result}= Run Process ${PYTHON} ${HELPER} compute-metrics cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-metrics-ok
|
|
|
|
Verify Metric Key Count
|
|
[Documentation] All 14 operational metric keys should be defined
|
|
${result}= Run Process ${PYTHON} ${HELPER} metric-key-count cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-metric-keys-ok
|
|
|
|
Verify LLM Trace Model Validation
|
|
[Documentation] Model should reject invalid inputs
|
|
${result}= Run Process ${PYTHON} ${HELPER} validation cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-validation-ok
|
|
|
|
Verify Lifecycle Hooks
|
|
[Documentation] Lifecycle hooks should produce correct metric entries
|
|
${result}= Run Process ${PYTHON} ${HELPER} lifecycle-hooks cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-hooks-ok
|