freemo
3f14cbbf7e
CI / lint (pull_request) Successful in 40s
CI / security (pull_request) Successful in 32s
CI / quality (pull_request) Successful in 42s
CI / typecheck (pull_request) Successful in 1m1s
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 14s
CI / unit_tests (pull_request) Successful in 3m58s
CI / integration_tests (pull_request) Successful in 4m34s
CI / docker (pull_request) Successful in 1m6s
CI / coverage (pull_request) Successful in 6m21s
CI / quality (push) Successful in 16s
CI / lint (push) Successful in 21s
CI / security (push) Successful in 32s
CI / typecheck (push) Successful in 35s
CI / benchmark-regression (push) Has been skipped
CI / build (push) Successful in 39s
CI / unit_tests (push) Successful in 2m38s
CI / integration_tests (push) Successful in 3m2s
CI / docker (push) Successful in 40s
CI / coverage (push) Successful in 4m9s
CI / benchmark-publish (push) Successful in 14m42s
CI / benchmark-regression (pull_request) Successful in 26m29s
Add LLMTrace Pydantic v2 domain model with all required fields (trace_id, plan_id, decision_id, actor, provider, model, prompt_tokens, completion_tokens, cost_usd, latency_ms, tool_calls, context_hash, streaming, retry_count, error). Define 14 OperationalMetricKey values (PLAN_DURATION_MS, PLAN_TOTAL_COST_USD, PLAN_DECISION_COUNT, ACTOR_INVOCATION_COUNT, ACTOR_LATENCY_MS, TOOL_INVOCATION_COUNT, TOOL_ERROR_RATE, CONTEXT_BUILD_TIME_MS, CONTEXT_TOKEN_COUNT, LLM_CALL_COUNT, LLM_TOTAL_TOKENS, LLM_TOTAL_COST_USD, LLM_AVG_LATENCY_MS, SUBPLAN_COUNT) with MetricEntry model and MetricCollector. Add llm_traces database table (LLMTraceModel) with LLMTraceRepository for persistence. TraceService provides recording, querying, metric computation, plan lifecycle hooks, and optional LangSmith forwarding when LANGCHAIN_TRACING_V2=true. Wired into DI container as trace_service. Includes 28 Behave BDD scenarios, 6 Robot Framework smoke tests, 3 ASV benchmark suites, and reference documentation. Fix pre-existing cli_core server_mode test flake by mocking resolve_server_mode in test steps to avoid stale config file interference. Closes #500
48 lines
2.1 KiB
Plaintext
48 lines
2.1 KiB
Plaintext
*** Settings ***
|
|
Documentation Smoke tests for LLM trace observability
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_llm_trace.py
|
|
|
|
*** Test Cases ***
|
|
Record And Retrieve LLM Trace
|
|
[Documentation] Record a trace and verify retrieval by ID
|
|
${result}= Run Process ${PYTHON} ${HELPER} record-and-retrieve cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-record-ok
|
|
|
|
List Traces By Plan
|
|
[Documentation] Record multiple traces and list by plan ID
|
|
${result}= Run Process ${PYTHON} ${HELPER} list-by-plan cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-list-plan-ok
|
|
|
|
Compute Metrics For Plan
|
|
[Documentation] Compute operational metrics from recorded traces
|
|
${result}= Run Process ${PYTHON} ${HELPER} compute-metrics cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-metrics-ok
|
|
|
|
Verify Metric Key Count
|
|
[Documentation] All 14 operational metric keys should be defined
|
|
${result}= Run Process ${PYTHON} ${HELPER} metric-key-count cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-metric-keys-ok
|
|
|
|
Verify LLM Trace Model Validation
|
|
[Documentation] Model should reject invalid inputs
|
|
${result}= Run Process ${PYTHON} ${HELPER} validation cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-validation-ok
|
|
|
|
Verify Lifecycle Hooks
|
|
[Documentation] Lifecycle hooks should produce correct metric entries
|
|
${result}= Run Process ${PYTHON} ${HELPER} lifecycle-hooks cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} llm-trace-hooks-ok
|