freemo
3f14cbbf7e
CI / lint (pull_request) Successful in 40s
CI / security (pull_request) Successful in 32s
CI / quality (pull_request) Successful in 42s
CI / typecheck (pull_request) Successful in 1m1s
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 14s
CI / unit_tests (pull_request) Successful in 3m58s
CI / integration_tests (pull_request) Successful in 4m34s
CI / docker (pull_request) Successful in 1m6s
CI / coverage (pull_request) Successful in 6m21s
CI / quality (push) Successful in 16s
CI / lint (push) Successful in 21s
CI / security (push) Successful in 32s
CI / typecheck (push) Successful in 35s
CI / benchmark-regression (push) Has been skipped
CI / build (push) Successful in 39s
CI / unit_tests (push) Successful in 2m38s
CI / integration_tests (push) Successful in 3m2s
CI / docker (push) Successful in 40s
CI / coverage (push) Successful in 4m9s
CI / benchmark-publish (push) Successful in 14m42s
CI / benchmark-regression (pull_request) Successful in 26m29s
Add LLMTrace Pydantic v2 domain model with all required fields (trace_id, plan_id, decision_id, actor, provider, model, prompt_tokens, completion_tokens, cost_usd, latency_ms, tool_calls, context_hash, streaming, retry_count, error). Define 14 OperationalMetricKey values (PLAN_DURATION_MS, PLAN_TOTAL_COST_USD, PLAN_DECISION_COUNT, ACTOR_INVOCATION_COUNT, ACTOR_LATENCY_MS, TOOL_INVOCATION_COUNT, TOOL_ERROR_RATE, CONTEXT_BUILD_TIME_MS, CONTEXT_TOKEN_COUNT, LLM_CALL_COUNT, LLM_TOTAL_TOKENS, LLM_TOTAL_COST_USD, LLM_AVG_LATENCY_MS, SUBPLAN_COUNT) with MetricEntry model and MetricCollector. Add llm_traces database table (LLMTraceModel) with LLMTraceRepository for persistence. TraceService provides recording, querying, metric computation, plan lifecycle hooks, and optional LangSmith forwarding when LANGCHAIN_TRACING_V2=true. Wired into DI container as trace_service. Includes 28 Behave BDD scenarios, 6 Robot Framework smoke tests, 3 ASV benchmark suites, and reference documentation. Fix pre-existing cli_core server_mode test flake by mocking resolve_server_mode in test steps to avoid stale config file interference. Closes #500
277 lines
11 KiB
Gherkin
277 lines
11 KiB
Gherkin
Feature: LLM trace observability
|
|
As a platform operator
|
|
I want to record LLM call telemetry and compute operational metrics
|
|
So that I can analyse cost, latency, and usage patterns
|
|
|
|
Background:
|
|
Given a trace service with an in-memory repository
|
|
|
|
# --- model validation --------------------------------------------------
|
|
|
|
Scenario: Create a valid LLM trace
|
|
Given a valid LLM trace payload
|
|
When I create the LLM trace model
|
|
Then the trace should have all required fields populated
|
|
And the trace should be frozen
|
|
|
|
Scenario: Reject trace with invalid trace_id
|
|
When I create a trace with an invalid trace_id
|
|
Then a validation error should be raised
|
|
|
|
Scenario: Reject trace with negative prompt tokens
|
|
When I create a trace with negative prompt_tokens
|
|
Then a validation error should be raised
|
|
|
|
Scenario: Reject trace with negative cost
|
|
When I create a trace with negative cost_usd
|
|
Then a validation error should be raised
|
|
|
|
Scenario: Trace with optional fields omitted
|
|
Given a minimal LLM trace payload
|
|
When I create the LLM trace model
|
|
Then the trace decision_id should be None
|
|
And the trace tool_calls should be empty
|
|
And the trace context_hash should be None
|
|
And the trace streaming should be False
|
|
And the trace retry_count should be 0
|
|
And the trace error should be None
|
|
|
|
# --- query model -------------------------------------------------------
|
|
|
|
Scenario: Create a valid trace query
|
|
When I create a trace query with plan_id filter
|
|
Then the query should have default limit 100
|
|
And the query should have default offset 0
|
|
|
|
Scenario: Reject query with limit below 1
|
|
When I create a trace query with limit 0
|
|
Then a validation error should be raised
|
|
|
|
# --- metric keys -------------------------------------------------------
|
|
|
|
Scenario: All 14 operational metric keys are defined
|
|
Then the OperationalMetricKey enum should have exactly 14 members
|
|
|
|
Scenario: Metric keys have correct string values
|
|
Then PLAN_DURATION_MS should equal "plan_duration_ms"
|
|
And LLM_AVG_LATENCY_MS should equal "llm_avg_latency_ms"
|
|
And SUBPLAN_COUNT should equal "subplan_count"
|
|
|
|
# --- metric entry ------------------------------------------------------
|
|
|
|
Scenario: Create a metric entry via collector
|
|
Given a plan_id for metrics
|
|
When I record a PLAN_DURATION_MS metric with value 1500.0
|
|
Then the metric entry key should be PLAN_DURATION_MS
|
|
And the metric entry value should be 1500.0
|
|
And the metric entry should have a timestamp
|
|
|
|
Scenario: Metric entry with labels
|
|
Given a plan_id for metrics
|
|
When I record an ACTOR_LATENCY_MS metric with actor label
|
|
Then the metric entry labels should contain actor
|
|
|
|
# --- trace recording ---------------------------------------------------
|
|
|
|
Scenario: Record and retrieve a trace
|
|
Given a valid LLM trace
|
|
When I record the trace via the service
|
|
Then I should be able to retrieve it by trace_id
|
|
And I should be able to list it by plan_id
|
|
|
|
Scenario: Record multiple traces for a plan
|
|
Given three valid LLM traces for the same plan
|
|
When I record all traces via the service
|
|
Then listing by plan_id should return 3 traces
|
|
|
|
Scenario: List traces by decision
|
|
Given two traces with the same decision_id
|
|
When I record all traces via the service
|
|
Then listing by decision_id should return 2 traces
|
|
|
|
Scenario: Get non-existent trace returns None
|
|
When I query a non-existent trace_id
|
|
Then the trace query result should be None
|
|
|
|
# --- metric computation ------------------------------------------------
|
|
|
|
Scenario: Compute metrics for a plan with traces
|
|
Given three valid LLM traces for the same plan
|
|
When I record all traces via the service
|
|
And I compute metrics for the plan
|
|
Then I should get LLM_CALL_COUNT equal to 3
|
|
And I should get LLM_TOTAL_TOKENS greater than 0
|
|
And I should get LLM_TOTAL_COST_USD greater than 0
|
|
And I should get LLM_AVG_LATENCY_MS greater than 0
|
|
|
|
Scenario: Compute metrics for a plan with no traces
|
|
When I compute metrics for a plan with no traces
|
|
Then the metrics list should be empty
|
|
|
|
# --- lifecycle hooks ---------------------------------------------------
|
|
|
|
Scenario: Plan start hook returns decision count metric
|
|
Given a plan_id for metrics
|
|
When I call on_plan_start
|
|
Then I should get a PLAN_DECISION_COUNT metric with value 0
|
|
|
|
Scenario: Actor invocation hook returns metrics
|
|
Given a plan_id for metrics
|
|
When I call on_actor_invocation with actor "planner" and latency 250.0
|
|
Then I should get ACTOR_INVOCATION_COUNT and ACTOR_LATENCY_MS metrics
|
|
|
|
Scenario: Tool execution hook returns metrics
|
|
Given a plan_id for metrics
|
|
When I call on_tool_execution with tool "file_write" and no error
|
|
Then I should get a TOOL_INVOCATION_COUNT metric
|
|
And I should not get a TOOL_ERROR_RATE metric
|
|
|
|
Scenario: Tool execution hook with error returns error rate
|
|
Given a plan_id for metrics
|
|
When I call on_tool_execution with tool "file_write" and error
|
|
Then I should get both TOOL_INVOCATION_COUNT and TOOL_ERROR_RATE metrics
|
|
|
|
# --- LangSmith forwarding ----------------------------------------------
|
|
|
|
Scenario: LangSmith forwarding when env var is set
|
|
Given LANGCHAIN_TRACING_V2 is set to "true"
|
|
And a valid LLM trace
|
|
When I record the trace via the service
|
|
Then the LangSmith forwarder should have been called
|
|
|
|
Scenario: LangSmith forwarding skipped when env var is not set
|
|
Given LANGCHAIN_TRACING_V2 is not set
|
|
And a valid LLM trace
|
|
When I record the trace via the service
|
|
Then the LangSmith forwarder should not have been called
|
|
|
|
Scenario: LangSmith forwarding failure is logged not raised
|
|
Given LANGCHAIN_TRACING_V2 is set to "true"
|
|
And a valid LLM trace
|
|
And the LangSmith forwarder will raise an exception
|
|
When I record the trace via the service
|
|
Then no exception should propagate
|
|
And the trace should still be persisted
|
|
|
|
# --- convenience collectors --------------------------------------------
|
|
|
|
Scenario: MetricCollector plan_duration convenience
|
|
Given a plan_id for metrics
|
|
When I call MetricCollector.plan_duration with 2000.0
|
|
Then the metric key should be PLAN_DURATION_MS
|
|
And the metric value should be 2000.0
|
|
|
|
Scenario: MetricCollector plan_cost convenience
|
|
Given a plan_id for metrics
|
|
When I call MetricCollector.plan_cost with 0.05
|
|
Then the metric key should be PLAN_TOTAL_COST_USD
|
|
And the metric value should be 0.05
|
|
|
|
Scenario: MetricCollector llm_call_count convenience
|
|
Given a plan_id for metrics
|
|
When I call MetricCollector.llm_call_count with 10
|
|
Then the metric key should be LLM_CALL_COUNT
|
|
And the metric value should be 10.0
|
|
|
|
# --- database model ----------------------------------------------------
|
|
|
|
Scenario: LLMTraceModel table exists
|
|
Then the LLMTraceModel should have tablename "llm_traces"
|
|
|
|
# --- repository integration (SQLAlchemy in-memory) ---------------------
|
|
|
|
Scenario: Repository rejects None session_factory
|
|
When I create a repository with None session_factory
|
|
Then a ValueError should be raised for the repository
|
|
|
|
Scenario: Repository save and get roundtrip
|
|
Given a SQLAlchemy in-memory repository
|
|
And a valid LLM trace
|
|
When I save the trace via the repository
|
|
Then I should be able to get it by trace_id from the repository
|
|
|
|
Scenario: Repository save trace with tool calls
|
|
Given a SQLAlchemy in-memory repository
|
|
And a valid LLM trace with tool calls
|
|
When I save the trace via the repository
|
|
Then I should be able to get it by trace_id from the repository
|
|
And the retrieved trace should have tool calls
|
|
|
|
Scenario: Repository get non-existent trace returns None
|
|
Given a SQLAlchemy in-memory repository
|
|
When I get a non-existent trace from the repository
|
|
Then the repository result should be None
|
|
|
|
Scenario: Repository list by plan returns matching traces
|
|
Given a SQLAlchemy in-memory repository
|
|
And two traces for the same plan in the repository
|
|
When I save both traces via the repository
|
|
Then listing by plan from the repository should return 2 traces
|
|
|
|
Scenario: Repository list by plan returns empty for unknown plan
|
|
Given a SQLAlchemy in-memory repository
|
|
When I list traces for a non-existent plan from the repository
|
|
Then the repository list should be empty
|
|
|
|
Scenario: Repository list by decision returns matching traces
|
|
Given a SQLAlchemy in-memory repository
|
|
And two traces for the same decision in the repository
|
|
When I save both traces via the repository
|
|
Then listing by decision from the repository should return 2 traces
|
|
|
|
Scenario: Repository list by decision returns empty for unknown decision
|
|
Given a SQLAlchemy in-memory repository
|
|
When I list traces for a non-existent decision from the repository
|
|
Then the repository list should be empty
|
|
|
|
Scenario: Repository save raises DatabaseError on failure
|
|
Given a SQLAlchemy repository with a broken session
|
|
And a valid LLM trace
|
|
When I save the trace via the broken repository
|
|
Then a DatabaseError should be raised with message containing "save"
|
|
|
|
Scenario: Repository get raises DatabaseError on failure
|
|
Given a SQLAlchemy repository with a broken session
|
|
When I get a trace from the broken repository
|
|
Then a DatabaseError should be raised with message containing "get"
|
|
|
|
Scenario: Repository list_by_plan raises DatabaseError on failure
|
|
Given a SQLAlchemy repository with a broken session
|
|
When I list by plan from the broken repository
|
|
Then a DatabaseError should be raised with message containing "list"
|
|
|
|
Scenario: Repository list_by_decision raises DatabaseError on failure
|
|
Given a SQLAlchemy repository with a broken session
|
|
When I list by decision from the broken repository
|
|
Then a DatabaseError should be raised with message containing "list"
|
|
|
|
# --- LangSmith internal forwarder detail --------------------------------
|
|
|
|
Scenario: LangSmith forwarder sends trace data via SDK
|
|
Given a valid LLM trace
|
|
And the langsmith SDK is mocked as available
|
|
When I call the internal LangSmith forwarder
|
|
Then the langsmith Client create_run should have been called
|
|
|
|
Scenario: LangSmith forwarder includes error field when present
|
|
Given a valid LLM trace with error "connection timeout"
|
|
And the langsmith SDK is mocked as available
|
|
When I call the internal LangSmith forwarder
|
|
Then the langsmith Client create_run should have been called with error
|
|
|
|
Scenario: LangSmith forwarder skips when SDK not installed
|
|
Given a valid LLM trace
|
|
And the langsmith SDK is not importable
|
|
When I call the internal LangSmith forwarder
|
|
Then no langsmith Client should have been created
|
|
|
|
Scenario: LangSmith enabled check is case insensitive
|
|
Given LANGCHAIN_TRACING_V2 is set to "TRUE"
|
|
Then the langsmith_enabled check should return True
|
|
|
|
Scenario: Repository save trace with no tool calls stores null
|
|
Given a SQLAlchemy in-memory repository
|
|
And a valid LLM trace
|
|
When I save the trace via the repository
|
|
Then the raw tool_calls_json in the database should be null
|