HAL9000
b4b6fe6484
Refactor: replace session.commit() with session.flush() in LLMTraceRepository.save() to ensure changes are persisted within the UnitOfWork without prematurely committing the database transaction. - Updated LLMTraceRepository.save() to call session.flush() instead of session.commit() in src/cleveragents/infrastructure/database/llm_trace_repository.py. - Added two new BDD scenarios to features/llm_trace.feature: - 'Repository save() calls flush not commit' to verify save() uses flush not commit. - 'LLM trace rolled back when UnitOfWork transaction rolls back' to verify rollback. - Added corresponding step definitions to features/steps/llm_trace_steps.py. ISSUES CLOSED: #10034
292 lines
12 KiB
Gherkin
292 lines
12 KiB
Gherkin
Feature: LLM trace observability
|
|
As a platform operator
|
|
I want to record LLM call telemetry and compute operational metrics
|
|
So that I can analyse cost, latency, and usage patterns
|
|
|
|
Background:
|
|
Given a trace service with an in-memory repository
|
|
|
|
# --- model validation --------------------------------------------------
|
|
|
|
Scenario: Create a valid LLM trace
|
|
Given a valid LLM trace payload
|
|
When I create the LLM trace model
|
|
Then the trace should have all required fields populated
|
|
And the trace should be frozen
|
|
|
|
Scenario: Reject trace with invalid trace_id
|
|
When I create a trace with an invalid trace_id
|
|
Then a validation error should be raised
|
|
|
|
Scenario: Reject trace with negative prompt tokens
|
|
When I create a trace with negative prompt_tokens
|
|
Then a validation error should be raised
|
|
|
|
Scenario: Reject trace with negative cost
|
|
When I create a trace with negative cost_usd
|
|
Then a validation error should be raised
|
|
|
|
Scenario: Trace with optional fields omitted
|
|
Given a minimal LLM trace payload
|
|
When I create the LLM trace model
|
|
Then the trace decision_id should be None
|
|
And the trace tool_calls should be empty
|
|
And the trace context_hash should be None
|
|
And the trace streaming should be False
|
|
And the trace retry_count should be 0
|
|
And the trace error should be None
|
|
|
|
# --- query model -------------------------------------------------------
|
|
|
|
Scenario: Create a valid trace query
|
|
When I create a trace query with plan_id filter
|
|
Then the query should have default limit 100
|
|
And the query should have default offset 0
|
|
|
|
Scenario: Reject query with limit below 1
|
|
When I create a trace query with limit 0
|
|
Then a validation error should be raised
|
|
|
|
# --- metric keys -------------------------------------------------------
|
|
|
|
Scenario: All 14 operational metric keys are defined
|
|
Then the OperationalMetricKey enum should have exactly 14 members
|
|
|
|
Scenario: Metric keys have correct string values
|
|
Then PLAN_DURATION_MS should equal "plan_duration_ms"
|
|
And LLM_AVG_LATENCY_MS should equal "llm_avg_latency_ms"
|
|
And SUBPLAN_COUNT should equal "subplan_count"
|
|
|
|
# --- metric entry ------------------------------------------------------
|
|
|
|
Scenario: Create a metric entry via collector
|
|
Given a plan_id for metrics
|
|
When I record a PLAN_DURATION_MS metric with value 1500.0
|
|
Then the metric entry key should be PLAN_DURATION_MS
|
|
And the metric entry value should be 1500.0
|
|
And the metric entry should have a timestamp
|
|
|
|
Scenario: Metric entry with labels
|
|
Given a plan_id for metrics
|
|
When I record an ACTOR_LATENCY_MS metric with actor label
|
|
Then the metric entry labels should contain actor
|
|
|
|
# --- trace recording ---------------------------------------------------
|
|
|
|
Scenario: Record and retrieve a trace
|
|
Given a valid LLM trace
|
|
When I record the trace via the service
|
|
Then I should be able to retrieve it by trace_id
|
|
And I should be able to list it by plan_id
|
|
|
|
Scenario: Record multiple traces for a plan
|
|
Given three valid LLM traces for the same plan
|
|
When I record all traces via the service
|
|
Then listing by plan_id should return 3 traces
|
|
|
|
Scenario: List traces by decision
|
|
Given two traces with the same decision_id
|
|
When I record all traces via the service
|
|
Then listing by decision_id should return 2 traces
|
|
|
|
Scenario: Get non-existent trace returns None
|
|
When I query a non-existent trace_id
|
|
Then the trace query result should be None
|
|
|
|
# --- metric computation ------------------------------------------------
|
|
|
|
Scenario: Compute metrics for a plan with traces
|
|
Given three valid LLM traces for the same plan
|
|
When I record all traces via the service
|
|
And I compute metrics for the plan
|
|
Then I should get LLM_CALL_COUNT equal to 3
|
|
And I should get LLM_TOTAL_TOKENS greater than 0
|
|
And I should get LLM_TOTAL_COST_USD greater than 0
|
|
And I should get LLM_AVG_LATENCY_MS greater than 0
|
|
|
|
Scenario: Compute metrics for a plan with no traces
|
|
When I compute metrics for a plan with no traces
|
|
Then the metrics list should be empty
|
|
|
|
# --- lifecycle hooks ---------------------------------------------------
|
|
|
|
Scenario: Plan start hook returns decision count metric
|
|
Given a plan_id for metrics
|
|
When I call on_plan_start
|
|
Then I should get a PLAN_DECISION_COUNT metric with value 0
|
|
|
|
Scenario: Actor invocation hook returns metrics
|
|
Given a plan_id for metrics
|
|
When I call on_actor_invocation with actor "planner" and latency 250.0
|
|
Then I should get ACTOR_INVOCATION_COUNT and ACTOR_LATENCY_MS metrics
|
|
|
|
Scenario: Tool execution hook returns metrics
|
|
Given a plan_id for metrics
|
|
When I call on_tool_execution with tool "file_write" and no error
|
|
Then I should get a TOOL_INVOCATION_COUNT metric
|
|
And I should not get a TOOL_ERROR_RATE metric
|
|
|
|
Scenario: Tool execution hook with error returns error rate
|
|
Given a plan_id for metrics
|
|
When I call on_tool_execution with tool "file_write" and error
|
|
Then I should get both TOOL_INVOCATION_COUNT and TOOL_ERROR_RATE metrics
|
|
|
|
# --- LangSmith forwarding ----------------------------------------------
|
|
|
|
Scenario: LangSmith forwarding when env var is set
|
|
Given LANGCHAIN_TRACING_V2 is set to "true"
|
|
And a valid LLM trace
|
|
When I record the trace via the service
|
|
Then the LangSmith forwarder should have been called
|
|
|
|
Scenario: LangSmith forwarding skipped when env var is not set
|
|
Given LANGCHAIN_TRACING_V2 is not set
|
|
And a valid LLM trace
|
|
When I record the trace via the service
|
|
Then the LangSmith forwarder should not have been called
|
|
|
|
Scenario: LangSmith forwarding failure is logged not raised
|
|
Given LANGCHAIN_TRACING_V2 is set to "true"
|
|
And a valid LLM trace
|
|
And the LangSmith forwarder will raise an exception
|
|
When I record the trace via the service
|
|
Then no exception should propagate
|
|
And the trace should still be persisted
|
|
|
|
# --- convenience collectors --------------------------------------------
|
|
|
|
Scenario: MetricCollector plan_duration convenience
|
|
Given a plan_id for metrics
|
|
When I call MetricCollector.plan_duration with 2000.0
|
|
Then the metric key should be PLAN_DURATION_MS
|
|
And the metric value should be 2000.0
|
|
|
|
Scenario: MetricCollector plan_cost convenience
|
|
Given a plan_id for metrics
|
|
When I call MetricCollector.plan_cost with 0.05
|
|
Then the metric key should be PLAN_TOTAL_COST_USD
|
|
And the metric value should be 0.05
|
|
|
|
Scenario: MetricCollector llm_call_count convenience
|
|
Given a plan_id for metrics
|
|
When I call MetricCollector.llm_call_count with 10
|
|
Then the metric key should be LLM_CALL_COUNT
|
|
And the metric value should be 10.0
|
|
|
|
# --- database model ----------------------------------------------------
|
|
|
|
Scenario: LLMTraceModel table exists
|
|
Then the LLMTraceModel should have tablename "llm_traces"
|
|
|
|
# --- repository integration (SQLAlchemy in-memory) ---------------------
|
|
|
|
Scenario: Repository rejects None session_factory
|
|
When I create a repository with None session_factory
|
|
Then a ValueError should be raised for the repository
|
|
|
|
Scenario: Repository save and get roundtrip
|
|
Given a SQLAlchemy in-memory repository
|
|
And a valid LLM trace
|
|
When I save the trace via the repository
|
|
Then I should be able to get it by trace_id from the repository
|
|
|
|
Scenario: Repository save trace with tool calls
|
|
Given a SQLAlchemy in-memory repository
|
|
And a valid LLM trace with tool calls
|
|
When I save the trace via the repository
|
|
Then I should be able to get it by trace_id from the repository
|
|
And the retrieved trace should have tool calls
|
|
|
|
Scenario: Repository get non-existent trace returns None
|
|
Given a SQLAlchemy in-memory repository
|
|
When I get a non-existent trace from the repository
|
|
Then the repository result should be None
|
|
|
|
Scenario: Repository list by plan returns matching traces
|
|
Given a SQLAlchemy in-memory repository
|
|
And two traces for the same plan in the repository
|
|
When I save both traces via the repository
|
|
Then listing by plan from the repository should return 2 traces
|
|
|
|
Scenario: Repository list by plan returns empty for unknown plan
|
|
Given a SQLAlchemy in-memory repository
|
|
When I list traces for a non-existent plan from the repository
|
|
Then the repository list should be empty
|
|
|
|
Scenario: Repository list by decision returns matching traces
|
|
Given a SQLAlchemy in-memory repository
|
|
And two traces for the same decision in the repository
|
|
When I save both traces via the repository
|
|
Then listing by decision from the repository should return 2 traces
|
|
|
|
Scenario: Repository list by decision returns empty for unknown decision
|
|
Given a SQLAlchemy in-memory repository
|
|
When I list traces for a non-existent decision from the repository
|
|
Then the repository list should be empty
|
|
|
|
Scenario: Repository save raises DatabaseError on failure
|
|
Given a SQLAlchemy repository with a broken session
|
|
And a valid LLM trace
|
|
When I save the trace via the broken repository
|
|
Then a DatabaseError should be raised with message containing "save"
|
|
|
|
Scenario: Repository get raises DatabaseError on failure
|
|
Given a SQLAlchemy repository with a broken session
|
|
When I get a trace from the broken repository
|
|
Then a DatabaseError should be raised with message containing "get"
|
|
|
|
Scenario: Repository list_by_plan raises DatabaseError on failure
|
|
Given a SQLAlchemy repository with a broken session
|
|
When I list by plan from the broken repository
|
|
Then a DatabaseError should be raised with message containing "list"
|
|
|
|
Scenario: Repository list_by_decision raises DatabaseError on failure
|
|
Given a SQLAlchemy repository with a broken session
|
|
When I list by decision from the broken repository
|
|
Then a DatabaseError should be raised with message containing "list"
|
|
|
|
# --- LangSmith internal forwarder detail --------------------------------
|
|
|
|
Scenario: LangSmith forwarder sends trace data via SDK
|
|
Given a valid LLM trace
|
|
And the langsmith SDK is mocked as available
|
|
When I call the internal LangSmith forwarder
|
|
Then the langsmith Client create_run should have been called
|
|
|
|
Scenario: LangSmith forwarder includes error field when present
|
|
Given a valid LLM trace with error "connection timeout"
|
|
And the langsmith SDK is mocked as available
|
|
When I call the internal LangSmith forwarder
|
|
Then the langsmith Client create_run should have been called with error
|
|
|
|
Scenario: LangSmith forwarder skips when SDK not installed
|
|
Given a valid LLM trace
|
|
And the langsmith SDK is not importable
|
|
When I call the internal LangSmith forwarder
|
|
Then no langsmith Client should have been created
|
|
|
|
Scenario: LangSmith enabled check is case insensitive
|
|
Given LANGCHAIN_TRACING_V2 is set to "TRUE"
|
|
Then the langsmith_enabled check should return True
|
|
|
|
Scenario: Repository save trace with no tool calls stores null
|
|
Given a SQLAlchemy in-memory repository
|
|
And a valid LLM trace
|
|
When I save the trace via the repository
|
|
Then the raw tool_calls_json in the database should be null
|
|
|
|
# --- session contract (flush not commit) --------------------------------
|
|
|
|
Scenario: Repository save() calls flush not commit
|
|
Given a SQLAlchemy in-memory repository
|
|
And a valid LLM trace
|
|
When I save the trace via the repository and spy on the session
|
|
Then the session flush should have been called
|
|
And the session commit should not have been called
|
|
|
|
Scenario: LLM trace rolled back when UnitOfWork transaction rolls back
|
|
Given a UnitOfWork with an in-memory database
|
|
And a valid LLM trace
|
|
When I save the trace inside a UnitOfWork transaction that rolls back
|
|
Then the LLM trace should not be persisted in the database
|