freemo
958eb0c060
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 15s
CI / build (pull_request) Successful in 17s
CI / quality (pull_request) Successful in 19s
CI / typecheck (pull_request) Successful in 37s
CI / security (pull_request) Successful in 51s
CI / unit_tests (pull_request) Successful in 2m41s
CI / integration_tests (pull_request) Successful in 3m19s
CI / docker (pull_request) Successful in 51s
CI / coverage (pull_request) Successful in 5m7s
CI / benchmark-regression (pull_request) Successful in 33m26s
CI / lint (push) Successful in 13s
CI / build (push) Successful in 15s
CI / quality (push) Successful in 16s
CI / security (push) Successful in 33s
CI / typecheck (push) Successful in 35s
CI / benchmark-regression (push) Has been skipped
CI / unit_tests (push) Successful in 3m2s
CI / integration_tests (push) Successful in 3m12s
CI / docker (push) Successful in 49s
CI / coverage (push) Successful in 6m10s
CI / benchmark-publish (push) Has been cancelled
Implement the structured metrics collection framework covering 14 operational metric types with proper Histogram, Counter, and Gauge semantics per the spec (Architecture > Observability > Metrics Collection, lines ~43805-43825). Domain layer: - Add MetricType enum (HISTOGRAM, COUNTER, GAUGE) to metrics.py - Add MetricDefinition model and METRIC_DEFINITIONS registry mapping all 14 OperationalMetricKey values to their metric types - Extend MetricCollector with typed factory methods (histogram, counter, gauge) and 14 convenience methods (plan_duration, plan_cost, plan_decision_count, subplan_count, actor_invocation_count, actor_latency, tool_invocation_count, tool_error_rate, context_build_time, context_token_count, llm_call_count, llm_total_tokens, llm_total_cost, llm_avg_latency) - Extend MetricEntry with optional metric_type field that auto-resolves from METRIC_DEFINITIONS via MetricCollector.record() Infrastructure layer: - Add MetricsEmitter (infrastructure/observability/metrics_emitter.py) with emit(), emit_batch(), from_settings(), and enabled/disabled support for structured log emission in local mode - Add metrics_log_processor (config/metrics_processor.py) for structlog integration Configuration: - Add metrics_enabled and metrics_export_prometheus settings - Register MetricsEmitter as DI Singleton in application container Instrumentation: - Add best-effort metric emission in PlanExecutor for plan_duration (both runtime and stub execute paths) and plan_decision_count (strategize path) via _try_emit_metric helper that tolerates invalid plan IDs in test fixtures Testing: - 34 Behave BDD scenarios (features/observability/metrics_collection.feature) - 8 Robot Framework integration tests (robot/metrics_collection.robot) - ASV benchmark suite (benchmarks/bench_metrics_collection.py) Closes #579
197 lines
7.9 KiB
Gherkin
197 lines
7.9 KiB
Gherkin
Feature: Metrics Collection Framework
|
|
As a platform operator
|
|
I want structured metric collection with typed semantics
|
|
So that I can monitor plan execution via histograms, counters, and gauges
|
|
|
|
# --- MetricType enum ---
|
|
|
|
Scenario: MetricType enum has three members
|
|
Then MetricType should have exactly 3 members
|
|
|
|
Scenario: MetricType values are correct
|
|
Then MetricType HISTOGRAM should equal "histogram"
|
|
And MetricType COUNTER should equal "counter"
|
|
And MetricType GAUGE should equal "gauge"
|
|
|
|
# --- MetricDefinition registry ---
|
|
|
|
Scenario: All 14 metric keys have definitions
|
|
Then METRIC_DEFINITIONS should have exactly 14 entries
|
|
|
|
Scenario: PLAN_DURATION_MS is a histogram
|
|
Then the definition for PLAN_DURATION_MS should have metric_type HISTOGRAM
|
|
|
|
Scenario: LLM_CALL_COUNT is a counter
|
|
Then the definition for LLM_CALL_COUNT should have metric_type COUNTER
|
|
|
|
Scenario: PLAN_DECISION_COUNT is a gauge
|
|
Then the definition for PLAN_DECISION_COUNT should have metric_type GAUGE
|
|
|
|
Scenario: TOOL_ERROR_RATE is a gauge
|
|
Then the definition for TOOL_ERROR_RATE should have metric_type GAUGE
|
|
|
|
# --- MetricCollector typed factory methods ---
|
|
|
|
Scenario: histogram method creates a metric with histogram type
|
|
Given a plan_id for metrics collection
|
|
When I create a histogram metric for PLAN_DURATION_MS with value 1200.0
|
|
Then the result metric_type should be HISTOGRAM
|
|
And the result value should be 1200.0
|
|
|
|
Scenario: counter method creates a metric with counter type
|
|
Given a plan_id for metrics collection
|
|
When I create a counter metric for LLM_CALL_COUNT with value 5.0
|
|
Then the result metric_type should be COUNTER
|
|
And the result value should be 5.0
|
|
|
|
Scenario: gauge method creates a metric with gauge type
|
|
Given a plan_id for metrics collection
|
|
When I create a gauge metric for PLAN_DECISION_COUNT with value 3.0
|
|
Then the result metric_type should be GAUGE
|
|
And the result value should be 3.0
|
|
|
|
# --- All 14 convenience methods ---
|
|
|
|
Scenario: plan_duration convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the plan_duration convenience with value 2500.0
|
|
Then the collected metric key should equal PLAN_DURATION_MS
|
|
And the collected metric value should equal 2500.0
|
|
And the metric_type should be HISTOGRAM
|
|
|
|
Scenario: plan_cost convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the plan_cost convenience with value 0.12
|
|
Then the collected metric key should equal PLAN_TOTAL_COST_USD
|
|
And the collected metric value should equal 0.12
|
|
|
|
Scenario: plan_decision_count convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the plan_decision_count convenience with value 7.0
|
|
Then the collected metric key should equal PLAN_DECISION_COUNT
|
|
And the collected metric value should equal 7.0
|
|
|
|
Scenario: subplan_count convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the subplan_count convenience with value 2.0
|
|
Then the collected metric key should equal SUBPLAN_COUNT
|
|
And the collected metric value should equal 2.0
|
|
|
|
Scenario: actor_invocation_count convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the actor_invocation_count convenience with value 1.0
|
|
Then the collected metric key should equal ACTOR_INVOCATION_COUNT
|
|
And the collected metric value should equal 1.0
|
|
|
|
Scenario: actor_latency convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the actor_latency convenience with value 350.0
|
|
Then the collected metric key should equal ACTOR_LATENCY_MS
|
|
And the collected metric value should equal 350.0
|
|
|
|
Scenario: tool_invocation_count convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the tool_invocation_count convenience with value 10.0
|
|
Then the collected metric key should equal TOOL_INVOCATION_COUNT
|
|
And the collected metric value should equal 10.0
|
|
|
|
Scenario: tool_error_rate convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the tool_error_rate convenience with value 0.05
|
|
Then the collected metric key should equal TOOL_ERROR_RATE
|
|
And the collected metric value should equal 0.05
|
|
|
|
Scenario: context_build_time convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the context_build_time convenience with value 45.0
|
|
Then the collected metric key should equal CONTEXT_BUILD_TIME_MS
|
|
And the collected metric value should equal 45.0
|
|
|
|
Scenario: context_token_count convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the context_token_count convenience with value 8192.0
|
|
Then the collected metric key should equal CONTEXT_TOKEN_COUNT
|
|
And the collected metric value should equal 8192.0
|
|
|
|
Scenario: llm_call_count convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the llm_call_count convenience with value 15.0
|
|
Then the collected metric key should equal LLM_CALL_COUNT
|
|
And the collected metric value should equal 15.0
|
|
|
|
Scenario: llm_total_tokens convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the llm_total_tokens convenience with value 50000.0
|
|
Then the collected metric key should equal LLM_TOTAL_TOKENS
|
|
And the collected metric value should equal 50000.0
|
|
|
|
Scenario: llm_total_cost convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the llm_total_cost convenience with value 1.25
|
|
Then the collected metric key should equal LLM_TOTAL_COST_USD
|
|
And the collected metric value should equal 1.25
|
|
|
|
Scenario: llm_avg_latency convenience method
|
|
Given a plan_id for metrics collection
|
|
When I use the llm_avg_latency convenience with value 200.0
|
|
Then the collected metric key should equal LLM_AVG_LATENCY_MS
|
|
And the collected metric value should equal 200.0
|
|
|
|
# --- MetricsEmitter ---
|
|
|
|
Scenario: MetricsEmitter emits metric as structured log
|
|
Given a plan_id for metrics collection
|
|
And a MetricsEmitter in local mode
|
|
When I emit a plan duration metric with value 1500.0
|
|
Then the emitter should have emitted successfully
|
|
|
|
Scenario: MetricsEmitter respects disabled setting
|
|
Given a plan_id for metrics collection
|
|
And a MetricsEmitter that is disabled
|
|
When I emit a plan duration metric with value 1500.0
|
|
Then the emitter should not have emitted
|
|
|
|
Scenario: MetricsEmitter emits batch
|
|
Given a plan_id for metrics collection
|
|
And a MetricsEmitter in local mode
|
|
When I emit a batch of 3 metrics
|
|
Then the emitter should report 3 emitted
|
|
|
|
Scenario: MetricsEmitter disabled batch returns zero
|
|
Given a plan_id for metrics collection
|
|
And a MetricsEmitter that is disabled
|
|
When I emit a batch of 3 metrics
|
|
Then the emitter should report 0 emitted
|
|
|
|
Scenario: MetricsEmitter from_settings uses defaults
|
|
When I create a MetricsEmitter from default settings
|
|
Then the emitter should be enabled
|
|
And the emitter prometheus should be disabled
|
|
|
|
# --- Configuration ---
|
|
|
|
Scenario: metrics_enabled defaults to True
|
|
When I load default settings for metrics
|
|
Then metrics_enabled should be True
|
|
|
|
Scenario: metrics_export_prometheus defaults to False
|
|
When I load default settings for metrics
|
|
Then metrics_export_prometheus should be False
|
|
|
|
# --- Metrics processor ---
|
|
|
|
Scenario: metrics_log_processor tags metric events
|
|
When I pass a metric event through the metrics processor
|
|
Then the event should have event_category metric
|
|
|
|
Scenario: metrics_log_processor ignores non-metric events
|
|
When I pass a non-metric event through the metrics processor
|
|
Then the event should not have event_category
|
|
|
|
# --- MetricEntry includes metric_type ---
|
|
|
|
Scenario: MetricEntry from record includes resolved metric_type
|
|
Given a plan_id for metrics collection
|
|
When I record a metric via collector record for ACTOR_LATENCY_MS
|
|
Then the entry metric_type should be HISTOGRAM
|