forked from cleveragents/cleveragents-core
freemo
958eb0c060
Implement the structured metrics collection framework covering 14 operational metric types with proper Histogram, Counter, and Gauge semantics per the spec (Architecture > Observability > Metrics Collection, lines ~43805-43825). Domain layer: - Add MetricType enum (HISTOGRAM, COUNTER, GAUGE) to metrics.py - Add MetricDefinition model and METRIC_DEFINITIONS registry mapping all 14 OperationalMetricKey values to their metric types - Extend MetricCollector with typed factory methods (histogram, counter, gauge) and 14 convenience methods (plan_duration, plan_cost, plan_decision_count, subplan_count, actor_invocation_count, actor_latency, tool_invocation_count, tool_error_rate, context_build_time, context_token_count, llm_call_count, llm_total_tokens, llm_total_cost, llm_avg_latency) - Extend MetricEntry with optional metric_type field that auto-resolves from METRIC_DEFINITIONS via MetricCollector.record() Infrastructure layer: - Add MetricsEmitter (infrastructure/observability/metrics_emitter.py) with emit(), emit_batch(), from_settings(), and enabled/disabled support for structured log emission in local mode - Add metrics_log_processor (config/metrics_processor.py) for structlog integration Configuration: - Add metrics_enabled and metrics_export_prometheus settings - Register MetricsEmitter as DI Singleton in application container Instrumentation: - Add best-effort metric emission in PlanExecutor for plan_duration (both runtime and stub execute paths) and plan_decision_count (strategize path) via _try_emit_metric helper that tolerates invalid plan IDs in test fixtures Testing: - 34 Behave BDD scenarios (features/observability/metrics_collection.feature) - 8 Robot Framework integration tests (robot/metrics_collection.robot) - ASV benchmark suite (benchmarks/bench_metrics_collection.py) Closes #579
60 lines
2.8 KiB
Plaintext
60 lines
2.8 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration tests for metrics collection framework
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_metrics_collection.py
|
|
|
|
*** Test Cases ***
|
|
MetricType Enum Has Three Members
|
|
[Documentation] MetricType should have HISTOGRAM, COUNTER, GAUGE
|
|
${result}= Run Process ${PYTHON} ${HELPER} metric-type-enum cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-type-enum-ok
|
|
|
|
All 14 Metric Definitions Exist
|
|
[Documentation] METRIC_DEFINITIONS should have 14 entries
|
|
${result}= Run Process ${PYTHON} ${HELPER} metric-definitions cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-definitions-ok
|
|
|
|
Histogram Counter Gauge Factory Methods
|
|
[Documentation] Typed factory methods should set correct metric_type
|
|
${result}= Run Process ${PYTHON} ${HELPER} typed-factories cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-typed-factories-ok
|
|
|
|
All 14 Convenience Methods
|
|
[Documentation] All 14 convenience methods should produce correct keys
|
|
${result}= Run Process ${PYTHON} ${HELPER} convenience-methods cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-convenience-ok
|
|
|
|
MetricsEmitter Local Mode
|
|
[Documentation] MetricsEmitter emits structured log entries
|
|
${result}= Run Process ${PYTHON} ${HELPER} emitter-local cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-emitter-local-ok
|
|
|
|
MetricsEmitter Disabled
|
|
[Documentation] MetricsEmitter respects disabled setting
|
|
${result}= Run Process ${PYTHON} ${HELPER} emitter-disabled cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-emitter-disabled-ok
|
|
|
|
Metrics Settings Configuration
|
|
[Documentation] Settings should expose metrics_enabled and metrics_export_prometheus
|
|
${result}= Run Process ${PYTHON} ${HELPER} settings-config cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-settings-ok
|
|
|
|
Metrics Log Processor
|
|
[Documentation] Structlog processor should tag metric events
|
|
${result}= Run Process ${PYTHON} ${HELPER} log-processor cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-processor-ok
|