freemo
958eb0c060
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 15s
CI / build (pull_request) Successful in 17s
CI / quality (pull_request) Successful in 19s
CI / typecheck (pull_request) Successful in 37s
CI / security (pull_request) Successful in 51s
CI / unit_tests (pull_request) Successful in 2m41s
CI / integration_tests (pull_request) Successful in 3m19s
CI / docker (pull_request) Successful in 51s
CI / coverage (pull_request) Successful in 5m7s
CI / benchmark-regression (pull_request) Successful in 33m26s
CI / lint (push) Successful in 13s
CI / build (push) Successful in 15s
CI / quality (push) Successful in 16s
CI / security (push) Successful in 33s
CI / typecheck (push) Successful in 35s
CI / benchmark-regression (push) Has been skipped
CI / unit_tests (push) Successful in 3m2s
CI / integration_tests (push) Successful in 3m12s
CI / docker (push) Successful in 49s
CI / coverage (push) Successful in 6m10s
CI / benchmark-publish (push) Has been cancelled
Implement the structured metrics collection framework covering 14 operational metric types with proper Histogram, Counter, and Gauge semantics per the spec (Architecture > Observability > Metrics Collection, lines ~43805-43825). Domain layer: - Add MetricType enum (HISTOGRAM, COUNTER, GAUGE) to metrics.py - Add MetricDefinition model and METRIC_DEFINITIONS registry mapping all 14 OperationalMetricKey values to their metric types - Extend MetricCollector with typed factory methods (histogram, counter, gauge) and 14 convenience methods (plan_duration, plan_cost, plan_decision_count, subplan_count, actor_invocation_count, actor_latency, tool_invocation_count, tool_error_rate, context_build_time, context_token_count, llm_call_count, llm_total_tokens, llm_total_cost, llm_avg_latency) - Extend MetricEntry with optional metric_type field that auto-resolves from METRIC_DEFINITIONS via MetricCollector.record() Infrastructure layer: - Add MetricsEmitter (infrastructure/observability/metrics_emitter.py) with emit(), emit_batch(), from_settings(), and enabled/disabled support for structured log emission in local mode - Add metrics_log_processor (config/metrics_processor.py) for structlog integration Configuration: - Add metrics_enabled and metrics_export_prometheus settings - Register MetricsEmitter as DI Singleton in application container Instrumentation: - Add best-effort metric emission in PlanExecutor for plan_duration (both runtime and stub execute paths) and plan_decision_count (strategize path) via _try_emit_metric helper that tolerates invalid plan IDs in test fixtures Testing: - 34 Behave BDD scenarios (features/observability/metrics_collection.feature) - 8 Robot Framework integration tests (robot/metrics_collection.robot) - ASV benchmark suite (benchmarks/bench_metrics_collection.py) Closes #579
60 lines
2.8 KiB
Plaintext
60 lines
2.8 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration tests for metrics collection framework
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_metrics_collection.py
|
|
|
|
*** Test Cases ***
|
|
MetricType Enum Has Three Members
|
|
[Documentation] MetricType should have HISTOGRAM, COUNTER, GAUGE
|
|
${result}= Run Process ${PYTHON} ${HELPER} metric-type-enum cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-type-enum-ok
|
|
|
|
All 14 Metric Definitions Exist
|
|
[Documentation] METRIC_DEFINITIONS should have 14 entries
|
|
${result}= Run Process ${PYTHON} ${HELPER} metric-definitions cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-definitions-ok
|
|
|
|
Histogram Counter Gauge Factory Methods
|
|
[Documentation] Typed factory methods should set correct metric_type
|
|
${result}= Run Process ${PYTHON} ${HELPER} typed-factories cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-typed-factories-ok
|
|
|
|
All 14 Convenience Methods
|
|
[Documentation] All 14 convenience methods should produce correct keys
|
|
${result}= Run Process ${PYTHON} ${HELPER} convenience-methods cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-convenience-ok
|
|
|
|
MetricsEmitter Local Mode
|
|
[Documentation] MetricsEmitter emits structured log entries
|
|
${result}= Run Process ${PYTHON} ${HELPER} emitter-local cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-emitter-local-ok
|
|
|
|
MetricsEmitter Disabled
|
|
[Documentation] MetricsEmitter respects disabled setting
|
|
${result}= Run Process ${PYTHON} ${HELPER} emitter-disabled cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-emitter-disabled-ok
|
|
|
|
Metrics Settings Configuration
|
|
[Documentation] Settings should expose metrics_enabled and metrics_export_prometheus
|
|
${result}= Run Process ${PYTHON} ${HELPER} settings-config cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-settings-ok
|
|
|
|
Metrics Log Processor
|
|
[Documentation] Structlog processor should tag metric events
|
|
${result}= Run Process ${PYTHON} ${HELPER} log-processor cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} metrics-processor-ok
|