cleveragents-core/docs/adr/ADR-025-observability-and-logging.md

adr_number, title, status_history, tier, authors, superseded_by, related_adrs, acceptance

adr_number	title	status_history	tier	authors	superseded_by	related_adrs	acceptance
25	Observability and Logging	2026-02-16 Proposed Jeffrey Phillips Freeman 2026-02-16 Accepted Jeffrey Phillips Freeman	4	Jeffrey Phillips Freeman	null	number title relationship 1 Layered Architecture Logging is a cross-cutting concern handled through the event bus in the Application Layer number title relationship 5 Technical Stack structlog is the chosen structured logging library number title relationship 24 Configuration System Log verbosity levels are controlled through the configuration system and CLI flags	votes_for votes_against abstentions voter comment Jeffrey Phillips Freeman <Jeffrey.Freeman@CleverThis.com> Structured JSON logging with context binding gives us the observability needed for debugging LLM-driven workflows

Context

CleverAgents orchestrates complex, multi-step AI workflows involving LLM calls, tool invocations, plan hierarchies, and concurrent execution. When something goes wrong — a plan fails, a tool produces unexpected output, an LLM makes a bad decision — the operator needs to trace the chain of events, understand the context, and identify the root cause. Traditional unstructured logging is insufficient for correlating events across plan hierarchies, actor invocations, and tool calls.

Decision Drivers

• Multi-step AI workflows (LLM calls, tool invocations, plan hierarchies, concurrent execution) require correlated tracing across nested scopes to diagnose failures
• Log events must carry contextual identifiers (plan_id, actor_name, decision_id, tool_name) for structured querying and filtering
• Log output must not contaminate structured command output on stdout; logs and results must flow through separate streams
• LLM-specific observability (token usage, latency, cost) is needed but must remain optional and not create a hard dependency
• Default verbosity must be minimal (FATAL-only) with intuitive CLI-flag escalation for debugging

Decision

CleverAgents uses structlog (>= 24.4.0) for structured JSON logging with context binding. Every log event carries contextual identifiers (plan_id, decision_id, actor_name, tool_name) that enable correlation across the plan lifecycle. Optional LangSmith integration provides LLM-specific observability (tracing, token usage, latency, cost).

Design

Structured Logging with structlog

structlog provides:

• JSON-structured output: Every log event is a JSON object with a consistent schema, enabling log aggregation, filtering, and analysis with standard tools.
• Context binding: Contextual key-value pairs are bound to the logger and automatically included in every subsequent log event. Plan execution binds plan_id, decision_id, actor_name, and tool_name at the appropriate scope.
• Processor pipeline: Log events flow through a configurable processor chain (timestamp injection, level filtering, context enrichment, output formatting).

Context Binding Pattern

plan_lifecycle.start → bind(plan_id=...)
  strategize → bind(actor_name="local/strategy-actor")
    decision_made → bind(decision_id=...)
  execute → bind(actor_name="local/execution-actor")
    tool_invoked → bind(tool_name="local/edit-file")
    tool_completed → unbind(tool_name)
  apply → bind(phase="apply")
    apply_completed

Each nested scope adds its identifiers to the bound context. Log events at any depth carry the full context chain, enabling queries like "show all events for plan X, decision Y, tool Z."

Log Levels

Level	Default Behavior	-v Flag
FATAL	Always shown	(default)
ERROR	Suppressed by default	-v
WARN	Suppressed by default	-vv
INFO	Suppressed by default	-vvv
DEBUG	Suppressed by default	-vvvv
TRACE	Suppressed by default	-vvvvv

The default log level is FATAL — only fatal errors produce log output. The -v flag (repeatable up to 5 times) raises the effective verbosity for a single invocation.

Log Output Destinations

• File output: Log files written to core.log.dir (default: <data-dir>/logs). Controlled by core.log.file-enabled (default: true). Retention: core.log.retention-days (default: 30).
• Terminal output: Controlled by core.log.terminal (auto, always, never). In auto mode, terminal output is shown only when -v is used. Terminal log output goes to core.log.terminal-stream (default: stderr) to keep logs separate from command output on stdout.

Domain Events

The event-driven architecture (ADR-001) emits structured domain events through the event bus:

• PlanCreated, PhaseChanged, DecisionMade, ToolInvoked, ApplyCompleted, CorrectionRequested.
• Events flow through RxPY reactive streams for real-time processing.
• Events are captured by the structured logging pipeline for persistence.
• Events enable decoupled observability without tight coupling between components.

LangSmith Integration (Optional)

LangSmith provides LLM-specific observability:

• Tracing: Full trace of LLM calls including prompts, completions, tool calls, and intermediate steps.
• Token usage: Per-call and aggregate token consumption tracking.
• Latency: Per-call and end-to-end latency measurement.
• Cost: Estimated cost per call and per plan based on provider pricing.

Configuration: CLEVERAGENTS_LANGSMITH_* environment variables, which sync to LANGCHAIN_TRACING_V2. LangSmith is entirely optional — the system functions fully without it.

Metrics Collection

The MetricsCollector in the Application Layer aggregates:

• Plan execution counts and durations.
• Tool invocation counts, success/failure rates, and latencies.
• LLM call counts, token usage, and estimated costs.
• Context assembly metrics (strategy execution times, budget utilization, cache hit rates).

Metrics are exposed through structured log events and are available for performance benchmarking via ASV.

Constraints

• All log events must be structured JSON. Unstructured string messages are prohibited.
• Context binding must include at minimum plan_id and actor_name for events within plan execution. decision_id and tool_name must be included when available.
• Terminal log output must go to stderr (or the configured stream), never to stdout, to avoid contaminating structured command output.
• LangSmith integration must be optional — the system must function identically (except for LLM-specific tracing) without LangSmith configured.
• Log file retention must be enforced automatically. Logs older than core.log.retention-days must be purged.
• Provider API keys and other secrets must never appear in log output.

Consequences

Positive

• Structured JSON logging enables powerful querying and correlation across complex plan hierarchies.
• Context binding automatically enriches events with identifiers, reducing the need for manual log annotation.
• The -v flag convention provides intuitive verbosity control without configuration file changes.
• Separate log and command output streams enable clean scripting with --format json while retaining debug log access.
• LangSmith integration provides deep LLM-specific insights without custom instrumentation.

Negative

• Structured JSON logs are less human-readable than traditional text logs when viewed directly.
• Context binding adds overhead to every log call (maintaining the bound context dictionary).
• The six log levels with a default of FATAL may hide important information from users who do not know about the -v flag.

Risks

• High-volume log events during concurrent plan execution could impact performance or fill disk storage.
• LangSmith dependency on an external service introduces a potential single point of failure for LLM tracing.
• Secret leakage in log output is a persistent risk that requires ongoing vigilance.

Alternatives Considered

Python stdlib logging — Supports structured logging via formatters but lacks structlog's first-class context binding, processor pipeline, and JSON output capabilities. Would require significant custom code to achieve equivalent functionality.

OpenTelemetry for tracing — A more general-purpose observability framework, but adds significant complexity. LangSmith provides LLM-specific tracing that is more targeted to CleverAgents' needs. OpenTelemetry could be added later as a complementary integration.

Compliance

• Structured output tests: Tests verify that all log events are valid JSON with the expected schema fields.
• Context binding tests: Tests verify that plan_id, actor_name, decision_id, and tool_name are correctly bound and unbound at the appropriate scopes.
• Secret masking tests: Tests verify that provider API keys and authentication tokens never appear in log output.
• Stderr separation tests: Tests verify that log output goes to stderr and does not appear in stdout regardless of verbosity level.
• Retention tests: Tests verify that log file cleanup correctly removes files older than the configured retention period.
• LangSmith opt-in tests: Tests verify that the system functions correctly without LangSmith configured and that enabling LangSmith produces the expected traces.