cleveragents-core/docs/observability.md

Observability

This document describes the observability stack for CleverAgents Core: tracing, structured logging, metrics, audit trail, and token/cost tracking.

---

LangSmith Tracing

CleverAgents integrates with LangSmith for distributed tracing of LangChain/LangGraph agent runs. Tracing is off by default and is enabled by setting the environment variables below.

Quick Setup

export CLEVERAGENTS_LANGSMITH_ENABLED=true
export CLEVERAGENTS_LANGSMITH_PROJECT=my-project
export CLEVERAGENTS_LANGSMITH_API_KEY=ls__...

The settings module automatically mirrors these values to the LangChain environment variables (LANGCHAIN_TRACING_V2, LANGCHAIN_PROJECT, LANGCHAIN_API_KEY), so no additional wiring is required.

Environment Variables

Variable	Required	Default	Description
CLEVERAGENTS_LANGSMITH_ENABLED	No	false	Set to true to enable LangSmith tracing
CLEVERAGENTS_LANGSMITH_PROJECT	When enabled	—	LangSmith project name (maps to LANGCHAIN_PROJECT)
CLEVERAGENTS_LANGSMITH_API_KEY	When enabled	—	LangSmith API key (maps to LANGCHAIN_API_KEY)
CLEVERAGENTS_LANGSMITH_ENDPOINT	No	https://api.smith.langchain.com	Custom LangSmith endpoint for self-hosted deployments
CLEVERAGENTS_LANGSMITH_USER_ID	No	—	Optional user identifier attached to all traces
CLEVERAGENTS_LANGSMITH_TAGS	No	—	Comma-separated tags attached to every run (e.g. env:prod,team:core)

Mirrored LangChain Variables

The settings module sets these automatically when LangSmith is enabled. You do not need to set them manually:

LangChain Variable	Source
LANGCHAIN_TRACING_V2	Set to "true" when CLEVERAGENTS_LANGSMITH_ENABLED=true
LANGCHAIN_PROJECT	Copied from CLEVERAGENTS_LANGSMITH_PROJECT
LANGCHAIN_API_KEY	Copied from CLEVERAGENTS_LANGSMITH_API_KEY
LANGCHAIN_ENDPOINT	Copied from CLEVERAGENTS_LANGSMITH_ENDPOINT (if set)

Verifying Tracing

# Check that the settings module sees your credentials
agents diagnostics

# Run a quick plan and look for trace URLs in the output
agents tell "Hello" --actor openai/gpt-4o

Traces appear in the LangSmith UI under the configured project within a few seconds of each run.

---

Structured Logging

CleverAgents uses structlog for structured, machine-readable logging throughout the application.

Log Format

In development (TTY detected), logs are rendered as human-friendly colored key-value pairs:

2026-04-13 12:00:00 [info     ] Plan execution started   actor=openai/gpt-4o plan_id=01HZ...

In production (non-TTY / JSON mode), logs are emitted as newline-delimited JSON:

{"timestamp": "2026-04-13T12:00:00Z", "level": "info", "event": "Plan execution started", "actor": "openai/gpt-4o", "plan_id": "01HZ..."}

Log Level Configuration

Variable	Default	Description
CLEVERAGENTS_LOG_LEVEL	INFO	Minimum log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
CLEVERAGENTS_LOG_FORMAT	auto	Force output format: auto (detect TTY), json, console

Standard Context Fields

Every log record emitted by CleverAgents includes these fields when available:

Field	Description
actor	Active actor name (e.g. openai/gpt-4o)
plan_id	ULID of the current plan
session_id	Active session identifier
provider	LLM provider name
component	Source component (e.g. plan_service, context_tier)

Adding Structured Logs in Code

import structlog

log = structlog.get_logger(__name__)

log.info("context_hydration_complete", file_count=42, budget_used_mb=3.7)
log.warning("provider_fallback", from_provider="openai", to_provider="anthropic")
log.error("plan_execution_failed", plan_id=plan.id, exc_info=True)

---

Metrics and Audit Trail

Audit Trail

CleverAgents records an immutable audit trail of all plan lifecycle events. Each event captures:

• Event type — e.g. PLAN_CREATED, PLAN_EXECUTED, PLAN_APPLIED, INVARIANT_VIOLATED, CORRECTION_APPLIED, VALIDATION_FAILED
• Timestamp — ISO 8601 UTC
• Actor — which actor/agent triggered the event
• Plan ID — ULID of the affected plan
• Metadata — event-specific payload (e.g. validation results, invariant name)

Audit events are written to the structlog pipeline (so they appear in both console and JSON output) and optionally forwarded to LangSmith as run metadata when tracing is enabled.

Metrics

Runtime metrics are collected via the internal MetricsCollector and exposed through:

• Structured log fields — counters and gauges are emitted as log fields on key events
• agents diagnostics — prints a summary of session-level metrics

Key metrics tracked:

Metric	Description
plans.created	Total plans created in the session
plans.executed	Total plan executions
plans.applied	Total successful applies
plans.failed	Total failed executions
validations.passed	Validation checks that passed
validations.failed	Validation checks that failed
invariants.violated	Invariant violations detected
corrections.applied	Corrections applied to plans
context.files_hydrated	Files loaded into context tier
context.budget_used_bytes	Total context budget consumed

---

Token and Cost Tracking

CleverAgents tracks LLM token consumption and estimated cost for every plan execution.

Per-Run Tracking

Each LLM call records:

• Prompt tokens — tokens in the input
• Completion tokens — tokens in the output
• Total tokens — sum of prompt + completion
• Estimated cost (USD) — calculated from the provider's published pricing

These values are:

1. Emitted as structured log fields on every LLM call completion
2. Aggregated into the plan's audit record
3. Forwarded to LangSmith as run metadata (when tracing is enabled)

Session Totals

At the end of each session (or on agents diagnostics), cumulative totals are reported:

Session token usage:
  Prompt tokens:     12,450
  Completion tokens:  3,210
  Total tokens:      15,660
  Estimated cost:    $0.0423 USD

Cost Configuration

Pricing tables are bundled with the provider registry and updated with each release. To override pricing for a custom deployment:

export CLEVERAGENTS_PROVIDER_COST_OVERRIDE='{"openai/gpt-4o": {"prompt": 0.000005, "completion": 0.000015}}'

The value is a JSON object mapping provider/model to per-token costs in USD.

---

Environment Variable Reference

Complete list of observability-related environment variables:

Variable	Default	Description
CLEVERAGENTS_LANGSMITH_ENABLED	false	Enable LangSmith tracing
CLEVERAGENTS_LANGSMITH_PROJECT	—	LangSmith project name
CLEVERAGENTS_LANGSMITH_API_KEY	—	LangSmith API key
CLEVERAGENTS_LANGSMITH_ENDPOINT	https://api.smith.langchain.com	LangSmith endpoint URL
CLEVERAGENTS_LANGSMITH_USER_ID	—	User ID attached to traces
CLEVERAGENTS_LANGSMITH_TAGS	—	Comma-separated run tags
CLEVERAGENTS_LOG_LEVEL	INFO	Minimum log level
CLEVERAGENTS_LOG_FORMAT	auto	Log output format (auto/json/console)
CLEVERAGENTS_PROVIDER_COST_OVERRIDE	—	JSON override for provider pricing

---

Diagnostics

# Print observability configuration and session metrics
agents diagnostics

# Check LangSmith connectivity (requires CLEVERAGENTS_LANGSMITH_ENABLED=true)
agents diagnostics --check-tracing

# Show token usage for the last N plans
agents plan list --show-tokens --limit 10