Observability

Stage 2.7.2 introduces an explicit observability surface for CleverAgents. LangSmith tracing is disabled by default but can be toggled on via environment variables so that LangChain graphs emit structured telemetry without any code changes.

LangSmith tracing

Required environment variables

Variable(s)	Purpose
`CLEVERAGENTS_LANGSMITH_ENABLED` / `LANGCHAIN_TRACING_V2`	Primary toggle for LangSmith tracing. `Settings` automatically propagates this flag to the LangChain runtime.
`CLEVERAGENTS_LANGSMITH_API_KEY` / `LANGCHAIN_API_KEY`	Authenticates with the LangSmith service. Validation fails if a key is missing while tracing is enabled.
`CLEVERAGENTS_LANGSMITH_PROJECT` / `LANGCHAIN_PROJECT`	Logical project name shown inside LangSmith.
`CLEVERAGENTS_LANGSMITH_ENDPOINT` / `LANGCHAIN_ENDPOINT`	Optional override when targeting a self-hosted LangSmith deployment.
`CLEVERAGENTS_LANGSMITH_USER_ID`	Optional identifier used to enrich trace metadata with human-friendly ownership.
`CLEVERAGENTS_LANGSMITH_TAGS`	(Comma separated) list of global tags appended to every trace.

Tip: src/cleveragents/config/settings.py synchronizes these values for you. Once Settings.is_langsmith_enabled returns True, the module writes the LANGCHAIN_* variables so downstream libraries do not need bespoke configuration.

Quick start

Export the required environment variables (API key + project name).
Toggle tracing on: export CLEVERAGENTS_LANGSMITH_ENABLED=true.
Run any LangGraph-powered command, e.g. agents tell "add tracing" --stream.
Open LangSmith and filter by tags such as service:plan, auto-debug, or context-analysis to inspect the run.

Metadata & tags

Every LangGraph invocation builds a metadata payload via Settings.build_langsmith_config() and merges the following fields automatically:

project_id, project_name, plan_id, plan_name
user_id (when CLEVERAGENTS_LANGSMITH_USER_ID is set)
error_message, retry_attempt, retry_limit for auto-debug retries
Context file counts and execution mode for ContextService

Tags combine global tags (CLEVERAGENTS_LANGSMITH_TAGS) with per-service tags:

service:plan, service:context
project:<id>, plan:<id>
auto-debug, retry:<attempt> during error recovery

Sample trace

The example trace highlights how metadata, retry counters, and per-node events show up once the CLI streaming workflow (agents tell --stream) runs with tracing enabled.

Troubleshooting & validation

Settings.validate_langsmith_configuration() (see src/cleveragents/config/settings.py) returns both the activation status and any validation errors. Behave scenarios in features/langsmith_config.feature assert the following safeguards:

Missing API keys cause tracing to stay disabled with a descriptive error.
LangChain-only environment variables (LANGCHAIN_PROJECT, LANGCHAIN_API_KEY) are auto-detected.
The runtime always sets LANGCHAIN_TRACING_V2=true once a valid configuration is present.

If you see LangSmith validation errors should mention "API key" in test output, export the key or disable tracing.

Additional resources

LangSmith documentation
Implementation details: src/cleveragents/application/services/plan_service.py and src/cleveragents/application/services/context_service.py for how metadata is attached to LangGraph runs.

3.5 KiB Raw Permalink Blame History