forked from cleveragents/cleveragents-core
62 lines
3.5 KiB
Markdown
62 lines
3.5 KiB
Markdown
# Observability
|
|
|
|
Stage 2.7.2 introduces an explicit observability surface for CleverAgents. LangSmith tracing is disabled by default but can be toggled on via environment variables so that LangChain graphs emit structured telemetry without any code changes.
|
|
|
|
## LangSmith tracing
|
|
|
|
### Required environment variables
|
|
|
|
| Variable(s) | Purpose |
|
|
| --- | --- |
|
|
| `CLEVERAGENTS_LANGSMITH_ENABLED` / `LANGCHAIN_TRACING_V2` | Primary toggle for LangSmith tracing. `Settings` automatically propagates this flag to the LangChain runtime. |
|
|
| `CLEVERAGENTS_LANGSMITH_API_KEY` / `LANGCHAIN_API_KEY` | Authenticates with the LangSmith service. Validation fails if a key is missing while tracing is enabled. |
|
|
| `CLEVERAGENTS_LANGSMITH_PROJECT` / `LANGCHAIN_PROJECT` | Logical project name shown inside LangSmith. |
|
|
| `CLEVERAGENTS_LANGSMITH_ENDPOINT` / `LANGCHAIN_ENDPOINT` | Optional override when targeting a self-hosted LangSmith deployment. |
|
|
| `CLEVERAGENTS_LANGSMITH_USER_ID` | Optional identifier used to enrich trace metadata with human-friendly ownership. |
|
|
| `CLEVERAGENTS_LANGSMITH_TAGS` | (Comma separated) list of global tags appended to every trace. |
|
|
|
|
> **Tip:** `src/cleveragents/config/settings.py` synchronizes these values for you. Once `Settings.is_langsmith_enabled` returns `True`, the module writes the `LANGCHAIN_*` variables so downstream libraries do not need bespoke configuration.
|
|
|
|
### Quick start
|
|
|
|
1. Export the required environment variables (API key + project name).
|
|
2. Toggle tracing on: `export CLEVERAGENTS_LANGSMITH_ENABLED=true`.
|
|
3. Run any LangGraph-powered command, e.g. `agents tell "add tracing" --stream`.
|
|
4. Open LangSmith and filter by tags such as `service:plan`, `auto-debug`, or `context-analysis` to inspect the run.
|
|
|
|
### Metadata & tags
|
|
|
|
Every LangGraph invocation builds a metadata payload via `Settings.build_langsmith_config()` and merges the following fields automatically:
|
|
|
|
- `project_id`, `project_name`, `plan_id`, `plan_name`
|
|
- `user_id` (when `CLEVERAGENTS_LANGSMITH_USER_ID` is set)
|
|
- `error_message`, `retry_attempt`, `retry_limit` for auto-debug retries
|
|
- Context file counts and execution mode for `ContextService`
|
|
|
|
Tags combine global tags (`CLEVERAGENTS_LANGSMITH_TAGS`) with per-service tags:
|
|
|
|
- `service:plan`, `service:context`
|
|
- `project:<id>`, `plan:<id>`
|
|
- `auto-debug`, `retry:<attempt>` during error recovery
|
|
|
|
### Sample trace
|
|
|
|
|
|
|
|
The example trace highlights how metadata, retry counters, and per-node events show up once the CLI streaming workflow (`agents tell --stream`) runs with tracing enabled.
|
|
|
|
### Troubleshooting & validation
|
|
|
|
`Settings.validate_langsmith_configuration()` (see `src/cleveragents/config/settings.py`) returns both the activation status and any validation errors. Behave scenarios in `features/langsmith_config.feature` assert the following safeguards:
|
|
|
|
- Missing API keys cause tracing to stay disabled with a descriptive error.
|
|
- LangChain-only environment variables (`LANGCHAIN_PROJECT`, `LANGCHAIN_API_KEY`) are auto-detected.
|
|
- The runtime always sets `LANGCHAIN_TRACING_V2=true` once a valid configuration is present.
|
|
|
|
If you see `LangSmith validation errors should mention "API key"` in test output, export the key or disable tracing.
|
|
|
|
### Additional resources
|
|
|
|
- [LangSmith documentation](https://docs.smith.langchain.com/)
|
|
- Implementation details: `src/cleveragents/application/services/plan_service.py` and `src/cleveragents/application/services/context_service.py` for how metadata is attached to LangGraph runs.
|