Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a18a87b6ed |
+57
-72
@@ -5,35 +5,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Automation Profile Silent Fallback** (#8232): `_resolve_profile_for_plan` in
|
||||
`PlanLifecycleService` now raises a clear `ValidationError` when a plan's
|
||||
automation profile name is not a known built-in profile, instead of silently
|
||||
falling back to `"manual"`. Users who configured custom automation profiles
|
||||
(e.g. `"semi-auto"`, `"acme/strict"`) will now receive an actionable error
|
||||
message listing available built-in profiles. The resolved profile name is also
|
||||
logged at debug level for observability.
|
||||
<!-- Development focus: v3.2.0 milestone — Decisions + Validations + Invariants.
|
||||
This section accumulates all changes since the [3.8.0] release (2026-04-05).
|
||||
Entries span agent-system hardening, automation tracking, PR workflow improvements,
|
||||
TDD infrastructure, and plan-lifecycle fixes. -->
|
||||
|
||||
### Added
|
||||
|
||||
- **TDD Issue-Capture Test Activation** (#7025): Replaced 234 bare `@skip` tags
|
||||
across 82 Behave feature files with the correct `@tdd_expected_fail @tdd_issue
|
||||
@tdd_issue_<N>` tag system. Scenarios whose referenced bugs were already fixed
|
||||
had `@tdd_expected_fail` removed and now run as permanent regression guards.
|
||||
Net result: 629 features active in CI (up from ~545), zero `@skip` tags remain.
|
||||
|
||||
- **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges
|
||||
LLM-generated changes via `git merge` from an isolated worktree branch
|
||||
instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary
|
||||
(plan ID, artifacts, insertions/deletions, project, timestamp), Sandbox
|
||||
Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall
|
||||
back to the original flat file copy.
|
||||
|
||||
- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types
|
||||
(`detail_depth` and `relevance_score` must be strings, not int/float) that
|
||||
caused Pydantic validation errors during context assembly, resulting in the
|
||||
LLM receiving zero file context.
|
||||
#### Agent System & Automation
|
||||
|
||||
- **Automation Tracking System**: Replaced shared session state issue tracking with
|
||||
individual per-agent tracking issues. Each agent now creates its own `[AUTO-<PREFIX>]`
|
||||
@@ -49,6 +28,34 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
creates high-priority diagnostic issues, and closes stale tracking issues with recovery
|
||||
notes.
|
||||
|
||||
- **Centralized Automation Tracking Manager** (`automation-tracking-manager`): The manager
|
||||
is now the single interface for all tracking issue operations (`CREATE_TRACKING_ISSUE`,
|
||||
`UPDATE_TRACKING_ISSUE`, `CLOSE_TRACKING_ISSUE`, `READ_TRACKING_STATE`,
|
||||
`GET_NEXT_CYCLE_NUMBER`). Agents delegate to the manager rather than calling the Forgejo
|
||||
API directly, ensuring sequential cycle numbers across restarts and preventing
|
||||
duplicate issues. Migrated agents include `system-watchdog`,
|
||||
`implementation-pool-supervisor`, `timeline-update-pool-supervisor`,
|
||||
`project-owner-pool-supervisor`, `product-builder`, and
|
||||
`backlog-grooming-pool-supervisor`. The legacy `shared/automation_tracking.md` module
|
||||
was removed.
|
||||
|
||||
- **Automation Tracking Announcements**: Extended `automation-tracking-manager` with
|
||||
announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`,
|
||||
`LIST_TRACKING_ISSUES`, `READ_ANNOUNCEMENTS`, `REVIEW_OWN_ANNOUNCEMENTS`). Supervisors
|
||||
and workers now read critical announcements before each cycle for cross-agent awareness.
|
||||
Priority-based filtering (Critical/High/Medium/Low) reduces noise. Backlog-groomer
|
||||
performs intelligent cleanup with age thresholds by priority.
|
||||
|
||||
- **Documentation Writer Tracking** (`docs-writer`): The documentation writer now
|
||||
participates in the automation tracking system by creating individual
|
||||
`[AUTO-DOCS] Documentation Report (Cycle N)` issues every 10 cycles (~3.3 hours).
|
||||
The manager applies the mandatory `Automation Tracking` label automatically, while
|
||||
teams may add additional workflow labels as needed. See
|
||||
`docs/development/automation-tracking.md` and the new
|
||||
`docs/development/docs-writer.md` reference.
|
||||
|
||||
#### Label & PR Management
|
||||
|
||||
- **Centralized Label Management** (`forgejo-label-manager`): A new specialized subagent
|
||||
centralizes all Forgejo label operations across the agent system. Enforces the
|
||||
organization-level label system, prohibits label creation, and validates label compliance.
|
||||
@@ -62,13 +69,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
PR–issue label synchronization. The `issue-state-updater` syncs PR state labels whenever
|
||||
issue states change.
|
||||
|
||||
- **Automation Tracking Announcements**: Extended `automation-tracking-manager` with
|
||||
announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`,
|
||||
`LIST_TRACKING_ISSUES`, `READ_ANNOUNCEMENTS`, `REVIEW_OWN_ANNOUNCEMENTS`). Supervisors
|
||||
and workers now read critical announcements before each cycle for cross-agent awareness.
|
||||
Priority-based filtering (Critical/High/Medium/Low) reduces noise. Backlog-groomer
|
||||
performs intelligent cleanup with age thresholds by priority.
|
||||
|
||||
- **PR Agent Reorganization**: All PR-related agents renamed and reorganized to follow
|
||||
the `*-pool-supervisor` naming pattern. New agents added: `pr-editor` (safe PR editing
|
||||
with description preservation), `pr-manager` (unified PR interface), and
|
||||
@@ -82,6 +82,15 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
(approvals, CI passing, no conflicts). Supports both formal reviews and comment-based
|
||||
approvals (LGTM, ✅, "ready to merge", etc.).
|
||||
|
||||
#### Plan Lifecycle & Implementation
|
||||
|
||||
- **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges
|
||||
LLM-generated changes via `git merge` from an isolated worktree branch
|
||||
instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary
|
||||
(plan ID, artifacts, insertions/deletions, project, timestamp), Sandbox
|
||||
Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall
|
||||
back to the original flat file copy.
|
||||
|
||||
- **Implementation Worker Workflow Completion**: `implementation-worker` now implements
|
||||
work claiming protocols with conflict detection, comprehensive review feedback handling
|
||||
with intelligent parsing, sophisticated merge conflict resolution with multiple
|
||||
@@ -91,24 +100,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
- **Container Resource Stop Support**: `agents resource stop` now correctly stops
|
||||
`container-instance` and `devcontainer-instance` resource types.
|
||||
|
||||
- **Centralized Automation Tracking Manager** (`automation-tracking-manager`): The manager
|
||||
is now the single interface for all tracking issue operations (`CREATE_TRACKING_ISSUE`,
|
||||
`UPDATE_TRACKING_ISSUE`, `CLOSE_TRACKING_ISSUE`, `READ_TRACKING_STATE`,
|
||||
`GET_NEXT_CYCLE_NUMBER`). Agents delegate to the manager rather than calling the Forgejo
|
||||
API directly, ensuring sequential cycle numbers across restarts and preventing
|
||||
duplicate issues. Migrated agents include `system-watchdog`,
|
||||
`implementation-pool-supervisor`, `timeline-update-pool-supervisor`,
|
||||
`project-owner-pool-supervisor`, `product-builder`, and
|
||||
`backlog-grooming-pool-supervisor`. The legacy `shared/automation_tracking.md` module
|
||||
was removed.
|
||||
#### Test Infrastructure
|
||||
|
||||
- **Documentation Writer Tracking** (`docs-writer`): The documentation writer now
|
||||
participates in the automation tracking system by creating individual
|
||||
`[AUTO-DOCS] Documentation Report (Cycle N)` issues every 10 cycles (~3.3 hours).
|
||||
The manager applies the mandatory `Automation Tracking` label automatically, while
|
||||
teams may add additional workflow labels as needed. See
|
||||
`docs/development/automation-tracking.md` and the new
|
||||
`docs/development/docs-writer.md` reference.
|
||||
- **TDD Issue-Capture Test Activation** (#7025): Replaced 234 bare `@skip` tags
|
||||
across 82 Behave feature files with the correct `@tdd_expected_fail @tdd_issue
|
||||
@tdd_issue_<N>` tag system. Scenarios whose referenced bugs were already fixed
|
||||
had `@tdd_expected_fail` removed and now run as permanent regression guards.
|
||||
Net result: 629 features active in CI (up from ~545), zero `@skip` tags remain.
|
||||
|
||||
### Changed
|
||||
|
||||
@@ -143,14 +141,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Plan Concurrency Race Condition** (#7989): Fixed critical race condition in `execute_plan()` and
|
||||
`apply_plan()` where concurrent CLI/worker sessions could simultaneously modify the same plan,
|
||||
corrupting plan state. `LockService` is now wired into the plan lifecycle with plan-level advisory
|
||||
locking. Each invocation generates a unique caller identity (UUID) to prevent re-entrant lock
|
||||
acquisition by concurrent sessions on the same plan. Concurrent attempts now raise `LockConflictError`
|
||||
instead of silently racing. Lock is acquired before phase transition and released in a `finally`
|
||||
block to ensure cleanup even on error.
|
||||
|
||||
- **Validation Gate Empty-Run Guard** (#7508): Fixed `ApplyValidationSummary.all_required_passed`
|
||||
returning `True` when zero validations were run, silently bypassing the apply gate. The property
|
||||
now returns `False` when the validation result set is empty (`is_empty` is `True`), ensuring
|
||||
@@ -158,28 +148,23 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
`required_total` property for completeness. Updated `consolidated_validation.feature` scenarios
|
||||
to reflect the corrected blocking behavior for empty summaries and no-attachment runs.
|
||||
|
||||
- **ACMS context tier hydration**: `ContextTierService` no longer starts empty
|
||||
- **ACMS context tier hydration** (#1028): `ContextTierService` no longer starts empty
|
||||
on every CLI invocation. A new `context_tier_hydrator.py` reads files from
|
||||
linked project resources (via `git ls-files` or `os.walk`), creates
|
||||
`TieredFragment` objects, and stores them in the tier service before context
|
||||
assembly in `LLMExecuteActor.execute()`. The LLM now receives real file
|
||||
context during plan execution. Respects max file size (256 KB), total budget
|
||||
(10 MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__`
|
||||
directory skipping. (#1028)
|
||||
directory skipping.
|
||||
|
||||
- **Sandbox root wiring**: `_get_plan_executor()` now passes
|
||||
- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types
|
||||
(`detail_depth` and `relevance_score` must be strings, not int/float) that
|
||||
caused Pydantic validation errors during context assembly, resulting in the
|
||||
LLM receiving zero file context.
|
||||
|
||||
- **Sandbox root wiring** (#4222): `_get_plan_executor()` now passes
|
||||
`sandbox_root=.cleveragents/sandbox/`, so LLM file output (`FILE:` blocks)
|
||||
is written to disk during the execute phase. (#4222)
|
||||
|
||||
- **SubplanExecutionService fail_fast cancellation** (#7582): Fixed a race condition where
|
||||
already-running parallel subplans were not cancelled when `fail_fast` fired. Previously,
|
||||
`Future.cancel()` only prevented queued futures from starting but had no effect on
|
||||
in-flight futures that completed after `stop_flag` was set — their `COMPLETE` results
|
||||
were incorrectly included in the merge output. The fix adds a post-completion guard that
|
||||
overrides any non-`ERRORED`/non-`CANCELLED` result to `CANCELLED` when `stop_flag` is
|
||||
active, and clears the associated output to prevent it from entering the merge. Also
|
||||
replaces the O(n) linear `status` lookup in the `as_completed()` loop with an O(1)
|
||||
`status_map` dict pre-computed before the executor block.
|
||||
is written to disk during the execute phase.
|
||||
|
||||
- **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the
|
||||
`tdd_expected_fail_listener` `end_test()` function to prevent blindly inverting ALL test
|
||||
|
||||
@@ -17,13 +17,37 @@ nox -s lint # Ruff linting
|
||||
nox -s format # Ruff formatting
|
||||
nox -s typecheck # Pyright type checking
|
||||
nox -s unit_tests # Behave unit tests
|
||||
nox -s integration_tests # Robot Framework integration tests
|
||||
nox -s coverage_report # Coverage (must be >=97%)
|
||||
nox -s security_scan # Bandit + Semgrep + Vulture security scanning
|
||||
nox -s dead_code # Vulture dead code detection
|
||||
nox -s complexity # Radon complexity analysis
|
||||
nox -s pre_commit # Run all pre-commit hooks
|
||||
nox -s adr_compliance # Verify ADR compliance
|
||||
nox -s docs # Build MkDocs documentation
|
||||
nox -s serve_docs # Serve docs locally (http://localhost:8000)
|
||||
```
|
||||
|
||||
## Nox Session Reference
|
||||
|
||||
All available Nox sessions and their purpose:
|
||||
|
||||
| Session | Tool(s) | Purpose | Blocks CI |
|
||||
|---|---|---|---|
|
||||
| `format` | Ruff | Auto-format source code | Yes (diff check) |
|
||||
| `lint` | Ruff | Lint with safe auto-fixes | Yes |
|
||||
| `typecheck` | Pyright | Static type checking (`strict` mode) | Yes |
|
||||
| `unit_tests` | Behave | BDD unit test suite | Yes |
|
||||
| `integration_tests` | Robot Framework | End-to-end integration tests | Yes |
|
||||
| `coverage_report` | Coverage.py | Measure test coverage (≥97% required) | Yes |
|
||||
| `security_scan` | Bandit + Semgrep + Vulture | Security and dead-code scanning | Yes |
|
||||
| `dead_code` | Vulture | Dead code detection (standalone) | No |
|
||||
| `complexity` | Radon | Cyclomatic complexity analysis | Yes (grade F) |
|
||||
| `pre_commit` | pre-commit | Run all pre-commit hooks | No |
|
||||
| `adr_compliance` | Custom script | Verify ADR references are consistent | No |
|
||||
| `docs` | MkDocs | Build documentation site | No |
|
||||
| `serve_docs` | MkDocs | Serve docs locally for preview | No |
|
||||
|
||||
## Pre-commit Hooks
|
||||
|
||||
Pre-commit hooks run automatically on every `git commit`. They are configured in
|
||||
@@ -71,17 +95,19 @@ The CI pipeline runs on **Forgejo Actions** (`.forgejo/workflows/ci.yml`).
|
||||
|
||||
### CI Jobs
|
||||
|
||||
| Job | Trigger | Purpose | Failure Impact |
|
||||
| ----------- | ------- | ----------------------------------- | ---------------------- |
|
||||
| `lint` | Push/PR | Ruff format + lint check | Blocks merge |
|
||||
| `typecheck` | Push/PR | Pyright type checking | Blocks merge |
|
||||
| `security` | Push/PR | Bandit + Semgrep + Vulture | Blocks merge |
|
||||
| `quality` | Push/PR | Radon complexity check | Blocks merge (grade F) |
|
||||
| `unit_tests`| Push/PR | Behave BDD tests (Python 3.13) | Blocks merge |
|
||||
| `coverage` | Push/PR | Coverage measurement | Blocks merge (<97%) |
|
||||
| `build` | Push/PR | Wheel build | Blocks release |
|
||||
| `docker` | Push/PR | Docker image build + test | Blocks deployment |
|
||||
| `helm` | Push/PR | Helm chart lint + template | Blocks deployment |
|
||||
| Job | Trigger | Purpose | Failure Impact |
|
||||
| ------------------- | ------- | ----------------------------------- | ---------------------- |
|
||||
| `lint` | Push/PR | Ruff format + lint check | Blocks merge |
|
||||
| `typecheck` | Push/PR | Pyright type checking | Blocks merge |
|
||||
| `security` | Push/PR | Bandit + Semgrep + Vulture | Blocks merge |
|
||||
| `quality` | Push/PR | Radon complexity check | Blocks merge (grade F) |
|
||||
| `unit_tests` | Push/PR | Behave BDD tests (Python 3.13) | Blocks merge |
|
||||
| `integration_tests` | Push/PR | Robot Framework e2e tests | Blocks merge |
|
||||
| `coverage` | Push/PR | Coverage measurement | Blocks merge (<97%) |
|
||||
| `adr_compliance` | Push/PR | ADR reference consistency check | Blocks merge |
|
||||
| `build` | Push/PR | Wheel build | Blocks release |
|
||||
| `docker` | Push/PR | Docker image build + test | Blocks deployment |
|
||||
| `helm` | Push/PR | Helm chart lint + template | Blocks deployment |
|
||||
|
||||
### Nightly Quality
|
||||
|
||||
|
||||
@@ -0,0 +1,229 @@
|
||||
# Observability
|
||||
|
||||
This document describes the observability stack for CleverAgents Core: tracing,
|
||||
structured logging, metrics, audit trail, and token/cost tracking.
|
||||
|
||||
---
|
||||
|
||||
## LangSmith Tracing
|
||||
|
||||
CleverAgents integrates with [LangSmith](https://smith.langchain.com/) for distributed
|
||||
tracing of LangChain/LangGraph agent runs. Tracing is **off by default** and is enabled
|
||||
by setting the environment variables below.
|
||||
|
||||
### Quick Setup
|
||||
|
||||
```bash
|
||||
export CLEVERAGENTS_LANGSMITH_ENABLED=true
|
||||
export CLEVERAGENTS_LANGSMITH_PROJECT=my-project
|
||||
export CLEVERAGENTS_LANGSMITH_API_KEY=ls__...
|
||||
```
|
||||
|
||||
The settings module automatically mirrors these values to the LangChain environment
|
||||
variables (`LANGCHAIN_TRACING_V2`, `LANGCHAIN_PROJECT`, `LANGCHAIN_API_KEY`), so no
|
||||
additional wiring is required.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `CLEVERAGENTS_LANGSMITH_ENABLED` | No | `false` | Set to `true` to enable LangSmith tracing |
|
||||
| `CLEVERAGENTS_LANGSMITH_PROJECT` | When enabled | — | LangSmith project name (maps to `LANGCHAIN_PROJECT`) |
|
||||
| `CLEVERAGENTS_LANGSMITH_API_KEY` | When enabled | — | LangSmith API key (maps to `LANGCHAIN_API_KEY`) |
|
||||
| `CLEVERAGENTS_LANGSMITH_ENDPOINT` | No | `https://api.smith.langchain.com` | Custom LangSmith endpoint for self-hosted deployments |
|
||||
| `CLEVERAGENTS_LANGSMITH_USER_ID` | No | — | Optional user identifier attached to all traces |
|
||||
| `CLEVERAGENTS_LANGSMITH_TAGS` | No | — | Comma-separated tags attached to every run (e.g. `env:prod,team:core`) |
|
||||
|
||||
### Mirrored LangChain Variables
|
||||
|
||||
The settings module sets these automatically when LangSmith is enabled. You do **not**
|
||||
need to set them manually:
|
||||
|
||||
| LangChain Variable | Source |
|
||||
|---|---|
|
||||
| `LANGCHAIN_TRACING_V2` | Set to `"true"` when `CLEVERAGENTS_LANGSMITH_ENABLED=true` |
|
||||
| `LANGCHAIN_PROJECT` | Copied from `CLEVERAGENTS_LANGSMITH_PROJECT` |
|
||||
| `LANGCHAIN_API_KEY` | Copied from `CLEVERAGENTS_LANGSMITH_API_KEY` |
|
||||
| `LANGCHAIN_ENDPOINT` | Copied from `CLEVERAGENTS_LANGSMITH_ENDPOINT` (if set) |
|
||||
|
||||
### Verifying Tracing
|
||||
|
||||
```bash
|
||||
# Check that the settings module sees your credentials
|
||||
agents diagnostics
|
||||
|
||||
# Run a quick plan and look for trace URLs in the output
|
||||
agents tell "Hello" --actor openai/gpt-4o
|
||||
```
|
||||
|
||||
Traces appear in the LangSmith UI under the configured project within a few seconds of
|
||||
each run.
|
||||
|
||||
---
|
||||
|
||||
## Structured Logging
|
||||
|
||||
CleverAgents uses [structlog](https://www.structlog.org/) for structured, machine-readable
|
||||
logging throughout the application.
|
||||
|
||||
### Log Format
|
||||
|
||||
In development (TTY detected), logs are rendered as human-friendly colored key-value pairs:
|
||||
|
||||
```
|
||||
2026-04-13 12:00:00 [info ] Plan execution started actor=openai/gpt-4o plan_id=01HZ...
|
||||
```
|
||||
|
||||
In production (non-TTY / JSON mode), logs are emitted as newline-delimited JSON:
|
||||
|
||||
```json
|
||||
{"timestamp": "2026-04-13T12:00:00Z", "level": "info", "event": "Plan execution started", "actor": "openai/gpt-4o", "plan_id": "01HZ..."}
|
||||
```
|
||||
|
||||
### Log Level Configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `CLEVERAGENTS_LOG_LEVEL` | `INFO` | Minimum log level (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`) |
|
||||
| `CLEVERAGENTS_LOG_FORMAT` | `auto` | Force output format: `auto` (detect TTY), `json`, `console` |
|
||||
|
||||
### Standard Context Fields
|
||||
|
||||
Every log record emitted by CleverAgents includes these fields when available:
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| `actor` | Active actor name (e.g. `openai/gpt-4o`) |
|
||||
| `plan_id` | ULID of the current plan |
|
||||
| `session_id` | Active session identifier |
|
||||
| `provider` | LLM provider name |
|
||||
| `component` | Source component (e.g. `plan_service`, `context_tier`) |
|
||||
|
||||
### Adding Structured Logs in Code
|
||||
|
||||
```python
|
||||
import structlog
|
||||
|
||||
log = structlog.get_logger(__name__)
|
||||
|
||||
log.info("context_hydration_complete", file_count=42, budget_used_mb=3.7)
|
||||
log.warning("provider_fallback", from_provider="openai", to_provider="anthropic")
|
||||
log.error("plan_execution_failed", plan_id=plan.id, exc_info=True)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Metrics and Audit Trail
|
||||
|
||||
### Audit Trail
|
||||
|
||||
CleverAgents records an immutable audit trail of all plan lifecycle events. Each event
|
||||
captures:
|
||||
|
||||
- **Event type** — e.g. `PLAN_CREATED`, `PLAN_EXECUTED`, `PLAN_APPLIED`,
|
||||
`INVARIANT_VIOLATED`, `CORRECTION_APPLIED`, `VALIDATION_FAILED`
|
||||
- **Timestamp** — ISO 8601 UTC
|
||||
- **Actor** — which actor/agent triggered the event
|
||||
- **Plan ID** — ULID of the affected plan
|
||||
- **Metadata** — event-specific payload (e.g. validation results, invariant name)
|
||||
|
||||
Audit events are written to the structlog pipeline (so they appear in both console and
|
||||
JSON output) and optionally forwarded to LangSmith as run metadata when tracing is enabled.
|
||||
|
||||
### Metrics
|
||||
|
||||
Runtime metrics are collected via the internal `MetricsCollector` and exposed through:
|
||||
|
||||
- **Structured log fields** — counters and gauges are emitted as log fields on key events
|
||||
- **`agents diagnostics`** — prints a summary of session-level metrics
|
||||
|
||||
Key metrics tracked:
|
||||
|
||||
| Metric | Description |
|
||||
|---|---|
|
||||
| `plans.created` | Total plans created in the session |
|
||||
| `plans.executed` | Total plan executions |
|
||||
| `plans.applied` | Total successful applies |
|
||||
| `plans.failed` | Total failed executions |
|
||||
| `validations.passed` | Validation checks that passed |
|
||||
| `validations.failed` | Validation checks that failed |
|
||||
| `invariants.violated` | Invariant violations detected |
|
||||
| `corrections.applied` | Corrections applied to plans |
|
||||
| `context.files_hydrated` | Files loaded into context tier |
|
||||
| `context.budget_used_bytes` | Total context budget consumed |
|
||||
|
||||
---
|
||||
|
||||
## Token and Cost Tracking
|
||||
|
||||
CleverAgents tracks LLM token consumption and estimated cost for every plan execution.
|
||||
|
||||
### Per-Run Tracking
|
||||
|
||||
Each LLM call records:
|
||||
|
||||
- **Prompt tokens** — tokens in the input
|
||||
- **Completion tokens** — tokens in the output
|
||||
- **Total tokens** — sum of prompt + completion
|
||||
- **Estimated cost (USD)** — calculated from the provider's published pricing
|
||||
|
||||
These values are:
|
||||
1. Emitted as structured log fields on every LLM call completion
|
||||
2. Aggregated into the plan's audit record
|
||||
3. Forwarded to LangSmith as run metadata (when tracing is enabled)
|
||||
|
||||
### Session Totals
|
||||
|
||||
At the end of each session (or on `agents diagnostics`), cumulative totals are reported:
|
||||
|
||||
```
|
||||
Session token usage:
|
||||
Prompt tokens: 12,450
|
||||
Completion tokens: 3,210
|
||||
Total tokens: 15,660
|
||||
Estimated cost: $0.0423 USD
|
||||
```
|
||||
|
||||
### Cost Configuration
|
||||
|
||||
Pricing tables are bundled with the provider registry and updated with each release.
|
||||
To override pricing for a custom deployment:
|
||||
|
||||
```bash
|
||||
export CLEVERAGENTS_PROVIDER_COST_OVERRIDE='{"openai/gpt-4o": {"prompt": 0.000005, "completion": 0.000015}}'
|
||||
```
|
||||
|
||||
The value is a JSON object mapping `provider/model` to per-token costs in USD.
|
||||
|
||||
---
|
||||
|
||||
## Environment Variable Reference
|
||||
|
||||
Complete list of observability-related environment variables:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `CLEVERAGENTS_LANGSMITH_ENABLED` | `false` | Enable LangSmith tracing |
|
||||
| `CLEVERAGENTS_LANGSMITH_PROJECT` | — | LangSmith project name |
|
||||
| `CLEVERAGENTS_LANGSMITH_API_KEY` | — | LangSmith API key |
|
||||
| `CLEVERAGENTS_LANGSMITH_ENDPOINT` | `https://api.smith.langchain.com` | LangSmith endpoint URL |
|
||||
| `CLEVERAGENTS_LANGSMITH_USER_ID` | — | User ID attached to traces |
|
||||
| `CLEVERAGENTS_LANGSMITH_TAGS` | — | Comma-separated run tags |
|
||||
| `CLEVERAGENTS_LOG_LEVEL` | `INFO` | Minimum log level |
|
||||
| `CLEVERAGENTS_LOG_FORMAT` | `auto` | Log output format (`auto`/`json`/`console`) |
|
||||
| `CLEVERAGENTS_PROVIDER_COST_OVERRIDE` | — | JSON override for provider pricing |
|
||||
|
||||
---
|
||||
|
||||
## Diagnostics
|
||||
|
||||
```bash
|
||||
# Print observability configuration and session metrics
|
||||
agents diagnostics
|
||||
|
||||
# Check LangSmith connectivity (requires CLEVERAGENTS_LANGSMITH_ENABLED=true)
|
||||
agents diagnostics --check-tracing
|
||||
|
||||
# Show token usage for the last N plans
|
||||
agents plan list --show-tokens --limit 10
|
||||
```
|
||||
Reference in New Issue
Block a user