Compare commits

...

1 Commits

Author SHA1 Message Date
Repo Isolator a18a87b6ed docs: organize CHANGELOG [Unreleased] section and update observability docs [AUTO-DOCS-4]
CI / lint (pull_request) Successful in 36s
CI / typecheck (pull_request) Successful in 44s
CI / security (pull_request) Successful in 52s
CI / quality (pull_request) Successful in 30s
CI / build (pull_request) Successful in 24s
CI / helm (pull_request) Successful in 34s
CI / push-validation (pull_request) Successful in 21s
CI / e2e_tests (pull_request) Successful in 3m7s
CI / integration_tests (pull_request) Successful in 4m4s
CI / unit_tests (pull_request) Successful in 5m57s
CI / docker (pull_request) Successful in 22s
CI / coverage (pull_request) Successful in 13m22s
CI / status-check (pull_request) Successful in 1s
2026-04-14 06:03:55 +00:00
3 changed files with 323 additions and 83 deletions
+57 -72
View File
@@ -5,35 +5,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
## [Unreleased]
### Fixed
- **Automation Profile Silent Fallback** (#8232): `_resolve_profile_for_plan` in
`PlanLifecycleService` now raises a clear `ValidationError` when a plan's
automation profile name is not a known built-in profile, instead of silently
falling back to `"manual"`. Users who configured custom automation profiles
(e.g. `"semi-auto"`, `"acme/strict"`) will now receive an actionable error
message listing available built-in profiles. The resolved profile name is also
logged at debug level for observability.
<!-- Development focus: v3.2.0 milestone — Decisions + Validations + Invariants.
This section accumulates all changes since the [3.8.0] release (2026-04-05).
Entries span agent-system hardening, automation tracking, PR workflow improvements,
TDD infrastructure, and plan-lifecycle fixes. -->
### Added
- **TDD Issue-Capture Test Activation** (#7025): Replaced 234 bare `@skip` tags
across 82 Behave feature files with the correct `@tdd_expected_fail @tdd_issue
@tdd_issue_<N>` tag system. Scenarios whose referenced bugs were already fixed
had `@tdd_expected_fail` removed and now run as permanent regression guards.
Net result: 629 features active in CI (up from ~545), zero `@skip` tags remain.
- **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges
LLM-generated changes via `git merge` from an isolated worktree branch
instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary
(plan ID, artifacts, insertions/deletions, project, timestamp), Sandbox
Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall
back to the original flat file copy.
- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types
(`detail_depth` and `relevance_score` must be strings, not int/float) that
caused Pydantic validation errors during context assembly, resulting in the
LLM receiving zero file context.
#### Agent System & Automation
- **Automation Tracking System**: Replaced shared session state issue tracking with
individual per-agent tracking issues. Each agent now creates its own `[AUTO-<PREFIX>]`
@@ -49,6 +28,34 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
creates high-priority diagnostic issues, and closes stale tracking issues with recovery
notes.
- **Centralized Automation Tracking Manager** (`automation-tracking-manager`): The manager
is now the single interface for all tracking issue operations (`CREATE_TRACKING_ISSUE`,
`UPDATE_TRACKING_ISSUE`, `CLOSE_TRACKING_ISSUE`, `READ_TRACKING_STATE`,
`GET_NEXT_CYCLE_NUMBER`). Agents delegate to the manager rather than calling the Forgejo
API directly, ensuring sequential cycle numbers across restarts and preventing
duplicate issues. Migrated agents include `system-watchdog`,
`implementation-pool-supervisor`, `timeline-update-pool-supervisor`,
`project-owner-pool-supervisor`, `product-builder`, and
`backlog-grooming-pool-supervisor`. The legacy `shared/automation_tracking.md` module
was removed.
- **Automation Tracking Announcements**: Extended `automation-tracking-manager` with
announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`,
`LIST_TRACKING_ISSUES`, `READ_ANNOUNCEMENTS`, `REVIEW_OWN_ANNOUNCEMENTS`). Supervisors
and workers now read critical announcements before each cycle for cross-agent awareness.
Priority-based filtering (Critical/High/Medium/Low) reduces noise. Backlog-groomer
performs intelligent cleanup with age thresholds by priority.
- **Documentation Writer Tracking** (`docs-writer`): The documentation writer now
participates in the automation tracking system by creating individual
`[AUTO-DOCS] Documentation Report (Cycle N)` issues every 10 cycles (~3.3 hours).
The manager applies the mandatory `Automation Tracking` label automatically, while
teams may add additional workflow labels as needed. See
`docs/development/automation-tracking.md` and the new
`docs/development/docs-writer.md` reference.
#### Label & PR Management
- **Centralized Label Management** (`forgejo-label-manager`): A new specialized subagent
centralizes all Forgejo label operations across the agent system. Enforces the
organization-level label system, prohibits label creation, and validates label compliance.
@@ -62,13 +69,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
PRissue label synchronization. The `issue-state-updater` syncs PR state labels whenever
issue states change.
- **Automation Tracking Announcements**: Extended `automation-tracking-manager` with
announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`,
`LIST_TRACKING_ISSUES`, `READ_ANNOUNCEMENTS`, `REVIEW_OWN_ANNOUNCEMENTS`). Supervisors
and workers now read critical announcements before each cycle for cross-agent awareness.
Priority-based filtering (Critical/High/Medium/Low) reduces noise. Backlog-groomer
performs intelligent cleanup with age thresholds by priority.
- **PR Agent Reorganization**: All PR-related agents renamed and reorganized to follow
the `*-pool-supervisor` naming pattern. New agents added: `pr-editor` (safe PR editing
with description preservation), `pr-manager` (unified PR interface), and
@@ -82,6 +82,15 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
(approvals, CI passing, no conflicts). Supports both formal reviews and comment-based
approvals (LGTM, ✅, "ready to merge", etc.).
#### Plan Lifecycle & Implementation
- **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges
LLM-generated changes via `git merge` from an isolated worktree branch
instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary
(plan ID, artifacts, insertions/deletions, project, timestamp), Sandbox
Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall
back to the original flat file copy.
- **Implementation Worker Workflow Completion**: `implementation-worker` now implements
work claiming protocols with conflict detection, comprehensive review feedback handling
with intelligent parsing, sophisticated merge conflict resolution with multiple
@@ -91,24 +100,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
- **Container Resource Stop Support**: `agents resource stop` now correctly stops
`container-instance` and `devcontainer-instance` resource types.
- **Centralized Automation Tracking Manager** (`automation-tracking-manager`): The manager
is now the single interface for all tracking issue operations (`CREATE_TRACKING_ISSUE`,
`UPDATE_TRACKING_ISSUE`, `CLOSE_TRACKING_ISSUE`, `READ_TRACKING_STATE`,
`GET_NEXT_CYCLE_NUMBER`). Agents delegate to the manager rather than calling the Forgejo
API directly, ensuring sequential cycle numbers across restarts and preventing
duplicate issues. Migrated agents include `system-watchdog`,
`implementation-pool-supervisor`, `timeline-update-pool-supervisor`,
`project-owner-pool-supervisor`, `product-builder`, and
`backlog-grooming-pool-supervisor`. The legacy `shared/automation_tracking.md` module
was removed.
#### Test Infrastructure
- **Documentation Writer Tracking** (`docs-writer`): The documentation writer now
participates in the automation tracking system by creating individual
`[AUTO-DOCS] Documentation Report (Cycle N)` issues every 10 cycles (~3.3 hours).
The manager applies the mandatory `Automation Tracking` label automatically, while
teams may add additional workflow labels as needed. See
`docs/development/automation-tracking.md` and the new
`docs/development/docs-writer.md` reference.
- **TDD Issue-Capture Test Activation** (#7025): Replaced 234 bare `@skip` tags
across 82 Behave feature files with the correct `@tdd_expected_fail @tdd_issue
@tdd_issue_<N>` tag system. Scenarios whose referenced bugs were already fixed
had `@tdd_expected_fail` removed and now run as permanent regression guards.
Net result: 629 features active in CI (up from ~545), zero `@skip` tags remain.
### Changed
@@ -143,14 +141,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
### Fixed
- **Plan Concurrency Race Condition** (#7989): Fixed critical race condition in `execute_plan()` and
`apply_plan()` where concurrent CLI/worker sessions could simultaneously modify the same plan,
corrupting plan state. `LockService` is now wired into the plan lifecycle with plan-level advisory
locking. Each invocation generates a unique caller identity (UUID) to prevent re-entrant lock
acquisition by concurrent sessions on the same plan. Concurrent attempts now raise `LockConflictError`
instead of silently racing. Lock is acquired before phase transition and released in a `finally`
block to ensure cleanup even on error.
- **Validation Gate Empty-Run Guard** (#7508): Fixed `ApplyValidationSummary.all_required_passed`
returning `True` when zero validations were run, silently bypassing the apply gate. The property
now returns `False` when the validation result set is empty (`is_empty` is `True`), ensuring
@@ -158,28 +148,23 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
`required_total` property for completeness. Updated `consolidated_validation.feature` scenarios
to reflect the corrected blocking behavior for empty summaries and no-attachment runs.
- **ACMS context tier hydration**: `ContextTierService` no longer starts empty
- **ACMS context tier hydration** (#1028): `ContextTierService` no longer starts empty
on every CLI invocation. A new `context_tier_hydrator.py` reads files from
linked project resources (via `git ls-files` or `os.walk`), creates
`TieredFragment` objects, and stores them in the tier service before context
assembly in `LLMExecuteActor.execute()`. The LLM now receives real file
context during plan execution. Respects max file size (256 KB), total budget
(10 MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__`
directory skipping. (#1028)
directory skipping.
- **Sandbox root wiring**: `_get_plan_executor()` now passes
- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types
(`detail_depth` and `relevance_score` must be strings, not int/float) that
caused Pydantic validation errors during context assembly, resulting in the
LLM receiving zero file context.
- **Sandbox root wiring** (#4222): `_get_plan_executor()` now passes
`sandbox_root=.cleveragents/sandbox/`, so LLM file output (`FILE:` blocks)
is written to disk during the execute phase. (#4222)
- **SubplanExecutionService fail_fast cancellation** (#7582): Fixed a race condition where
already-running parallel subplans were not cancelled when `fail_fast` fired. Previously,
`Future.cancel()` only prevented queued futures from starting but had no effect on
in-flight futures that completed after `stop_flag` was set — their `COMPLETE` results
were incorrectly included in the merge output. The fix adds a post-completion guard that
overrides any non-`ERRORED`/non-`CANCELLED` result to `CANCELLED` when `stop_flag` is
active, and clears the associated output to prevent it from entering the merge. Also
replaces the O(n) linear `status` lookup in the `as_completed()` loop with an O(1)
`status_map` dict pre-computed before the executor block.
is written to disk during the execute phase.
- **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the
`tdd_expected_fail_listener` `end_test()` function to prevent blindly inverting ALL test
+37 -11
View File
@@ -17,13 +17,37 @@ nox -s lint # Ruff linting
nox -s format # Ruff formatting
nox -s typecheck # Pyright type checking
nox -s unit_tests # Behave unit tests
nox -s integration_tests # Robot Framework integration tests
nox -s coverage_report # Coverage (must be >=97%)
nox -s security_scan # Bandit + Semgrep + Vulture security scanning
nox -s dead_code # Vulture dead code detection
nox -s complexity # Radon complexity analysis
nox -s pre_commit # Run all pre-commit hooks
nox -s adr_compliance # Verify ADR compliance
nox -s docs # Build MkDocs documentation
nox -s serve_docs # Serve docs locally (http://localhost:8000)
```
## Nox Session Reference
All available Nox sessions and their purpose:
| Session | Tool(s) | Purpose | Blocks CI |
|---|---|---|---|
| `format` | Ruff | Auto-format source code | Yes (diff check) |
| `lint` | Ruff | Lint with safe auto-fixes | Yes |
| `typecheck` | Pyright | Static type checking (`strict` mode) | Yes |
| `unit_tests` | Behave | BDD unit test suite | Yes |
| `integration_tests` | Robot Framework | End-to-end integration tests | Yes |
| `coverage_report` | Coverage.py | Measure test coverage (≥97% required) | Yes |
| `security_scan` | Bandit + Semgrep + Vulture | Security and dead-code scanning | Yes |
| `dead_code` | Vulture | Dead code detection (standalone) | No |
| `complexity` | Radon | Cyclomatic complexity analysis | Yes (grade F) |
| `pre_commit` | pre-commit | Run all pre-commit hooks | No |
| `adr_compliance` | Custom script | Verify ADR references are consistent | No |
| `docs` | MkDocs | Build documentation site | No |
| `serve_docs` | MkDocs | Serve docs locally for preview | No |
## Pre-commit Hooks
Pre-commit hooks run automatically on every `git commit`. They are configured in
@@ -71,17 +95,19 @@ The CI pipeline runs on **Forgejo Actions** (`.forgejo/workflows/ci.yml`).
### CI Jobs
| Job | Trigger | Purpose | Failure Impact |
| ----------- | ------- | ----------------------------------- | ---------------------- |
| `lint` | Push/PR | Ruff format + lint check | Blocks merge |
| `typecheck` | Push/PR | Pyright type checking | Blocks merge |
| `security` | Push/PR | Bandit + Semgrep + Vulture | Blocks merge |
| `quality` | Push/PR | Radon complexity check | Blocks merge (grade F) |
| `unit_tests`| Push/PR | Behave BDD tests (Python 3.13) | Blocks merge |
| `coverage` | Push/PR | Coverage measurement | Blocks merge (<97%) |
| `build` | Push/PR | Wheel build | Blocks release |
| `docker` | Push/PR | Docker image build + test | Blocks deployment |
| `helm` | Push/PR | Helm chart lint + template | Blocks deployment |
| Job | Trigger | Purpose | Failure Impact |
| ------------------- | ------- | ----------------------------------- | ---------------------- |
| `lint` | Push/PR | Ruff format + lint check | Blocks merge |
| `typecheck` | Push/PR | Pyright type checking | Blocks merge |
| `security` | Push/PR | Bandit + Semgrep + Vulture | Blocks merge |
| `quality` | Push/PR | Radon complexity check | Blocks merge (grade F) |
| `unit_tests` | Push/PR | Behave BDD tests (Python 3.13) | Blocks merge |
| `integration_tests` | Push/PR | Robot Framework e2e tests | Blocks merge |
| `coverage` | Push/PR | Coverage measurement | Blocks merge (<97%) |
| `adr_compliance` | Push/PR | ADR reference consistency check | Blocks merge |
| `build` | Push/PR | Wheel build | Blocks release |
| `docker` | Push/PR | Docker image build + test | Blocks deployment |
| `helm` | Push/PR | Helm chart lint + template | Blocks deployment |
### Nightly Quality
+229
View File
@@ -0,0 +1,229 @@
# Observability
This document describes the observability stack for CleverAgents Core: tracing,
structured logging, metrics, audit trail, and token/cost tracking.
---
## LangSmith Tracing
CleverAgents integrates with [LangSmith](https://smith.langchain.com/) for distributed
tracing of LangChain/LangGraph agent runs. Tracing is **off by default** and is enabled
by setting the environment variables below.
### Quick Setup
```bash
export CLEVERAGENTS_LANGSMITH_ENABLED=true
export CLEVERAGENTS_LANGSMITH_PROJECT=my-project
export CLEVERAGENTS_LANGSMITH_API_KEY=ls__...
```
The settings module automatically mirrors these values to the LangChain environment
variables (`LANGCHAIN_TRACING_V2`, `LANGCHAIN_PROJECT`, `LANGCHAIN_API_KEY`), so no
additional wiring is required.
### Environment Variables
| Variable | Required | Default | Description |
|---|---|---|---|
| `CLEVERAGENTS_LANGSMITH_ENABLED` | No | `false` | Set to `true` to enable LangSmith tracing |
| `CLEVERAGENTS_LANGSMITH_PROJECT` | When enabled | — | LangSmith project name (maps to `LANGCHAIN_PROJECT`) |
| `CLEVERAGENTS_LANGSMITH_API_KEY` | When enabled | — | LangSmith API key (maps to `LANGCHAIN_API_KEY`) |
| `CLEVERAGENTS_LANGSMITH_ENDPOINT` | No | `https://api.smith.langchain.com` | Custom LangSmith endpoint for self-hosted deployments |
| `CLEVERAGENTS_LANGSMITH_USER_ID` | No | — | Optional user identifier attached to all traces |
| `CLEVERAGENTS_LANGSMITH_TAGS` | No | — | Comma-separated tags attached to every run (e.g. `env:prod,team:core`) |
### Mirrored LangChain Variables
The settings module sets these automatically when LangSmith is enabled. You do **not**
need to set them manually:
| LangChain Variable | Source |
|---|---|
| `LANGCHAIN_TRACING_V2` | Set to `"true"` when `CLEVERAGENTS_LANGSMITH_ENABLED=true` |
| `LANGCHAIN_PROJECT` | Copied from `CLEVERAGENTS_LANGSMITH_PROJECT` |
| `LANGCHAIN_API_KEY` | Copied from `CLEVERAGENTS_LANGSMITH_API_KEY` |
| `LANGCHAIN_ENDPOINT` | Copied from `CLEVERAGENTS_LANGSMITH_ENDPOINT` (if set) |
### Verifying Tracing
```bash
# Check that the settings module sees your credentials
agents diagnostics
# Run a quick plan and look for trace URLs in the output
agents tell "Hello" --actor openai/gpt-4o
```
Traces appear in the LangSmith UI under the configured project within a few seconds of
each run.
---
## Structured Logging
CleverAgents uses [structlog](https://www.structlog.org/) for structured, machine-readable
logging throughout the application.
### Log Format
In development (TTY detected), logs are rendered as human-friendly colored key-value pairs:
```
2026-04-13 12:00:00 [info ] Plan execution started actor=openai/gpt-4o plan_id=01HZ...
```
In production (non-TTY / JSON mode), logs are emitted as newline-delimited JSON:
```json
{"timestamp": "2026-04-13T12:00:00Z", "level": "info", "event": "Plan execution started", "actor": "openai/gpt-4o", "plan_id": "01HZ..."}
```
### Log Level Configuration
| Variable | Default | Description |
|---|---|---|
| `CLEVERAGENTS_LOG_LEVEL` | `INFO` | Minimum log level (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`) |
| `CLEVERAGENTS_LOG_FORMAT` | `auto` | Force output format: `auto` (detect TTY), `json`, `console` |
### Standard Context Fields
Every log record emitted by CleverAgents includes these fields when available:
| Field | Description |
|---|---|
| `actor` | Active actor name (e.g. `openai/gpt-4o`) |
| `plan_id` | ULID of the current plan |
| `session_id` | Active session identifier |
| `provider` | LLM provider name |
| `component` | Source component (e.g. `plan_service`, `context_tier`) |
### Adding Structured Logs in Code
```python
import structlog
log = structlog.get_logger(__name__)
log.info("context_hydration_complete", file_count=42, budget_used_mb=3.7)
log.warning("provider_fallback", from_provider="openai", to_provider="anthropic")
log.error("plan_execution_failed", plan_id=plan.id, exc_info=True)
```
---
## Metrics and Audit Trail
### Audit Trail
CleverAgents records an immutable audit trail of all plan lifecycle events. Each event
captures:
- **Event type** — e.g. `PLAN_CREATED`, `PLAN_EXECUTED`, `PLAN_APPLIED`,
`INVARIANT_VIOLATED`, `CORRECTION_APPLIED`, `VALIDATION_FAILED`
- **Timestamp** — ISO 8601 UTC
- **Actor** — which actor/agent triggered the event
- **Plan ID** — ULID of the affected plan
- **Metadata** — event-specific payload (e.g. validation results, invariant name)
Audit events are written to the structlog pipeline (so they appear in both console and
JSON output) and optionally forwarded to LangSmith as run metadata when tracing is enabled.
### Metrics
Runtime metrics are collected via the internal `MetricsCollector` and exposed through:
- **Structured log fields** — counters and gauges are emitted as log fields on key events
- **`agents diagnostics`** — prints a summary of session-level metrics
Key metrics tracked:
| Metric | Description |
|---|---|
| `plans.created` | Total plans created in the session |
| `plans.executed` | Total plan executions |
| `plans.applied` | Total successful applies |
| `plans.failed` | Total failed executions |
| `validations.passed` | Validation checks that passed |
| `validations.failed` | Validation checks that failed |
| `invariants.violated` | Invariant violations detected |
| `corrections.applied` | Corrections applied to plans |
| `context.files_hydrated` | Files loaded into context tier |
| `context.budget_used_bytes` | Total context budget consumed |
---
## Token and Cost Tracking
CleverAgents tracks LLM token consumption and estimated cost for every plan execution.
### Per-Run Tracking
Each LLM call records:
- **Prompt tokens** — tokens in the input
- **Completion tokens** — tokens in the output
- **Total tokens** — sum of prompt + completion
- **Estimated cost (USD)** — calculated from the provider's published pricing
These values are:
1. Emitted as structured log fields on every LLM call completion
2. Aggregated into the plan's audit record
3. Forwarded to LangSmith as run metadata (when tracing is enabled)
### Session Totals
At the end of each session (or on `agents diagnostics`), cumulative totals are reported:
```
Session token usage:
Prompt tokens: 12,450
Completion tokens: 3,210
Total tokens: 15,660
Estimated cost: $0.0423 USD
```
### Cost Configuration
Pricing tables are bundled with the provider registry and updated with each release.
To override pricing for a custom deployment:
```bash
export CLEVERAGENTS_PROVIDER_COST_OVERRIDE='{"openai/gpt-4o": {"prompt": 0.000005, "completion": 0.000015}}'
```
The value is a JSON object mapping `provider/model` to per-token costs in USD.
---
## Environment Variable Reference
Complete list of observability-related environment variables:
| Variable | Default | Description |
|---|---|---|
| `CLEVERAGENTS_LANGSMITH_ENABLED` | `false` | Enable LangSmith tracing |
| `CLEVERAGENTS_LANGSMITH_PROJECT` | — | LangSmith project name |
| `CLEVERAGENTS_LANGSMITH_API_KEY` | — | LangSmith API key |
| `CLEVERAGENTS_LANGSMITH_ENDPOINT` | `https://api.smith.langchain.com` | LangSmith endpoint URL |
| `CLEVERAGENTS_LANGSMITH_USER_ID` | — | User ID attached to traces |
| `CLEVERAGENTS_LANGSMITH_TAGS` | — | Comma-separated run tags |
| `CLEVERAGENTS_LOG_LEVEL` | `INFO` | Minimum log level |
| `CLEVERAGENTS_LOG_FORMAT` | `auto` | Log output format (`auto`/`json`/`console`) |
| `CLEVERAGENTS_PROVIDER_COST_OVERRIDE` | — | JSON override for provider pricing |
---
## Diagnostics
```bash
# Print observability configuration and session metrics
agents diagnostics
# Check LangSmith connectivity (requires CLEVERAGENTS_LANGSMITH_ENABLED=true)
agents diagnostics --check-tracing
# Show token usage for the last N plans
agents plan list --show-tokens --limit 10
```