diff --git a/CHANGELOG.md b/CHANGELOG.md index 0cf791e12..7ea75f08b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,35 +5,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] -### Fixed - -- **Automation Profile Silent Fallback** (#8232): `_resolve_profile_for_plan` in - `PlanLifecycleService` now raises a clear `ValidationError` when a plan's - automation profile name is not a known built-in profile, instead of silently - falling back to `"manual"`. Users who configured custom automation profiles - (e.g. `"semi-auto"`, `"acme/strict"`) will now receive an actionable error - message listing available built-in profiles. The resolved profile name is also - logged at debug level for observability. + ### Added -- **TDD Issue-Capture Test Activation** (#7025): Replaced 234 bare `@skip` tags - across 82 Behave feature files with the correct `@tdd_expected_fail @tdd_issue - @tdd_issue_` tag system. Scenarios whose referenced bugs were already fixed - had `@tdd_expected_fail` removed and now run as permanent regression guards. - Net result: 629 features active in CI (up from ~545), zero `@skip` tags remain. - -- **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges - LLM-generated changes via `git merge` from an isolated worktree branch - instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary - (plan ID, artifacts, insertions/deletions, project, timestamp), Sandbox - Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall - back to the original flat file copy. - -- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types - (`detail_depth` and `relevance_score` must be strings, not int/float) that - caused Pydantic validation errors during context assembly, resulting in the - LLM receiving zero file context. +#### Agent System & Automation - **Automation Tracking System**: Replaced shared session state issue tracking with individual per-agent tracking issues. Each agent now creates its own `[AUTO-]` @@ -49,6 +28,34 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). creates high-priority diagnostic issues, and closes stale tracking issues with recovery notes. +- **Centralized Automation Tracking Manager** (`automation-tracking-manager`): The manager + is now the single interface for all tracking issue operations (`CREATE_TRACKING_ISSUE`, + `UPDATE_TRACKING_ISSUE`, `CLOSE_TRACKING_ISSUE`, `READ_TRACKING_STATE`, + `GET_NEXT_CYCLE_NUMBER`). Agents delegate to the manager rather than calling the Forgejo + API directly, ensuring sequential cycle numbers across restarts and preventing + duplicate issues. Migrated agents include `system-watchdog`, + `implementation-pool-supervisor`, `timeline-update-pool-supervisor`, + `project-owner-pool-supervisor`, `product-builder`, and + `backlog-grooming-pool-supervisor`. The legacy `shared/automation_tracking.md` module + was removed. + +- **Automation Tracking Announcements**: Extended `automation-tracking-manager` with + announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`, + `LIST_TRACKING_ISSUES`, `READ_ANNOUNCEMENTS`, `REVIEW_OWN_ANNOUNCEMENTS`). Supervisors + and workers now read critical announcements before each cycle for cross-agent awareness. + Priority-based filtering (Critical/High/Medium/Low) reduces noise. Backlog-groomer + performs intelligent cleanup with age thresholds by priority. + +- **Documentation Writer Tracking** (`docs-writer`): The documentation writer now + participates in the automation tracking system by creating individual + `[AUTO-DOCS] Documentation Report (Cycle N)` issues every 10 cycles (~3.3 hours). + The manager applies the mandatory `Automation Tracking` label automatically, while + teams may add additional workflow labels as needed. See + `docs/development/automation-tracking.md` and the new + `docs/development/docs-writer.md` reference. + +#### Label & PR Management + - **Centralized Label Management** (`forgejo-label-manager`): A new specialized subagent centralizes all Forgejo label operations across the agent system. Enforces the organization-level label system, prohibits label creation, and validates label compliance. @@ -62,13 +69,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). PR–issue label synchronization. The `issue-state-updater` syncs PR state labels whenever issue states change. -- **Automation Tracking Announcements**: Extended `automation-tracking-manager` with - announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`, - `LIST_TRACKING_ISSUES`, `READ_ANNOUNCEMENTS`, `REVIEW_OWN_ANNOUNCEMENTS`). Supervisors - and workers now read critical announcements before each cycle for cross-agent awareness. - Priority-based filtering (Critical/High/Medium/Low) reduces noise. Backlog-groomer - performs intelligent cleanup with age thresholds by priority. - - **PR Agent Reorganization**: All PR-related agents renamed and reorganized to follow the `*-pool-supervisor` naming pattern. New agents added: `pr-editor` (safe PR editing with description preservation), `pr-manager` (unified PR interface), and @@ -82,6 +82,15 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). (approvals, CI passing, no conflicts). Supports both formal reviews and comment-based approvals (LGTM, ✅, "ready to merge", etc.). +#### Plan Lifecycle & Implementation + +- **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges + LLM-generated changes via `git merge` from an isolated worktree branch + instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary + (plan ID, artifacts, insertions/deletions, project, timestamp), Sandbox + Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall + back to the original flat file copy. + - **Implementation Worker Workflow Completion**: `implementation-worker` now implements work claiming protocols with conflict detection, comprehensive review feedback handling with intelligent parsing, sophisticated merge conflict resolution with multiple @@ -91,24 +100,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Container Resource Stop Support**: `agents resource stop` now correctly stops `container-instance` and `devcontainer-instance` resource types. -- **Centralized Automation Tracking Manager** (`automation-tracking-manager`): The manager - is now the single interface for all tracking issue operations (`CREATE_TRACKING_ISSUE`, - `UPDATE_TRACKING_ISSUE`, `CLOSE_TRACKING_ISSUE`, `READ_TRACKING_STATE`, - `GET_NEXT_CYCLE_NUMBER`). Agents delegate to the manager rather than calling the Forgejo - API directly, ensuring sequential cycle numbers across restarts and preventing - duplicate issues. Migrated agents include `system-watchdog`, - `implementation-pool-supervisor`, `timeline-update-pool-supervisor`, - `project-owner-pool-supervisor`, `product-builder`, and - `backlog-grooming-pool-supervisor`. The legacy `shared/automation_tracking.md` module - was removed. +#### Test Infrastructure -- **Documentation Writer Tracking** (`docs-writer`): The documentation writer now - participates in the automation tracking system by creating individual - `[AUTO-DOCS] Documentation Report (Cycle N)` issues every 10 cycles (~3.3 hours). - The manager applies the mandatory `Automation Tracking` label automatically, while - teams may add additional workflow labels as needed. See - `docs/development/automation-tracking.md` and the new - `docs/development/docs-writer.md` reference. +- **TDD Issue-Capture Test Activation** (#7025): Replaced 234 bare `@skip` tags + across 82 Behave feature files with the correct `@tdd_expected_fail @tdd_issue + @tdd_issue_` tag system. Scenarios whose referenced bugs were already fixed + had `@tdd_expected_fail` removed and now run as permanent regression guards. + Net result: 629 features active in CI (up from ~545), zero `@skip` tags remain. ### Changed @@ -143,14 +141,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed -- **Plan Concurrency Race Condition** (#7989): Fixed critical race condition in `execute_plan()` and - `apply_plan()` where concurrent CLI/worker sessions could simultaneously modify the same plan, - corrupting plan state. `LockService` is now wired into the plan lifecycle with plan-level advisory - locking. Each invocation generates a unique caller identity (UUID) to prevent re-entrant lock - acquisition by concurrent sessions on the same plan. Concurrent attempts now raise `LockConflictError` - instead of silently racing. Lock is acquired before phase transition and released in a `finally` - block to ensure cleanup even on error. - - **Validation Gate Empty-Run Guard** (#7508): Fixed `ApplyValidationSummary.all_required_passed` returning `True` when zero validations were run, silently bypassing the apply gate. The property now returns `False` when the validation result set is empty (`is_empty` is `True`), ensuring @@ -158,28 +148,23 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). `required_total` property for completeness. Updated `consolidated_validation.feature` scenarios to reflect the corrected blocking behavior for empty summaries and no-attachment runs. -- **ACMS context tier hydration**: `ContextTierService` no longer starts empty +- **ACMS context tier hydration** (#1028): `ContextTierService` no longer starts empty on every CLI invocation. A new `context_tier_hydrator.py` reads files from linked project resources (via `git ls-files` or `os.walk`), creates `TieredFragment` objects, and stores them in the tier service before context assembly in `LLMExecuteActor.execute()`. The LLM now receives real file context during plan execution. Respects max file size (256 KB), total budget (10 MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__` - directory skipping. (#1028) + directory skipping. -- **Sandbox root wiring**: `_get_plan_executor()` now passes +- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types + (`detail_depth` and `relevance_score` must be strings, not int/float) that + caused Pydantic validation errors during context assembly, resulting in the + LLM receiving zero file context. + +- **Sandbox root wiring** (#4222): `_get_plan_executor()` now passes `sandbox_root=.cleveragents/sandbox/`, so LLM file output (`FILE:` blocks) - is written to disk during the execute phase. (#4222) - -- **SubplanExecutionService fail_fast cancellation** (#7582): Fixed a race condition where - already-running parallel subplans were not cancelled when `fail_fast` fired. Previously, - `Future.cancel()` only prevented queued futures from starting but had no effect on - in-flight futures that completed after `stop_flag` was set — their `COMPLETE` results - were incorrectly included in the merge output. The fix adds a post-completion guard that - overrides any non-`ERRORED`/non-`CANCELLED` result to `CANCELLED` when `stop_flag` is - active, and clears the associated output to prevent it from entering the merge. Also - replaces the O(n) linear `status` lookup in the `as_completed()` loop with an O(1) - `status_map` dict pre-computed before the executor block. + is written to disk during the execute phase. - **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the `tdd_expected_fail_listener` `end_test()` function to prevent blindly inverting ALL test diff --git a/docs/development/quality-automation.md b/docs/development/quality-automation.md index 93371d2b5..0c2109f75 100644 --- a/docs/development/quality-automation.md +++ b/docs/development/quality-automation.md @@ -17,13 +17,37 @@ nox -s lint # Ruff linting nox -s format # Ruff formatting nox -s typecheck # Pyright type checking nox -s unit_tests # Behave unit tests +nox -s integration_tests # Robot Framework integration tests nox -s coverage_report # Coverage (must be >=97%) nox -s security_scan # Bandit + Semgrep + Vulture security scanning nox -s dead_code # Vulture dead code detection nox -s complexity # Radon complexity analysis nox -s pre_commit # Run all pre-commit hooks +nox -s adr_compliance # Verify ADR compliance +nox -s docs # Build MkDocs documentation +nox -s serve_docs # Serve docs locally (http://localhost:8000) ``` +## Nox Session Reference + +All available Nox sessions and their purpose: + +| Session | Tool(s) | Purpose | Blocks CI | +|---|---|---|---| +| `format` | Ruff | Auto-format source code | Yes (diff check) | +| `lint` | Ruff | Lint with safe auto-fixes | Yes | +| `typecheck` | Pyright | Static type checking (`strict` mode) | Yes | +| `unit_tests` | Behave | BDD unit test suite | Yes | +| `integration_tests` | Robot Framework | End-to-end integration tests | Yes | +| `coverage_report` | Coverage.py | Measure test coverage (≥97% required) | Yes | +| `security_scan` | Bandit + Semgrep + Vulture | Security and dead-code scanning | Yes | +| `dead_code` | Vulture | Dead code detection (standalone) | No | +| `complexity` | Radon | Cyclomatic complexity analysis | Yes (grade F) | +| `pre_commit` | pre-commit | Run all pre-commit hooks | No | +| `adr_compliance` | Custom script | Verify ADR references are consistent | No | +| `docs` | MkDocs | Build documentation site | No | +| `serve_docs` | MkDocs | Serve docs locally for preview | No | + ## Pre-commit Hooks Pre-commit hooks run automatically on every `git commit`. They are configured in @@ -71,17 +95,19 @@ The CI pipeline runs on **Forgejo Actions** (`.forgejo/workflows/ci.yml`). ### CI Jobs -| Job | Trigger | Purpose | Failure Impact | -| ----------- | ------- | ----------------------------------- | ---------------------- | -| `lint` | Push/PR | Ruff format + lint check | Blocks merge | -| `typecheck` | Push/PR | Pyright type checking | Blocks merge | -| `security` | Push/PR | Bandit + Semgrep + Vulture | Blocks merge | -| `quality` | Push/PR | Radon complexity check | Blocks merge (grade F) | -| `unit_tests`| Push/PR | Behave BDD tests (Python 3.13) | Blocks merge | -| `coverage` | Push/PR | Coverage measurement | Blocks merge (<97%) | -| `build` | Push/PR | Wheel build | Blocks release | -| `docker` | Push/PR | Docker image build + test | Blocks deployment | -| `helm` | Push/PR | Helm chart lint + template | Blocks deployment | +| Job | Trigger | Purpose | Failure Impact | +| ------------------- | ------- | ----------------------------------- | ---------------------- | +| `lint` | Push/PR | Ruff format + lint check | Blocks merge | +| `typecheck` | Push/PR | Pyright type checking | Blocks merge | +| `security` | Push/PR | Bandit + Semgrep + Vulture | Blocks merge | +| `quality` | Push/PR | Radon complexity check | Blocks merge (grade F) | +| `unit_tests` | Push/PR | Behave BDD tests (Python 3.13) | Blocks merge | +| `integration_tests` | Push/PR | Robot Framework e2e tests | Blocks merge | +| `coverage` | Push/PR | Coverage measurement | Blocks merge (<97%) | +| `adr_compliance` | Push/PR | ADR reference consistency check | Blocks merge | +| `build` | Push/PR | Wheel build | Blocks release | +| `docker` | Push/PR | Docker image build + test | Blocks deployment | +| `helm` | Push/PR | Helm chart lint + template | Blocks deployment | ### Nightly Quality diff --git a/docs/observability.md b/docs/observability.md new file mode 100644 index 000000000..4416a34f7 --- /dev/null +++ b/docs/observability.md @@ -0,0 +1,229 @@ +# Observability + +This document describes the observability stack for CleverAgents Core: tracing, +structured logging, metrics, audit trail, and token/cost tracking. + +--- + +## LangSmith Tracing + +CleverAgents integrates with [LangSmith](https://smith.langchain.com/) for distributed +tracing of LangChain/LangGraph agent runs. Tracing is **off by default** and is enabled +by setting the environment variables below. + +### Quick Setup + +```bash +export CLEVERAGENTS_LANGSMITH_ENABLED=true +export CLEVERAGENTS_LANGSMITH_PROJECT=my-project +export CLEVERAGENTS_LANGSMITH_API_KEY=ls__... +``` + +The settings module automatically mirrors these values to the LangChain environment +variables (`LANGCHAIN_TRACING_V2`, `LANGCHAIN_PROJECT`, `LANGCHAIN_API_KEY`), so no +additional wiring is required. + +### Environment Variables + +| Variable | Required | Default | Description | +|---|---|---|---| +| `CLEVERAGENTS_LANGSMITH_ENABLED` | No | `false` | Set to `true` to enable LangSmith tracing | +| `CLEVERAGENTS_LANGSMITH_PROJECT` | When enabled | — | LangSmith project name (maps to `LANGCHAIN_PROJECT`) | +| `CLEVERAGENTS_LANGSMITH_API_KEY` | When enabled | — | LangSmith API key (maps to `LANGCHAIN_API_KEY`) | +| `CLEVERAGENTS_LANGSMITH_ENDPOINT` | No | `https://api.smith.langchain.com` | Custom LangSmith endpoint for self-hosted deployments | +| `CLEVERAGENTS_LANGSMITH_USER_ID` | No | — | Optional user identifier attached to all traces | +| `CLEVERAGENTS_LANGSMITH_TAGS` | No | — | Comma-separated tags attached to every run (e.g. `env:prod,team:core`) | + +### Mirrored LangChain Variables + +The settings module sets these automatically when LangSmith is enabled. You do **not** +need to set them manually: + +| LangChain Variable | Source | +|---|---| +| `LANGCHAIN_TRACING_V2` | Set to `"true"` when `CLEVERAGENTS_LANGSMITH_ENABLED=true` | +| `LANGCHAIN_PROJECT` | Copied from `CLEVERAGENTS_LANGSMITH_PROJECT` | +| `LANGCHAIN_API_KEY` | Copied from `CLEVERAGENTS_LANGSMITH_API_KEY` | +| `LANGCHAIN_ENDPOINT` | Copied from `CLEVERAGENTS_LANGSMITH_ENDPOINT` (if set) | + +### Verifying Tracing + +```bash +# Check that the settings module sees your credentials +agents diagnostics + +# Run a quick plan and look for trace URLs in the output +agents tell "Hello" --actor openai/gpt-4o +``` + +Traces appear in the LangSmith UI under the configured project within a few seconds of +each run. + +--- + +## Structured Logging + +CleverAgents uses [structlog](https://www.structlog.org/) for structured, machine-readable +logging throughout the application. + +### Log Format + +In development (TTY detected), logs are rendered as human-friendly colored key-value pairs: + +``` +2026-04-13 12:00:00 [info ] Plan execution started actor=openai/gpt-4o plan_id=01HZ... +``` + +In production (non-TTY / JSON mode), logs are emitted as newline-delimited JSON: + +```json +{"timestamp": "2026-04-13T12:00:00Z", "level": "info", "event": "Plan execution started", "actor": "openai/gpt-4o", "plan_id": "01HZ..."} +``` + +### Log Level Configuration + +| Variable | Default | Description | +|---|---|---| +| `CLEVERAGENTS_LOG_LEVEL` | `INFO` | Minimum log level (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`) | +| `CLEVERAGENTS_LOG_FORMAT` | `auto` | Force output format: `auto` (detect TTY), `json`, `console` | + +### Standard Context Fields + +Every log record emitted by CleverAgents includes these fields when available: + +| Field | Description | +|---|---| +| `actor` | Active actor name (e.g. `openai/gpt-4o`) | +| `plan_id` | ULID of the current plan | +| `session_id` | Active session identifier | +| `provider` | LLM provider name | +| `component` | Source component (e.g. `plan_service`, `context_tier`) | + +### Adding Structured Logs in Code + +```python +import structlog + +log = structlog.get_logger(__name__) + +log.info("context_hydration_complete", file_count=42, budget_used_mb=3.7) +log.warning("provider_fallback", from_provider="openai", to_provider="anthropic") +log.error("plan_execution_failed", plan_id=plan.id, exc_info=True) +``` + +--- + +## Metrics and Audit Trail + +### Audit Trail + +CleverAgents records an immutable audit trail of all plan lifecycle events. Each event +captures: + +- **Event type** — e.g. `PLAN_CREATED`, `PLAN_EXECUTED`, `PLAN_APPLIED`, + `INVARIANT_VIOLATED`, `CORRECTION_APPLIED`, `VALIDATION_FAILED` +- **Timestamp** — ISO 8601 UTC +- **Actor** — which actor/agent triggered the event +- **Plan ID** — ULID of the affected plan +- **Metadata** — event-specific payload (e.g. validation results, invariant name) + +Audit events are written to the structlog pipeline (so they appear in both console and +JSON output) and optionally forwarded to LangSmith as run metadata when tracing is enabled. + +### Metrics + +Runtime metrics are collected via the internal `MetricsCollector` and exposed through: + +- **Structured log fields** — counters and gauges are emitted as log fields on key events +- **`agents diagnostics`** — prints a summary of session-level metrics + +Key metrics tracked: + +| Metric | Description | +|---|---| +| `plans.created` | Total plans created in the session | +| `plans.executed` | Total plan executions | +| `plans.applied` | Total successful applies | +| `plans.failed` | Total failed executions | +| `validations.passed` | Validation checks that passed | +| `validations.failed` | Validation checks that failed | +| `invariants.violated` | Invariant violations detected | +| `corrections.applied` | Corrections applied to plans | +| `context.files_hydrated` | Files loaded into context tier | +| `context.budget_used_bytes` | Total context budget consumed | + +--- + +## Token and Cost Tracking + +CleverAgents tracks LLM token consumption and estimated cost for every plan execution. + +### Per-Run Tracking + +Each LLM call records: + +- **Prompt tokens** — tokens in the input +- **Completion tokens** — tokens in the output +- **Total tokens** — sum of prompt + completion +- **Estimated cost (USD)** — calculated from the provider's published pricing + +These values are: +1. Emitted as structured log fields on every LLM call completion +2. Aggregated into the plan's audit record +3. Forwarded to LangSmith as run metadata (when tracing is enabled) + +### Session Totals + +At the end of each session (or on `agents diagnostics`), cumulative totals are reported: + +``` +Session token usage: + Prompt tokens: 12,450 + Completion tokens: 3,210 + Total tokens: 15,660 + Estimated cost: $0.0423 USD +``` + +### Cost Configuration + +Pricing tables are bundled with the provider registry and updated with each release. +To override pricing for a custom deployment: + +```bash +export CLEVERAGENTS_PROVIDER_COST_OVERRIDE='{"openai/gpt-4o": {"prompt": 0.000005, "completion": 0.000015}}' +``` + +The value is a JSON object mapping `provider/model` to per-token costs in USD. + +--- + +## Environment Variable Reference + +Complete list of observability-related environment variables: + +| Variable | Default | Description | +|---|---|---| +| `CLEVERAGENTS_LANGSMITH_ENABLED` | `false` | Enable LangSmith tracing | +| `CLEVERAGENTS_LANGSMITH_PROJECT` | — | LangSmith project name | +| `CLEVERAGENTS_LANGSMITH_API_KEY` | — | LangSmith API key | +| `CLEVERAGENTS_LANGSMITH_ENDPOINT` | `https://api.smith.langchain.com` | LangSmith endpoint URL | +| `CLEVERAGENTS_LANGSMITH_USER_ID` | — | User ID attached to traces | +| `CLEVERAGENTS_LANGSMITH_TAGS` | — | Comma-separated run tags | +| `CLEVERAGENTS_LOG_LEVEL` | `INFO` | Minimum log level | +| `CLEVERAGENTS_LOG_FORMAT` | `auto` | Log output format (`auto`/`json`/`console`) | +| `CLEVERAGENTS_PROVIDER_COST_OVERRIDE` | — | JSON override for provider pricing | + +--- + +## Diagnostics + +```bash +# Print observability configuration and session metrics +agents diagnostics + +# Check LangSmith connectivity (requires CLEVERAGENTS_LANGSMITH_ENABLED=true) +agents diagnostics --check-tracing + +# Show token usage for the last N plans +agents plan list --show-tokens --limit 10 +```