From b5dbd5f6a13860a2a5dab104bb22c7736d5e9d58 Mon Sep 17 00:00:00 2001 From: CleverThis Date: Fri, 10 Apr 2026 05:01:48 +0000 Subject: [PATCH] docs: update documentation for v3.9.0 features and recent merges - CHANGELOG.md: add entries for plan use UNIQUE constraint fix (#4197), validation attach named option format (#3837), container resource stop fix (#3250), agent system reorganization (18 supervisors, *-pool-supervisor naming), and PR review policy update (1 approval required) - README.md: add git worktree sandbox and ACMS context hydration highlights - docs/architecture.md: add Git Worktree Sandbox and ACMS Context Hydration sections with diagrams and cross-references - docs/modules/git-worktree-sandbox.md: new module guide for the git worktree sandbox execute/apply lifecycle (PR #5998) - docs/modules/context-tier-hydrator.md: new module guide for context tier hydration from project resources (PR #4219, fixes #1028) - docs/development/automation-tracking.md: add pr-fix-pool-supervisor and pr-merge-pool-supervisor to agent prefix table - mkdocs.yml: add git-worktree-sandbox and context-tier-hydrator to Modules nav ISSUES CLOSED: #6933 --- CHANGELOG.md | 85 +++++++----- README.md | 5 + docs/architecture.md | 45 +++++++ docs/modules/context-tier-hydrator.md | 179 ++++++++++++++++++++++++++ docs/modules/git-worktree-sandbox.md | 115 +++++++++++++++++ mkdocs.yml | 2 + 6 files changed, 397 insertions(+), 34 deletions(-) create mode 100644 docs/modules/context-tier-hydrator.md create mode 100644 docs/modules/git-worktree-sandbox.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 89d67e2c3..d3ed08286 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,17 +7,17 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added -- **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges - LLM-generated changes via `git merge` from an isolated worktree branch - instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary - (plan ID, artifacts, insertions/deletions, project, timestamp), Sandbox - Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall - back to the original flat file copy. +- **Git Worktree Sandbox Apply** (#4454, #2265): The `plan apply` command now merges + LLM-generated changes via `git merge` from an isolated worktree branch instead of flat + `shutil.copy2`. Displays spec-aligned Apply Summary (plan ID, artifacts, insertions/ + deletions, project, timestamp), Sandbox Cleanup panel, and `✓ OK Changes applied` footer. + Non-git projects fall back to the original flat file copy. `_handle_plan_execute` now + short-circuits repeat execute dispatches so plans already in execute/apply phase do not + raise "Invalid phase transition" errors. -- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types - (`detail_depth` and `relevance_score` must be strings, not int/float) that - caused Pydantic validation errors during context assembly, resulting in the - LLM receiving zero file context. +- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types (`detail_depth` + and `relevance_score` must be strings, not int/float) that caused Pydantic validation + errors during context assembly, resulting in the LLM receiving zero file context. - **Automation Tracking System**: Replaced shared session state issue tracking with individual per-agent tracking issues. Each agent now creates its own `[AUTO-]` @@ -28,8 +28,8 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Automated Health Monitoring and Recovery**: The `system-watchdog` now runs `audit_automation_tracking_health()` every 5 minutes, detecting stalled agents when - tracking issues are >20% overdue from their declared reporting interval. On detection, - it terminates stalled sessions via the OpenCode Server API, performs root-cause analysis, + tracking issues are >20% overdue from their declared reporting interval. On detection, it + terminates stalled sessions via the OpenCode Server API, performs root-cause analysis, creates high-priority diagnostic issues, and closes stale tracking issues with recovery notes. @@ -41,25 +41,23 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). subagent. - **PR–Issue Label Synchronization**: PRs now inherit `Priority/`, `MoSCoW/`, `Points/`, - and `State/` labels from their associated issues at creation time - (`pr-api-creator`). The `backlog-groomer` adds a continuous Pass 19 for ongoing - PR–issue label synchronization. The `issue-state-updater` syncs PR state labels whenever - issue states change. + and `State/` labels from their associated issues at creation time (`pr-api-creator`). The + `backlog-groomer` adds a continuous Pass 19 for ongoing PR–issue label synchronization. + The `issue-state-updater` syncs PR state labels whenever issue states change. - **Automation Tracking Announcements**: Extended `automation-tracking-manager` with announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`, - `LIST_TRACKING_ISSUES`, `READ_ANNOUNCEMENTS`, `REVIEW_OWN_ANNOUNCEMENTS`). Supervisors - and workers now read critical announcements before each cycle for cross-agent awareness. + `LIST_TRACKING_ISSUES`, `READ_ANNOUNCEMENTS`, `REVIEW_OWN_ANNOUNCEMENTS`). Supervisors and + workers now read critical announcements before each cycle for cross-agent awareness. Priority-based filtering (Critical/High/Medium/Low) reduces noise. Backlog-groomer performs intelligent cleanup with age thresholds by priority. -- **PR Agent Reorganization**: All PR-related agents renamed and reorganized to follow - the `*-pool-supervisor` naming pattern. New agents added: `pr-editor` (safe PR editing - with description preservation), `pr-manager` (unified PR interface), and - `pr-merge-pool-supervisor` (automated PR merging supervisor). Renamed: - `pr-api-creator` → `pr-creator`, `pr-checker` → `pr-ci-test-fixer`, - `pr-status-checker` → `pr-status-analyzer`, `pr-self-reviewer` → `pr-reviewer`, - `pr-fix-orchestrator` → `pr-fix-pool-supervisor`. +- **Agent System Reorganization**: All PR-related agents were consolidated under the + `*-pool-supervisor` naming standard, bringing the supervisor roster to 18. Agent renames + include `pr-api-creator` → `pr-creator`, `pr-checker` → `pr-ci-test-fixer`, + `pr-status-checker` → `pr-status-analyzer`, and `pr-self-reviewer` → `pr-reviewer`. New + agents `pr-editor`, `pr-manager`, and `pr-merge-pool-supervisor` provide safe editing, + unified PR management, and automated merge coordination respectively. - **Automated PR Merging** (`pr-merge-pool-supervisor`): New supervisor continuously monitors for merge-ready PRs and merges them automatically when all criteria are met @@ -69,10 +67,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Implementation Worker Workflow Completion**: `implementation-worker` now implements work claiming protocols with conflict detection, comprehensive review feedback handling with intelligent parsing, sophisticated merge conflict resolution with multiple - strategies, and parallel subtask execution with wave-based dependency analysis. - Pass rate improved from 48.15% to 84.8%. + strategies, and parallel subtask execution with wave-based dependency analysis. Pass rate + improved from 48.15% to 84.8%. -- **Container Resource Stop Support**: `agents resource stop` now correctly stops +- **Container Resource Stop Support** (#3250): `agents resource stop` now correctly stops `container-instance` and `devcontainer-instance` resource types. - **Centralized Automation Tracking Manager** (`automation-tracking-manager`): The manager @@ -125,6 +123,12 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). `AUTO-INF-POOL`, `AUTO-ARCH`, `AUTO-EPIC`, `AUTO-EVLV`, `AUTO-GUARD`, `AUTO-SPEC`, `AUTO-TIME`, `AUTO-PROJ-OWN`, and `AUTO-PROD-BLDR`. +- **ACMS Context Hydration** (#1028): Fixed the CLI wiring so `ContextTierService` no + longer starts empty during plan execution. Added `context_tier_hydrator.py` to read files + from project resources (`git ls-files` or `os.walk`) and populate tiered fragments prior + to context assembly, respecting the 256 KB per-file and 10 MB total limits while skipping + binary artefacts and common ignore directories. + ### Fixed - **Validation Gate Empty-Run Guard** (#7508): Fixed `ApplyValidationSummary.all_required_passed` @@ -149,11 +153,11 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the `tdd_expected_fail_listener` `end_test()` function to prevent blindly inverting ALL test - failures to passes, which was masking infrastructure errors and causing flaky CI behavior. - Guards: setup/teardown error detection, non-assertion failure detection (infrastructure - errors), and dry-run mode detection. Also fixed `Variable Should Exist` syntax errors in - e2e test files and removed `tdd_expected_fail` from 4 context assembly e2e tests where - bugs were already fixed. + failures to passes, which was masking infrastructure errors and causing flaky CI + behavior. Guards: setup/teardown error detection, non-assertion failure detection + (infrastructure errors), and dry-run mode detection. Also fixed `Variable Should Exist` + syntax errors in e2e test files and removed `tdd_expected_fail` from 4 context assembly + e2e tests where bugs were already fixed. - **`issue-state-updater` Bash Script Errors**: Removed problematic bash script examples that tried to invoke `task forgejo-label-manager` as a bash command (the Task tool cannot @@ -167,7 +171,20 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **`product-builder` Missing Supervisors**: Added missing `pr-fix-pool-supervisor` and `pr-merge-pool-supervisor` to the product-builder's supervisor launch list (18 total - supervisors). Updated all numeric references, pre-flight checklists, and validation logic. + supervisors). Updated all numeric references, pre-flight checklists, and validation + logic. + +- **`plan use` action arguments UNIQUE constraint** (#4174): Eliminated + `sqlite3.IntegrityError: UNIQUE constraint failed` by replacing the SQLAlchemy + `.clear()`/`.append()` pattern with explicit `sa_delete()` + `session.flush()` before + re-inserting child rows. Added `UniqueConstraint` and `CheckConstraint` to + `ActionArgumentModel` and `ActionInvariantModel` along with migration + `a5_006_action_invariants_unique_constraint.py`. + +- **`agents validation attach` named option format** (#3683): The `attach` subcommand now + accepts extra arguments as `--key value` named options (e.g. `--coverage-threshold 90`) + instead of positional `key=value` tokens, and rejects the old format with a descriptive + error. Hyphenated option names are normalized to underscores. --- diff --git a/README.md b/README.md index 733986eda..b382ac9fc 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,11 @@ embracing modern Python tooling. `validFrom`, and `isCurrent` metadata; a revision chain enables temporal queries - **JSON-RPC 2.0 A2A wire format** — `A2aRequest`/`A2aResponse` fields renamed to standard JSON-RPC 2.0 names (`method`, `id`, `result`, `error`) +- **Git worktree sandbox** — `plan apply` merges LLM-generated changes via `git merge` + from an isolated worktree branch; non-git projects fall back to flat file copy +- **ACMS context hydration** — `ContextTierService` is now populated automatically + before every plan execution via `context_tier_hydrator.py`; LLM receives real file + context from linked project resources instead of an empty context window - Fast Typer/Click-based interface with parity for help/version behavior - Behavior-driven coverage via Behave and Robot Framework - Nox automation for linting, typing, testing, docs, builds, and benchmarks diff --git a/docs/architecture.md b/docs/architecture.md index e355f0c5e..ffe48dd3a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -259,6 +259,51 @@ result = actor.run(plan_id="...", project_name="...", action_name="...") --- +## Git Worktree Sandbox + +The plan execute/apply lifecycle uses an isolated git worktree for file changes +(introduced in v3.9.0, PR #5998): + +``` +Execute phase: + ┌─────────────────────────────────────────────────────┐ + │ LLM generates FILE: blocks │ + │ → written to .cleveragents/sandbox/plan-/ │ + │ → committed on branch cleveragents/plan- │ + └─────────────────────────────────────────────────────┘ + +Apply phase: + ┌─────────────────────────────────────────────────────┐ + │ git merge cleveragents/plan- │ + │ → worktree removed │ + │ → Apply Summary + Sandbox Cleanup panels shown │ + └─────────────────────────────────────────────────────┘ +``` + +Non-git projects fall back to flat `shutil.copy2`. The A2A facade +`_handle_plan_execute` is idempotent — re-dispatching execute for a plan already +in execute/apply phase is a no-op (fixes #2265). + +See [`docs/modules/git-worktree-sandbox.md`](modules/git-worktree-sandbox.md) for +the full module guide. + +--- + +## ACMS Context Hydration + +The `context_tier_hydrator` module populates `ContextTierService` from linked +project resources before every plan execution (introduced in v3.9.0, fixes #1028): + +- For `git-checkout` resources: uses `git ls-files` to enumerate tracked files +- For other resource types: uses `os.walk` with skip-dir filtering +- Limits: 256 KB per file, 10 MB total; binary files and build directories skipped +- Hydration runs automatically in `LLMExecuteActor.execute()` before context assembly + +See [`docs/modules/context-tier-hydrator.md`](modules/context-tier-hydrator.md) for +the full module guide. + +--- + ## A2A Protocol Boundary All CLI, TUI, and server interactions cross the A2A boundary diff --git a/docs/modules/context-tier-hydrator.md b/docs/modules/context-tier-hydrator.md new file mode 100644 index 000000000..78489f914 --- /dev/null +++ b/docs/modules/context-tier-hydrator.md @@ -0,0 +1,179 @@ +# Context Tier Hydrator Module + +**Package:** `cleveragents.application.services.context_tier_hydrator` +**Introduced:** v3.9.0 (fixes #1028) + +The context tier hydrator bridges the gap between the resource registry (files on +disk) and the ACMS context tier (in-memory fragments). Without this module, the +`ContextTierService` starts empty on every CLI process invocation and the LLM +receives zero file context during plan execution. + +For the ACMS context tier architecture, see +[`docs/reference/context_tiers.md`](../reference/context_tiers.md). +For the full ACMS documentation, see +[`docs/reference/acms.md`](../reference/acms.md). + +--- + +## Purpose + +When a plan is executed, the LLM needs file context from the project being worked on. +The `ContextTierService` is an in-memory store that holds `TieredFragment` objects +representing file contents. However, because the CLI is a short-lived process, the +tier service is empty at startup. + +`context_tier_hydrator.py` solves this by reading files from linked project resources +(via `git ls-files` for git-checkout resources, or `os.walk` for other types) and +populating the tier service before context assembly begins. + +--- + +## Key Functions + +### `hydrate_tiers_from_project()` + +Reads files from a single resource and stores them as `TieredFragment` objects in +the `ContextTierService`. + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_from_project, +) + +fragments_stored = hydrate_tiers_from_project( + tier_service=tier_service, + project_name="local/my-project", + resource_id="01ABCDEF...", + resource_location="/path/to/project", + resource_type="git-checkout", # default +) +``` + +**Parameters:** + +| Parameter | Type | Description | +|-----------|------|-------------| +| `tier_service` | `ContextTierService` | The tier service to populate | +| `project_name` | `str` | Namespaced project name (e.g. `local/my-project`) | +| `resource_id` | `str` | ULID of the resource | +| `resource_location` | `str` | Filesystem path to the resource root | +| `resource_type` | `str` | Resource type — affects file listing strategy | + +**Returns:** Number of fragments stored. + +--- + +### `hydrate_tiers_for_plan()` + +Hydrates tiers for all projects linked to a plan. Called automatically by +`LLMExecuteActor.execute()` before context assembly. + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_for_plan, +) + +total_fragments = hydrate_tiers_for_plan( + tier_service=tier_service, + project_names=["local/my-project"], + project_repository=project_repository, + resource_registry=resource_registry, +) +``` + +--- + +## Limits and Filters + +The hydrator applies several guards to prevent excessive memory usage: + +| Limit | Value | Description | +|-------|-------|-------------| +| Max file size | 256 KB | Files larger than this are skipped | +| Max total bytes | 10 MB | Hydration stops when this budget is reached | +| Binary extensions | `.pyc`, `.so`, `.png`, `.pdf`, etc. | Binary files are skipped | +| Skip directories | `.git`, `__pycache__`, `node_modules`, `.venv`, etc. | These directories are never traversed | + +--- + +## File Listing Strategy + +For `git-checkout` resources, the hydrator uses `git ls-files` to enumerate tracked +files. This ensures only version-controlled files are indexed and respects +`.gitignore` rules. For other resource types, `os.walk` is used with the skip-dir +filter applied. + +--- + +## Fragment Metadata + +Each `TieredFragment` is stored in the `HOT` tier with the following metadata: + +```python +{ + "path": "relative/path/to/file.py", + "detail_depth": "1", # string, not int + "relevance_score": "0.5", # string, not float +} +``` + +> **Note:** `detail_depth` and `relevance_score` must be strings. Passing `int` or +> `float` values causes a Pydantic validation error in `ContextFragment` (fixed in +> PR #5998). + +--- + +## Integration Point + +`LLMExecuteActor.execute()` calls `hydrate_tiers_for_plan()` before invoking the +context assembler. The actor receives `tier_service`, `project_repository`, and +`resource_registry` via constructor injection — no `get_container()` calls are made +from within the service. + +```python +# Simplified from llm_actors.py +class LLMExecuteActor: + def __init__( + self, + tier_service: ContextTierService, + project_repository: NamespacedProjectRepository, + resource_registry: ResourceRegistryService, + ... + ): + self._tier_service = tier_service + self._project_repository = project_repository + self._resource_registry = resource_registry + + def execute(self, plan_id: str, ...) -> ExecuteResult: + # Hydrate before assembly + hydrate_tiers_for_plan( + self._tier_service, + project_names, + self._project_repository, + self._resource_registry, + ) + # Context assembly now has real file fragments + context = self._assembler.assemble(...) + ... +``` + +--- + +## Logging + +The hydrator emits structured log events: + +| Event | Level | Description | +|-------|-------|-------------| +| `context_hydrator.skip_missing_location` | WARNING | Resource location does not exist | +| `context_hydrator.hydrated` | INFO | Hydration complete — reports fragment count and bytes | +| `context_hydrator.project_not_found` | DEBUG | Project not found in repository | +| `context_hydrator.store_failed` | DEBUG | Failed to store a single fragment | + +--- + +## Related Fixes + +| Issue | Description | +|-------|-------------| +| #1028 | ACMS indexing pipeline not wired into CLI — ContextTierService started empty | diff --git a/docs/modules/git-worktree-sandbox.md b/docs/modules/git-worktree-sandbox.md new file mode 100644 index 000000000..359800f21 --- /dev/null +++ b/docs/modules/git-worktree-sandbox.md @@ -0,0 +1,115 @@ +# Git Worktree Sandbox Module + +**Package:** `cleveragents.infrastructure.sandbox.git_worktree` +**Introduced:** v3.9.0 (PR #5998, closes #4454, #2265) + +The git worktree sandbox implements spec-aligned isolated execution for the plan +execute/apply lifecycle. LLM-generated file changes are written to an isolated git +worktree branch and merged into the project's current branch during `plan apply`. + +For the plan lifecycle overview, see +[`docs/architecture.md`](../architecture.md#plan-lifecycle). +For the sandbox and checkpoint specification, see +[`docs/reference/sandbox.md`](../reference/sandbox.md). + +--- + +## Purpose + +Before this feature, `plan apply` used `shutil.copy2` to copy files from a flat +sandbox directory into the project. This approach: + +- Did not produce a clean git history +- Could not show meaningful diffs +- Did not integrate with git workflows + +The git worktree sandbox replaces this with a proper git-based workflow: + +1. **Execute phase** — creates an isolated git worktree for the plan's linked + `git-checkout` resource and commits LLM output to a dedicated branch + (`cleveragents/plan-`). +2. **Apply phase** — merges the worktree branch into the project's current branch + via `git merge`, then removes the worktree. + +Non-git projects fall back to the original flat directory sandbox with `shutil.copy2`. + +--- + +## Execute Phase + +When `agents plan execute` runs: + +1. The CLI detects whether the plan's linked resource is a `git-checkout` type. +2. If so, `GitWorktreeSandbox` creates a new worktree at a temporary path: + ``` + .cleveragents/sandbox/plan-/ + ``` +3. LLM file output (`FILE:` blocks) is written to the worktree directory. +4. Changes are committed on branch `cleveragents/plan-` — no merge yet. + +--- + +## Apply Phase + +When `agents plan apply` runs: + +1. The CLI merges the worktree branch into the project's current branch: + ``` + git merge cleveragents/plan- + ``` +2. The worktree is removed after a successful merge. +3. Spec-aligned output panels are displayed: + + | Panel | Contents | + |-------|----------| + | **Apply Summary** | Plan ID, artifact count, insertions/deletions, project name, applied-at timestamp | + | **Sandbox Cleanup** | Worktree removed, branch merged to main | + | **Next Steps** | Review `git diff`, commit changes | + | **Footer** | `✓ OK Changes applied` | + +--- + +## Idempotent Execute Dispatch + +The A2A facade `_handle_plan_execute` now checks whether the plan has already +reached the execute or apply phase before attempting a phase transition. This +eliminates the noisy `"Invalid phase transition from execute to execute"` error +that occurred when the execute command was dispatched more than once for the same +plan. (#2265) + +--- + +## Fallback Behaviour + +If the plan's linked resource is **not** a `git-checkout` type (e.g. a plain +directory or database resource), the sandbox falls back to the original flat file +copy using `shutil.copy2`. The apply panels are still displayed but the Sandbox +Cleanup panel notes that a flat copy was used. + +--- + +## Usage + +```bash +# Execute the plan (writes to worktree branch) +agents plan execute --plan-id + +# Apply the plan (merges worktree branch into current branch) +agents plan apply --plan-id +``` + +After apply, review the merged changes: + +```bash +git log --oneline -5 +git diff HEAD~1 +``` + +--- + +## Related Fixes + +| Issue | Description | +|-------|-------------| +| #4454 | Context assembly failure — `ContextFragment` metadata values must be strings | +| #2265 | Duplicate execute dispatch — idempotent phase transition check | diff --git a/mkdocs.yml b/mkdocs.yml index 2bb05974c..17d59b0ea 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,8 @@ nav: - Shell Safety: modules/shell-safety.md - UKO Provenance Tracking: modules/uko-provenance.md - Invariant Reconciliation: modules/invariant-reconciliation.md + - Git Worktree Sandbox: modules/git-worktree-sandbox.md + - Context Tier Hydrator: modules/context-tier-hydrator.md - Development: - Agent System Specification: development/agent-system-specification.md - CI/CD Pipeline: development/ci-cd.md -- 2.52.0