docs: add git worktree sandbox and context tier hydrator module guides

- Add docs/modules/git-worktree-sandbox.md: full lifecycle guide for GitWorktreeSandbox including state machine, method reference, error table, and CLI Apply Summary panel documentation (PR #5998) - Add docs/modules/context-tier-hydrator.md: module guide for ContextTierHydrator covering public API, limits/filters, fragment structure, and structured log events (PR #4219, bug #1028) - Update mkdocs.yml nav to include both new module pages - Update README.md highlights with git worktree sandbox, ACMS context tier hydration, and container/devcontainer resource stop features - Update CHANGELOG.md [Unreleased] section with entries for PRs #5998, #4219, #3250, #4197, #4175, #3837
2026-04-10 04:31:12 +00:00
parent 4029a3331a
commit b299da392f
5 changed files with 518 additions and 2 deletions
@@ -7,6 +7,33 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

 ### Added

+- **Git Worktree Sandbox — Full Execute + Apply** (#5998): The plan execute
+  and apply phases now use an isolated git worktree sandbox for all
+  `git-checkout` resources.  Changes are staged on a dedicated
+  `cleveragents/plan-<id>` branch and merged back to the original branch on
+  `plan apply`.  Non-git resources fall back to the copy-on-write sandbox.
+  Includes atomic rollback support via pre-merge commit tracking, a
+  spec-aligned Apply Summary panel (plan ID, artifacts, insertions/deletions,
+  project, timestamp), Sandbox Cleanup panel, and `✓ OK Changes applied`
+  footer.  See [`docs/modules/git-worktree-sandbox.md`](docs/modules/git-worktree-sandbox.md).
+
+- **ACMS Context Tier Hydration** (#4219): Fixed a critical bug (#1028) where
+  `ContextTierService` started empty on every CLI invocation, causing the LLM
+  to receive zero file context during plan execution.  A new
+  `ContextTierHydrator` reads files from linked project resources (via
+  `git ls-files` for git-checkout resources or `os.walk` for fs-directory
+  resources) and stores them as `TieredFragment` objects in the tier service.
+  Hydration runs automatically before context assembly in
+  `LLMExecuteActor.execute()`.  Respects max file size (256 KB), total budget
+  (10 MB), binary file exclusion, and skip-directory rules.
+  See [`docs/modules/context-tier-hydrator.md`](docs/modules/context-tier-hydrator.md).
+
+- **Container/Devcontainer Resource Stop** (#3250): `agents resource stop` now
+  correctly stops `container-instance` and `devcontainer-instance` resource
+  types in addition to the previously supported types.  The resource handler
+  dispatch table was extended to route stop requests to the container and
+  devcontainer lifecycle handlers.
+
 - **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges
  LLM-generated changes via `git merge` from an isolated worktree branch
  instead of flat `shutil.copy2`.  Displays spec-aligned Apply Summary
@@ -72,8 +99,25 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
  strategies, and parallel subtask execution with wave-based dependency analysis.
  Pass rate improved from 48.15% to 84.8%.

- **Container Resource Stop Support**: `agents resource stop` now correctly stops
-  `container-instance` and `devcontainer-instance` resource types.
+- **Container Resource Stop Support** (#3250): `agents resource stop` now correctly stops
+  `container-instance` and `devcontainer-instance` resource types in addition to the
+  previously supported types. The resource handler dispatch table now routes stop requests
+  to the container and devcontainer lifecycle handlers.
+
+### Fixed
+
+- **Plan action argument upsert** (#4197): Fixed a `UNIQUE constraint violation`
+  that occurred when `plan use` attempted to insert action arguments that already
+  existed in the database.  The repository layer now uses an upsert (insert-or-update)
+  strategy so re-using a plan with the same action arguments succeeds idempotently.
+
+- **CI quality tests** (#4175): Restored all CI quality test sessions to passing
+  state after lint rule changes introduced regressions.  Removed stale
+  `@tdd_expected_fail` tags from Behave scenarios that had been fixed.
+
+- **Validation attach CLI format** (#3837): `agents validation attach` extra
+  arguments now use `--key value` named option format instead of positional
+  arguments, matching the spec and other CLI commands.

 ### Changed

@@ -41,6 +41,14 @@ embracing modern Python tooling.
  `validFrom`, and `isCurrent` metadata; a revision chain enables temporal queries
 - **JSON-RPC 2.0 A2A wire format** — `A2aRequest`/`A2aResponse` fields renamed to
  standard JSON-RPC 2.0 names (`method`, `id`, `result`, `error`)
+- **Git worktree sandbox** — plan execute and apply phases use an isolated git worktree
+  for `git-checkout` resources; changes are committed on a dedicated branch and merged
+  back on `plan apply`; atomic rollback via pre-merge commit tracking
+- **ACMS context tier hydration** — `ContextTierHydrator` reads files from linked project
+  resources at plan execution time so the LLM always receives file context; supports
+  `git ls-files` for git resources and `os.walk` for directory resources
+- **Container/devcontainer resource stop** — `agents resource stop` now supports
+  `container-instance` and `devcontainer-instance` resource types
 - Fast Typer/Click-based interface with parity for help/version behavior
 - Behavior-driven coverage via Behave and Robot Framework
 - Nox automation for linting, typing, testing, docs, builds, and benchmarks
@@ -0,0 +1,189 @@
+# Context Tier Hydrator Module
+
+**Package:** `cleveragents.application.services.context_tier_hydrator`
+**Introduced:** v3.9.0 (PR #4219, fixes bug #1028)
+
+The Context Tier Hydrator bridges the gap between the resource registry (files
+on disk) and the ACMS `ContextTierService` (in-memory fragments).  Without
+hydration, `ContextTierService` starts empty on every CLI process invocation,
+causing the LLM to receive zero file context during plan execution.
+
+For the ACMS context tier architecture, see
+[ADR-014](../adr/ADR-014-context-management-acms.md).
+For the `ContextTierService` API, see [`docs/api/core.md`](../api/core.md).
+
+---
+
+## Problem Solved
+
+The ACMS three-tier context system (`HOT`, `WARM`, `COLD`) stores file fragments
+in memory via `ContextTierService`.  In a long-running server process, these
+fragments accumulate over time.  However, the CLI spawns a fresh process for
+every invocation, so the tier service always starts empty.
+
+Before this module, `LLMExecuteActor.execute()` would assemble context from an
+empty tier service, sending the LLM a prompt with no file content.  The hydrator
+fixes this by reading files from linked project resources at the start of each
+plan execution.
+
+---
+
+## Automatic Invocation
+
+`LLMExecuteActor.execute()` calls `hydrate_tiers_for_plan()` automatically
+before context assembly.  No manual wiring is required.
+
+---
+
+## Public API
+
+### `hydrate_tiers_from_project()`
+
+```python
+from cleveragents.application.services.context_tier_hydrator import (
+    hydrate_tiers_from_project,
+)
+
+count = hydrate_tiers_from_project(
+    tier_service=tier_service,
+    project_name="local/my-project",
+    resource_id="01HXYZ...",
+    resource_location="/home/user/my-project",
+    resource_type="git-checkout",   # optional, default "git-checkout"
+)
+print(f"Stored {count} fragments")
+```
+
+Reads files from a single resource directory and stores them as
+`TieredFragment` objects in the `ContextTierService`.
+
+**Parameters:**
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `tier_service` | `ContextTierService` | The tier service to populate |
+| `project_name` | `str` | Namespaced project name (e.g. `local/my-project`) |
+| `resource_id` | `str` | ULID of the resource |
+| `resource_location` | `str` | Filesystem path to the resource root |
+| `resource_type` | `str` | Resource type name — affects file listing strategy |
+
+**Returns:** Number of fragments stored.
+
+**File listing strategy:**
+
+- `git-checkout` / `git` resources: uses `git ls-files --cached --others
+  --exclude-standard` to list tracked and untracked (non-ignored) files
+- All other resource types: uses `os.walk` with skip-directory filtering
+
+### `hydrate_tiers_for_plan()`
+
+```python
+from cleveragents.application.services.context_tier_hydrator import (
+    hydrate_tiers_for_plan,
+)
+
+total = hydrate_tiers_for_plan(
+    tier_service=tier_service,
+    project_names=["local/my-project", "local/other-project"],
+    project_repository=project_repo,
+    resource_registry=resource_registry_service,
+)
+print(f"Stored {total} fragments across all projects")
+```
+
+Hydrates tiers for all projects linked to a plan.  Iterates over each project,
+resolves its linked resources, and calls `hydrate_tiers_from_project()` for
+each resource that has a filesystem location.
+
+**Parameters:**
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `tier_service` | `ContextTierService` | The tier service to populate |
+| `project_names` | `list[str]` | Namespaced project names |
+| `project_repository` | `NamespacedProjectRepository` | Repository for project lookup |
+| `resource_registry` | `ResourceRegistryService` | Registry for resource lookup |
+
+**Returns:** Total number of fragments stored across all projects.
+
+---
+
+## Limits and Filters
+
+The hydrator applies several limits to prevent memory exhaustion:
+
+| Limit | Value | Description |
+|-------|-------|-------------|
+| Max file size | 256 KB | Files larger than this are skipped |
+| Max total bytes | 10 MB | Hydration stops when this budget is reached |
+| Binary extensions | `.pyc`, `.so`, `.png`, `.pdf`, etc. | Skipped entirely |
+| Skip directories | `.git`, `__pycache__`, `node_modules`, `.venv`, etc. | Not traversed |
+
+### Skipped Directories
+
+The following directories are never traversed:
+
+```
+.git  .hg  .svn  __pycache__  node_modules  .venv  venv  .nox  .tox
+.mypy_cache  .pytest_cache  .ruff_cache  dist  build  .eggs  .cleveragents
+```
+
+Additionally, any directory starting with `.` or ending with `.egg-info` is
+skipped during `os.walk` traversal.
+
+### Skipped File Extensions
+
+```
+.pyc  .pyo  .so  .o  .a  .dll  .exe
+.png  .jpg  .jpeg  .gif  .bmp  .ico  .pdf
+.zip  .tar  .gz  .bz2  .xz  .whl  .egg
+.db  .sqlite  .sqlite3
+```
+
+---
+
+## Fragment Structure
+
+Each file is stored as a `TieredFragment` in the `HOT` tier:
+
+```python
+TieredFragment(
+    fragment_id=f"{resource_id}:{rel_path}",  # e.g. "01HXYZ...:src/main.py"
+    content=file_content,
+    tier=ContextTier.HOT,
+    resource_id=resource_id,
+    project_name=project_name,
+    token_count=len(content) // 4,            # approximate token count
+    metadata={
+        "path": rel_path,
+        "detail_depth": "1",
+        "relevance_score": "0.5",
+    },
+)
+```
+
+> **Note:** `detail_depth` and `relevance_score` are stored as strings (not
+> int/float) to satisfy `ContextFragment` Pydantic validation requirements.
+> This was a source of the original bug (#1028).
+
+---
+
+## Logging
+
+The hydrator emits structured log events via `structlog`:
+
+| Event | Level | Description |
+|-------|-------|-------------|
+| `context_hydrator.skip_missing_location` | WARNING | Resource location does not exist on disk |
+| `context_hydrator.project_not_found` | DEBUG | Project not found in repository |
+| `context_hydrator.resource_not_found` | DEBUG | Resource not found in registry |
+| `context_hydrator.store_failed` | DEBUG | Failed to store a fragment (non-fatal) |
+| `context_hydrator.hydrated` | INFO | Hydration complete for a resource |
+
+---
+
+## See Also
+
+- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md) — Three-tier context architecture
+- [`docs/api/core.md`](../api/core.md) — `ContextTierService` API reference
+- [`docs/modules/invariant-reconciliation.md`](invariant-reconciliation.md) — Related plan lifecycle module
@@ -0,0 +1,273 @@
+# Git Worktree Sandbox Module
+
+**Package:** `cleveragents.infrastructure.sandbox.git_worktree`
+**Introduced:** v3.9.0 (PR #5998)
+
+The Git Worktree Sandbox provides fully isolated, git-native sandboxing for
+`git-checkout` resources during plan execution and apply phases.  Changes are
+staged on a dedicated branch in a temporary git worktree and merged back to the
+original branch only on a successful `plan apply`.  If the plan is rolled back,
+the merge is undone atomically.
+
+For the sandbox protocol and other sandbox strategies, see
+[`docs/development/custom_sandbox_strategy.md`](../development/custom_sandbox_strategy.md).
+For the plan lifecycle that invokes sandboxes, see
+[ADR-006](../adr/ADR-006-plan-lifecycle.md) and
+[ADR-015](../adr/ADR-015-sandbox-and-checkpoint.md).
+
+---
+
+## Purpose
+
+When an actor executes a plan against a `git-checkout` resource, it must not
+write changes directly to the working tree — doing so would corrupt the
+repository state if the plan fails or is rolled back.  The Git Worktree Sandbox
+solves this by:
+
+1. Creating a temporary git worktree at a new branch (`cleveragents/plan-<id>`)
+2. Routing all actor file writes into the worktree
+3. Committing changes in the worktree on `plan apply`
+4. Merging the sandbox branch back to the original branch
+5. Cleaning up the worktree and branch after the merge
+
+Non-git resources (e.g. `fs-directory`) continue to use the copy-on-write
+sandbox strategy.
+
+---
+
+## Lifecycle
+
+```
+GitWorktreeSandbox(resource_id, original_path)
+    │
+    ▼
+sandbox.create(plan_id)          # PENDING → CREATED
+    │  Creates worktree at /tmp/ca-sandbox-<plan_id>-XXXX
+    │  Creates branch cleveragents/plan-<plan_id>
+    │
+    ▼
+sandbox.get_path("src/foo.py")   # CREATED/ACTIVE → ACTIVE
+    │  Returns /tmp/ca-sandbox-.../src/foo.py
+    │  Actor writes to this path
+    │
+    ▼
+sandbox.commit("msg")            # ACTIVE → COMMITTED
+    │  git add -A  (in worktree)
+    │  git commit  (in worktree)
+    │  git merge cleveragents/plan-<id>  (in original repo)
+    │
+    ▼
+sandbox.cleanup()                # → CLEANED_UP
+       git worktree remove --force
+       git branch -D cleveragents/plan-<id>
+       git worktree prune
+```
+
+Rollback is possible from `ACTIVE` or `COMMITTED`:
+
+```
+sandbox.rollback()               # ACTIVE/COMMITTED → ROLLED_BACK
+    │  If COMMITTED: git reset --hard <pre_merge_commit>  (original repo)
+    │  git reset --hard <base_commit>  (worktree)
+    │  git clean -fd              (worktree)
+```
+
+---
+
+## Key Class: `GitWorktreeSandbox`
+
+```python
+from cleveragents.infrastructure.sandbox.git_worktree import GitWorktreeSandbox
+
+sandbox = GitWorktreeSandbox(
+    resource_id="01HXYZ...",
+    original_path="/home/user/my-project",
+    git_timeout=30,          # optional, default 30 s
+)
+```
+
+### Constructor Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `resource_id` | `str` | required | ULID of the resource being sandboxed |
+| `original_path` | `str` | required | Absolute path to the git repository root |
+| `git_timeout` | `int` | `30` | Timeout in seconds for each git command |
+
+### Methods
+
+#### `create(plan_id: str) → SandboxContext`
+
+Creates the worktree and sandbox branch.  Must be called before any other
+method.
+
+- Creates branch `cleveragents/plan-<sanitised_plan_id>` from `HEAD`
+- Creates a temporary directory at `/tmp/ca-sandbox-<plan_id>-XXXX`
+- Returns a `SandboxContext` with `sandbox_path` pointing to the worktree
+
+**Raises:**
+- `ValueError` — if `plan_id` is empty
+- `SandboxStateError` — if not in `PENDING` status
+- `SandboxCreationError` — if git worktree creation fails (e.g. not a git repo,
+  git command timeout)
+
+#### `get_path(resource_path: str) → str`
+
+Translates a resource-relative path to its absolute path inside the worktree.
+
+```python
+worktree_path = sandbox.get_path("src/cleveragents/cli/main.py")
+# → "/tmp/ca-sandbox-abc123-XXXX/src/cleveragents/cli/main.py"
+```
+
+**Raises:**
+- `SandboxStateError` — if sandbox is not in a usable status
+- `ValueError` — if `resource_path` contains `..` (directory traversal)
+
+#### `commit(message: str | None = None) → CommitResult`
+
+Stages all changes in the worktree, commits them, and merges the sandbox branch
+into the original branch.
+
+```python
+result = sandbox.commit("feat: add new feature")
+print(result.commit_ref)      # git SHA of the sandbox commit
+print(result.changed_files)   # list of modified files
+print(result.added_files)     # list of new files
+print(result.deleted_files)   # list of deleted files
+```
+
+If there are no staged changes, returns a `CommitResult` with `success=True`
+and empty file lists (no-op commit).
+
+**Raises:**
+- `SandboxStateError` — if not in `CREATED` or `ACTIVE` status
+- `SandboxCommitError` — if the git commit or merge fails
+
+#### `rollback() → None`
+
+Discards all worktree changes.
+
+- From `ACTIVE`: resets the worktree branch to the base commit
+- From `COMMITTED`: additionally resets the original branch to the pre-merge
+  commit, fully undoing the merge
+
+> **Warning:** Rolling back from `COMMITTED` executes `git reset --hard` on
+> the original branch.  If other git worktrees or external processes are
+> tracking the same branch, they will be affected.
+
+**Raises:**
+- `SandboxStateError` — if called in an invalid status
+- `SandboxRollbackError` — if the rollback fails
+
+#### `cleanup() → None`
+
+Removes the worktree directory and deletes the sandbox branch.  Idempotent —
+safe to call multiple times.
+
+Internally runs:
+1. `git worktree remove --force <worktree_path>`
+2. `git branch -D cleveragents/plan-<id>`
+3. `git worktree prune`
+
+Falls back to `shutil.rmtree` if `git worktree remove` fails.
+
+---
+
+## `SandboxContext` Metadata
+
+After `create()`, the `sandbox.context.metadata` dict contains:
+
+| Key | Description |
+|-----|-------------|
+| `strategy` | Always `"git_worktree"` |
+| `branch` | Sandbox branch name, e.g. `cleveragents/plan-abc123` |
+| `original_branch` | Branch that was active before sandboxing |
+| `base_commit` | SHA of the HEAD commit at sandbox creation time |
+| `worktree_path` | Absolute path to the temporary worktree directory |
+
+---
+
+## Status State Machine
+
+```
+PENDING
+  │ create()
+  ▼
+CREATED ──── get_path() ──► ACTIVE
+  │                           │
+  │ commit()                  │ commit()
+  ▼                           ▼
+COMMITTED ◄─────────────── COMMITTED
+  │                           │
+  │ rollback()                │ rollback()
+  ▼                           ▼
+ROLLED_BACK              ROLLED_BACK
+  │                           │
+  │ cleanup()                 │ cleanup()
+  ▼                           ▼
+CLEANED_UP               CLEANED_UP
+
+Any state ──► ERRORED (on git command failure)
+```
+
+---
+
+## Error Reference
+
+| Exception | When raised |
+|-----------|-------------|
+| `SandboxCreationError` | `create()` fails (not a git repo, git timeout, etc.) |
+| `SandboxCommitError` | `commit()` fails (git commit or merge error) |
+| `SandboxRollbackError` | `rollback()` fails (git reset error) |
+| `SandboxStateError` | Method called in wrong lifecycle state |
+
+All exceptions are importable from `cleveragents.infrastructure.sandbox.protocol`.
+
+---
+
+## Integration with Plan Lifecycle
+
+The `SandboxFactory` selects `GitWorktreeSandbox` automatically when the
+resource type is `git-checkout` and the sandbox strategy is `git_worktree`.
+The `PlanApplyService` calls `sandbox.commit()` and then `sandbox.cleanup()` on
+success, or `sandbox.rollback()` followed by `sandbox.cleanup()` on failure.
+
+The CLI `plan apply` command displays a spec-aligned Apply Summary panel:
+
+```
+╭─ Apply Summary ──────────────────────────────────────────────────╮
+│ Plan ID   : 01HXYZ...                                            │
+│ Project   : local/my-project                                     │
+│ Artifacts : 3 changed, 1 added, 0 deleted                        │
+│ Timestamp : 2026-04-09T12:34:56                                  │
+╰──────────────────────────────────────────────────────────────────╯
+╭─ Sandbox Cleanup ────────────────────────────────────────────────╮
+│ Worktree removed · Branch deleted · Prune complete               │
+╰──────────────────────────────────────────────────────────────────╯
+✓ OK Changes applied
+```
+
+---
+
+## Configuration
+
+The git command timeout defaults to 30 seconds.  Override it by passing
+`git_timeout` to the constructor, or configure it via the sandbox strategy
+registry if using the factory.
+
+```python
+sandbox = GitWorktreeSandbox(
+    resource_id=resource.id,
+    original_path=resource.location,
+    git_timeout=60,   # increase for large repos on slow storage
+)
+```
+
+---
+
+## See Also
+
+- [`docs/development/custom_sandbox_strategy.md`](../development/custom_sandbox_strategy.md) — How to implement a custom sandbox strategy
+- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) — Design rationale
+- [ADR-038 Cross-Mechanism Sandbox Coordination](../adr/ADR-038-cross-mechanism-sandbox-coordination.md) — Multi-sandbox coordination
@@ -27,6 +27,8 @@ nav:
    - Shell Safety: modules/shell-safety.md
    - UKO Provenance Tracking: modules/uko-provenance.md
    - Invariant Reconciliation: modules/invariant-reconciliation.md
+    - Git Worktree Sandbox: modules/git-worktree-sandbox.md
+    - Context Tier Hydrator: modules/context-tier-hydrator.md
  - Development:
    - Agent System Specification: development/agent-system-specification.md
    - CI/CD Pipeline: development/ci-cd.md