docs(modules): add git worktree sandbox and context hydration docs

ISSUES CLOSED: #6841

docs/modules/context-tier-hydration.md

@@ -0,0 +1,235 @@
+# Context Tier Hydration
+
+The **Context Tier Hydrator** bridges the gap between the resource registry (files on
+disk) and the ACMS `ContextTierService` (in-memory fragments).  Without it, the tier
+service starts empty on every CLI process invocation and the LLM receives zero file
+context during plan execution.
+
+Introduced in PR #5998 (merged 2026-04-09), fixing bug #1028.
+
+---
+
+## Problem
+
+The Advanced Context Management System (ACMS) stores file content as `TieredFragment`
+objects in `ContextTierService`.  This service is in-memory and is not persisted
+between CLI invocations.  When a plan is executed, the LLM actor calls the context
+assembler to build a context window from the tier service — but the service is empty
+because no files have been loaded.
+
+The hydrator solves this by reading files from the project's linked resources and
+storing them as `TieredFragment` objects before context assembly begins.
+
+---
+
+## Architecture
+
+```
+CLI: agents plan execute
+        │
+        ▼
+LLMExecuteActor.execute()
+        │
+        ├─── hydrate_tiers_for_plan()   ← Context Tier Hydrator
+        │         │
+        │         ├── project_repository.get(project_name)
+        │         ├── resource_registry.show_resource(resource_id)
+        │         └── hydrate_tiers_from_project()
+        │                   │
+        │                   ├── _list_files()
+        │                   │       ├── git ls-files (git-checkout resources)
+        │                   │       └── os.walk (other resource types)
+        │                   └── tier_service.store(TieredFragment)
+        │
+        └─── context_assembler.assemble()   ← reads from tier_service
+```
+
+---
+
+## Key Functions
+
+**Module**: `cleveragents.application.services.context_tier_hydrator`
+
+### `hydrate_tiers_for_plan`
+
+```python
+def hydrate_tiers_for_plan(
+    tier_service: ContextTierService,
+    project_names: list[str],
+    project_repository: Any,
+    resource_registry: Any,
+) -> int:
+```
+
+Top-level entry point.  Iterates over all projects linked to a plan, resolves their
+linked resources, and calls `hydrate_tiers_from_project` for each.
+
+**Parameters**:
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `tier_service` | `ContextTierService` | The tier service to populate |
+| `project_names` | `list[str]` | Namespaced project names (e.g. `local/my-project`) |
+| `project_repository` | `NamespacedProjectRepository` | Repository for project lookups |
+| `resource_registry` | `ResourceRegistryService` | Registry for resource lookups |
+
+**Returns**: Total number of `TieredFragment` objects stored across all projects.
+
+---
+
+### `hydrate_tiers_from_project`
+
+```python
+def hydrate_tiers_from_project(
+    tier_service: ContextTierService,
+    project_name: str,
+    resource_id: str,
+    resource_location: str,
+    resource_type: str = "git-checkout",
+) -> int:
+```
+
+Reads files from a single resource directory and stores them as `TieredFragment`
+objects in the tier service.
+
+**Parameters**:
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `tier_service` | `ContextTierService` | — | The tier service to populate |
+| `project_name` | `str` | — | Namespaced project name |
+| `resource_id` | `str` | — | ULID of the resource |
+| `resource_location` | `str` | — | Filesystem path to the resource root |
+| `resource_type` | `str` | `"git-checkout"` | Resource type (affects file listing strategy) |
+
+**Returns**: Number of fragments stored.
+
+**Limits enforced**:
+
+| Limit | Value | Description |
+|-------|-------|-------------|
+| Max file size | 256 KB | Files larger than this are skipped |
+| Max total bytes | 10 MB | Hydration stops when this budget is reached |
+
+---
+
+## File Listing Strategy
+
+The hydrator uses two strategies to enumerate files, selected by resource type:
+
+### `git ls-files` (git-checkout resources)
+
+For `git-checkout` and `git` resource types, the hydrator runs:
+
+```bash
+git ls-files --cached --others --exclude-standard
+```
+
+This lists tracked files and untracked files that are not gitignored.  Binary
+extensions (`.pyc`, `.so`, `.png`, etc.) are filtered out.
+
+Falls back to `os.walk` if the git command fails or times out (30 s timeout).
+
+### `os.walk` (other resource types)
+
+For non-git resources, the hydrator walks the directory tree, skipping:
+
+**Directories**: `.git`, `.hg`, `.svn`, `__pycache__`, `node_modules`, `.venv`,
+`venv`, `.nox`, `.tox`, `.mypy_cache`, `.pytest_cache`, `.ruff_cache`, `dist`,
+`build`, `.eggs`, `.cleveragents`, and any directory starting with `.` or ending
+with `.egg-info`.
+
+**File extensions**: `.pyc`, `.pyo`, `.so`, `.o`, `.a`, `.dll`, `.exe`, `.png`,
+`.jpg`, `.jpeg`, `.gif`, `.bmp`, `.ico`, `.pdf`, `.zip`, `.tar`, `.gz`, `.bz2`,
+`.xz`, `.whl`, `.egg`, `.db`, `.sqlite`, `.sqlite3`.
+
+**Hidden files**: Files starting with `.` are skipped.
+
+---
+
+## Fragment Format
+
+Each file is stored as a `TieredFragment` with:
+
+| Field | Value |
+|-------|-------|
+| `fragment_id` | `<resource_id>:<relative_path>` |
+| `content` | UTF-8 file content |
+| `tier` | `ContextTier.HOT` |
+| `resource_id` | Resource ULID |
+| `project_name` | Namespaced project name |
+| `token_count` | `len(content) // 4` (rough estimate) |
+| `metadata.path` | Relative path within the resource |
+| `metadata.detail_depth` | `"1"` (string, not int) |
+| `metadata.relevance_score` | `"0.5"` (string, not float) |
+
+!!! note "Metadata types"
+    `detail_depth` and `relevance_score` must be **strings**, not `int` or `float`.
+    Pydantic v2 validation on `ContextFragment` enforces this.  Passing numeric
+    values causes a `ValidationError` that silently drops all context from the LLM
+    call.  This was the root cause of bug #1028.
+
+---
+
+## Usage Example
+
+The hydrator is called automatically by `LLMExecuteActor.execute()` before context
+assembly.  You can also call it directly for testing or custom integrations:
+
+```python
+from cleveragents.application.services.context_tier_hydrator import (
+    hydrate_tiers_from_project,
+)
+from cleveragents.application.services.context_tiers import ContextTierService
+
+tier_service = ContextTierService()
+
+count = hydrate_tiers_from_project(
+    tier_service=tier_service,
+    project_name="local/my-project",
+    resource_id="01ABCDEF...",
+    resource_location="/path/to/repo",
+    resource_type="git-checkout",
+)
+print(f"Stored {count} fragments")
+```
+
+---
+
+## Logging
+
+The hydrator emits structured log events via `structlog`:
+
+| Event | Level | Fields |
+|-------|-------|--------|
+| `context_hydrator.skip_missing_location` | WARNING | `project`, `resource_id`, `location` |
+| `context_hydrator.store_failed` | DEBUG | `fragment_id`, `exc_info` |
+| `context_hydrator.hydrated` | INFO | `project`, `resource_id`, `fragments_stored`, `total_bytes` |
+| `context_hydrator.project_not_found` | DEBUG | `project` |
+| `context_hydrator.no_linked_resources` | DEBUG | `project` |
+| `context_hydrator.resource_not_found` | DEBUG | `resource_id` |
+
+---
+
+## Gotchas
+
+1. **Binary files are silently skipped**.  If a file cannot be decoded as UTF-8, it
+   is skipped without error.
+
+2. **The 10 MB budget is per-hydration call**, not per-project.  If a project has
+   many large files, the budget may be exhausted before all files are indexed.
+
+3. **`git ls-files` requires a 30-second timeout**.  On very large repositories or
+   slow disks, the command may time out and fall back to `os.walk`.
+
+4. **Fragments are stored in the HOT tier**.  All files are treated as equally
+   relevant.  Future versions may use relevance scoring to assign files to
+   HOT/WARM/COLD tiers.
+
+---
+
+## Related
+
+- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md)
+- [Git Worktree Sandbox](git-worktree-sandbox.md)
+- [API Reference — ACMS](../api/core.md)

docs/modules/git-worktree-sandbox.md

@@ -0,0 +1,244 @@
+# Git Worktree Sandbox
+
+The **Git Worktree Sandbox** provides isolated, reversible staging for LLM-generated
+file changes during plan execution.  It uses `git worktree` to create a detached
+working tree on a dedicated branch, so modifications never touch the original branch
+until the plan is explicitly applied.
+
+Introduced in PR #5998 (merged 2026-04-09) as part of the M1 plan lifecycle
+implementation.
+
+---
+
+## Overview
+
+When a plan is executed against a `git-checkout` resource, CleverAgents creates a
+temporary git worktree on a branch named `cleveragents/plan-<plan_id>`.  The LLM
+actor writes files into this worktree.  On `plan apply`, the sandbox branch is merged
+back into the project's current branch via `git merge`.  If anything goes wrong, the
+sandbox can be rolled back without affecting the original working tree.
+
+Non-git projects fall back to the flat directory sandbox (`shutil.copy2`).
+
+---
+
+## Lifecycle
+
+```
+create(plan_id)
+    │  Creates branch cleveragents/plan-<plan_id>
+    │  Creates worktree at /tmp/ca-sandbox-<plan_id>-<random>/
+    ▼
+get_path(resource_path)
+    │  Translates resource-relative paths to worktree paths
+    │  Actor writes files here
+    ▼
+commit(message)
+    │  git add -A  →  git commit  →  git merge <branch> (in original repo)
+    ▼
+cleanup()
+    │  git worktree remove --force
+    │  git branch -D cleveragents/plan-<plan_id>
+    │  git worktree prune
+    ▼
+  CLEANED_UP
+```
+
+Rollback is available from `ACTIVE` or `COMMITTED` states:
+
+```
+rollback()
+    │  ACTIVE   → git reset --hard <base_commit> + git clean -fd (in worktree)
+    │  COMMITTED → git reset --hard <pre_merge_commit> (in original repo)
+    │              + git reset --hard <base_commit> (in worktree)
+    ▼
+  ROLLED_BACK
+```
+
+---
+
+## Status State Machine
+
+| Status | Meaning |
+|--------|---------|
+| `PENDING` | Sandbox created but `create()` not yet called |
+| `CREATED` | Worktree and branch exist; no writes yet |
+| `ACTIVE` | Actor has written at least one file |
+| `COMMITTED` | Changes committed and merged to original branch |
+| `ROLLED_BACK` | Changes discarded; worktree reset to base commit |
+| `ERRORED` | A git command failed; sandbox is unusable |
+| `CLEANED_UP` | Worktree and branch removed |
+
+---
+
+## Key Class: `GitWorktreeSandbox`
+
+**Module**: `cleveragents.infrastructure.sandbox.git_worktree`
+
+```python
+from cleveragents.infrastructure.sandbox.git_worktree import GitWorktreeSandbox
+
+sandbox = GitWorktreeSandbox(
+    resource_id="01ABCDEF...",   # ULID of the git-checkout resource
+    original_path="/path/to/repo",
+    git_timeout=30,              # seconds per git command (default: 30)
+)
+
+# Create the worktree
+ctx = sandbox.create(plan_id="01PLAN...")
+print(ctx.sandbox_path)          # /tmp/ca-sandbox-01PLAN-xxxx/
+print(ctx.metadata["branch"])    # cleveragents/plan-01PLAN...
+
+# Resolve a path inside the worktree
+worktree_file = sandbox.get_path("src/calculator.py")
+
+# After the actor writes files...
+result = sandbox.commit("feat: implement calculator")
+print(result.commit_ref)         # git SHA of the sandbox commit
+print(result.added_files)        # ["src/calculator.py"]
+
+# Cleanup
+sandbox.cleanup()
+```
+
+### Constructor Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `resource_id` | `str` | — | ULID of the resource being sandboxed |
+| `original_path` | `str` | — | Absolute path to the git repository root |
+| `git_timeout` | `int` | `30` | Timeout in seconds for each git command |
+
+### Methods
+
+#### `create(plan_id: str) → SandboxContext`
+
+Creates the worktree and branch.  Must be called before any other method.
+
+- Verifies `original_path` is a git repository root (not a subdirectory).
+- Creates branch `cleveragents/plan-<sanitised_plan_id>`.
+- Creates a temporary directory for the worktree.
+- Returns a `SandboxContext` with `sandbox_path`, `original_path`, `resource_id`,
+  `plan_id`, `created_at`, and `metadata` (strategy, branch, original_branch,
+  base_commit, worktree_path).
+
+#### `get_path(resource_path: str) → str`
+
+Translates a resource-relative path to an absolute path inside the worktree.
+Raises `ValueError` on directory traversal attempts (`..` components).
+
+#### `commit(message: str | None = None) → CommitResult`
+
+Stages all changes (`git add -A`), commits them in the worktree, then merges
+the sandbox branch into the original branch (`git merge --no-edit`).
+
+Returns a `CommitResult` with:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `sandbox_id` | `str` | Sandbox ULID |
+| `success` | `bool` | `True` on success |
+| `commit_ref` | `str` | SHA of the sandbox commit |
+| `changed_files` | `list[str]` | Modified files |
+| `added_files` | `list[str]` | New files |
+| `deleted_files` | `list[str]` | Removed files |
+| `error` | `str \| None` | Error message on failure |
+| `timestamp` | `datetime` | Commit timestamp |
+
+If there are no staged changes, returns a `CommitResult` with `success=True`
+and empty file lists (no-op commit).
+
+#### `rollback() → None`
+
+Discards all changes.  Behaviour depends on current status:
+
+- **`ACTIVE`**: Resets the worktree branch to `base_commit` and runs `git clean -fd`.
+- **`COMMITTED`**: Also resets the original branch to `pre_merge_commit` (undoes the merge).
+
+!!! warning "Multi-worktree safety"
+    Rolling back from `COMMITTED` executes `git reset --hard` on the original
+    branch.  Other git worktrees or external processes tracking the same branch
+    will be affected.
+
+#### `cleanup() → None`
+
+Removes the worktree and deletes the sandbox branch.  Idempotent — safe to call
+multiple times.  Falls back to `shutil.rmtree` if `git worktree remove` fails.
+
+---
+
+## Error Types
+
+All errors are defined in `cleveragents.infrastructure.sandbox.protocol`.
+
+| Exception | When raised |
+|-----------|-------------|
+| `SandboxCreationError` | `git worktree add` or prerequisite git commands fail |
+| `SandboxCommitError` | `git commit` or `git merge` fails |
+| `SandboxRollbackError` | `git reset` or `git clean` fails |
+| `SandboxStateError` | Method called in an invalid status |
+
+---
+
+## Branch Naming
+
+Branch names are sanitised to be git-safe:
+
+- Only alphanumeric characters, hyphens, underscores, slashes, and dots are kept.
+- Runs of disallowed characters are collapsed into a single hyphen.
+- The final name is `cleveragents/plan-<sanitised_plan_id>`.
+
+Example: plan ID `01JQABC-xyz!foo` → branch `cleveragents/plan-01JQABC-xyz-foo`.
+
+---
+
+## Integration with Plan Lifecycle
+
+The `GitWorktreeSandbox` is instantiated by the CLI `plan execute` command when the
+plan's linked resource is a `git-checkout` type.  The `plan apply` command calls
+`commit()` followed by `cleanup()`.
+
+```
+agents plan execute <plan_id>
+    → GitWorktreeSandbox.create(plan_id)
+    → LLM actor writes files via get_path()
+
+agents plan apply <plan_id>
+    → GitWorktreeSandbox.commit("cleveragents: apply plan <plan_id>")
+    → GitWorktreeSandbox.cleanup()
+    → Displays Apply Summary panel
+```
+
+The Apply Summary panel shows:
+
+- Plan ID
+- Artifact count (added + changed files)
+- Insertions / deletions (from `git diff --stat`)
+- Project name
+- Applied-at timestamp
+
+---
+
+## Gotchas
+
+1. **`original_path` must be the repository root**, not a subdirectory.  The sandbox
+   verifies this with `git rev-parse --show-toplevel`.
+
+2. **Merge conflicts are not auto-resolved**.  If the original branch has diverged
+   since the sandbox was created, `git merge` may fail.  The sandbox transitions to
+   `ERRORED` and raises `SandboxCommitError`.
+
+3. **Cleanup is not automatic on error**.  Always call `cleanup()` in a `finally`
+   block or use a context manager wrapper to avoid orphaned worktrees.
+
+4. **Git timeout defaults to 30 seconds**.  For large repositories or slow disks,
+   increase `git_timeout` when constructing the sandbox.
+
+---
+
+## Related
+
+- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md)
+- [ADR-038 Cross-Mechanism Sandbox Coordination](../adr/ADR-038-cross-mechanism-sandbox-coordination.md)
+- [Context Tier Hydration](context-tier-hydration.md)
+- [Custom Sandbox Strategy](../development/custom_sandbox_strategy.md)

mkdocs.yml

@@ -27,6 +27,8 @@ nav:
     - Shell Safety: modules/shell-safety.md
     - UKO Provenance Tracking: modules/uko-provenance.md
     - Invariant Reconciliation: modules/invariant-reconciliation.md
+    - Git Worktree Sandbox: modules/git-worktree-sandbox.md
+    - Context Tier Hydration: modules/context-tier-hydration.md
   - Development:
     - CI/CD Pipeline: development/ci-cd.md
     - Quality Automation: development/quality-automation.md