docs(modules): add git worktree sandbox and context hydration docs

ISSUES CLOSED: #6841

CHANGELOG.md

@@ -45,6 +45,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
   (`pr-api-creator`). The `backlog-groomer` adds a continuous Pass 19 for ongoing
   PR–issue label synchronization. The `issue-state-updater` syncs PR state labels whenever
   issue states change.
+- **Module documentation** (#6841): Added comprehensive module guides for
+  `GitWorktreeSandbox` and the context tier hydration runtime, covering
+  lifecycle diagrams, metadata, configuration, logging, and apply-panel
+  behaviour.  Linked navigation entries under *Modules* in MkDocs.
 
 - **Automation Tracking Announcements**: Extended `automation-tracking-manager` with
   announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`,
@@ -106,7 +110,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - **Automation Tracking Format**: All automation tracking issues now use a standardized
   header format with mandatory `Reporting Interval: <interval> (Next report expected: <ts>)`
   declarations, enabling precise staleness detection.
-
 - **PR Review Policy**: Reduced PR review requirement from 2 approvals to 1. Self-approval
   is now permitted including for automated bot PRs. Approval can be a formal review OR an
   approval comment (LGTM, Approved, ✅, "ready to merge").
@@ -134,6 +137,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
   (10MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__` directory
   skipping. (#1028)
 
+- **Architecture cross-references** (#6841): Updated the architecture overview with a
+  dedicated *Sandbox System* section and cross-references to the new hydration guide so
+  runtime documentation is discoverable from the high-level design.
+
 ### Fixed
 
 - **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the

docs/architecture.md

@@ -217,6 +217,28 @@ Key UKO capabilities:
 - **Graph persistence** — `UKOGraphPersistence` serialises/restores state via
   JSON-file or in-memory backends across application restarts
 
+See the
+[`Context Tier Hydration`](modules/context-tier-hydration.md) module guide for the
+runtime responsible for populating ACMS fragments before execution.
+
+---
+
+## Sandbox System
+
+Sandboxing isolates plan execution side effects from the source repository until
+they are explicitly applied ([ADR-015](adr/ADR-015-sandbox-and-checkpoint.md)).
+
+- `SandboxFactory` chooses the appropriate strategy (git worktree vs.
+  copy-on-write) based on resource type and configuration.
+- `GitWorktreeSandbox` maintains a dedicated branch and temporary worktree for
+  `git-checkout` resources, enabling deterministic rollback and spec-aligned
+  apply summaries.
+
+For lifecycle details and usage examples see the
+[`Git Worktree Sandbox`](modules/git-worktree-sandbox.md) module guide.  Custom
+strategies can be implemented via
+[`docs/development/custom_sandbox_strategy.md`](development/custom_sandbox_strategy.md).
+
 ---
 
 ## Invariant Reconciliation

docs/modules/context-tier-hydration.md

@@ -0,0 +1,210 @@
+# Context Tier Hydration Module
+
+**Package:** `cleveragents.application.services.context_tier_hydrator`  
+**Introduced:** v3.9.0 (PR #4219, fixes bug #1028)
+
+The Context Tier Hydrator bridges the gap between files on disk and the ACMS
+`ContextTierService`.  Without hydration, CLI invocations start with an empty
+tier store and the LLM receives no file context during plan execution.  The
+hydrator loads project resources into the HOT tier before context assembly
+begins.
+
+> **Naming note:** The implementation is function-based; there is no
+> `ContextTierHydrator` class.  The documentation refers to the module by the
+> common shorthand used in changelog and issue discussions.
+
+---
+
+## Problem Solved
+
+The ACMS three-tier system stores `TieredFragment` objects in-memory via
+`ContextTierService`.  In a long-running server process these fragments persist,
+but each CLI invocation starts a new process with an empty tier store.  Prior to
+this module, `LLMExecuteActor.execute()` assembled prompts from an empty tier
+service, so the LLM received zero file context.  The hydrator remedies this by
+loading project resources on-demand before execution.
+
+---
+
+## Automatic Invocation
+
+`LLMExecuteActor.execute()` calls `hydrate_tiers_for_plan()` automatically
+before context assembly.  No manual wiring is required for standard plan
+execution.
+
+---
+
+## Public API
+
+### `hydrate_tiers_from_project()`
+
+```python
+from cleveragents.application.services.context_tier_hydrator import (
+    hydrate_tiers_from_project,
+)
+from cleveragents.application.services.context_tiers import ContextTierService
+
+tier_service = ContextTierService()
+
+count = hydrate_tiers_from_project(
+    tier_service=tier_service,
+    project_name="local/my-project",
+    resource_id="01HXYZ...",
+    resource_location="/home/user/my-project",
+    resource_type="git-checkout",  # optional, defaults to "git-checkout"
+)
+print(f"Stored {count} fragments")
+```
+
+Reads files from a single resource directory and stores them as
+`TieredFragment` objects in the tier service.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `tier_service` | `ContextTierService` | Tier store to populate |
+| `project_name` | `str` | Namespaced project name |
+| `resource_id` | `str` | ULID of the resource |
+| `resource_location` | `str` | Filesystem path to the resource root |
+| `resource_type` | `str` | Resource type; impacts file enumeration strategy |
+
+**Returns:** Number of fragments stored for that resource.
+
+### `hydrate_tiers_for_plan()`
+
+```python
+from cleveragents.application.services.context_tier_hydrator import (
+    hydrate_tiers_for_plan,
+)
+from cleveragents.application.services.resource_registry_service import (
+    ResourceRegistryService,
+)
+from cleveragents.infrastructure.database.repositories import (
+    NamespacedProjectRepository,
+)
+
+total = hydrate_tiers_for_plan(
+    tier_service=tier_service,
+    project_names=["local/my-project", "local/other-project"],
+    project_repository=project_repo,
+    resource_registry=resource_registry_service,
+)
+print(f"Stored {total} fragments across all projects")
+```
+
+Hydrates tiers for every project linked to a plan.  It resolves project
+metadata, enumerates linked resources, and delegates to
+`hydrate_tiers_from_project()` when the resource has a filesystem location.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `tier_service` | `ContextTierService` | Tier store to populate |
+| `project_names` | `list[str]` | Namespaced project names |
+| `project_repository` | `NamespacedProjectRepository` | Project lookup service |
+| `resource_registry` | `ResourceRegistryService` | Resource lookup service |
+
+**Returns:** Total fragments stored across all projects.
+
+---
+
+## File Enumeration Strategy
+
+Hydration uses two strategies, selected by resource type:
+
+### `git ls-files` — git-based resources
+
+For `git-checkout` and `git` resources the hydrator runs:
+
+```bash
+git ls-files --cached --others --exclude-standard
+```
+
+This lists tracked files and untracked files that are not gitignored.  If the
+git command fails or times out (30 seconds), hydration falls back to
+`os.walk`.
+
+### `os.walk` — other resources
+
+For non-git resources the hydrator walks the directory tree, skipping:
+
+- Directories: `.git`, `.hg`, `.svn`, `__pycache__`, `node_modules`, `.venv`,
+  `venv`, `.nox`, `.tox`, `.mypy_cache`, `.pytest_cache`, `.ruff_cache`, `dist`,
+  `build`, `.eggs`, `.cleveragents`, and any directory starting with `.` or
+  ending with `.egg-info`
+- File extensions: `.pyc`, `.pyo`, `.so`, `.o`, `.a`, `.dll`, `.exe`, `.png`,
+  `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.ico`, `.pdf`, `.zip`, `.tar`, `.gz`, `.bz2`,
+  `.xz`, `.whl`, `.egg`, `.db`, `.sqlite`, `.sqlite3`
+- Hidden files (anything beginning with `.`)
+
+---
+
+## Limits and Budgets
+
+To keep hydration deterministic and memory-safe, the module enforces:
+
+| Limit | Value | Description |
+|-------|-------|-------------|
+| Max file size | 256 KB | Larger files are skipped |
+| Max total bytes | 10 MB | Hydration stops once budget is reached |
+
+The 10 MB budget applies per hydration invocation (plan-wide), not per project.
+
+---
+
+## Fragment Structure
+
+Each file becomes a `TieredFragment` stored in the HOT tier:
+
+| Field | Value |
+|-------|-------|
+| `fragment_id` | `<resource_id>:<relative_path>` |
+| `content` | UTF-8 file content |
+| `tier` | `ContextTier.HOT` |
+| `resource_id` | Resource ULID |
+| `project_name` | Namespaced project name |
+| `token_count` | Approximate (`len(content) // 4`) |
+| `metadata.path` | Relative path within the resource |
+| `metadata.detail_depth` | `"1"` (string) |
+| `metadata.relevance_score` | `"0.5"` (string) |
+
+> **Metadata types:** `detail_depth` and `relevance_score` must be strings.  The
+> original bug #1028 stemmed from passing numeric types, which caused Pydantic
+> validation to drop every fragment.
+
+All fragments are loaded into the HOT tier so that plan execution has immediate
+access.  Tier management can subsequently demote fragments based on age or
+budget.
+
+---
+
+## Logging
+
+Structured log events (via `structlog`) include:
+
+| Event | Level | Description |
+|-------|-------|-------------|
+| `context_hydrator.hydrated` | INFO | Hydration completed (`project`, `resource_id`, `fragments_stored`, `total_bytes`) |
+| `context_hydrator.skip_missing_location` | WARNING | Resource path missing |
+| `context_hydrator.store_failed` | DEBUG | Storing a fragment failed (non-fatal) |
+| `context_hydrator.project_not_found` | DEBUG | Project lookup failed |
+| `context_hydrator.resource_not_found` | DEBUG | Resource lookup failed |
+| `context_hydrator.no_linked_resources` | DEBUG | Project has no linked resources |
+
+---
+
+## Gotchas
+
+1. Binary files and non-UTF-8 content are silently skipped.
+2. The 10 MB budget applies per hydration call.  Large monorepos may require
+   follow-up hydration runs or selective resource linking.
+3. `git ls-files` enforces a 30-second timeout to avoid hanging the CLI.  Large
+   repositories may fall back to `os.walk`.
+4. Fragments are always HOT-tier; future enhancements may distribute fragments
+   across tiers based on relevance.
+
+---
+
+## Related Documentation
+
+- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md)
+- [Git Worktree Sandbox](git-worktree-sandbox.md)
+- [ACMS API Reference](../api/core.md)

docs/modules/git-worktree-sandbox.md

@@ -0,0 +1,248 @@
+# Git Worktree Sandbox Module
+
+**Package:** `cleveragents.infrastructure.sandbox.git_worktree`  
+**Introduced:** v3.9.0 (PR #5998)
+
+The Git Worktree Sandbox provides isolated, git-native staging for
+`git-checkout` resources during plan execution and apply phases.  All file
+writes are routed to a temporary git worktree on a dedicated branch, ensuring
+the original working tree stays untouched until `plan apply` succeeds.  If
+apply fails or the plan rolls back, the sandbox can undo the merge atomically.
+
+> **Background:** Non-git resources (for example `fs-directory`) continue to use
+> the copy-on-write sandbox strategy.  See
+> [`docs/development/custom_sandbox_strategy.md`](../development/custom_sandbox_strategy.md)
+> for the sandbox protocol and alternative strategies.
+
+---
+
+## Purpose
+
+When an actor executes a plan against a git repository it must avoid direct
+writes to the repository’s active branch.  The Git Worktree Sandbox solves
+this by:
+
+1. Creating a temporary git worktree at `/tmp/ca-sandbox-<plan_id>-XXXX`
+2. Creating a detached branch named `cleveragents/plan-<plan_id>`
+3. Routing all file writes through `GitWorktreeSandbox.get_path()`
+4. Committing and merging the sandbox branch during `plan apply`
+5. Cleaning up (or rolling back) the sandbox branch once apply finishes
+
+This preserves repository integrity, enables deterministic rollbacks, and
+aligns with the specification’s apply summary output.
+
+---
+
+## Lifecycle
+
+```
+GitWorktreeSandbox(resource_id, original_path)
+    │
+    ▼
+create(plan_id)                 # PENDING → CREATED
+    │  Creates worktree + sandbox branch
+    │
+    ├─ get_path("src/foo.py")   # CREATED → ACTIVE
+    │     Returns worktree path for writes
+    │
+    ├─ commit("msg")            # ACTIVE → COMMITTED (no staged changes → ACTIVE)
+    │     git add -A • git commit • git merge sandbox → original
+    │
+    ├─ rollback()                # ACTIVE/COMMITTED → ROLLED_BACK
+    │     Resets worktree (+ original branch if COMMITTED)
+    │
+    └─ cleanup()                 # any state → CLEANED_UP
+          git worktree remove --force
+          git branch -D cleveragents/plan-<plan_id>
+          git worktree prune
+```
+
+Any git command failure transitions the sandbox to `ERRORED`.
+
+---
+
+## Status State Machine
+
+| Status | Meaning |
+|--------|---------|
+| `PENDING` | Sandbox instantiated; `create()` not yet invoked |
+| `CREATED` | Worktree and branch exist; no writes yet |
+| `ACTIVE` | At least one write occurred via `get_path()` |
+| `COMMITTED` | Changes committed and merged into the original branch |
+| `ROLLED_BACK` | Changes discarded and branches reset |
+| `ERRORED` | A git command failed; sandbox unusable until cleanup |
+| `CLEANED_UP` | Worktree directory and branch removed |
+
+---
+
+## API Reference
+
+```python
+from cleveragents.infrastructure.sandbox.git_worktree import GitWorktreeSandbox
+
+sandbox = GitWorktreeSandbox(
+    resource_id="01HXYZ...",
+    original_path="/home/user/my-project",
+    git_timeout=30,       # optional
+)
+
+ctx = sandbox.create(plan_id="01PLAN...")
+worktree_file = sandbox.get_path("src/cleveragents/cli/main.py")
+# actor writes to worktree_file
+
+result = sandbox.commit("docs: update module guides")
+print(result.commit_ref, result.changed_files)
+
+sandbox.cleanup()
+```
+
+### Constructor Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `resource_id` | `str` | required | ULID of the sandboxed resource |
+| `original_path` | `str` | required | Absolute path to the git repository root |
+| `git_timeout` | `int` | `30` | Timeout (seconds) for each git command |
+
+### `create(plan_id: str) → SandboxContext`
+
+Creates the temporary worktree and sandbox branch.  Returns a
+`SandboxContext` whose `metadata` records operational data.  Raises
+`ValueError` for empty plan IDs, `SandboxStateError` outside `PENDING`, and
+`SandboxCreationError` when git operations fail (missing repo, timeout, etc.).
+
+### `get_path(resource_path: str) → str`
+
+Translates a resource-relative path into the sandbox worktree.  Rejects
+directory traversal (`..`) with `ValueError` and invalid lifecycle states with
+`SandboxStateError`.
+
+### `commit(message: str | None = None) → CommitResult`
+
+Stages, commits, and merges sandbox changes.  The returned `CommitResult`
+includes the sandbox ID, commit SHA, changed/added/deleted file lists, and a
+timestamp.  Raises `SandboxStateError` for invalid lifecycle states and
+`SandboxCommitError` when git commit or merge fails.  If nothing is staged the
+method returns a `success=True` no-op result.
+
+### `rollback() → None`
+
+Discards sandbox changes.  From `ACTIVE` it resets the sandbox branch to the
+base commit; from `COMMITTED` it also resets the original branch to the
+pre-merge commit.  Errors raise `SandboxRollbackError`.
+
+> **Multi-worktree caution:** Rolling back from `COMMITTED` executes
+> `git reset --hard` on the original branch.  Coordinate with other worktrees
+> that track the same branch before rolling back.
+
+### `cleanup() → None`
+
+Removes the worktree directory, deletes the sandbox branch, and prunes stale
+worktrees.  The method is idempotent and falls back to `shutil.rmtree` when
+`git worktree remove` fails.
+
+---
+
+## `SandboxContext` Metadata
+
+After `create()` the returned context contains:
+
+| Key | Description |
+|-----|-------------|
+| `strategy` | Always `"git_worktree"` |
+| `branch` | Sandbox branch name (`cleveragents/plan-<plan_id>`) |
+| `original_branch` | Branch active before sandboxing |
+| `base_commit` | SHA captured at sandbox creation |
+| `worktree_path` | Absolute path to the temporary worktree directory |
+
+Use these fields for downstream logging, apply summaries, or cleanup.
+
+---
+
+## Branch Naming Rules
+
+Sandbox branches follow the `cleveragents/plan-<sanitised_plan_id>` pattern.
+Plan IDs are sanitised to retain alphanumerics, hyphens, underscores, slashes,
+and dots; disallowed characters collapse into `-`.
+
+Example: `01JQABC-xyz!foo` → `cleveragents/plan-01JQABC-xyz-foo`.
+
+---
+
+## Error Reference
+
+All exceptions live in `cleveragents.infrastructure.sandbox.protocol`:
+
+| Exception | When raised |
+|-----------|-------------|
+| `SandboxCreationError` | `create()` fails (no repo, timeout, etc.) |
+| `SandboxCommitError` | `commit()` git commit or merge fails |
+| `SandboxRollbackError` | `rollback()` git reset fails |
+| `SandboxStateError` | Lifecycle state invalid for the attempted action |
+
+---
+
+## Integration with Plan Lifecycle
+
+`SandboxFactory` automatically selects `GitWorktreeSandbox` when a plan uses a
+`git-checkout` resource and the `git_worktree` strategy.  During `agents plan
+execute` the sandbox is created, and during `agents plan apply` the
+`PlanApplyService` invokes `commit()` followed by `cleanup()`.
+
+```
+agents plan execute <plan_id>
+    → GitWorktreeSandbox.create(plan_id)
+    → Actor writes via get_path()
+
+agents plan apply <plan_id>
+    → GitWorktreeSandbox.commit("cleveragents: apply plan <plan_id>")
+    → GitWorktreeSandbox.cleanup()
+```
+
+The CLI shows the spec-aligned Apply Summary panel:
+
+```
+╭─ Apply Summary ───────────────────────────────────────────────╮
+│ Plan ID   : 01HXYZ...                                         │
+│ Project   : local/my-project                                  │
+│ Artifacts : 3 changed · 1 added · 0 deleted                   │
+│ Timestamp : 2026-04-09T12:34:56Z                              │
+╰───────────────────────────────────────────────────────────────╯
+╭─ Sandbox Cleanup ────────────────────────────────────────────╮
+│ Worktree removed · Branch deleted · Prune complete            │
+╰───────────────────────────────────────────────────────────────╯
+✓ OK Changes applied
+```
+
+---
+
+## Configuration & Tuning
+
+- **Git timeout:** Increase `git_timeout` for very large repositories or slow
+  storage layers.
+- **Cleanup responsibility:** Always call `cleanup()` in a `finally` block (or
+  use a context manager wrapper) to avoid orphaned worktrees.
+- **Conflict handling:** Merge conflicts surface as `SandboxCommitError` and
+  leave the sandbox in `ERRORED`.  Resolve conflicts and re-run apply or
+  discard the sandbox.
+
+---
+
+## Gotchas
+
+1. `original_path` must be the repository root.  The sandbox validates this via
+   `git rev-parse --show-toplevel`.
+2. Rolling back from `COMMITTED` affects the original branch with a hard reset.
+   Ensure no other worktrees rely on the branch without coordination.
+3. Cleanup is not automatic when exceptions propagate; always invoke it.
+4. Long-running git operations (large repos, network storage) may require a
+   higher `git_timeout`.
+
+---
+
+## Related Documentation
+
+- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md)
+- [ADR-038 Cross-Mechanism Sandbox Coordination](../adr/ADR-038-cross-mechanism-sandbox-coordination.md)
+- [Context Tier Hydration](context-tier-hydration.md)
+- [Custom Sandbox Strategy Guide](../development/custom_sandbox_strategy.md)

mkdocs.yml

@@ -27,6 +27,8 @@ nav:
     - Shell Safety: modules/shell-safety.md
     - UKO Provenance Tracking: modules/uko-provenance.md
     - Invariant Reconciliation: modules/invariant-reconciliation.md
+    - Git Worktree Sandbox: modules/git-worktree-sandbox.md
+    - Context Tier Hydration: modules/context-tier-hydration.md
   - Development:
     - Agent System Specification: development/agent-system-specification.md
     - CI/CD Pipeline: development/ci-cd.md