diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a76a7292..85528f852 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall back to the original flat file copy. +- **Documentation**: Added module guides for the Git Worktree Sandbox and Context + Tier Hydration systems, refreshed the architecture overview with a Sandbox System + section, and updated MkDocs navigation accordingly. + - **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types (`detail_depth` and `relevance_score` must be strings, not int/float) that caused Pydantic validation errors during context assembly, resulting in the diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index f5091deaa..10338a451 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -6,6 +6,7 @@ * Jeffrey Phillips Freeman * Luis Mendes * Rui Hu +* HAL 9000 # Details diff --git a/docs/architecture.md b/docs/architecture.md index e355f0c5e..06d441914 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -194,6 +194,58 @@ Resources are managed external entities --- +## Sandbox System + +The sandbox system provides isolated, reversible environments for resource +modifications during plan execution +([ADR-015](adr/ADR-015-sandbox-and-checkpoint.md), +[ADR-038](adr/ADR-038-cross-mechanism-sandbox-coordination.md)). + +### Git Worktree Sandbox (default for git-checkout resources) + +For resources of type `git-checkout`, the **Git Worktree Sandbox** +(`infrastructure.sandbox.git_worktree.GitWorktreeSandbox`) creates an +isolated git branch and worktree: + +``` +Execute phase: + 1. Create branch cleveragents/plan- from HEAD + 2. Add git worktree at a temp directory on that branch + 3. LLM writes files into the worktree (not the main working tree) + +Apply phase: + 4. git merge cleveragents/plan- into the project branch + 5. Display Apply Summary, Sandbox Cleanup, and Next Steps panels + 6. Remove worktree and delete sandbox branch +``` + +Rollback from `ACTIVE` resets the worktree to the base commit. +Rollback from `COMMITTED` also resets the original branch to the +pre-merge commit, fully undoing the merge. + +Non-git projects fall back to the flat-directory sandbox using +`shutil.copy2`. + +See [`docs/modules/git-worktree-sandbox.md`](modules/git-worktree-sandbox.md) +for the full module guide. + +### Context Tier Hydration + +Before context assembly in the Execute phase, the +`context_tier_hydrator` reads files from linked project resources and +stores them as `TieredFragment` objects in `ContextTierService`. This +ensures the LLM has full file context on every CLI invocation (the +service is in-memory and starts empty each process). + +- **git-checkout resources**: uses `git ls-files` (respects `.gitignore`) +- **Other resources**: uses `os.walk` with skip-list filtering +- **Limits**: 256 KB per file, 10 MB total per project + +See [`docs/modules/context-tier-hydration.md`](modules/context-tier-hydration.md) +for the full module guide. + +--- + ## Context Management (ACMS) The Advanced Context Management System manages what context is sent to the diff --git a/docs/modules/context-tier-hydration.md b/docs/modules/context-tier-hydration.md new file mode 100644 index 000000000..46aae412f --- /dev/null +++ b/docs/modules/context-tier-hydration.md @@ -0,0 +1,202 @@ +# Context Tier Hydration + +The **Context Tier Hydration** module bridges the gap between the resource +registry (files on disk) and the ACMS context tier (in-memory fragments). +Without it, `ContextTierService` starts empty on every CLI process invocation +and the LLM receives zero file context during plan execution. + +Implemented in PR #4219 (merged 2026-04-09). Resolves bug #1028. + +--- + +## Problem + +The `ContextTierService` is an in-memory store. On each new CLI process +invocation it starts empty. Before this module, the LLM executing a plan +had no awareness of the project's files — it was operating blind. + +--- + +## Solution + +`context_tier_hydrator.py` reads files from linked project resources at the +start of plan execution and stores them as `TieredFragment` objects in the +`ContextTierService`. The LLM then has full file context available during +context assembly. + +--- + +## Architecture + +```mermaid +graph TD + CLI["CLI: plan execute"] --> HYDRATOR["hydrate_tiers_for_plan()"] + HYDRATOR --> PROJ["NamespacedProjectRepository\n.get(project_name)"] + PROJ --> RES["ResourceRegistryService\n.show_resource(resource_id)"] + RES --> FILES["_list_files()\n(git ls-files or os.walk)"] + FILES --> FRAGS["TieredFragment objects\n(HOT tier)"] + FRAGS --> TIER["ContextTierService\n.store(fragment)"] + TIER --> LLM["LLMExecuteActor\n(context assembly)"] +``` + +Hydration runs **before** context assembly in `LLMExecuteActor.execute()`. + +--- + +## API Reference + +### `hydrate_tiers_for_plan` + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_for_plan, +) + +count = hydrate_tiers_for_plan( + tier_service=tier_service, + project_names=["local/my-project"], + project_repository=project_repository, + resource_registry=resource_registry, +) +``` + +Iterates over all projects and their linked resources, calling +`hydrate_tiers_from_project` for each. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `tier_service` | `ContextTierService` | Service to populate. | +| `project_names` | `list[str]` | Namespaced project names (e.g. `local/my-project`). | +| `project_repository` | `NamespacedProjectRepository` | Repository for project lookup. | +| `resource_registry` | `ResourceRegistryService` | Registry for resource lookup. | + +**Returns:** Total number of `TieredFragment` objects stored. + +--- + +### `hydrate_tiers_from_project` + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_from_project, +) + +count = hydrate_tiers_from_project( + tier_service=tier_service, + project_name="local/my-project", + resource_id="01HZ...", + resource_location="/home/user/my-project", + resource_type="git-checkout", +) +``` + +Reads files from a single resource and stores them as `TieredFragment` +objects in the `HOT` tier. + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `tier_service` | `ContextTierService` | — | Service to populate. | +| `project_name` | `str` | — | Namespaced project name. | +| `resource_id` | `str` | — | ULID of the resource. | +| `resource_location` | `str` | — | Filesystem path to the resource root. | +| `resource_type` | `str` | `"git-checkout"` | Resource type (affects file listing strategy). | + +**Returns:** Number of fragments stored. + +--- + +## File Listing Strategies + +The hydrator uses two strategies depending on resource type: + +| Strategy | Trigger | Description | +|----------|---------|-------------| +| `git ls-files` | `resource_type` is `git-checkout` or `git` | Lists tracked and untracked (non-ignored) files via git. Respects `.gitignore`. | +| `os.walk` | All other resource types | Walks the directory tree, skipping hidden files and configured directories. | + +If `git ls-files` fails (e.g. not a git repo), the hydrator falls back to +`os.walk` automatically. + +--- + +## Limits and Filters + +| Limit | Value | Description | +|-------|-------|-------------| +| Max file size | 256 KB | Files larger than this are skipped. | +| Max total bytes | 10 MB | Hydration stops when this budget is reached. | +| Binary extensions | `.pyc`, `.so`, `.png`, `.pdf`, etc. | Skipped entirely. | +| Skip directories | `.git`, `__pycache__`, `node_modules`, `.venv`, `.cleveragents`, etc. | Never traversed. | + +--- + +## Fragment Metadata + +Each `TieredFragment` is stored in the `HOT` tier with the following metadata: + +```python +TieredFragment( + fragment_id=f"{resource_id}:{rel_path}", + content=file_content, + tier=ContextTier.HOT, + resource_id=resource_id, + project_name=project_name, + token_count=len(content) // 4, # approximate + metadata={ + "path": rel_path, + "detail_depth": "1", # string, not int + "relevance_score": "0.5", # string, not float + }, +) +``` + +!!! note "Metadata type requirement" + `detail_depth` and `relevance_score` **must be strings**, not `int` or + `float`. Pydantic validation in `ContextFragment` enforces this. Passing + numeric values causes a `ValidationError` that crashes context assembly, + leaving the LLM with zero context. This was the root cause of bug #4454. + +--- + +## Integration with LLMExecuteActor + +The hydrator is called from `LLMExecuteActor.execute()` via constructor +injection — no `get_container()` calls from within the service: + +```python +# In CLI factory (_get_plan_executor) +actor = LLMExecuteActor( + tier_service=container.context_tier_service(), + project_repository=container.project_repository(), + resource_registry=container.resource_registry(), + # ... +) +``` + +The `ACMSExecutePhaseContextAssembler` exposes `tier_service` as a public +property so the actor can pass it to the hydrator. + +--- + +## Logging + +The hydrator emits structured log events via `structlog`: + +| Event | Level | Description | +|-------|-------|-------------| +| `context_hydrator.skip_missing_location` | WARNING | Resource location is missing or not a directory. | +| `context_hydrator.hydrated` | INFO | Hydration complete; includes fragment count and total bytes. | +| `context_hydrator.project_not_found` | DEBUG | Project not found in repository. | +| `context_hydrator.no_linked_resources` | DEBUG | Project has no linked resources. | +| `context_hydrator.resource_not_found` | DEBUG | Resource not found in registry. | +| `context_hydrator.store_failed` | DEBUG | Fragment storage failed (non-fatal). | + +--- + +## Related + +- [`docs/modules/git-worktree-sandbox.md`](git-worktree-sandbox.md) — sandbox for git-checkout resources +- [`docs/architecture.md`](../architecture.md) — ACMS context management section +- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md) +- Source: `src/cleveragents/application/services/context_tier_hydrator.py` +- Source: `src/cleveragents/application/services/context_tiers.py` diff --git a/docs/modules/git-worktree-sandbox.md b/docs/modules/git-worktree-sandbox.md new file mode 100644 index 000000000..c82e6f1c4 --- /dev/null +++ b/docs/modules/git-worktree-sandbox.md @@ -0,0 +1,219 @@ +# Git Worktree Sandbox + +The **Git Worktree Sandbox** provides isolated, branch-based staging for +LLM-generated file changes during plan execution. It is the default sandbox +strategy for resources of type `git-checkout` and replaces the earlier +flat-directory `shutil.copy2` approach for git-backed projects. + +Implemented in PR #5998 (merged 2026-04-09). Spec reference: [Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md). + +--- + +## Overview + +When a plan enters the **Execute** phase for a git-checkout resource, the +sandbox: + +1. Creates a new git branch `cleveragents/plan-` from the current + `HEAD` of the repository. +2. Adds a git worktree at a temporary directory pointing to that branch. +3. Writes all LLM file output into the worktree (not the main working tree). +4. On **Apply**, merges the worktree branch back into the project's current + branch via `git merge`. +5. Displays spec-aligned panels: **Apply Summary**, **Sandbox Cleanup**, and + **Next Steps**. +6. Removes the worktree and deletes the sandbox branch on cleanup. + +Non-git projects fall back to the original flat-directory sandbox using +`shutil.copy2`. + +--- + +## Architecture + +```mermaid +graph LR + subgraph "Execute Phase" + LLM["LLM Actor"] -->|writes files| WT["Git Worktree\n(temp dir)"] + WT -->|on branch| SB["cleveragents/plan-"] + end + + subgraph "Apply Phase" + SB -->|git merge| MAIN["Project Branch\n(original HEAD)"] + MAIN -->|cleanup| DEL["Worktree removed\nBranch deleted"] + end +``` + +The worktree is completely isolated from the main working tree. Concurrent +plans on the same repository each get their own branch and worktree directory, +preventing interference. + +--- + +## Class Reference + +### `GitWorktreeSandbox` + +```python +from cleveragents.infrastructure.sandbox.git_worktree import GitWorktreeSandbox +``` + +Implements the `Sandbox` protocol. + +#### Constructor + +```python +GitWorktreeSandbox( + resource_id: str, + original_path: str, + git_timeout: int = 30, +) +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `resource_id` | `str` | Identifier of the resource being sandboxed. | +| `original_path` | `str` | Absolute path to the git repository root. | +| `git_timeout` | `int` | Timeout in seconds for each git command (default: 30). | + +**Raises:** `ValueError` if any argument is invalid. + +#### Lifecycle Methods + +| Method | Description | +|--------|-------------| +| `create(plan_id)` | Creates the worktree and branch. Returns `SandboxContext`. | +| `get_path(resource_path)` | Translates a resource-relative path to the worktree path. | +| `commit(message=None)` | Stages all changes, commits in the worktree, and merges to the original branch. | +| `rollback()` | Resets the worktree to the base commit; if already committed, also resets the original branch to the pre-merge commit. | +| `cleanup()` | Removes the worktree and deletes the sandbox branch. Idempotent. | + +#### Status Transitions + +``` +PENDING → CREATED → ACTIVE → COMMITTED → CLEANED_UP + ↘ ROLLED_BACK ↗ + ERRORED (any failure) +``` + +--- + +## Usage Example + +```python +from cleveragents.infrastructure.sandbox.git_worktree import GitWorktreeSandbox + +sandbox = GitWorktreeSandbox( + resource_id="01HZ...", + original_path="/home/user/my-project", +) + +# Create the worktree +ctx = sandbox.create(plan_id="01HZ...") +print(ctx.sandbox_path) # /tmp/ca-sandbox-01HZ...-xxxxx + +# Resolve a path inside the worktree +worktree_file = sandbox.get_path("src/main.py") + +# Write files (done by LLM actor) +with open(worktree_file, "w") as f: + f.write("# LLM-generated content\n") + +# Commit and merge back +result = sandbox.commit("feat: LLM-generated changes") +print(result.commit_ref) # git commit hash +print(result.added_files) # ["src/main.py"] +print(result.changed_files) # [] + +# Cleanup +sandbox.cleanup() +``` + +--- + +## Branch Naming + +Sandbox branches follow the pattern: + +``` +cleveragents/plan- +``` + +The plan ID is sanitised to be git-safe: only alphanumerics, hyphens, +underscores, slashes, and dots are kept; runs of other characters are +collapsed to a single hyphen. + +--- + +## Rollback Semantics + +`rollback()` supports two scenarios: + +| Status at rollback | Behaviour | +|--------------------|-----------| +| `ACTIVE` | Resets the worktree branch to the base commit and cleans untracked files. | +| `COMMITTED` | Also resets the original branch to the pre-merge commit (`git reset --hard`), fully undoing the merge. | + +!!! warning "Multi-worktree safety" + Rolling back from `COMMITTED` executes `git reset --hard` on the original + branch. If other git worktrees or external processes are tracking the same + branch, they will be affected. + +--- + +## Apply Phase Panels + +When `plan apply` completes for a git-checkout resource, the CLI renders +three spec-aligned panels: + +=== "Apply Summary" + ``` + ┌─ Apply Summary ──────────────────────────────────────────────────┐ + │ Plan ID: 01HZ... │ + │ Artifacts: 3 files │ + │ Changes: +42 / -7 │ + │ Project: local/my-project │ + │ Applied at: 2026-04-09 14:33:34 │ + └──────────────────────────────────────────────────────────────────┘ + ``` + +=== "Sandbox Cleanup" + ``` + ┌─ Sandbox Cleanup ────────────────────────────────────────────────┐ + │ ✓ Worktree removed │ + │ ✓ Branch merged to main │ + └──────────────────────────────────────────────────────────────────┘ + ``` + +=== "Next Steps" + ``` + ┌─ Next Steps ─────────────────────────────────────────────────────┐ + │ 1. Review changes: git diff HEAD~1 │ + │ 2. Commit if satisfied: git commit --amend │ + └──────────────────────────────────────────────────────────────────┘ + ✓ OK Changes applied + ``` + +--- + +## Error Handling + +All git operations are wrapped with: + +- **Timeout**: Each command has a configurable timeout (default 30 s). + `subprocess.TimeoutExpired` raises `SandboxCreationError`, + `SandboxCommitError`, or `SandboxRollbackError` as appropriate. +- **Non-zero exit**: `subprocess.CalledProcessError` is caught and re-raised + as the appropriate sandbox error with the git stderr included. +- **Cleanup fallback**: If `git worktree remove` fails, the directory is + removed manually with `shutil.rmtree`. + +--- + +## Related + +- [`docs/modules/shell-safety.md`](shell-safety.md) — shell danger detection +- [`docs/architecture.md`](../architecture.md) — sandbox section +- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) +- [ADR-038 Cross-Mechanism Sandbox Coordination](../adr/ADR-038-cross-mechanism-sandbox-coordination.md) +- Source: `src/cleveragents/infrastructure/sandbox/git_worktree.py` diff --git a/mkdocs.yml b/mkdocs.yml index f7c707117..11bda8613 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,8 @@ nav: - Shell Safety: modules/shell-safety.md - UKO Provenance Tracking: modules/uko-provenance.md - Invariant Reconciliation: modules/invariant-reconciliation.md + - Git Worktree Sandbox: modules/git-worktree-sandbox.md + - Context Tier Hydration: modules/context-tier-hydration.md - Development: - Agent System Specification: development/agent-system-specification.md - CI/CD Pipeline: development/ci-cd.md