diff --git a/docs/modules/context-tier-hydration.md b/docs/modules/context-tier-hydration.md new file mode 100644 index 000000000..7f19bd9b6 --- /dev/null +++ b/docs/modules/context-tier-hydration.md @@ -0,0 +1,235 @@ +# Context Tier Hydration + +The **Context Tier Hydrator** bridges the gap between the resource registry (files on +disk) and the ACMS `ContextTierService` (in-memory fragments). Without it, the tier +service starts empty on every CLI process invocation and the LLM receives zero file +context during plan execution. + +Introduced in PR #5998 (merged 2026-04-09), fixing bug #1028. + +--- + +## Problem + +The Advanced Context Management System (ACMS) stores file content as `TieredFragment` +objects in `ContextTierService`. This service is in-memory and is not persisted +between CLI invocations. When a plan is executed, the LLM actor calls the context +assembler to build a context window from the tier service — but the service is empty +because no files have been loaded. + +The hydrator solves this by reading files from the project's linked resources and +storing them as `TieredFragment` objects before context assembly begins. + +--- + +## Architecture + +``` +CLI: agents plan execute + │ + ▼ +LLMExecuteActor.execute() + │ + ├─── hydrate_tiers_for_plan() ← Context Tier Hydrator + │ │ + │ ├── project_repository.get(project_name) + │ ├── resource_registry.show_resource(resource_id) + │ └── hydrate_tiers_from_project() + │ │ + │ ├── _list_files() + │ │ ├── git ls-files (git-checkout resources) + │ │ └── os.walk (other resource types) + │ └── tier_service.store(TieredFragment) + │ + └─── context_assembler.assemble() ← reads from tier_service +``` + +--- + +## Key Functions + +**Module**: `cleveragents.application.services.context_tier_hydrator` + +### `hydrate_tiers_for_plan` + +```python +def hydrate_tiers_for_plan( + tier_service: ContextTierService, + project_names: list[str], + project_repository: Any, + resource_registry: Any, +) -> int: +``` + +Top-level entry point. Iterates over all projects linked to a plan, resolves their +linked resources, and calls `hydrate_tiers_from_project` for each. + +**Parameters**: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `tier_service` | `ContextTierService` | The tier service to populate | +| `project_names` | `list[str]` | Namespaced project names (e.g. `local/my-project`) | +| `project_repository` | `NamespacedProjectRepository` | Repository for project lookups | +| `resource_registry` | `ResourceRegistryService` | Registry for resource lookups | + +**Returns**: Total number of `TieredFragment` objects stored across all projects. + +--- + +### `hydrate_tiers_from_project` + +```python +def hydrate_tiers_from_project( + tier_service: ContextTierService, + project_name: str, + resource_id: str, + resource_location: str, + resource_type: str = "git-checkout", +) -> int: +``` + +Reads files from a single resource directory and stores them as `TieredFragment` +objects in the tier service. + +**Parameters**: + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `tier_service` | `ContextTierService` | — | The tier service to populate | +| `project_name` | `str` | — | Namespaced project name | +| `resource_id` | `str` | — | ULID of the resource | +| `resource_location` | `str` | — | Filesystem path to the resource root | +| `resource_type` | `str` | `"git-checkout"` | Resource type (affects file listing strategy) | + +**Returns**: Number of fragments stored. + +**Limits enforced**: + +| Limit | Value | Description | +|-------|-------|-------------| +| Max file size | 256 KB | Files larger than this are skipped | +| Max total bytes | 10 MB | Hydration stops when this budget is reached | + +--- + +## File Listing Strategy + +The hydrator uses two strategies to enumerate files, selected by resource type: + +### `git ls-files` (git-checkout resources) + +For `git-checkout` and `git` resource types, the hydrator runs: + +```bash +git ls-files --cached --others --exclude-standard +``` + +This lists tracked files and untracked files that are not gitignored. Binary +extensions (`.pyc`, `.so`, `.png`, etc.) are filtered out. + +Falls back to `os.walk` if the git command fails or times out (30 s timeout). + +### `os.walk` (other resource types) + +For non-git resources, the hydrator walks the directory tree, skipping: + +**Directories**: `.git`, `.hg`, `.svn`, `__pycache__`, `node_modules`, `.venv`, +`venv`, `.nox`, `.tox`, `.mypy_cache`, `.pytest_cache`, `.ruff_cache`, `dist`, +`build`, `.eggs`, `.cleveragents`, and any directory starting with `.` or ending +with `.egg-info`. + +**File extensions**: `.pyc`, `.pyo`, `.so`, `.o`, `.a`, `.dll`, `.exe`, `.png`, +`.jpg`, `.jpeg`, `.gif`, `.bmp`, `.ico`, `.pdf`, `.zip`, `.tar`, `.gz`, `.bz2`, +`.xz`, `.whl`, `.egg`, `.db`, `.sqlite`, `.sqlite3`. + +**Hidden files**: Files starting with `.` are skipped. + +--- + +## Fragment Format + +Each file is stored as a `TieredFragment` with: + +| Field | Value | +|-------|-------| +| `fragment_id` | `:` | +| `content` | UTF-8 file content | +| `tier` | `ContextTier.HOT` | +| `resource_id` | Resource ULID | +| `project_name` | Namespaced project name | +| `token_count` | `len(content) // 4` (rough estimate) | +| `metadata.path` | Relative path within the resource | +| `metadata.detail_depth` | `"1"` (string, not int) | +| `metadata.relevance_score` | `"0.5"` (string, not float) | + +!!! note "Metadata types" + `detail_depth` and `relevance_score` must be **strings**, not `int` or `float`. + Pydantic v2 validation on `ContextFragment` enforces this. Passing numeric + values causes a `ValidationError` that silently drops all context from the LLM + call. This was the root cause of bug #1028. + +--- + +## Usage Example + +The hydrator is called automatically by `LLMExecuteActor.execute()` before context +assembly. You can also call it directly for testing or custom integrations: + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_from_project, +) +from cleveragents.application.services.context_tiers import ContextTierService + +tier_service = ContextTierService() + +count = hydrate_tiers_from_project( + tier_service=tier_service, + project_name="local/my-project", + resource_id="01ABCDEF...", + resource_location="/path/to/repo", + resource_type="git-checkout", +) +print(f"Stored {count} fragments") +``` + +--- + +## Logging + +The hydrator emits structured log events via `structlog`: + +| Event | Level | Fields | +|-------|-------|--------| +| `context_hydrator.skip_missing_location` | WARNING | `project`, `resource_id`, `location` | +| `context_hydrator.store_failed` | DEBUG | `fragment_id`, `exc_info` | +| `context_hydrator.hydrated` | INFO | `project`, `resource_id`, `fragments_stored`, `total_bytes` | +| `context_hydrator.project_not_found` | DEBUG | `project` | +| `context_hydrator.no_linked_resources` | DEBUG | `project` | +| `context_hydrator.resource_not_found` | DEBUG | `resource_id` | + +--- + +## Gotchas + +1. **Binary files are silently skipped**. If a file cannot be decoded as UTF-8, it + is skipped without error. + +2. **The 10 MB budget is per-hydration call**, not per-project. If a project has + many large files, the budget may be exhausted before all files are indexed. + +3. **`git ls-files` requires a 30-second timeout**. On very large repositories or + slow disks, the command may time out and fall back to `os.walk`. + +4. **Fragments are stored in the HOT tier**. All files are treated as equally + relevant. Future versions may use relevance scoring to assign files to + HOT/WARM/COLD tiers. + +--- + +## Related + +- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md) +- [Git Worktree Sandbox](git-worktree-sandbox.md) +- [API Reference — ACMS](../api/core.md) diff --git a/docs/modules/git-worktree-sandbox.md b/docs/modules/git-worktree-sandbox.md new file mode 100644 index 000000000..5dc5110b4 --- /dev/null +++ b/docs/modules/git-worktree-sandbox.md @@ -0,0 +1,244 @@ +# Git Worktree Sandbox + +The **Git Worktree Sandbox** provides isolated, reversible staging for LLM-generated +file changes during plan execution. It uses `git worktree` to create a detached +working tree on a dedicated branch, so modifications never touch the original branch +until the plan is explicitly applied. + +Introduced in PR #5998 (merged 2026-04-09) as part of the M1 plan lifecycle +implementation. + +--- + +## Overview + +When a plan is executed against a `git-checkout` resource, CleverAgents creates a +temporary git worktree on a branch named `cleveragents/plan-`. The LLM +actor writes files into this worktree. On `plan apply`, the sandbox branch is merged +back into the project's current branch via `git merge`. If anything goes wrong, the +sandbox can be rolled back without affecting the original working tree. + +Non-git projects fall back to the flat directory sandbox (`shutil.copy2`). + +--- + +## Lifecycle + +``` +create(plan_id) + │ Creates branch cleveragents/plan- + │ Creates worktree at /tmp/ca-sandbox--/ + ▼ +get_path(resource_path) + │ Translates resource-relative paths to worktree paths + │ Actor writes files here + ▼ +commit(message) + │ git add -A → git commit → git merge (in original repo) + ▼ +cleanup() + │ git worktree remove --force + │ git branch -D cleveragents/plan- + │ git worktree prune + ▼ + CLEANED_UP +``` + +Rollback is available from `ACTIVE` or `COMMITTED` states: + +``` +rollback() + │ ACTIVE → git reset --hard + git clean -fd (in worktree) + │ COMMITTED → git reset --hard (in original repo) + │ + git reset --hard (in worktree) + ▼ + ROLLED_BACK +``` + +--- + +## Status State Machine + +| Status | Meaning | +|--------|---------| +| `PENDING` | Sandbox created but `create()` not yet called | +| `CREATED` | Worktree and branch exist; no writes yet | +| `ACTIVE` | Actor has written at least one file | +| `COMMITTED` | Changes committed and merged to original branch | +| `ROLLED_BACK` | Changes discarded; worktree reset to base commit | +| `ERRORED` | A git command failed; sandbox is unusable | +| `CLEANED_UP` | Worktree and branch removed | + +--- + +## Key Class: `GitWorktreeSandbox` + +**Module**: `cleveragents.infrastructure.sandbox.git_worktree` + +```python +from cleveragents.infrastructure.sandbox.git_worktree import GitWorktreeSandbox + +sandbox = GitWorktreeSandbox( + resource_id="01ABCDEF...", # ULID of the git-checkout resource + original_path="/path/to/repo", + git_timeout=30, # seconds per git command (default: 30) +) + +# Create the worktree +ctx = sandbox.create(plan_id="01PLAN...") +print(ctx.sandbox_path) # /tmp/ca-sandbox-01PLAN-xxxx/ +print(ctx.metadata["branch"]) # cleveragents/plan-01PLAN... + +# Resolve a path inside the worktree +worktree_file = sandbox.get_path("src/calculator.py") + +# After the actor writes files... +result = sandbox.commit("feat: implement calculator") +print(result.commit_ref) # git SHA of the sandbox commit +print(result.added_files) # ["src/calculator.py"] + +# Cleanup +sandbox.cleanup() +``` + +### Constructor Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `resource_id` | `str` | — | ULID of the resource being sandboxed | +| `original_path` | `str` | — | Absolute path to the git repository root | +| `git_timeout` | `int` | `30` | Timeout in seconds for each git command | + +### Methods + +#### `create(plan_id: str) → SandboxContext` + +Creates the worktree and branch. Must be called before any other method. + +- Verifies `original_path` is a git repository root (not a subdirectory). +- Creates branch `cleveragents/plan-`. +- Creates a temporary directory for the worktree. +- Returns a `SandboxContext` with `sandbox_path`, `original_path`, `resource_id`, + `plan_id`, `created_at`, and `metadata` (strategy, branch, original_branch, + base_commit, worktree_path). + +#### `get_path(resource_path: str) → str` + +Translates a resource-relative path to an absolute path inside the worktree. +Raises `ValueError` on directory traversal attempts (`..` components). + +#### `commit(message: str | None = None) → CommitResult` + +Stages all changes (`git add -A`), commits them in the worktree, then merges +the sandbox branch into the original branch (`git merge --no-edit`). + +Returns a `CommitResult` with: + +| Field | Type | Description | +|-------|------|-------------| +| `sandbox_id` | `str` | Sandbox ULID | +| `success` | `bool` | `True` on success | +| `commit_ref` | `str` | SHA of the sandbox commit | +| `changed_files` | `list[str]` | Modified files | +| `added_files` | `list[str]` | New files | +| `deleted_files` | `list[str]` | Removed files | +| `error` | `str \| None` | Error message on failure | +| `timestamp` | `datetime` | Commit timestamp | + +If there are no staged changes, returns a `CommitResult` with `success=True` +and empty file lists (no-op commit). + +#### `rollback() → None` + +Discards all changes. Behaviour depends on current status: + +- **`ACTIVE`**: Resets the worktree branch to `base_commit` and runs `git clean -fd`. +- **`COMMITTED`**: Also resets the original branch to `pre_merge_commit` (undoes the merge). + +!!! warning "Multi-worktree safety" + Rolling back from `COMMITTED` executes `git reset --hard` on the original + branch. Other git worktrees or external processes tracking the same branch + will be affected. + +#### `cleanup() → None` + +Removes the worktree and deletes the sandbox branch. Idempotent — safe to call +multiple times. Falls back to `shutil.rmtree` if `git worktree remove` fails. + +--- + +## Error Types + +All errors are defined in `cleveragents.infrastructure.sandbox.protocol`. + +| Exception | When raised | +|-----------|-------------| +| `SandboxCreationError` | `git worktree add` or prerequisite git commands fail | +| `SandboxCommitError` | `git commit` or `git merge` fails | +| `SandboxRollbackError` | `git reset` or `git clean` fails | +| `SandboxStateError` | Method called in an invalid status | + +--- + +## Branch Naming + +Branch names are sanitised to be git-safe: + +- Only alphanumeric characters, hyphens, underscores, slashes, and dots are kept. +- Runs of disallowed characters are collapsed into a single hyphen. +- The final name is `cleveragents/plan-`. + +Example: plan ID `01JQABC-xyz!foo` → branch `cleveragents/plan-01JQABC-xyz-foo`. + +--- + +## Integration with Plan Lifecycle + +The `GitWorktreeSandbox` is instantiated by the CLI `plan execute` command when the +plan's linked resource is a `git-checkout` type. The `plan apply` command calls +`commit()` followed by `cleanup()`. + +``` +agents plan execute + → GitWorktreeSandbox.create(plan_id) + → LLM actor writes files via get_path() + +agents plan apply + → GitWorktreeSandbox.commit("cleveragents: apply plan ") + → GitWorktreeSandbox.cleanup() + → Displays Apply Summary panel +``` + +The Apply Summary panel shows: + +- Plan ID +- Artifact count (added + changed files) +- Insertions / deletions (from `git diff --stat`) +- Project name +- Applied-at timestamp + +--- + +## Gotchas + +1. **`original_path` must be the repository root**, not a subdirectory. The sandbox + verifies this with `git rev-parse --show-toplevel`. + +2. **Merge conflicts are not auto-resolved**. If the original branch has diverged + since the sandbox was created, `git merge` may fail. The sandbox transitions to + `ERRORED` and raises `SandboxCommitError`. + +3. **Cleanup is not automatic on error**. Always call `cleanup()` in a `finally` + block or use a context manager wrapper to avoid orphaned worktrees. + +4. **Git timeout defaults to 30 seconds**. For large repositories or slow disks, + increase `git_timeout` when constructing the sandbox. + +--- + +## Related + +- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) +- [ADR-038 Cross-Mechanism Sandbox Coordination](../adr/ADR-038-cross-mechanism-sandbox-coordination.md) +- [Context Tier Hydration](context-tier-hydration.md) +- [Custom Sandbox Strategy](../development/custom_sandbox_strategy.md) diff --git a/mkdocs.yml b/mkdocs.yml index af4eba30e..539ff0beb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,8 @@ nav: - Shell Safety: modules/shell-safety.md - UKO Provenance Tracking: modules/uko-provenance.md - Invariant Reconciliation: modules/invariant-reconciliation.md + - Git Worktree Sandbox: modules/git-worktree-sandbox.md + - Context Tier Hydration: modules/context-tier-hydration.md - Development: - CI/CD Pipeline: development/ci-cd.md - Quality Automation: development/quality-automation.md