diff --git a/CHANGELOG.md b/CHANGELOG.md index 81a54b269..7dc6a3098 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -45,6 +45,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). (`pr-api-creator`). The `backlog-groomer` adds a continuous Pass 19 for ongoing PR–issue label synchronization. The `issue-state-updater` syncs PR state labels whenever issue states change. +- **Module documentation** (#6841): Added comprehensive module guides for + `GitWorktreeSandbox` and the context tier hydration runtime, covering + lifecycle diagrams, metadata, configuration, logging, and apply-panel + behaviour. Linked navigation entries under *Modules* in MkDocs. - **Automation Tracking Announcements**: Extended `automation-tracking-manager` with announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`, @@ -106,7 +110,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Automation Tracking Format**: All automation tracking issues now use a standardized header format with mandatory `Reporting Interval: (Next report expected: )` declarations, enabling precise staleness detection. - - **PR Review Policy**: Reduced PR review requirement from 2 approvals to 1. Self-approval is now permitted including for automated bot PRs. Approval can be a formal review OR an approval comment (LGTM, Approved, ✅, "ready to merge"). @@ -134,6 +137,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). (10MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__` directory skipping. (#1028) +- **Architecture cross-references** (#6841): Updated the architecture overview with a + dedicated *Sandbox System* section and cross-references to the new hydration guide so + runtime documentation is discoverable from the high-level design. + ### Fixed - **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the diff --git a/docs/architecture.md b/docs/architecture.md index e355f0c5e..13155dad7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -217,6 +217,28 @@ Key UKO capabilities: - **Graph persistence** — `UKOGraphPersistence` serialises/restores state via JSON-file or in-memory backends across application restarts +See the +[`Context Tier Hydration`](modules/context-tier-hydration.md) module guide for the +runtime responsible for populating ACMS fragments before execution. + +--- + +## Sandbox System + +Sandboxing isolates plan execution side effects from the source repository until +they are explicitly applied ([ADR-015](adr/ADR-015-sandbox-and-checkpoint.md)). + +- `SandboxFactory` chooses the appropriate strategy (git worktree vs. + copy-on-write) based on resource type and configuration. +- `GitWorktreeSandbox` maintains a dedicated branch and temporary worktree for + `git-checkout` resources, enabling deterministic rollback and spec-aligned + apply summaries. + +For lifecycle details and usage examples see the +[`Git Worktree Sandbox`](modules/git-worktree-sandbox.md) module guide. Custom +strategies can be implemented via +[`docs/development/custom_sandbox_strategy.md`](development/custom_sandbox_strategy.md). + --- ## Invariant Reconciliation diff --git a/docs/modules/context-tier-hydration.md b/docs/modules/context-tier-hydration.md new file mode 100644 index 000000000..8054b61d7 --- /dev/null +++ b/docs/modules/context-tier-hydration.md @@ -0,0 +1,210 @@ +# Context Tier Hydration Module + +**Package:** `cleveragents.application.services.context_tier_hydrator` +**Introduced:** v3.9.0 (PR #4219, fixes bug #1028) + +The Context Tier Hydrator bridges the gap between files on disk and the ACMS +`ContextTierService`. Without hydration, CLI invocations start with an empty +tier store and the LLM receives no file context during plan execution. The +hydrator loads project resources into the HOT tier before context assembly +begins. + +> **Naming note:** The implementation is function-based; there is no +> `ContextTierHydrator` class. The documentation refers to the module by the +> common shorthand used in changelog and issue discussions. + +--- + +## Problem Solved + +The ACMS three-tier system stores `TieredFragment` objects in-memory via +`ContextTierService`. In a long-running server process these fragments persist, +but each CLI invocation starts a new process with an empty tier store. Prior to +this module, `LLMExecuteActor.execute()` assembled prompts from an empty tier +service, so the LLM received zero file context. The hydrator remedies this by +loading project resources on-demand before execution. + +--- + +## Automatic Invocation + +`LLMExecuteActor.execute()` calls `hydrate_tiers_for_plan()` automatically +before context assembly. No manual wiring is required for standard plan +execution. + +--- + +## Public API + +### `hydrate_tiers_from_project()` + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_from_project, +) +from cleveragents.application.services.context_tiers import ContextTierService + +tier_service = ContextTierService() + +count = hydrate_tiers_from_project( + tier_service=tier_service, + project_name="local/my-project", + resource_id="01HXYZ...", + resource_location="/home/user/my-project", + resource_type="git-checkout", # optional, defaults to "git-checkout" +) +print(f"Stored {count} fragments") +``` + +Reads files from a single resource directory and stores them as +`TieredFragment` objects in the tier service. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `tier_service` | `ContextTierService` | Tier store to populate | +| `project_name` | `str` | Namespaced project name | +| `resource_id` | `str` | ULID of the resource | +| `resource_location` | `str` | Filesystem path to the resource root | +| `resource_type` | `str` | Resource type; impacts file enumeration strategy | + +**Returns:** Number of fragments stored for that resource. + +### `hydrate_tiers_for_plan()` + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_for_plan, +) +from cleveragents.application.services.resource_registry_service import ( + ResourceRegistryService, +) +from cleveragents.infrastructure.database.repositories import ( + NamespacedProjectRepository, +) + +total = hydrate_tiers_for_plan( + tier_service=tier_service, + project_names=["local/my-project", "local/other-project"], + project_repository=project_repo, + resource_registry=resource_registry_service, +) +print(f"Stored {total} fragments across all projects") +``` + +Hydrates tiers for every project linked to a plan. It resolves project +metadata, enumerates linked resources, and delegates to +`hydrate_tiers_from_project()` when the resource has a filesystem location. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `tier_service` | `ContextTierService` | Tier store to populate | +| `project_names` | `list[str]` | Namespaced project names | +| `project_repository` | `NamespacedProjectRepository` | Project lookup service | +| `resource_registry` | `ResourceRegistryService` | Resource lookup service | + +**Returns:** Total fragments stored across all projects. + +--- + +## File Enumeration Strategy + +Hydration uses two strategies, selected by resource type: + +### `git ls-files` — git-based resources + +For `git-checkout` and `git` resources the hydrator runs: + +```bash +git ls-files --cached --others --exclude-standard +``` + +This lists tracked files and untracked files that are not gitignored. If the +git command fails or times out (30 seconds), hydration falls back to +`os.walk`. + +### `os.walk` — other resources + +For non-git resources the hydrator walks the directory tree, skipping: + +- Directories: `.git`, `.hg`, `.svn`, `__pycache__`, `node_modules`, `.venv`, + `venv`, `.nox`, `.tox`, `.mypy_cache`, `.pytest_cache`, `.ruff_cache`, `dist`, + `build`, `.eggs`, `.cleveragents`, and any directory starting with `.` or + ending with `.egg-info` +- File extensions: `.pyc`, `.pyo`, `.so`, `.o`, `.a`, `.dll`, `.exe`, `.png`, + `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.ico`, `.pdf`, `.zip`, `.tar`, `.gz`, `.bz2`, + `.xz`, `.whl`, `.egg`, `.db`, `.sqlite`, `.sqlite3` +- Hidden files (anything beginning with `.`) + +--- + +## Limits and Budgets + +To keep hydration deterministic and memory-safe, the module enforces: + +| Limit | Value | Description | +|-------|-------|-------------| +| Max file size | 256 KB | Larger files are skipped | +| Max total bytes | 10 MB | Hydration stops once budget is reached | + +The 10 MB budget applies per hydration invocation (plan-wide), not per project. + +--- + +## Fragment Structure + +Each file becomes a `TieredFragment` stored in the HOT tier: + +| Field | Value | +|-------|-------| +| `fragment_id` | `:` | +| `content` | UTF-8 file content | +| `tier` | `ContextTier.HOT` | +| `resource_id` | Resource ULID | +| `project_name` | Namespaced project name | +| `token_count` | Approximate (`len(content) // 4`) | +| `metadata.path` | Relative path within the resource | +| `metadata.detail_depth` | `"1"` (string) | +| `metadata.relevance_score` | `"0.5"` (string) | + +> **Metadata types:** `detail_depth` and `relevance_score` must be strings. The +> original bug #1028 stemmed from passing numeric types, which caused Pydantic +> validation to drop every fragment. + +All fragments are loaded into the HOT tier so that plan execution has immediate +access. Tier management can subsequently demote fragments based on age or +budget. + +--- + +## Logging + +Structured log events (via `structlog`) include: + +| Event | Level | Description | +|-------|-------|-------------| +| `context_hydrator.hydrated` | INFO | Hydration completed (`project`, `resource_id`, `fragments_stored`, `total_bytes`) | +| `context_hydrator.skip_missing_location` | WARNING | Resource path missing | +| `context_hydrator.store_failed` | DEBUG | Storing a fragment failed (non-fatal) | +| `context_hydrator.project_not_found` | DEBUG | Project lookup failed | +| `context_hydrator.resource_not_found` | DEBUG | Resource lookup failed | +| `context_hydrator.no_linked_resources` | DEBUG | Project has no linked resources | + +--- + +## Gotchas + +1. Binary files and non-UTF-8 content are silently skipped. +2. The 10 MB budget applies per hydration call. Large monorepos may require + follow-up hydration runs or selective resource linking. +3. `git ls-files` enforces a 30-second timeout to avoid hanging the CLI. Large + repositories may fall back to `os.walk`. +4. Fragments are always HOT-tier; future enhancements may distribute fragments + across tiers based on relevance. + +--- + +## Related Documentation + +- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md) +- [Git Worktree Sandbox](git-worktree-sandbox.md) +- [ACMS API Reference](../api/core.md) diff --git a/docs/modules/git-worktree-sandbox.md b/docs/modules/git-worktree-sandbox.md new file mode 100644 index 000000000..1fc3cade2 --- /dev/null +++ b/docs/modules/git-worktree-sandbox.md @@ -0,0 +1,248 @@ +# Git Worktree Sandbox Module + +**Package:** `cleveragents.infrastructure.sandbox.git_worktree` +**Introduced:** v3.9.0 (PR #5998) + +The Git Worktree Sandbox provides isolated, git-native staging for +`git-checkout` resources during plan execution and apply phases. All file +writes are routed to a temporary git worktree on a dedicated branch, ensuring +the original working tree stays untouched until `plan apply` succeeds. If +apply fails or the plan rolls back, the sandbox can undo the merge atomically. + +> **Background:** Non-git resources (for example `fs-directory`) continue to use +> the copy-on-write sandbox strategy. See +> [`docs/development/custom_sandbox_strategy.md`](../development/custom_sandbox_strategy.md) +> for the sandbox protocol and alternative strategies. + +--- + +## Purpose + +When an actor executes a plan against a git repository it must avoid direct +writes to the repository’s active branch. The Git Worktree Sandbox solves +this by: + +1. Creating a temporary git worktree at `/tmp/ca-sandbox--XXXX` +2. Creating a detached branch named `cleveragents/plan-` +3. Routing all file writes through `GitWorktreeSandbox.get_path()` +4. Committing and merging the sandbox branch during `plan apply` +5. Cleaning up (or rolling back) the sandbox branch once apply finishes + +This preserves repository integrity, enables deterministic rollbacks, and +aligns with the specification’s apply summary output. + +--- + +## Lifecycle + +``` +GitWorktreeSandbox(resource_id, original_path) + │ + ▼ +create(plan_id) # PENDING → CREATED + │ Creates worktree + sandbox branch + │ + ├─ get_path("src/foo.py") # CREATED → ACTIVE + │ Returns worktree path for writes + │ + ├─ commit("msg") # ACTIVE → COMMITTED (no staged changes → ACTIVE) + │ git add -A • git commit • git merge sandbox → original + │ + ├─ rollback() # ACTIVE/COMMITTED → ROLLED_BACK + │ Resets worktree (+ original branch if COMMITTED) + │ + └─ cleanup() # any state → CLEANED_UP + git worktree remove --force + git branch -D cleveragents/plan- + git worktree prune +``` + +Any git command failure transitions the sandbox to `ERRORED`. + +--- + +## Status State Machine + +| Status | Meaning | +|--------|---------| +| `PENDING` | Sandbox instantiated; `create()` not yet invoked | +| `CREATED` | Worktree and branch exist; no writes yet | +| `ACTIVE` | At least one write occurred via `get_path()` | +| `COMMITTED` | Changes committed and merged into the original branch | +| `ROLLED_BACK` | Changes discarded and branches reset | +| `ERRORED` | A git command failed; sandbox unusable until cleanup | +| `CLEANED_UP` | Worktree directory and branch removed | + +--- + +## API Reference + +```python +from cleveragents.infrastructure.sandbox.git_worktree import GitWorktreeSandbox + +sandbox = GitWorktreeSandbox( + resource_id="01HXYZ...", + original_path="/home/user/my-project", + git_timeout=30, # optional +) + +ctx = sandbox.create(plan_id="01PLAN...") +worktree_file = sandbox.get_path("src/cleveragents/cli/main.py") +# actor writes to worktree_file + +result = sandbox.commit("docs: update module guides") +print(result.commit_ref, result.changed_files) + +sandbox.cleanup() +``` + +### Constructor Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `resource_id` | `str` | required | ULID of the sandboxed resource | +| `original_path` | `str` | required | Absolute path to the git repository root | +| `git_timeout` | `int` | `30` | Timeout (seconds) for each git command | + +### `create(plan_id: str) → SandboxContext` + +Creates the temporary worktree and sandbox branch. Returns a +`SandboxContext` whose `metadata` records operational data. Raises +`ValueError` for empty plan IDs, `SandboxStateError` outside `PENDING`, and +`SandboxCreationError` when git operations fail (missing repo, timeout, etc.). + +### `get_path(resource_path: str) → str` + +Translates a resource-relative path into the sandbox worktree. Rejects +directory traversal (`..`) with `ValueError` and invalid lifecycle states with +`SandboxStateError`. + +### `commit(message: str | None = None) → CommitResult` + +Stages, commits, and merges sandbox changes. The returned `CommitResult` +includes the sandbox ID, commit SHA, changed/added/deleted file lists, and a +timestamp. Raises `SandboxStateError` for invalid lifecycle states and +`SandboxCommitError` when git commit or merge fails. If nothing is staged the +method returns a `success=True` no-op result. + +### `rollback() → None` + +Discards sandbox changes. From `ACTIVE` it resets the sandbox branch to the +base commit; from `COMMITTED` it also resets the original branch to the +pre-merge commit. Errors raise `SandboxRollbackError`. + +> **Multi-worktree caution:** Rolling back from `COMMITTED` executes +> `git reset --hard` on the original branch. Coordinate with other worktrees +> that track the same branch before rolling back. + +### `cleanup() → None` + +Removes the worktree directory, deletes the sandbox branch, and prunes stale +worktrees. The method is idempotent and falls back to `shutil.rmtree` when +`git worktree remove` fails. + +--- + +## `SandboxContext` Metadata + +After `create()` the returned context contains: + +| Key | Description | +|-----|-------------| +| `strategy` | Always `"git_worktree"` | +| `branch` | Sandbox branch name (`cleveragents/plan-`) | +| `original_branch` | Branch active before sandboxing | +| `base_commit` | SHA captured at sandbox creation | +| `worktree_path` | Absolute path to the temporary worktree directory | + +Use these fields for downstream logging, apply summaries, or cleanup. + +--- + +## Branch Naming Rules + +Sandbox branches follow the `cleveragents/plan-` pattern. +Plan IDs are sanitised to retain alphanumerics, hyphens, underscores, slashes, +and dots; disallowed characters collapse into `-`. + +Example: `01JQABC-xyz!foo` → `cleveragents/plan-01JQABC-xyz-foo`. + +--- + +## Error Reference + +All exceptions live in `cleveragents.infrastructure.sandbox.protocol`: + +| Exception | When raised | +|-----------|-------------| +| `SandboxCreationError` | `create()` fails (no repo, timeout, etc.) | +| `SandboxCommitError` | `commit()` git commit or merge fails | +| `SandboxRollbackError` | `rollback()` git reset fails | +| `SandboxStateError` | Lifecycle state invalid for the attempted action | + +--- + +## Integration with Plan Lifecycle + +`SandboxFactory` automatically selects `GitWorktreeSandbox` when a plan uses a +`git-checkout` resource and the `git_worktree` strategy. During `agents plan +execute` the sandbox is created, and during `agents plan apply` the +`PlanApplyService` invokes `commit()` followed by `cleanup()`. + +``` +agents plan execute + → GitWorktreeSandbox.create(plan_id) + → Actor writes via get_path() + +agents plan apply + → GitWorktreeSandbox.commit("cleveragents: apply plan ") + → GitWorktreeSandbox.cleanup() +``` + +The CLI shows the spec-aligned Apply Summary panel: + +``` +╭─ Apply Summary ───────────────────────────────────────────────╮ +│ Plan ID : 01HXYZ... │ +│ Project : local/my-project │ +│ Artifacts : 3 changed · 1 added · 0 deleted │ +│ Timestamp : 2026-04-09T12:34:56Z │ +╰───────────────────────────────────────────────────────────────╯ +╭─ Sandbox Cleanup ────────────────────────────────────────────╮ +│ Worktree removed · Branch deleted · Prune complete │ +╰───────────────────────────────────────────────────────────────╯ +✓ OK Changes applied +``` + +--- + +## Configuration & Tuning + +- **Git timeout:** Increase `git_timeout` for very large repositories or slow + storage layers. +- **Cleanup responsibility:** Always call `cleanup()` in a `finally` block (or + use a context manager wrapper) to avoid orphaned worktrees. +- **Conflict handling:** Merge conflicts surface as `SandboxCommitError` and + leave the sandbox in `ERRORED`. Resolve conflicts and re-run apply or + discard the sandbox. + +--- + +## Gotchas + +1. `original_path` must be the repository root. The sandbox validates this via + `git rev-parse --show-toplevel`. +2. Rolling back from `COMMITTED` affects the original branch with a hard reset. + Ensure no other worktrees rely on the branch without coordination. +3. Cleanup is not automatic when exceptions propagate; always invoke it. +4. Long-running git operations (large repos, network storage) may require a + higher `git_timeout`. + +--- + +## Related Documentation + +- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) +- [ADR-038 Cross-Mechanism Sandbox Coordination](../adr/ADR-038-cross-mechanism-sandbox-coordination.md) +- [Context Tier Hydration](context-tier-hydration.md) +- [Custom Sandbox Strategy Guide](../development/custom_sandbox_strategy.md) diff --git a/mkdocs.yml b/mkdocs.yml index 2bb05974c..77d0bcb25 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,8 @@ nav: - Shell Safety: modules/shell-safety.md - UKO Provenance Tracking: modules/uko-provenance.md - Invariant Reconciliation: modules/invariant-reconciliation.md + - Git Worktree Sandbox: modules/git-worktree-sandbox.md + - Context Tier Hydration: modules/context-tier-hydration.md - Development: - Agent System Specification: development/agent-system-specification.md - CI/CD Pipeline: development/ci-cd.md