diff --git a/CHANGELOG.md b/CHANGELOG.md index 81a54b269..a2a8c8624 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,17 +7,21 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added -- **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges - LLM-generated changes via `git merge` from an isolated worktree branch - instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary - (plan ID, artifacts, insertions/deletions, project, timestamp), Sandbox - Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall - back to the original flat file copy. +- **Git Worktree Sandbox — Full Execute + Apply** (#5998): The plan execute + and apply phases now use an isolated git worktree sandbox for all + `git-checkout` resources. Changes are staged on a dedicated + `cleveragents/plan-` branch and merged back to the original branch on + `plan apply`. Non-git resources fall back to the copy-on-write sandbox. + Includes atomic rollback support via pre-merge commit tracking, a + spec-aligned Apply Summary panel (plan ID, artifacts, insertions/deletions, + project, timestamp), Sandbox Cleanup panel, and `✓ OK Changes applied` + footer. See [`docs/modules/git-worktree-sandbox.md`](docs/modules/git-worktree-sandbox.md). -- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types - (`detail_depth` and `relevance_score` must be strings, not int/float) that - caused Pydantic validation errors during context assembly, resulting in the - LLM receiving zero file context. +- **Container/Devcontainer Resource Stop** (#3250): `agents resource stop` now + correctly stops `container-instance` and `devcontainer-instance` resource + types in addition to the previously supported types. The resource handler + dispatch table was extended to route stop requests to the container and + devcontainer lifecycle handlers. - **Automation Tracking System**: Replaced shared session state issue tracking with individual per-agent tracking issues. Each agent now creates its own `[AUTO-]` @@ -72,27 +76,46 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). strategies, and parallel subtask execution with wave-based dependency analysis. Pass rate improved from 48.15% to 84.8%. -- **Container Resource Stop Support**: `agents resource stop` now correctly stops - `container-instance` and `devcontainer-instance` resource types. +- **Container Resource Stop Support** (#3250): `agents resource stop` now correctly stops + `container-instance` and `devcontainer-instance` resource types in addition to the + previously supported types. The resource handler dispatch table now routes stop requests + to the container and devcontainer lifecycle handlers. -- **Centralized Automation Tracking Manager** (`automation-tracking-manager`): The manager - is now the single interface for all tracking issue operations (`CREATE_TRACKING_ISSUE`, - `UPDATE_TRACKING_ISSUE`, `CLOSE_TRACKING_ISSUE`, `READ_TRACKING_STATE`, - `GET_NEXT_CYCLE_NUMBER`). Agents delegate to the manager rather than calling the Forgejo - API directly, ensuring sequential cycle numbers across restarts and preventing - duplicate issues. Migrated agents include `system-watchdog`, - `implementation-pool-supervisor`, `timeline-update-pool-supervisor`, - `project-owner-pool-supervisor`, `product-builder`, and - `backlog-grooming-pool-supervisor`. The legacy `shared/automation_tracking.md` module - was removed. +### Fixed -- **Documentation Writer Tracking** (`docs-writer`): The documentation writer now - participates in the automation tracking system by creating individual - `[AUTO-DOCS] Documentation Report (Cycle N)` issues every 10 cycles (~3.3 hours). - The manager applies the mandatory `Automation Tracking` label automatically, while - teams may add additional workflow labels as needed. See - `docs/development/automation-tracking.md` and the new - `docs/development/docs-writer.md` reference. +- **Plan action argument upsert** (#4197): Fixed a `UNIQUE constraint violation` + that occurred when `plan use` attempted to insert action arguments that already + existed in the database. The repository layer now uses an upsert (insert-or-update) + strategy so re-using a plan with the same action arguments succeeds idempotently. + +- **CI quality tests** (#4175): Restored all CI quality test sessions to passing + state after lint rule changes introduced regressions. Removed stale + `@tdd_expected_fail` tags from Behave scenarios that had been fixed. + +- **Validation attach CLI format** (#3837): `agents validation attach` extra + arguments now use `--key value` named option format instead of positional + arguments, matching the spec and other CLI commands. + +- **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the + `tdd_expected_fail_listener` `end_test()` function to prevent blindly inverting ALL test + failures to passes, which was masking infrastructure errors and causing flaky CI behavior. + Guards include setup/teardown error detection, non-assertion failure detection (infrastructure + errors), and dry-run mode detection. Also fixed `Variable Should Exist` syntax errors in + e2e test files and removed `tdd_expected_fail` from four context assembly e2e tests where + bugs were already fixed. + +- **`issue-state-updater` Bash Script Errors**: Removed problematic bash script examples + that attempted to invoke `task forgejo-label-manager` as a shell command. Replaced with + step-by-step operational instructions and direct label management via API. + +- **`automation-tracking-manager` Label Delegation Syntax**: Fixed incorrect delegation + syntax when calling `forgejo-label-manager`. The manager now uses natural-language + requests (for example, "Apply labels to issue #123: Automation Tracking") instead of + structured parameters, ensuring tracking issues receive proper labels. + +- **`product-builder` Missing Supervisors**: Added missing `pr-fix-pool-supervisor` and + `pr-merge-pool-supervisor` to the product-builder's supervisor launch list (18 total + supervisors). Updated all numeric references, pre-flight checklists, and validation logic. ### Changed @@ -113,50 +136,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Label Delegation Enforcement**: `automation-tracking-manager` now enforces delegation to `forgejo-label-manager` for all label operations, preventing "invalid label ID" errors - and ensuring label application uses correct name-to-ID mapping. - -- **Automation Tracking Label Guidance**: Documentation now clarifies that the manager - automatically applies the `Automation Tracking` label and that additional labels such as - `Type/Automation`, `State/In Progress`, or `Priority/Medium` remain optional workflow - choices rather than mandatory. - -- **Automation Tracking Agent Prefix Registry**: Expanded from 5 agents to 18 agents. - New prefixes include `AUTO-DOCS`, `AUTO-REV-POOL`, `AUTO-UAT-POOL`, `AUTO-BUG-POOL`, - `AUTO-INF-POOL`, `AUTO-ARCH`, `AUTO-EPIC`, `AUTO-EVLV`, `AUTO-GUARD`, `AUTO-SPEC`, - `AUTO-TIME`, `AUTO-PROJ-OWN`, and `AUTO-PROD-BLDR`. - -- **ACMS Context Hydration**: Fixed ACMS indexing pipeline not wired into CLI — - `ContextTierService` started empty on every CLI invocation so LLM received zero file - context during plan execution. Added `context_tier_hydrator.py` that reads files from - linked project resources (via `git ls-files` or `os.walk`) and stores them as - `TieredFragment` objects in the tier service. Hydration runs automatically before context - assembly in `LLMExecuteActor.execute()`. Respects max file size (256KB), total budget - (10MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__` directory - skipping. (#1028) - -### Fixed - -- **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the - `tdd_expected_fail_listener` `end_test()` function to prevent blindly inverting ALL test - failures to passes, which was masking infrastructure errors and causing flaky CI behavior. - Guards: setup/teardown error detection, non-assertion failure detection (infrastructure - errors), and dry-run mode detection. Also fixed `Variable Should Exist` syntax errors in - e2e test files and removed `tdd_expected_fail` from 4 context assembly e2e tests where - bugs were already fixed. - -- **`issue-state-updater` Bash Script Errors**: Removed problematic bash script examples - that tried to invoke `task forgejo-label-manager` as a bash command (the Task tool cannot - be invoked from bash). Replaced with clear step-by-step operational instructions and - direct label management via API. - -- **`automation-tracking-manager` Label Delegation Syntax**: Fixed incorrect delegation - syntax when calling `forgejo-label-manager`. The manager now uses correct natural language - requests (e.g., "Apply labels to issue #123: Automation Tracking") instead of structured - parameters, ensuring tracking issues receive proper labels. - -- **`product-builder` Missing Supervisors**: Added missing `pr-fix-pool-supervisor` and - `pr-merge-pool-supervisor` to the product-builder's supervisor launch list (18 total - supervisors). Updated all numeric references, pre-flight checklists, and validation logic. + and ensuring all tracking issues receive proper labels via name-to-ID mapping. --- diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index f5091deaa..4ff595ff8 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -3,6 +3,7 @@ * Aditya Chhabra * Brent E. Edwards * Hamza Khyari +* HAL 9000 * Jeffrey Phillips Freeman * Luis Mendes * Rui Hu diff --git a/README.md b/README.md index 733986eda..6df9cd01c 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,11 @@ embracing modern Python tooling. `validFrom`, and `isCurrent` metadata; a revision chain enables temporal queries - **JSON-RPC 2.0 A2A wire format** — `A2aRequest`/`A2aResponse` fields renamed to standard JSON-RPC 2.0 names (`method`, `id`, `result`, `error`) +- **Git worktree sandbox** — plan execute and apply phases use an isolated git worktree + for `git-checkout` resources; changes are committed on a dedicated branch and merged + back on `plan apply`; atomic rollback via pre-merge commit tracking +- **Container/devcontainer resource stop** — `agents resource stop` now supports + `container-instance` and `devcontainer-instance` resource types - Fast Typer/Click-based interface with parity for help/version behavior - Behavior-driven coverage via Behave and Robot Framework - Nox automation for linting, typing, testing, docs, builds, and benchmarks diff --git a/docs/modules/git-worktree-sandbox.md b/docs/modules/git-worktree-sandbox.md new file mode 100644 index 000000000..4935f06c5 --- /dev/null +++ b/docs/modules/git-worktree-sandbox.md @@ -0,0 +1,276 @@ +# Git Worktree Sandbox Module + +**Package:** `cleveragents.infrastructure.sandbox.git_worktree` +**Introduced:** v3.9.0 (PR #5998) + +The Git Worktree Sandbox provides fully isolated, git-native sandboxing for +`git-checkout` resources during plan execution and apply phases. Changes are +staged on a dedicated branch in a temporary git worktree and merged back to the +original branch only on a successful `plan apply`. If the plan is rolled back, +the merge is undone atomically. + +For the sandbox protocol and other sandbox strategies, see +[`docs/development/custom_sandbox_strategy.md`](../development/custom_sandbox_strategy.md). +For the plan lifecycle that invokes sandboxes, see +[ADR-006](../adr/ADR-006-plan-lifecycle.md) and +[ADR-015](../adr/ADR-015-sandbox-and-checkpoint.md). + +--- + +## Purpose + +When an actor executes a plan against a `git-checkout` resource, it must not +write changes directly to the working tree — doing so would corrupt the +repository state if the plan fails or is rolled back. The Git Worktree Sandbox +solves this by: + +1. Creating a temporary git worktree at a new branch (`cleveragents/plan-`) +2. Routing all actor file writes into the worktree +3. Committing changes in the worktree on `plan apply` +4. Merging the sandbox branch back to the original branch +5. Cleaning up the worktree and branch after the merge + +Non-git resources (e.g. `fs-directory`) continue to use the copy-on-write +sandbox strategy. + +--- + +## Lifecycle + +``` +GitWorktreeSandbox(resource_id, original_path) + │ + ▼ +sandbox.create(plan_id) # PENDING → CREATED + │ Creates worktree at /tmp/ca-sandbox--XXXX + │ Creates branch cleveragents/plan- + │ + ├─ sandbox.commit("msg") # CREATED → COMMITTED (no staged changes) + │ Returns CommitResult(success=True) with empty diffs when nothing is staged + │ + ▼ +sandbox.get_path("src/foo.py") # CREATED/ACTIVE → ACTIVE + │ Returns /tmp/ca-sandbox-.../src/foo.py + │ Actor writes to this path + │ + ▼ +sandbox.commit("msg") # ACTIVE → COMMITTED + │ git add -A (in worktree) + │ git commit (in worktree) + │ git merge cleveragents/plan- (in original repo) + │ + ▼ +sandbox.cleanup() # → CLEANED_UP + git worktree remove --force (falls back to shutil.rmtree on failure) + git branch -D cleveragents/plan- + git worktree prune +``` + +Rollback is possible from `ACTIVE` or `COMMITTED`: + +``` +sandbox.rollback() # ACTIVE/COMMITTED → ROLLED_BACK + │ If COMMITTED: git reset --hard (original repo) + │ git reset --hard (worktree) + │ git clean -fd (worktree) +``` + +--- + +## Key Class: `GitWorktreeSandbox` + +```python +from cleveragents.infrastructure.sandbox.git_worktree import GitWorktreeSandbox + +sandbox = GitWorktreeSandbox( + resource_id="01HXYZ...", + original_path="/home/user/my-project", + git_timeout=30, # optional, default 30 s +) +``` + +### Constructor Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `resource_id` | `str` | required | ULID of the resource being sandboxed | +| `original_path` | `str` | required | Absolute path to the git repository root | +| `git_timeout` | `int` | `30` | Timeout in seconds for each git command | + +### Methods + +#### `create(plan_id: str) → SandboxContext` + +Creates the worktree and sandbox branch. Must be called before any other +method. + +- Creates branch `cleveragents/plan-` from `HEAD` +- Creates a temporary directory at `/tmp/ca-sandbox--XXXX` +- Returns a `SandboxContext` with `sandbox_path` pointing to the worktree + +**Raises:** +- `ValueError` — if `plan_id` is empty +- `SandboxStateError` — if not in `PENDING` status +- `SandboxCreationError` — if git worktree creation fails (e.g. not a git repo, + git command timeout) + +#### `get_path(resource_path: str) → str` + +Translates a resource-relative path to its absolute path inside the worktree. + +```python +worktree_path = sandbox.get_path("src/cleveragents/cli/main.py") +# → "/tmp/ca-sandbox-abc123-XXXX/src/cleveragents/cli/main.py" +``` + +**Raises:** +- `SandboxStateError` — if sandbox is not in a usable status +- `ValueError` — if `resource_path` contains `..` (directory traversal) + +#### `commit(message: str | None = None) → CommitResult` + +Stages all changes in the worktree, commits them, and merges the sandbox branch +into the original branch. + +```python +result = sandbox.commit("feat: add new feature") +print(result.commit_ref) # git SHA of the sandbox commit +print(result.changed_files) # list of modified files +print(result.added_files) # list of new files +print(result.deleted_files) # list of deleted files +``` + +If there are no staged changes, returns a `CommitResult` with `success=True` +and empty file lists (no-op commit). + +**Raises:** +- `SandboxStateError` — if not in `CREATED` or `ACTIVE` status +- `SandboxCommitError` — if the git commit or merge fails + +#### `rollback() → None` + +Discards all worktree changes. + +- From `ACTIVE`: resets the worktree branch to the base commit +- From `COMMITTED`: additionally resets the original branch to the pre-merge + commit, fully undoing the merge + +> **Warning:** Rolling back from `COMMITTED` executes `git reset --hard` on +> the original branch. If other git worktrees or external processes are +> tracking the same branch, they will be affected. + +**Raises:** +- `SandboxStateError` — if called in an invalid status +- `SandboxRollbackError` — if the rollback fails + +#### `cleanup() → None` + +Removes the worktree directory and deletes the sandbox branch. Idempotent — +safe to call multiple times. + +Internally runs: +1. `git worktree remove --force ` +2. `git branch -D cleveragents/plan-` +3. `git worktree prune` + +Falls back to `shutil.rmtree` if `git worktree remove` fails. + +--- + +## `SandboxContext` Metadata + +After `create()`, the `sandbox.context.metadata` dict contains: + +| Key | Description | +|-----|-------------| +| `strategy` | Always `"git_worktree"` | +| `branch` | Sandbox branch name, e.g. `cleveragents/plan-abc123` | +| `original_branch` | Branch that was active before sandboxing | +| `base_commit` | SHA of the HEAD commit at sandbox creation time | +| `worktree_path` | Absolute path to the temporary worktree directory | + +--- + +## Status State Machine + +``` +PENDING + │ create() + ▼ +CREATED ──── get_path() ──► ACTIVE + │ │ + │ commit() │ commit() + ▼ ▼ +COMMITTED ◄─────────────── COMMITTED + │ │ + │ rollback() │ rollback() + ▼ ▼ +ROLLED_BACK ROLLED_BACK + │ │ + │ cleanup() │ cleanup() + ▼ ▼ +CLEANED_UP CLEANED_UP + +Any state ──► ERRORED (on git command failure) +``` + +--- + +## Error Reference + +| Exception | When raised | +|-----------|-------------| +| `SandboxCreationError` | `create()` fails (not a git repo, git timeout, etc.) | +| `SandboxCommitError` | `commit()` fails (git commit or merge error) | +| `SandboxRollbackError` | `rollback()` fails (git reset error) | +| `SandboxStateError` | Method called in wrong lifecycle state | + +All exceptions are importable from `cleveragents.infrastructure.sandbox.protocol`. + +--- + +## Integration with Plan Lifecycle + +The `SandboxFactory` selects `GitWorktreeSandbox` automatically when the +resource type is `git-checkout` and the sandbox strategy is `git_worktree`. +The `PlanApplyService` calls `sandbox.commit()` and then `sandbox.cleanup()` on +success, or `sandbox.rollback()` followed by `sandbox.cleanup()` on failure. + +The CLI `plan apply` command displays a spec-aligned Apply Summary panel: + +``` +╭─ Apply Summary ──────────────────────────────────────────────────╮ +│ Plan ID : 01HXYZ... │ +│ Project : local/my-project │ +│ Artifacts : 3 changed, 1 added, 0 deleted │ +│ Timestamp : 2026-04-09T12:34:56 │ +╰──────────────────────────────────────────────────────────────────╯ +╭─ Sandbox Cleanup ────────────────────────────────────────────────╮ +│ Worktree removed · Branch deleted · Prune complete │ +╰──────────────────────────────────────────────────────────────────╯ +✓ OK Changes applied +``` + +--- + +## Configuration + +The git command timeout defaults to 30 seconds. Override it by passing +`git_timeout` to the constructor, or configure it via the sandbox strategy +registry if using the factory. + +```python +sandbox = GitWorktreeSandbox( + resource_id=resource.id, + original_path=resource.location, + git_timeout=60, # increase for large repos on slow storage +) +``` + +--- + +## See Also + +- [`docs/development/custom_sandbox_strategy.md`](../development/custom_sandbox_strategy.md) — How to implement a custom sandbox strategy +- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) — Design rationale +- [ADR-038 Cross-Mechanism Sandbox Coordination](../adr/ADR-038-cross-mechanism-sandbox-coordination.md) — Multi-sandbox coordination diff --git a/mkdocs.yml b/mkdocs.yml index 2bb05974c..921fc43e5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,7 @@ nav: - Shell Safety: modules/shell-safety.md - UKO Provenance Tracking: modules/uko-provenance.md - Invariant Reconciliation: modules/invariant-reconciliation.md + - Git Worktree Sandbox: modules/git-worktree-sandbox.md - Development: - Agent System Specification: development/agent-system-specification.md - CI/CD Pipeline: development/ci-cd.md