From 60a25d4ba78641b97b7abba8f3e7e2cbd39c37ea Mon Sep 17 00:00:00 2001 From: CleverThis Date: Fri, 10 Apr 2026 04:31:12 +0000 Subject: [PATCH 1/3] docs: add git worktree sandbox and context tier hydrator module guides - Add docs/modules/git-worktree-sandbox.md: full lifecycle guide for GitWorktreeSandbox including state machine, method reference, error table, and CLI Apply Summary panel documentation (PR #5998) - Add docs/modules/context-tier-hydrator.md: module guide for ContextTierHydrator covering public API, limits/filters, fragment structure, and structured log events (PR #4219, bug #1028) - Update mkdocs.yml nav to include both new module pages - Update README.md highlights with git worktree sandbox, ACMS context tier hydration, and container/devcontainer resource stop features - Update CHANGELOG.md [Unreleased] section with entries for PRs #5998, #4219, #3250, #4197, #4175, #3837 --- CHANGELOG.md | 48 ++++- README.md | 8 + docs/modules/context-tier-hydrator.md | 189 ++++++++++++++++++ docs/modules/git-worktree-sandbox.md | 273 ++++++++++++++++++++++++++ mkdocs.yml | 2 + 5 files changed, 518 insertions(+), 2 deletions(-) create mode 100644 docs/modules/context-tier-hydrator.md create mode 100644 docs/modules/git-worktree-sandbox.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 81a54b269..c00c559f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,33 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added +- **Git Worktree Sandbox — Full Execute + Apply** (#5998): The plan execute + and apply phases now use an isolated git worktree sandbox for all + `git-checkout` resources. Changes are staged on a dedicated + `cleveragents/plan-` branch and merged back to the original branch on + `plan apply`. Non-git resources fall back to the copy-on-write sandbox. + Includes atomic rollback support via pre-merge commit tracking, a + spec-aligned Apply Summary panel (plan ID, artifacts, insertions/deletions, + project, timestamp), Sandbox Cleanup panel, and `✓ OK Changes applied` + footer. See [`docs/modules/git-worktree-sandbox.md`](docs/modules/git-worktree-sandbox.md). + +- **ACMS Context Tier Hydration** (#4219): Fixed a critical bug (#1028) where + `ContextTierService` started empty on every CLI invocation, causing the LLM + to receive zero file context during plan execution. A new + `ContextTierHydrator` reads files from linked project resources (via + `git ls-files` for git-checkout resources or `os.walk` for fs-directory + resources) and stores them as `TieredFragment` objects in the tier service. + Hydration runs automatically before context assembly in + `LLMExecuteActor.execute()`. Respects max file size (256 KB), total budget + (10 MB), binary file exclusion, and skip-directory rules. + See [`docs/modules/context-tier-hydrator.md`](docs/modules/context-tier-hydrator.md). + +- **Container/Devcontainer Resource Stop** (#3250): `agents resource stop` now + correctly stops `container-instance` and `devcontainer-instance` resource + types in addition to the previously supported types. The resource handler + dispatch table was extended to route stop requests to the container and + devcontainer lifecycle handlers. + - **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges LLM-generated changes via `git merge` from an isolated worktree branch instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary @@ -72,8 +99,25 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). strategies, and parallel subtask execution with wave-based dependency analysis. Pass rate improved from 48.15% to 84.8%. -- **Container Resource Stop Support**: `agents resource stop` now correctly stops - `container-instance` and `devcontainer-instance` resource types. +- **Container Resource Stop Support** (#3250): `agents resource stop` now correctly stops + `container-instance` and `devcontainer-instance` resource types in addition to the + previously supported types. The resource handler dispatch table now routes stop requests + to the container and devcontainer lifecycle handlers. + +### Fixed + +- **Plan action argument upsert** (#4197): Fixed a `UNIQUE constraint violation` + that occurred when `plan use` attempted to insert action arguments that already + existed in the database. The repository layer now uses an upsert (insert-or-update) + strategy so re-using a plan with the same action arguments succeeds idempotently. + +- **CI quality tests** (#4175): Restored all CI quality test sessions to passing + state after lint rule changes introduced regressions. Removed stale + `@tdd_expected_fail` tags from Behave scenarios that had been fixed. + +- **Validation attach CLI format** (#3837): `agents validation attach` extra + arguments now use `--key value` named option format instead of positional + arguments, matching the spec and other CLI commands. - **Centralized Automation Tracking Manager** (`automation-tracking-manager`): The manager is now the single interface for all tracking issue operations (`CREATE_TRACKING_ISSUE`, diff --git a/README.md b/README.md index 733986eda..ecf558b62 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,14 @@ embracing modern Python tooling. `validFrom`, and `isCurrent` metadata; a revision chain enables temporal queries - **JSON-RPC 2.0 A2A wire format** — `A2aRequest`/`A2aResponse` fields renamed to standard JSON-RPC 2.0 names (`method`, `id`, `result`, `error`) +- **Git worktree sandbox** — plan execute and apply phases use an isolated git worktree + for `git-checkout` resources; changes are committed on a dedicated branch and merged + back on `plan apply`; atomic rollback via pre-merge commit tracking +- **ACMS context tier hydration** — `ContextTierHydrator` reads files from linked project + resources at plan execution time so the LLM always receives file context; supports + `git ls-files` for git resources and `os.walk` for directory resources +- **Container/devcontainer resource stop** — `agents resource stop` now supports + `container-instance` and `devcontainer-instance` resource types - Fast Typer/Click-based interface with parity for help/version behavior - Behavior-driven coverage via Behave and Robot Framework - Nox automation for linting, typing, testing, docs, builds, and benchmarks diff --git a/docs/modules/context-tier-hydrator.md b/docs/modules/context-tier-hydrator.md new file mode 100644 index 000000000..b7fccfbf8 --- /dev/null +++ b/docs/modules/context-tier-hydrator.md @@ -0,0 +1,189 @@ +# Context Tier Hydrator Module + +**Package:** `cleveragents.application.services.context_tier_hydrator` +**Introduced:** v3.9.0 (PR #4219, fixes bug #1028) + +The Context Tier Hydrator bridges the gap between the resource registry (files +on disk) and the ACMS `ContextTierService` (in-memory fragments). Without +hydration, `ContextTierService` starts empty on every CLI process invocation, +causing the LLM to receive zero file context during plan execution. + +For the ACMS context tier architecture, see +[ADR-014](../adr/ADR-014-context-management-acms.md). +For the `ContextTierService` API, see [`docs/api/core.md`](../api/core.md). + +--- + +## Problem Solved + +The ACMS three-tier context system (`HOT`, `WARM`, `COLD`) stores file fragments +in memory via `ContextTierService`. In a long-running server process, these +fragments accumulate over time. However, the CLI spawns a fresh process for +every invocation, so the tier service always starts empty. + +Before this module, `LLMExecuteActor.execute()` would assemble context from an +empty tier service, sending the LLM a prompt with no file content. The hydrator +fixes this by reading files from linked project resources at the start of each +plan execution. + +--- + +## Automatic Invocation + +`LLMExecuteActor.execute()` calls `hydrate_tiers_for_plan()` automatically +before context assembly. No manual wiring is required. + +--- + +## Public API + +### `hydrate_tiers_from_project()` + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_from_project, +) + +count = hydrate_tiers_from_project( + tier_service=tier_service, + project_name="local/my-project", + resource_id="01HXYZ...", + resource_location="/home/user/my-project", + resource_type="git-checkout", # optional, default "git-checkout" +) +print(f"Stored {count} fragments") +``` + +Reads files from a single resource directory and stores them as +`TieredFragment` objects in the `ContextTierService`. + +**Parameters:** + +| Parameter | Type | Description | +|-----------|------|-------------| +| `tier_service` | `ContextTierService` | The tier service to populate | +| `project_name` | `str` | Namespaced project name (e.g. `local/my-project`) | +| `resource_id` | `str` | ULID of the resource | +| `resource_location` | `str` | Filesystem path to the resource root | +| `resource_type` | `str` | Resource type name — affects file listing strategy | + +**Returns:** Number of fragments stored. + +**File listing strategy:** + +- `git-checkout` / `git` resources: uses `git ls-files --cached --others + --exclude-standard` to list tracked and untracked (non-ignored) files +- All other resource types: uses `os.walk` with skip-directory filtering + +### `hydrate_tiers_for_plan()` + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_for_plan, +) + +total = hydrate_tiers_for_plan( + tier_service=tier_service, + project_names=["local/my-project", "local/other-project"], + project_repository=project_repo, + resource_registry=resource_registry_service, +) +print(f"Stored {total} fragments across all projects") +``` + +Hydrates tiers for all projects linked to a plan. Iterates over each project, +resolves its linked resources, and calls `hydrate_tiers_from_project()` for +each resource that has a filesystem location. + +**Parameters:** + +| Parameter | Type | Description | +|-----------|------|-------------| +| `tier_service` | `ContextTierService` | The tier service to populate | +| `project_names` | `list[str]` | Namespaced project names | +| `project_repository` | `NamespacedProjectRepository` | Repository for project lookup | +| `resource_registry` | `ResourceRegistryService` | Registry for resource lookup | + +**Returns:** Total number of fragments stored across all projects. + +--- + +## Limits and Filters + +The hydrator applies several limits to prevent memory exhaustion: + +| Limit | Value | Description | +|-------|-------|-------------| +| Max file size | 256 KB | Files larger than this are skipped | +| Max total bytes | 10 MB | Hydration stops when this budget is reached | +| Binary extensions | `.pyc`, `.so`, `.png`, `.pdf`, etc. | Skipped entirely | +| Skip directories | `.git`, `__pycache__`, `node_modules`, `.venv`, etc. | Not traversed | + +### Skipped Directories + +The following directories are never traversed: + +``` +.git .hg .svn __pycache__ node_modules .venv venv .nox .tox +.mypy_cache .pytest_cache .ruff_cache dist build .eggs .cleveragents +``` + +Additionally, any directory starting with `.` or ending with `.egg-info` is +skipped during `os.walk` traversal. + +### Skipped File Extensions + +``` +.pyc .pyo .so .o .a .dll .exe +.png .jpg .jpeg .gif .bmp .ico .pdf +.zip .tar .gz .bz2 .xz .whl .egg +.db .sqlite .sqlite3 +``` + +--- + +## Fragment Structure + +Each file is stored as a `TieredFragment` in the `HOT` tier: + +```python +TieredFragment( + fragment_id=f"{resource_id}:{rel_path}", # e.g. "01HXYZ...:src/main.py" + content=file_content, + tier=ContextTier.HOT, + resource_id=resource_id, + project_name=project_name, + token_count=len(content) // 4, # approximate token count + metadata={ + "path": rel_path, + "detail_depth": "1", + "relevance_score": "0.5", + }, +) +``` + +> **Note:** `detail_depth` and `relevance_score` are stored as strings (not +> int/float) to satisfy `ContextFragment` Pydantic validation requirements. +> This was a source of the original bug (#1028). + +--- + +## Logging + +The hydrator emits structured log events via `structlog`: + +| Event | Level | Description | +|-------|-------|-------------| +| `context_hydrator.skip_missing_location` | WARNING | Resource location does not exist on disk | +| `context_hydrator.project_not_found` | DEBUG | Project not found in repository | +| `context_hydrator.resource_not_found` | DEBUG | Resource not found in registry | +| `context_hydrator.store_failed` | DEBUG | Failed to store a fragment (non-fatal) | +| `context_hydrator.hydrated` | INFO | Hydration complete for a resource | + +--- + +## See Also + +- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md) — Three-tier context architecture +- [`docs/api/core.md`](../api/core.md) — `ContextTierService` API reference +- [`docs/modules/invariant-reconciliation.md`](invariant-reconciliation.md) — Related plan lifecycle module diff --git a/docs/modules/git-worktree-sandbox.md b/docs/modules/git-worktree-sandbox.md new file mode 100644 index 000000000..0add0c8fc --- /dev/null +++ b/docs/modules/git-worktree-sandbox.md @@ -0,0 +1,273 @@ +# Git Worktree Sandbox Module + +**Package:** `cleveragents.infrastructure.sandbox.git_worktree` +**Introduced:** v3.9.0 (PR #5998) + +The Git Worktree Sandbox provides fully isolated, git-native sandboxing for +`git-checkout` resources during plan execution and apply phases. Changes are +staged on a dedicated branch in a temporary git worktree and merged back to the +original branch only on a successful `plan apply`. If the plan is rolled back, +the merge is undone atomically. + +For the sandbox protocol and other sandbox strategies, see +[`docs/development/custom_sandbox_strategy.md`](../development/custom_sandbox_strategy.md). +For the plan lifecycle that invokes sandboxes, see +[ADR-006](../adr/ADR-006-plan-lifecycle.md) and +[ADR-015](../adr/ADR-015-sandbox-and-checkpoint.md). + +--- + +## Purpose + +When an actor executes a plan against a `git-checkout` resource, it must not +write changes directly to the working tree — doing so would corrupt the +repository state if the plan fails or is rolled back. The Git Worktree Sandbox +solves this by: + +1. Creating a temporary git worktree at a new branch (`cleveragents/plan-`) +2. Routing all actor file writes into the worktree +3. Committing changes in the worktree on `plan apply` +4. Merging the sandbox branch back to the original branch +5. Cleaning up the worktree and branch after the merge + +Non-git resources (e.g. `fs-directory`) continue to use the copy-on-write +sandbox strategy. + +--- + +## Lifecycle + +``` +GitWorktreeSandbox(resource_id, original_path) + │ + ▼ +sandbox.create(plan_id) # PENDING → CREATED + │ Creates worktree at /tmp/ca-sandbox--XXXX + │ Creates branch cleveragents/plan- + │ + ▼ +sandbox.get_path("src/foo.py") # CREATED/ACTIVE → ACTIVE + │ Returns /tmp/ca-sandbox-.../src/foo.py + │ Actor writes to this path + │ + ▼ +sandbox.commit("msg") # ACTIVE → COMMITTED + │ git add -A (in worktree) + │ git commit (in worktree) + │ git merge cleveragents/plan- (in original repo) + │ + ▼ +sandbox.cleanup() # → CLEANED_UP + git worktree remove --force + git branch -D cleveragents/plan- + git worktree prune +``` + +Rollback is possible from `ACTIVE` or `COMMITTED`: + +``` +sandbox.rollback() # ACTIVE/COMMITTED → ROLLED_BACK + │ If COMMITTED: git reset --hard (original repo) + │ git reset --hard (worktree) + │ git clean -fd (worktree) +``` + +--- + +## Key Class: `GitWorktreeSandbox` + +```python +from cleveragents.infrastructure.sandbox.git_worktree import GitWorktreeSandbox + +sandbox = GitWorktreeSandbox( + resource_id="01HXYZ...", + original_path="/home/user/my-project", + git_timeout=30, # optional, default 30 s +) +``` + +### Constructor Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `resource_id` | `str` | required | ULID of the resource being sandboxed | +| `original_path` | `str` | required | Absolute path to the git repository root | +| `git_timeout` | `int` | `30` | Timeout in seconds for each git command | + +### Methods + +#### `create(plan_id: str) → SandboxContext` + +Creates the worktree and sandbox branch. Must be called before any other +method. + +- Creates branch `cleveragents/plan-` from `HEAD` +- Creates a temporary directory at `/tmp/ca-sandbox--XXXX` +- Returns a `SandboxContext` with `sandbox_path` pointing to the worktree + +**Raises:** +- `ValueError` — if `plan_id` is empty +- `SandboxStateError` — if not in `PENDING` status +- `SandboxCreationError` — if git worktree creation fails (e.g. not a git repo, + git command timeout) + +#### `get_path(resource_path: str) → str` + +Translates a resource-relative path to its absolute path inside the worktree. + +```python +worktree_path = sandbox.get_path("src/cleveragents/cli/main.py") +# → "/tmp/ca-sandbox-abc123-XXXX/src/cleveragents/cli/main.py" +``` + +**Raises:** +- `SandboxStateError` — if sandbox is not in a usable status +- `ValueError` — if `resource_path` contains `..` (directory traversal) + +#### `commit(message: str | None = None) → CommitResult` + +Stages all changes in the worktree, commits them, and merges the sandbox branch +into the original branch. + +```python +result = sandbox.commit("feat: add new feature") +print(result.commit_ref) # git SHA of the sandbox commit +print(result.changed_files) # list of modified files +print(result.added_files) # list of new files +print(result.deleted_files) # list of deleted files +``` + +If there are no staged changes, returns a `CommitResult` with `success=True` +and empty file lists (no-op commit). + +**Raises:** +- `SandboxStateError` — if not in `CREATED` or `ACTIVE` status +- `SandboxCommitError` — if the git commit or merge fails + +#### `rollback() → None` + +Discards all worktree changes. + +- From `ACTIVE`: resets the worktree branch to the base commit +- From `COMMITTED`: additionally resets the original branch to the pre-merge + commit, fully undoing the merge + +> **Warning:** Rolling back from `COMMITTED` executes `git reset --hard` on +> the original branch. If other git worktrees or external processes are +> tracking the same branch, they will be affected. + +**Raises:** +- `SandboxStateError` — if called in an invalid status +- `SandboxRollbackError` — if the rollback fails + +#### `cleanup() → None` + +Removes the worktree directory and deletes the sandbox branch. Idempotent — +safe to call multiple times. + +Internally runs: +1. `git worktree remove --force ` +2. `git branch -D cleveragents/plan-` +3. `git worktree prune` + +Falls back to `shutil.rmtree` if `git worktree remove` fails. + +--- + +## `SandboxContext` Metadata + +After `create()`, the `sandbox.context.metadata` dict contains: + +| Key | Description | +|-----|-------------| +| `strategy` | Always `"git_worktree"` | +| `branch` | Sandbox branch name, e.g. `cleveragents/plan-abc123` | +| `original_branch` | Branch that was active before sandboxing | +| `base_commit` | SHA of the HEAD commit at sandbox creation time | +| `worktree_path` | Absolute path to the temporary worktree directory | + +--- + +## Status State Machine + +``` +PENDING + │ create() + ▼ +CREATED ──── get_path() ──► ACTIVE + │ │ + │ commit() │ commit() + ▼ ▼ +COMMITTED ◄─────────────── COMMITTED + │ │ + │ rollback() │ rollback() + ▼ ▼ +ROLLED_BACK ROLLED_BACK + │ │ + │ cleanup() │ cleanup() + ▼ ▼ +CLEANED_UP CLEANED_UP + +Any state ──► ERRORED (on git command failure) +``` + +--- + +## Error Reference + +| Exception | When raised | +|-----------|-------------| +| `SandboxCreationError` | `create()` fails (not a git repo, git timeout, etc.) | +| `SandboxCommitError` | `commit()` fails (git commit or merge error) | +| `SandboxRollbackError` | `rollback()` fails (git reset error) | +| `SandboxStateError` | Method called in wrong lifecycle state | + +All exceptions are importable from `cleveragents.infrastructure.sandbox.protocol`. + +--- + +## Integration with Plan Lifecycle + +The `SandboxFactory` selects `GitWorktreeSandbox` automatically when the +resource type is `git-checkout` and the sandbox strategy is `git_worktree`. +The `PlanApplyService` calls `sandbox.commit()` and then `sandbox.cleanup()` on +success, or `sandbox.rollback()` followed by `sandbox.cleanup()` on failure. + +The CLI `plan apply` command displays a spec-aligned Apply Summary panel: + +``` +╭─ Apply Summary ──────────────────────────────────────────────────╮ +│ Plan ID : 01HXYZ... │ +│ Project : local/my-project │ +│ Artifacts : 3 changed, 1 added, 0 deleted │ +│ Timestamp : 2026-04-09T12:34:56 │ +╰──────────────────────────────────────────────────────────────────╯ +╭─ Sandbox Cleanup ────────────────────────────────────────────────╮ +│ Worktree removed · Branch deleted · Prune complete │ +╰──────────────────────────────────────────────────────────────────╯ +✓ OK Changes applied +``` + +--- + +## Configuration + +The git command timeout defaults to 30 seconds. Override it by passing +`git_timeout` to the constructor, or configure it via the sandbox strategy +registry if using the factory. + +```python +sandbox = GitWorktreeSandbox( + resource_id=resource.id, + original_path=resource.location, + git_timeout=60, # increase for large repos on slow storage +) +``` + +--- + +## See Also + +- [`docs/development/custom_sandbox_strategy.md`](../development/custom_sandbox_strategy.md) — How to implement a custom sandbox strategy +- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) — Design rationale +- [ADR-038 Cross-Mechanism Sandbox Coordination](../adr/ADR-038-cross-mechanism-sandbox-coordination.md) — Multi-sandbox coordination diff --git a/mkdocs.yml b/mkdocs.yml index 2bb05974c..17d59b0ea 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,8 @@ nav: - Shell Safety: modules/shell-safety.md - UKO Provenance Tracking: modules/uko-provenance.md - Invariant Reconciliation: modules/invariant-reconciliation.md + - Git Worktree Sandbox: modules/git-worktree-sandbox.md + - Context Tier Hydrator: modules/context-tier-hydrator.md - Development: - Agent System Specification: development/agent-system-specification.md - CI/CD Pipeline: development/ci-cd.md -- 2.52.0 From 68aa27120572b616a4e0a157c62141fb94844f78 Mon Sep 17 00:00:00 2001 From: CleverThis Date: Fri, 10 Apr 2026 08:18:27 +0000 Subject: [PATCH 2/3] docs: address review feedback on module guides - deduplicate changelog coverage for git worktree sandbox and context hydration - clarify module guides with lifecycle notes, import examples, and tier rationale - add HAL 9000 to CONTRIBUTORS for documentation work ISSUES CLOSED: #7150 --- CHANGELOG.md | 85 +++++++-------------------- CONTRIBUTORS.md | 1 + docs/modules/context-tier-hydrator.md | 21 +++++++ docs/modules/git-worktree-sandbox.md | 5 +- 4 files changed, 46 insertions(+), 66 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c00c559f2..bd80be688 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,18 +34,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). dispatch table was extended to route stop requests to the container and devcontainer lifecycle handlers. -- **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges - LLM-generated changes via `git merge` from an isolated worktree branch - instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary - (plan ID, artifacts, insertions/deletions, project, timestamp), Sandbox - Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall - back to the original flat file copy. - -- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types - (`detail_depth` and `relevance_score` must be strings, not int/float) that - caused Pydantic validation errors during context assembly, resulting in the - LLM receiving zero file context. - - **Automation Tracking System**: Replaced shared session state issue tracking with individual per-agent tracking issues. Each agent now creates its own `[AUTO-]` titled issues with standardized headers, reporting intervals, and health indicators. @@ -119,24 +107,26 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). arguments now use `--key value` named option format instead of positional arguments, matching the spec and other CLI commands. -- **Centralized Automation Tracking Manager** (`automation-tracking-manager`): The manager - is now the single interface for all tracking issue operations (`CREATE_TRACKING_ISSUE`, - `UPDATE_TRACKING_ISSUE`, `CLOSE_TRACKING_ISSUE`, `READ_TRACKING_STATE`, - `GET_NEXT_CYCLE_NUMBER`). Agents delegate to the manager rather than calling the Forgejo - API directly, ensuring sequential cycle numbers across restarts and preventing - duplicate issues. Migrated agents include `system-watchdog`, - `implementation-pool-supervisor`, `timeline-update-pool-supervisor`, - `project-owner-pool-supervisor`, `product-builder`, and - `backlog-grooming-pool-supervisor`. The legacy `shared/automation_tracking.md` module - was removed. +- **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the + `tdd_expected_fail_listener` `end_test()` function to prevent blindly inverting ALL test + failures to passes, which was masking infrastructure errors and causing flaky CI behavior. + Guards include setup/teardown error detection, non-assertion failure detection (infrastructure + errors), and dry-run mode detection. Also fixed `Variable Should Exist` syntax errors in + e2e test files and removed `tdd_expected_fail` from four context assembly e2e tests where + bugs were already fixed. -- **Documentation Writer Tracking** (`docs-writer`): The documentation writer now - participates in the automation tracking system by creating individual - `[AUTO-DOCS] Documentation Report (Cycle N)` issues every 10 cycles (~3.3 hours). - The manager applies the mandatory `Automation Tracking` label automatically, while - teams may add additional workflow labels as needed. See - `docs/development/automation-tracking.md` and the new - `docs/development/docs-writer.md` reference. +- **`issue-state-updater` Bash Script Errors**: Removed problematic bash script examples + that attempted to invoke `task forgejo-label-manager` as a shell command. Replaced with + step-by-step operational instructions and direct label management via API. + +- **`automation-tracking-manager` Label Delegation Syntax**: Fixed incorrect delegation + syntax when calling `forgejo-label-manager`. The manager now uses natural-language + requests (for example, "Apply labels to issue #123: Automation Tracking") instead of + structured parameters, ensuring tracking issues receive proper labels. + +- **`product-builder` Missing Supervisors**: Added missing `pr-fix-pool-supervisor` and + `pr-merge-pool-supervisor` to the product-builder's supervisor launch list (18 total + supervisors). Updated all numeric references, pre-flight checklists, and validation logic. ### Changed @@ -157,17 +147,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Label Delegation Enforcement**: `automation-tracking-manager` now enforces delegation to `forgejo-label-manager` for all label operations, preventing "invalid label ID" errors - and ensuring label application uses correct name-to-ID mapping. - -- **Automation Tracking Label Guidance**: Documentation now clarifies that the manager - automatically applies the `Automation Tracking` label and that additional labels such as - `Type/Automation`, `State/In Progress`, or `Priority/Medium` remain optional workflow - choices rather than mandatory. - -- **Automation Tracking Agent Prefix Registry**: Expanded from 5 agents to 18 agents. - New prefixes include `AUTO-DOCS`, `AUTO-REV-POOL`, `AUTO-UAT-POOL`, `AUTO-BUG-POOL`, - `AUTO-INF-POOL`, `AUTO-ARCH`, `AUTO-EPIC`, `AUTO-EVLV`, `AUTO-GUARD`, `AUTO-SPEC`, - `AUTO-TIME`, `AUTO-PROJ-OWN`, and `AUTO-PROD-BLDR`. + and ensuring all tracking issues receive proper labels via name-to-ID mapping. - **ACMS Context Hydration**: Fixed ACMS indexing pipeline not wired into CLI — `ContextTierService` started empty on every CLI invocation so LLM received zero file @@ -177,31 +157,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). assembly in `LLMExecuteActor.execute()`. Respects max file size (256KB), total budget (10MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__` directory skipping. (#1028) - -### Fixed - -- **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the - `tdd_expected_fail_listener` `end_test()` function to prevent blindly inverting ALL test - failures to passes, which was masking infrastructure errors and causing flaky CI behavior. - Guards: setup/teardown error detection, non-assertion failure detection (infrastructure - errors), and dry-run mode detection. Also fixed `Variable Should Exist` syntax errors in - e2e test files and removed `tdd_expected_fail` from 4 context assembly e2e tests where - bugs were already fixed. - -- **`issue-state-updater` Bash Script Errors**: Removed problematic bash script examples - that tried to invoke `task forgejo-label-manager` as a bash command (the Task tool cannot - be invoked from bash). Replaced with clear step-by-step operational instructions and - direct label management via API. - -- **`automation-tracking-manager` Label Delegation Syntax**: Fixed incorrect delegation - syntax when calling `forgejo-label-manager`. The manager now uses correct natural language - requests (e.g., "Apply labels to issue #123: Automation Tracking") instead of structured - parameters, ensuring tracking issues receive proper labels. - -- **`product-builder` Missing Supervisors**: Added missing `pr-fix-pool-supervisor` and - `pr-merge-pool-supervisor` to the product-builder's supervisor launch list (18 total - supervisors). Updated all numeric references, pre-flight checklists, and validation logic. - --- ## [3.8.0] — 2026-04-05 diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index f5091deaa..4ff595ff8 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -3,6 +3,7 @@ * Aditya Chhabra * Brent E. Edwards * Hamza Khyari +* HAL 9000 * Jeffrey Phillips Freeman * Luis Mendes * Rui Hu diff --git a/docs/modules/context-tier-hydrator.md b/docs/modules/context-tier-hydrator.md index b7fccfbf8..f6111e888 100644 --- a/docs/modules/context-tier-hydrator.md +++ b/docs/modules/context-tier-hydrator.md @@ -8,6 +8,10 @@ on disk) and the ACMS `ContextTierService` (in-memory fragments). Without hydration, `ContextTierService` starts empty on every CLI process invocation, causing the LLM to receive zero file context during plan execution. +> **Naming note:** The implementation is a function-based module; there is no +> `ContextTierHydrator` class. Documentation and changelog references use the +> module name as shorthand for the public functions exported here. + For the ACMS context tier architecture, see [ADR-014](../adr/ADR-014-context-management-acms.md). For the `ContextTierService` API, see [`docs/api/core.md`](../api/core.md). @@ -81,6 +85,15 @@ Reads files from a single resource directory and stores them as from cleveragents.application.services.context_tier_hydrator import ( hydrate_tiers_for_plan, ) +from cleveragents.application.services.resource_registry_service import ( + ResourceRegistryService, +) +from cleveragents.infrastructure.database.repositories import ( + NamespacedProjectRepository, +) + +project_repo: NamespacedProjectRepository = ... +resource_registry_service: ResourceRegistryService = ... total = hydrate_tiers_for_plan( tier_service=tier_service, @@ -104,6 +117,9 @@ each resource that has a filesystem location. | `project_repository` | `NamespacedProjectRepository` | Repository for project lookup | | `resource_registry` | `ResourceRegistryService` | Registry for resource lookup | +`NamespacedProjectRepository` is defined in `cleveragents.infrastructure.database.repositories`, and +`ResourceRegistryService` lives in `cleveragents.application.services.resource_registry_service`. + **Returns:** Total number of fragments stored across all projects. --- @@ -166,6 +182,11 @@ TieredFragment( > int/float) to satisfy `ContextFragment` Pydantic validation requirements. > This was a source of the original bug (#1028). +All fragments are written to the `HOT` tier so that plan execution has +immediate access to freshly hydrated context. Tier management services may +subsequently demote fragments to `WARM` or `COLD` tiers according to budget and +aging policies. + --- ## Logging diff --git a/docs/modules/git-worktree-sandbox.md b/docs/modules/git-worktree-sandbox.md index 0add0c8fc..4935f06c5 100644 --- a/docs/modules/git-worktree-sandbox.md +++ b/docs/modules/git-worktree-sandbox.md @@ -45,6 +45,9 @@ sandbox.create(plan_id) # PENDING → CREATED │ Creates worktree at /tmp/ca-sandbox--XXXX │ Creates branch cleveragents/plan- │ + ├─ sandbox.commit("msg") # CREATED → COMMITTED (no staged changes) + │ Returns CommitResult(success=True) with empty diffs when nothing is staged + │ ▼ sandbox.get_path("src/foo.py") # CREATED/ACTIVE → ACTIVE │ Returns /tmp/ca-sandbox-.../src/foo.py @@ -58,7 +61,7 @@ sandbox.commit("msg") # ACTIVE → COMMITTED │ ▼ sandbox.cleanup() # → CLEANED_UP - git worktree remove --force + git worktree remove --force (falls back to shutil.rmtree on failure) git branch -D cleveragents/plan- git worktree prune ``` -- 2.52.0 From 842fd49c4a3d8617717756eccc794153514fb933 Mon Sep 17 00:00:00 2001 From: CleverThis Date: Sat, 11 Apr 2026 02:14:56 +0000 Subject: [PATCH 3/3] docs: drop context-tier hydrator overlap --- CHANGELOG.md | 19 --- README.md | 3 - docs/modules/context-tier-hydrator.md | 210 -------------------------- mkdocs.yml | 1 - 4 files changed, 233 deletions(-) delete mode 100644 docs/modules/context-tier-hydrator.md diff --git a/CHANGELOG.md b/CHANGELOG.md index bd80be688..a2a8c8624 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,17 +17,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). project, timestamp), Sandbox Cleanup panel, and `✓ OK Changes applied` footer. See [`docs/modules/git-worktree-sandbox.md`](docs/modules/git-worktree-sandbox.md). -- **ACMS Context Tier Hydration** (#4219): Fixed a critical bug (#1028) where - `ContextTierService` started empty on every CLI invocation, causing the LLM - to receive zero file context during plan execution. A new - `ContextTierHydrator` reads files from linked project resources (via - `git ls-files` for git-checkout resources or `os.walk` for fs-directory - resources) and stores them as `TieredFragment` objects in the tier service. - Hydration runs automatically before context assembly in - `LLMExecuteActor.execute()`. Respects max file size (256 KB), total budget - (10 MB), binary file exclusion, and skip-directory rules. - See [`docs/modules/context-tier-hydrator.md`](docs/modules/context-tier-hydrator.md). - - **Container/Devcontainer Resource Stop** (#3250): `agents resource stop` now correctly stops `container-instance` and `devcontainer-instance` resource types in addition to the previously supported types. The resource handler @@ -149,14 +138,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). to `forgejo-label-manager` for all label operations, preventing "invalid label ID" errors and ensuring all tracking issues receive proper labels via name-to-ID mapping. -- **ACMS Context Hydration**: Fixed ACMS indexing pipeline not wired into CLI — - `ContextTierService` started empty on every CLI invocation so LLM received zero file - context during plan execution. Added `context_tier_hydrator.py` that reads files from - linked project resources (via `git ls-files` or `os.walk`) and stores them as - `TieredFragment` objects in the tier service. Hydration runs automatically before context - assembly in `LLMExecuteActor.execute()`. Respects max file size (256KB), total budget - (10MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__` directory - skipping. (#1028) --- ## [3.8.0] — 2026-04-05 diff --git a/README.md b/README.md index ecf558b62..6df9cd01c 100644 --- a/README.md +++ b/README.md @@ -44,9 +44,6 @@ embracing modern Python tooling. - **Git worktree sandbox** — plan execute and apply phases use an isolated git worktree for `git-checkout` resources; changes are committed on a dedicated branch and merged back on `plan apply`; atomic rollback via pre-merge commit tracking -- **ACMS context tier hydration** — `ContextTierHydrator` reads files from linked project - resources at plan execution time so the LLM always receives file context; supports - `git ls-files` for git resources and `os.walk` for directory resources - **Container/devcontainer resource stop** — `agents resource stop` now supports `container-instance` and `devcontainer-instance` resource types - Fast Typer/Click-based interface with parity for help/version behavior diff --git a/docs/modules/context-tier-hydrator.md b/docs/modules/context-tier-hydrator.md deleted file mode 100644 index f6111e888..000000000 --- a/docs/modules/context-tier-hydrator.md +++ /dev/null @@ -1,210 +0,0 @@ -# Context Tier Hydrator Module - -**Package:** `cleveragents.application.services.context_tier_hydrator` -**Introduced:** v3.9.0 (PR #4219, fixes bug #1028) - -The Context Tier Hydrator bridges the gap between the resource registry (files -on disk) and the ACMS `ContextTierService` (in-memory fragments). Without -hydration, `ContextTierService` starts empty on every CLI process invocation, -causing the LLM to receive zero file context during plan execution. - -> **Naming note:** The implementation is a function-based module; there is no -> `ContextTierHydrator` class. Documentation and changelog references use the -> module name as shorthand for the public functions exported here. - -For the ACMS context tier architecture, see -[ADR-014](../adr/ADR-014-context-management-acms.md). -For the `ContextTierService` API, see [`docs/api/core.md`](../api/core.md). - ---- - -## Problem Solved - -The ACMS three-tier context system (`HOT`, `WARM`, `COLD`) stores file fragments -in memory via `ContextTierService`. In a long-running server process, these -fragments accumulate over time. However, the CLI spawns a fresh process for -every invocation, so the tier service always starts empty. - -Before this module, `LLMExecuteActor.execute()` would assemble context from an -empty tier service, sending the LLM a prompt with no file content. The hydrator -fixes this by reading files from linked project resources at the start of each -plan execution. - ---- - -## Automatic Invocation - -`LLMExecuteActor.execute()` calls `hydrate_tiers_for_plan()` automatically -before context assembly. No manual wiring is required. - ---- - -## Public API - -### `hydrate_tiers_from_project()` - -```python -from cleveragents.application.services.context_tier_hydrator import ( - hydrate_tiers_from_project, -) - -count = hydrate_tiers_from_project( - tier_service=tier_service, - project_name="local/my-project", - resource_id="01HXYZ...", - resource_location="/home/user/my-project", - resource_type="git-checkout", # optional, default "git-checkout" -) -print(f"Stored {count} fragments") -``` - -Reads files from a single resource directory and stores them as -`TieredFragment` objects in the `ContextTierService`. - -**Parameters:** - -| Parameter | Type | Description | -|-----------|------|-------------| -| `tier_service` | `ContextTierService` | The tier service to populate | -| `project_name` | `str` | Namespaced project name (e.g. `local/my-project`) | -| `resource_id` | `str` | ULID of the resource | -| `resource_location` | `str` | Filesystem path to the resource root | -| `resource_type` | `str` | Resource type name — affects file listing strategy | - -**Returns:** Number of fragments stored. - -**File listing strategy:** - -- `git-checkout` / `git` resources: uses `git ls-files --cached --others - --exclude-standard` to list tracked and untracked (non-ignored) files -- All other resource types: uses `os.walk` with skip-directory filtering - -### `hydrate_tiers_for_plan()` - -```python -from cleveragents.application.services.context_tier_hydrator import ( - hydrate_tiers_for_plan, -) -from cleveragents.application.services.resource_registry_service import ( - ResourceRegistryService, -) -from cleveragents.infrastructure.database.repositories import ( - NamespacedProjectRepository, -) - -project_repo: NamespacedProjectRepository = ... -resource_registry_service: ResourceRegistryService = ... - -total = hydrate_tiers_for_plan( - tier_service=tier_service, - project_names=["local/my-project", "local/other-project"], - project_repository=project_repo, - resource_registry=resource_registry_service, -) -print(f"Stored {total} fragments across all projects") -``` - -Hydrates tiers for all projects linked to a plan. Iterates over each project, -resolves its linked resources, and calls `hydrate_tiers_from_project()` for -each resource that has a filesystem location. - -**Parameters:** - -| Parameter | Type | Description | -|-----------|------|-------------| -| `tier_service` | `ContextTierService` | The tier service to populate | -| `project_names` | `list[str]` | Namespaced project names | -| `project_repository` | `NamespacedProjectRepository` | Repository for project lookup | -| `resource_registry` | `ResourceRegistryService` | Registry for resource lookup | - -`NamespacedProjectRepository` is defined in `cleveragents.infrastructure.database.repositories`, and -`ResourceRegistryService` lives in `cleveragents.application.services.resource_registry_service`. - -**Returns:** Total number of fragments stored across all projects. - ---- - -## Limits and Filters - -The hydrator applies several limits to prevent memory exhaustion: - -| Limit | Value | Description | -|-------|-------|-------------| -| Max file size | 256 KB | Files larger than this are skipped | -| Max total bytes | 10 MB | Hydration stops when this budget is reached | -| Binary extensions | `.pyc`, `.so`, `.png`, `.pdf`, etc. | Skipped entirely | -| Skip directories | `.git`, `__pycache__`, `node_modules`, `.venv`, etc. | Not traversed | - -### Skipped Directories - -The following directories are never traversed: - -``` -.git .hg .svn __pycache__ node_modules .venv venv .nox .tox -.mypy_cache .pytest_cache .ruff_cache dist build .eggs .cleveragents -``` - -Additionally, any directory starting with `.` or ending with `.egg-info` is -skipped during `os.walk` traversal. - -### Skipped File Extensions - -``` -.pyc .pyo .so .o .a .dll .exe -.png .jpg .jpeg .gif .bmp .ico .pdf -.zip .tar .gz .bz2 .xz .whl .egg -.db .sqlite .sqlite3 -``` - ---- - -## Fragment Structure - -Each file is stored as a `TieredFragment` in the `HOT` tier: - -```python -TieredFragment( - fragment_id=f"{resource_id}:{rel_path}", # e.g. "01HXYZ...:src/main.py" - content=file_content, - tier=ContextTier.HOT, - resource_id=resource_id, - project_name=project_name, - token_count=len(content) // 4, # approximate token count - metadata={ - "path": rel_path, - "detail_depth": "1", - "relevance_score": "0.5", - }, -) -``` - -> **Note:** `detail_depth` and `relevance_score` are stored as strings (not -> int/float) to satisfy `ContextFragment` Pydantic validation requirements. -> This was a source of the original bug (#1028). - -All fragments are written to the `HOT` tier so that plan execution has -immediate access to freshly hydrated context. Tier management services may -subsequently demote fragments to `WARM` or `COLD` tiers according to budget and -aging policies. - ---- - -## Logging - -The hydrator emits structured log events via `structlog`: - -| Event | Level | Description | -|-------|-------|-------------| -| `context_hydrator.skip_missing_location` | WARNING | Resource location does not exist on disk | -| `context_hydrator.project_not_found` | DEBUG | Project not found in repository | -| `context_hydrator.resource_not_found` | DEBUG | Resource not found in registry | -| `context_hydrator.store_failed` | DEBUG | Failed to store a fragment (non-fatal) | -| `context_hydrator.hydrated` | INFO | Hydration complete for a resource | - ---- - -## See Also - -- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md) — Three-tier context architecture -- [`docs/api/core.md`](../api/core.md) — `ContextTierService` API reference -- [`docs/modules/invariant-reconciliation.md`](invariant-reconciliation.md) — Related plan lifecycle module diff --git a/mkdocs.yml b/mkdocs.yml index 17d59b0ea..921fc43e5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -28,7 +28,6 @@ nav: - UKO Provenance Tracking: modules/uko-provenance.md - Invariant Reconciliation: modules/invariant-reconciliation.md - Git Worktree Sandbox: modules/git-worktree-sandbox.md - - Context Tier Hydrator: modules/context-tier-hydrator.md - Development: - Agent System Specification: development/agent-system-specification.md - CI/CD Pipeline: development/ci-cd.md -- 2.52.0