diff --git a/CHANGELOG.md b/CHANGELOG.md index 89d67e2c3..0b62dc612 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,11 +14,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall back to the original flat file copy. -- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types - (`detail_depth` and `relevance_score` must be strings, not int/float) that - caused Pydantic validation errors during context assembly, resulting in the - LLM receiving zero file context. - - **Automation Tracking System**: Replaced shared session state issue tracking with individual per-agent tracking issues. Each agent now creates its own `[AUTO-]` titled issues with standardized headers, reporting intervals, and health indicators. @@ -125,6 +120,12 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). `AUTO-INF-POOL`, `AUTO-ARCH`, `AUTO-EPIC`, `AUTO-EVLV`, `AUTO-GUARD`, `AUTO-SPEC`, `AUTO-TIME`, `AUTO-PROJ-OWN`, and `AUTO-PROD-BLDR`. +- **Implementation Orchestrator**: Updated to use `async-agent-starter` for worker + dispatch, enabling proper async agent lifecycle management. + +- **Label Management Hardening**: Label creation is now restricted to the + `forgejo-label-manager` subagent; all other agents must delegate label operations + to prevent unauthorized label proliferation. ### Fixed - **Validation Gate Empty-Run Guard** (#7508): Fixed `ApplyValidationSummary.all_required_passed` @@ -169,6 +170,22 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). `pr-merge-pool-supervisor` to the product-builder's supervisor launch list (18 total supervisors). Updated all numeric references, pre-flight checklists, and validation logic. +- **ACMS Indexing Pipeline** (#1028): `ContextTierService` was never populated on + CLI invocations, causing the LLM to receive zero file context during plan execution. + Added `context_tier_hydrator.py` that reads files from linked project resources + (via `git ls-files` or `os.walk`) and stores them as `TieredFragment` objects in + the tier service. Hydration runs automatically before context assembly in + `LLMExecuteActor.execute()`. Respects max file size (256 KB), total budget (10 MB), + binary file exclusion, and skip-directory list (`.git`, `node_modules`, + `__pycache__`, etc.). + +- **Plan Action Arguments** (#4197): Upsert action arguments during `plan use` to + avoid `UNIQUE constraint violation` when the same action is used across multiple + plan invocations. + +- **CI Quality Tests**: Resolved lint errors and removed stale `@tdd_expected_fail` + tag that was masking a passing test (#5264). + --- ## [3.8.0] — 2026-04-05 diff --git a/docs/api/acms.md b/docs/api/acms.md new file mode 100644 index 000000000..5424f1045 --- /dev/null +++ b/docs/api/acms.md @@ -0,0 +1,181 @@ +# `cleveragents.application.services` — ACMS Services + +This page documents the Advanced Context Management System (ACMS) application +services, including the context tier hydrator that bridges the resource +registry with the in-memory context tier service. + +For the ACMS architecture overview, see +[Architecture — Context Management](../architecture.md#context-management-acms). +For the ADR, see +[ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md). + +--- + +## `context_tier_hydrator` — ACMS Indexing Pipeline + +**Module:** `cleveragents.application.services.context_tier_hydrator` +**Introduced:** v3.4.0 — fixes bug #1028 + +### Problem Solved + +Without this module, `ContextTierService` starts empty on every CLI process +invocation. The LLM receives zero file context during plan execution because +the tier service is never populated from the project's linked resources. + +`context_tier_hydrator` bridges the gap: it reads files from the resource +registry and stores them as `TieredFragment` objects in the tier service, +so the LLM always has project context available. + +--- + +### Functions + +#### `hydrate_tiers_for_plan` + +```python +def hydrate_tiers_for_plan( + tier_service: ContextTierService, + project_names: list[str], + project_repository: Any, + resource_registry: Any, +) -> int: +``` + +High-level entry point. Hydrates tiers for all projects linked to a plan. +Called automatically by `LLMExecuteActor.execute()` before context assembly. + +**Parameters:** + +| Parameter | Type | Description | +|-----------|------|-------------| +| `tier_service` | `ContextTierService` | The service to populate with fragments | +| `project_names` | `list[str]` | Namespaced project names (e.g. `["local/my-project"]`) | +| `project_repository` | `NamespacedProjectRepository` | Provides project → linked resource lookup | +| `resource_registry` | `ResourceRegistryService` | Provides resource location and type | + +**Returns:** Total number of `TieredFragment` objects stored across all projects. + +!!! note "Implementation detail" + The production implementation annotates `project_repository` and + `resource_registry` as `Any` to accommodate dependency-injection wiring. + At runtime these parameters are instances of `NamespacedProjectRepository` + and `ResourceRegistryService` respectively. Future refactors will tighten + the function signature once the container exposes stable protocols. + +**Example:** + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_for_plan, +) + +count = hydrate_tiers_for_plan( + tier_service=container.context_tier_service(), + project_names=["local/my-project"], + project_repository=container.project_repository(), + resource_registry=container.resource_registry(), +) +print(f"Hydrated {count} fragments") +``` + +--- + +#### `hydrate_tiers_from_project` + +```python +def hydrate_tiers_from_project( + tier_service: ContextTierService, + project_name: str, + resource_id: str, + resource_location: str, + resource_type: str = "git-checkout", +) -> int: +``` + +Lower-level function. Reads files from a single resource directory and stores +them as `TieredFragment` objects in `ContextTier.HOT`. + +**Parameters:** + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `tier_service` | `ContextTierService` | — | Service to populate | +| `project_name` | `str` | — | Namespaced project name | +| `resource_id` | `str` | — | ULID of the resource | +| `resource_location` | `str` | — | Filesystem path to the resource root | +| `resource_type` | `str` | `"git-checkout"` | Affects file listing strategy | + +**Returns:** Number of fragments stored. + +**File listing strategy:** + +- For `git-checkout` / `git` resources: uses `git ls-files --cached --others --exclude-standard` +- For all other resource types: falls back to `os.walk` + +--- + +### Limits and Filters + +| Limit | Value | Description | +|-------|-------|-------------| +| Max file size | 256 KB | Files larger than this are skipped | +| Max total bytes | 10 MB | Hydration stops when this budget is reached | +| Binary extensions | `.pyc`, `.so`, `.png`, `.pdf`, `.zip`, etc. | Skipped entirely | +| Skip directories | `.git`, `__pycache__`, `node_modules`, `.venv`, `dist`, `build`, etc. | Not traversed | + +--- + +### Fragment Metadata + +Each `TieredFragment` stored by the hydrator has the following metadata: + +```python +metadata = { + "path": "src/cleveragents/foo.py", # relative path within resource + "detail_depth": "1", # string (Pydantic requirement) + "relevance_score": "0.5", # string (Pydantic requirement) +} +``` + +!!! note "String metadata types" + `detail_depth` and `relevance_score` must be strings, not `int`/`float`. + Using numeric types causes a Pydantic validation error during context + assembly (bug #1028). + +--- + +### Automatic Invocation + +`hydrate_tiers_for_plan` is called automatically by `LLMExecuteActor.execute()` +before context assembly. No manual invocation is required in normal usage. + +``` +LLMExecuteActor.execute() + │ + ├─ hydrate_tiers_for_plan(...) ← populates ContextTierService + │ + └─ ContextAssembler.assemble() ← now has file context available +``` + +--- + +### Structured Log Events + +The hydrator emits structured log events at `DEBUG` and `INFO` levels: + +| Event | Level | Description | +|-------|-------|-------------| +| `context_hydrator.skip_missing_location` | `WARNING` | Resource location does not exist | +| `context_hydrator.project_not_found` | `DEBUG` | Project not found in repository | +| `context_hydrator.no_linked_resources` | `DEBUG` | Project has no linked resources | +| `context_hydrator.resource_not_found` | `DEBUG` | Resource not found in registry | +| `context_hydrator.store_failed` | `DEBUG` | Failed to store a fragment | +| `context_hydrator.hydrated` | `INFO` | Hydration complete — includes `fragments_stored` and `total_bytes` | + +--- + +### Related Documentation + +- [Architecture — Context Management (ACMS)](../architecture.md#context-management-acms) +- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md) +- [Modules — UKO Provenance Tracking](../modules/uko-provenance.md) diff --git a/docs/api/index.md b/docs/api/index.md index 4eee7341e..1aab31712 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -18,6 +18,7 @@ classes, functions, and exceptions with signatures and usage examples. | [`cleveragents.config`](config.md) | Application settings, logging, metrics, and security scanning | | [`cleveragents.providers`](providers.md) | AI provider registry — discovery, selection, LLM factory, and capability metadata | | [`cleveragents.tui`](tui.md) | Interactive Terminal UI — app, persona system, input routing, slash commands, session export/import | +| [ACMS Services](acms.md) | Advanced Context Management System services — context tier hydrator, fragment indexing pipeline | > **Note:** Internal modules (prefixed with `_`) and implementation details > not listed in a module's `__all__` are considered private and may change diff --git a/docs/architecture.md b/docs/architecture.md index e355f0c5e..b2e8e7d71 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -206,6 +206,13 @@ LLM at each phase ([ADR-014](adr/ADR-014-context-management-acms.md)): The **UKO Runtime** (Universal Knowledge Ontology) provides graph-based knowledge inference and persistence for Tier 3 context strategies. +The **Context Tier Hydrator** (`context_tier_hydrator.py`) bridges the +resource registry and the in-memory tier service: it reads files from linked +project resources (via `git ls-files` or `os.walk`) and stores them as +`TieredFragment` objects before context assembly. Without it, the tier service +starts empty on every CLI invocation and the LLM receives no file context. +See [`docs/api/acms.md`](api/acms.md) for the full API reference. + Key UKO capabilities: - **Typed triples with provenance** — every triple carries `sourceResource`, @@ -259,6 +266,42 @@ result = actor.run(plan_id="...", project_name="...", action_name="...") --- +## Git Worktree Sandbox + +The **Git Worktree Sandbox** (`cleveragents.infrastructure.sandbox.git_worktree`) +provides git-native isolation for the Execute and Apply phases of the plan +lifecycle when the project resource is a `git-checkout`. + +Instead of copying files to a temporary directory, it creates a real git +worktree on a dedicated branch (`cleveragents/plan-`). Changes are +committed in the worktree and merged back to the original branch on Apply, or +discarded entirely on rollback. + +``` +Execute phase: + GitWorktreeSandbox.create(plan_id) + → creates branch cleveragents/plan- + → creates worktree at /tmp/ca-sandbox--/ + LLM writes files into worktree + GitWorktreeSandbox.commit("...") + → git add -A && git commit (in worktree) + → git merge cleveragents/plan- (in original repo) + +Apply phase: + Displays Apply Summary (plan ID, artifacts, insertions/deletions) + Displays Sandbox Cleanup panel + GitWorktreeSandbox.cleanup() + → git worktree remove --force + → git branch -D cleveragents/plan- +``` + +Non-git resources fall back to the original flat-copy strategy. + +For full API documentation, see +[Modules — Git Worktree Sandbox](modules/git-worktree-sandbox.md). + +--- + ## A2A Protocol Boundary All CLI, TUI, and server interactions cross the A2A boundary diff --git a/docs/modules/git-worktree-sandbox.md b/docs/modules/git-worktree-sandbox.md new file mode 100644 index 000000000..225ad6ebf --- /dev/null +++ b/docs/modules/git-worktree-sandbox.md @@ -0,0 +1,269 @@ +# Git Worktree Sandbox Module + +**Package:** `cleveragents.infrastructure.sandbox.git_worktree` +**Introduced:** v3.5.0 + +The Git Worktree Sandbox provides isolated, git-native sandboxing for +`git-checkout` resources. Instead of copying files to a temporary directory, +it creates a real git worktree on a dedicated branch. Changes are committed +in the worktree and merged back to the original branch on `commit`, or +discarded entirely on `rollback`. + +For the sandbox protocol interface, see +[`docs/development/custom_sandbox_strategy.md`](../development/custom_sandbox_strategy.md). +For the sandbox ADR, see +[`ADR-015 Sandbox & Checkpoint`](../adr/ADR-015-sandbox-and-checkpoint.md). + +--- + +## Purpose + +The previous flat-copy sandbox (`shutil.copy2`) had no awareness of git +history, making it impossible to produce meaningful diffs or leverage git's +merge machinery. The git worktree sandbox: + +- Preserves full git history for every plan's changes +- Enables `git merge` for conflict detection and resolution +- Produces accurate insertions/deletions counts for the Apply Summary panel +- Falls back gracefully to flat-copy for non-git resources + +--- + +## Lifecycle + +``` +GitWorktreeSandbox(resource_id, original_path) + │ + ▼ +sandbox.create(plan_id) + │ Creates branch cleveragents/plan- + │ Creates worktree at /tmp/ca-sandbox--/ + ▼ +sandbox.get_path("src/foo.py") + │ Returns /tmp/ca-sandbox-.../src/foo.py + │ Actor writes changes here + ▼ +sandbox.commit("feat: ...") + │ git add -A (in worktree) + │ git commit (in worktree) + │ git merge cleveragents/plan- (in original repo) + ▼ +sandbox.cleanup() + │ git worktree remove --force + │ git branch -D cleveragents/plan- + │ git worktree prune + ▼ + CLEANED_UP +``` + +**Rollback path:** + +``` +sandbox.rollback() + │ If COMMITTED: git reset --hard (original repo) + │ git reset --hard (worktree) + │ git clean -fd (worktree) + ▼ + ROLLED_BACK +``` + +--- + +## Status State Machine + +| Status | Description | +|--------|-------------| +| `PENDING` | Sandbox created but `create()` not yet called | +| `CREATED` | Worktree and branch exist; no writes yet | +| `ACTIVE` | `get_path()` called; actor is writing | +| `COMMITTED` | Changes committed and merged to original branch | +| `ROLLED_BACK` | Changes discarded; worktree reset to base commit | +| `ERRORED` | A git command failed; sandbox is unusable | +| `CLEANED_UP` | Worktree and branch removed | + +--- + +## Key Class: `GitWorktreeSandbox` + +```python +from cleveragents.infrastructure.sandbox.git_worktree import GitWorktreeSandbox + +sandbox = GitWorktreeSandbox( + resource_id="res-01JXYZ", + original_path="/path/to/project", + git_timeout=30, # seconds; default +) +``` + +### Constructor Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `resource_id` | `str` | — | Identifier of the resource being sandboxed | +| `original_path` | `str` | — | Absolute path to the git repository root | +| `git_timeout` | `int` | `30` | Timeout in seconds for each git command | + +### Methods + +#### `create(plan_id: str) → SandboxContext` + +Creates the worktree and branch. Must be called before any other method. + +- Branch name: `cleveragents/plan-` +- Worktree path: `/tmp/ca-sandbox--/` +- `SandboxContext.metadata` includes `strategy`, `branch`, `original_branch`, + `base_commit`, and `worktree_path` + +**Raises:** `SandboxCreationError` if git commands fail; `SandboxStateError` +if not in `PENDING` status. + +#### `get_path(resource_path: str) → str` + +Translates a resource-relative path to its worktree equivalent. + +```python +worktree_file = sandbox.get_path("src/cleveragents/foo.py") +# → /tmp/ca-sandbox-.../src/cleveragents/foo.py +``` + +**Raises:** `ValueError` on path traversal (`..` components); `SandboxStateError` +if sandbox is not usable. + +#### `commit(message: str | None = None) → CommitResult` + +Stages all changes, commits in the worktree, then merges back to the original +branch. Returns a `CommitResult` with `changed_files`, `added_files`, +`deleted_files`, and `commit_ref`. + +If there are no staged changes, returns a no-op `CommitResult` with the base +commit ref and empty file lists. + +**Raises:** `SandboxCommitError` on git failure; `SandboxStateError` if not +in `CREATED` or `ACTIVE` status. + +#### `rollback() → None` + +Discards all changes: + +- From `ACTIVE`: resets the worktree branch to `base_commit` +- From `COMMITTED`: also resets the original branch to `pre_merge_commit` + (undoes the merge) + +!!! warning "Multi-worktree safety" + Rolling back from `COMMITTED` executes `git reset --hard` on the original + branch. Other git worktrees or external processes tracking the same branch + will be affected. + +**Raises:** `SandboxRollbackError` on git failure. + +#### `cleanup() → None` + +Removes the worktree directory and deletes the sandbox branch. Idempotent — +safe to call multiple times. Falls back to `shutil.rmtree` if +`git worktree remove` fails. + +--- + +## CommitResult + +```python +@dataclass +class CommitResult: + sandbox_id: str + success: bool + commit_ref: str | None # SHA of the commit in the worktree + changed_files: list[str] # Modified files (M status) + added_files: list[str] # New files (A status) + deleted_files: list[str] # Deleted files (D status) + error: str | None + timestamp: datetime +``` + +--- + +## Error Types + +| Exception | When raised | +|-----------|-------------| +| `SandboxCreationError` | `git worktree add` or `git rev-parse` fails | +| `SandboxCommitError` | `git commit` or `git merge` fails | +| `SandboxRollbackError` | `git reset --hard` or `git clean` fails | +| `SandboxStateError` | Method called in wrong lifecycle state | + +All exceptions are subclasses of `SandboxError` from +`cleveragents.infrastructure.sandbox.protocol`. + +--- + +## Non-Git Fallback + +For resources that are not git repositories, `plan apply` falls back to the +original flat-copy strategy (`shutil.copy2`). The `GitWorktreeSandbox` itself +raises `SandboxCreationError` if the `original_path` is not the root of a git +repository (verified via `git rev-parse --show-toplevel`). + +--- + +## Usage Example + +```python +from cleveragents.infrastructure.sandbox.git_worktree import GitWorktreeSandbox + +sandbox = GitWorktreeSandbox( + resource_id="res-01JXYZ", + original_path="/home/user/my-project", +) + +ctx = sandbox.create(plan_id="plan-01JABC") +print(f"Worktree at: {ctx.sandbox_path}") +print(f"Branch: {ctx.metadata['branch']}") + +# Actor writes files into the worktree +import pathlib +target = pathlib.Path(sandbox.get_path("src/new_feature.py")) +target.write_text("# new feature\n") + +# Commit and merge back +result = sandbox.commit("feat: add new_feature module") +print(f"Commit: {result.commit_ref}") +print(f"Added: {result.added_files}") + +# Always clean up +sandbox.cleanup() +``` + +--- + +## Apply Summary Output + +When `plan apply` uses the git worktree sandbox, the CLI renders a +spec-aligned Apply Summary panel: + +``` +╭─ Apply Summary ──────────────────────────────────────────────╮ +│ Plan ID plan-01JABC │ +│ Project local/my-project │ +│ Timestamp 2026-04-09 14:33:34 UTC │ +│ │ +│ Artifacts │ +│ src/new_feature.py │ +│ │ +│ +1 insertion 0 deletions │ +╰──────────────────────────────────────────────────────────────╯ + +╭─ Sandbox Cleanup ────────────────────────────────────────────╮ +│ Worktree removed /tmp/ca-sandbox-plan-01JABC-xxxxx/ │ +│ Branch deleted cleveragents/plan-01JABC │ +╰──────────────────────────────────────────────────────────────╯ + +✓ OK Changes applied +``` + +--- + +## Related Documentation + +- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) — design rationale +- [Custom Sandbox Strategy](../development/custom_sandbox_strategy.md) — implementing your own sandbox +- [Architecture — Plan Lifecycle](../architecture.md#plan-lifecycle) — where apply fits +- [Shell Safety Module](shell-safety.md) — another infrastructure safety module diff --git a/mkdocs.yml b/mkdocs.yml index 2bb05974c..9b117b4cf 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -23,10 +23,12 @@ nav: - Configuration: api/config.md - AI Providers: api/providers.md - TUI: api/tui.md + - ACMS Services: api/acms.md - Modules: - Shell Safety: modules/shell-safety.md - UKO Provenance Tracking: modules/uko-provenance.md - Invariant Reconciliation: modules/invariant-reconciliation.md + - Git Worktree Sandbox: modules/git-worktree-sandbox.md - Development: - Agent System Specification: development/agent-system-specification.md - CI/CD Pipeline: development/ci-cd.md