From 681285b05d8f22c5cdb01d36d9956eedbfbe8754 Mon Sep 17 00:00:00 2001 From: CleverThis Date: Fri, 10 Apr 2026 03:39:33 +0000 Subject: [PATCH] docs(modules): add ACMS context tier hydrator module guide Document hydrate_tiers APIs and integrate the module into the MkDocs navigation. ISSUES CLOSED: #6076 --- CHANGELOG.md | 12 ++ docs/modules/context-tier-hydrator.md | 236 ++++++++++++++++++++++++++ mkdocs.yml | 1 + 3 files changed, 249 insertions(+) create mode 100644 docs/modules/context-tier-hydrator.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 89d67e2c3..f8fbaf49e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,11 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added +- **Documentation — Context Tier Hydrator Guide** (#6076): Documented the + `context_tier_hydrator` service with a dedicated module guide and wired it into the + MkDocs navigation so teams can reference hydration budgets, file selection rules, and + integration patterns with `LLMExecuteActor`. + - **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges LLM-generated changes via `git merge` from an isolated worktree branch instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary @@ -133,6 +138,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). that apply is blocked unless at least one validation was actually executed. Also added `required_total` property for completeness. Updated `consolidated_validation.feature` scenarios to reflect the corrected blocking behavior for empty summaries and no-attachment runs. +- **ACMS Context Hydration** (#1028): Fixed ACMS indexing pipeline not wired into the CLI — + `ContextTierService` started empty on every invocation so the LLM received zero file context. + Added `context_tier_hydrator.py` to read files from linked project resources (via `git ls-files` + or `os.walk`) and store them as `TieredFragment` objects in the tier service. Hydration now runs + automatically before context assembly in `LLMExecuteActor.execute()` and respects max file size + (256KB), total budget (10MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__` + directory skipping. - **ACMS context tier hydration**: `ContextTierService` no longer starts empty on every CLI invocation. A new `context_tier_hydrator.py` reads files from diff --git a/docs/modules/context-tier-hydrator.md b/docs/modules/context-tier-hydrator.md new file mode 100644 index 000000000..42ee6dca7 --- /dev/null +++ b/docs/modules/context-tier-hydrator.md @@ -0,0 +1,236 @@ +# Context Tier Hydrator + +**Module:** `cleveragents.application.services.context_tier_hydrator` +**Introduced:** v3.4.0 (PR #4219, fixes #1028) + +The context tier hydrator bridges the gap between the **resource registry** (files on +disk) and the **ACMS context tier** (in-memory fragments). Without it, the +`ContextTierService` starts empty on every CLI process invocation and the LLM receives +zero file context during plan execution. + +For the full ACMS pipeline architecture, see +[ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md). + +--- + +## Problem Solved + +Before this module, the `ContextTierService` was populated only during long-running +server sessions. Each CLI invocation started a fresh process, so the context tier was +always empty. The LLM execute actor had no file context to work with, producing +low-quality output that ignored the actual project files. + +The hydrator solves this by reading files from linked project resources at the start +of each plan execution, populating the `ContextTierService` with `TieredFragment` +objects before the LLM is called. + +--- + +## Public API + +### `hydrate_tiers_from_project()` + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_from_project, +) + +count = hydrate_tiers_from_project( + tier_service=tier_service, + project_name="local/my-project", + resource_id="01JXYZ...", + resource_location="/home/user/my-project", + resource_type="git-checkout", # optional, default "git-checkout" +) +print(f"Stored {count} fragments") +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `tier_service` | `ContextTierService` | The service to populate | +| `project_name` | `str` | Namespaced project name (e.g. `local/my-project`) | +| `resource_id` | `str` | ULID of the resource | +| `resource_location` | `str` | Filesystem path to the resource root | +| `resource_type` | `str` | Resource type name; affects file listing strategy | + +**Returns:** Number of fragments stored. + +**Limits:** +- Files larger than 256 KB are skipped +- Total indexed content capped at 10 MB per project +- Binary file extensions are skipped (`.pyc`, `.so`, `.png`, `.db`, etc.) +- Hidden directories and build artifacts are skipped + +### `hydrate_tiers_for_plan()` + +Higher-level function that hydrates all projects linked to a plan: + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_for_plan, +) + +total = hydrate_tiers_for_plan( + tier_service=tier_service, + project_names=["local/my-project", "local/shared-lib"], + project_repository=project_repo, + resource_registry=resource_registry_service, +) +print(f"Stored {total} fragments across all projects") +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `tier_service` | `ContextTierService` | The service to populate | +| `project_names` | `list[str]` | Namespaced project names to hydrate | +| `project_repository` | `NamespacedProjectRepository` | For looking up linked resources | +| `resource_registry` | `ResourceRegistryService` | For resolving resource locations | + +**Returns:** Total number of fragments stored across all projects. + +--- + +## File Listing Strategies + +The hydrator uses two strategies for listing files, selected by `resource_type`: + +### `git ls-files` (for `git-checkout` and `git` resources) + +Uses `git ls-files --cached --others --exclude-standard` to list tracked and +untracked-but-not-ignored files. This respects `.gitignore` and is fast for +large repositories. + +```python +# Internally called as: +subprocess.run( + ["git", "ls-files", "--cached", "--others", "--exclude-standard"], + cwd=resource_location, + timeout=30, +) +``` + +Falls back to `os.walk` if `git ls-files` fails (non-git directory, timeout, etc.). + +### `os.walk` (for all other resource types) + +Recursively walks the directory tree, skipping: +- Hidden directories (starting with `.`) +- Build artifacts: `__pycache__`, `node_modules`, `.venv`, `dist`, `build`, etc. +- Binary file extensions + +--- + +## Fragment Format + +Each file is stored as a `TieredFragment` in the `HOT` context tier: + +```python +TieredFragment( + fragment_id=f"{resource_id}:{rel_path}", # e.g. "01JXYZ...:src/main.py" + content=file_content, + tier=ContextTier.HOT, + resource_id=resource_id, + project_name=project_name, + token_count=len(file_content) // 4, # rough estimate: 4 chars per token + metadata={ + "path": rel_path, + "detail_depth": 1, + "relevance_score": 0.5, + }, +) +``` + +--- + +## Integration with `LLMExecuteActor` + +The `LLMExecuteActor` calls `hydrate_tiers_for_plan()` at the start of its +`execute()` method, before assembling context for the LLM: + +```python +class LLMExecuteActor: + def __init__( + self, + tier_service: ContextTierService, + project_repository: NamespacedProjectRepository, + resource_registry: ResourceRegistryService, + # ... other deps + ): + self._tier_service = tier_service + self._project_repository = project_repository + self._resource_registry = resource_registry + + async def execute(self, plan: Plan, context: ExecutionContext) -> ActorResult: + # Hydrate context tiers before assembling context + hydrate_tiers_for_plan( + tier_service=self._tier_service, + project_names=plan.project_names, + project_repository=self._project_repository, + resource_registry=self._resource_registry, + ) + # ... proceed with context assembly and LLM call +``` + +Dependencies are injected at the CLI factory level (`_get_plan_executor`), not +inside application services. No `get_container()` calls from services. + +--- + +## Sandbox Root Wiring + +PR #4219 also wires the `sandbox_root` into `_get_plan_executor()`: + +```python +# In CLI factory +executor = _get_plan_executor( + sandbox_root=Path(".cleveragents/sandbox/"), + # ... other args +) +``` + +LLM file output (`FILE:` blocks in the LLM response) is written to the sandbox +directory during execute. The `apply` command then copies files from the sandbox +to the project directory. + +--- + +## Apply File Writing + +The CLI `apply` command copies files from the sandbox to the project directory: + +```bash +agents plan apply --plan-id +``` + +**Safety guards:** +- Path traversal protection on both write and apply +- Protected directories are skipped: `.cleveragents`, `.git`, `.hg`, `.svn` +- Per-file error handling — sandbox is preserved on partial failure +- Cleanup only on full success + +**Known limitation:** Apply uses flat file copy instead of git worktree merge. +This is tracked as a spec divergence in issue #4454. + +--- + +## Observability + +The hydrator emits structured log events: + +| Event | Level | Description | +|-------|-------|-------------| +| `context_hydrator.skip_missing_location` | WARNING | Resource location does not exist | +| `context_hydrator.hydrated` | INFO | Hydration complete — includes `fragments_stored` and `total_bytes` | +| `context_hydrator.project_not_found` | DEBUG | Project not found in repository | +| `context_hydrator.no_linked_resources` | DEBUG | Project has no linked resources | +| `context_hydrator.resource_not_found` | DEBUG | Resource not found in registry | +| `context_hydrator.store_failed` | DEBUG | Fragment storage failed | + +--- + +## Related Documentation + +- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md) — full ACMS architecture +- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) — sandbox design +- [ADR-008 Resource System](../adr/ADR-008-resource-system.md) — resource registry +- [UKO Provenance Tracking](uko-provenance.md) — knowledge ontology layer diff --git a/mkdocs.yml b/mkdocs.yml index 2bb05974c..53dd660ed 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,7 @@ nav: - Shell Safety: modules/shell-safety.md - UKO Provenance Tracking: modules/uko-provenance.md - Invariant Reconciliation: modules/invariant-reconciliation.md + - Context Tier Hydrator: modules/context-tier-hydrator.md - Development: - Agent System Specification: development/agent-system-specification.md - CI/CD Pipeline: development/ci-cd.md -- 2.52.0