Compare commits

...

1 Commits

Author SHA1 Message Date
HAL9000 681285b05d docs(modules): add ACMS context tier hydrator module guide
CI / build (pull_request) Successful in 26s
CI / typecheck (pull_request) Successful in 51s
CI / push-validation (pull_request) Successful in 28s
CI / quality (pull_request) Successful in 49s
CI / helm (pull_request) Successful in 45s
CI / lint (pull_request) Successful in 3m21s
CI / security (pull_request) Successful in 4m6s
CI / integration_tests (pull_request) Successful in 7m1s
CI / e2e_tests (pull_request) Successful in 7m30s
CI / unit_tests (pull_request) Successful in 8m52s
CI / docker (pull_request) Successful in 23s
CI / coverage (pull_request) Successful in 11m3s
CI / status-check (pull_request) Successful in 15s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 1h0m34s
Document hydrate_tiers APIs and integrate the module into the MkDocs navigation.

ISSUES CLOSED: #6076
2026-04-12 16:55:32 +00:00
3 changed files with 249 additions and 0 deletions
+12
View File
@@ -7,6 +7,11 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
### Added
- **Documentation — Context Tier Hydrator Guide** (#6076): Documented the
`context_tier_hydrator` service with a dedicated module guide and wired it into the
MkDocs navigation so teams can reference hydration budgets, file selection rules, and
integration patterns with `LLMExecuteActor`.
- **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges
LLM-generated changes via `git merge` from an isolated worktree branch
instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary
@@ -133,6 +138,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
that apply is blocked unless at least one validation was actually executed. Also added
`required_total` property for completeness. Updated `consolidated_validation.feature` scenarios
to reflect the corrected blocking behavior for empty summaries and no-attachment runs.
- **ACMS Context Hydration** (#1028): Fixed ACMS indexing pipeline not wired into the CLI —
`ContextTierService` started empty on every invocation so the LLM received zero file context.
Added `context_tier_hydrator.py` to read files from linked project resources (via `git ls-files`
or `os.walk`) and store them as `TieredFragment` objects in the tier service. Hydration now runs
automatically before context assembly in `LLMExecuteActor.execute()` and respects max file size
(256KB), total budget (10MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__`
directory skipping.
- **ACMS context tier hydration**: `ContextTierService` no longer starts empty
on every CLI invocation. A new `context_tier_hydrator.py` reads files from
+236
View File
@@ -0,0 +1,236 @@
# Context Tier Hydrator
**Module:** `cleveragents.application.services.context_tier_hydrator`
**Introduced:** v3.4.0 (PR #4219, fixes #1028)
The context tier hydrator bridges the gap between the **resource registry** (files on
disk) and the **ACMS context tier** (in-memory fragments). Without it, the
`ContextTierService` starts empty on every CLI process invocation and the LLM receives
zero file context during plan execution.
For the full ACMS pipeline architecture, see
[ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md).
---
## Problem Solved
Before this module, the `ContextTierService` was populated only during long-running
server sessions. Each CLI invocation started a fresh process, so the context tier was
always empty. The LLM execute actor had no file context to work with, producing
low-quality output that ignored the actual project files.
The hydrator solves this by reading files from linked project resources at the start
of each plan execution, populating the `ContextTierService` with `TieredFragment`
objects before the LLM is called.
---
## Public API
### `hydrate_tiers_from_project()`
```python
from cleveragents.application.services.context_tier_hydrator import (
hydrate_tiers_from_project,
)
count = hydrate_tiers_from_project(
tier_service=tier_service,
project_name="local/my-project",
resource_id="01JXYZ...",
resource_location="/home/user/my-project",
resource_type="git-checkout", # optional, default "git-checkout"
)
print(f"Stored {count} fragments")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `tier_service` | `ContextTierService` | The service to populate |
| `project_name` | `str` | Namespaced project name (e.g. `local/my-project`) |
| `resource_id` | `str` | ULID of the resource |
| `resource_location` | `str` | Filesystem path to the resource root |
| `resource_type` | `str` | Resource type name; affects file listing strategy |
**Returns:** Number of fragments stored.
**Limits:**
- Files larger than 256 KB are skipped
- Total indexed content capped at 10 MB per project
- Binary file extensions are skipped (`.pyc`, `.so`, `.png`, `.db`, etc.)
- Hidden directories and build artifacts are skipped
### `hydrate_tiers_for_plan()`
Higher-level function that hydrates all projects linked to a plan:
```python
from cleveragents.application.services.context_tier_hydrator import (
hydrate_tiers_for_plan,
)
total = hydrate_tiers_for_plan(
tier_service=tier_service,
project_names=["local/my-project", "local/shared-lib"],
project_repository=project_repo,
resource_registry=resource_registry_service,
)
print(f"Stored {total} fragments across all projects")
```
| Parameter | Type | Description |
|-----------|------|-------------|
| `tier_service` | `ContextTierService` | The service to populate |
| `project_names` | `list[str]` | Namespaced project names to hydrate |
| `project_repository` | `NamespacedProjectRepository` | For looking up linked resources |
| `resource_registry` | `ResourceRegistryService` | For resolving resource locations |
**Returns:** Total number of fragments stored across all projects.
---
## File Listing Strategies
The hydrator uses two strategies for listing files, selected by `resource_type`:
### `git ls-files` (for `git-checkout` and `git` resources)
Uses `git ls-files --cached --others --exclude-standard` to list tracked and
untracked-but-not-ignored files. This respects `.gitignore` and is fast for
large repositories.
```python
# Internally called as:
subprocess.run(
["git", "ls-files", "--cached", "--others", "--exclude-standard"],
cwd=resource_location,
timeout=30,
)
```
Falls back to `os.walk` if `git ls-files` fails (non-git directory, timeout, etc.).
### `os.walk` (for all other resource types)
Recursively walks the directory tree, skipping:
- Hidden directories (starting with `.`)
- Build artifacts: `__pycache__`, `node_modules`, `.venv`, `dist`, `build`, etc.
- Binary file extensions
---
## Fragment Format
Each file is stored as a `TieredFragment` in the `HOT` context tier:
```python
TieredFragment(
fragment_id=f"{resource_id}:{rel_path}", # e.g. "01JXYZ...:src/main.py"
content=file_content,
tier=ContextTier.HOT,
resource_id=resource_id,
project_name=project_name,
token_count=len(file_content) // 4, # rough estimate: 4 chars per token
metadata={
"path": rel_path,
"detail_depth": 1,
"relevance_score": 0.5,
},
)
```
---
## Integration with `LLMExecuteActor`
The `LLMExecuteActor` calls `hydrate_tiers_for_plan()` at the start of its
`execute()` method, before assembling context for the LLM:
```python
class LLMExecuteActor:
def __init__(
self,
tier_service: ContextTierService,
project_repository: NamespacedProjectRepository,
resource_registry: ResourceRegistryService,
# ... other deps
):
self._tier_service = tier_service
self._project_repository = project_repository
self._resource_registry = resource_registry
async def execute(self, plan: Plan, context: ExecutionContext) -> ActorResult:
# Hydrate context tiers before assembling context
hydrate_tiers_for_plan(
tier_service=self._tier_service,
project_names=plan.project_names,
project_repository=self._project_repository,
resource_registry=self._resource_registry,
)
# ... proceed with context assembly and LLM call
```
Dependencies are injected at the CLI factory level (`_get_plan_executor`), not
inside application services. No `get_container()` calls from services.
---
## Sandbox Root Wiring
PR #4219 also wires the `sandbox_root` into `_get_plan_executor()`:
```python
# In CLI factory
executor = _get_plan_executor(
sandbox_root=Path(".cleveragents/sandbox/"),
# ... other args
)
```
LLM file output (`FILE:` blocks in the LLM response) is written to the sandbox
directory during execute. The `apply` command then copies files from the sandbox
to the project directory.
---
## Apply File Writing
The CLI `apply` command copies files from the sandbox to the project directory:
```bash
agents plan apply --plan-id <ID>
```
**Safety guards:**
- Path traversal protection on both write and apply
- Protected directories are skipped: `.cleveragents`, `.git`, `.hg`, `.svn`
- Per-file error handling — sandbox is preserved on partial failure
- Cleanup only on full success
**Known limitation:** Apply uses flat file copy instead of git worktree merge.
This is tracked as a spec divergence in issue #4454.
---
## Observability
The hydrator emits structured log events:
| Event | Level | Description |
|-------|-------|-------------|
| `context_hydrator.skip_missing_location` | WARNING | Resource location does not exist |
| `context_hydrator.hydrated` | INFO | Hydration complete — includes `fragments_stored` and `total_bytes` |
| `context_hydrator.project_not_found` | DEBUG | Project not found in repository |
| `context_hydrator.no_linked_resources` | DEBUG | Project has no linked resources |
| `context_hydrator.resource_not_found` | DEBUG | Resource not found in registry |
| `context_hydrator.store_failed` | DEBUG | Fragment storage failed |
---
## Related Documentation
- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md) — full ACMS architecture
- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) — sandbox design
- [ADR-008 Resource System](../adr/ADR-008-resource-system.md) — resource registry
- [UKO Provenance Tracking](uko-provenance.md) — knowledge ontology layer
+1
View File
@@ -27,6 +27,7 @@ nav:
- Shell Safety: modules/shell-safety.md
- UKO Provenance Tracking: modules/uko-provenance.md
- Invariant Reconciliation: modules/invariant-reconciliation.md
- Context Tier Hydrator: modules/context-tier-hydrator.md
- Development:
- Agent System Specification: development/agent-system-specification.md
- CI/CD Pipeline: development/ci-cd.md