|
|
|
@@ -0,0 +1,236 @@
|
|
|
|
|
# Context Tier Hydrator
|
|
|
|
|
|
|
|
|
|
**Module:** `cleveragents.application.services.context_tier_hydrator`
|
|
|
|
|
**Introduced:** v3.4.0 (PR #4219, fixes #1028)
|
|
|
|
|
|
|
|
|
|
The context tier hydrator bridges the gap between the **resource registry** (files on
|
|
|
|
|
disk) and the **ACMS context tier** (in-memory fragments). Without it, the
|
|
|
|
|
`ContextTierService` starts empty on every CLI process invocation and the LLM receives
|
|
|
|
|
zero file context during plan execution.
|
|
|
|
|
|
|
|
|
|
For the full ACMS pipeline architecture, see
|
|
|
|
|
[ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md).
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Problem Solved
|
|
|
|
|
|
|
|
|
|
Before this module, the `ContextTierService` was populated only during long-running
|
|
|
|
|
server sessions. Each CLI invocation started a fresh process, so the context tier was
|
|
|
|
|
always empty. The LLM execute actor had no file context to work with, producing
|
|
|
|
|
low-quality output that ignored the actual project files.
|
|
|
|
|
|
|
|
|
|
The hydrator solves this by reading files from linked project resources at the start
|
|
|
|
|
of each plan execution, populating the `ContextTierService` with `TieredFragment`
|
|
|
|
|
objects before the LLM is called.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Public API
|
|
|
|
|
|
|
|
|
|
### `hydrate_tiers_from_project()`
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
from cleveragents.application.services.context_tier_hydrator import (
|
|
|
|
|
hydrate_tiers_from_project,
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
count = hydrate_tiers_from_project(
|
|
|
|
|
tier_service=tier_service,
|
|
|
|
|
project_name="local/my-project",
|
|
|
|
|
resource_id="01JXYZ...",
|
|
|
|
|
resource_location="/home/user/my-project",
|
|
|
|
|
resource_type="git-checkout", # optional, default "git-checkout"
|
|
|
|
|
)
|
|
|
|
|
print(f"Stored {count} fragments")
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
| Parameter | Type | Description |
|
|
|
|
|
|-----------|------|-------------|
|
|
|
|
|
| `tier_service` | `ContextTierService` | The service to populate |
|
|
|
|
|
| `project_name` | `str` | Namespaced project name (e.g. `local/my-project`) |
|
|
|
|
|
| `resource_id` | `str` | ULID of the resource |
|
|
|
|
|
| `resource_location` | `str` | Filesystem path to the resource root |
|
|
|
|
|
| `resource_type` | `str` | Resource type name; affects file listing strategy |
|
|
|
|
|
|
|
|
|
|
**Returns:** Number of fragments stored.
|
|
|
|
|
|
|
|
|
|
**Limits:**
|
|
|
|
|
- Files larger than 256 KB are skipped
|
|
|
|
|
- Total indexed content capped at 10 MB per project
|
|
|
|
|
- Binary file extensions are skipped (`.pyc`, `.so`, `.png`, `.db`, etc.)
|
|
|
|
|
- Hidden directories and build artifacts are skipped
|
|
|
|
|
|
|
|
|
|
### `hydrate_tiers_for_plan()`
|
|
|
|
|
|
|
|
|
|
Higher-level function that hydrates all projects linked to a plan:
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
from cleveragents.application.services.context_tier_hydrator import (
|
|
|
|
|
hydrate_tiers_for_plan,
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
total = hydrate_tiers_for_plan(
|
|
|
|
|
tier_service=tier_service,
|
|
|
|
|
project_names=["local/my-project", "local/shared-lib"],
|
|
|
|
|
project_repository=project_repo,
|
|
|
|
|
resource_registry=resource_registry_service,
|
|
|
|
|
)
|
|
|
|
|
print(f"Stored {total} fragments across all projects")
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
| Parameter | Type | Description |
|
|
|
|
|
|-----------|------|-------------|
|
|
|
|
|
| `tier_service` | `ContextTierService` | The service to populate |
|
|
|
|
|
| `project_names` | `list[str]` | Namespaced project names to hydrate |
|
|
|
|
|
| `project_repository` | `NamespacedProjectRepository` | For looking up linked resources |
|
|
|
|
|
| `resource_registry` | `ResourceRegistryService` | For resolving resource locations |
|
|
|
|
|
|
|
|
|
|
**Returns:** Total number of fragments stored across all projects.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## File Listing Strategies
|
|
|
|
|
|
|
|
|
|
The hydrator uses two strategies for listing files, selected by `resource_type`:
|
|
|
|
|
|
|
|
|
|
### `git ls-files` (for `git-checkout` and `git` resources)
|
|
|
|
|
|
|
|
|
|
Uses `git ls-files --cached --others --exclude-standard` to list tracked and
|
|
|
|
|
untracked-but-not-ignored files. This respects `.gitignore` and is fast for
|
|
|
|
|
large repositories.
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
# Internally called as:
|
|
|
|
|
subprocess.run(
|
|
|
|
|
["git", "ls-files", "--cached", "--others", "--exclude-standard"],
|
|
|
|
|
cwd=resource_location,
|
|
|
|
|
timeout=30,
|
|
|
|
|
)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Falls back to `os.walk` if `git ls-files` fails (non-git directory, timeout, etc.).
|
|
|
|
|
|
|
|
|
|
### `os.walk` (for all other resource types)
|
|
|
|
|
|
|
|
|
|
Recursively walks the directory tree, skipping:
|
|
|
|
|
- Hidden directories (starting with `.`)
|
|
|
|
|
- Build artifacts: `__pycache__`, `node_modules`, `.venv`, `dist`, `build`, etc.
|
|
|
|
|
- Binary file extensions
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Fragment Format
|
|
|
|
|
|
|
|
|
|
Each file is stored as a `TieredFragment` in the `HOT` context tier:
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
TieredFragment(
|
|
|
|
|
fragment_id=f"{resource_id}:{rel_path}", # e.g. "01JXYZ...:src/main.py"
|
|
|
|
|
content=file_content,
|
|
|
|
|
tier=ContextTier.HOT,
|
|
|
|
|
resource_id=resource_id,
|
|
|
|
|
project_name=project_name,
|
|
|
|
|
token_count=len(file_content) // 4, # rough estimate: 4 chars per token
|
|
|
|
|
metadata={
|
|
|
|
|
"path": rel_path,
|
|
|
|
|
"detail_depth": 1,
|
|
|
|
|
"relevance_score": 0.5,
|
|
|
|
|
},
|
|
|
|
|
)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Integration with `LLMExecuteActor`
|
|
|
|
|
|
|
|
|
|
The `LLMExecuteActor` calls `hydrate_tiers_for_plan()` at the start of its
|
|
|
|
|
`execute()` method, before assembling context for the LLM:
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
class LLMExecuteActor:
|
|
|
|
|
def __init__(
|
|
|
|
|
self,
|
|
|
|
|
tier_service: ContextTierService,
|
|
|
|
|
project_repository: NamespacedProjectRepository,
|
|
|
|
|
resource_registry: ResourceRegistryService,
|
|
|
|
|
# ... other deps
|
|
|
|
|
):
|
|
|
|
|
self._tier_service = tier_service
|
|
|
|
|
self._project_repository = project_repository
|
|
|
|
|
self._resource_registry = resource_registry
|
|
|
|
|
|
|
|
|
|
async def execute(self, plan: Plan, context: ExecutionContext) -> ActorResult:
|
|
|
|
|
# Hydrate context tiers before assembling context
|
|
|
|
|
hydrate_tiers_for_plan(
|
|
|
|
|
tier_service=self._tier_service,
|
|
|
|
|
project_names=plan.project_names,
|
|
|
|
|
project_repository=self._project_repository,
|
|
|
|
|
resource_registry=self._resource_registry,
|
|
|
|
|
)
|
|
|
|
|
# ... proceed with context assembly and LLM call
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Dependencies are injected at the CLI factory level (`_get_plan_executor`), not
|
|
|
|
|
inside application services. No `get_container()` calls from services.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Sandbox Root Wiring
|
|
|
|
|
|
|
|
|
|
PR #4219 also wires the `sandbox_root` into `_get_plan_executor()`:
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
# In CLI factory
|
|
|
|
|
executor = _get_plan_executor(
|
|
|
|
|
sandbox_root=Path(".cleveragents/sandbox/"),
|
|
|
|
|
# ... other args
|
|
|
|
|
)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
LLM file output (`FILE:` blocks in the LLM response) is written to the sandbox
|
|
|
|
|
directory during execute. The `apply` command then copies files from the sandbox
|
|
|
|
|
to the project directory.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Apply File Writing
|
|
|
|
|
|
|
|
|
|
The CLI `apply` command copies files from the sandbox to the project directory:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
agents plan apply --plan-id <ID>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Safety guards:**
|
|
|
|
|
- Path traversal protection on both write and apply
|
|
|
|
|
- Protected directories are skipped: `.cleveragents`, `.git`, `.hg`, `.svn`
|
|
|
|
|
- Per-file error handling — sandbox is preserved on partial failure
|
|
|
|
|
- Cleanup only on full success
|
|
|
|
|
|
|
|
|
|
**Known limitation:** Apply uses flat file copy instead of git worktree merge.
|
|
|
|
|
This is tracked as a spec divergence in issue #4454.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Observability
|
|
|
|
|
|
|
|
|
|
The hydrator emits structured log events:
|
|
|
|
|
|
|
|
|
|
| Event | Level | Description |
|
|
|
|
|
|-------|-------|-------------|
|
|
|
|
|
| `context_hydrator.skip_missing_location` | WARNING | Resource location does not exist |
|
|
|
|
|
| `context_hydrator.hydrated` | INFO | Hydration complete — includes `fragments_stored` and `total_bytes` |
|
|
|
|
|
| `context_hydrator.project_not_found` | DEBUG | Project not found in repository |
|
|
|
|
|
| `context_hydrator.no_linked_resources` | DEBUG | Project has no linked resources |
|
|
|
|
|
| `context_hydrator.resource_not_found` | DEBUG | Resource not found in registry |
|
|
|
|
|
| `context_hydrator.store_failed` | DEBUG | Fragment storage failed |
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Related Documentation
|
|
|
|
|
|
|
|
|
|
- [ADR-014 Context Management (ACMS)](../adr/ADR-014-context-management-acms.md) — full ACMS architecture
|
|
|
|
|
- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) — sandbox design
|
|
|
|
|
- [ADR-008 Resource System](../adr/ADR-008-resource-system.md) — resource registry
|
|
|
|
|
- [UKO Provenance Tracking](uko-provenance.md) — knowledge ontology layer
|