cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

docs: add context-tier-hydrator module guide, fix CHANGELOG placement #6119

Closed
HAL9000 wants to merge 1 commits from docs/acms-context-hydrator-cycle2 into master
pull from: docs/acms-context-hydrator-cycle2
merge into: cleveragents:master
Conversation 12 Commits 1 Files Changed 3
+243 -33
3 changed files with 243 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+44 -33
View File
@@ -14,11 +14,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall
back to the original flat file copy.
- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types
(`detail_depth` and `relevance_score` must be strings, not int/float) that
caused Pydantic validation errors during context assembly, resulting in the
LLM receiving zero file context.
- **Automation Tracking System**: Replaced shared session state issue tracking with
individual per-agent tracking issues. Each agent now creates its own `[AUTO-<PREFIX>]`
titled issues with standardized headers, reporting intervals, and health indicators.
@@ -27,24 +22,24 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
`docs/development/automation-tracking.md`.
- **Automated Health Monitoring and Recovery**: The `system-watchdog` now runs
`audit_automation_tracking_health()` every 5 minutes, detecting stalled agents when
tracking issues are >20% overdue from their declared reporting interval. On detection,
it terminates stalled sessions via the OpenCode Server API, performs root-cause analysis,
creates high-priority diagnostic issues, and closes stale tracking issues with recovery
notes.
`audit_automation_tracking_health()` every 5 minutes. It flags agents whose tracking
issues are more than 20% overdue relative to their reporting interval. When triggered,
the watchdog terminates stalled sessions through the OpenCode Server API, performs
root-cause analysis, files high-priority diagnostic issues, and closes stale tracking
issues with recovery notes.
- **Centralized Label Management** (`forgejo-label-manager`): A new specialized subagent
centralizes all Forgejo label operations across the agent system. Enforces the
organization-level label system, prohibits label creation, and validates label compliance.
Agents `backlog-groomer`, `human-liaison`, `project-owner`, `epic-planner`,
`new-issue-creator`, and `issue-state-updater` now delegate all label operations to this
subagent.
- **Centralized Label Management** (`forgejo-label-manager`): A new subagent now
centralizes Forgejo label operations across every automation agent. It
enforces the organization-wide label set, forbids ad-hoc label creation,
validates compliance, and serves requests from agents such as
`backlog-groomer`, `human-liaison`, and `project-owner`. It also supports
`epic-planner`, `new-issue-creator`, and `issue-state-updater`.
- **PR–Issue Label Synchronization**: PRs now inherit `Priority/`, `MoSCoW/`, `Points/`,
and `State/` labels from their associated issues at creation time
(`pr-api-creator`). The `backlog-groomer` adds a continuous Pass 19 for ongoing
PR–issue label synchronization. The `issue-state-updater` syncs PR state labels whenever
issue states change.
- **PR–Issue Label Synchronization**: `pr-api-creator` now copies `Priority/`,
`MoSCoW/`, `Points/`, and `State/` labels from linked issues when a PR opens.
`backlog-groomer` adds a continuous Pass 19 task to keep labels synchronized,
and `issue-state-updater` refreshes PR state labels whenever the issue state
changes.
- **Automation Tracking Announcements**: Extended `automation-tracking-manager` with
announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`,
@@ -103,9 +98,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
ID reconstruction. Added "Decision IDs (for correction)" section with human-readable
labels for easy reference. Applies to both table and rich/plain text output formats.
- **Automation Tracking Format**: All automation tracking issues now use a standardized
header format with mandatory `Reporting Interval: <interval> (Next report expected: <ts>)`
declarations, enabling precise staleness detection.
- **Automation Tracking Format**: Automation tracking issues now use a
standardized header with a `Reporting Interval: <interval> (Next report
expected: <ts>)` line. The consistent metadata makes overdue detection
precise.
- **PR Review Policy**: Reduced PR review requirement from 2 approvals to 1. Self-approval
is now permitted including for automated bot PRs. Approval can be a formal review OR an
@@ -125,17 +121,32 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
`AUTO-INF-POOL`, `AUTO-ARCH`, `AUTO-EPIC`, `AUTO-EVLV`, `AUTO-GUARD`, `AUTO-SPEC`,
`AUTO-TIME`, `AUTO-PROJ-OWN`, and `AUTO-PROD-BLDR`.
- **ACMS Context Hydration**: Fixed ACMS indexing pipeline not wired into CLI —
`ContextTierService` started empty on every CLI invocation so LLM received zero file
context during plan execution. Added `context_tier_hydrator.py` that reads files from
linked project resources (via `git ls-files` or `os.walk`) and stores them as
`TieredFragment` objects in the tier service. Hydration runs automatically before context
assembly in `LLMExecuteActor.execute()`. Respects max file size (256KB), total budget
(10MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__` directory
skipping. (#1028)
### Fixed
- **ACMS context tier hydration** (`context_tier_hydrator`):
`ContextTierService` used to start empty on each CLI invocation, so the LLM
received zero file context. `context_tier_hydrator.py` now reads files from
linked resources with `git ls-files` for git checkouts and falls back to
`os.walk` for other types. The hydrator stores results as `TieredFragment`
entries in the tier service. It runs before context assembly in
`LLMExecuteActor.execute()`. Limits remain 256 KB per file and 10 MB overall,
and binary extensions plus build directories such as `.git`, `node_modules`,
`__pycache__`, and `.venv` are skipped. (#1028, #4219)
- **Apply phase file writing** (`plan apply`): The CLI now copies files from
the sandbox directory into the project directory while enforcing path
traversal guards. It skips protected directories such as `.cleveragents`,
`.git`, `.hg`, and `.svn`, handles per-file failures without discarding the
sandbox, and only cleans up on full success. The sandbox root is also wired
to `.cleveragents/sandbox/` in `_get_plan_executor()`. (#4222, #4219)
- **`plan use` UNIQUE constraint violation** (`ActionRepository.update`):
Replaced the `.clear()` plus `.append()` pattern with explicit `sa_delete()`
calls followed by `session.flush()` before re-inserting related rows. Added
`session.expire()` so the collections reload and avoid the stale identity-map
references that triggered `sqlite3.IntegrityError` when reusing actions.
(#4174, #4197)
- **Validation Gate Empty-Run Guard** (#7508): Fixed `ApplyValidationSummary.all_required_passed`
returning `True` when zero validations were run, silently bypassing the apply gate. The property
now returns `False` when the validation result set is empty (`is_empty` is `True`), ensuring
docs/modules/context-tier-hydrator.md
+198
View File
@@ -0,0 +1,198 @@
# Context Tier Hydrator
**Module:** `cleveragents.application.services.context_tier_hydrator`
The context tier hydrator bridges the gap between the resource registry (files
on disk) and the ACMS context tier (in-memory fragments). Without it, the
`ContextTierService` starts empty on every CLI process invocation and the LLM
receives zero file context during plan execution.
This module resolves issue [#1028].
---
## Overview
When a plan is executed, the LLM needs file context from the project's linked
resources. The hydrator:
1. Iterates over all projects linked to the plan
2. For each project, finds its linked resources
3. For each resource, lists files using `git ls-files` (git-checkout) or `os.walk`
4. Reads each file and stores it as a `TieredFragment` in `ContextTierService`
5. Respects size and budget limits to avoid overwhelming the LLM context window
Hydration runs automatically before context assembly in
`LLMExecuteActor.execute()`.
---
## Functions
### `hydrate_tiers_for_plan`
```python
from cleveragents.application.services.context_tier_hydrator import (
hydrate_tiers_for_plan,
)
total_fragments = hydrate_tiers_for_plan(
tier_service=tier_service,
project_names=["local/my-project"],
project_repository=project_repo,
resource_registry=resource_registry,
)
```
Hydrates tiers for all projects linked to a plan. This is the primary entry
point called by `LLMExecuteActor`.
| Parameter | Type | Description |
|-----------|------|-------------|
| `tier_service` | `ContextTierService` | The tier service to populate |
| `project_names` | `list[str]` | Namespaced project names (e.g. `local/my-project`) |
| `project_repository` | `NamespacedProjectRepository` | Project metadata repository |
| `resource_registry` | `ResourceRegistryService` | Resolves resource locations |
**Returns:** Total number of `TieredFragment` objects stored across all projects.
---
### `hydrate_tiers_from_project`
```python
from cleveragents.application.services.context_tier_hydrator import (
hydrate_tiers_from_project,
)
count = hydrate_tiers_from_project(
tier_service=tier_service,
project_name="local/my-project",
resource_id="01ABCDEF...",
resource_location="/path/to/project",
resource_type="git-checkout",
)
```
Reads files from a single resource and stores them as `TieredFragment` objects.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `tier_service` | `ContextTierService` | — | The tier service to populate |
| `project_name` | `str` | — | Namespaced project name |
| `resource_id` | `str` | — | ULID of the resource |
| `resource_location` | `str` | — | Filesystem path to the resource root |
| `resource_type` | `str` | `"git-checkout"` | Resource type; selects listing mode |
**Returns:** Number of fragments stored.
**Behaviour when `resource_location` is missing or not a directory:** logs a
warning and returns `0`.
---
## Limits and Filters
| Limit | Value | Description |
|-------|-------|-------------|
| Max file size | 256 KB | Files larger than this are skipped |
| Max total budget | 10 MB | Hydration stops when this is reached |
### Skipped directories
`.git`, `.hg`, `.svn`, `__pycache__`, `node_modules`, `.venv`, `venv`,
`.nox`, `.tox`, `.mypy_cache`, `.pytest_cache`, `.ruff_cache`, `dist`,
`build`, `.eggs`, `.cleveragents`
### Skipped file extensions (binary)
`.pyc`, `.pyo`, `.so`, `.o`, `.a`, `.dll`, `.exe`, `.png`, `.jpg`, `.jpeg`,
`.gif`, `.bmp`, `.ico`, `.pdf`, `.zip`, `.tar`, `.gz`, `.bz2`, `.xz`,
`.whl`, `.egg`, `.db`, `.sqlite`, `.sqlite3`
---
## File Listing Strategy
| Resource type | Strategy |
|---------------|----------|
| `git-checkout`, `git` | `git ls-files --cached --others --exclude-standard` |
| All others | `os.walk` with directory and extension filters |
For git-checkout resources, `git ls-files` is preferred because it respects
`.gitignore` and only returns tracked/untracked-but-not-ignored files. If the
`git` command fails (e.g. not a git repo), the hydrator falls back to
`os.walk`.
---
## Fragment Structure
Each file is stored as a `TieredFragment` with:
```python
TieredFragment(
fragment_id=f"{resource_id}:{rel_path}",
content=file_content,
tier=ContextTier.HOT,
resource_id=resource_id,
project_name=project_name,
token_count=len(content) // 4, # approximate
metadata={
"path": rel_path,
"detail_depth": "1",
"relevance_score": "0.5",
},
)
```
All fragments are stored at `ContextTier.HOT` (always included in context
assembly). Token count is approximated as `len(content) // 4`.
---
## Integration with `LLMExecuteActor`
The hydrator is called inside `LLMExecuteActor.execute()` before context
assembly:
```python
# Simplified from llm_actors.py
from cleveragents.application.services.context_tier_hydrator import (
hydrate_tiers_for_plan,
)
class LLMExecuteActor:
def execute(self, plan, ...):
# Hydrate context tiers from project resources
hydrate_tiers_for_plan(
tier_service=self._tier_service,
project_names=plan.project_names,
project_repository=self._project_repository,
resource_registry=self._resource_registry,
)
# Then assemble context and call LLM
context = self._assembler.assemble(plan)
...
```
Dependencies (`tier_service`, `project_repository`, `resource_registry`) are
injected at the CLI factory level in `_get_plan_executor()`, not via
`get_container()` calls inside services.
---
## Logging
The hydrator uses `structlog` with the following event keys:
| Event | Level | Description |
|-------|-------|-------------|
| `context_hydrator.skip_missing_location` | WARNING | Missing path or not directory |
| `context_hydrator.hydrated` | INFO | Hydration complete — reports fragments + bytes |
| `context_hydrator.project_not_found` | DEBUG | Project not found in repository |
| `context_hydrator.no_linked_resources` | DEBUG | Project has no linked resources |
| `context_hydrator.resource_not_found` | DEBUG | Resource not found in registry |
| `context_hydrator.store_failed` | DEBUG | Fragment store failed (non-fatal) |
[#1028]: https://git.cleverthis.com/cleveragents/cleveragents-core/issues/1028
mkdocs.yml
+1
View File
@@ -27,6 +27,7 @@ nav:
- Shell Safety: modules/shell-safety.md
- UKO Provenance Tracking: modules/uko-provenance.md
- Invariant Reconciliation: modules/invariant-reconciliation.md
- Context Tier Hydrator: modules/context-tier-hydrator.md
- Development:
- Agent System Specification: development/agent-system-specification.md
- CI/CD Pipeline: development/ci-cd.md