diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c1a7dca9..dfab42106 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,11 +14,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall back to the original flat file copy. -- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types - (`detail_depth` and `relevance_score` must be strings, not int/float) that - caused Pydantic validation errors during context assembly, resulting in the - LLM receiving zero file context. - - **Automation Tracking System**: Replaced shared session state issue tracking with individual per-agent tracking issues. Each agent now creates its own `[AUTO-]` titled issues with standardized headers, reporting intervals, and health indicators. @@ -27,24 +22,24 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). `docs/development/automation-tracking.md`. - **Automated Health Monitoring and Recovery**: The `system-watchdog` now runs - `audit_automation_tracking_health()` every 5 minutes, detecting stalled agents when - tracking issues are >20% overdue from their declared reporting interval. On detection, - it terminates stalled sessions via the OpenCode Server API, performs root-cause analysis, - creates high-priority diagnostic issues, and closes stale tracking issues with recovery - notes. + `audit_automation_tracking_health()` every 5 minutes. It flags agents whose tracking + issues are more than 20% overdue relative to their reporting interval. When triggered, + the watchdog terminates stalled sessions through the OpenCode Server API, performs + root-cause analysis, files high-priority diagnostic issues, and closes stale tracking + issues with recovery notes. -- **Centralized Label Management** (`forgejo-label-manager`): A new specialized subagent - centralizes all Forgejo label operations across the agent system. Enforces the - organization-level label system, prohibits label creation, and validates label compliance. - Agents `backlog-groomer`, `human-liaison`, `project-owner`, `epic-planner`, - `new-issue-creator`, and `issue-state-updater` now delegate all label operations to this - subagent. +- **Centralized Label Management** (`forgejo-label-manager`): A new subagent now + centralizes Forgejo label operations across every automation agent. It + enforces the organization-wide label set, forbids ad-hoc label creation, + validates compliance, and serves requests from agents such as + `backlog-groomer`, `human-liaison`, and `project-owner`. It also supports + `epic-planner`, `new-issue-creator`, and `issue-state-updater`. -- **PR–Issue Label Synchronization**: PRs now inherit `Priority/`, `MoSCoW/`, `Points/`, - and `State/` labels from their associated issues at creation time - (`pr-api-creator`). The `backlog-groomer` adds a continuous Pass 19 for ongoing - PR–issue label synchronization. The `issue-state-updater` syncs PR state labels whenever - issue states change. +- **PR–Issue Label Synchronization**: `pr-api-creator` now copies `Priority/`, + `MoSCoW/`, `Points/`, and `State/` labels from linked issues when a PR opens. + `backlog-groomer` adds a continuous Pass 19 task to keep labels synchronized, + and `issue-state-updater` refreshes PR state labels whenever the issue state + changes. - **Automation Tracking Announcements**: Extended `automation-tracking-manager` with announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`, @@ -103,9 +98,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ID reconstruction. Added "Decision IDs (for correction)" section with human-readable labels for easy reference. Applies to both table and rich/plain text output formats. -- **Automation Tracking Format**: All automation tracking issues now use a standardized - header format with mandatory `Reporting Interval: (Next report expected: )` - declarations, enabling precise staleness detection. +- **Automation Tracking Format**: Automation tracking issues now use a + standardized header with a `Reporting Interval: (Next report + expected: )` line. The consistent metadata makes overdue detection + precise. - **PR Review Policy**: Reduced PR review requirement from 2 approvals to 1. Self-approval is now permitted including for automated bot PRs. Approval can be a formal review OR an @@ -125,17 +121,32 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). `AUTO-INF-POOL`, `AUTO-ARCH`, `AUTO-EPIC`, `AUTO-EVLV`, `AUTO-GUARD`, `AUTO-SPEC`, `AUTO-TIME`, `AUTO-PROJ-OWN`, and `AUTO-PROD-BLDR`. -- **ACMS Context Hydration**: Fixed ACMS indexing pipeline not wired into CLI — - `ContextTierService` started empty on every CLI invocation so LLM received zero file - context during plan execution. Added `context_tier_hydrator.py` that reads files from - linked project resources (via `git ls-files` or `os.walk`) and stores them as - `TieredFragment` objects in the tier service. Hydration runs automatically before context - assembly in `LLMExecuteActor.execute()`. Respects max file size (256KB), total budget - (10MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__` directory - skipping. (#1028) - ### Fixed +- **ACMS context tier hydration** (`context_tier_hydrator`): + `ContextTierService` used to start empty on each CLI invocation, so the LLM + received zero file context. `context_tier_hydrator.py` now reads files from + linked resources with `git ls-files` for git checkouts and falls back to + `os.walk` for other types. The hydrator stores results as `TieredFragment` + entries in the tier service. It runs before context assembly in + `LLMExecuteActor.execute()`. Limits remain 256 KB per file and 10 MB overall, + and binary extensions plus build directories such as `.git`, `node_modules`, + `__pycache__`, and `.venv` are skipped. (#1028, #4219) + +- **Apply phase file writing** (`plan apply`): The CLI now copies files from + the sandbox directory into the project directory while enforcing path + traversal guards. It skips protected directories such as `.cleveragents`, + `.git`, `.hg`, and `.svn`, handles per-file failures without discarding the + sandbox, and only cleans up on full success. The sandbox root is also wired + to `.cleveragents/sandbox/` in `_get_plan_executor()`. (#4222, #4219) + +- **`plan use` UNIQUE constraint violation** (`ActionRepository.update`): + Replaced the `.clear()` plus `.append()` pattern with explicit `sa_delete()` + calls followed by `session.flush()` before re-inserting related rows. Added + `session.expire()` so the collections reload and avoid the stale identity-map + references that triggered `sqlite3.IntegrityError` when reusing actions. + (#4174, #4197) + - **Validation Gate Empty-Run Guard** (#7508): Fixed `ApplyValidationSummary.all_required_passed` returning `True` when zero validations were run, silently bypassing the apply gate. The property now returns `False` when the validation result set is empty (`is_empty` is `True`), ensuring diff --git a/docs/modules/context-tier-hydrator.md b/docs/modules/context-tier-hydrator.md new file mode 100644 index 000000000..b09eaf772 --- /dev/null +++ b/docs/modules/context-tier-hydrator.md @@ -0,0 +1,198 @@ +# Context Tier Hydrator + +**Module:** `cleveragents.application.services.context_tier_hydrator` + +The context tier hydrator bridges the gap between the resource registry (files +on disk) and the ACMS context tier (in-memory fragments). Without it, the +`ContextTierService` starts empty on every CLI process invocation and the LLM +receives zero file context during plan execution. + +This module resolves issue [#1028]. + +--- + +## Overview + +When a plan is executed, the LLM needs file context from the project's linked +resources. The hydrator: + +1. Iterates over all projects linked to the plan +2. For each project, finds its linked resources +3. For each resource, lists files using `git ls-files` (git-checkout) or `os.walk` +4. Reads each file and stores it as a `TieredFragment` in `ContextTierService` +5. Respects size and budget limits to avoid overwhelming the LLM context window + +Hydration runs automatically before context assembly in +`LLMExecuteActor.execute()`. + +--- + +## Functions + +### `hydrate_tiers_for_plan` + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_for_plan, +) + +total_fragments = hydrate_tiers_for_plan( + tier_service=tier_service, + project_names=["local/my-project"], + project_repository=project_repo, + resource_registry=resource_registry, +) +``` + +Hydrates tiers for all projects linked to a plan. This is the primary entry +point called by `LLMExecuteActor`. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `tier_service` | `ContextTierService` | The tier service to populate | +| `project_names` | `list[str]` | Namespaced project names (e.g. `local/my-project`) | +| `project_repository` | `NamespacedProjectRepository` | Project metadata repository | +| `resource_registry` | `ResourceRegistryService` | Resolves resource locations | + +**Returns:** Total number of `TieredFragment` objects stored across all projects. + +--- + +### `hydrate_tiers_from_project` + +```python +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_from_project, +) + +count = hydrate_tiers_from_project( + tier_service=tier_service, + project_name="local/my-project", + resource_id="01ABCDEF...", + resource_location="/path/to/project", + resource_type="git-checkout", +) +``` + +Reads files from a single resource and stores them as `TieredFragment` objects. + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `tier_service` | `ContextTierService` | — | The tier service to populate | +| `project_name` | `str` | — | Namespaced project name | +| `resource_id` | `str` | — | ULID of the resource | +| `resource_location` | `str` | — | Filesystem path to the resource root | +| `resource_type` | `str` | `"git-checkout"` | Resource type; selects listing mode | + +**Returns:** Number of fragments stored. + +**Behaviour when `resource_location` is missing or not a directory:** logs a +warning and returns `0`. + +--- + +## Limits and Filters + +| Limit | Value | Description | +|-------|-------|-------------| +| Max file size | 256 KB | Files larger than this are skipped | +| Max total budget | 10 MB | Hydration stops when this is reached | + +### Skipped directories + +`.git`, `.hg`, `.svn`, `__pycache__`, `node_modules`, `.venv`, `venv`, +`.nox`, `.tox`, `.mypy_cache`, `.pytest_cache`, `.ruff_cache`, `dist`, +`build`, `.eggs`, `.cleveragents` + +### Skipped file extensions (binary) + +`.pyc`, `.pyo`, `.so`, `.o`, `.a`, `.dll`, `.exe`, `.png`, `.jpg`, `.jpeg`, +`.gif`, `.bmp`, `.ico`, `.pdf`, `.zip`, `.tar`, `.gz`, `.bz2`, `.xz`, +`.whl`, `.egg`, `.db`, `.sqlite`, `.sqlite3` + +--- + +## File Listing Strategy + +| Resource type | Strategy | +|---------------|----------| +| `git-checkout`, `git` | `git ls-files --cached --others --exclude-standard` | +| All others | `os.walk` with directory and extension filters | + +For git-checkout resources, `git ls-files` is preferred because it respects +`.gitignore` and only returns tracked/untracked-but-not-ignored files. If the +`git` command fails (e.g. not a git repo), the hydrator falls back to +`os.walk`. + +--- + +## Fragment Structure + +Each file is stored as a `TieredFragment` with: + +```python +TieredFragment( + fragment_id=f"{resource_id}:{rel_path}", + content=file_content, + tier=ContextTier.HOT, + resource_id=resource_id, + project_name=project_name, + token_count=len(content) // 4, # approximate + metadata={ + "path": rel_path, + "detail_depth": "1", + "relevance_score": "0.5", + }, +) +``` + +All fragments are stored at `ContextTier.HOT` (always included in context +assembly). Token count is approximated as `len(content) // 4`. + +--- + +## Integration with `LLMExecuteActor` + +The hydrator is called inside `LLMExecuteActor.execute()` before context +assembly: + +```python +# Simplified from llm_actors.py +from cleveragents.application.services.context_tier_hydrator import ( + hydrate_tiers_for_plan, +) + +class LLMExecuteActor: + def execute(self, plan, ...): + # Hydrate context tiers from project resources + hydrate_tiers_for_plan( + tier_service=self._tier_service, + project_names=plan.project_names, + project_repository=self._project_repository, + resource_registry=self._resource_registry, + ) + # Then assemble context and call LLM + context = self._assembler.assemble(plan) + ... +``` + +Dependencies (`tier_service`, `project_repository`, `resource_registry`) are +injected at the CLI factory level in `_get_plan_executor()`, not via +`get_container()` calls inside services. + +--- + +## Logging + +The hydrator uses `structlog` with the following event keys: + +| Event | Level | Description | +|-------|-------|-------------| +| `context_hydrator.skip_missing_location` | WARNING | Missing path or not directory | +| `context_hydrator.hydrated` | INFO | Hydration complete — reports fragments + bytes | +| `context_hydrator.project_not_found` | DEBUG | Project not found in repository | +| `context_hydrator.no_linked_resources` | DEBUG | Project has no linked resources | +| `context_hydrator.resource_not_found` | DEBUG | Resource not found in registry | +| `context_hydrator.store_failed` | DEBUG | Fragment store failed (non-fatal) | + +[#1028]: https://git.cleverthis.com/cleveragents/cleveragents-core/issues/1028 diff --git a/mkdocs.yml b/mkdocs.yml index 2bb05974c..53dd660ed 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,7 @@ nav: - Shell Safety: modules/shell-safety.md - UKO Provenance Tracking: modules/uko-provenance.md - Invariant Reconciliation: modules/invariant-reconciliation.md + - Context Tier Hydrator: modules/context-tier-hydrator.md - Development: - Agent System Specification: development/agent-system-specification.md - CI/CD Pipeline: development/ci-cd.md