bug(acms): ACMS indexing pipeline not wired into CLI — ContextTierService starts empty on every invocation #1028

Closed
opened 2026-03-17 13:11:40 +00:00 by hurui200320 · 7 comments
Member

Metadata

  • Commit Message: fix(acms): wire ACMS indexing pipeline into CLI so ContextTierService is populated during context operations
  • Branch: bugfix/m5-acms-cli-indexing-pipeline-wiring

Background

The v3.4.0 (M5) milestone goal states:

The Advanced Context Management System v1 is operational. Projects with 10,000+ files can be indexed and queried. The context assembly pipeline produces scoped, budget-constrained context views for actors. Hot/warm/cold storage tiers manage context lifecycle.

However, ContextTierService (src/cleveragents/application/services/context_tiers.py) is an in-memory singleton that uses plain Python dicts for tier storage (self._hot, self._warm, self._cold). Because each CLI invocation is a separate Python process, the singleton starts empty every time. There is no integration point that:

  1. Triggers the ACMS indexing pipeline when context CLI commands are invoked
  2. Populates ContextTierService with indexed data from the project's filesystem
  3. Persists tier state across CLI process boundaries (or hydrates from the database on startup)

This means:

  • project context simulate operates on zero data — the ContextTierService has no fragments
  • project context inspect reports no indexed resources
  • Budget enforcement cannot be validated behaviorally — there's nothing to constrain
  • 10K+ file indexing is never triggered — files exist on disk but are never scanned
  • The E2E acceptance tests (PR !811 / ticket #745) can only validate structural plumbing (JSON storage, config), not actual ACMS behavioral output

Expected Behavior

When a user runs context CLI commands (project context simulate, project context inspect, context list, context add, etc.), the ACMS indexing pipeline should:

  1. Scan the project's filesystem for resources
  2. Index discovered resources into the ContextTierService tiers
  3. Apply budget constraints (max_file_size, max_total_size) during indexing
  4. Produce meaningful tier data that simulate and inspect can report on
  5. Handle 10,000+ file projects without timeout

Actual Behavior

The ContextTierService starts empty on every CLI process invocation. No indexing occurs. All context commands operate on zero data.

Relevant Code Locations

  • src/cleveragents/application/services/context_tiers.pyContextTierService.__init__ creates empty dicts
  • src/cleveragents/application/container.py:515ContextTierService registered as providers.Singleton
  • src/cleveragents/cli/commands/project_context.py — CLI commands that should trigger indexing
  • Individual pipeline component issues: #935 (epic), #540 (Phase 2), #821 (tier runtime logic)
  • #745 / PR !811 — E2E acceptance tests that document this limitation as "structural plumbing only"
  • #935 — Epic: ACMS Context Assembly Pipeline
  • #540 — Pipeline Phase 2 components
  • #821 — ContextTierService runtime promotion/demotion/eviction logic
  • #847 — Budget enforcement
  • #848 — Context assembly CLI functional
  • #849 — Context analysis
  • #851 — 10K+ file indexing

Subtasks

  • Wire the ACMS indexing pipeline into CLI context commands so ContextTierService is populated with real filesystem data
  • Add persistence or hydration mechanism so tier state survives across CLI process boundaries
  • Connect ContextTierService to the DI container with proper lifecycle management
  • Ensure budget constraints (max_file_size, max_total_size) are applied during indexing
  • Validate that 10K+ file projects index without timeout
  • Remove @tdd_expected_fail tag from the TDD test once the fix is implemented
  • Run nox (all default sessions), fix any errors
  • Verify coverage >=97% via nox -s coverage_report

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • The TDD behavioral test (tagged @tdd_expected_fail in the companion TDD issue) passes with the @tdd_expected_fail tag removed.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
## Metadata - **Commit Message**: `fix(acms): wire ACMS indexing pipeline into CLI so ContextTierService is populated during context operations` - **Branch**: `bugfix/m5-acms-cli-indexing-pipeline-wiring` ## Background The v3.4.0 (M5) milestone goal states: > **The Advanced Context Management System v1 is operational.** Projects with 10,000+ files can be indexed and queried. The context assembly pipeline produces scoped, budget-constrained context views for actors. Hot/warm/cold storage tiers manage context lifecycle. However, `ContextTierService` (`src/cleveragents/application/services/context_tiers.py`) is an **in-memory singleton** that uses plain Python dicts for tier storage (`self._hot`, `self._warm`, `self._cold`). Because each CLI invocation is a separate Python process, the singleton starts **empty** every time. There is no integration point that: 1. Triggers the ACMS indexing pipeline when context CLI commands are invoked 2. Populates `ContextTierService` with indexed data from the project's filesystem 3. Persists tier state across CLI process boundaries (or hydrates from the database on startup) This means: - `project context simulate` operates on **zero data** — the ContextTierService has no fragments - `project context inspect` reports **no indexed resources** - **Budget enforcement cannot be validated behaviorally** — there's nothing to constrain - **10K+ file indexing is never triggered** — files exist on disk but are never scanned - The E2E acceptance tests (PR !811 / ticket #745) can only validate structural plumbing (JSON storage, config), not actual ACMS behavioral output ## Expected Behavior When a user runs context CLI commands (`project context simulate`, `project context inspect`, `context list`, `context add`, etc.), the ACMS indexing pipeline should: 1. Scan the project's filesystem for resources 2. Index discovered resources into the `ContextTierService` tiers 3. Apply budget constraints (`max_file_size`, `max_total_size`) during indexing 4. Produce meaningful tier data that `simulate` and `inspect` can report on 5. Handle 10,000+ file projects without timeout ## Actual Behavior The `ContextTierService` starts empty on every CLI process invocation. No indexing occurs. All context commands operate on zero data. ## Relevant Code Locations - `src/cleveragents/application/services/context_tiers.py` — `ContextTierService.__init__` creates empty dicts - `src/cleveragents/application/container.py:515` — `ContextTierService` registered as `providers.Singleton` - `src/cleveragents/cli/commands/project_context.py` — CLI commands that should trigger indexing - Individual pipeline component issues: #935 (epic), #540 (Phase 2), #821 (tier runtime logic) ## Related Issues - #745 / PR !811 — E2E acceptance tests that document this limitation as "structural plumbing only" - #935 — Epic: ACMS Context Assembly Pipeline - #540 — Pipeline Phase 2 components - #821 — ContextTierService runtime promotion/demotion/eviction logic - #847 — Budget enforcement - #848 — Context assembly CLI functional - #849 — Context analysis - #851 — 10K+ file indexing ## Subtasks - [ ] Wire the ACMS indexing pipeline into CLI context commands so `ContextTierService` is populated with real filesystem data - [ ] Add persistence or hydration mechanism so tier state survives across CLI process boundaries - [ ] Connect `ContextTierService` to the DI container with proper lifecycle management - [ ] Ensure budget constraints (`max_file_size`, `max_total_size`) are applied during indexing - [ ] Validate that 10K+ file projects index without timeout - [ ] Remove `@tdd_expected_fail` tag from the TDD test once the fix is implemented - [ ] Run `nox` (all default sessions), fix any errors - [ ] Verify coverage >=97% via `nox -s coverage_report` ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - The TDD behavioral test (tagged `@tdd_expected_fail` in the companion TDD issue) passes with the `@tdd_expected_fail` tag removed. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.
hurui200320 added this to the v3.4.0 milestone 2026-03-17 13:11:46 +00:00
Author
Member

TDD companion issue: #1029

Per the Bug Fix Workflow, this bug issue is blocked by the TDD issue. The TDD test must be merged to master first, then the bug fix branch removes the @tdd_expected_fail tag when the fix is implemented.

**TDD companion issue:** #1029 Per the [Bug Fix Workflow](CONTRIBUTING.md#bug-fix-workflow), this bug issue is blocked by the TDD issue. The TDD test must be merged to `master` first, then the bug fix branch removes the `@tdd_expected_fail` tag when the fix is implemented.
Owner

Assigned to @hamza.khyari for bug fix based on developer expertise (ACMS pipeline infrastructure — Hamza). State changed from Unverified to Verified. This bug and its TDD counterpart (#1029) are top priority per project policy — bugs always take precedence over feature work.

Assigned to @hamza.khyari for bug fix based on developer expertise (ACMS pipeline infrastructure — Hamza). State changed from Unverified to Verified. This bug and its TDD counterpart (#1029) are top priority per project policy — bugs always take precedence over feature work.
Owner

Planning Agent — Discussion Review

@hurui200320 — Correct citation of CONTRIBUTING.md §Bug Fix Workflow. The dependency chain is properly documented: this bug (#1028) is blocked by TDD issue #1029, and the workflow sequence (TDD test merged → bugfix branch → remove @tdd_expected_fail tag) is accurate.

@hamza.khyari — You are the assigned bug fix developer for this issue (ACMS pipeline infrastructure). Once TDD issue #1029 is merged to master by @hurui200320, you should:

  1. Create branch bugfix/m5-acms-cli-indexing-pipeline-wiring from master.
  2. Wire the ACMS indexing pipeline into the CLI so ContextTierService is populated on invocation.
  3. Remove the @tdd_expected_fail tag from the test.
  4. Open a PR with Closes #1028.

This is a 13-point issue at Priority/Critical — the largest bug fix currently open. Make sure to break the implementation into atomic commits per CONTRIBUTING.md §Atomic Commits if the wiring touches multiple subsystems.

## Planning Agent — Discussion Review @hurui200320 — Correct citation of CONTRIBUTING.md §Bug Fix Workflow. The dependency chain is properly documented: this bug (#1028) is blocked by TDD issue #1029, and the workflow sequence (TDD test merged → bugfix branch → remove `@tdd_expected_fail` tag) is accurate. @hamza.khyari — You are the assigned bug fix developer for this issue (ACMS pipeline infrastructure). Once TDD issue #1029 is merged to `master` by @hurui200320, you should: 1. Create branch `bugfix/m5-acms-cli-indexing-pipeline-wiring` from `master`. 2. Wire the ACMS indexing pipeline into the CLI so `ContextTierService` is populated on invocation. 3. Remove the `@tdd_expected_fail` tag from the test. 4. Open a PR with `Closes #1028`. This is a 13-point issue at Priority/Critical — the largest bug fix currently open. Make sure to break the implementation into atomic commits per CONTRIBUTING.md §Atomic Commits if the wiring touches multiple subsystems.
Member

Implementation Journal — Awaiting TDD Merge

Picking up this issue. Per CONTRIBUTING.md §Bug Fix Workflow, this bug is blocked by TDD companion #1029, which must be merged to master before the bugfix branch can proceed.

Current status of #1029:

  • Branch tdd/m5-acms-cli-indexing-pipeline-wiring is pushed with commit 640e90a2
  • All 7 subtasks complete, self-QA approved (0C/0M), all nox gates passing (coverage 98.4%)
  • No PR has been created yet — the branch exists on remote but awaits a PR to master

Blocked until #1029 is merged. Will begin implementation on bugfix/m5-acms-cli-indexing-pipeline-wiring once that lands on master.

## Implementation Journal — Awaiting TDD Merge Picking up this issue. Per CONTRIBUTING.md §Bug Fix Workflow, this bug is blocked by TDD companion #1029, which must be merged to `master` before the bugfix branch can proceed. **Current status of #1029:** - Branch `tdd/m5-acms-cli-indexing-pipeline-wiring` is pushed with commit `640e90a2` - All 7 subtasks complete, self-QA approved (0C/0M), all nox gates passing (coverage 98.4%) - **No PR has been created yet** — the branch exists on remote but awaits a PR to `master` Blocked until #1029 is merged. Will begin implementation on `bugfix/m5-acms-cli-indexing-pipeline-wiring` once that lands on `master`.
Owner

Architect Guidance: Critical Wiring Gap — ACMS Indexing Pipeline

This is a critical architectural gap for M5 (v3.4.0). The ACMS is the core deliverable for this milestone and it's not operational.

Root Cause Analysis

The fundamental issue is that ContextTierService is an in-memory singleton that resets on every CLI invocation. This is a process-boundary problem:

Architecture Decision: Two-Phase Solution

Phase 1 (CLI mode — immediate fix):

  • Add a hydration step at CLI startup that populates ContextTierService from the database
  • When context commands are invoked, trigger the indexing pipeline to scan project resources
  • Persist tier state to the database after indexing (so subsequent invocations don't re-scan unchanged files)
  • Use file modification timestamps to detect changes and only re-index modified files

Phase 2 (Server mode — M9):

  • In server mode, ContextTierService lives in the long-running server process
  • The indexing pipeline runs as a background task triggered by resource change events
  • Tier state is persisted to the database for crash recovery

Immediate Priority

For M5, the minimum viable fix is:

  1. Wire the indexing pipeline into context CLI commands
  2. Persist indexed fragments to the database
  3. Hydrate ContextTierService from the database on startup
  4. Apply budget constraints during indexing

This unblocks the entire ACMS feature set and makes the E2E acceptance tests meaningful.

  • #4559 — DI container wires base ACMSPipeline instead of production ContextAssemblyPipeline
  • #4560ContextStrategy protocol wrong signature

All three issues (#1028, #4559, #4560) need to be fixed together to make ACMS operational.


🤖 CleverAgents Bot (architect-1)

## Architect Guidance: Critical Wiring Gap — ACMS Indexing Pipeline This is a **critical architectural gap** for M5 (v3.4.0). The ACMS is the core deliverable for this milestone and it's not operational. ### Root Cause Analysis The fundamental issue is that `ContextTierService` is an in-memory singleton that resets on every CLI invocation. This is a process-boundary problem: ### Architecture Decision: Two-Phase Solution **Phase 1 (CLI mode — immediate fix):** - Add a **hydration step** at CLI startup that populates `ContextTierService` from the database - When context commands are invoked, trigger the indexing pipeline to scan project resources - Persist tier state to the database after indexing (so subsequent invocations don't re-scan unchanged files) - Use file modification timestamps to detect changes and only re-index modified files **Phase 2 (Server mode — M9):** - In server mode, `ContextTierService` lives in the long-running server process - The indexing pipeline runs as a background task triggered by resource change events - Tier state is persisted to the database for crash recovery ### Immediate Priority For M5, the minimum viable fix is: 1. Wire the indexing pipeline into context CLI commands 2. Persist indexed fragments to the database 3. Hydrate `ContextTierService` from the database on startup 4. Apply budget constraints during indexing This unblocks the entire ACMS feature set and makes the E2E acceptance tests meaningful. ### Related Issues - **#4559** — DI container wires base `ACMSPipeline` instead of production `ContextAssemblyPipeline` - **#4560** — `ContextStrategy` protocol wrong signature All three issues (#1028, #4559, #4560) need to be fixed together to make ACMS operational. --- *🤖 CleverAgents Bot (architect-1)*
Owner

@HAL9000 I am passing this issue onto you to address and fix.

@HAL9000 I am passing this issue onto you to address and fix.
Owner

State label reconciliation:

  • Previous state: State/In Progress
  • Corrected to: State/Completed
  • Reason: Issue is closed but had incorrect/missing terminal state label. Work was completed via merged PR #4219.

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: backlog-groomer

State label reconciliation: - Previous state: State/In Progress - Corrected to: State/Completed - Reason: Issue is closed but had incorrect/missing terminal state label. Work was completed via merged PR #4219. --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: backlog-groomer
Sign in to join this conversation.
No milestone
No project
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1028
No description provided.