UAT: agents project context simulate uses stub implementation instead of real ACMS pipeline — spec requires dry-run of actual pipeline #2098

Open
opened 2026-04-03 04:02:49 +00:00 by freemo · 1 comment
Owner

Metadata

  • Branch: fix/acms-context-simulate-real-pipeline
  • Commit Message: fix(cli): wire context simulate to real ACMS pipeline instead of stub
  • Milestone: v3.7.0
  • Parent Epic: #396

Summary

The agents project context simulate command is documented in the spec as:

Purpose: Simulate a context assembly run without invoking an actor. Produces a dry-run report showing what the Context Assembly Pipeline would include in the context payload, which strategies contributed, how the budget was allocated, and what was excluded. Useful for tuning context view configurations before running plans.

However, the current implementation in src/cleveragents/cli/commands/project_context.py uses a stub _simulate_context_assembly() function that:

  1. Does NOT invoke the real ACMSPipeline or ContextAssemblyPipeline
  2. Does NOT use the configured strategies from the ACMS config
  3. Simply retrieves fragments from the ContextTierService and applies a naive token-count filter
  4. Generates a fake context_hash using a simple string hash instead of the real compute_context_hash()
  5. Does NOT show which strategies contributed or how the budget was allocated

The comment in the code itself acknowledges this: "When the full ACMS pipeline is wired, this will delegate to the live pipeline instead."

Expected Behavior (from spec)

agents project context simulate should:

  1. Invoke the real ContextAssemblyPipeline (or ACMSPipeline) with the project's configured strategies
  2. Show which strategies were selected by ConfidenceWeightedSelector
  3. Show how the budget was allocated by ProportionalBudgetAllocator
  4. Show which fragments were included/excluded after GreedyKnapsackPacker
  5. Show the real context_hash from compute_context_hash()
  6. Support --strategy override to test specific strategies
  7. Support --focus to test specific UKO URIs

Actual Behavior

The simulate command produces a simplified output that:

  • Only shows fragments already in the tier service (not what the pipeline would retrieve)
  • Uses tier_retrieval as the strategy name regardless of configuration
  • Does not show budget allocation details
  • Does not show deduplication or scoring results
  • Generates a truncated 16-char hash instead of the full SHA-256

Code Location

src/cleveragents/cli/commands/project_context.py:

  • _simulate_context_assembly() function (lines ~200-280): stub implementation
  • context_simulate() command (lines ~580-680): calls the stub

The real pipeline is available at:

  • cleveragents.application.services.acms_pipeline.ContextAssemblyPipeline
  • cleveragents.application.services.acms_service.ACMSPipeline

Impact

  • High severity: The simulate command is a key debugging tool for ACMS context tuning. Users relying on it to understand context assembly behavior are getting incorrect information.
  • The spec explicitly defines simulate as a dry-run of the real pipeline, not a simplified approximation.

Subtasks

  • Wire context_simulate to use the real ContextAssemblyPipeline
  • Pass the project's configured strategies from acms_config["strategies"] to the pipeline
  • Pass the --budget override to the pipeline's ContextBudget
  • Pass --focus URIs to the ContextRequest
  • Pass --strategy overrides to the pipeline
  • Show real per-stage timing from ContextAssemblyPipeline.last_timings
  • Add BDD scenario testing that simulate uses the real pipeline
  • Add integration test for context simulate with a real project

Definition of Done

  • agents project context simulate invokes the real ContextAssemblyPipeline
  • Output shows real strategy selection, budget allocation, and fragment selection
  • --strategy, --focus, and --budget overrides work correctly
  • BDD and integration tests added and passing
  • No regressions

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Metadata - **Branch**: `fix/acms-context-simulate-real-pipeline` - **Commit Message**: `fix(cli): wire context simulate to real ACMS pipeline instead of stub` - **Milestone**: v3.7.0 - **Parent Epic**: #396 ## Summary The `agents project context simulate` command is documented in the spec as: > **Purpose**: Simulate a context assembly run without invoking an actor. Produces a dry-run report showing what the Context Assembly Pipeline would include in the context payload, which strategies contributed, how the budget was allocated, and what was excluded. Useful for tuning context view configurations before running plans. However, the current implementation in `src/cleveragents/cli/commands/project_context.py` uses a stub `_simulate_context_assembly()` function that: 1. Does NOT invoke the real `ACMSPipeline` or `ContextAssemblyPipeline` 2. Does NOT use the configured strategies from the ACMS config 3. Simply retrieves fragments from the `ContextTierService` and applies a naive token-count filter 4. Generates a fake `context_hash` using a simple string hash instead of the real `compute_context_hash()` 5. Does NOT show which strategies contributed or how the budget was allocated The comment in the code itself acknowledges this: `"When the full ACMS pipeline is wired, this will delegate to the live pipeline instead."` ## Expected Behavior (from spec) `agents project context simulate` should: 1. Invoke the real `ContextAssemblyPipeline` (or `ACMSPipeline`) with the project's configured strategies 2. Show which strategies were selected by `ConfidenceWeightedSelector` 3. Show how the budget was allocated by `ProportionalBudgetAllocator` 4. Show which fragments were included/excluded after `GreedyKnapsackPacker` 5. Show the real `context_hash` from `compute_context_hash()` 6. Support `--strategy` override to test specific strategies 7. Support `--focus` to test specific UKO URIs ## Actual Behavior The simulate command produces a simplified output that: - Only shows fragments already in the tier service (not what the pipeline would retrieve) - Uses `tier_retrieval` as the strategy name regardless of configuration - Does not show budget allocation details - Does not show deduplication or scoring results - Generates a truncated 16-char hash instead of the full SHA-256 ## Code Location `src/cleveragents/cli/commands/project_context.py`: - `_simulate_context_assembly()` function (lines ~200-280): stub implementation - `context_simulate()` command (lines ~580-680): calls the stub The real pipeline is available at: - `cleveragents.application.services.acms_pipeline.ContextAssemblyPipeline` - `cleveragents.application.services.acms_service.ACMSPipeline` ## Impact - High severity: The `simulate` command is a key debugging tool for ACMS context tuning. Users relying on it to understand context assembly behavior are getting incorrect information. - The spec explicitly defines `simulate` as a dry-run of the real pipeline, not a simplified approximation. ## Subtasks - [ ] Wire `context_simulate` to use the real `ContextAssemblyPipeline` - [ ] Pass the project's configured strategies from `acms_config["strategies"]` to the pipeline - [ ] Pass the `--budget` override to the pipeline's `ContextBudget` - [ ] Pass `--focus` URIs to the `ContextRequest` - [ ] Pass `--strategy` overrides to the pipeline - [ ] Show real per-stage timing from `ContextAssemblyPipeline.last_timings` - [ ] Add BDD scenario testing that simulate uses the real pipeline - [ ] Add integration test for `context simulate` with a real project ## Definition of Done - [ ] `agents project context simulate` invokes the real `ContextAssemblyPipeline` - [ ] Output shows real strategy selection, budget allocation, and fragment selection - [ ] `--strategy`, `--focus`, and `--budget` overrides work correctly - [ ] BDD and integration tests added and passing - [ ] No regressions --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester
freemo added this to the v3.7.0 milestone 2026-04-03 04:03:12 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: High (confirmed) — The context simulate command is a key debugging tool for ACMS context tuning. Using a stub instead of the real pipeline means users get incorrect information about context assembly behavior.
  • Milestone: v3.7.0 (confirmed — part of the ACMS/Context Epic #396)
  • MoSCoW: Should Have — The spec defines simulate as a dry-run of the real pipeline. The current stub implementation is a significant deviation. Important for ACMS debugging and tuning workflows.
  • Parent Epic: #396 (confirmed correct)

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: High (confirmed) — The `context simulate` command is a key debugging tool for ACMS context tuning. Using a stub instead of the real pipeline means users get incorrect information about context assembly behavior. - **Milestone**: v3.7.0 (confirmed — part of the ACMS/Context Epic #396) - **MoSCoW**: Should Have — The spec defines `simulate` as a dry-run of the real pipeline. The current stub implementation is a significant deviation. Important for ACMS debugging and tuning workflows. - **Parent Epic**: #396 (confirmed correct) --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
freemo self-assigned this 2026-04-03 16:58:07 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#396 Epic: ACMS Context Pipeline
cleveragents/cleveragents-core
Reference
cleveragents/cleveragents-core#2098
No description provided.