UAT: `agents project context simulate` output missing spec-required `strategy_results` and `fusion_result` fields — returns flat assembled context instead #2507

New issue

Open

opened 2026-04-03 18:40:51 +00:00 by freemo · 1 comment

freemo commented

2026-04-03 18:40:51 +00:00

Owner

Summary

agents project context simulate returns a flat assembled context object (total_tokens, budget_used, strategies_used, fragment_count, fragments) instead of the spec-required structured simulation output (simulation, strategy_results per-strategy breakdown, fusion_result with deduplication and skeleton statistics). The command also does not include the spec-required message "Simulation complete (dry run -- no context was assembled)".

What Was Tested

Code-level analysis of src/cleveragents/cli/commands/project_context.py (the context_simulate command, lines 1164-1307) against docs/specification.md §4324-4553.

Expected Behavior (from spec §4428-4486)

The JSON output must contain:

{
  "command": "project context simulate",
  "status": "ok",
  "data": {
    "simulation": {
      "project": "local/api-service",
      "view": "strategize",
      "budget": 16000,
      "budget_source": "override",
      "focus": ["uko-py:module/src.auth", "uko-py:module/src.db"]
    },
    "strategy_results": [
      {
        "name": "breadth-depth-navigator",
        "confidence": 0.85,
        "budget_tokens": 7200,
        "fragments": 12,
        "top": [
          { "path": "src/auth/__init__.py", "depth": 9, "tokens": 1240, "score": 0.95 }
        ]
      },
      {
        "name": "semantic-embedding",
        "confidence": 0.60,
        "budget_tokens": 5600,
        "fragments": 8,
        "top": [ ... ]
      },
      {
        "name": "simple-keyword",
        "confidence": 0.30,
        "budget_tokens": 3200,
        "fragments": 5,
        "top": [ ... ]
      }
    ],
    "fusion_result": {
      "included_fragments": 18,
      "included_tokens": 14820,
      "budget_utilization": 0.926,
      "excluded_fragments": 7,
      "excluded_tokens": 3200,
      "deduplicated_fragments": 4,
      "skeleton_tokens": 2400,
      "skeleton_ratio": 0.15,
      "preamble_tokens": 180
    }
  },
  "messages": ["Simulation complete (dry run -- no context was assembled)"]
}

Actual Behavior

The command returns:

{
  "total_tokens": ...,
  "budget_used": ...,
  "strategies_used": [...],
  "context_hash": "...",
  "preamble": "...",
  "fragment_count": ...,
  "fragments": [...],
  "project": "...",
  "view": "...",
  "acms_config": {...},
  "phase_analysis": {...}
}

Missing from output:

simulation object with budget_source field (spec requires "override" when --budget is used, "config" otherwise)
strategy_results array with per-strategy breakdown (confidence, budget_tokens, fragments, top fragments with path/depth/tokens/score)
fusion_result with included_fragments, included_tokens, budget_utilization, excluded_fragments, excluded_tokens, deduplicated_fragments, skeleton_tokens, skeleton_ratio, preamble_tokens
messages array with "Simulation complete (dry run -- no context was assembled)"

The current implementation uses _simulate_context_assembly() which creates a fake AssembledContext from tier data rather than running the actual ACMS pipeline in dry-run mode.

Code Location

src/cleveragents/cli/commands/project_context.py lines 1164-1307 (context_simulate function) and lines 286-385 (_simulate_context_assembly helper)

Steps to Reproduce

Create a project: agents project add local/test-project
Run: agents project context simulate --budget 16000 --format json local/test-project
Observe the output lacks strategy_results and fusion_result fields

Severity

High — The command's output structure is fundamentally different from the spec. The per-strategy breakdown (strategy_results) is the primary value of the simulate command — it shows how each strategy contributes to the context window. The fusion_result shows deduplication and skeleton statistics. Both are entirely absent.

Metadata

Branch: fix/context-simulate-output-structure
Commit Message: fix(cli): align context simulate output with spec — add strategy_results and fusion_result fields
Parent Epic: #396

Subtasks

Add simulation object to output with budget_source field ("override" when --budget is used, "config" otherwise)
Add strategy_results array: run each enabled strategy against the project and collect per-strategy results (confidence, budget_tokens, fragments, top fragments)
Add fusion_result object: run the full pipeline fusion and collect deduplication/skeleton/preamble statistics
Add messages array with "Simulation complete (dry run -- no context was assembled)"
Update Rich output to show per-strategy table and fusion result panel
Add unit tests for the corrected output structure

Definition of Done

agents project context simulate --format json returns simulation, strategy_results, and fusion_result fields matching the spec
messages contains "Simulation complete (dry run -- no context was assembled)"
Rich output shows per-strategy breakdown and fusion result
All tests pass with ≥97% coverage

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Summary `agents project context simulate` returns a flat assembled context object (`total_tokens`, `budget_used`, `strategies_used`, `fragment_count`, `fragments`) instead of the spec-required structured simulation output (`simulation`, `strategy_results` per-strategy breakdown, `fusion_result` with deduplication and skeleton statistics). The command also does not include the spec-required message `"Simulation complete (dry run -- no context was assembled)"`. ## What Was Tested Code-level analysis of `src/cleveragents/cli/commands/project_context.py` (the `context_simulate` command, lines 1164-1307) against `docs/specification.md` §4324-4553. ## Expected Behavior (from spec §4428-4486) The JSON output must contain: ```json { "command": "project context simulate", "status": "ok", "data": { "simulation": { "project": "local/api-service", "view": "strategize", "budget": 16000, "budget_source": "override", "focus": ["uko-py:module/src.auth", "uko-py:module/src.db"] }, "strategy_results": [ { "name": "breadth-depth-navigator", "confidence": 0.85, "budget_tokens": 7200, "fragments": 12, "top": [ { "path": "src/auth/__init__.py", "depth": 9, "tokens": 1240, "score": 0.95 } ] }, { "name": "semantic-embedding", "confidence": 0.60, "budget_tokens": 5600, "fragments": 8, "top": [ ... ] }, { "name": "simple-keyword", "confidence": 0.30, "budget_tokens": 3200, "fragments": 5, "top": [ ... ] } ], "fusion_result": { "included_fragments": 18, "included_tokens": 14820, "budget_utilization": 0.926, "excluded_fragments": 7, "excluded_tokens": 3200, "deduplicated_fragments": 4, "skeleton_tokens": 2400, "skeleton_ratio": 0.15, "preamble_tokens": 180 } }, "messages": ["Simulation complete (dry run -- no context was assembled)"] } ``` ## Actual Behavior The command returns: ```json { "total_tokens": ..., "budget_used": ..., "strategies_used": [...], "context_hash": "...", "preamble": "...", "fragment_count": ..., "fragments": [...], "project": "...", "view": "...", "acms_config": {...}, "phase_analysis": {...} } ``` **Missing from output:** - `simulation` object with `budget_source` field (spec requires `"override"` when `--budget` is used, `"config"` otherwise) - `strategy_results` array with per-strategy breakdown (confidence, budget_tokens, fragments, top fragments with path/depth/tokens/score) - `fusion_result` with `included_fragments`, `included_tokens`, `budget_utilization`, `excluded_fragments`, `excluded_tokens`, `deduplicated_fragments`, `skeleton_tokens`, `skeleton_ratio`, `preamble_tokens` - `messages` array with `"Simulation complete (dry run -- no context was assembled)"` The current implementation uses `_simulate_context_assembly()` which creates a fake `AssembledContext` from tier data rather than running the actual ACMS pipeline in dry-run mode. ## Code Location `src/cleveragents/cli/commands/project_context.py` lines 1164-1307 (`context_simulate` function) and lines 286-385 (`_simulate_context_assembly` helper) ## Steps to Reproduce 1. Create a project: `agents project add local/test-project` 2. Run: `agents project context simulate --budget 16000 --format json local/test-project` 3. Observe the output lacks `strategy_results` and `fusion_result` fields ## Severity **High** — The command's output structure is fundamentally different from the spec. The per-strategy breakdown (`strategy_results`) is the primary value of the simulate command — it shows how each strategy contributes to the context window. The `fusion_result` shows deduplication and skeleton statistics. Both are entirely absent. ## Metadata - **Branch**: `fix/context-simulate-output-structure` - **Commit Message**: `fix(cli): align context simulate output with spec — add strategy_results and fusion_result fields` - **Parent Epic**: #396 ## Subtasks - [ ] Add `simulation` object to output with `budget_source` field (`"override"` when `--budget` is used, `"config"` otherwise) - [ ] Add `strategy_results` array: run each enabled strategy against the project and collect per-strategy results (confidence, budget_tokens, fragments, top fragments) - [ ] Add `fusion_result` object: run the full pipeline fusion and collect deduplication/skeleton/preamble statistics - [ ] Add `messages` array with `"Simulation complete (dry run -- no context was assembled)"` - [ ] Update Rich output to show per-strategy table and fusion result panel - [ ] Add unit tests for the corrected output structure ## Definition of Done - `agents project context simulate --format json` returns `simulation`, `strategy_results`, and `fusion_result` fields matching the spec - `messages` contains `"Simulation complete (dry run -- no context was assembled)"` - Rich output shows per-strategy breakdown and fusion result - All tests pass with ≥97% coverage --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo referenced this issue

2026-04-03 18:43:28 +00:00

[UAT-SUPERVISOR] Progress Report — Cycle 1571 — v3.7.0 #2516

freemo commented

2026-04-03 18:46:48 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
MoSCoW: Should Have — Spec compliance or quality improvement that should be included in the milestone.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **MoSCoW**: Should Have — Spec compliance or quality improvement that should be included in the milestone. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo referenced this issue

2026-04-03 18:54:29 +00:00

[CA-AUTO] UAT Pool Supervisor — v3.x — Session Tracker #2435

freemo referenced this issue

2026-04-04 20:50:23 +00:00

[CA-AUTO] Project Owner — project-owner-1 — Session Tracker #2793

freemo added this to the v3.6.0 milestone

2026-04-05 05:09:42 +00:00