UAT: `agents project context simulate` JSON/YAML output uses wrong structure — missing `simulation`, `strategy_results`, `fusion_result` keys #6332

New issue

Open

opened 2026-04-09 20:11:10 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-09 20:11:10 +00:00

Owner

Summary

The agents project context simulate command's non-rich output does not match the spec. The implementation outputs total_tokens, budget_used, strategies_used, context_hash, preamble, fragment_count, fragments, project, view, acms_config, and phase_analysis, while the spec requires simulation, strategy_results, and fusion_result as the three data sub-objects. The per-strategy breakdown (confidence, budget_tokens, top fragments) and fusion result statistics (excluded_fragments, deduplicated_fragments, skeleton_tokens, preamble_tokens) are entirely absent.

Spec Reference

docs/specification.md lines 4425–4555 (agents project context simulate JSON/YAML example).

Expected JSON output structure:

{
  "command": "project context simulate",
  "status": "ok",
  "exit_code": 0,
  "data": {
    "simulation": {
      "project": "local/api-service",
      "view": "strategize",
      "budget": 16000,
      "budget_source": "override",
      "focus": ["uko-py:module/src.auth", "uko-py:module/src.db"]
    },
    "strategy_results": [
      {
        "name": "breadth-depth-navigator",
        "confidence": 0.85,
        "budget_tokens": 7200,
        "fragments": 12,
        "top": [
          { "path": "src/auth/__init__.py", "depth": 9, "tokens": 1240, "score": 0.95 },
          ...
        ]
      },
      ...
    ],
    "fusion_result": {
      "included_fragments": 18,
      "included_tokens": 14820,
      "budget_utilization": 0.926,
      "excluded_fragments": 7,
      "excluded_tokens": 3200,
      "deduplicated_fragments": 4,
      "skeleton_tokens": 2400,
      "skeleton_ratio": 0.15,
      "preamble_tokens": 180
    }
  },
  "timing": { "duration_ms": 310 },
  "messages": ["Simulation complete (dry run -- no context was assembled)"]
}

Code Location

src/cleveragents/cli/commands/project_context.py:

context_simulate() command (lines 1164–1307)
Lines 1245–1249: data dict built from _assembled_to_dict(assembled) which returns total_tokens, budget_used, strategies_used, context_hash, preamble, fragment_count, fragments
Lines 1246–1249: extended with project, view, acms_config, phase_analysis
_assembled_to_dict() (lines 260–283) — outputs raw AssembledContext fields directly

Actual Behavior

Non-rich output (abbreviated):

{
  "total_tokens": 0,
  "budget_used": 0.0,
  "strategies_used": ["tier_retrieval"],
  "context_hash": "abc123...",
  "preamble": "Simulated context for local/api-service (view: strategize, budget: 8000 tokens)",
  "fragment_count": 0,
  "fragments": [],
  "project": "local/api-service",
  "view": "strategize",
  "acms_config": { ... },
  "phase_analysis": { ... }
}

Missing:

Top-level envelope (command, status, exit_code, timing, messages)
data.simulation (project, view, budget, budget_source, focus)
data.strategy_results (per-strategy confidence, budget_tokens, fragments count, top fragments with path/depth/tokens/score)
data.fusion_result (included_fragments, included_tokens, budget_utilization, excluded_fragments, excluded_tokens, deduplicated_fragments, skeleton_tokens, skeleton_ratio, preamble_tokens)

Also: the rich output (lines 1251–1305) shows a summary panel and fragment table, but is missing the spec-required Strategy Results panel with per-strategy breakdown (spec lines 4360–4374). The Fusion Result panel (spec lines 4376–4382) is also absent.

Steps to Reproduce

Inspect project_context.py lines 1245–1249 and _assembled_to_dict() (lines 260–283) to see what is currently included.

Expected vs Actual

Field	Expected	Actual
Top-level	`{command, status, exit_code, data, timing, messages}`	Flat merged dict
`data.simulation`	`{project, view, budget, budget_source, focus}`	Absent (scattered at top level)
`data.strategy_results`	Per-strategy `{name, confidence, budget_tokens, fragments, top[]}`	Absent (only `strategies_used` as list of names)
`data.fusion_result`	`{included_fragments, excluded_fragments, deduplicated_fragments, skeleton_tokens, preamble_tokens, ...}`	Absent
Rich `Strategy Results` panel	Per-strategy details	Absent
Rich `Fusion Result` panel	Fusion statistics	Absent

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Summary The `agents project context simulate` command's non-rich output does not match the spec. The implementation outputs `total_tokens`, `budget_used`, `strategies_used`, `context_hash`, `preamble`, `fragment_count`, `fragments`, `project`, `view`, `acms_config`, and `phase_analysis`, while the spec requires `simulation`, `strategy_results`, and `fusion_result` as the three `data` sub-objects. The per-strategy breakdown (confidence, budget_tokens, top fragments) and fusion result statistics (excluded_fragments, deduplicated_fragments, skeleton_tokens, preamble_tokens) are entirely absent. ## Spec Reference `docs/specification.md` lines 4425–4555 (`agents project context simulate` JSON/YAML example). Expected JSON output structure: ```json { "command": "project context simulate", "status": "ok", "exit_code": 0, "data": { "simulation": { "project": "local/api-service", "view": "strategize", "budget": 16000, "budget_source": "override", "focus": ["uko-py:module/src.auth", "uko-py:module/src.db"] }, "strategy_results": [ { "name": "breadth-depth-navigator", "confidence": 0.85, "budget_tokens": 7200, "fragments": 12, "top": [ { "path": "src/auth/__init__.py", "depth": 9, "tokens": 1240, "score": 0.95 }, ... ] }, ... ], "fusion_result": { "included_fragments": 18, "included_tokens": 14820, "budget_utilization": 0.926, "excluded_fragments": 7, "excluded_tokens": 3200, "deduplicated_fragments": 4, "skeleton_tokens": 2400, "skeleton_ratio": 0.15, "preamble_tokens": 180 } }, "timing": { "duration_ms": 310 }, "messages": ["Simulation complete (dry run -- no context was assembled)"] } ``` ## Code Location `src/cleveragents/cli/commands/project_context.py`: - `context_simulate()` command (lines 1164–1307) - Lines 1245–1249: data dict built from `_assembled_to_dict(assembled)` which returns `total_tokens`, `budget_used`, `strategies_used`, `context_hash`, `preamble`, `fragment_count`, `fragments` - Lines 1246–1249: extended with `project`, `view`, `acms_config`, `phase_analysis` - `_assembled_to_dict()` (lines 260–283) — outputs raw `AssembledContext` fields directly ## Actual Behavior Non-rich output (abbreviated): ```json { "total_tokens": 0, "budget_used": 0.0, "strategies_used": ["tier_retrieval"], "context_hash": "abc123...", "preamble": "Simulated context for local/api-service (view: strategize, budget: 8000 tokens)", "fragment_count": 0, "fragments": [], "project": "local/api-service", "view": "strategize", "acms_config": { ... }, "phase_analysis": { ... } } ``` Missing: - Top-level envelope (`command`, `status`, `exit_code`, `timing`, `messages`) - `data.simulation` (project, view, budget, budget_source, focus) - `data.strategy_results` (per-strategy confidence, budget_tokens, fragments count, top fragments with path/depth/tokens/score) - `data.fusion_result` (included_fragments, included_tokens, budget_utilization, excluded_fragments, excluded_tokens, deduplicated_fragments, skeleton_tokens, skeleton_ratio, preamble_tokens) Also: the rich output (lines 1251–1305) shows a summary panel and fragment table, but is missing the spec-required `Strategy Results` panel with per-strategy breakdown (spec lines 4360–4374). The `Fusion Result` panel (spec lines 4376–4382) is also absent. ## Steps to Reproduce Inspect `project_context.py` lines 1245–1249 and `_assembled_to_dict()` (lines 260–283) to see what is currently included. ## Expected vs Actual | Field | Expected | Actual | |-------|----------|--------| | Top-level | `{command, status, exit_code, data, timing, messages}` | Flat merged dict | | `data.simulation` | `{project, view, budget, budget_source, focus}` | Absent (scattered at top level) | | `data.strategy_results` | Per-strategy `{name, confidence, budget_tokens, fragments, top[]}` | Absent (only `strategies_used` as list of names) | | `data.fusion_result` | `{included_fragments, excluded_fragments, deduplicated_fragments, skeleton_tokens, preamble_tokens, ...}` | Absent | | Rich `Strategy Results` panel | Per-strategy details | Absent | | Rich `Fusion Result` panel | Fusion statistics | Absent | --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 referenced this issue

2026-04-09 20:18:09 +00:00

[AUTO-UAT-POOL] UAT Testing Report (Cycle 3) #6294

HAL9000 added the

labels

2026-04-09 20:18:44 +00:00