cleveragents/cleveragents-core

Fork 3

UAT: `agents project context simulate` output missing spec-required per-strategy results and fusion details #4766

New issue

Open

opened 2026-04-08 18:55:09 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-08 18:55:09 +00:00

Owner

Bug Report

Feature Area: ACMS — agents project context simulate command

Severity: High — the simulation output is structurally incomplete vs. the spec, missing the per-strategy breakdown and fusion result details that make the command useful for "tuning context view configurations"

What Was Tested

Code-level analysis comparing the context_simulate function in src/cleveragents/cli/commands/project_context.py against the spec's defined output format at docs/specification.md §4325-4500.

Expected Behavior (from spec)

The spec defines the context simulate output (docs/specification.md lines 4349-4384) as showing:

Rich output:

╭─ Strategy Results ───────────────────────────────────────────────╮
│ breadth-depth-navigator                                          │
│   Confidence: 0.85  │  Budget: 7,200 tokens  │  12 fragments     │
│   Top: src/auth/__init__.py (depth 9, 1,240t, score=0.95)        │
│        src/db/models.py (depth 3, 480t, score=0.88)              │
│                                                                  │
│ semantic-embedding                                               │
│   Confidence: 0.60  │  Budget: 5,600 tokens  │  8 fragments      │
│   Top: src/auth/jwt.py (depth 3, 380t, score=0.82)               │
│                                                                  │
│ simple-keyword                                                   │
│   Confidence: 0.30  │  Budget: 3,200 tokens  │  5 fragments      │
│   Top: src/auth/utils.py (depth 6, 210t, score=0.71)             │
╰──────────────────────────────────────────────────────────────────╯

╭─ Fusion Result ──────────────────────────────────────────────────╮
│ Included: 18 fragments (14,820 tokens, 92.6% of budget)          │
│ Excluded: 7 fragments (3,200 tokens, below budget cutoff)        │
│ Deduplicated: 4 fragments merged                                 │
│ Skeleton: 2,400 tokens (15% reserve)                             │
│ Preamble: 180 tokens                                             │
╰──────────────────────────────────────────────────────────────────╯

JSON output (docs/specification.md lines 4425-4500) should include:

{
  "simulation": {
    "project": "...",
    "view": "strategize",
    "budget": 16000,
    "budget_source": "override",
    "focus": [...]
  },
  "strategy_results": [
    {
      "name": "breadth-depth-navigator",
      "confidence": 0.85,
      "budget_tokens": 7200,
      "fragments_returned": 12,
      "top_fragments": [...]
    }
  ],
  "fusion_result": {
    "included_fragments": 18,
    "included_tokens": 14820,
    "budget_utilization": 0.926,
    "excluded_fragments": 7,
    "excluded_tokens": 3200,
    "deduplicated_count": 4,
    "skeleton_tokens": 2400,
    "skeleton_ratio": 0.15,
    "preamble_tokens": 180
  }
}

Actual Behavior

The implementation (src/cleveragents/cli/commands/project_context.py lines 1164-1307) returns a simplified AssembledContext object:

data = {
    "project": project,
    "view": effective_view,
    "total_tokens": assembled.total_tokens,
    "budget_used": assembled.budget_used,
    "strategies_used": assembled.strategies_used,
    "context_hash": assembled.context_hash,
    "preamble": assembled.preamble,
    "fragment_count": len(assembled.fragments),
    "fragments": [...],
    "acms_config": acms,
    "phase_analysis": phase_analysis
}

Missing fields:

strategy_results — no per-strategy confidence, budget allocation, or top fragments
fusion_result.excluded_fragments — how many fragments were excluded
fusion_result.excluded_tokens — tokens excluded
fusion_result.deduplicated_count — how many fragments were merged
fusion_result.skeleton_tokens — tokens reserved for skeleton
fusion_result.preamble_tokens — tokens used by preamble
simulation.budget_source — whether budget was overridden or computed

The Rich output also lacks the "Strategy Results" panel showing per-strategy breakdown.

The _simulate_context_assembly function (lines 286-385) is a stub that only queries the tier service for existing fragments — it does not actually invoke the ACMS pipeline strategies. This means the simulation doesn't show what strategies would contribute or how budget would be allocated.

Impact

The context simulate command cannot be used for its stated purpose: "tuning context view configurations before running plans"
Users cannot see which strategies would contribute, how budget would be allocated, or what would be excluded
The simulation is not a true dry-run of the ACMS pipeline — it just returns existing tier fragments

Code Location

src/cleveragents/cli/commands/project_context.py:

_simulate_context_assembly() function (lines 286-385) — stub implementation
context_simulate() function (lines 1164-1307) — incomplete output

Fix Required

_simulate_context_assembly() should invoke the actual ACMS pipeline (or a dry-run mode) to get per-strategy results
The output should include strategy_results with per-strategy confidence, budget, and top fragments
The output should include fusion_result with excluded/deduplicated/skeleton/preamble details
The Rich output should add a "Strategy Results" panel and update the "Fusion Result" panel

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area:** ACMS — `agents project context simulate` command **Severity:** High — the simulation output is structurally incomplete vs. the spec, missing the per-strategy breakdown and fusion result details that make the command useful for "tuning context view configurations" ### What Was Tested Code-level analysis comparing the `context_simulate` function in `src/cleveragents/cli/commands/project_context.py` against the spec's defined output format at `docs/specification.md` §4325-4500. ### Expected Behavior (from spec) The spec defines the `context simulate` output (`docs/specification.md` lines 4349-4384) as showing: **Rich output:** ``` ╭─ Strategy Results ───────────────────────────────────────────────╮ │ breadth-depth-navigator │ │ Confidence: 0.85 │ Budget: 7,200 tokens │ 12 fragments │ │ Top: src/auth/__init__.py (depth 9, 1,240t, score=0.95) │ │ src/db/models.py (depth 3, 480t, score=0.88) │ │ │ │ semantic-embedding │ │ Confidence: 0.60 │ Budget: 5,600 tokens │ 8 fragments │ │ Top: src/auth/jwt.py (depth 3, 380t, score=0.82) │ │ │ │ simple-keyword │ │ Confidence: 0.30 │ Budget: 3,200 tokens │ 5 fragments │ │ Top: src/auth/utils.py (depth 6, 210t, score=0.71) │ ╰──────────────────────────────────────────────────────────────────╯ ╭─ Fusion Result ──────────────────────────────────────────────────╮ │ Included: 18 fragments (14,820 tokens, 92.6% of budget) │ │ Excluded: 7 fragments (3,200 tokens, below budget cutoff) │ │ Deduplicated: 4 fragments merged │ │ Skeleton: 2,400 tokens (15% reserve) │ │ Preamble: 180 tokens │ ╰──────────────────────────────────────────────────────────────────╯ ``` **JSON output** (`docs/specification.md` lines 4425-4500) should include: ```json { "simulation": { "project": "...", "view": "strategize", "budget": 16000, "budget_source": "override", "focus": [...] }, "strategy_results": [ { "name": "breadth-depth-navigator", "confidence": 0.85, "budget_tokens": 7200, "fragments_returned": 12, "top_fragments": [...] } ], "fusion_result": { "included_fragments": 18, "included_tokens": 14820, "budget_utilization": 0.926, "excluded_fragments": 7, "excluded_tokens": 3200, "deduplicated_count": 4, "skeleton_tokens": 2400, "skeleton_ratio": 0.15, "preamble_tokens": 180 } } ``` ### Actual Behavior The implementation (`src/cleveragents/cli/commands/project_context.py` lines 1164-1307) returns a simplified `AssembledContext` object: ```python data = { "project": project, "view": effective_view, "total_tokens": assembled.total_tokens, "budget_used": assembled.budget_used, "strategies_used": assembled.strategies_used, "context_hash": assembled.context_hash, "preamble": assembled.preamble, "fragment_count": len(assembled.fragments), "fragments": [...], "acms_config": acms, "phase_analysis": phase_analysis } ``` **Missing fields:** - `strategy_results` — no per-strategy confidence, budget allocation, or top fragments - `fusion_result.excluded_fragments` — how many fragments were excluded - `fusion_result.excluded_tokens` — tokens excluded - `fusion_result.deduplicated_count` — how many fragments were merged - `fusion_result.skeleton_tokens` — tokens reserved for skeleton - `fusion_result.preamble_tokens` — tokens used by preamble - `simulation.budget_source` — whether budget was overridden or computed The Rich output also lacks the "Strategy Results" panel showing per-strategy breakdown. The `_simulate_context_assembly` function (lines 286-385) is a stub that only queries the tier service for existing fragments — it does not actually invoke the ACMS pipeline strategies. This means the simulation doesn't show what strategies would contribute or how budget would be allocated. ### Impact - The `context simulate` command cannot be used for its stated purpose: "tuning context view configurations before running plans" - Users cannot see which strategies would contribute, how budget would be allocated, or what would be excluded - The simulation is not a true dry-run of the ACMS pipeline — it just returns existing tier fragments ### Code Location `src/cleveragents/cli/commands/project_context.py`: - `_simulate_context_assembly()` function (lines 286-385) — stub implementation - `context_simulate()` function (lines 1164-1307) — incomplete output ### Fix Required 1. `_simulate_context_assembly()` should invoke the actual ACMS pipeline (or a dry-run mode) to get per-strategy results 2. The output should include `strategy_results` with per-strategy confidence, budget, and top fragments 3. The output should include `fusion_result` with excluded/deduplicated/skeleton/preamble details 4. The Rich output should add a "Strategy Results" panel and update the "Fusion Result" panel --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 referenced this issue

2026-04-08 18:57:54 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4743

HAL9000 added the

labels

2026-04-08 23:00:36 +00:00

HAL9000 commented

2026-04-09 03:04:35 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: Medium — spec compliance bug identified by UAT testing
Story Points: 3 (M) — targeted fix to align implementation with spec
MoSCoW: Must Have — spec compliance is required for correct system behavior

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium — spec compliance bug identified by UAT testing - **Story Points**: 3 (M) — targeted fix to align implementation with spec - **MoSCoW**: Must Have — spec compliance is required for correct system behavior --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner

HAL9000 added

and removed

labels

2026-04-09 03:04:46 +00:00

HAL9000 added this to the v3.5.0 milestone