feat(acms): context analysis produces meaningful summaries #849

Closed
opened 2026-03-13 21:59:15 +00:00 by freemo · 2 comments
Owner

Metadata

  • Commit Message: feat(acms): context analysis produces meaningful summaries
  • Branch: feature/m5-context-analysis

Background

M5 (v3.4.0) acceptance criterion: context analysis must produce meaningful summaries of the assembled context. When context is assembled for a plan phase, the system should be able to generate a human-readable summary describing what context is included, its relevance, and any budget constraints that affected the assembly.

Per the E2E suite (m5_e2e_verification.robot line 78), the scoped context output test verifies that ACMS produces scoped context output for each phase with narrowing size limits from strategize through execute to apply.

Expected Behavior

  1. Context analysis summarizes what resources are included in each phase's context
  2. Summaries indicate size/budget utilization per phase
  3. Narrowing context behavior is visible (strategize has most, apply has least)
  4. Analysis output is human-readable and useful for debugging context assembly issues

Acceptance Criteria

  • Context analysis produces per-phase summaries
  • Summaries include resource counts and size utilization
  • Narrowing context from strategize -> execute -> apply is reflected
  • Output is human-readable and actionable
  • Robot E2E test ACMS Scoped Context Output Per Phase passes
  • Unit tests cover: empty context, single resource, multiple resources, budget-constrained

Supporting Information

  • E2E test: robot/m5_e2e_verification.robot line 78
  • Related: budget enforcement and context CLI (sibling issues)

Subtasks

  • Implement context analysis summary generation
  • Add per-phase summary with resource counts and size info
  • Implement narrowing context visualization
  • Tests (Behave): Add scenarios for context analysis edge cases
  • Tests (Robot): Verify E2E acceptance test passes
  • Verify coverage >=97% via nox -s coverage_report
  • Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
## Metadata - **Commit Message**: `feat(acms): context analysis produces meaningful summaries` - **Branch**: `feature/m5-context-analysis` ## Background M5 (v3.4.0) acceptance criterion: context analysis must produce meaningful summaries of the assembled context. When context is assembled for a plan phase, the system should be able to generate a human-readable summary describing what context is included, its relevance, and any budget constraints that affected the assembly. Per the E2E suite (`m5_e2e_verification.robot` line 78), the scoped context output test verifies that ACMS produces scoped context output for each phase with narrowing size limits from strategize through execute to apply. ## Expected Behavior 1. Context analysis summarizes what resources are included in each phase's context 2. Summaries indicate size/budget utilization per phase 3. Narrowing context behavior is visible (strategize has most, apply has least) 4. Analysis output is human-readable and useful for debugging context assembly issues ## Acceptance Criteria - [x] Context analysis produces per-phase summaries - [x] Summaries include resource counts and size utilization - [x] Narrowing context from strategize -> execute -> apply is reflected - [x] Output is human-readable and actionable - [x] Robot E2E test `ACMS Scoped Context Output Per Phase` passes - [x] Unit tests cover: empty context, single resource, multiple resources, budget-constrained ## Supporting Information - E2E test: `robot/m5_e2e_verification.robot` line 78 - Related: budget enforcement and context CLI (sibling issues) ## Subtasks - [x] Implement context analysis summary generation - [x] Add per-phase summary with resource counts and size info - [x] Implement narrowing context visualization - [x] Tests (Behave): Add scenarios for context analysis edge cases - [x] Tests (Robot): Verify E2E acceptance test passes - [x] Verify coverage >=97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.
freemo added this to the v3.4.0 milestone 2026-03-13 21:59:49 +00:00
Member

Started implementation for #849 on branch feature/m5-context-analysis.

Planned implementation shape (spec-aligned):

  1. Add a dedicated phase-analysis utility that computes per-phase summaries (strategize, execute, apply) from ProjectContextPolicy + tiered fragments, including:
    • resource_count, fragment_count, total_bytes, total_tokens, budget_used_ratio
    • exclusion counters for resource/path/max_file_size/max_total_size
    • human-readable summary text
    • narrowing diagnostics (strategize >= execute >= apply)
  2. Integrate this summary into agents project context inspect and agents project context simulate output (structured output + rich table/panel).
  3. Upgrade Robot M5 scoped-context acceptance helper to verify meaningful per-phase analysis output and narrowing behavior (not only view inheritance).
  4. Add Behave coverage for required edge cases: empty context, single resource, multiple resources, and budget-constrained narrowing.

Files being updated:

  • src/cleveragents/application/services/context_phase_analysis.py (new)
  • src/cleveragents/cli/commands/project_context.py
  • robot/helper_m5_e2e_verification.py
  • features/project_context_phase_analysis.feature (new)
  • features/steps/project_context_phase_analysis_steps.py (new)

I will run targeted Behave/Robot first and store logs under build/test-logs/, then run required nox gates per prompt/contributing.

Started implementation for #849 on branch `feature/m5-context-analysis`. Planned implementation shape (spec-aligned): 1. Add a dedicated phase-analysis utility that computes per-phase summaries (`strategize`, `execute`, `apply`) from `ProjectContextPolicy` + tiered fragments, including: - resource_count, fragment_count, total_bytes, total_tokens, budget_used_ratio - exclusion counters for resource/path/max_file_size/max_total_size - human-readable summary text - narrowing diagnostics (strategize >= execute >= apply) 2. Integrate this summary into `agents project context inspect` and `agents project context simulate` output (structured output + rich table/panel). 3. Upgrade Robot M5 scoped-context acceptance helper to verify meaningful per-phase analysis output and narrowing behavior (not only view inheritance). 4. Add Behave coverage for required edge cases: empty context, single resource, multiple resources, and budget-constrained narrowing. Files being updated: - `src/cleveragents/application/services/context_phase_analysis.py` (new) - `src/cleveragents/cli/commands/project_context.py` - `robot/helper_m5_e2e_verification.py` - `features/project_context_phase_analysis.feature` (new) - `features/steps/project_context_phase_analysis_steps.py` (new) I will run targeted Behave/Robot first and store logs under `build/test-logs/`, then run required nox gates per prompt/contributing.
Member

Implementation update for #849 (branch feature/m5-context-analysis):

What was implemented

1) New phase-analysis service

Added src/cleveragents/application/services/context_phase_analysis.py with analyze_phase_summaries(...).

Behavior:

  • Builds per-phase summaries for strategize, execute, apply from:
    • ProjectContextPolicy (resolved view rules)
    • scoped tier fragments (TieredFragment)
    • budget token limit
  • Applies include/exclude resource filtering, include/exclude path filtering, max-file-size checks, and max-total-size checks.
  • Computes per-phase metrics:
    • fragment_count, resource_count, total_bytes, total_tokens, budget_used_ratio
    • exclusion counters by reason
    • human-readable summary text
  • Computes narrowing diagnostics across phases:
    • tokens/bytes/fragments monotonic checks and overall verdict.

2) CLI integration for meaningful summaries

Updated src/cleveragents/cli/commands/project_context.py:

  • context_inspect now includes phase_analysis in structured output.
  • context_simulate now includes phase_analysis in structured output.
  • Rich output now renders:
    • per-phase analysis table (fragments/resources/bytes/tokens/budget)
    • narrowing diagnostic panel (strategize >= execute >= apply).

3) Robot acceptance upgrade

Updated robot/helper_m5_e2e_verification.py acms_scoped_context():

  • Added concrete sample fragments and validates analyze_phase_summaries(...) output.
  • Asserts summary presence + resource/size metrics for all phases.
  • Asserts narrowing diagnostics overall=True for the configured policy.

4) Behave coverage for required edge cases

Added:

  • features/project_context_phase_analysis.feature
  • features/steps/project_context_phase_analysis_steps.py

Scenarios cover:

  • empty context
  • single resource
  • multiple resources
  • budget-constrained narrowing

5) Changelog

Updated CHANGELOG.md Unreleased section with #849 user-facing entry.

Validation evidence

Targeted affected-file runs

  • build/test-logs/issue-849-behave-phase-analysis.log
  • build/test-logs/issue-849-behave-project-context-cli.log
  • build/test-logs/issue-849-robot-scoped-context.log

Required nox sessions run

  • build/test-logs/issue-849-nox-lint.log
  • build/test-logs/issue-849-nox-typecheck.log
  • build/test-logs/issue-849-nox-integration_tests.log
  • build/test-logs/issue-849-nox-coverage_report.log (coverage XML indicates 97.64%)
  • build/test-logs/issue-849-nox-unit_tests.log
    • Full suite reports failures not isolated to #849 changes (cross-suite failures in unrelated areas).
  • build/test-logs/issue-849-nox-e2e_tests.log
    • Fails in unrelated M1/M2 E2E tests while M5 E2E section passes.

I left the final "Run nox (all default sessions), fix any errors" subtask unchecked in the issue body because this environment run still reports unrelated suite failures even after re-run.

Implementation update for #849 (branch `feature/m5-context-analysis`): ## What was implemented ### 1) New phase-analysis service Added `src/cleveragents/application/services/context_phase_analysis.py` with `analyze_phase_summaries(...)`. Behavior: - Builds per-phase summaries for `strategize`, `execute`, `apply` from: - `ProjectContextPolicy` (resolved view rules) - scoped tier fragments (`TieredFragment`) - budget token limit - Applies include/exclude resource filtering, include/exclude path filtering, max-file-size checks, and max-total-size checks. - Computes per-phase metrics: - `fragment_count`, `resource_count`, `total_bytes`, `total_tokens`, `budget_used_ratio` - exclusion counters by reason - human-readable `summary` text - Computes narrowing diagnostics across phases: - tokens/bytes/fragments monotonic checks and `overall` verdict. ### 2) CLI integration for meaningful summaries Updated `src/cleveragents/cli/commands/project_context.py`: - `context_inspect` now includes `phase_analysis` in structured output. - `context_simulate` now includes `phase_analysis` in structured output. - Rich output now renders: - per-phase analysis table (fragments/resources/bytes/tokens/budget) - narrowing diagnostic panel (`strategize >= execute >= apply`). ### 3) Robot acceptance upgrade Updated `robot/helper_m5_e2e_verification.py` `acms_scoped_context()`: - Added concrete sample fragments and validates `analyze_phase_summaries(...)` output. - Asserts summary presence + resource/size metrics for all phases. - Asserts narrowing diagnostics `overall=True` for the configured policy. ### 4) Behave coverage for required edge cases Added: - `features/project_context_phase_analysis.feature` - `features/steps/project_context_phase_analysis_steps.py` Scenarios cover: - empty context - single resource - multiple resources - budget-constrained narrowing ### 5) Changelog Updated `CHANGELOG.md` Unreleased section with #849 user-facing entry. ## Validation evidence ### Targeted affected-file runs - `build/test-logs/issue-849-behave-phase-analysis.log` ✅ - `build/test-logs/issue-849-behave-project-context-cli.log` ✅ - `build/test-logs/issue-849-robot-scoped-context.log` ✅ ### Required nox sessions run - `build/test-logs/issue-849-nox-lint.log` ✅ - `build/test-logs/issue-849-nox-typecheck.log` ✅ - `build/test-logs/issue-849-nox-integration_tests.log` ✅ - `build/test-logs/issue-849-nox-coverage_report.log` ✅ (coverage XML indicates 97.64%) - `build/test-logs/issue-849-nox-unit_tests.log` ❌ - Full suite reports failures not isolated to #849 changes (cross-suite failures in unrelated areas). - `build/test-logs/issue-849-nox-e2e_tests.log` ❌ - Fails in unrelated M1/M2 E2E tests while M5 E2E section passes. I left the final "Run nox (all default sessions), fix any errors" subtask unchecked in the issue body because this environment run still reports unrelated suite failures even after re-run.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleveragents/cleveragents-core#849
No description provided.