aditya
ef8f9640fa
CI / lint (pull_request) Successful in 30s
CI / typecheck (pull_request) Successful in 1m7s
CI / quality (pull_request) Successful in 46s
CI / security (pull_request) Successful in 1m0s
CI / build (pull_request) Successful in 23s
CI / helm (pull_request) Successful in 41s
CI / integration_tests (pull_request) Successful in 4m11s
CI / unit_tests (pull_request) Successful in 4m38s
CI / docker (pull_request) Successful in 1m22s
CI / coverage (pull_request) Successful in 11m26s
CI / e2e_tests (pull_request) Successful in 18m21s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 58m2s
Add phase-wise context analysis summaries for strategize, execute, and apply, and integrate them into `agents project context inspect` and `agents project context simulate` with per-phase resource counts, size and token utilization, and narrowing diagnostics.\n\nAdd Behave and Robot coverage for empty, single-resource, multi-resource, and budget-constrained context analysis, and tighten the M5 smoke coverage after review feedback with exact summary assertions and a missing-path regression scenario for `agents context add`.\n\nISSUES CLOSED: #849
288 lines
13 KiB
Gherkin
288 lines
13 KiB
Gherkin
Feature: M5 ACMS pipeline and large-project context smoke tests
|
|
As a developer working with the CleverAgents M5 milestone
|
|
I want to verify the ACMS context pipeline end-to-end
|
|
So that context assembly, budget enforcement, and multi-project context work correctly
|
|
|
|
Background:
|
|
Given a m5 smoke test runner
|
|
And a m5 smoke mocked context service
|
|
|
|
# --- Fixture loading ---
|
|
|
|
Scenario: M5 smoke load ACMS context policy fixture
|
|
When I m5 smoke load the ACMS context policy fixture
|
|
Then the m5 smoke policy fixture should have a default view
|
|
And the m5 smoke policy fixture should have a strategize view
|
|
And the m5 smoke policy fixture should have an execute view
|
|
|
|
Scenario: M5 smoke load large project context fixture
|
|
When I m5 smoke load the large project context fixture
|
|
Then the m5 smoke large project fixture should have file entries
|
|
And the m5 smoke large project fixture should have context tiers
|
|
|
|
Scenario: M5 smoke load context analysis results fixture
|
|
When I m5 smoke load the context analysis results fixture
|
|
Then the m5 smoke analysis fixture should have a summary
|
|
And the m5 smoke analysis fixture should have dependencies
|
|
And the m5 smoke analysis fixture should have relevance scores
|
|
|
|
# --- Context policy resolution ---
|
|
|
|
Scenario: M5 smoke resolve default view from empty policy
|
|
Given a m5 smoke empty context policy
|
|
When I m5 smoke resolve the view for phase "default"
|
|
Then the m5 smoke resolved view should include all paths
|
|
|
|
Scenario: M5 smoke resolve strategize view inherits from default
|
|
Given a m5 smoke policy with only default view
|
|
When I m5 smoke resolve the view for phase "strategize"
|
|
Then the m5 smoke resolved view should match the default view
|
|
|
|
Scenario: M5 smoke resolve strategize view with override
|
|
Given a m5 smoke policy with strategize override
|
|
When I m5 smoke resolve the view for phase "strategize"
|
|
Then the m5 smoke resolved view should use the strategize override
|
|
|
|
Scenario: M5 smoke resolve execute view inherits from strategize
|
|
Given a m5 smoke policy with strategize override
|
|
When I m5 smoke resolve the view for phase "execute"
|
|
Then the m5 smoke resolved view should use the strategize override
|
|
|
|
Scenario: M5 smoke resolve apply view falls through to default
|
|
Given a m5 smoke policy with only default view
|
|
When I m5 smoke resolve the view for phase "apply"
|
|
Then the m5 smoke resolved view should match the default view
|
|
|
|
Scenario: M5 smoke invalid phase raises ValueError
|
|
Given a m5 smoke empty context policy
|
|
When I m5 smoke resolve the view for invalid phase "invalid"
|
|
Then a m5 smoke ValueError should be raised
|
|
|
|
# --- Budget enforcement ---
|
|
|
|
Scenario: M5 smoke max file size rejects oversized files
|
|
Given a m5 smoke context view with max_file_size 1024
|
|
Then a m5 smoke file of size 2048 should exceed the budget
|
|
And a m5 smoke file of size 512 should be within the budget
|
|
|
|
Scenario: M5 smoke max total size limits aggregate context
|
|
Given a m5 smoke context view with max_total_size 8192
|
|
Then a m5 smoke aggregate of size 10000 should exceed the budget
|
|
And a m5 smoke aggregate of size 4096 should be within the budget
|
|
|
|
Scenario: M5 smoke zero max file size is invalid
|
|
When I m5 smoke create a context view with max_file_size 0
|
|
Then a m5 smoke validation error should be raised
|
|
|
|
Scenario: M5 smoke negative max total size is invalid
|
|
When I m5 smoke create a context view with max_total_size -1
|
|
Then a m5 smoke validation error should be raised
|
|
|
|
Scenario: M5 smoke None size limits allow unlimited
|
|
Given a m5 smoke context view with no size limits
|
|
Then a m5 smoke file of size 999999999 should be within the budget
|
|
And a m5 smoke aggregate of size 999999999 should be within the budget
|
|
|
|
# --- Context assembly via CLI ---
|
|
|
|
Scenario: M5 smoke context list with empty project
|
|
Given a m5 smoke project with no context
|
|
When I m5 smoke invoke context list
|
|
Then the m5 smoke context list should succeed
|
|
And the m5 smoke context list output should be empty
|
|
|
|
Scenario: M5 smoke context add file to project
|
|
Given a m5 smoke project with mocked context service
|
|
When I m5 smoke invoke context add with path "src/main.py"
|
|
Then the m5 smoke context add should succeed
|
|
|
|
Scenario: M5 smoke context add missing path reports an error
|
|
Given a m5 smoke project with mocked context service
|
|
When I m5 smoke invoke context add with missing path "src/missing.py"
|
|
Then the m5 smoke context add output should report the missing path
|
|
And the m5 smoke context service should not add any files
|
|
|
|
Scenario: M5 smoke context show displays file content
|
|
Given a m5 smoke project with context entries
|
|
When I m5 smoke invoke context show for path "src/main.py"
|
|
Then the m5 smoke context show should succeed
|
|
And the m5 smoke context show output should contain content
|
|
|
|
Scenario: M5 smoke context clear removes all entries
|
|
Given a m5 smoke project with context entries
|
|
When I m5 smoke invoke context clear
|
|
Then the m5 smoke context clear should succeed
|
|
|
|
# --- Context CLI: multiple resources ---
|
|
|
|
Scenario: M5 smoke context list with multiple resources
|
|
Given a m5 smoke project with multiple context entries
|
|
When I m5 smoke invoke context list
|
|
Then the m5 smoke context list should succeed
|
|
And the m5 smoke context list output should contain multiple files
|
|
|
|
Scenario: M5 smoke context add multiple files to project
|
|
Given a m5 smoke project with mocked multi-add context service
|
|
When I m5 smoke invoke context add with paths "src/a.py" and "src/b.py"
|
|
Then the m5 smoke context add should succeed
|
|
|
|
Scenario: M5 smoke context show with multiple resources shows summary
|
|
Given a m5 smoke project with multiple context entries
|
|
When I m5 smoke invoke context show without path
|
|
Then the m5 smoke context show should succeed
|
|
And the m5 smoke context show output should contain an exact summary
|
|
|
|
# --- Context CLI: clear and re-add ---
|
|
|
|
Scenario: M5 smoke context clear then re-add restores entries
|
|
Given a m5 smoke project with context entries
|
|
When I m5 smoke invoke context clear
|
|
Then the m5 smoke context clear should succeed
|
|
When I m5 smoke re-add context entry "src/main.py" after clear
|
|
Then the m5 smoke context add should succeed
|
|
|
|
Scenario: M5 smoke context clear on empty project succeeds
|
|
Given a m5 smoke project with no context
|
|
When I m5 smoke invoke context clear
|
|
Then the m5 smoke context clear should succeed
|
|
|
|
# --- Project context policy CLI ---
|
|
|
|
Scenario: M5 smoke project context show displays policy
|
|
Given a m5 smoke project with a saved context policy
|
|
When I m5 smoke invoke project context show
|
|
Then the m5 smoke project context show should succeed
|
|
And the m5 smoke project context output should contain phase views
|
|
|
|
Scenario: M5 smoke project context inspect is wired to ACMS
|
|
When I m5 smoke invoke project context inspect
|
|
Then the m5 smoke project context inspect should succeed
|
|
|
|
Scenario: M5 smoke project context simulate is wired to ACMS
|
|
When I m5 smoke invoke project context simulate
|
|
Then the m5 smoke project context simulate should succeed
|
|
|
|
# --- Context analysis agent ---
|
|
|
|
Scenario: M5 smoke context analysis produces summary
|
|
Given a m5 smoke mocked context analysis agent
|
|
When I m5 smoke invoke context analysis
|
|
Then the m5 smoke analysis result should have a non-empty summary
|
|
|
|
Scenario: M5 smoke context analysis produces dependencies
|
|
Given a m5 smoke mocked context analysis agent
|
|
When I m5 smoke invoke context analysis
|
|
Then the m5 smoke analysis result should have dependency entries
|
|
|
|
Scenario: M5 smoke context analysis produces relevance scores
|
|
Given a m5 smoke mocked context analysis agent
|
|
When I m5 smoke invoke context analysis
|
|
Then the m5 smoke analysis result should have relevance scores
|
|
And all m5 smoke relevance scores should be between 0 and 1
|
|
|
|
# --- Multi-project context ---
|
|
|
|
Scenario: M5 smoke multiple projects have independent context
|
|
Given m5 smoke project "proj-a" with 3 context files
|
|
And m5 smoke project "proj-b" with 5 context files
|
|
Then m5 smoke project "proj-a" should have 3 context entries
|
|
And m5 smoke project "proj-b" should have 5 context entries
|
|
|
|
Scenario: M5 smoke context exclusion patterns filter correctly
|
|
Given a m5 smoke context view excluding "**/__pycache__/**"
|
|
Then the m5 smoke path "src/__pycache__/module.pyc" should be excluded
|
|
And the m5 smoke path "src/module.py" should not be excluded
|
|
|
|
# --- Budget enforcement (enforce_size_budget) ---
|
|
|
|
Scenario: M5 smoke enforce_size_budget excludes oversized fragments
|
|
Given m5 smoke fragments of sizes 50 200 80
|
|
And a m5 smoke context view with max_file_size 100
|
|
When I m5 smoke enforce the size budget
|
|
Then 2 m5 smoke fragments should be accepted
|
|
And 1 m5 smoke fragments should be violated
|
|
And the m5 smoke violation type should be "max_file_size"
|
|
And the m5 smoke total accepted size should be 130
|
|
|
|
Scenario: M5 smoke enforce_size_budget caps total size
|
|
Given m5 smoke fragments of sizes 50 60 70
|
|
And a m5 smoke context view with max_total_size 120
|
|
When I m5 smoke enforce the size budget
|
|
Then 2 m5 smoke fragments should be accepted
|
|
And 1 m5 smoke fragments should be violated
|
|
And the m5 smoke violation type should be "max_total_size"
|
|
And the m5 smoke total accepted size should be 110
|
|
|
|
Scenario: M5 smoke enforce_size_budget with mixed limits
|
|
Given m5 smoke fragments of sizes 30 200 40 50
|
|
And a m5 smoke context view with max_file_size 100 and max_total_size 100
|
|
When I m5 smoke enforce the size budget
|
|
Then 2 m5 smoke fragments should be accepted
|
|
And 2 m5 smoke fragments should be violated
|
|
And the m5 smoke violations should include type "max_file_size"
|
|
And the m5 smoke violations should include type "max_total_size"
|
|
|
|
Scenario: M5 smoke enforce_size_budget with no limits accepts all
|
|
Given m5 smoke fragments of sizes 500 600 700
|
|
And a m5 smoke context view with no size limits
|
|
When I m5 smoke enforce the size budget
|
|
Then 3 m5 smoke fragments should be accepted
|
|
And 0 m5 smoke fragments should be violated
|
|
And the m5 smoke total accepted size should be 1800
|
|
|
|
Scenario: M5 smoke enforce_size_budget boundary exact fit
|
|
Given m5 smoke fragments of sizes 50 50
|
|
And a m5 smoke context view with max_total_size 100
|
|
When I m5 smoke enforce the size budget
|
|
Then 2 m5 smoke fragments should be accepted
|
|
And 0 m5 smoke fragments should be violated
|
|
And the m5 smoke total accepted size should be 100
|
|
|
|
Scenario: M5 smoke enforce_size_budget reports violation details
|
|
Given m5 smoke fragments of sizes 200
|
|
And a m5 smoke context view with max_file_size 100
|
|
When I m5 smoke enforce the size budget
|
|
Then the m5 smoke violation reason should contain "exceeded max_file_size"
|
|
And the m5 smoke violation content size should be 200
|
|
|
|
Scenario: M5 smoke pipeline applies size budget with context view
|
|
Given m5 smoke fragments of sizes 50 200 80
|
|
And a m5 smoke context view with max_file_size 100
|
|
When I m5 smoke assemble via pipeline with the context view
|
|
Then the m5 smoke pipeline should return 2 fragments
|
|
And the m5 smoke pipeline enforcement result should have 1 violation(s)
|
|
|
|
# --- Budget enforcement edge cases ---
|
|
|
|
Scenario: M5 smoke enforce_size_budget with empty fragments list
|
|
Given m5 smoke fragments of sizes
|
|
And a m5 smoke context view with max_file_size 100
|
|
When I m5 smoke enforce the size budget
|
|
Then 0 m5 smoke fragments should be accepted
|
|
And 0 m5 smoke fragments should be violated
|
|
And the m5 smoke total accepted size should be 0
|
|
|
|
Scenario: M5 smoke enforce_size_budget all fragments exceed max_file_size
|
|
Given m5 smoke fragments of sizes 200 300 400
|
|
And a m5 smoke context view with max_file_size 100
|
|
When I m5 smoke enforce the size budget
|
|
Then 0 m5 smoke fragments should be accepted
|
|
And 3 m5 smoke fragments should be violated
|
|
And the m5 smoke total accepted size should be 0
|
|
|
|
Scenario: M5 smoke enforce_size_budget single fragment at exact max_file_size boundary
|
|
Given m5 smoke fragments of sizes 100
|
|
And a m5 smoke context view with max_file_size 100
|
|
When I m5 smoke enforce the size budget
|
|
Then 1 m5 smoke fragments should be accepted
|
|
And 0 m5 smoke fragments should be violated
|
|
And the m5 smoke total accepted size should be 100
|
|
|
|
Scenario: M5 smoke enforce_size_budget with multi-byte Unicode content
|
|
Given m5 smoke fragments with unicode content
|
|
And a m5 smoke context view with max_file_size 10
|
|
When I m5 smoke enforce the size budget
|
|
Then 1 m5 smoke fragments should be accepted
|
|
And 1 m5 smoke fragments should be violated
|
|
And the m5 smoke violation type should be "max_file_size"
|