HAL9000
3b8ff5c566
Implemented core ACMS pipeline components: ActorPhaseStrategySelector, SpecBudgetAllocator, RelevanceRecencyPriorityScorer, ConstrainedKnapsackPacker, and PriorityCoherenceOrderer. These components implement the Protocol interfaces from acms_service.py and coordinate strategy selection, budget allocation, fragment scoring, content packing, and fragment ordering to optimize LLM usage. The StrategySelector selects context strategies based on actor type and plan phase with confidence boosts; the BudgetAllocator computes per-strategy budgets using the spec formula (confidence * quality_score proportional allocation); the FragmentScorer scores fragments by a weighted composite of relevance, recency, and priority; the Packer performs greedy knapsack packing respecting max_file_size and max_total_size. The Orderer groups related content to maximize coherence and overall throughput. The work also includes a 44-scenario BDD feature file covering all components and edge cases. All quality gates pass: lint, typecheck, unit tests. ISSUES CLOSED: #10015
365 lines
17 KiB
Gherkin
365 lines
17 KiB
Gherkin
@acms @acms_core_pipeline
|
|
Feature: ACMS Core Pipeline Components
|
|
As a CleverAgents developer
|
|
I want core ACMS pipeline components implemented
|
|
So that the pipeline can select strategies, allocate budgets, score fragments,
|
|
pack within constraints, and order for optimal LLM consumption
|
|
|
|
# ===========================================================================
|
|
# ActorPhaseStrategySelector
|
|
# ===========================================================================
|
|
|
|
@strategy_selector
|
|
Scenario: Select strategies with no actor type or plan phase
|
|
Given a set of core pipeline strategies
|
|
When I select strategies with no actor_type or plan_phase
|
|
Then all core strategies with positive confidence should be returned
|
|
And the core strategies should be sorted by confidence descending
|
|
|
|
@strategy_selector
|
|
Scenario: Select strategies boosts preferred strategies for planner actor
|
|
Given a set of core pipeline strategies
|
|
When I select strategies with actor_type "planner" only
|
|
Then the "relevance" strategy should have boosted confidence
|
|
|
|
@strategy_selector
|
|
Scenario: Select strategies boosts preferred strategies for executor actor
|
|
Given a set of core pipeline strategies
|
|
When I select strategies with actor_type "executor" only
|
|
Then the "tiered" strategy should have boosted confidence
|
|
|
|
@strategy_selector
|
|
Scenario: Select strategies boosts preferred strategies for strategize phase
|
|
Given a set of core pipeline strategies
|
|
When I select strategies with plan_phase "strategize" only
|
|
Then the "relevance" strategy should have boosted confidence
|
|
|
|
@strategy_selector
|
|
Scenario: Select strategies applies both actor and phase boosts
|
|
Given a set of core pipeline strategies
|
|
When I select strategies with actor_type "planner" and plan_phase "strategize"
|
|
Then the "relevance" strategy should have maximum boosted confidence
|
|
|
|
@strategy_selector
|
|
Scenario: Select strategies excludes zero-confidence strategies
|
|
Given a set of core pipeline strategies including a zero-confidence strategy
|
|
When I select strategies with no actor_type or plan_phase
|
|
Then the core zero-confidence strategy should not be in the results
|
|
|
|
@strategy_selector
|
|
Scenario: Select strategies with empty strategy list returns empty
|
|
Given an empty strategy list
|
|
When I select strategies with no actor_type or plan_phase
|
|
Then 0 strategies should be selected
|
|
|
|
@strategy_selector
|
|
Scenario: Confidence is clamped to 1.0 after boost
|
|
Given a strategy with confidence 0.9
|
|
When I select strategies with actor_type "planner" and plan_phase "strategize"
|
|
Then the strategy confidence should not exceed 1.0
|
|
|
|
# ===========================================================================
|
|
# SpecBudgetAllocator
|
|
# ===========================================================================
|
|
|
|
@budget_allocator
|
|
Scenario: Allocate budget to empty candidates returns empty
|
|
Given an empty candidate list
|
|
When I allocate budget 1000 with SpecBudgetAllocator
|
|
Then 0 allocations should be returned
|
|
|
|
@budget_allocator
|
|
Scenario: Allocate budget to single candidate gives full budget
|
|
Given a single candidate with confidence 0.8
|
|
When I allocate budget 1000 with SpecBudgetAllocator
|
|
Then 1 allocation should be returned
|
|
And the single allocation should receive 1000 tokens
|
|
|
|
@budget_allocator
|
|
Scenario: Allocate budget proportionally to two candidates
|
|
Given two candidates with equal confidence 0.5
|
|
When I allocate budget 1000 with SpecBudgetAllocator
|
|
Then 2 allocations should be returned
|
|
And the total allocated tokens should equal 1000
|
|
|
|
@budget_allocator
|
|
Scenario: Allocate budget uses spec formula with quality scores
|
|
Given two candidates with different quality scores
|
|
When I allocate budget 1000 with SpecBudgetAllocator
|
|
Then the higher quality candidate should receive more tokens
|
|
|
|
@budget_allocator
|
|
Scenario: Allocate budget with zero total weight falls back to equal split
|
|
Given two candidates with zero confidence
|
|
When I allocate budget 1000 with SpecBudgetAllocator
|
|
Then 2 allocations should be returned
|
|
And the total allocated tokens should equal 1000
|
|
|
|
@budget_allocator
|
|
Scenario: Allocate budget with min_useful_budget excludes small candidates
|
|
Given three candidates where one would receive very few tokens
|
|
When I allocate budget 100 with SpecBudgetAllocator and min_useful_budget 30
|
|
Then the small candidate should be excluded from allocations
|
|
|
|
@budget_allocator
|
|
Scenario: Allocate budget total never exceeds budget
|
|
Given three candidates with varying confidence
|
|
When I allocate budget 500 with SpecBudgetAllocator
|
|
Then the total allocated tokens should equal 500
|
|
|
|
# ===========================================================================
|
|
# RelevanceRecencyPriorityScorer
|
|
# ===========================================================================
|
|
|
|
@scorer
|
|
Scenario: Score empty fragment list returns empty
|
|
Given an empty core pipeline fragment list
|
|
When I score the fragments with RelevanceRecencyPriorityScorer
|
|
Then 0 core scored fragments should be returned
|
|
|
|
@scorer
|
|
Scenario: Score single fragment preserves metadata
|
|
Given a core pipeline fragment with relevance 0.8
|
|
When I score the fragments with RelevanceRecencyPriorityScorer
|
|
Then the scored fragment should have _original_relevance metadata "0.8"
|
|
|
|
@scorer
|
|
Scenario: Score produces composite from relevance recency and priority
|
|
Given a core pipeline fragment with relevance 0.8 and priority 0.9
|
|
When I score the fragments with RelevanceRecencyPriorityScorer
|
|
Then the scored fragment composite score should be between 0.0 and 1.0
|
|
|
|
@scorer
|
|
Scenario: Score is deterministic for identical inputs
|
|
Given two identical core pipeline fragments
|
|
When I score both fragment sets with RelevanceRecencyPriorityScorer
|
|
Then both scored fragments should have identical composite scores
|
|
|
|
@scorer
|
|
Scenario: Score recency normalises across fragment timestamps
|
|
Given two core pipeline fragments with different timestamps
|
|
When I score the fragments with RelevanceRecencyPriorityScorer
|
|
Then the newer fragment should have higher recency score
|
|
|
|
@scorer
|
|
Scenario: Score with same timestamps gives recency 1.0
|
|
Given two core pipeline fragments with identical timestamps
|
|
When I score the fragments with RelevanceRecencyPriorityScorer
|
|
Then all scored fragments should have recency score 1.0
|
|
|
|
@scorer
|
|
Scenario: Score composite is clamped to 1.0
|
|
Given a core pipeline fragment with maximum relevance and priority
|
|
When I score the fragments with RelevanceRecencyPriorityScorer
|
|
Then the scored fragment composite score should be at most 1.0
|
|
|
|
@scorer
|
|
Scenario: Score composite is clamped to 0.0
|
|
Given a core pipeline fragment with zero relevance and zero priority
|
|
When I score the fragments with RelevanceRecencyPriorityScorer
|
|
Then the scored fragment composite score should be at least 0.0
|
|
|
|
@scorer
|
|
Scenario: Score with custom weights uses configured weights
|
|
Given a core pipeline fragment with relevance 1.0 and priority 0.0
|
|
And scorer configured with relevance_weight 1.0 recency_weight 0.0 priority_weight 0.0
|
|
When I score the fragments with custom RelevanceRecencyPriorityScorer
|
|
Then the scored fragment composite score should be 1.0
|
|
|
|
@scorer
|
|
Scenario: Score extracts priority from metadata
|
|
Given a core pipeline fragment with metadata priority "0.9"
|
|
When I score the fragments with RelevanceRecencyPriorityScorer
|
|
Then the scored fragment should have _score_priority metadata near "0.9"
|
|
|
|
@scorer
|
|
Scenario: Score defaults priority to 0.5 when not in metadata
|
|
Given a core pipeline fragment with no priority metadata
|
|
When I score the fragments with RelevanceRecencyPriorityScorer
|
|
Then the scored fragment should have _score_priority metadata "0.5"
|
|
|
|
# ===========================================================================
|
|
# ConstrainedKnapsackPacker
|
|
# ===========================================================================
|
|
|
|
@packer
|
|
Scenario: Pack empty fragment list returns empty
|
|
Given an empty core pipeline fragment list
|
|
And a core pipeline budget with max_tokens 1000 and reserved_tokens 0
|
|
When I pack the fragments with ConstrainedKnapsackPacker
|
|
Then 0 core packed fragments should be returned
|
|
|
|
@packer
|
|
Scenario: Pack fragments within token budget
|
|
Given the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/main.py | alpha | 0.9 | 100 | 3 |
|
|
| project://app/io.py | beta | 0.7 | 100 | 3 |
|
|
| project://app/util.py | gamma | 0.5 | 100 | 3 |
|
|
And a core pipeline budget with max_tokens 250 and reserved_tokens 0
|
|
When I pack the fragments with ConstrainedKnapsackPacker
|
|
Then 2 core packed fragments should be returned
|
|
And the core packed total tokens should be at most 250
|
|
|
|
@packer
|
|
Scenario: Pack respects max_file_size constraint
|
|
Given the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/main.py | short content | 0.9 | 10 | 3 |
|
|
| project://app/io.py | this is a much longer text | 0.7 | 20 | 3 |
|
|
And a core pipeline budget with max_tokens 1000 and reserved_tokens 0
|
|
And a ConstrainedKnapsackPacker with max_file_size 15
|
|
When I pack the fragments with ConstrainedKnapsackPacker
|
|
Then 1 core packed fragment should be returned
|
|
And the core packed fragment uko_node should be "project://app/main.py"
|
|
|
|
@packer
|
|
Scenario: Pack respects max_total_size constraint
|
|
Given the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/main.py | hello | 0.9 | 10 | 3 |
|
|
| project://app/io.py | world | 0.7 | 10 | 3 |
|
|
| project://app/util.py | extra | 0.5 | 10 | 3 |
|
|
And a core pipeline budget with max_tokens 1000 and reserved_tokens 0
|
|
And a ConstrainedKnapsackPacker with max_total_size 4
|
|
When I pack the fragments with ConstrainedKnapsackPacker
|
|
Then 0 core packed fragments should be returned
|
|
|
|
@packer
|
|
Scenario: Pack prefers higher scored fragments
|
|
Given the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/main.py | alpha | 0.3 | 100 | 3 |
|
|
| project://app/io.py | beta | 0.9 | 100 | 3 |
|
|
And a core pipeline budget with max_tokens 150 and reserved_tokens 0
|
|
When I pack the fragments with ConstrainedKnapsackPacker
|
|
Then 1 core packed fragment should be returned
|
|
And the core packed fragment uko_node should be "project://app/io.py"
|
|
|
|
@packer
|
|
Scenario: Pack with context_view uses view constraints
|
|
Given the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/main.py | hello | 0.9 | 10 | 3 |
|
|
| project://app/io.py | world | 0.7 | 10 | 3 |
|
|
And a core pipeline budget with max_tokens 1000 and reserved_tokens 0
|
|
And a ConstrainedKnapsackPacker with context_view max_total_size 4
|
|
When I pack the fragments with ConstrainedKnapsackPacker
|
|
Then 0 core packed fragments should be returned
|
|
|
|
@packer
|
|
Scenario: Pack with budget=0 returns empty
|
|
Given the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/main.py | hello | 0.9 | 10 | 3 |
|
|
And a core pipeline budget with max_tokens 1 and reserved_tokens 0
|
|
When I pack the fragments with ConstrainedKnapsackPacker
|
|
Then 0 core packed fragments should be returned
|
|
|
|
# ===========================================================================
|
|
# PriorityCoherenceOrderer
|
|
# ===========================================================================
|
|
|
|
@orderer
|
|
Scenario: Order empty fragment list returns empty
|
|
Given an empty core pipeline fragment list
|
|
When I order the fragments with PriorityCoherenceOrderer
|
|
Then 0 core ordered fragments should be returned
|
|
|
|
@orderer
|
|
Scenario: Order single fragment returns it unchanged
|
|
Given a core pipeline fragment with relevance 0.8
|
|
When I order the fragments with PriorityCoherenceOrderer
|
|
Then 1 core ordered fragment should be returned
|
|
|
|
@orderer
|
|
Scenario: Order preserves all fragments
|
|
Given the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/src/main.py | alpha | 0.9 | 10 | 3 |
|
|
| project://app/src/io.py | beta | 0.7 | 10 | 3 |
|
|
| project://lib/util/util.py | gamma | 0.5 | 10 | 3 |
|
|
When I order the fragments with PriorityCoherenceOrderer
|
|
Then 3 core ordered fragments should be returned
|
|
|
|
@orderer
|
|
Scenario: Order places high-priority fragments first
|
|
Given the following core pipeline fragments with priorities:
|
|
| uko_node | content | score | tokens | depth | priority |
|
|
| project://app/src/main.py | alpha | 0.5 | 10 | 3 | 0.9 |
|
|
| project://app/src/io.py | beta | 0.9 | 10 | 3 | 0.1 |
|
|
When I order the fragments with PriorityCoherenceOrderer
|
|
Then the first core ordered fragment should have uko_node "project://app/src/main.py"
|
|
|
|
@orderer
|
|
Scenario: Order groups related fragments together
|
|
Given the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/src/main.py | alpha | 0.9 | 10 | 3 |
|
|
| project://lib/util/util.py | beta | 0.8 | 10 | 3 |
|
|
| project://app/src/io.py | gamma | 0.7 | 10 | 3 |
|
|
When I order the fragments with PriorityCoherenceOrderer
|
|
Then fragments from "project://app" should be adjacent
|
|
|
|
@orderer
|
|
Scenario: Order with default priority 0.5 falls back to relevance
|
|
Given the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/src/main.py | alpha | 0.3 | 10 | 3 |
|
|
| project://lib/util/util.py | beta | 0.9 | 10 | 3 |
|
|
When I order the fragments with PriorityCoherenceOrderer
|
|
Then the first core ordered fragment should have uko_node "project://lib/util/util.py"
|
|
|
|
# ===========================================================================
|
|
# Pipeline Integration
|
|
# ===========================================================================
|
|
|
|
@pipeline_integration
|
|
Scenario: Inject ActorPhaseStrategySelector into pipeline
|
|
Given the ACMS pipeline with an ActorPhaseStrategySelector
|
|
And the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/main.py | hello | 0.8 | 10 | 3 |
|
|
When I assemble context through the core pipeline
|
|
Then the core pipeline output should contain 1 fragment
|
|
|
|
@pipeline_integration
|
|
Scenario: Inject SpecBudgetAllocator into pipeline
|
|
Given the ACMS pipeline with a SpecBudgetAllocator
|
|
And the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/main.py | hello | 0.8 | 10 | 3 |
|
|
When I assemble context through the core pipeline
|
|
Then the core pipeline output should contain 1 fragment
|
|
|
|
@pipeline_integration
|
|
Scenario: Inject RelevanceRecencyPriorityScorer into pipeline
|
|
Given the ACMS pipeline with a RelevanceRecencyPriorityScorer
|
|
And the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/main.py | hello | 0.8 | 10 | 3 |
|
|
When I assemble context through the core pipeline
|
|
Then the core pipeline output fragments should have updated relevance scores
|
|
|
|
@pipeline_integration
|
|
Scenario: Inject ConstrainedKnapsackPacker into pipeline
|
|
Given the ACMS pipeline with a ConstrainedKnapsackPacker
|
|
And the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/main.py | alpha | 0.9 | 100 | 3 |
|
|
| project://app/io.py | beta | 0.7 | 100 | 3 |
|
|
| project://app/util.py | gamma | 0.5 | 100 | 3 |
|
|
And a core pipeline budget with max_tokens 250 and reserved_tokens 0
|
|
When I assemble context through the core pipeline with budget
|
|
Then the core pipeline output total tokens should be at most 250
|
|
|
|
@pipeline_integration
|
|
Scenario: Inject PriorityCoherenceOrderer into pipeline
|
|
Given the ACMS pipeline with a PriorityCoherenceOrderer
|
|
And the following core pipeline fragments:
|
|
| uko_node | content | score | tokens | depth |
|
|
| project://app/main.py | hello | 0.8 | 10 | 3 |
|
|
| project://lib/util.py | world | 0.6 | 15 | 5 |
|
|
When I assemble context through the core pipeline
|
|
Then the core pipeline output should contain 2 fragments
|