aditya
137d040c4d
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 18s
CI / helm (pull_request) Successful in 22s
CI / quality (pull_request) Successful in 56s
CI / lint (pull_request) Successful in 3m21s
CI / typecheck (pull_request) Successful in 4m0s
CI / security (pull_request) Successful in 4m19s
CI / unit_tests (pull_request) Successful in 9m32s
CI / docker (pull_request) Successful in 1m22s
CI / coverage (pull_request) Successful in 12m27s
CI / e2e_tests (pull_request) Successful in 20m3s
CI / integration_tests (pull_request) Successful in 21m20s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-regression (pull_request) Successful in 54m56s
Add a production skeleton compressor that re-renders inherited fragments to overview depths via the UKO detail-level map chain, fits the result within the configured skeleton budget, and wires the pipeline default to the new compressor. Address prior review feedback by extracting the render visitors into a dedicated module, restoring projected metadata to native runtime types, constraining builtin component resolution with an allowlist, and keeping child-context inheritance compatible with CRP context fragments for the Robot integration path. Reproduced the Forgejo lint job in a clean python:3.13-slim container with the CI commands All checks passed! and 1740 files already formatted; both passed, so the earlier lint failure appears to have been transient runner behavior rather than a source-level defect. ISSUES CLOSED: #919
274 lines
14 KiB
Gherkin
274 lines
14 KiB
Gherkin
@phase2 @acms @acms_pipeline_orchestrator
|
|
Feature: ACMS Pipeline Orchestrator and Phase 1 Components
|
|
As a CleverAgents developer
|
|
I want production-quality Phase 1 pipeline components
|
|
So that context strategies are selected, budgeted, and executed
|
|
with confidence weighting, proportional allocation, parallel
|
|
execution, and circuit-breaker fault tolerance
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ConfidenceWeightedSelector
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@orchestrator @selector
|
|
Scenario: ConfidenceWeightedSelector selects strategies by confidence
|
|
Given the pipeline orchestrator modules are available
|
|
When I use ConfidenceWeightedSelector to select from 3 strategies
|
|
Then all strategies with positive confidence should be returned
|
|
And the results should be sorted by confidence descending
|
|
|
|
@orchestrator @selector
|
|
Scenario: ConfidenceWeightedSelector boosts preferred strategies
|
|
Given the pipeline orchestrator modules are available
|
|
When I select with preferred_strategies containing "recency"
|
|
Then the "recency" strategy should have a boosted confidence
|
|
And the boosted confidence should not exceed 1.0
|
|
|
|
@orchestrator @selector
|
|
Scenario: ConfidenceWeightedSelector excludes zero-confidence strategies
|
|
Given the pipeline orchestrator modules are available
|
|
And a test strategy with zero confidence
|
|
When I use ConfidenceWeightedSelector with the zero-confidence strategy
|
|
Then the zero-confidence strategy should not be in the results
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ProportionalBudgetAllocator
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@orchestrator @allocator
|
|
Scenario: ProportionalBudgetAllocator distributes proportionally
|
|
Given the pipeline orchestrator modules are available
|
|
When I use ProportionalBudgetAllocator for candidates with confidences 0.8 and 0.2 and budget 1000
|
|
Then the first candidate should receive approximately 800 tokens from proportional allocator
|
|
And the second candidate should receive approximately 200 tokens from proportional allocator
|
|
And the total proportional allocation should equal exactly 1000
|
|
|
|
@orchestrator @allocator
|
|
Scenario: ProportionalBudgetAllocator enforces min_useful_budget
|
|
Given the pipeline orchestrator modules are available
|
|
When I use ProportionalBudgetAllocator with min_useful_budget 200 for candidates with confidences 0.99 and 0.01 and budget 300
|
|
Then only the high-confidence candidate should receive tokens
|
|
And the total proportional allocation should equal exactly 300
|
|
|
|
@orchestrator @allocator
|
|
Scenario: ProportionalBudgetAllocator handles single candidate
|
|
Given the pipeline orchestrator modules are available
|
|
When I use ProportionalBudgetAllocator for a single candidate with budget 500
|
|
Then the single candidate should receive all 500 tokens
|
|
|
|
@orchestrator @allocator
|
|
Scenario: ProportionalBudgetAllocator handles zero confidence with equal split
|
|
Given the pipeline orchestrator modules are available
|
|
When I use ProportionalBudgetAllocator for 3 zero-confidence candidates with budget 999
|
|
Then the total proportional allocation should equal exactly 999
|
|
And each proportional allocation should be 333 or 334
|
|
|
|
@orchestrator @allocator
|
|
Scenario: ProportionalBudgetAllocator falls back when all excluded
|
|
Given the pipeline orchestrator modules are available
|
|
When I use ProportionalBudgetAllocator with min_useful_budget 1000 for 2 candidates with budget 500
|
|
Then the highest-confidence candidate should receive the full budget
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# CircuitBreaker
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@orchestrator @circuit_breaker
|
|
Scenario: CircuitBreaker opens after threshold failures
|
|
Given the pipeline orchestrator modules are available
|
|
And a circuit breaker with threshold 3
|
|
When I record 3 consecutive failures for strategy "flaky"
|
|
Then the circuit for "flaky" should be open
|
|
|
|
@orchestrator @circuit_breaker
|
|
Scenario: CircuitBreaker resets on success
|
|
Given the pipeline orchestrator modules are available
|
|
And a circuit breaker with threshold 3
|
|
When I record 2 failures then 1 success for strategy "recovering"
|
|
Then the circuit for "recovering" should be closed
|
|
|
|
@orchestrator @circuit_breaker
|
|
Scenario: CircuitBreaker can be explicitly reset
|
|
Given the pipeline orchestrator modules are available
|
|
And a circuit breaker with threshold 2
|
|
When I record 2 failures for strategy "resettable"
|
|
And I reset the circuit for "resettable"
|
|
Then the circuit for "resettable" should be closed
|
|
|
|
@orchestrator @circuit_breaker
|
|
Scenario: CircuitBreaker reset_all clears all circuits
|
|
Given the pipeline orchestrator modules are available
|
|
And a circuit breaker with threshold 1
|
|
When I record 1 failure for strategy "a" and 1 failure for strategy "b"
|
|
And I reset all circuits
|
|
Then the circuit for "a" should be closed
|
|
And the circuit for "b" should be closed
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ParallelStrategyExecutor
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@orchestrator @executor
|
|
Scenario: ParallelStrategyExecutor invokes strategy and returns fragments
|
|
Given the pipeline orchestrator modules are available
|
|
And a tracking test strategy
|
|
And context fragments for executor testing
|
|
When I execute the strategy via ParallelStrategyExecutor
|
|
Then the executor should return fragments from the strategy
|
|
And the tracking strategy should have been invoked
|
|
|
|
@orchestrator @pipeline
|
|
Scenario: ContextAssemblyPipeline uses DepthReductionCompressor by default
|
|
Given the pipeline orchestrator modules are available
|
|
And a ContextAssemblyPipeline with default components
|
|
Then the orchestrator pipeline should use DepthReductionCompressor
|
|
|
|
@orchestrator @executor
|
|
Scenario: ParallelStrategyExecutor skips circuit-broken strategies
|
|
Given the pipeline orchestrator modules are available
|
|
And a ParallelStrategyExecutor with a pre-broken circuit for "broken-strat"
|
|
And context fragments for executor testing
|
|
When I execute with the circuit-broken strategy
|
|
Then the executor should return 0 fragments
|
|
|
|
@orchestrator @executor
|
|
Scenario: ParallelStrategyExecutor handles strategy failures gracefully
|
|
Given the pipeline orchestrator modules are available
|
|
And a strategy that always raises an exception
|
|
And context fragments for executor testing
|
|
When I execute the failing strategy via ParallelStrategyExecutor
|
|
Then the executor should return 0 fragments
|
|
And the circuit breaker should have recorded a failure for "failing"
|
|
|
|
@orchestrator @executor
|
|
Scenario: ParallelStrategyExecutor skips zero-budget allocations
|
|
Given the pipeline orchestrator modules are available
|
|
And a tracking test strategy
|
|
And context fragments for executor testing
|
|
When I execute with zero-budget allocation
|
|
Then the executor should return 0 fragments
|
|
|
|
@orchestrator @executor
|
|
Scenario: ParallelStrategyExecutor executes multiple strategies in parallel
|
|
Given the pipeline orchestrator modules are available
|
|
And context fragments for executor testing
|
|
When I execute 2 tracking strategies in parallel via ParallelStrategyExecutor
|
|
Then the executor should return fragments from all parallel strategies
|
|
And both parallel strategies should have been invoked
|
|
|
|
@orchestrator @executor
|
|
Scenario: ParallelStrategyExecutor handles mixed success and failure in parallel
|
|
Given the pipeline orchestrator modules are available
|
|
And context fragments for executor testing
|
|
When I execute a tracking strategy and a failing strategy in parallel
|
|
Then the executor should return fragments only from the successful parallel strategy
|
|
And the parallel circuit breaker should record a failure for "failing"
|
|
|
|
@orchestrator @executor
|
|
Scenario: ParallelStrategyExecutor returns empty for empty allocations
|
|
Given the pipeline orchestrator modules are available
|
|
And context fragments for executor testing
|
|
When I execute with empty allocations via ParallelStrategyExecutor
|
|
Then the executor should return 0 fragments
|
|
|
|
@orchestrator @executor
|
|
Scenario: ParallelStrategyExecutor circuit_breaker property is accessible
|
|
Given the pipeline orchestrator modules are available
|
|
When I create a ParallelStrategyExecutor with a custom circuit breaker
|
|
Then the circuit_breaker property should return the custom breaker
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ProportionalBudgetAllocator — Additional edge cases
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@orchestrator @allocator
|
|
Scenario: ProportionalBudgetAllocator handles empty candidate list
|
|
Given the pipeline orchestrator modules are available
|
|
When I use ProportionalBudgetAllocator for empty candidates with budget 1000
|
|
Then the proportional allocator should return an empty list
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ContextAssemblyPipeline — Additional edge cases
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@orchestrator @pipeline
|
|
Scenario: ContextAssemblyPipeline rejects unknown strategy name
|
|
Given a ContextAssemblyPipeline with default components
|
|
And the following orchestrator test fragments:
|
|
| uko_node | content | score | tokens |
|
|
| project://app/a.py | alpha | 0.9 | 100 |
|
|
And a pipeline budget with max_tokens 500 and reserved_tokens 0
|
|
When I assemble via the orchestrator with strategy "nonexistent_strategy"
|
|
Then an orchestrator ValueError should be raised mentioning "Unknown strategy"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ContextAssemblyPipeline — Full integration
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@orchestrator @pipeline
|
|
Scenario: ContextAssemblyPipeline assembles with timing metadata
|
|
Given a ContextAssemblyPipeline with default components
|
|
And the following orchestrator test fragments:
|
|
| uko_node | content | score | tokens |
|
|
| project://app/a.py | alpha | 0.9 | 100 |
|
|
| project://app/b.py | beta | 0.3 | 100 |
|
|
And a pipeline budget with max_tokens 250 and reserved_tokens 0
|
|
When I assemble via the orchestrator with strategy "relevance"
|
|
Then the orchestrator payload should contain 2 fragments
|
|
And the orchestrator should have timing metadata
|
|
And all timing values should be non-negative
|
|
|
|
@orchestrator @pipeline
|
|
Scenario: ContextAssemblyPipeline inherits ACMSPipeline behavior
|
|
Given a ContextAssemblyPipeline with default components
|
|
And the following orchestrator test fragments:
|
|
| uko_node | content | score | tokens |
|
|
| project://app/a.py | alpha | 0.9 | 50 |
|
|
And a pipeline budget with max_tokens 500 and reserved_tokens 0
|
|
When I assemble via the orchestrator with strategy "relevance"
|
|
Then the orchestrator payload strategies used should include "relevance"
|
|
And the orchestrator payload should be within budget
|
|
|
|
@orchestrator @pipeline
|
|
Scenario: ContextAssemblyPipeline rejects invalid plan_id
|
|
Given a ContextAssemblyPipeline with default components
|
|
And the following orchestrator test fragments:
|
|
| uko_node | content | score | tokens |
|
|
| project://app/a.py | alpha | 0.9 | 100 |
|
|
And a pipeline budget with max_tokens 500 and reserved_tokens 0
|
|
When I assemble via the orchestrator with invalid plan_id "not-valid"
|
|
Then an orchestrator ValueError should be raised mentioning "ULID"
|
|
|
|
@orchestrator @pipeline
|
|
Scenario: ContextAssemblyPipeline accepts custom Phase 1 components
|
|
Given a ContextAssemblyPipeline with a custom tracking selector
|
|
And the following orchestrator test fragments:
|
|
| uko_node | content | score | tokens |
|
|
| project://app/a.py | alpha | 0.9 | 100 |
|
|
And a pipeline budget with max_tokens 500 and reserved_tokens 0
|
|
When I assemble via the orchestrator with strategy "relevance"
|
|
Then the custom orchestrator selector should have been called
|
|
|
|
@orchestrator @pipeline
|
|
Scenario: ContextAssemblyPipeline can register custom strategies
|
|
Given a ContextAssemblyPipeline with default components
|
|
And a custom orchestrator strategy that returns fragments in reverse
|
|
And the following orchestrator test fragments:
|
|
| uko_node | content | score | tokens |
|
|
| project://app/a.py | first | 0.5 | 50 |
|
|
| project://app/b.py | second | 0.9 | 50 |
|
|
And a pipeline budget with max_tokens 200 and reserved_tokens 0
|
|
When I assemble via the orchestrator with strategy "reverse_orch"
|
|
Then the first orchestrator fragment content should be "second"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# StageTimings data model
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@orchestrator @timings
|
|
Scenario: StageTimings is a frozen dataclass with named fields
|
|
Given the pipeline orchestrator modules are available
|
|
When I create a StageTimings with total_ms 42.5
|
|
Then the timings total_ms should be 42.5
|
|
And the timings should have all 10 named fields
|