HAL9000
ebb543a9c3
- Remove duplicate @then decorator on step_verify_peak_concurrency_limit (caused AmbiguousStep error crashing all 8 unit test feature files) - Rename "the subplans should have been executed in order" to "the subplans should have been executed in sequential order" to avoid conflict with pre-existing step in subplan_execution_steps.py - Remove 13 additional @then step definitions that duplicated steps in subplan_execution_steps.py; alias context.exec_result and context.validation_error in @when steps so pre-existing steps work - Replace two # type: ignore comments (lines 438, 453) with typed Any variables per zero-tolerance policy - Apply ruff format to fix formatting (long import wrapping, list comps) - Add CHANGELOG entry and CONTRIBUTORS entry for #9555 ISSUES CLOSED: #9609
320 lines
13 KiB
Gherkin
320 lines
13 KiB
Gherkin
@phase1 @scheduler @parallel
|
|
Feature: Parallel Subplan Execution Scheduler with max_parallel Concurrency Control
|
|
As a plan orchestrator
|
|
I want to execute subplans in parallel with a configurable max_parallel limit
|
|
And automatically queue additional subplans when the limit is reached
|
|
So that I can efficiently utilize resources while preventing unbounded concurrency
|
|
|
|
# --- Basic parallel execution ---
|
|
|
|
@basic
|
|
Scenario: Scheduler executes subplans in parallel up to max_parallel limit
|
|
Given a parallel subplan scheduler with max_parallel 3
|
|
And 3 subplans to execute
|
|
When the scheduler executes all subplans
|
|
Then all 3 subplans should complete successfully
|
|
And the execution result should report all succeeded
|
|
|
|
@basic
|
|
Scenario: Scheduler queues subplans when max_parallel limit is reached
|
|
Given a parallel subplan scheduler with max_parallel 2
|
|
And 5 subplans to execute
|
|
When the scheduler executes all subplans
|
|
Then all 5 subplans should complete successfully
|
|
And the peak concurrent execution should not exceed 2
|
|
|
|
@basic
|
|
Scenario: Scheduler with max_parallel 1 executes sequentially
|
|
Given a parallel subplan scheduler with max_parallel 1
|
|
And 3 subplans to execute
|
|
When the scheduler executes all subplans
|
|
Then all 3 subplans should complete successfully
|
|
And the subplans should have been executed in sequential order
|
|
|
|
# --- Concurrency control ---
|
|
|
|
@concurrency
|
|
Scenario: Scheduler respects max_parallel with 10 subplans and limit 5
|
|
Given a parallel subplan scheduler with max_parallel 5
|
|
And 10 subplans to execute with concurrency tracking
|
|
When the scheduler executes all subplans
|
|
Then all 10 subplans should complete successfully
|
|
And the peak concurrent execution should not exceed 5
|
|
|
|
@concurrency
|
|
Scenario: Scheduler respects max_parallel with 15 subplans and limit 3
|
|
Given a parallel subplan scheduler with max_parallel 3
|
|
And 15 subplans to execute with concurrency tracking
|
|
When the scheduler executes all subplans
|
|
Then all 15 subplans should complete successfully
|
|
And the peak concurrent execution should not exceed 3
|
|
|
|
@concurrency
|
|
Scenario: Scheduler respects max_parallel with 50 subplans and limit 10
|
|
Given a parallel subplan scheduler with max_parallel 10
|
|
And 50 subplans to execute with concurrency tracking
|
|
When the scheduler executes all subplans
|
|
Then all 50 subplans should complete successfully
|
|
And the peak concurrent execution should not exceed 10
|
|
|
|
# --- Queue management ---
|
|
|
|
@queue
|
|
Scenario: Scheduler queue status reflects pending, active, and completed
|
|
Given a parallel subplan scheduler with max_parallel 2
|
|
And 5 subplans to execute
|
|
When the scheduler starts execution
|
|
Then the queue should have 5 pending subplans
|
|
And the queue should have 0 active subplans
|
|
And the queue should have 0 completed subplans
|
|
|
|
@queue
|
|
Scenario: Scheduler queue updates as subplans complete
|
|
Given a parallel subplan scheduler with max_parallel 2
|
|
And 4 subplans to execute with staggered completion
|
|
When the scheduler executes all subplans
|
|
Then the queue should eventually have 0 pending subplans
|
|
And the queue should eventually have 0 active subplans
|
|
And the queue should eventually have 4 completed subplans
|
|
|
|
@queue
|
|
Scenario: Scheduler available slots decrease as subplans start
|
|
Given a parallel subplan scheduler with max_parallel 3
|
|
And 5 subplans to execute
|
|
When the scheduler starts execution
|
|
Then the available slots should be 3
|
|
And after 2 subplans start, the available slots should be 1
|
|
|
|
# --- Parent plan blocking ---
|
|
|
|
@blocking
|
|
Scenario: Parent plan blocks until all subplans complete
|
|
Given a parallel subplan scheduler with max_parallel 2
|
|
And 3 subplans to execute
|
|
When the scheduler executes all subplans
|
|
Then the scheduler should block until all subplans finish
|
|
And the execution result should contain all 3 subplan statuses
|
|
|
|
@blocking
|
|
Scenario: Parent plan blocks even with max_parallel 1
|
|
Given a parallel subplan scheduler with max_parallel 1
|
|
And 5 subplans to execute
|
|
When the scheduler executes all subplans
|
|
Then the scheduler should block until all subplans finish
|
|
And the execution result should contain all 5 subplan statuses
|
|
|
|
# --- Failure handling ---
|
|
|
|
@failure
|
|
Scenario: Scheduler handles subplan failure with fail_fast disabled
|
|
Given a parallel subplan scheduler with max_parallel 3 and fail_fast disabled
|
|
And 3 subplans where the second will fail
|
|
When the scheduler executes all subplans
|
|
Then the first subplan should complete successfully
|
|
And the second subplan should be errored
|
|
And the third subplan should complete successfully
|
|
|
|
@failure
|
|
Scenario: Scheduler stops other subplans with fail_fast enabled
|
|
Given a parallel subplan scheduler with max_parallel 3 and fail_fast enabled
|
|
And 3 subplans where the first will fail
|
|
When the scheduler executes all subplans
|
|
Then the first subplan should be errored
|
|
And the remaining subplans should be cancelled
|
|
|
|
@failure
|
|
Scenario: Scheduler retries retriable failures
|
|
Given a parallel subplan scheduler with max_parallel 2 with retry enabled
|
|
And 2 subplans where the first will fail once with TimeoutError then succeed
|
|
When the scheduler executes all subplans
|
|
Then both subplans should complete successfully
|
|
And the first subplan should have 1 previous attempt recorded
|
|
|
|
# --- Merge strategies ---
|
|
|
|
@merge
|
|
Scenario: Scheduler merges subplan outputs with git_three_way strategy
|
|
Given a parallel subplan scheduler with max_parallel 2 and git_three_way merge
|
|
And 2 subplans with non-overlapping file changes
|
|
When the scheduler executes all subplans
|
|
Then the execution result should include a merge result
|
|
And the merge result should have no conflicts
|
|
|
|
@merge
|
|
Scenario: Scheduler merges subplan outputs with last_wins strategy
|
|
Given a parallel subplan scheduler with max_parallel 2 and last_wins merge
|
|
And 2 subplans with overlapping file changes
|
|
When the scheduler executes all subplans
|
|
Then the execution result should include a merge result
|
|
And the merged content should be from the last subplan
|
|
|
|
# --- Execution modes ---
|
|
|
|
@modes
|
|
Scenario: Scheduler supports SEQUENTIAL execution mode
|
|
Given a parallel subplan scheduler in SEQUENTIAL mode with max_parallel 5
|
|
And 3 subplans to execute
|
|
When the scheduler executes all subplans
|
|
Then all 3 subplans should complete successfully
|
|
And the subplans should have been executed in sequential order
|
|
|
|
@modes
|
|
Scenario: Scheduler supports PARALLEL execution mode
|
|
Given a parallel subplan scheduler in PARALLEL mode with max_parallel 3
|
|
And 3 subplans to execute
|
|
When the scheduler executes all subplans
|
|
Then all 3 subplans should complete successfully
|
|
|
|
@modes
|
|
Scenario: Scheduler supports DEPENDENCY_ORDERED execution mode
|
|
Given a parallel subplan scheduler in DEPENDENCY_ORDERED mode with max_parallel 5
|
|
And 3 subplans where C depends on B which depends on A
|
|
When the scheduler executes all subplans
|
|
Then all 3 subplans should complete successfully
|
|
And subplan A should complete before subplan B
|
|
And subplan B should complete before subplan C
|
|
|
|
# --- Timeout enforcement ---
|
|
|
|
@timeout
|
|
Scenario: Scheduler enforces per-subplan timeout
|
|
Given a parallel subplan scheduler with max_parallel 2 and 1 second timeout
|
|
And 2 subplans where the first will block for 3 seconds
|
|
When the scheduler executes all subplans
|
|
Then at least one subplan should be errored with timeout
|
|
|
|
@timeout
|
|
Scenario: Scheduler timeout does not affect other subplans
|
|
Given a parallel subplan scheduler with max_parallel 2 and 1 second timeout
|
|
And 2 subplans where the first will block for 3 seconds and the second completes quickly
|
|
When the scheduler executes all subplans
|
|
Then the first subplan should be errored with timeout
|
|
And the second subplan should complete successfully
|
|
|
|
# --- Validation ---
|
|
|
|
@validation
|
|
Scenario: Scheduler rejects None config
|
|
When a ParallelSubplanScheduler is created with None config
|
|
Then a config validation error should be raised
|
|
|
|
@validation
|
|
Scenario: Scheduler rejects None executor
|
|
When a ParallelSubplanScheduler is created with None executor
|
|
Then an executor validation error should be raised
|
|
|
|
@validation
|
|
Scenario: Scheduler rejects empty subplan list
|
|
Given a valid parallel subplan scheduler
|
|
When schedule is called with empty subplan statuses
|
|
Then an empty statuses error should be raised
|
|
|
|
@validation
|
|
Scenario: Scheduler requires dependency graph for DEPENDENCY_ORDERED mode
|
|
Given a parallel subplan scheduler in DEPENDENCY_ORDERED mode
|
|
And 2 subplans to execute
|
|
When schedule is called without a dependency graph
|
|
Then a missing dependency graph error should be raised
|
|
|
|
# --- State tracking ---
|
|
|
|
@state
|
|
Scenario: Scheduler state reflects execution progress
|
|
Given a parallel subplan scheduler with max_parallel 2
|
|
And 4 subplans to execute
|
|
When the scheduler executes all subplans
|
|
Then the scheduler state should show started_at timestamp
|
|
And the scheduler state should show completed_at timestamp
|
|
And the scheduler state should show all 4 subplans completed
|
|
|
|
@state
|
|
Scenario: Scheduler state is_running reflects execution status
|
|
Given a parallel subplan scheduler with max_parallel 2
|
|
And 3 subplans to execute
|
|
When the scheduler starts execution
|
|
Then the scheduler state is_running should be true
|
|
And after execution completes, is_running should be false
|
|
|
|
# --- Property accessors ---
|
|
|
|
@accessor
|
|
Scenario: Scheduler exposes config property
|
|
Given a parallel subplan scheduler with max_parallel 3
|
|
Then the scheduler config property should return the configured config
|
|
And the config max_parallel should be 3
|
|
|
|
@accessor
|
|
Scenario: Scheduler exposes state property
|
|
Given a parallel subplan scheduler with max_parallel 2
|
|
Then the scheduler state property should return a SchedulerState
|
|
And the state max_parallel should be 2
|
|
|
|
@accessor
|
|
Scenario: Scheduler exposes max_parallel property
|
|
Given a parallel subplan scheduler with max_parallel 7
|
|
Then the scheduler max_parallel property should return 7
|
|
|
|
@accessor
|
|
Scenario: Scheduler exposes execution_mode property
|
|
Given a parallel subplan scheduler in PARALLEL mode
|
|
Then the scheduler execution_mode property should return PARALLEL
|
|
|
|
# --- Queue status methods ---
|
|
|
|
@queue_status
|
|
Scenario: Scheduler get_queue_status returns correct counts
|
|
Given a parallel subplan scheduler with max_parallel 2
|
|
And 5 subplans to execute
|
|
When the scheduler executes all subplans
|
|
Then get_queue_status should return pending=0, active=0, completed=5
|
|
|
|
@queue_status
|
|
Scenario: Scheduler get_available_slots returns correct count
|
|
Given a parallel subplan scheduler with max_parallel 3
|
|
And 5 subplans to execute
|
|
When the scheduler starts execution
|
|
Then get_available_slots should return 3
|
|
|
|
@queue_status
|
|
Scenario: Scheduler can_accept_more returns true when pending exist
|
|
Given a parallel subplan scheduler with max_parallel 2
|
|
And 5 subplans to execute
|
|
When the scheduler starts execution
|
|
Then can_accept_more should return true
|
|
|
|
@queue_status
|
|
Scenario: Scheduler can_accept_more returns false when no pending
|
|
Given a parallel subplan scheduler with max_parallel 2
|
|
And 2 subplans to execute
|
|
When the scheduler executes all subplans
|
|
Then can_accept_more should return false
|
|
|
|
# --- Integration scenarios ---
|
|
|
|
@integration
|
|
Scenario: Scheduler with 20 subplans and max_parallel 5 completes successfully
|
|
Given a parallel subplan scheduler with max_parallel 5
|
|
And 20 subplans to execute with concurrency tracking
|
|
When the scheduler executes all subplans
|
|
Then all 20 subplans should complete successfully
|
|
And the peak concurrent execution should not exceed 5
|
|
And the execution result should report all succeeded
|
|
|
|
@integration
|
|
Scenario: Scheduler with mixed success and failure handles correctly
|
|
Given a parallel subplan scheduler with max_parallel 3 with fail_fast disabled
|
|
And 5 subplans where 2 will fail
|
|
When the scheduler executes all subplans
|
|
Then 3 subplans should complete successfully
|
|
And 2 subplans should be errored
|
|
And the execution result should report not all succeeded
|
|
|
|
@integration
|
|
Scenario: Scheduler with dependency graph and max_parallel respects both
|
|
Given a parallel subplan scheduler in DEPENDENCY_ORDERED mode with max_parallel 2
|
|
And 4 subplans with dependencies: B depends on A, C depends on A, D depends on B and C
|
|
When the scheduler executes all subplans
|
|
Then all 4 subplans should complete successfully
|
|
And the peak concurrent execution should not exceed 2
|
|
And the dependency order should be respected
|