HAL9000
3cfa24854a
Ensure fail_fast cancels in-flight futures and reports them as CANCELLED. Add Behave coverage that reproduces the concurrency regression. ISSUES CLOSED: #7582
327 lines
14 KiB
Gherkin
327 lines
14 KiB
Gherkin
@phase1 @subplan @execution
|
|
Feature: Subplan Execution and Merge
|
|
As a system orchestrating parallel workloads
|
|
I want to execute subplans in sequential, parallel, and dependency-ordered modes
|
|
And merge their sandbox outputs using configurable strategies
|
|
So that parent plans can decompose work into coordinated child plans
|
|
|
|
# --- Sequential execution ---
|
|
|
|
@sequential
|
|
Scenario: Sequential execution completes all subplans in order
|
|
Given a parent plan with 3 subplans in sequential mode
|
|
When the subplans are executed
|
|
Then all 3 subplans should complete successfully
|
|
And the subplans should have been executed in order
|
|
And the execution result should report all succeeded
|
|
|
|
@sequential
|
|
Scenario: Sequential execution stops on failure when fail_fast is enabled
|
|
Given a parent plan with 3 subplans in sequential mode with fail_fast
|
|
And the second subplan will fail with "ValidationError: bad input"
|
|
When the subplans are executed
|
|
Then the first subplan should complete successfully
|
|
And the second subplan should be errored
|
|
And the third subplan should be cancelled
|
|
|
|
@sequential
|
|
Scenario: Sequential execution retries retriable failures
|
|
Given a parent plan with 1 subplan in sequential mode with retry enabled
|
|
And the subplan will fail once with "TimeoutError: timed out" then succeed
|
|
When the subplans are executed
|
|
Then the subplan should complete successfully
|
|
And the subplan should have 1 previous attempt recorded
|
|
|
|
# --- Parallel execution ---
|
|
|
|
@parallel
|
|
Scenario: Parallel execution runs subplans concurrently
|
|
Given a parent plan with 3 subplans in parallel mode with max_parallel 3
|
|
When the subplans are executed
|
|
Then all 3 subplans should complete successfully
|
|
And the execution result should report all succeeded
|
|
|
|
@parallel
|
|
Scenario: Parallel execution respects max_parallel limit
|
|
Given a parent plan with 5 subplans in parallel mode with max_parallel 2
|
|
When the subplans are executed
|
|
Then all 5 subplans should complete successfully
|
|
|
|
@parallel
|
|
Scenario: Parallel execution scales to 15 subplans while respecting max_parallel
|
|
Given a parent plan with 15 subplans in parallel mode with max_parallel 10
|
|
And parallel execution concurrency tracking is enabled
|
|
When the subplans are executed
|
|
Then all 15 subplans should complete successfully
|
|
And the peak concurrent execution count should not exceed 10
|
|
|
|
@parallel
|
|
Scenario: Parallel execution with fail_fast cancels remaining on failure
|
|
Given a parent plan with 3 subplans in parallel mode with fail_fast
|
|
And the first subplan will fail with "ValidationError: schema mismatch"
|
|
When the subplans are executed
|
|
Then at least one subplan should be errored
|
|
And the execution result should have failed subplan ids
|
|
|
|
# --- Dependency-ordered execution ---
|
|
|
|
@dependency_ordered
|
|
Scenario: Dependency-ordered execution respects DAG ordering
|
|
Given a parent plan with 3 subplans in dependency_ordered mode
|
|
And subplan C depends on subplan B which depends on subplan A
|
|
When the subplans are executed
|
|
Then all 3 subplans should complete successfully
|
|
And subplan A should have completed before subplan B
|
|
And subplan B should have completed before subplan C
|
|
|
|
@dependency_ordered
|
|
Scenario: Dependency-ordered execution rejects circular dependencies
|
|
Given a parent plan with 2 subplans in dependency_ordered mode
|
|
And the subplans have circular dependencies
|
|
When the subplans are executed expecting an error
|
|
Then a circular dependency error should be raised
|
|
|
|
@dependency_ordered
|
|
Scenario: Dependency-ordered mode requires a dependency graph
|
|
Given a parent plan with 2 subplans in dependency_ordered mode
|
|
When the subplans are executed without a dependency graph
|
|
Then a missing dependency graph error should be raised
|
|
|
|
# --- Merge strategies ---
|
|
|
|
@merge @git_three_way
|
|
Scenario: Git three-way merge combines non-overlapping changes
|
|
Given subplan outputs with non-overlapping changes to the same file
|
|
And the merge strategy is git_three_way
|
|
When the subplan outputs are merged
|
|
Then the merge should succeed
|
|
And the merged file should contain changes from both subplans
|
|
|
|
@merge @sequential_apply
|
|
Scenario: Sequential apply merges changes in order
|
|
Given subplan outputs with overlapping changes to the same file
|
|
And the merge strategy is sequential_apply
|
|
When the subplan outputs are merged
|
|
Then the merge should succeed
|
|
And the final content should reflect sequential application
|
|
|
|
@merge @fail_on_conflict
|
|
Scenario: Fail-on-conflict raises error when conflicts exist
|
|
Given subplan outputs with conflicting changes to the same file
|
|
And the merge strategy is fail_on_conflict
|
|
When the subplan outputs are merged expecting a conflict error
|
|
Then a merge conflict error should be raised
|
|
|
|
@merge @last_wins
|
|
Scenario: Last-wins takes the final subplan output
|
|
Given subplan outputs with overlapping changes to the same file
|
|
And the merge strategy is last_wins
|
|
When the subplan outputs are merged
|
|
Then the merge should succeed
|
|
And the merged content should be from the last subplan
|
|
|
|
@merge
|
|
Scenario: Merge with single subplan output returns unchanged content
|
|
Given a single subplan output with file changes
|
|
And the merge strategy is git_three_way
|
|
When the subplan outputs are merged
|
|
Then the merge should succeed
|
|
And the merged content should match the single subplan output
|
|
|
|
@merge
|
|
Scenario: Merge with empty subplan outputs raises error
|
|
Given no subplan outputs
|
|
And the merge strategy is git_three_way
|
|
When the subplan outputs are merged expecting a validation error
|
|
Then an empty outputs error should be raised
|
|
|
|
# --- Integration: execution + merge ---
|
|
|
|
@integration
|
|
Scenario: Sequential execution with git merge produces combined result
|
|
Given a parent plan with 2 subplans in sequential mode with git merge
|
|
And both subplans produce non-overlapping file changes
|
|
When the subplans are executed
|
|
Then the execution result should include a merge result
|
|
And the merge result should have no conflicts
|
|
|
|
@integration
|
|
Scenario: Parallel execution with last-wins merge resolves overlaps
|
|
Given a parent plan with 2 subplans in parallel mode with last_wins merge
|
|
And both subplans modify the same file differently
|
|
When the subplans are executed
|
|
Then the execution result should include a merge result
|
|
And the merge result should succeed
|
|
|
|
# --- Service validation ---
|
|
|
|
@validation
|
|
Scenario: SubplanExecutionService rejects None config
|
|
When a SubplanExecutionService is created with None config
|
|
Then a config validation error should be raised
|
|
|
|
@validation
|
|
Scenario: SubplanExecutionService rejects None executor
|
|
When a SubplanExecutionService is created with None executor
|
|
Then an executor validation error should be raised
|
|
|
|
@validation
|
|
Scenario: SubplanMergeService rejects None strategy
|
|
When a SubplanMergeService is created with None strategy
|
|
Then a strategy validation error should be raised
|
|
|
|
@validation
|
|
Scenario: SubplanExecutionService rejects empty subplan list
|
|
Given a valid SubplanExecutionService
|
|
When execute_all is called with empty subplan statuses
|
|
Then an empty statuses error should be raised
|
|
|
|
# --- Property accessors ---
|
|
|
|
@accessor
|
|
Scenario: SubplanExecutionService exposes config and merge_service properties
|
|
Given a valid SubplanExecutionService
|
|
Then the service config property should return the configured config
|
|
And the service merge_service property should return a merge service
|
|
|
|
@accessor
|
|
Scenario: SubplanMergeService exposes strategy property
|
|
Given a SubplanMergeService with git_three_way strategy
|
|
Then the merge strategy property should return git_three_way
|
|
|
|
# --- Retry edge cases ---
|
|
|
|
@retry
|
|
Scenario: Non-retriable error is not retried
|
|
Given a parent plan with 1 subplan in sequential mode with retry enabled
|
|
And the subplan will fail with "ConfigurationError: invalid config"
|
|
When the subplans are executed
|
|
Then the subplan should be errored
|
|
And the subplan attempt number should be 1
|
|
|
|
@retry
|
|
Scenario: Retry exhaustion marks subplan as permanently failed
|
|
Given a parent plan with 1 subplan in sequential mode with max_retries 1
|
|
And the subplan will always fail with "TimeoutError: timed out"
|
|
When the subplans are executed
|
|
Then the subplan should be errored
|
|
And the subplan should have 2 previous attempt recorded
|
|
|
|
@retry
|
|
Scenario: Executor exception triggers retry for retriable errors
|
|
Given a parent plan with 1 subplan in sequential mode with retry enabled
|
|
And the executor will raise "TimeoutError" once then succeed
|
|
When the subplans are executed
|
|
Then the subplan should complete successfully
|
|
And the subplan should have 1 previous attempt recorded
|
|
|
|
@retry
|
|
Scenario: Executor exception with non-retriable error is not retried
|
|
Given a parent plan with 1 subplan in sequential mode with retry enabled
|
|
And the executor will raise "ConfigurationError" permanently
|
|
When the subplans are executed
|
|
Then the subplan should be errored
|
|
And the subplan should have 1 previous attempt recorded
|
|
|
|
# --- Timeout enforcement ---
|
|
|
|
@timeout
|
|
Scenario: Sequential execution enforces per-subplan timeout
|
|
Given a parent plan with 1 subplan in sequential mode with a 1 second timeout
|
|
And the subplan executor will block for 3 seconds
|
|
When the subplans are executed
|
|
Then the subplan should be errored
|
|
And the subplan error should mention timeout
|
|
|
|
@timeout
|
|
Scenario: Parallel execution enforces per-subplan timeout
|
|
Given a parent plan with 2 subplans in parallel mode with a 1 second timeout
|
|
And the first subplan executor will block for 3 seconds
|
|
When the subplans are executed
|
|
Then at least one subplan error should mention timeout
|
|
|
|
# --- Completion order for parallel merge ---
|
|
|
|
@parallel @completion_order
|
|
Scenario: Parallel execution returns outputs in completion order
|
|
Given a parent plan with 3 subplans in parallel mode with staggered completion
|
|
When the subplans are executed
|
|
Then the execution result statuses should be in completion order
|
|
|
|
# --- Cancelled status in parallel fail_fast ---
|
|
|
|
@parallel @cancel_status
|
|
Scenario: Parallel fail_fast marks unstarted futures as CANCELLED not ERRORED
|
|
Given a parent plan with 3 subplans in parallel mode with max_parallel 1 and fail_fast
|
|
And the first subplan will fail with "ValidationError: schema mismatch"
|
|
When the subplans are executed
|
|
Then the first subplan should be errored
|
|
And the remaining subplans should have CANCELLED status
|
|
|
|
@parallel @cancel_status
|
|
Scenario: Parallel fail_fast marks in-flight futures as CANCELLED
|
|
Given a parent plan with 3 subplans in parallel mode with fail_fast
|
|
And the subplan executor will block for 1 seconds
|
|
And the first subplan will fail with "ValidationError: schema mismatch"
|
|
When the subplans are executed
|
|
Then the first subplan should be errored
|
|
And the remaining subplans should have CANCELLED status
|
|
|
|
# --- Dependency-ordered concurrent execution ---
|
|
|
|
@dependency_ordered @concurrent
|
|
Scenario: Dependency-ordered mode runs independent subplans concurrently
|
|
Given a parent plan with 3 subplans in dependency_ordered mode with concurrency tracking
|
|
And subplans A and B are independent while C depends on both
|
|
When the subplans are executed
|
|
Then all 3 subplans should complete successfully
|
|
And the peak concurrent execution count should be at least 2
|
|
And subplan C should have started after A and B completed
|
|
|
|
# --- Dependency-ordered fail_fast ---
|
|
|
|
@dependency_ordered @fail_fast
|
|
Scenario: Dependency-ordered fail_fast cancels subsequent waves
|
|
Given a parent plan with 3 subplans in dependency_ordered mode with fail_fast
|
|
And subplan C depends on subplan B which depends on subplan A
|
|
And the first subplan will fail with "ValidationError: bad input"
|
|
When the subplans are executed
|
|
Then the first subplan should be errored
|
|
And the second and third subplans should have CANCELLED status
|
|
|
|
# --- Dependency-ordered timeout enforcement ---
|
|
|
|
@dependency_ordered @timeout
|
|
Scenario: Dependency-ordered mode enforces timeout on single-node wave
|
|
Given a parent plan with 2 subplans in dependency_ordered mode with a 1 second timeout
|
|
And subplan B depends on subplan A
|
|
And the first subplan executor will block for 3 seconds
|
|
When the subplans are executed
|
|
Then at least one subplan error should mention timeout
|
|
|
|
@dependency_ordered @timeout
|
|
Scenario: Dependency-ordered mode enforces timeout on multi-node wave
|
|
Given a parent plan with 3 subplans in dependency_ordered mode with concurrency and a 1 second timeout
|
|
And subplans A and B are independent while C depends on both
|
|
And the subplan executor will block for 3 seconds
|
|
When the subplans are executed
|
|
Then at least one subplan error should mention timeout
|
|
|
|
# --- Parallel executor exception bubbling ---
|
|
|
|
@parallel @error
|
|
Scenario: Parallel execution handles executor exception as ERRORED
|
|
Given a parent plan with 2 subplans in parallel mode with max_parallel 2
|
|
And the executor will raise "RuntimeError" for the first subplan in parallel
|
|
When the subplans are executed
|
|
Then at least one subplan should be errored
|
|
|
|
# --- Integration: merge conflict during execution ---
|
|
|
|
@integration
|
|
Scenario: Execution with merge conflict marks result as not all succeeded
|
|
Given a parent plan with 2 subplans in sequential mode with fail_on_conflict merge
|
|
And both subplans produce conflicting file changes
|
|
When the subplans are executed
|
|
Then the execution result should report not all succeeded
|