HAL9000
195fbac109
CI / lint (pull_request) Successful in 40s
CI / helm (pull_request) Successful in 38s
CI / build (pull_request) Successful in 48s
CI / quality (pull_request) Successful in 1m0s
CI / typecheck (pull_request) Successful in 1m29s
CI / security (pull_request) Successful in 1m28s
CI / push-validation (pull_request) Successful in 20s
CI / unit_tests (pull_request) Successful in 5m8s
CI / docker (pull_request) Successful in 1m27s
CI / integration_tests (pull_request) Successful in 8m39s
CI / coverage (pull_request) Successful in 11m1s
CI / status-check (pull_request) Successful in 3s
Add domain-scenario Gherkin tags to all A2A, session, and CLI feature files (30 files) so tests can be filtered individually via behave. - 8 A2A feature files: @a2a tag - 7 session feature files: @session tag - 15 CLI feature files: @cli tag ISSUES CLOSED: #9124
398 lines
20 KiB
Gherkin
398 lines
20 KiB
Gherkin
@cli
|
|
Feature: CLI lifecycle command coverage for action and plan commands
|
|
As a developer
|
|
I want comprehensive CLI coverage for action/plan lifecycle commands
|
|
So that success paths, error paths, and terminal outcomes are verified
|
|
|
|
Background:
|
|
Given a lifecycle coverage CLI runner
|
|
And a lifecycle coverage mocked lifecycle service
|
|
|
|
# ===================================================================
|
|
# Action create/list/show/archive success + error paths
|
|
# ===================================================================
|
|
|
|
Scenario: Lifecycle coverage action create from valid config succeeds
|
|
Given a lifecycle coverage valid action config file
|
|
When I run lifecycle coverage action create with config
|
|
Then lifecycle coverage action create should succeed
|
|
And lifecycle coverage created action name should be "local/lc-action"
|
|
|
|
Scenario: Lifecycle coverage action create with missing config file fails
|
|
When I run lifecycle coverage action create with missing config
|
|
Then lifecycle coverage action CLI should abort
|
|
|
|
Scenario: Lifecycle coverage action create surfaces pydantic validation error
|
|
Given a lifecycle coverage invalid schema config file
|
|
When I run lifecycle coverage action create with invalid schema config
|
|
Then lifecycle coverage action CLI should abort
|
|
|
|
Scenario: Lifecycle coverage action create surfaces value error from config
|
|
Given a lifecycle coverage invalid value config file
|
|
When I run lifecycle coverage action create with value error config
|
|
Then lifecycle coverage action CLI should abort
|
|
|
|
Scenario: Lifecycle coverage action list returns all actions in table
|
|
Given lifecycle coverage mocked actions exist
|
|
When I run lifecycle coverage action list
|
|
Then lifecycle coverage action list should show table with 3 actions
|
|
|
|
Scenario: Lifecycle coverage action list with json format
|
|
Given lifecycle coverage mocked actions exist
|
|
When I run lifecycle coverage action list with format "json"
|
|
Then lifecycle coverage action list should succeed
|
|
|
|
Scenario: Lifecycle coverage action list with regex filter
|
|
Given lifecycle coverage mocked actions exist
|
|
When I run lifecycle coverage action list with regex "alpha"
|
|
Then lifecycle coverage action list filtered should succeed
|
|
|
|
Scenario: Lifecycle coverage action list with invalid regex aborts
|
|
Given lifecycle coverage mocked actions exist
|
|
When I run lifecycle coverage action list with invalid regex "[bad"
|
|
Then lifecycle coverage action CLI should abort
|
|
|
|
Scenario: Lifecycle coverage action show by namespaced name
|
|
Given lifecycle coverage a specific action "local/alpha-action" exists
|
|
When I run lifecycle coverage action show "local/alpha-action" in default format
|
|
Then lifecycle coverage action show should display details
|
|
|
|
Scenario: Lifecycle coverage action show with json format
|
|
Given lifecycle coverage a specific action "local/alpha-action" exists
|
|
When I run lifecycle coverage action show "local/alpha-action" with format "json"
|
|
Then lifecycle coverage action show should succeed
|
|
|
|
Scenario: Lifecycle coverage action show not found aborts
|
|
Given lifecycle coverage action not found for show
|
|
When I run lifecycle coverage action show "local/no-such-action" in default format
|
|
Then lifecycle coverage action not found should abort
|
|
|
|
Scenario: Lifecycle coverage action archive succeeds
|
|
Given lifecycle coverage an archivable action exists
|
|
When I run lifecycle coverage action archive "local/alpha-action" in default format
|
|
Then lifecycle coverage action archive should succeed
|
|
|
|
Scenario: Lifecycle coverage action archive with json format
|
|
Given lifecycle coverage an archivable action exists
|
|
When I run lifecycle coverage action archive "local/alpha-action" using format "json"
|
|
Then lifecycle coverage action archive json should succeed
|
|
|
|
Scenario: Lifecycle coverage action archive not found aborts
|
|
Given lifecycle coverage action not found for archive
|
|
When I run lifecycle coverage action archive "local/no-such-action" in default format
|
|
Then lifecycle coverage action not found should abort
|
|
|
|
# ===================================================================
|
|
# Plan use success + error paths
|
|
# ===================================================================
|
|
|
|
Scenario: Lifecycle coverage plan use creates plan in strategize phase
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use "local/lc-action" targeting project "proj-a"
|
|
Then lifecycle coverage plan use should succeed
|
|
And lifecycle coverage plan should be in strategize phase
|
|
|
|
Scenario: Lifecycle coverage plan use with multi-project args
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use "local/lc-action" with projects "proj-a" "proj-b" "proj-c"
|
|
Then lifecycle coverage plan use should succeed
|
|
And lifecycle coverage plan should link 3 projects
|
|
|
|
Scenario: Lifecycle coverage plan use with automation profile override
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use with automation profile "cautious"
|
|
Then lifecycle coverage plan use should succeed
|
|
And lifecycle coverage plan automation profile should be "cautious"
|
|
|
|
Scenario: Lifecycle coverage plan use with invariants
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use with invariants "No warnings" and "Keep API compat"
|
|
Then lifecycle coverage plan use should succeed
|
|
And lifecycle coverage plan should have 2 invariants
|
|
|
|
Scenario: Lifecycle coverage plan use with strategy actor override
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use with strategy actor "anthropic/claude-3"
|
|
Then lifecycle coverage plan use should succeed
|
|
And lifecycle coverage plan strategy actor should be "anthropic/claude-3"
|
|
|
|
Scenario: Lifecycle coverage plan use with execution actor override
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use with execution actor "openai/gpt-4o"
|
|
Then lifecycle coverage plan use should succeed
|
|
And lifecycle coverage plan execution actor should be "openai/gpt-4o"
|
|
|
|
Scenario: Lifecycle coverage plan use with estimation actor override
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use with estimation actor "openai/gpt-4-turbo"
|
|
Then lifecycle coverage plan use should succeed
|
|
|
|
Scenario: Lifecycle coverage plan use with invariant actor override
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use with invariant actor "anthropic/claude-3-opus"
|
|
Then lifecycle coverage plan use should succeed
|
|
|
|
Scenario: Lifecycle coverage plan use with arg name=value
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use with arg "target_coverage=90"
|
|
Then lifecycle coverage plan use should succeed
|
|
|
|
Scenario: Lifecycle coverage plan use with removed automation level flag rejected
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use with automation level "full_automation"
|
|
Then lifecycle coverage plan use should abort
|
|
|
|
Scenario: Lifecycle coverage plan use invalid automation level also rejected
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use with invalid automation level "mega_auto"
|
|
Then lifecycle coverage plan use should abort
|
|
|
|
Scenario: Lifecycle coverage plan use invalid arg format aborts
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use with invalid arg "badformat"
|
|
Then lifecycle coverage plan use should abort
|
|
|
|
Scenario: Lifecycle coverage plan use action not available error
|
|
Given lifecycle coverage an action not available for plan use
|
|
When I run lifecycle coverage plan use "local/archived-action" targeting project "proj-a"
|
|
Then lifecycle coverage plan use should abort
|
|
|
|
Scenario: Lifecycle coverage plan use validation error
|
|
Given lifecycle coverage an action with validation error for plan use
|
|
When I run lifecycle coverage plan use "local/lc-action" targeting project "proj-a"
|
|
Then lifecycle coverage plan use should abort
|
|
|
|
Scenario: Lifecycle coverage plan use general error
|
|
Given lifecycle coverage an action with general error for plan use
|
|
When I run lifecycle coverage plan use "local/lc-action" targeting project "proj-a"
|
|
Then lifecycle coverage plan use should abort
|
|
|
|
Scenario: Lifecycle coverage plan use with json format output
|
|
Given lifecycle coverage an action for plan use exists
|
|
When I run lifecycle coverage plan use "local/lc-action" on project "proj-a" with format "json"
|
|
Then lifecycle coverage plan use should succeed
|
|
|
|
# ===================================================================
|
|
# Plan list success + error paths
|
|
# ===================================================================
|
|
|
|
Scenario: Lifecycle coverage plan list shows plans
|
|
Given lifecycle coverage plans exist for listing
|
|
When I run lifecycle coverage plan list
|
|
Then lifecycle coverage plan list should show table
|
|
|
|
Scenario: Lifecycle coverage plan list with json format
|
|
Given lifecycle coverage plans exist for listing
|
|
When I run lifecycle coverage plan list with format "json"
|
|
Then lifecycle coverage plan list should succeed
|
|
|
|
Scenario: Lifecycle coverage plan list empty shows message
|
|
Given lifecycle coverage no plans exist for listing
|
|
When I run lifecycle coverage plan list
|
|
Then lifecycle coverage plan list should show no plans message
|
|
|
|
Scenario: Lifecycle coverage plan list invalid phase aborts
|
|
When I run lifecycle coverage plan list with invalid phase "bogus"
|
|
Then lifecycle coverage plan list should abort
|
|
|
|
Scenario: Lifecycle coverage plan list invalid state aborts
|
|
When I run lifecycle coverage plan list with invalid state "bogus"
|
|
Then lifecycle coverage plan list should abort
|
|
|
|
# ===================================================================
|
|
# Plan status success + error paths
|
|
# ===================================================================
|
|
|
|
Scenario: Lifecycle coverage plan status shows specific plan
|
|
Given lifecycle coverage a specific plan exists for status
|
|
When I run lifecycle coverage plan status with plan ID
|
|
Then lifecycle coverage plan status should show details
|
|
|
|
Scenario: Lifecycle coverage plan status lists all when no ID
|
|
Given lifecycle coverage plans exist for listing
|
|
When I run lifecycle coverage plan status without ID
|
|
Then lifecycle coverage plan status should show table
|
|
|
|
Scenario: Lifecycle coverage plan status no plans message
|
|
Given lifecycle coverage no plans exist for listing
|
|
When I run lifecycle coverage plan status without ID
|
|
Then lifecycle coverage plan status should show no plans
|
|
|
|
Scenario: Lifecycle coverage plan status with json format
|
|
Given lifecycle coverage a specific plan exists for status
|
|
When I run lifecycle coverage plan status with plan ID and format "json"
|
|
Then lifecycle coverage plan status should succeed
|
|
|
|
Scenario: Lifecycle coverage plan status shows error on service failure
|
|
Given lifecycle coverage plan status service error
|
|
When I run lifecycle coverage plan status with plan ID
|
|
Then lifecycle coverage plan status should abort
|
|
|
|
# ===================================================================
|
|
# Plan execute success + error paths
|
|
# ===================================================================
|
|
|
|
Scenario: Lifecycle coverage plan execute transitions to execute phase
|
|
Given lifecycle coverage a plan ready for execute exists
|
|
When I run lifecycle coverage plan execute with plan ID
|
|
Then lifecycle coverage plan execute should succeed
|
|
|
|
Scenario: Lifecycle coverage plan execute auto-selects single ready plan
|
|
Given lifecycle coverage a single plan ready for auto-execute exists
|
|
When I run lifecycle coverage plan execute without ID
|
|
Then lifecycle coverage plan execute should succeed
|
|
|
|
Scenario: Lifecycle coverage plan execute no ready plans aborts
|
|
Given lifecycle coverage no plans ready for execute
|
|
When I run lifecycle coverage plan execute without ID
|
|
Then lifecycle coverage plan execute should abort
|
|
|
|
Scenario: Lifecycle coverage plan execute multiple ready plans aborts
|
|
Given lifecycle coverage multiple plans ready for execute
|
|
When I run lifecycle coverage plan execute without ID
|
|
Then lifecycle coverage plan execute should abort
|
|
|
|
Scenario: Lifecycle coverage plan execute invalid transition aborts
|
|
Given lifecycle coverage a plan with invalid transition for execute
|
|
When I run lifecycle coverage plan execute with plan ID
|
|
Then lifecycle coverage plan execute should abort
|
|
|
|
Scenario: Lifecycle coverage plan execute plan not ready aborts
|
|
Given lifecycle coverage a plan not ready for execute
|
|
When I run lifecycle coverage plan execute with plan ID
|
|
Then lifecycle coverage plan execute should abort
|
|
|
|
Scenario: Lifecycle coverage plan execute with json format
|
|
Given lifecycle coverage a plan ready for execute exists
|
|
When I run lifecycle coverage plan execute with plan ID and format "json"
|
|
Then lifecycle coverage plan execute should succeed
|
|
|
|
# ===================================================================
|
|
# Plan apply success + error paths
|
|
# ===================================================================
|
|
|
|
Scenario: Lifecycle coverage plan apply transitions to apply phase
|
|
Given lifecycle coverage a plan ready for apply exists
|
|
When I run lifecycle coverage plan apply with plan ID
|
|
Then lifecycle coverage plan apply should succeed
|
|
|
|
Scenario: Lifecycle coverage plan apply auto-selects single ready plan
|
|
Given lifecycle coverage a single plan ready for auto-apply exists
|
|
When I run lifecycle coverage plan apply without ID
|
|
Then lifecycle coverage plan apply should succeed
|
|
|
|
Scenario: Lifecycle coverage plan apply no ready plans aborts
|
|
Given lifecycle coverage no plans ready for apply
|
|
When I run lifecycle coverage plan apply without ID
|
|
Then lifecycle coverage plan apply should abort
|
|
|
|
Scenario: Lifecycle coverage plan apply multiple ready plans aborts
|
|
Given lifecycle coverage multiple plans ready for apply
|
|
When I run lifecycle coverage plan apply without ID
|
|
Then lifecycle coverage plan apply should abort
|
|
|
|
Scenario: Lifecycle coverage plan apply with json format
|
|
Given lifecycle coverage a plan ready for apply exists
|
|
When I run lifecycle coverage plan apply with plan ID and format "json"
|
|
Then lifecycle coverage plan apply should succeed
|
|
|
|
# ===================================================================
|
|
# Plan cancel success + error paths
|
|
# ===================================================================
|
|
|
|
Scenario: Lifecycle coverage plan cancel with reason
|
|
Given lifecycle coverage a cancellable plan exists
|
|
When I run lifecycle coverage plan cancel providing reason "No longer needed"
|
|
Then lifecycle coverage plan cancel should succeed
|
|
And lifecycle coverage cancel output should contain "No longer needed"
|
|
|
|
Scenario: Lifecycle coverage plan cancel without reason
|
|
Given lifecycle coverage a cancellable plan exists
|
|
When I run lifecycle coverage plan cancel without reason
|
|
Then lifecycle coverage plan cancel should succeed
|
|
|
|
Scenario: Lifecycle coverage plan cancel terminal plan aborts
|
|
Given lifecycle coverage a terminal plan exists for cancel
|
|
When I run lifecycle coverage plan cancel providing reason "try cancel"
|
|
Then lifecycle coverage plan cancel should abort
|
|
|
|
Scenario: Lifecycle coverage plan cancel with json format
|
|
Given lifecycle coverage a cancellable plan exists
|
|
When I run lifecycle coverage plan cancel with reason "cleanup" and format "json"
|
|
Then lifecycle coverage plan cancel should succeed
|
|
|
|
# ===================================================================
|
|
# Action phase visibility in plan status
|
|
# ===================================================================
|
|
|
|
Scenario: Lifecycle coverage plan status shows action phase as strategize
|
|
Given lifecycle coverage a plan in strategize phase exists
|
|
When I run lifecycle coverage plan status with plan ID
|
|
Then lifecycle coverage plan status should show phase "strategize"
|
|
|
|
Scenario: Lifecycle coverage plan status shows action phase as execute
|
|
Given lifecycle coverage a plan in execute phase exists
|
|
When I run lifecycle coverage plan status with plan ID
|
|
Then lifecycle coverage plan status should show phase "execute"
|
|
|
|
Scenario: Lifecycle coverage plan status shows action phase as apply
|
|
Given lifecycle coverage a plan in apply phase exists
|
|
When I run lifecycle coverage plan status with plan ID
|
|
Then lifecycle coverage plan status should show phase "apply"
|
|
|
|
# ===================================================================
|
|
# Apply terminal outcomes in plan status output
|
|
# ===================================================================
|
|
|
|
Scenario: Lifecycle coverage plan status shows applied terminal outcome
|
|
Given lifecycle coverage a plan with applied outcome exists
|
|
When I run lifecycle coverage plan status with plan ID
|
|
Then lifecycle coverage plan status should show state "applied"
|
|
And lifecycle coverage plan should be terminal
|
|
|
|
Scenario: Lifecycle coverage plan status shows constrained terminal outcome
|
|
Given lifecycle coverage a plan with constrained outcome exists
|
|
When I run lifecycle coverage plan status with plan ID
|
|
Then lifecycle coverage plan status should show state "constrained"
|
|
And lifecycle coverage plan should be terminal
|
|
|
|
Scenario: Lifecycle coverage plan status shows errored outcome
|
|
Given lifecycle coverage a plan with errored outcome exists
|
|
When I run lifecycle coverage plan status with plan ID
|
|
Then lifecycle coverage plan status should show state "errored"
|
|
|
|
Scenario: Lifecycle coverage plan status shows cancelled terminal outcome
|
|
Given lifecycle coverage a plan with cancelled outcome exists
|
|
When I run lifecycle coverage plan status with plan ID
|
|
Then lifecycle coverage plan status should show state "cancelled"
|
|
And lifecycle coverage plan should be terminal
|
|
|
|
# ===================================================================
|
|
# Negative cases
|
|
# ===================================================================
|
|
|
|
Scenario: Lifecycle coverage action create with empty config file fails
|
|
Given a lifecycle coverage empty config file
|
|
When I run lifecycle coverage action create with empty config
|
|
Then lifecycle coverage action CLI should abort
|
|
|
|
Scenario: Lifecycle coverage plan use with unknown action name
|
|
Given lifecycle coverage unknown action for plan use
|
|
When I run lifecycle coverage plan use "local/does-not-exist" targeting project "proj-a"
|
|
Then lifecycle coverage plan use should abort
|
|
|
|
Scenario: Lifecycle coverage plan execute with general CleverAgentsError
|
|
Given lifecycle coverage a plan execute with general error
|
|
When I run lifecycle coverage plan execute with plan ID
|
|
Then lifecycle coverage plan execute should abort
|
|
|
|
Scenario: Lifecycle coverage plan apply with general error
|
|
Given lifecycle coverage a plan apply with general error
|
|
When I run lifecycle coverage plan apply with plan ID
|
|
Then lifecycle coverage plan apply should abort
|
|
|
|
Scenario: Lifecycle coverage plan cancel with general error
|
|
Given lifecycle coverage a plan cancel with general error
|
|
When I run lifecycle coverage plan cancel providing reason "err"
|
|
Then lifecycle coverage plan cancel should abort
|