test(cli): expand lifecycle command coverage
CI / lint (pull_request) Successful in 17s
CI / typecheck (pull_request) Successful in 35s
CI / security (pull_request) Successful in 36s
CI / quality (pull_request) Successful in 19s
CI / build (pull_request) Successful in 26s
CI / integration_tests (pull_request) Successful in 5m19s
CI / unit_tests (pull_request) Failing after 32m35s
CI / docker (pull_request) Has been skipped
CI / coverage (pull_request) Failing after 12m0s
CI / lint (pull_request) Successful in 17s
CI / typecheck (pull_request) Successful in 35s
CI / security (pull_request) Successful in 36s
CI / quality (pull_request) Successful in 19s
CI / build (pull_request) Successful in 26s
CI / integration_tests (pull_request) Successful in 5m19s
CI / unit_tests (pull_request) Failing after 32m35s
CI / docker (pull_request) Has been skipped
CI / coverage (pull_request) Failing after 12m0s
This commit is contained in:
@@ -0,0 +1,396 @@
|
||||
Feature: CLI lifecycle command coverage for action and plan commands
|
||||
As a developer
|
||||
I want comprehensive CLI coverage for action/plan lifecycle commands
|
||||
So that success paths, error paths, and terminal outcomes are verified
|
||||
|
||||
Background:
|
||||
Given a lifecycle coverage CLI runner
|
||||
And a lifecycle coverage mocked lifecycle service
|
||||
|
||||
# ===================================================================
|
||||
# Action create/list/show/archive success + error paths
|
||||
# ===================================================================
|
||||
|
||||
Scenario: Lifecycle coverage action create from valid config succeeds
|
||||
Given a lifecycle coverage valid action config file
|
||||
When I run lifecycle coverage action create with config
|
||||
Then lifecycle coverage action create should succeed
|
||||
And lifecycle coverage created action name should be "local/lc-action"
|
||||
|
||||
Scenario: Lifecycle coverage action create with missing config file fails
|
||||
When I run lifecycle coverage action create with missing config
|
||||
Then lifecycle coverage action CLI should abort
|
||||
|
||||
Scenario: Lifecycle coverage action create surfaces pydantic validation error
|
||||
Given a lifecycle coverage invalid schema config file
|
||||
When I run lifecycle coverage action create with invalid schema config
|
||||
Then lifecycle coverage action CLI should abort
|
||||
|
||||
Scenario: Lifecycle coverage action create surfaces value error from config
|
||||
Given a lifecycle coverage invalid value config file
|
||||
When I run lifecycle coverage action create with value error config
|
||||
Then lifecycle coverage action CLI should abort
|
||||
|
||||
Scenario: Lifecycle coverage action list returns all actions in table
|
||||
Given lifecycle coverage mocked actions exist
|
||||
When I run lifecycle coverage action list
|
||||
Then lifecycle coverage action list should show table with 3 actions
|
||||
|
||||
Scenario: Lifecycle coverage action list with json format
|
||||
Given lifecycle coverage mocked actions exist
|
||||
When I run lifecycle coverage action list with format "json"
|
||||
Then lifecycle coverage action list should succeed
|
||||
|
||||
Scenario: Lifecycle coverage action list with regex filter
|
||||
Given lifecycle coverage mocked actions exist
|
||||
When I run lifecycle coverage action list with regex "alpha"
|
||||
Then lifecycle coverage action list filtered should succeed
|
||||
|
||||
Scenario: Lifecycle coverage action list with invalid regex aborts
|
||||
Given lifecycle coverage mocked actions exist
|
||||
When I run lifecycle coverage action list with invalid regex "[bad"
|
||||
Then lifecycle coverage action CLI should abort
|
||||
|
||||
Scenario: Lifecycle coverage action show by namespaced name
|
||||
Given lifecycle coverage a specific action "local/alpha-action" exists
|
||||
When I run lifecycle coverage action show "local/alpha-action" in default format
|
||||
Then lifecycle coverage action show should display details
|
||||
|
||||
Scenario: Lifecycle coverage action show with json format
|
||||
Given lifecycle coverage a specific action "local/alpha-action" exists
|
||||
When I run lifecycle coverage action show "local/alpha-action" with format "json"
|
||||
Then lifecycle coverage action show should succeed
|
||||
|
||||
Scenario: Lifecycle coverage action show not found aborts
|
||||
Given lifecycle coverage action not found for show
|
||||
When I run lifecycle coverage action show "local/no-such-action" in default format
|
||||
Then lifecycle coverage action not found should abort
|
||||
|
||||
Scenario: Lifecycle coverage action archive succeeds
|
||||
Given lifecycle coverage an archivable action exists
|
||||
When I run lifecycle coverage action archive "local/alpha-action" in default format
|
||||
Then lifecycle coverage action archive should succeed
|
||||
|
||||
Scenario: Lifecycle coverage action archive with json format
|
||||
Given lifecycle coverage an archivable action exists
|
||||
When I run lifecycle coverage action archive "local/alpha-action" using format "json"
|
||||
Then lifecycle coverage action archive json should succeed
|
||||
|
||||
Scenario: Lifecycle coverage action archive not found aborts
|
||||
Given lifecycle coverage action not found for archive
|
||||
When I run lifecycle coverage action archive "local/no-such-action" in default format
|
||||
Then lifecycle coverage action not found should abort
|
||||
|
||||
# ===================================================================
|
||||
# Plan use success + error paths
|
||||
# ===================================================================
|
||||
|
||||
Scenario: Lifecycle coverage plan use creates plan in strategize phase
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use "local/lc-action" targeting project "proj-a"
|
||||
Then lifecycle coverage plan use should succeed
|
||||
And lifecycle coverage plan should be in strategize phase
|
||||
|
||||
Scenario: Lifecycle coverage plan use with multi-project args
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use "local/lc-action" with projects "proj-a" "proj-b" "proj-c"
|
||||
Then lifecycle coverage plan use should succeed
|
||||
And lifecycle coverage plan should link 3 projects
|
||||
|
||||
Scenario: Lifecycle coverage plan use with automation profile override
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use with automation profile "careful-auto"
|
||||
Then lifecycle coverage plan use should succeed
|
||||
And lifecycle coverage plan automation profile should be "careful-auto"
|
||||
|
||||
Scenario: Lifecycle coverage plan use with invariants
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use with invariants "No warnings" and "Keep API compat"
|
||||
Then lifecycle coverage plan use should succeed
|
||||
And lifecycle coverage plan should have 2 invariants
|
||||
|
||||
Scenario: Lifecycle coverage plan use with strategy actor override
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use with strategy actor "anthropic/claude-3"
|
||||
Then lifecycle coverage plan use should succeed
|
||||
And lifecycle coverage plan strategy actor should be "anthropic/claude-3"
|
||||
|
||||
Scenario: Lifecycle coverage plan use with execution actor override
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use with execution actor "openai/gpt-4o"
|
||||
Then lifecycle coverage plan use should succeed
|
||||
And lifecycle coverage plan execution actor should be "openai/gpt-4o"
|
||||
|
||||
Scenario: Lifecycle coverage plan use with estimation actor override
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use with estimation actor "openai/gpt-4-turbo"
|
||||
Then lifecycle coverage plan use should succeed
|
||||
|
||||
Scenario: Lifecycle coverage plan use with invariant actor override
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use with invariant actor "anthropic/claude-3-opus"
|
||||
Then lifecycle coverage plan use should succeed
|
||||
|
||||
Scenario: Lifecycle coverage plan use with arg name=value
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use with arg "target_coverage=90"
|
||||
Then lifecycle coverage plan use should succeed
|
||||
|
||||
Scenario: Lifecycle coverage plan use with automation level override
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use with automation level "full_automation"
|
||||
Then lifecycle coverage plan use should succeed
|
||||
|
||||
Scenario: Lifecycle coverage plan use invalid automation level aborts
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use with invalid automation level "mega_auto"
|
||||
Then lifecycle coverage plan use should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan use invalid arg format aborts
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use with invalid arg "badformat"
|
||||
Then lifecycle coverage plan use should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan use action not available error
|
||||
Given lifecycle coverage an action not available for plan use
|
||||
When I run lifecycle coverage plan use "local/archived-action" targeting project "proj-a"
|
||||
Then lifecycle coverage plan use should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan use validation error
|
||||
Given lifecycle coverage an action with validation error for plan use
|
||||
When I run lifecycle coverage plan use "local/lc-action" targeting project "proj-a"
|
||||
Then lifecycle coverage plan use should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan use general error
|
||||
Given lifecycle coverage an action with general error for plan use
|
||||
When I run lifecycle coverage plan use "local/lc-action" targeting project "proj-a"
|
||||
Then lifecycle coverage plan use should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan use with json format output
|
||||
Given lifecycle coverage an action for plan use exists
|
||||
When I run lifecycle coverage plan use "local/lc-action" on project "proj-a" with format "json"
|
||||
Then lifecycle coverage plan use should succeed
|
||||
|
||||
# ===================================================================
|
||||
# Plan lifecycle-list success + error paths
|
||||
# ===================================================================
|
||||
|
||||
Scenario: Lifecycle coverage plan lifecycle-list shows plans
|
||||
Given lifecycle coverage plans exist for listing
|
||||
When I run lifecycle coverage plan lifecycle-list
|
||||
Then lifecycle coverage plan list should show table
|
||||
|
||||
Scenario: Lifecycle coverage plan lifecycle-list with json format
|
||||
Given lifecycle coverage plans exist for listing
|
||||
When I run lifecycle coverage plan lifecycle-list with format "json"
|
||||
Then lifecycle coverage plan list should succeed
|
||||
|
||||
Scenario: Lifecycle coverage plan lifecycle-list empty shows message
|
||||
Given lifecycle coverage no plans exist for listing
|
||||
When I run lifecycle coverage plan lifecycle-list
|
||||
Then lifecycle coverage plan list should show no plans message
|
||||
|
||||
Scenario: Lifecycle coverage plan lifecycle-list invalid phase aborts
|
||||
When I run lifecycle coverage plan lifecycle-list with invalid phase "bogus"
|
||||
Then lifecycle coverage plan list should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan lifecycle-list invalid state aborts
|
||||
When I run lifecycle coverage plan lifecycle-list with invalid state "bogus"
|
||||
Then lifecycle coverage plan list should abort
|
||||
|
||||
# ===================================================================
|
||||
# Plan status success + error paths
|
||||
# ===================================================================
|
||||
|
||||
Scenario: Lifecycle coverage plan status shows specific plan
|
||||
Given lifecycle coverage a specific plan exists for status
|
||||
When I run lifecycle coverage plan status with plan ID
|
||||
Then lifecycle coverage plan status should show details
|
||||
|
||||
Scenario: Lifecycle coverage plan status lists all when no ID
|
||||
Given lifecycle coverage plans exist for listing
|
||||
When I run lifecycle coverage plan status without ID
|
||||
Then lifecycle coverage plan status should show table
|
||||
|
||||
Scenario: Lifecycle coverage plan status no plans message
|
||||
Given lifecycle coverage no plans exist for listing
|
||||
When I run lifecycle coverage plan status without ID
|
||||
Then lifecycle coverage plan status should show no plans
|
||||
|
||||
Scenario: Lifecycle coverage plan status with json format
|
||||
Given lifecycle coverage a specific plan exists for status
|
||||
When I run lifecycle coverage plan status with plan ID and format "json"
|
||||
Then lifecycle coverage plan status should succeed
|
||||
|
||||
Scenario: Lifecycle coverage plan status shows error on service failure
|
||||
Given lifecycle coverage plan status service error
|
||||
When I run lifecycle coverage plan status with plan ID
|
||||
Then lifecycle coverage plan status should abort
|
||||
|
||||
# ===================================================================
|
||||
# Plan execute success + error paths
|
||||
# ===================================================================
|
||||
|
||||
Scenario: Lifecycle coverage plan execute transitions to execute phase
|
||||
Given lifecycle coverage a plan ready for execute exists
|
||||
When I run lifecycle coverage plan execute with plan ID
|
||||
Then lifecycle coverage plan execute should succeed
|
||||
|
||||
Scenario: Lifecycle coverage plan execute auto-selects single ready plan
|
||||
Given lifecycle coverage a single plan ready for auto-execute exists
|
||||
When I run lifecycle coverage plan execute without ID
|
||||
Then lifecycle coverage plan execute should succeed
|
||||
|
||||
Scenario: Lifecycle coverage plan execute no ready plans aborts
|
||||
Given lifecycle coverage no plans ready for execute
|
||||
When I run lifecycle coverage plan execute without ID
|
||||
Then lifecycle coverage plan execute should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan execute multiple ready plans aborts
|
||||
Given lifecycle coverage multiple plans ready for execute
|
||||
When I run lifecycle coverage plan execute without ID
|
||||
Then lifecycle coverage plan execute should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan execute invalid transition aborts
|
||||
Given lifecycle coverage a plan with invalid transition for execute
|
||||
When I run lifecycle coverage plan execute with plan ID
|
||||
Then lifecycle coverage plan execute should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan execute plan not ready aborts
|
||||
Given lifecycle coverage a plan not ready for execute
|
||||
When I run lifecycle coverage plan execute with plan ID
|
||||
Then lifecycle coverage plan execute should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan execute with json format
|
||||
Given lifecycle coverage a plan ready for execute exists
|
||||
When I run lifecycle coverage plan execute with plan ID and format "json"
|
||||
Then lifecycle coverage plan execute should succeed
|
||||
|
||||
# ===================================================================
|
||||
# Plan lifecycle-apply success + error paths
|
||||
# ===================================================================
|
||||
|
||||
Scenario: Lifecycle coverage plan lifecycle-apply transitions to apply phase
|
||||
Given lifecycle coverage a plan ready for apply exists
|
||||
When I run lifecycle coverage plan lifecycle-apply with plan ID
|
||||
Then lifecycle coverage plan apply should succeed
|
||||
|
||||
Scenario: Lifecycle coverage plan lifecycle-apply auto-selects single ready plan
|
||||
Given lifecycle coverage a single plan ready for auto-apply exists
|
||||
When I run lifecycle coverage plan lifecycle-apply without ID
|
||||
Then lifecycle coverage plan apply should succeed
|
||||
|
||||
Scenario: Lifecycle coverage plan lifecycle-apply no ready plans aborts
|
||||
Given lifecycle coverage no plans ready for apply
|
||||
When I run lifecycle coverage plan lifecycle-apply without ID
|
||||
Then lifecycle coverage plan apply should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan lifecycle-apply multiple ready plans aborts
|
||||
Given lifecycle coverage multiple plans ready for apply
|
||||
When I run lifecycle coverage plan lifecycle-apply without ID
|
||||
Then lifecycle coverage plan apply should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan lifecycle-apply with json format
|
||||
Given lifecycle coverage a plan ready for apply exists
|
||||
When I run lifecycle coverage plan lifecycle-apply with plan ID and format "json"
|
||||
Then lifecycle coverage plan apply should succeed
|
||||
|
||||
# ===================================================================
|
||||
# Plan cancel success + error paths
|
||||
# ===================================================================
|
||||
|
||||
Scenario: Lifecycle coverage plan cancel with reason
|
||||
Given lifecycle coverage a cancellable plan exists
|
||||
When I run lifecycle coverage plan cancel providing reason "No longer needed"
|
||||
Then lifecycle coverage plan cancel should succeed
|
||||
And lifecycle coverage cancel output should contain "No longer needed"
|
||||
|
||||
Scenario: Lifecycle coverage plan cancel without reason
|
||||
Given lifecycle coverage a cancellable plan exists
|
||||
When I run lifecycle coverage plan cancel without reason
|
||||
Then lifecycle coverage plan cancel should succeed
|
||||
|
||||
Scenario: Lifecycle coverage plan cancel terminal plan aborts
|
||||
Given lifecycle coverage a terminal plan exists for cancel
|
||||
When I run lifecycle coverage plan cancel providing reason "try cancel"
|
||||
Then lifecycle coverage plan cancel should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan cancel with json format
|
||||
Given lifecycle coverage a cancellable plan exists
|
||||
When I run lifecycle coverage plan cancel with reason "cleanup" and format "json"
|
||||
Then lifecycle coverage plan cancel should succeed
|
||||
|
||||
# ===================================================================
|
||||
# Action phase visibility in plan status
|
||||
# ===================================================================
|
||||
|
||||
Scenario: Lifecycle coverage plan status shows action phase as strategize
|
||||
Given lifecycle coverage a plan in strategize phase exists
|
||||
When I run lifecycle coverage plan status with plan ID
|
||||
Then lifecycle coverage plan status should show phase "strategize"
|
||||
|
||||
Scenario: Lifecycle coverage plan status shows action phase as execute
|
||||
Given lifecycle coverage a plan in execute phase exists
|
||||
When I run lifecycle coverage plan status with plan ID
|
||||
Then lifecycle coverage plan status should show phase "execute"
|
||||
|
||||
Scenario: Lifecycle coverage plan status shows action phase as apply
|
||||
Given lifecycle coverage a plan in apply phase exists
|
||||
When I run lifecycle coverage plan status with plan ID
|
||||
Then lifecycle coverage plan status should show phase "apply"
|
||||
|
||||
# ===================================================================
|
||||
# Apply terminal outcomes in plan status output
|
||||
# ===================================================================
|
||||
|
||||
Scenario: Lifecycle coverage plan status shows applied terminal outcome
|
||||
Given lifecycle coverage a plan with applied outcome exists
|
||||
When I run lifecycle coverage plan status with plan ID
|
||||
Then lifecycle coverage plan status should show state "applied"
|
||||
And lifecycle coverage plan should be terminal
|
||||
|
||||
Scenario: Lifecycle coverage plan status shows constrained terminal outcome
|
||||
Given lifecycle coverage a plan with constrained outcome exists
|
||||
When I run lifecycle coverage plan status with plan ID
|
||||
Then lifecycle coverage plan status should show state "constrained"
|
||||
And lifecycle coverage plan should be terminal
|
||||
|
||||
Scenario: Lifecycle coverage plan status shows errored outcome
|
||||
Given lifecycle coverage a plan with errored outcome exists
|
||||
When I run lifecycle coverage plan status with plan ID
|
||||
Then lifecycle coverage plan status should show state "errored"
|
||||
|
||||
Scenario: Lifecycle coverage plan status shows cancelled terminal outcome
|
||||
Given lifecycle coverage a plan with cancelled outcome exists
|
||||
When I run lifecycle coverage plan status with plan ID
|
||||
Then lifecycle coverage plan status should show state "cancelled"
|
||||
And lifecycle coverage plan should be terminal
|
||||
|
||||
# ===================================================================
|
||||
# Negative cases
|
||||
# ===================================================================
|
||||
|
||||
Scenario: Lifecycle coverage action create with empty config file fails
|
||||
Given a lifecycle coverage empty config file
|
||||
When I run lifecycle coverage action create with empty config
|
||||
Then lifecycle coverage action CLI should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan use with unknown action name
|
||||
Given lifecycle coverage unknown action for plan use
|
||||
When I run lifecycle coverage plan use "local/does-not-exist" targeting project "proj-a"
|
||||
Then lifecycle coverage plan use should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan execute with general CleverAgentsError
|
||||
Given lifecycle coverage a plan execute with general error
|
||||
When I run lifecycle coverage plan execute with plan ID
|
||||
Then lifecycle coverage plan execute should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan lifecycle-apply with general error
|
||||
Given lifecycle coverage a plan apply with general error
|
||||
When I run lifecycle coverage plan lifecycle-apply with plan ID
|
||||
Then lifecycle coverage plan apply should abort
|
||||
|
||||
Scenario: Lifecycle coverage plan cancel with general error
|
||||
Given lifecycle coverage a plan cancel with general error
|
||||
When I run lifecycle coverage plan cancel providing reason "err"
|
||||
Then lifecycle coverage plan cancel should abort
|
||||
Reference in New Issue
Block a user