HAL9000
355d3b7839
Refactor cli_lifecycle_e2e tests to address all review feedback from cycles 1-2: - Split the 752-line cli_lifecycle_e2e_steps.py into 4 modular step files (each under 500 lines): . cli_lifecycle_e2e_setup_steps.py: context init, mock env, action setup (80 lines) . cli_lifecycle_e2e_plan_use_steps.py: plan use step definitions (290 lines) . cli_lifecycle_e2e_plan_execute_steps.py: plan execute/apply steps (168 lines) . cli_lifecycle_e2e_assertions_steps.py: all assertions (286 lines) - Fix tautological phase assertions: step definitions now verify actual CLI output (context.last_output) instead of creating new Plan objects with hardcoded values - Fix JSON envelope validation: validates all spec-required keys (command, status, exit_code, data, timing, messages) instead of checking only status string presence - Add missing plan list invocations: each plan list assertion is now preceded by 'And cli_lifecycle_e2e plan list' to run the plan list command - Add missing plan status invocations: each plan status assertion is now preceded by 'And cli_lifecycle_e2e plan status with the created plan ID' to run status command - Fix os.environ cleanup: mock LLM actors env var tracked and restored after scenario - Updated feature file with 824→823 lines of Gherkin, adding When-steps for plan status and plan list invocations in scenarios 1, 3, and 4 Closes #9459
147 lines
8.4 KiB
Gherkin
147 lines
8.4 KiB
Gherkin
Feature: CLI lifecycle e2e full plan lifecycle integration test
|
|
As a developer
|
|
I want to verify the complete plan lifecycle workflow
|
|
So that plan use → plan execute → plan apply works end-to-end
|
|
|
|
Background:
|
|
Given a cli_lifecycle_e2e CLI runner
|
|
And a cli_lifecycle_e2e mocked lifecycle service
|
|
And cli_lifecycle_e2e mock LLM actors enabled
|
|
|
|
# ===================================================================
|
|
# Full lifecycle: plan use → plan execute → plan apply
|
|
# ===================================================================
|
|
|
|
Scenario: Full lifecycle plan use → execute → apply succeeds end-to-end
|
|
Given a cli_lifecycle_e2e action for full lifecycle exists
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" targeting project "proj-e2e"
|
|
Then cli_lifecycle_e2e plan use should succeed
|
|
And cli_lifecycle_e2e plan should be in strategize phase
|
|
And cli_lifecycle_e2e plan status with the created plan ID
|
|
And cli_lifecycle_e2e plan status should show state "queued"
|
|
When I run cli_lifecycle_e2e plan execute with the created plan ID
|
|
Then cli_lifecycle_e2e plan execute should succeed
|
|
And cli_lifecycle_e2e plan should be in execute phase
|
|
And cli_lifecycle_e2e plan status with the created plan ID
|
|
And cli_lifecycle_e2e plan status should show state "complete"
|
|
When I run cli_lifecycle_e2e plan apply with the created plan ID
|
|
Then cli_lifecycle_e2e plan apply should succeed
|
|
And cli_lifecycle_e2e plan should be in apply phase
|
|
And cli_lifecycle_e2e plan status with the created plan ID
|
|
And cli_lifecycle_e2e plan status should show state "applied"
|
|
|
|
Scenario: Full lifecycle with json format at each step returns spec envelopes
|
|
Given a cli_lifecycle_e2e action for full lifecycle exists
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" on project "proj-e2e" with format "json"
|
|
Then cli_lifecycle_e2e plan use should succeed
|
|
And cli_lifecycle_e2e json output should contain envelope with status "queued"
|
|
When I run cli_lifecycle_e2e plan execute with the created plan ID and format "json"
|
|
Then cli_lifecycle_e2e plan execute should succeed
|
|
And cli_lifecycle_e2e json output should contain envelope with status "complete"
|
|
When I run cli_lifecycle_e2e plan apply with the created plan ID and format "json"
|
|
Then cli_lifecycle_e2e plan apply should succeed
|
|
And cli_lifecycle_e2e json output should contain envelope with status "applied"
|
|
|
|
Scenario: Status transitions are correct at each phase
|
|
Given a cli_lifecycle_e2e action for full lifecycle exists
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" targeting project "proj-e2e"
|
|
Then cli_lifecycle_e2e plan should be in strategize phase
|
|
And cli_lifecycle_e2e plan status with the created plan ID
|
|
And cli_lifecycle_e2e plan status should show phase "strategize"
|
|
And cli_lifecycle_e2e plan status should show state "queued"
|
|
When I run cli_lifecycle_e2e plan execute with the created plan ID
|
|
Then cli_lifecycle_e2e plan should be in execute phase
|
|
And cli_lifecycle_e2e plan status with the created plan ID
|
|
And cli_lifecycle_e2e plan status should show phase "execute"
|
|
And cli_lifecycle_e2e plan status should show state "complete"
|
|
When I run cli_lifecycle_e2e plan apply with the created plan ID
|
|
Then cli_lifecycle_e2e plan should be in apply phase
|
|
And cli_lifecycle_e2e plan status with the created plan ID
|
|
And cli_lifecycle_e2e plan status should show phase "apply"
|
|
And cli_lifecycle_e2e plan status should show state "applied"
|
|
|
|
Scenario: Plan list shows plan at each lifecycle stage with correct phase/state
|
|
Given a cli_lifecycle_e2e action for full lifecycle exists
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" targeting project "proj-e2e"
|
|
And cli_lifecycle_e2e plan list
|
|
Then cli_lifecycle_e2e plan list should show 1 plan in strategize phase
|
|
When I run cli_lifecycle_e2e plan execute with the created plan ID
|
|
And cli_lifecycle_e2e plan list
|
|
Then cli_lifecycle_e2e plan list should show 1 plan in execute phase
|
|
When I run cli_lifecycle_e2e plan apply with the created plan ID
|
|
And cli_lifecycle_e2e plan list
|
|
Then cli_lifecycle_e2e plan list should show 1 plan in apply phase
|
|
|
|
Scenario: Full lifecycle with multiple projects
|
|
Given a cli_lifecycle_e2e action for full lifecycle exists
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" with projects "proj-a" "proj-b" "proj-c"
|
|
Then cli_lifecycle_e2e plan use should succeed
|
|
And cli_lifecycle_e2e plan should link 3 projects
|
|
When I run cli_lifecycle_e2e plan execute with the created plan ID
|
|
Then cli_lifecycle_e2e plan execute should succeed
|
|
When I run cli_lifecycle_e2e plan apply with the created plan ID
|
|
Then cli_lifecycle_e2e plan apply should succeed
|
|
|
|
Scenario: Full lifecycle with automation profile override
|
|
Given a cli_lifecycle_e2e action for full lifecycle exists
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" on project "proj-e2e" with automation profile "cautious"
|
|
Then cli_lifecycle_e2e plan use should succeed
|
|
And cli_lifecycle_e2e plan automation profile should be "cautious"
|
|
When I run cli_lifecycle_e2e plan execute with the created plan ID
|
|
Then cli_lifecycle_e2e plan execute should succeed
|
|
When I run cli_lifecycle_e2e plan apply with the created plan ID
|
|
Then cli_lifecycle_e2e plan apply should succeed
|
|
|
|
Scenario: Full lifecycle with invariants
|
|
Given a cli_lifecycle_e2e action for full lifecycle exists
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" on project "proj-e2e" with invariants "No warnings" and "Keep API compat"
|
|
Then cli_lifecycle_e2e plan use should succeed
|
|
And cli_lifecycle_e2e plan should have 2 invariants
|
|
When I run cli_lifecycle_e2e plan execute with the created plan ID
|
|
Then cli_lifecycle_e2e plan execute should succeed
|
|
When I run cli_lifecycle_e2e plan apply with the created plan ID
|
|
Then cli_lifecycle_e2e plan apply should succeed
|
|
|
|
Scenario: Full lifecycle with custom strategy actor
|
|
Given a cli_lifecycle_e2e action for full lifecycle exists
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" on project "proj-e2e" with strategy actor "anthropic/claude-3"
|
|
Then cli_lifecycle_e2e plan use should succeed
|
|
And cli_lifecycle_e2e plan strategy actor should be "anthropic/claude-3"
|
|
When I run cli_lifecycle_e2e plan execute with the created plan ID
|
|
Then cli_lifecycle_e2e plan execute should succeed
|
|
When I run cli_lifecycle_e2e plan apply with the created plan ID
|
|
Then cli_lifecycle_e2e plan apply should succeed
|
|
|
|
Scenario: Full lifecycle with custom execution actor
|
|
Given a cli_lifecycle_e2e action for full lifecycle exists
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" on project "proj-e2e" with execution actor "openai/gpt-4o"
|
|
Then cli_lifecycle_e2e plan use should succeed
|
|
And cli_lifecycle_e2e plan execution actor should be "openai/gpt-4o"
|
|
When I run cli_lifecycle_e2e plan execute with the created plan ID
|
|
Then cli_lifecycle_e2e plan execute should succeed
|
|
When I run cli_lifecycle_e2e plan apply with the created plan ID
|
|
Then cli_lifecycle_e2e plan apply should succeed
|
|
|
|
# ===================================================================
|
|
# Error handling in full lifecycle
|
|
# ===================================================================
|
|
|
|
Scenario: Full lifecycle fails if plan use fails
|
|
Given a cli_lifecycle_e2e action with validation error for plan use
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" targeting project "proj-e2e"
|
|
Then cli_lifecycle_e2e plan use should abort
|
|
|
|
Scenario: Full lifecycle fails if execute fails on invalid plan state
|
|
Given a cli_lifecycle_e2e action for full lifecycle exists
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" targeting project "proj-e2e"
|
|
Then cli_lifecycle_e2e plan use should succeed
|
|
When I run cli_lifecycle_e2e plan execute with the created plan ID twice
|
|
Then cli_lifecycle_e2e second plan execute should abort
|
|
|
|
Scenario: Full lifecycle fails if apply fails on invalid plan state
|
|
Given a cli_lifecycle_e2e action for full lifecycle exists
|
|
When I run cli_lifecycle_e2e plan use "local/e2e-action" targeting project "proj-e2e"
|
|
Then cli_lifecycle_e2e plan use should succeed
|
|
When I run cli_lifecycle_e2e plan apply with the created plan ID
|
|
Then cli_lifecycle_e2e plan apply should abort
|