freemo
48cff5cfe0
CI / build (push) Successful in 18s
CI / lint (push) Failing after 31s
CI / helm (push) Successful in 33s
CI / typecheck (push) Successful in 50s
CI / security (push) Failing after 51s
CI / coverage (push) Has been skipped
CI / benchmark-regression (push) Has been skipped
CI / unit_tests (push) Failing after 1m50s
CI / docker (push) Has been skipped
CI / quality (push) Successful in 3m43s
CI / integration_tests (push) Has been cancelled
CI / e2e_tests (push) Has been cancelled
CI / benchmark-publish (push) Has been cancelled
CI / status-check (push) Has been cancelled
Renames `plan lifecycle-list` to `plan list` and `plan lifecycle-apply` to `plan apply` to align with the specification's canonical command names. Removes legacy V2 plan commands that occupied those names. - Renamed CLI command registrations from lifecycle-list/lifecycle-apply to list/apply - Removed legacy V2 apply and list commands (~200 lines) - Updated apply shortcut in main.py to delegate to v3 lifecycle - Added defensive null check for plan existence in apply command - Updated 63+ test, doc, and benchmark files for consistency Closes #881 Co-authored-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me> Co-committed-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me>
364 lines
17 KiB
Gherkin
364 lines
17 KiB
Gherkin
Feature: CLI extensions for plan and action commands
|
|
As a developer using the CleverAgents CLI
|
|
I want automation profile, invariant, and actor override flags
|
|
So that I can customize plan creation and view extended details
|
|
|
|
Background:
|
|
Given a cli extensions test runner
|
|
And a cli extensions mocked lifecycle service
|
|
|
|
# --- plan use: automation profile ---
|
|
|
|
Scenario: Plan use with --automation-profile persists to plan metadata
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "trusted"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "trusted"
|
|
|
|
Scenario: Plan use with invalid automation profile is rejected
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "nonexistent-profile"
|
|
Then the cli extensions plan use should fail
|
|
|
|
# --- plan use: invariant flags ---
|
|
|
|
Scenario: Plan use with --invariant flags persists to plan metadata
|
|
Given a cli extensions action exists
|
|
When I run plan use with invariant flags "No warnings" and "Keep compat"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions service received invariants "No warnings" and "Keep compat"
|
|
|
|
Scenario: Plan use with single invariant flag
|
|
Given a cli extensions action exists
|
|
When I run plan use with single invariant "All tests must pass"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions service received single invariant "All tests must pass"
|
|
|
|
# --- plan use: actor override validation (valid namespaced) ---
|
|
|
|
Scenario: Plan use with valid namespaced strategy-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "openai/gpt-4"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has strategy actor "openai/gpt-4"
|
|
|
|
Scenario: Plan use with valid namespaced execution-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with execution actor flag "anthropic/claude-3"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has execution actor "anthropic/claude-3"
|
|
|
|
Scenario: Plan use with valid namespaced estimation-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with estimation actor flag "openai/gpt-4"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has estimation actor "openai/gpt-4"
|
|
|
|
Scenario: Plan use with valid namespaced invariant-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with invariant actor flag "openai/gpt-4"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has invariant actor "openai/gpt-4"
|
|
|
|
# --- plan use: actor override validation (invalid format) ---
|
|
|
|
Scenario: Plan use rejects invalid strategy-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "bad-format"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects invalid execution-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with execution actor flag "no_slash"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects invalid estimation-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with estimation actor flag "UPPERCASE/gpt"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects invalid invariant-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with invariant actor flag "spaces are bad"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
# --- plan status: showing automation_profile ---
|
|
|
|
Scenario: Plan status shows automation profile in table
|
|
Given cli extensions plans exist with automation profile "trusted"
|
|
When I run cli extensions plan status
|
|
Then the cli extensions status output should contain "trusted"
|
|
|
|
Scenario: Plan status shows automation profile in JSON
|
|
Given cli extensions plans exist with automation profile "manual"
|
|
When I run cli extensions plan status with format "json"
|
|
Then the cli extensions status json output should contain automation profile "manual"
|
|
|
|
# --- plan list: showing invariants when present ---
|
|
|
|
Scenario: Plan list shows invariant count in table
|
|
Given cli extensions plans exist with invariants
|
|
When I run cli extensions plan list
|
|
Then the cli extensions list output should contain invariant count
|
|
|
|
Scenario: Plan list shows invariants in JSON
|
|
Given cli extensions plans exist with invariants
|
|
When I run cli extensions plan list with format "json"
|
|
Then the cli extensions list json output should contain invariants
|
|
|
|
# --- action show: optional actors ---
|
|
|
|
Scenario: Action show displays estimation actor when present
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Estimation Actor"
|
|
And the cli extensions action output should contain "openai/gpt-4"
|
|
|
|
Scenario: Action show displays invariant actor when present
|
|
Given a cli extensions action with invariant actor "anthropic/claude-3"
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Invariant Actor"
|
|
And the cli extensions action output should contain "anthropic/claude-3"
|
|
|
|
Scenario: Action show displays invariants when present
|
|
Given a cli extensions action with invariants
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Invariants"
|
|
|
|
Scenario: Action show displays inputs schema when present
|
|
Given a cli extensions action with inputs schema
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Inputs Schema"
|
|
|
|
Scenario: Action show in JSON includes optional actors
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions action json should contain "estimation_actor"
|
|
|
|
Scenario: Action show in JSON includes inputs schema
|
|
Given a cli extensions action with inputs schema
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions action json should contain "inputs_schema"
|
|
|
|
# --- plan use: automation_profile + invariants combined ---
|
|
|
|
Scenario: Plan use with automation profile and invariants together
|
|
Given a cli extensions action exists
|
|
When I run plan use with profile "trusted" and invariant "No regressions"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "trusted"
|
|
And the cli extensions service received single invariant "No regressions"
|
|
|
|
# --- plan status: single plan view with invariants ---
|
|
|
|
Scenario: Plan status single plan view renders invariants in rich panel
|
|
Given cli extensions single plan exists with invariants
|
|
When I run cli extensions plan status for single plan
|
|
Then the cli extensions status output should contain "Invariants"
|
|
And the cli extensions status output should contain "No warnings"
|
|
And the cli extensions status output should contain "plan"
|
|
|
|
Scenario: Plan status single plan view renders automation profile in rich panel
|
|
Given cli extensions single plan exists with profile "trusted"
|
|
When I run cli extensions plan status for single plan
|
|
Then the cli extensions status output should contain "Automation Profile"
|
|
And the cli extensions status output should contain "trusted"
|
|
|
|
# --- action show: automation profile display ---
|
|
|
|
Scenario: Action show displays automation profile when present
|
|
Given a cli extensions action with automation profile "trusted"
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Automation Profile"
|
|
And the cli extensions action output should contain "trusted"
|
|
|
|
Scenario: Action show in JSON without invariant actor omits field
|
|
Given a cli extensions action without invariant actor
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions action json should not contain "invariant_actor"
|
|
|
|
# --- validate_namespaced_actor function ---
|
|
|
|
Scenario: validate_namespaced_actor accepts valid names
|
|
Then validate_namespaced_actor accepts "openai/gpt-4"
|
|
And validate_namespaced_actor accepts "anthropic/claude-3"
|
|
And validate_namespaced_actor accepts "local/my-actor"
|
|
|
|
Scenario: validate_namespaced_actor rejects invalid names
|
|
Then validate_namespaced_actor rejects "no-slash"
|
|
And validate_namespaced_actor rejects "UPPER/case"
|
|
And validate_namespaced_actor rejects "/leading-slash"
|
|
And validate_namespaced_actor rejects "trail/ing/"
|
|
|
|
# ==========================================================================
|
|
# DEEP COVERAGE: Automation profile resolution logic (#326)
|
|
# ==========================================================================
|
|
|
|
Scenario: Plan use resolves valid builtin profile "manual"
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "manual"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "manual"
|
|
|
|
Scenario: Plan use resolves valid builtin profile "review"
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "review"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "review"
|
|
|
|
Scenario: Plan use resolves valid builtin profile "supervised"
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "supervised"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "supervised"
|
|
|
|
Scenario: Plan use resolves valid builtin profile "cautious"
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "cautious"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "cautious"
|
|
|
|
Scenario: Plan use resolves valid builtin profile "auto"
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "auto"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "auto"
|
|
|
|
Scenario: Plan use rejects profile name with special characters
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "tr@sted!"
|
|
Then the cli extensions plan use should fail
|
|
|
|
Scenario: Plan use rejects profile name with spaces
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "my profile"
|
|
Then the cli extensions plan use should fail
|
|
|
|
Scenario: Plan use profile resolution provides available profiles on error
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "nonexistent"
|
|
Then the cli extensions plan use should fail
|
|
And the cli extensions error output mentions available profiles
|
|
|
|
# ==========================================================================
|
|
# DEEP COVERAGE: Invariant ordering preservation (#326)
|
|
# ==========================================================================
|
|
|
|
Scenario: Plan use with three invariants preserves insertion order
|
|
Given a cli extensions action exists
|
|
When I run plan use with three invariants "Alpha first" and "Beta second" and "Gamma third"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions service received invariants in order "Alpha first" then "Beta second" then "Gamma third"
|
|
|
|
Scenario: Plan use with five invariants preserves insertion order
|
|
Given a cli extensions action exists
|
|
When I run plan use with five invariants "A" and "B" and "C" and "D" and "E"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions service received five invariants in exact order "A" "B" "C" "D" "E"
|
|
|
|
# ==========================================================================
|
|
# DEEP COVERAGE: Actor override error cases (#326)
|
|
# ==========================================================================
|
|
|
|
Scenario: Plan use rejects actor with double slashes
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "open//ai"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects actor with special characters in namespace
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "open@ai/gpt-4"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects actor with numeric-start namespace
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "123/model"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects actor with only a slash
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "/"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects actor with whitespace
|
|
Given a cli extensions action exists
|
|
When I run plan use with execution actor flag "open ai/gpt"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects actor with missing name after slash
|
|
Given a cli extensions action exists
|
|
When I run plan use with estimation actor flag "openai/"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: validate_namespaced_actor rejects double slash
|
|
Then validate_namespaced_actor rejects "ns//name"
|
|
|
|
Scenario: validate_namespaced_actor rejects special characters
|
|
Then validate_namespaced_actor rejects "ns@special/name"
|
|
And validate_namespaced_actor rejects "ns/name#bad"
|
|
|
|
Scenario: validate_namespaced_actor rejects numeric-leading namespace
|
|
Then validate_namespaced_actor rejects "1abc/model"
|
|
|
|
# ==========================================================================
|
|
# DEEP COVERAGE: Output snapshot assertions JSON/YAML/table (#326)
|
|
# ==========================================================================
|
|
|
|
Scenario: Plan status in JSON format contains required keys
|
|
Given cli extensions plans exist with automation profile "trusted"
|
|
When I run cli extensions plan status with format "json"
|
|
Then the cli extensions output is valid JSON
|
|
And the cli extensions json output contains key "automation_profile"
|
|
And the cli extensions json output contains key "namespaced_name"
|
|
And the cli extensions json output contains key "phase"
|
|
|
|
Scenario: Plan status in YAML format contains required fields
|
|
Given cli extensions plans exist with automation profile "trusted"
|
|
When I run cli extensions plan status with format "yaml"
|
|
Then the cli extensions output contains yaml field "automation_profile"
|
|
And the cli extensions output contains yaml field "namespaced_name"
|
|
|
|
Scenario: Action show in JSON format contains all required keys
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions output is valid JSON
|
|
And the cli extensions json output contains key "estimation_actor"
|
|
And the cli extensions json output contains key "strategy_actor"
|
|
And the cli extensions json output contains key "execution_actor"
|
|
And the cli extensions json output contains key "description"
|
|
|
|
Scenario: Action show in YAML format contains required fields
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show with format "yaml"
|
|
Then the cli extensions output contains yaml field "estimation_actor"
|
|
And the cli extensions output contains yaml field "strategy_actor"
|
|
|
|
Scenario: Action show in JSON includes invariants list when present
|
|
Given a cli extensions action with invariants
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions output is valid JSON
|
|
And the cli extensions json output contains key "invariants"
|
|
And the cli extensions json invariants list has length 2
|
|
|
|
Scenario: Plan list in YAML shows invariant data
|
|
Given cli extensions plans exist with invariants
|
|
When I run cli extensions plan list with format "yaml"
|
|
Then the cli extensions output contains yaml field "invariants"
|
|
|
|
Scenario: Plan list in JSON preserves invariant text
|
|
Given cli extensions plans exist with invariants
|
|
When I run cli extensions plan list with format "json"
|
|
Then the cli extensions output is valid JSON
|
|
And the cli extensions json output string contains "No warnings"
|
|
And the cli extensions json output string contains "Keep compat"
|
|
|
|
Scenario: Action show in table format contains actor information
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show with format "table"
|
|
Then the cli extensions action output should contain "openai/gpt-4"
|
|
|
|
Scenario: Plan status in table format contains profile name
|
|
Given cli extensions plans exist with automation profile "cautious"
|
|
When I run cli extensions plan status
|
|
Then the cli extensions status output should contain "cautious"
|