freemo
f69224e7e9
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Failing after 15s
CI / quality (pull_request) Successful in 17s
CI / build (pull_request) Successful in 17s
CI / security (pull_request) Successful in 29s
CI / typecheck (pull_request) Successful in 31s
CI / coverage (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Has been skipped
CI / integration_tests (pull_request) Successful in 3m44s
CI / unit_tests (pull_request) Successful in 7m4s
CI / docker (pull_request) Has been skipped
164 lines
7.4 KiB
Gherkin
164 lines
7.4 KiB
Gherkin
Feature: CLI extensions for plan and action commands
|
|
As a developer using the CleverAgents CLI
|
|
I want automation profile, invariant, and actor override flags
|
|
So that I can customize plan creation and view extended details
|
|
|
|
Background:
|
|
Given a cli extensions test runner
|
|
And a cli extensions mocked lifecycle service
|
|
|
|
# --- plan use: automation profile ---
|
|
|
|
Scenario: Plan use with --automation-profile persists to plan metadata
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "trusted"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "trusted"
|
|
|
|
Scenario: Plan use with invalid automation profile is rejected
|
|
Given a cli extensions action exists
|
|
When I run plan use with automation profile flag "nonexistent-profile"
|
|
Then the cli extensions plan use should fail
|
|
|
|
# --- plan use: invariant flags ---
|
|
|
|
Scenario: Plan use with --invariant flags persists to plan metadata
|
|
Given a cli extensions action exists
|
|
When I run plan use with invariant flags "No warnings" and "Keep compat"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions service received invariants "No warnings" and "Keep compat"
|
|
|
|
Scenario: Plan use with single invariant flag
|
|
Given a cli extensions action exists
|
|
When I run plan use with single invariant "All tests must pass"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions service received single invariant "All tests must pass"
|
|
|
|
# --- plan use: actor override validation (valid namespaced) ---
|
|
|
|
Scenario: Plan use with valid namespaced strategy-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "openai/gpt-4"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has strategy actor "openai/gpt-4"
|
|
|
|
Scenario: Plan use with valid namespaced execution-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with execution actor flag "anthropic/claude-3"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has execution actor "anthropic/claude-3"
|
|
|
|
Scenario: Plan use with valid namespaced estimation-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with estimation actor flag "openai/gpt-4"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has estimation actor "openai/gpt-4"
|
|
|
|
Scenario: Plan use with valid namespaced invariant-actor
|
|
Given a cli extensions action exists
|
|
When I run plan use with invariant actor flag "openai/gpt-4"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan has invariant actor "openai/gpt-4"
|
|
|
|
# --- plan use: actor override validation (invalid format) ---
|
|
|
|
Scenario: Plan use rejects invalid strategy-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with strategy actor flag "bad-format"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects invalid execution-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with execution actor flag "no_slash"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects invalid estimation-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with estimation actor flag "UPPERCASE/gpt"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
Scenario: Plan use rejects invalid invariant-actor format
|
|
Given a cli extensions action exists
|
|
When I run plan use with invariant actor flag "spaces are bad"
|
|
Then the cli extensions plan use should fail with actor validation error
|
|
|
|
# --- plan status: showing automation_profile ---
|
|
|
|
Scenario: Plan status shows automation profile in table
|
|
Given cli extensions plans exist with automation profile "trusted"
|
|
When I run cli extensions plan status
|
|
Then the cli extensions status output should contain "trusted"
|
|
|
|
Scenario: Plan status shows automation profile in JSON
|
|
Given cli extensions plans exist with automation profile "manual"
|
|
When I run cli extensions plan status with format "json"
|
|
Then the cli extensions status json output should contain automation profile "manual"
|
|
|
|
# --- plan list: showing invariants when present ---
|
|
|
|
Scenario: Plan lifecycle-list shows invariant count in table
|
|
Given cli extensions plans exist with invariants
|
|
When I run cli extensions plan lifecycle-list
|
|
Then the cli extensions list output should contain invariant count
|
|
|
|
Scenario: Plan lifecycle-list shows invariants in JSON
|
|
Given cli extensions plans exist with invariants
|
|
When I run cli extensions plan lifecycle-list with format "json"
|
|
Then the cli extensions list json output should contain invariants
|
|
|
|
# --- action show: optional actors ---
|
|
|
|
Scenario: Action show displays estimation actor when present
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Estimation Actor"
|
|
And the cli extensions action output should contain "openai/gpt-4"
|
|
|
|
Scenario: Action show displays invariant actor when present
|
|
Given a cli extensions action with invariant actor "anthropic/claude-3"
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Invariant Actor"
|
|
And the cli extensions action output should contain "anthropic/claude-3"
|
|
|
|
Scenario: Action show displays invariants when present
|
|
Given a cli extensions action with invariants
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Invariants"
|
|
|
|
Scenario: Action show displays inputs schema when present
|
|
Given a cli extensions action with inputs schema
|
|
When I run cli extensions action show
|
|
Then the cli extensions action output should contain "Inputs Schema"
|
|
|
|
Scenario: Action show in JSON includes optional actors
|
|
Given a cli extensions action with estimation actor "openai/gpt-4"
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions action json should contain "estimation_actor"
|
|
|
|
Scenario: Action show in JSON includes inputs schema
|
|
Given a cli extensions action with inputs schema
|
|
When I run cli extensions action show with format "json"
|
|
Then the cli extensions action json should contain "inputs_schema"
|
|
|
|
# --- plan use: automation_profile + invariants combined ---
|
|
|
|
Scenario: Plan use with automation profile and invariants together
|
|
Given a cli extensions action exists
|
|
When I run plan use with profile "trusted" and invariant "No regressions"
|
|
Then the cli extensions plan use should succeed
|
|
And the cli extensions plan should have automation profile "trusted"
|
|
And the cli extensions service received single invariant "No regressions"
|
|
|
|
# --- validate_namespaced_actor function ---
|
|
|
|
Scenario: validate_namespaced_actor accepts valid names
|
|
Then validate_namespaced_actor accepts "openai/gpt-4"
|
|
And validate_namespaced_actor accepts "anthropic/claude-3"
|
|
And validate_namespaced_actor accepts "local/my-actor"
|
|
|
|
Scenario: validate_namespaced_actor rejects invalid names
|
|
Then validate_namespaced_actor rejects "no-slash"
|
|
And validate_namespaced_actor rejects "UPPER/case"
|
|
And validate_namespaced_actor rejects "/leading-slash"
|
|
And validate_namespaced_actor rejects "trail/ing/"
|