freemo
225eab25b1
ci.yml / fix(cli): change `agents validation attach` extra args to use `--key value` named option format (#3837) (push) Failing after 0s
Co-authored-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me> Co-committed-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me>
307 lines
13 KiB
Gherkin
307 lines
13 KiB
Gherkin
Feature: Tool and Validation CLI commands
|
|
As a developer
|
|
I want to manage tools and validations via CLI commands
|
|
So that I can register, list, show, and remove tools and validations
|
|
|
|
Background:
|
|
Given a tool CLI runner with mocks
|
|
And a mocked tool registry service
|
|
|
|
# Tool add command tests
|
|
Scenario: Add tool via config file
|
|
Given a valid tool config YAML file
|
|
When I run tool CLI add with --config pointing to the YAML file
|
|
Then the tool CLI add should succeed
|
|
And the tool CLI output should contain "Tool Registered"
|
|
|
|
Scenario: Add tool with --update flag when tool exists
|
|
Given a valid tool config YAML file
|
|
And the tool already exists in the registry
|
|
When I run tool CLI add with --config and --update
|
|
Then the tool CLI add should succeed
|
|
And the tool CLI output should contain "Tool Updated"
|
|
|
|
Scenario: Add tool with --update flag when tool does not exist
|
|
Given a valid tool config YAML file
|
|
And the tool does not exist in the registry
|
|
When I run tool CLI add with --config and --update
|
|
Then the tool CLI add should succeed
|
|
And the tool CLI output should contain "Tool Registered"
|
|
|
|
Scenario: Add tool with missing config file fails
|
|
When I run tool CLI add with --config pointing to a missing file
|
|
Then the tool CLI command should abort
|
|
|
|
Scenario: Add tool with invalid YAML fails
|
|
Given an invalid YAML tool config file
|
|
When I run tool CLI add with --config pointing to the invalid YAML file
|
|
Then the tool CLI command should abort
|
|
|
|
Scenario: Add tool with duplicate name fails
|
|
Given a valid tool config YAML file
|
|
And the tool registry raises a duplicate error on register
|
|
When I run tool CLI add with --config pointing to the YAML file
|
|
Then the tool CLI command should abort
|
|
|
|
Scenario: Add tool with json format output
|
|
Given a valid tool config YAML file
|
|
When I run tool CLI add with --config and --format json
|
|
Then the tool CLI add should succeed
|
|
And the tool CLI output should contain "local/test-tool"
|
|
|
|
# Tool list command tests
|
|
Scenario: List all tools
|
|
Given there are mocked tools in the registry
|
|
When I run tool CLI list
|
|
Then the tool CLI list should show all tools
|
|
|
|
Scenario: List tools with namespace filter
|
|
Given there are mocked tools in the registry
|
|
When I run tool CLI list with namespace filter "local"
|
|
Then the tool CLI list should succeed with results
|
|
|
|
Scenario: List tools with source filter
|
|
Given there are mocked tools in the registry
|
|
When I run tool CLI list with source filter "custom"
|
|
Then the tool CLI list should succeed with results
|
|
|
|
Scenario: List tools with type filter
|
|
Given there are mocked tools in the registry
|
|
When I run tool CLI list with type filter "tool"
|
|
Then the tool CLI list should succeed with results
|
|
|
|
Scenario: List tools with type filter validation
|
|
Given there are mocked tools in the registry
|
|
When I run tool CLI list with type filter "validation"
|
|
Then the tool CLI list should succeed with results
|
|
|
|
Scenario: List tools with invalid type filter
|
|
When I run tool CLI list with invalid type "unknown"
|
|
Then the tool CLI command should abort
|
|
|
|
Scenario: List tools with regex filter
|
|
Given there are mocked tools in the registry
|
|
When I run tool CLI list with regex filter "line-.*"
|
|
Then the tool CLI list should succeed with results
|
|
|
|
Scenario: List tools with invalid regex
|
|
When I run tool CLI list with invalid regex "[invalid"
|
|
Then the tool CLI command should abort
|
|
|
|
Scenario: List tools when empty
|
|
Given there are no mocked tools
|
|
When I run tool CLI list
|
|
Then the tool CLI should show no tools message
|
|
|
|
Scenario: List tools with json format
|
|
Given there are mocked tools in the registry
|
|
When I run tool CLI list with format "json"
|
|
Then the tool CLI list should succeed with results
|
|
|
|
Scenario: List tools with yaml format
|
|
Given there are mocked tools in the registry
|
|
When I run tool CLI list with format "yaml"
|
|
Then the tool CLI list should succeed with results
|
|
|
|
Scenario: List tools with plain format
|
|
Given there are mocked tools in the registry
|
|
When I run tool CLI list with format "plain"
|
|
Then the tool CLI list should succeed with results
|
|
|
|
Scenario: List tools with table format
|
|
Given there are mocked tools in the registry
|
|
When I run tool CLI list with format "table"
|
|
Then the tool CLI list should succeed with results
|
|
|
|
# Tool show command tests
|
|
Scenario: Show tool by name
|
|
Given there is a mocked tool with name "local/test-tool"
|
|
When I run tool CLI show with name "local/test-tool"
|
|
Then the tool CLI should show the tool details
|
|
|
|
Scenario: Show tool not found
|
|
Given there is no tool with name "local/missing"
|
|
When I run tool CLI show with name "local/missing"
|
|
Then the tool CLI command should abort
|
|
|
|
Scenario: Show tool with json format
|
|
Given there is a mocked tool with name "local/test-tool"
|
|
When I run tool CLI show for name "local/test-tool" using format "json"
|
|
Then the tool CLI show should succeed
|
|
|
|
Scenario: Show tool with yaml format
|
|
Given there is a mocked tool with name "local/test-tool"
|
|
When I run tool CLI show for name "local/test-tool" using format "yaml"
|
|
Then the tool CLI show should succeed
|
|
|
|
Scenario: Show tool with plain format
|
|
Given there is a mocked tool with name "local/test-tool"
|
|
When I run tool CLI show for name "local/test-tool" using format "plain"
|
|
Then the tool CLI show should succeed
|
|
|
|
# Tool remove command tests
|
|
Scenario: Remove tool by name with confirmation
|
|
Given there is a mocked tool with name "local/test-tool"
|
|
When I run tool CLI remove with name "local/test-tool" and --yes
|
|
Then the tool CLI remove should succeed
|
|
And the tool CLI output should contain "Removed tool"
|
|
|
|
Scenario: Remove tool not found
|
|
Given there is no tool with name "local/missing"
|
|
When I run tool CLI remove with name "local/missing" and --yes
|
|
Then the tool CLI command should abort
|
|
|
|
Scenario: Remove tool with json format
|
|
Given there is a mocked tool with name "local/test-tool"
|
|
When I run tool CLI remove confirmed name "local/test-tool" with fmt "json"
|
|
Then the tool CLI remove should succeed
|
|
|
|
Scenario: Remove tool with service error
|
|
Given there is a mocked tool with name "local/test-tool"
|
|
And the tool registry raises an error on remove
|
|
When I run tool CLI remove with name "local/test-tool" and --yes
|
|
Then the tool CLI command should abort
|
|
|
|
# Validation add command tests
|
|
Scenario: Add validation via config file
|
|
Given a valid validation config YAML file
|
|
When I run validation CLI add with --config pointing to the YAML file
|
|
Then the validation CLI add should succeed
|
|
And the validation CLI output should contain "Validation Registered"
|
|
|
|
Scenario: Add validation with --update flag when validation exists
|
|
Given a valid validation config YAML file
|
|
And the validation already exists in the registry
|
|
When I run validation CLI add with --config and --update
|
|
Then the validation CLI add should succeed
|
|
And the validation CLI output should contain "Validation Updated"
|
|
|
|
Scenario: Add validation with missing config file fails
|
|
When I run validation CLI add with --config pointing to a missing file
|
|
Then the validation CLI command should abort
|
|
|
|
Scenario: Add validation with invalid YAML fails
|
|
Given an invalid YAML validation config file
|
|
When I run validation CLI add with --config pointing to the invalid YAML file
|
|
Then the validation CLI command should abort
|
|
|
|
Scenario: Add validation with json format output
|
|
Given a valid validation config YAML file
|
|
When I run validation CLI add with --config and --format json
|
|
Then the validation CLI add should succeed
|
|
|
|
# Validation attach command tests
|
|
Scenario: Attach validation to resource
|
|
Given a mocked validation exists for attaching
|
|
When I run validation CLI attach "resource/r1" "local/test-val"
|
|
Then the validation CLI attach should succeed
|
|
And the validation CLI output should contain "Attached validation"
|
|
|
|
Scenario: Attach validation with project scope
|
|
Given a mocked validation exists for attaching
|
|
When I run validation CLI attach scoped "resource/r1" "local/test-val" with project "myproj"
|
|
Then the validation CLI attach should succeed
|
|
|
|
Scenario: Attach validation with extra args using named option format
|
|
Given a mocked validation exists for attaching
|
|
When I run validation CLI attach with named option "resource/r1" "local/test-val" option "--threshold" value "80"
|
|
Then the validation CLI attach should succeed
|
|
|
|
Scenario: Attach validation with invalid arg format (not a named option)
|
|
Given a mocked validation exists for attaching
|
|
When I run validation CLI attach with extra args "resource/r1" "local/test-val" arg is "badarg"
|
|
Then the validation CLI command should abort
|
|
|
|
Scenario: Attach validation not found
|
|
Given the validation is not found for attaching
|
|
When I run validation CLI attach "resource/r1" "local/missing-val"
|
|
Then the validation CLI command should abort
|
|
|
|
Scenario: Attach validation with json format
|
|
Given a mocked validation exists for attaching
|
|
When I run validation CLI attach formatted "resource/r1" "local/test-val" fmt is "json"
|
|
Then the validation CLI attach should succeed
|
|
|
|
# Validation detach command tests
|
|
Scenario: Detach validation with confirmation
|
|
Given a mocked attachment exists for detaching
|
|
When I run validation CLI detach "att-123" with --yes
|
|
Then the validation CLI detach should succeed
|
|
And the validation CLI output should contain "Detached validation"
|
|
|
|
Scenario: Detach validation not found
|
|
Given the attachment is not found for detaching
|
|
When I run validation CLI detach "att-missing" with --yes
|
|
Then the validation CLI command should abort
|
|
|
|
Scenario: Detach validation with json format
|
|
Given a mocked attachment exists for detaching
|
|
When I run validation CLI detach formatted "att-123" confirmed fmt "json"
|
|
Then the validation CLI detach should succeed
|
|
|
|
# Tool add with CleverAgentsError
|
|
Scenario: Add tool with CleverAgentsError from service
|
|
Given a valid tool config YAML file
|
|
And the tool registry raises a CleverAgentsError on register
|
|
When I run tool CLI add with --config pointing to the YAML file
|
|
Then the tool CLI command should abort
|
|
|
|
# Tool list with CleverAgentsError
|
|
Scenario: List tools with CleverAgentsError from service
|
|
Given the tool registry raises a CleverAgentsError on list
|
|
When I run tool CLI list
|
|
Then the tool CLI command should abort
|
|
|
|
# Tool show with CleverAgentsError
|
|
Scenario: Show tool with CleverAgentsError from service
|
|
Given the tool registry raises a CleverAgentsError on get
|
|
When I run tool CLI show with name "local/test-tool"
|
|
Then the tool CLI command should abort
|
|
|
|
# Validation add with CleverAgentsError
|
|
Scenario: Add validation with CleverAgentsError from service
|
|
Given a valid validation config YAML file
|
|
And the validation registry raises a CleverAgentsError on register
|
|
When I run validation CLI add with --config pointing to the YAML file
|
|
Then the validation CLI command should abort
|
|
|
|
# Validation attach with CleverAgentsError
|
|
Scenario: Attach validation with CleverAgentsError from service
|
|
Given the validation registry raises a CleverAgentsError on attach
|
|
When I run validation CLI attach "resource/r1" "local/test-val"
|
|
Then the validation CLI command should abort
|
|
|
|
# Validation detach with CleverAgentsError
|
|
Scenario: Detach validation with CleverAgentsError from service
|
|
Given the validation registry raises a CleverAgentsError on detach
|
|
When I run validation CLI detach "att-123" with --yes
|
|
Then the validation CLI command should abort
|
|
|
|
# Validation add with update and non-existent
|
|
Scenario: Add validation with --update flag when validation does not exist
|
|
Given a valid validation config YAML file
|
|
And the validation does not exist in the registry
|
|
When I run validation CLI add with --config and --update
|
|
Then the validation CLI add should succeed
|
|
And the validation CLI output should contain "Validation Registered"
|
|
|
|
# Tool remove with remove returning false
|
|
Scenario: Remove tool with remove returning false
|
|
Given there is a mocked tool with name "local/fail-remove"
|
|
And the tool registry returns false on remove
|
|
When I run tool CLI remove with name "local/fail-remove" and --yes
|
|
Then the tool CLI command should abort
|
|
|
|
# Validation add with validation error
|
|
Scenario: Add validation with validation error from service
|
|
Given a valid validation config YAML file
|
|
And the validation registry raises a validation error on register
|
|
When I run validation CLI add with --config pointing to the YAML file
|
|
Then the validation CLI command should abort
|
|
|
|
# Validation attach with validation error
|
|
Scenario: Attach validation with validation error from service
|
|
Given the validation registry raises a validation error on attach
|
|
When I run validation CLI attach "resource/r1" "local/test-val"
|
|
Then the validation CLI command should abort
|