255 lines
8.6 KiB
Gherkin
255 lines
8.6 KiB
Gherkin
Feature: Comprehensive CLI Testing
|
|
As a user of CleverAgents
|
|
I want robust CLI functionality with proper error handling
|
|
So that I can confidently use the command-line interface in production
|
|
|
|
Background:
|
|
Given I have a working directory for CLI tests
|
|
|
|
Scenario Outline: CLI argument parsing and validation
|
|
When I run "cleveragents <command> <args>"
|
|
Then the command should <result>
|
|
And the exit code should be <exit_code>
|
|
|
|
Examples:
|
|
| command | args | result | exit_code |
|
|
| run | --help | succeed | 0 |
|
|
| interactive | --help | succeed | 0 |
|
|
| visualize | --help | succeed | 0 |
|
|
| generate-examples| --help | succeed | 0 |
|
|
| invalid-command | | fail | 2 |
|
|
| run | -c nonexistent.yaml | fail | 2 |
|
|
| run | -c /dev/null -p test | fail | 1 |
|
|
|
|
Scenario: Complex configuration file handling
|
|
Given I have a configuration file "complex.yaml":
|
|
"""
|
|
agents:
|
|
echo_agent:
|
|
type: tool
|
|
config:
|
|
tools: ["echo"]
|
|
safe_mode: false
|
|
timeout: 2
|
|
|
|
math_agent:
|
|
type: tool
|
|
config:
|
|
tools: ["math", "json_parse"]
|
|
safe_mode: false
|
|
timeout: 2
|
|
|
|
routes:
|
|
preprocessing:
|
|
type: stream
|
|
stream_type: cold
|
|
operators:
|
|
- type: map
|
|
params:
|
|
agent: echo_agent
|
|
- type: filter
|
|
params:
|
|
condition:
|
|
type: content_length
|
|
min: 5
|
|
publications:
|
|
- main_processing
|
|
|
|
main_processing:
|
|
type: stream
|
|
stream_type: hot
|
|
operators:
|
|
- type: map
|
|
params:
|
|
agent: math_agent
|
|
- type: buffer
|
|
params:
|
|
count: 2
|
|
- type: debounce
|
|
params:
|
|
duration: 0.5
|
|
publications:
|
|
- __output__
|
|
|
|
merges:
|
|
- sources: [__input__]
|
|
target: preprocessing
|
|
|
|
splits:
|
|
- source: main_processing
|
|
targets:
|
|
error_handler:
|
|
type: metadata_has
|
|
key: error
|
|
"""
|
|
And I set environment variable "ENV_VAR" to "test_value"
|
|
When I run "cleveragents run -c complex.yaml -p 'Complex test message for processing' --verbose"
|
|
Then the command should succeed
|
|
And the output should include debug information
|
|
And it should show stream processing details
|
|
|
|
Scenario: Interactive session with history management
|
|
Given I have a configuration file "interactive.yaml":
|
|
"""
|
|
agents:
|
|
chat_agent:
|
|
type: llm
|
|
config:
|
|
provider: openai
|
|
model: gpt-3.5-turbo
|
|
system_prompt: "You are a helpful assistant"
|
|
|
|
routes:
|
|
chat_stream:
|
|
type: stream
|
|
stream_type: cold
|
|
operators:
|
|
- type: map
|
|
params:
|
|
agent: chat_agent
|
|
publications:
|
|
- __output__
|
|
|
|
merges:
|
|
- sources: [__input__]
|
|
target: chat_stream
|
|
"""
|
|
And I have a history file "chat_history.json" with content:
|
|
"""
|
|
[
|
|
{"role": "user", "content": "Hello"},
|
|
{"role": "assistant", "content": "Hi there! How can I help you?"}
|
|
]
|
|
"""
|
|
When I start "cleveragents interactive -c interactive.yaml -h chat_history.json --verbose"
|
|
Then the interactive session should start
|
|
And I should see the welcome message
|
|
And the history should be loaded
|
|
And the prompt should be ready for input
|
|
|
|
Scenario: Configuration validation edge cases
|
|
Given I have an edge case configuration file "edge_case.yaml":
|
|
"""
|
|
agents: {}
|
|
|
|
routes:
|
|
empty_stream:
|
|
type: stream
|
|
stream_type: cold
|
|
operators: []
|
|
publications: []
|
|
|
|
merges: []
|
|
splits: []
|
|
"""
|
|
When I run "cleveragents run -c edge_case.yaml -p 'test'"
|
|
Then the command should fail
|
|
And the error message should mention configuration problems
|
|
And the exit code should be 2
|
|
|
|
Scenario: CLI output formatting and redirection
|
|
Given I have a working configuration file
|
|
When I run "cleveragents run -c config.yaml -p 'formatting test' -o output.json"
|
|
Then the command should succeed
|
|
And the file "output.json" should be created
|
|
And the output file should contain valid response data
|
|
And the stdout should confirm file creation
|
|
|
|
Scenario: Environment variable interpolation in CLI
|
|
Given I have a configuration file "env_config.yaml":
|
|
"""
|
|
agents:
|
|
env_agent:
|
|
type: llm
|
|
config:
|
|
provider: ${LLM_PROVIDER}
|
|
model: ${LLM_MODEL}
|
|
api_key: ${API_KEY}
|
|
|
|
routes:
|
|
env_stream:
|
|
type: stream
|
|
stream_type: cold
|
|
operators:
|
|
- type: map
|
|
params:
|
|
agent: env_agent
|
|
publications:
|
|
- __output__
|
|
|
|
merges:
|
|
- sources: [__input__]
|
|
target: env_stream
|
|
"""
|
|
And I set environment variable "LLM_PROVIDER" to "openai"
|
|
And I set environment variable "LLM_MODEL" to "gpt-3.5-turbo"
|
|
And I set environment variable "API_KEY" to "test-key"
|
|
When I run "cleveragents run -c env_config.yaml -p 'environment test'"
|
|
Then the command should succeed
|
|
And the configuration should use interpolated values
|
|
|
|
Scenario: CLI memory usage and performance monitoring
|
|
Given I have a large configuration file with 10 agents
|
|
When I run "cleveragents run -c large_config.yaml -p 'performance test' --verbose"
|
|
Then the command should succeed within 60 seconds
|
|
And the debug output should include performance metrics
|
|
And memory usage should remain reasonable
|
|
|
|
Scenario: CLI signal handling and graceful shutdown
|
|
Given I have a long-running configuration
|
|
When I start the command and send interrupt signal after 5 seconds
|
|
Then the process should terminate gracefully
|
|
And cleanup operations should complete
|
|
And temporary files should be removed
|
|
|
|
Scenario: CLI with multiple configuration sources
|
|
Given I have configuration files:
|
|
| filename | content_type |
|
|
| base.yaml | base |
|
|
| agents.yaml | agents |
|
|
| routes.yaml | routes |
|
|
| env.yaml | environment |
|
|
When I run "cleveragents run -c base.yaml -c agents.yaml -c routes.yaml -c env.yaml -p 'multi-config test'"
|
|
Then the command should succeed
|
|
And all configuration files should be loaded and merged correctly
|
|
And the merged configuration should contain all sections
|
|
|
|
Scenario: CLI visualization with different formats
|
|
Given I have a complex visualization configuration
|
|
When I run "cleveragents visualize -c complex.yaml -f mermaid -o diagram.mmd"
|
|
Then the command should succeed
|
|
And the file "diagram.mmd" should be created
|
|
And the diagram should contain valid Mermaid syntax
|
|
When I run "cleveragents visualize -c complex.yaml -f dot -o diagram.dot"
|
|
Then the command should succeed
|
|
And the file "diagram.dot" should contain valid DOT syntax
|
|
When I run "cleveragents visualize -c complex.yaml -f ascii"
|
|
Then the command should succeed
|
|
And the output should show ASCII network structure
|
|
|
|
Scenario: CLI error recovery and detailed error reporting
|
|
Given I have a configuration with intentional errors:
|
|
| error_type | description |
|
|
| missing_agent | Route references unknown agent |
|
|
| circular_dep | Circular stream dependencies |
|
|
| invalid_operator | Unknown operator type |
|
|
| malformed_yaml | Syntax errors in YAML |
|
|
When I run "cleveragents run -c error_config.yaml -p 'test'"
|
|
Then the command should fail
|
|
And the error message should be detailed and helpful
|
|
And the error should include line numbers when applicable
|
|
And suggested fixes should be provided
|
|
|
|
Scenario: CLI plugin and extension loading
|
|
Given I have a plugin configuration:
|
|
"""
|
|
plugins:
|
|
- name: custom_agent
|
|
path: ./plugins/custom_agent.py
|
|
- name: metrics_collector
|
|
path: ./plugins/metrics.py
|
|
"""
|
|
When I run "cleveragents run -c plugin_config.yaml -p 'plugin test'"
|
|
Then the command should load plugins successfully
|
|
And custom functionality should be available
|
|
And plugin errors should be reported clearly |