9ca1ceea49
Add 50+ Behave test scenarios covering all validation rules: Valid Scenarios (20): - LLM actors (minimal, with prompt, tools, memory, context) - TOOL actors (minimal, inline definitions) - GRAPH actors (linear, conditional, subgraphs) - All example YAML files (simple_llm, llm_with_tools, etc.) - Context views (strategist, executor, reviewer, full) - Memory configurations - Environment variables Invalid Name Scenarios (4): - Missing namespace - Multiple slashes - Empty namespace - Empty name Invalid LLM/TOOL/GRAPH Scenarios (7): - LLM without model - TOOL without tools - TOOL with empty tools list - GRAPH without model - GRAPH without route Graph Topology Validation (6): - Duplicate node IDs - Invalid entry node - Invalid exit nodes - Invalid edge references (from_node, to_node) - Cyclic graphs Tool/Node Validation (3): - Inline tool without namespace - Invalid parameter names - Invalid node ID formats Additional Scenarios (10): - YAML I/O (save/reload, missing files) - Edge priorities - Memory configurations (disabled, limits, summarization) Part 7 of C1.schema implementation (Actor YAML Schema Models).
356 lines
18 KiB
Gherkin
356 lines
18 KiB
Gherkin
Feature: Actor YAML schema validation
|
|
As a CleverAgents v3 user
|
|
I want to define actors via YAML configuration files
|
|
So that I can create validated, reusable actor workflows
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Valid LLM Actor scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Load a minimal valid LLM actor
|
|
Given an actor YAML string with minimal LLM configuration
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor config name should be "assistants/simple"
|
|
And the actor config type should be "llm"
|
|
And the actor config model should be "gpt-4"
|
|
|
|
Scenario: Load LLM actor with system prompt
|
|
Given an actor YAML string with LLM and system prompt
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor config system_prompt should contain "expert"
|
|
|
|
Scenario: Load LLM actor with tools
|
|
Given an actor YAML string with LLM and tools
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor config should have 2 tools
|
|
|
|
Scenario: Load LLM actor with memory config
|
|
Given an actor YAML string with memory configuration
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor memory enabled should be true
|
|
And the actor memory max_messages should be 50
|
|
|
|
Scenario: Load LLM actor with context config
|
|
Given an actor YAML string with context configuration
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor context should include 2 files
|
|
And the actor context should include 1 directory
|
|
|
|
Scenario: Load simple_llm.yaml example
|
|
Given the actor YAML file "examples/actors/simple_llm.yaml"
|
|
When I validate the actor schema from file
|
|
Then the actor schema validation should succeed
|
|
And the actor config type should be "llm"
|
|
|
|
Scenario: Load llm_with_tools.yaml example
|
|
Given the actor YAML file "examples/actors/llm_with_tools.yaml"
|
|
When I validate the actor schema from file
|
|
Then the actor schema validation should succeed
|
|
And the actor config should have at least 2 tools
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Valid TOOL Actor scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Load a minimal valid TOOL actor
|
|
Given an actor YAML string with TOOL type and tools
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor config type should be "tool"
|
|
And the actor config should have at least 1 tool
|
|
|
|
Scenario: Load TOOL actor with inline tool definition
|
|
Given an actor YAML string with inline tool definition
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor config should have 1 inline tool
|
|
|
|
Scenario: Load tool_collection.yaml example
|
|
Given the actor YAML file "examples/actors/tool_collection.yaml"
|
|
When I validate the actor schema from file
|
|
Then the actor schema validation should succeed
|
|
And the actor config type should be "tool"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Valid GRAPH Actor scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Load a minimal valid GRAPH actor
|
|
Given an actor YAML string with minimal GRAPH configuration
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor config type should be "graph"
|
|
And the actor route should have entry_node "start"
|
|
|
|
Scenario: Load GRAPH with linear topology
|
|
Given an actor YAML string with linear graph topology
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor route should have 3 nodes
|
|
And the actor route should have 2 edges
|
|
|
|
Scenario: Load GRAPH with conditional routing
|
|
Given an actor YAML string with conditional routing
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor route should have 1 conditional node
|
|
|
|
Scenario: Load GRAPH with subgraph node
|
|
Given an actor YAML string with subgraph node
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor route should have 1 subgraph node
|
|
|
|
Scenario: Load simple_graph.yaml example
|
|
Given the actor YAML file "examples/actors/simple_graph.yaml"
|
|
When I validate the actor schema from file
|
|
Then the actor schema validation should succeed
|
|
And the actor config type should be "graph"
|
|
|
|
Scenario: Load graph_workflow.yaml example
|
|
Given the actor YAML file "examples/actors/graph_workflow.yaml"
|
|
When I validate the actor schema from file
|
|
Then the actor schema validation should succeed
|
|
And the actor config type should be "graph"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Invalid Actor Name scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Reject actor without namespace
|
|
Given an actor YAML string with name "simple_actor"
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "namespaced"
|
|
|
|
Scenario: Reject actor with multiple slashes
|
|
Given an actor YAML string with name "namespace/sub/actor"
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "namespaced"
|
|
|
|
Scenario: Reject actor with empty namespace
|
|
Given an actor YAML string with name "/actor"
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "namespaced"
|
|
|
|
Scenario: Reject actor with empty name
|
|
Given an actor YAML string with name "namespace/"
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "namespaced"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Invalid LLM Actor scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Reject LLM actor without model
|
|
Given an actor YAML string with LLM type but no model
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "require 'model'"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Invalid TOOL Actor scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Reject TOOL actor without tools
|
|
Given an actor YAML string with TOOL type but no tools
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "require at least one tool"
|
|
|
|
Scenario: Reject TOOL actor with empty tools list
|
|
Given an actor YAML string with TOOL type and empty tools
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "require at least one tool"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Invalid GRAPH Actor scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Reject GRAPH actor without model
|
|
Given an actor YAML string with GRAPH type but no model
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "require 'model'"
|
|
|
|
Scenario: Reject GRAPH actor without route
|
|
Given an actor YAML string with GRAPH type but no route
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "require 'route'"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Graph Topology Validation scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Reject graph with duplicate node IDs
|
|
Given an actor YAML string with duplicate node IDs
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "Duplicate node IDs"
|
|
|
|
Scenario: Reject graph with invalid entry node
|
|
Given an actor YAML string with non-existent entry node
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "Entry node"
|
|
And the validation error should contain "not found"
|
|
|
|
Scenario: Reject graph with invalid exit node
|
|
Given an actor YAML string with non-existent exit node
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "Exit node"
|
|
And the validation error should contain "not found"
|
|
|
|
Scenario: Reject graph with invalid edge from_node
|
|
Given an actor YAML string with invalid edge from_node
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "from_node"
|
|
And the validation error should contain "not found"
|
|
|
|
Scenario: Reject graph with invalid edge to_node
|
|
Given an actor YAML string with invalid edge to_node
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "to_node"
|
|
And the validation error should contain "not found"
|
|
|
|
Scenario: Reject graph with cycles
|
|
Given an actor YAML string with cyclic graph
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "cycle"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Tool Definition Validation scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Reject inline tool without namespace
|
|
Given an actor YAML string with inline tool name "count_lines"
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "namespaced"
|
|
|
|
Scenario: Reject inline tool with invalid parameter name
|
|
Given an actor YAML string with invalid tool parameter name
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "valid Python identifier"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Node Definition Validation scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Reject node with invalid ID format
|
|
Given an actor YAML string with node ID containing spaces
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the validation error should contain "alphanumeric"
|
|
|
|
Scenario: Accept node with underscores and hyphens
|
|
Given an actor YAML string with node ID "node_1-test"
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Context View scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Load actor with strategist context view
|
|
Given an actor YAML string with context_view "strategist"
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor context_view should be "strategist"
|
|
|
|
Scenario: Load actor with executor context view
|
|
Given an actor YAML string with context_view "executor"
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor context_view should be "executor"
|
|
|
|
Scenario: Load actor with reviewer context view
|
|
Given an actor YAML string with context_view "reviewer"
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor context_view should be "reviewer"
|
|
|
|
Scenario: Load actor with full context view
|
|
Given an actor YAML string with context_view "full"
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor context_view should be "full"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Environment Variables scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Load actor with environment variables
|
|
Given an actor YAML string with env_vars
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor should have 2 env_vars
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# YAML I/O scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Save and reload actor configuration
|
|
Given a valid actor configuration object
|
|
When I save the actor to YAML file
|
|
And I reload the actor from YAML file
|
|
Then the reloaded actor should match the original
|
|
|
|
Scenario: Handle missing YAML file
|
|
Given a non-existent actor YAML file path
|
|
When I attempt to load the actor from file
|
|
Then a FileNotFoundError should be raised
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Edge Priority scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Load graph with edge priorities
|
|
Given an actor YAML string with edges having different priorities
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the highest priority edge should be 10
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Memory Configuration scenarios
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Load actor with memory disabled
|
|
Given an actor YAML string with memory enabled false
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor memory enabled should be false
|
|
|
|
Scenario: Load actor with message limit
|
|
Given an actor YAML string with max_messages 100
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor memory max_messages should be 100
|
|
|
|
Scenario: Load actor with token limit
|
|
Given an actor YAML string with max_tokens 8000
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor memory max_tokens should be 8000
|
|
|
|
Scenario: Load actor with summarize_old enabled
|
|
Given an actor YAML string with summarize_old true
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor memory summarize_old should be true
|