aditya
4ae7f2d6fe
- Add missing node descriptions in Behave test scenarios - Fix context_view and parallel execution step definitions - Update llm_with_tools.yaml max_context_tokens (10000 → 16000) - Add context.error compatibility for error assertions Results: Behave 25/25, Robot 16/16, ASV 7/7 all newly added tests passing.
676 lines
26 KiB
Gherkin
676 lines
26 KiB
Gherkin
Feature: Actor YAML examples validation
|
|
As a CleverAgents v3 user
|
|
I want to validate all actor YAML examples
|
|
So that I can be confident the examples are correct and usable
|
|
|
|
Background:
|
|
Given the CleverAgents actor schema validation is available
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Example File Loading - All examples must load successfully
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Load simple_llm.yaml example
|
|
Given the actor examples YAML file "examples/actors/simple_llm.yaml"
|
|
When I validate the actor schema from file
|
|
Then the actor schema validation should succeed
|
|
And the actor config name should be "assistants/code_reviewer"
|
|
And the actor config type should be "llm"
|
|
And the actor config model should be "gpt-4"
|
|
And the actor config description should contain "Reviews Python code"
|
|
And the actor config system_prompt should contain "expert Python code reviewer"
|
|
And the actor config context_view should be "reviewer"
|
|
And the actor memory enabled should be true
|
|
And the actor memory max_messages should be 20
|
|
And the actor memory max_tokens should be 4000
|
|
|
|
Scenario: Load llm_with_tools.yaml example
|
|
Given the actor examples YAML file "examples/actors/llm_with_tools.yaml"
|
|
When I validate the actor schema from file
|
|
Then the actor schema validation should succeed
|
|
And the actor config type should be "llm"
|
|
And the actor config should have at least 2 tools
|
|
And the actor memory enabled should be true
|
|
And the actor context max_context_tokens should be 16000
|
|
|
|
Scenario: Load tool_collection.yaml example
|
|
Given the actor examples YAML file "examples/actors/tool_collection.yaml"
|
|
When I validate the actor schema from file
|
|
Then the actor schema validation should succeed
|
|
And the actor config type should be "tool"
|
|
And the actor config should have at least 1 tool
|
|
And the actor config name should match pattern "^[a-zA-Z0-9_-]+/[a-zA-Z0-9_-]+$"
|
|
|
|
Scenario: Load simple_graph.yaml example
|
|
Given the actor examples YAML file "examples/actors/simple_graph.yaml"
|
|
When I validate the actor schema from file
|
|
Then the actor schema validation should succeed
|
|
And the actor config type should be "graph"
|
|
And the actor route should have at least 3 nodes
|
|
And the actor route should have at least 2 edges
|
|
And the actor route should have an entry_node
|
|
And the actor route should have exit_nodes
|
|
|
|
Scenario: Load graph_workflow.yaml example
|
|
Given the actor examples YAML file "examples/actors/graph_workflow.yaml"
|
|
When I validate the actor schema from file
|
|
Then the actor schema validation should succeed
|
|
And the actor config name should be "workflows/test_driven_dev"
|
|
And the actor config type should be "graph"
|
|
And the actor route should have at least 8 nodes
|
|
And the actor config should have environment variables
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Strategist Actor Pattern Validation
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Validate strategist actor pattern
|
|
Given an actor YAML string representing a strategist actor:
|
|
"""
|
|
name: strategists/task_planner
|
|
type: llm
|
|
description: Strategic planning and task decomposition
|
|
version: "1.0"
|
|
model: gpt-4
|
|
system_prompt: |
|
|
You are a strategic planner. Break down goals into actionable steps.
|
|
context_view: strategist
|
|
memory:
|
|
enabled: true
|
|
max_messages: 50
|
|
max_tokens: 16000
|
|
tools:
|
|
- files/read_file
|
|
- files/list_directory
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor config context_view should be "strategist"
|
|
And the actor memory max_messages should be at least 50
|
|
And the actor config should have read-only tools
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Executor Actor Pattern Validation
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Validate executor actor pattern
|
|
Given an actor YAML string representing an executor actor:
|
|
"""
|
|
name: executors/code_writer
|
|
type: llm
|
|
description: Code implementation and execution
|
|
version: "1.0"
|
|
model: gpt-4
|
|
system_prompt: "You are an expert software engineer."
|
|
context_view: executor
|
|
memory:
|
|
enabled: true
|
|
max_messages: 30
|
|
max_tokens: 8000
|
|
tools:
|
|
- files/read_file
|
|
- files/write_file
|
|
- files/create_directory
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor config context_view should be "executor"
|
|
And the actor config should have write-capable tools
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Reviewer Actor Pattern Validation
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Validate reviewer actor pattern
|
|
Given an actor YAML string representing a reviewer actor:
|
|
"""
|
|
name: reviewers/quality_gate
|
|
type: llm
|
|
description: Code quality and standards review
|
|
version: "1.0"
|
|
model: gpt-4
|
|
system_prompt: "You are a senior code reviewer."
|
|
context_view: reviewer
|
|
memory:
|
|
enabled: true
|
|
max_messages: 40
|
|
max_tokens: 12000
|
|
tools:
|
|
- files/read_file
|
|
- git/diff
|
|
- linting/run_ruff
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor config context_view should be "reviewer"
|
|
And the actor config should have linting tools
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Tool-Only Actor Validation
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Validate tool-only actor with file operations
|
|
Given an actor YAML string with tool-only configuration:
|
|
"""
|
|
name: utilities/file_ops
|
|
type: tool
|
|
description: File system operation tools
|
|
version: "1.0"
|
|
tools:
|
|
- files/read_file
|
|
- files/write_file
|
|
- files/create_directory
|
|
- files/delete_file
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor config type should be "tool"
|
|
And the actor config should have 4 tools
|
|
And the actor config should not have model field
|
|
And the actor config should not have system_prompt field
|
|
|
|
Scenario: Validate tool-only actor with git operations
|
|
Given an actor YAML string with git tools:
|
|
"""
|
|
name: utilities/git_ops
|
|
type: tool
|
|
description: Git operation tools
|
|
version: "1.0"
|
|
tools:
|
|
- git/status
|
|
- git/add
|
|
- git/commit
|
|
- git/diff
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor config should have 4 tools
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Validation-Node Actor Validation
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Validate validation-node actor pattern
|
|
Given an actor YAML string representing a validator actor:
|
|
"""
|
|
name: validators/python_checker
|
|
type: llm
|
|
description: Python code validation and linting
|
|
version: "1.0"
|
|
model: gpt-4
|
|
system_prompt: "You are a validation expert."
|
|
context_view: reviewer
|
|
memory:
|
|
enabled: false
|
|
tools:
|
|
- files/read_file
|
|
- linting/run_ruff
|
|
- linting/run_pyright
|
|
- testing/run_pytest
|
|
- security/run_bandit
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor memory enabled should be false
|
|
And the actor config should have validation tools
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Linear Graph Workflow Validation
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Validate linear three-step graph workflow
|
|
Given an actor YAML string with linear graph workflow:
|
|
"""
|
|
name: workflows/linear_review
|
|
type: graph
|
|
description: Linear three-step review workflow
|
|
version: "1.0"
|
|
model: gpt-4
|
|
route:
|
|
nodes:
|
|
- id: analyzer
|
|
type: agent
|
|
name: Analyzer
|
|
description: Analyzes code
|
|
config:
|
|
model: gpt-4
|
|
prompt: "Analyze the code"
|
|
tools:
|
|
- files/read_file
|
|
- id: reviewer
|
|
type: agent
|
|
name: Reviewer
|
|
description: Reviews code
|
|
config:
|
|
model: gpt-4
|
|
prompt: "Review the code"
|
|
tools:
|
|
- files/read_file
|
|
- id: reporter
|
|
type: agent
|
|
name: Reporter
|
|
description: Generates report
|
|
config:
|
|
model: gpt-4
|
|
prompt: "Generate report"
|
|
tools:
|
|
- files/write_file
|
|
edges:
|
|
- from_node: analyzer
|
|
to_node: reviewer
|
|
- from_node: reviewer
|
|
to_node: reporter
|
|
entry_node: analyzer
|
|
exit_nodes:
|
|
- reporter
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor route should have 3 nodes
|
|
And the actor route should have 2 edges
|
|
And the actor route entry_node should be "analyzer"
|
|
And the actor route should have 1 exit node
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Complex Graph with Conditionals
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Validate graph with conditional routing
|
|
Given an actor YAML string with conditional routing examples:
|
|
"""
|
|
name: workflows/conditional_flow
|
|
type: graph
|
|
description: Workflow with conditional routing
|
|
version: "1.0"
|
|
model: gpt-4
|
|
route:
|
|
nodes:
|
|
- id: checker
|
|
type: conditional
|
|
name: Checker
|
|
description: Checks condition
|
|
config:
|
|
conditions:
|
|
- check: "state.get('passed') == True"
|
|
route_to: success
|
|
- check: "state.get('passed') == False"
|
|
route_to: failure
|
|
- id: success
|
|
type: agent
|
|
name: Success Handler
|
|
description: Handles success case
|
|
config:
|
|
model: gpt-4
|
|
prompt: "Handle success"
|
|
- id: failure
|
|
type: agent
|
|
name: Failure Handler
|
|
description: Handles failure case
|
|
config:
|
|
model: gpt-4
|
|
prompt: "Handle failure"
|
|
edges:
|
|
- from_node: checker
|
|
to_node: success
|
|
- from_node: checker
|
|
to_node: failure
|
|
entry_node: checker
|
|
exit_nodes:
|
|
- success
|
|
- failure
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor route should have conditional node
|
|
And the actor route should have multiple exit nodes
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Graph with Tool Nodes
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Validate graph with tool execution nodes
|
|
Given an actor YAML string with tool nodes examples:
|
|
"""
|
|
name: workflows/tool_pipeline
|
|
type: graph
|
|
description: Pipeline with tool execution
|
|
version: "1.0"
|
|
model: gpt-4
|
|
route:
|
|
nodes:
|
|
- id: reader
|
|
type: agent
|
|
name: Reader
|
|
description: Reads and analyzes files
|
|
config:
|
|
model: gpt-4
|
|
prompt: "Read and analyze"
|
|
tools:
|
|
- files/read_file
|
|
- id: tester
|
|
type: tool
|
|
name: Test Runner
|
|
description: Runs tests
|
|
config:
|
|
tool_name: testing/run_pytest
|
|
parameters:
|
|
verbose: true
|
|
- id: writer
|
|
type: agent
|
|
name: Writer
|
|
description: Writes results to files
|
|
config:
|
|
model: gpt-4
|
|
prompt: "Write results"
|
|
tools:
|
|
- files/write_file
|
|
edges:
|
|
- from_node: reader
|
|
to_node: tester
|
|
- from_node: tester
|
|
to_node: writer
|
|
entry_node: reader
|
|
exit_nodes:
|
|
- writer
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor route should have tool node
|
|
And the actor route should have agent nodes
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Hierarchical Graph with Subgraphs
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Validate graph with subgraph invocation
|
|
Given an actor YAML string with subgraph examples:
|
|
"""
|
|
name: workflows/hierarchical
|
|
type: graph
|
|
description: Hierarchical workflow with subgraphs
|
|
version: "1.0"
|
|
model: gpt-4
|
|
route:
|
|
nodes:
|
|
- id: planner
|
|
type: agent
|
|
name: Planner
|
|
description: Creates execution plan
|
|
config:
|
|
model: gpt-4
|
|
prompt: "Create plan"
|
|
- id: executor_subgraph
|
|
type: subgraph
|
|
name: Executor Subgraph
|
|
description: Executes tasks via subgraph
|
|
config:
|
|
actor_path: examples/actors/executors/code_writer.yaml
|
|
parallel: true
|
|
max_parallel: 3
|
|
- id: reviewer_subgraph
|
|
type: subgraph
|
|
name: Reviewer Subgraph
|
|
description: Reviews execution via subgraph
|
|
config:
|
|
actor_path: examples/actors/reviewers/quality_gate.yaml
|
|
edges:
|
|
- from_node: planner
|
|
to_node: executor_subgraph
|
|
- from_node: executor_subgraph
|
|
to_node: reviewer_subgraph
|
|
entry_node: planner
|
|
exit_nodes:
|
|
- reviewer_subgraph
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor route should have subgraph nodes
|
|
And the actor route should have parallel execution config
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Graph with Retry Logic
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Validate graph with retry and failure handling
|
|
Given an actor YAML string with retry logic:
|
|
"""
|
|
name: workflows/retry_flow
|
|
type: graph
|
|
description: Workflow with retry logic
|
|
version: "1.0"
|
|
model: gpt-4
|
|
route:
|
|
nodes:
|
|
- id: executor
|
|
type: agent
|
|
name: Executor
|
|
description: Executes main task
|
|
config:
|
|
model: gpt-4
|
|
prompt: "Execute task"
|
|
- id: verifier
|
|
type: conditional
|
|
name: Verifier
|
|
description: Verifies task success or determines retry
|
|
config:
|
|
conditions:
|
|
- check: "state.get('success') == True"
|
|
route_to: success
|
|
- check: "state.get('retry_count', 0) < 3"
|
|
route_to: retry
|
|
- check: "state.get('retry_count', 0) >= 3"
|
|
route_to: escalate
|
|
- id: retry
|
|
type: agent
|
|
name: Retry Handler
|
|
description: Handles retry logic
|
|
config:
|
|
model: gpt-4
|
|
prompt: "Fix and retry"
|
|
- id: success
|
|
type: agent
|
|
name: Success Handler
|
|
description: Handles success case
|
|
config:
|
|
model: gpt-4
|
|
prompt: "Handle success"
|
|
- id: escalate
|
|
type: tool
|
|
name: Escalation
|
|
description: Escalates to alert
|
|
config:
|
|
tool_name: notifications/send_alert
|
|
edges:
|
|
- from_node: executor
|
|
to_node: verifier
|
|
- from_node: retry
|
|
to_node: executor
|
|
entry_node: executor
|
|
exit_nodes:
|
|
- success
|
|
- escalate
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor route should have retry loop
|
|
And the actor route should have escalation path
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Multi-Level Planner/Executor Pattern
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Validate multi-level planner-executor pattern
|
|
Given an actor YAML string with multi-level hierarchy:
|
|
"""
|
|
name: workflows/multi_level
|
|
type: graph
|
|
description: Multi-level planning and execution
|
|
version: "1.0"
|
|
model: gpt-4
|
|
route:
|
|
nodes:
|
|
- id: strategic_planner
|
|
type: agent
|
|
name: Strategic Planner
|
|
description: Creates high-level strategic plan
|
|
config:
|
|
model: gpt-4
|
|
prompt: "High-level planning"
|
|
- id: tactical_planner
|
|
type: subgraph
|
|
name: Tactical Planner
|
|
description: Plans tactical execution steps
|
|
config:
|
|
actor_path: examples/actors/strategists/task_planner.yaml
|
|
- id: executor_pool
|
|
type: subgraph
|
|
name: Executor Pool
|
|
description: Executes tasks in parallel
|
|
config:
|
|
actor_path: examples/actors/executors/code_writer.yaml
|
|
parallel: true
|
|
max_parallel: 5
|
|
- id: validator
|
|
type: subgraph
|
|
name: Validator
|
|
description: Validates execution results
|
|
config:
|
|
actor_path: examples/actors/validators/python_checker.yaml
|
|
edges:
|
|
- from_node: strategic_planner
|
|
to_node: tactical_planner
|
|
- from_node: tactical_planner
|
|
to_node: executor_pool
|
|
- from_node: executor_pool
|
|
to_node: validator
|
|
entry_node: strategic_planner
|
|
exit_nodes:
|
|
- validator
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should succeed
|
|
And the actor route should have 4 nodes
|
|
And the actor route should have multiple subgraph levels
|
|
And the actor route should support parallel execution
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Invalid Example Rejection
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Reject actor with missing required field
|
|
Given an actor YAML string missing required name field for examples:
|
|
"""
|
|
type: llm
|
|
description: Missing name
|
|
model: gpt-4
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the error message should contain "name"
|
|
|
|
Scenario: Reject actor with invalid namespaced name
|
|
Given an actor YAML string with invalid name for examples:
|
|
"""
|
|
name: InvalidName
|
|
type: llm
|
|
model: gpt-4
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the error message should contain "namespace"
|
|
|
|
Scenario: Reject LLM actor without model
|
|
Given an actor YAML string for LLM without model for examples:
|
|
"""
|
|
name: test/actor
|
|
type: llm
|
|
description: Missing model
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the error message should contain "model"
|
|
|
|
Scenario: Reject TOOL actor without tools
|
|
Given an actor YAML string for TOOL without tools for examples:
|
|
"""
|
|
name: test/tool_actor
|
|
type: tool
|
|
description: Missing tools
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the error message should contain "tools"
|
|
|
|
Scenario: Reject GRAPH actor without route
|
|
Given an actor YAML string for GRAPH without route for examples:
|
|
"""
|
|
name: test/graph_actor
|
|
type: graph
|
|
model: gpt-4
|
|
description: Missing route
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the error message should contain "route"
|
|
|
|
Scenario: Reject graph with cycle
|
|
Given an actor YAML string with cyclic graph for examples:
|
|
"""
|
|
name: test/cyclic
|
|
type: graph
|
|
model: gpt-4
|
|
description: Graph with cycle for testing
|
|
route:
|
|
nodes:
|
|
- id: a
|
|
type: agent
|
|
name: Node A
|
|
description: First node in cycle
|
|
config:
|
|
model: gpt-4
|
|
- id: b
|
|
type: agent
|
|
name: Node B
|
|
description: Second node in cycle
|
|
config:
|
|
model: gpt-4
|
|
edges:
|
|
- from_node: a
|
|
to_node: b
|
|
- from_node: b
|
|
to_node: a
|
|
entry_node: a
|
|
exit_nodes:
|
|
- b
|
|
"""
|
|
When I validate the actor schema
|
|
Then the actor schema validation should fail
|
|
And the error message should contain "cycle"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Example Count and Completeness
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Verify all example files exist
|
|
Given the examples/actors directory
|
|
Then there should be exactly 5 example YAML files
|
|
And the example files should include "simple_llm.yaml"
|
|
And the example files should include "llm_with_tools.yaml"
|
|
And the example files should include "tool_collection.yaml"
|
|
And the example files should include "simple_graph.yaml"
|
|
And the example files should include "graph_workflow.yaml"
|
|
|
|
Scenario: Verify all examples are documented
|
|
Given the documentation file "docs/reference/actors_examples.md"
|
|
Then the documentation should reference "simple_llm.yaml"
|
|
And the documentation should reference "llm_with_tools.yaml"
|
|
And the documentation should reference "tool_collection.yaml"
|
|
And the documentation should reference "simple_graph.yaml"
|
|
And the documentation should reference "graph_workflow.yaml"
|
|
And the documentation should have examples for strategist actors
|
|
And the documentation should have examples for executor actors
|
|
And the documentation should have examples for reviewer actors
|
|
And the documentation should have examples for tool-only actors
|
|
And the documentation should have examples for validation actors
|
|
And the documentation should have examples for hierarchical graphs
|