test(actor): add Robot Framework tests for actor examples
Create robot/actor_examples.robot with 17 integration test cases:
- Validate all 5 example YAML files load successfully
- Verify actor patterns (strategist, tool-only, graph, hierarchical)
- Check documentation completeness and pattern coverage
- Test invalid example rejection scenarios
Create robot/helper_actor_examples.py with validation commands.
Uses ${PYTHON} and ${CURDIR} per Robot Framework standards.
This commit is contained in:
@@ -0,0 +1,166 @@
|
||||
*** Settings ***
|
||||
Documentation Integration tests for actor YAML examples
|
||||
... Validates that all actor example files load correctly
|
||||
... and demonstrate expected patterns (strategist, executor,
|
||||
... reviewer, tool-only, validation, graphs, hierarchical)
|
||||
Resource ${CURDIR}/common.resource
|
||||
Suite Setup Setup Test Environment
|
||||
Suite Teardown Cleanup Test Environment
|
||||
|
||||
*** Variables ***
|
||||
${HELPER} ${CURDIR}/helper_actor_examples.py
|
||||
${EXAMPLES_DIR} ${CURDIR}/../examples/actors
|
||||
${DOCS_DIR} ${CURDIR}/../docs/reference
|
||||
|
||||
*** Test Cases ***
|
||||
Load All Example YAML Files
|
||||
[Documentation] Verify all 5 example YAML files can be loaded successfully
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} count-examples ${EXAMPLES_DIR} cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} example-count:5
|
||||
|
||||
Validate Simple LLM Example
|
||||
[Documentation] Load simple_llm.yaml and verify basic LLM actor structure
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/simple_llm.yaml cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} actor-ok
|
||||
Should Contain ${result.stdout} type:llm
|
||||
Should Contain ${result.stdout} name:assistants/code_reviewer
|
||||
|
||||
Validate LLM With Tools Example
|
||||
[Documentation] Load llm_with_tools.yaml and verify tools configuration
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/llm_with_tools.yaml cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} actor-ok
|
||||
Should Contain ${result.stdout} type:llm
|
||||
Should Contain ${result.stdout} has-tools
|
||||
|
||||
Validate Tool Collection Example
|
||||
[Documentation] Load tool_collection.yaml and verify tool-only actor
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/tool_collection.yaml cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} actor-ok
|
||||
Should Contain ${result.stdout} type:tool
|
||||
Should Contain ${result.stdout} has-tools
|
||||
|
||||
Validate Simple Graph Example
|
||||
[Documentation] Load simple_graph.yaml and verify linear graph structure
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/simple_graph.yaml cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} actor-ok
|
||||
Should Contain ${result.stdout} type:graph
|
||||
Should Contain ${result.stdout} has-route
|
||||
|
||||
Validate Complex Graph Workflow Example
|
||||
[Documentation] Load graph_workflow.yaml and verify complex graph with conditionals
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/graph_workflow.yaml cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} actor-ok
|
||||
Should Contain ${result.stdout} type:graph
|
||||
Should Contain ${result.stdout} name:workflows/test_driven_dev
|
||||
Should Contain ${result.stdout} has-route
|
||||
Should Contain ${result.stdout} has-env-vars
|
||||
|
||||
Verify Strategist Pattern
|
||||
[Documentation] Verify simple_llm.yaml follows strategist pattern (reviewer context)
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/simple_llm.yaml strategist cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} has-memory
|
||||
Should Contain ${result.stdout} context-view:reviewer
|
||||
|
||||
Verify Tool-Only Pattern
|
||||
[Documentation] Verify tool_collection.yaml follows tool-only pattern (no LLM)
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/tool_collection.yaml tool-only cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} pattern-ok:tool-only
|
||||
Should Contain ${result.stdout} no-model
|
||||
|
||||
Verify Graph Pattern
|
||||
[Documentation] Verify simple_graph.yaml follows graph workflow pattern
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/simple_graph.yaml graph cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} pattern-ok:graph
|
||||
Should Contain ${result.stdout} has-entry-node
|
||||
Should Contain ${result.stdout} has-exit-nodes
|
||||
|
||||
Verify Hierarchical Graph Pattern
|
||||
[Documentation] Verify graph_workflow.yaml has hierarchical structure
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/graph_workflow.yaml hierarchical cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} pattern-ok:hierarchical
|
||||
Should Contain ${result.stdout} has-subgraph
|
||||
|
||||
Verify Documentation References All Examples
|
||||
[Documentation] Verify docs/reference/actors_examples.md references all example files
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} check-docs ${DOCS_DIR}/actors_examples.md cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} doc-references:simple_llm.yaml
|
||||
Should Contain ${result.stdout} doc-references:llm_with_tools.yaml
|
||||
Should Contain ${result.stdout} doc-references:tool_collection.yaml
|
||||
Should Contain ${result.stdout} doc-references:simple_graph.yaml
|
||||
Should Contain ${result.stdout} doc-references:graph_workflow.yaml
|
||||
|
||||
Verify Documentation Has Pattern Examples
|
||||
[Documentation] Verify documentation includes all required actor patterns
|
||||
[Tags] slow
|
||||
${result}= Run Process ${PYTHON} ${HELPER} check-doc-patterns ${DOCS_DIR}/actors_examples.md cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} has-pattern:strategist
|
||||
Should Contain ${result.stdout} has-pattern:executor
|
||||
Should Contain ${result.stdout} has-pattern:reviewer
|
||||
Should Contain ${result.stdout} has-pattern:tool-only
|
||||
Should Contain ${result.stdout} has-pattern:validation
|
||||
Should Contain ${result.stdout} has-pattern:hierarchical
|
||||
|
||||
Reject Invalid Actor Name
|
||||
[Documentation] Verify that actor with invalid name (no namespace) is rejected
|
||||
[Tags] slow
|
||||
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_no_namespace.yaml
|
||||
Create File ${invalid_yaml} name: InvalidName\ntype: llm\nmodel: gpt-4\n
|
||||
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} expected-fail
|
||||
Should Contain ${result.stdout} namespace
|
||||
|
||||
Reject LLM Actor Without Model
|
||||
[Documentation] Verify that LLM actor without model field is rejected
|
||||
[Tags] slow
|
||||
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_llm_no_model.yaml
|
||||
Create File ${invalid_yaml} name: test/actor\ntype: llm\ndescription: Missing model\n
|
||||
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} expected-fail
|
||||
Should Contain ${result.stdout} model
|
||||
|
||||
Reject TOOL Actor Without Tools
|
||||
[Documentation] Verify that TOOL actor without tools list is rejected
|
||||
[Tags] slow
|
||||
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_tool_no_tools.yaml
|
||||
Create File ${invalid_yaml} name: test/tools\ntype: tool\ndescription: Missing tools\n
|
||||
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} expected-fail
|
||||
Should Contain ${result.stdout} tool
|
||||
|
||||
Reject GRAPH Actor Without Route
|
||||
[Documentation] Verify that GRAPH actor without route definition is rejected
|
||||
[Tags] slow
|
||||
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_graph_no_route.yaml
|
||||
Create File ${invalid_yaml} name: test/workflow\ntype: graph\nmodel: gpt-4\ndescription: Missing route\n
|
||||
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} expected-fail
|
||||
Should Contain ${result.stdout} route
|
||||
@@ -0,0 +1,246 @@
|
||||
"""Robot Framework helper for actor YAML examples validation.
|
||||
|
||||
Provides CLI-style interface for Robot to validate actor examples,
|
||||
check patterns, and verify documentation completeness.
|
||||
|
||||
Usage:
|
||||
python robot/helper_actor_examples.py validate <yaml_file>
|
||||
python robot/helper_actor_examples.py validate-invalid <yaml_file>
|
||||
python robot/helper_actor_examples.py count-examples <directory>
|
||||
python robot/helper_actor_examples.py check-pattern <yaml_file> <pattern>
|
||||
python robot/helper_actor_examples.py check-docs <doc_file>
|
||||
python robot/helper_actor_examples.py check-doc-patterns <doc_file>
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Ensure the src directory is on the import path.
|
||||
_SRC = str(Path(__file__).resolve().parents[1] / "src")
|
||||
if _SRC not in sys.path:
|
||||
sys.path.insert(0, _SRC)
|
||||
|
||||
from cleveragents.actor.schema import ActorConfigSchema, ActorType # noqa: E402
|
||||
|
||||
|
||||
def validate_actor(yaml_path: str) -> int:
|
||||
"""Validate actor YAML and print details."""
|
||||
try:
|
||||
config = ActorConfigSchema.from_yaml_file(yaml_path)
|
||||
print("actor-ok")
|
||||
print(f"name:{config.name}")
|
||||
print(f"type:{config.type.value}")
|
||||
|
||||
if config.tools:
|
||||
print("has-tools")
|
||||
|
||||
if config.route:
|
||||
print("has-route")
|
||||
|
||||
if config.env_vars:
|
||||
print("has-env-vars")
|
||||
|
||||
if config.memory:
|
||||
print("has-memory")
|
||||
|
||||
if config.context_view:
|
||||
print(f"context-view:{config.context_view.value}")
|
||||
|
||||
return 0
|
||||
except Exception as exc:
|
||||
print(f"actor-fail: {exc}")
|
||||
return 1
|
||||
|
||||
|
||||
def validate_invalid_actor(yaml_path: str) -> int:
|
||||
"""Expect validation to fail."""
|
||||
try:
|
||||
ActorConfigSchema.from_yaml_file(yaml_path)
|
||||
print("unexpected-success")
|
||||
return 1
|
||||
except Exception as exc:
|
||||
print("expected-fail")
|
||||
error_msg = str(exc).lower()
|
||||
if "namespace" in error_msg or "name" in error_msg:
|
||||
print("namespace")
|
||||
if "model" in error_msg:
|
||||
print("model")
|
||||
if "tool" in error_msg:
|
||||
print("tool")
|
||||
if "route" in error_msg:
|
||||
print("route")
|
||||
return 0
|
||||
|
||||
|
||||
def count_examples(directory: str) -> int:
|
||||
"""Count YAML example files."""
|
||||
try:
|
||||
examples_dir = Path(directory)
|
||||
yaml_files = list(examples_dir.glob("*.yaml"))
|
||||
print(f"example-count:{len(yaml_files)}")
|
||||
return 0
|
||||
except Exception as exc:
|
||||
print(f"count-fail: {exc}")
|
||||
return 1
|
||||
|
||||
|
||||
def check_pattern(yaml_path: str, pattern: str) -> int:
|
||||
"""Check if actor follows a specific pattern."""
|
||||
try:
|
||||
config = ActorConfigSchema.from_yaml_file(yaml_path)
|
||||
|
||||
if pattern == "strategist":
|
||||
# Strategist pattern: large context, read-only tools, memory enabled
|
||||
if config.memory and config.memory.enabled:
|
||||
print("has-memory")
|
||||
if config.context_view:
|
||||
print(f"context-view:{config.context_view.value}")
|
||||
print(f"pattern-ok:{pattern}")
|
||||
|
||||
elif pattern == "tool-only":
|
||||
# Tool-only pattern: no model, has tools
|
||||
if config.type == ActorType.TOOL:
|
||||
print("pattern-ok:tool-only")
|
||||
if not config.model:
|
||||
print("no-model")
|
||||
|
||||
elif pattern == "graph":
|
||||
# Graph pattern: has route with entry/exit nodes
|
||||
if config.route:
|
||||
print("pattern-ok:graph")
|
||||
if config.route.entry_node:
|
||||
print("has-entry-node")
|
||||
if config.route.exit_nodes:
|
||||
print("has-exit-nodes")
|
||||
|
||||
elif pattern == "hierarchical":
|
||||
# Hierarchical pattern: graph with subgraph nodes
|
||||
if config.route:
|
||||
print("pattern-ok:hierarchical")
|
||||
# Check for subgraph nodes
|
||||
from cleveragents.actor.schema import NodeType
|
||||
|
||||
subgraph_nodes = [
|
||||
node
|
||||
for node in config.route.nodes
|
||||
if node.type == NodeType.SUBGRAPH
|
||||
]
|
||||
if subgraph_nodes:
|
||||
print("has-subgraph")
|
||||
|
||||
return 0
|
||||
except Exception as exc:
|
||||
print(f"pattern-check-fail: {exc}")
|
||||
return 1
|
||||
|
||||
|
||||
def check_docs(doc_file: str) -> int:
|
||||
"""Check if documentation references all example files."""
|
||||
try:
|
||||
doc_path = Path(doc_file)
|
||||
doc_content = doc_path.read_text()
|
||||
|
||||
example_files = [
|
||||
"simple_llm.yaml",
|
||||
"llm_with_tools.yaml",
|
||||
"tool_collection.yaml",
|
||||
"simple_graph.yaml",
|
||||
"graph_workflow.yaml",
|
||||
]
|
||||
|
||||
for filename in example_files:
|
||||
if filename in doc_content:
|
||||
print(f"doc-references:{filename}")
|
||||
|
||||
return 0
|
||||
except Exception as exc:
|
||||
print(f"doc-check-fail: {exc}")
|
||||
return 1
|
||||
|
||||
|
||||
def check_doc_patterns(doc_file: str) -> int:
|
||||
"""Check if documentation includes all required patterns."""
|
||||
try:
|
||||
doc_path = Path(doc_file)
|
||||
doc_content = doc_path.read_text().lower()
|
||||
|
||||
patterns = [
|
||||
"strategist",
|
||||
"executor",
|
||||
"reviewer",
|
||||
"tool-only",
|
||||
"validation",
|
||||
"hierarchical",
|
||||
]
|
||||
|
||||
for pattern in patterns:
|
||||
if pattern in doc_content:
|
||||
print(f"has-pattern:{pattern}")
|
||||
|
||||
return 0
|
||||
except Exception as exc:
|
||||
print(f"doc-pattern-check-fail: {exc}")
|
||||
return 1
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Entry point called by Robot Framework ``Run Process``."""
|
||||
if len(sys.argv) < 2:
|
||||
print(
|
||||
"Usage: helper_actor_examples.py <command> [args...]\n"
|
||||
"Commands:\n"
|
||||
" validate <file>\n"
|
||||
" validate-invalid <file>\n"
|
||||
" count-examples <directory>\n"
|
||||
" check-pattern <file> <pattern>\n"
|
||||
" check-docs <doc_file>\n"
|
||||
" check-doc-patterns <doc_file>"
|
||||
)
|
||||
return 1
|
||||
|
||||
command = sys.argv[1]
|
||||
|
||||
if command == "validate":
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: validate <yaml_file>")
|
||||
return 1
|
||||
return validate_actor(sys.argv[2])
|
||||
|
||||
if command == "validate-invalid":
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: validate-invalid <yaml_file>")
|
||||
return 1
|
||||
return validate_invalid_actor(sys.argv[2])
|
||||
|
||||
if command == "count-examples":
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: count-examples <directory>")
|
||||
return 1
|
||||
return count_examples(sys.argv[2])
|
||||
|
||||
if command == "check-pattern":
|
||||
if len(sys.argv) < 4:
|
||||
print("Usage: check-pattern <yaml_file> <pattern>")
|
||||
return 1
|
||||
return check_pattern(sys.argv[2], sys.argv[3])
|
||||
|
||||
if command == "check-docs":
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: check-docs <doc_file>")
|
||||
return 1
|
||||
return check_docs(sys.argv[2])
|
||||
|
||||
if command == "check-doc-patterns":
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: check-doc-patterns <doc_file>")
|
||||
return 1
|
||||
return check_doc_patterns(sys.argv[2])
|
||||
|
||||
print(f"Unknown command: {command}")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user