aditya
1f05648b2a
Add Robot Framework smoke tests for actor schema validation: - Valid actor tests: All 5 example YAMLs (LLM, tool, graph actors) - Invalid name tests: Missing namespace validation - Invalid LLM tests: Missing model field validation - Invalid TOOL tests: Missing tools field validation - Invalid GRAPH tests: Missing route and duplicate node IDs - Helper script: CLI interface for schema validation from Robot Part 9 of C1.schema implementation (Actor YAML Schema Models).
120 lines
6.4 KiB
Plaintext
120 lines
6.4 KiB
Plaintext
*** Settings ***
|
|
Documentation Smoke tests for Actor YAML schema validation
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_actor_schema.py
|
|
|
|
*** Test Cases ***
|
|
Validate Simple LLM Actor YAML
|
|
[Documentation] Parse the simple_llm example actor YAML and assert success
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${CURDIR}/../examples/actors/simple_llm.yaml cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-ok
|
|
|
|
Validate LLM With Tools Actor YAML
|
|
[Documentation] Parse the llm_with_tools example actor YAML and assert success
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${CURDIR}/../examples/actors/llm_with_tools.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-ok
|
|
|
|
Validate Tool Collection Actor YAML
|
|
[Documentation] Parse the tool_collection example actor YAML and assert success
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${CURDIR}/../examples/actors/tool_collection.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-ok
|
|
|
|
Validate Simple Graph Actor YAML
|
|
[Documentation] Parse the simple_graph example actor YAML and assert success
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${CURDIR}/../examples/actors/simple_graph.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-ok
|
|
|
|
Validate Complex Graph Workflow Actor YAML
|
|
[Documentation] Parse the graph_workflow example actor YAML and assert success
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${CURDIR}/../examples/actors/graph_workflow.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-ok
|
|
|
|
Reject Actor YAML Without Namespace
|
|
[Documentation] Attempt to parse actor YAML with invalid name (no namespace)
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_actor_no_namespace.yaml
|
|
Create File ${invalid_yaml} name: simple_actor\ntype: llm\ndescription: Test\nmodel: gpt-4\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-expected-fail
|
|
Should Contain ${result.stdout} namespaced
|
|
|
|
Reject LLM Actor Without Model
|
|
[Documentation] Attempt to parse LLM actor without required model field
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_llm_no_model.yaml
|
|
Create File ${invalid_yaml} name: test/actor\ntype: llm\ndescription: Missing model\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-expected-fail
|
|
Should Contain ${result.stdout} model
|
|
|
|
Reject TOOL Actor Without Tools
|
|
[Documentation] Attempt to parse TOOL actor without tools list
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_tool_no_tools.yaml
|
|
Create File ${invalid_yaml} name: test/tools\ntype: tool\ndescription: Missing tools\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-expected-fail
|
|
Should Contain ${result.stdout} tool
|
|
|
|
Reject GRAPH Actor Without Route
|
|
[Documentation] Attempt to parse GRAPH actor without route definition
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_graph_no_route.yaml
|
|
Create File ${invalid_yaml} name: test/workflow\ntype: graph\ndescription: Missing route\nmodel: gpt-4\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-expected-fail
|
|
Should Contain ${result.stdout} route
|
|
|
|
Reject Graph With Duplicate Node IDs
|
|
[Documentation] Attempt to parse graph with duplicate node IDs
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_graph_duplicate_nodes.yaml
|
|
${yaml_content}= Catenate SEPARATOR=\n
|
|
... name: test/duplicate
|
|
... type: graph
|
|
... description: Duplicate nodes
|
|
... model: gpt-4
|
|
... route:
|
|
... ${SPACE}${SPACE}nodes:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}- id: node1
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}type: agent
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}name: First
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}description: First node
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}config:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}prompt: Test
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}- id: node1
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}type: agent
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}name: Duplicate
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}description: Duplicate ID
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}config:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}prompt: Test
|
|
... ${SPACE}${SPACE}edges: []
|
|
... ${SPACE}${SPACE}entry_node: node1
|
|
... ${SPACE}${SPACE}exit_nodes: [node1]
|
|
Create File ${invalid_yaml} ${yaml_content}
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-expected-fail
|
|
Should Contain ${result.stdout} Duplicate
|