freemo
e03fd2956d
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 18s
CI / lint (pull_request) Failing after 18s
CI / helm (pull_request) Successful in 24s
CI / typecheck (pull_request) Failing after 52s
CI / security (pull_request) Failing after 53s
CI / coverage (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Has been skipped
CI / unit_tests (pull_request) Failing after 1m48s
CI / docker (pull_request) Has been skipped
CI / quality (pull_request) Successful in 3m42s
CI / e2e_tests (pull_request) Failing after 13m56s
CI / integration_tests (pull_request) Failing after 21m12s
CI / status-check (pull_request) Failing after 2s
Add missing provider: field to all actor examples in examples/actors/. Fix llm_with_tools.yaml actor name from assistants/file_analyzer to local/assistants-file_analyzer (custom actors must use local/ namespace and cannot contain two slashes). Add validate-all command to helper_actor_examples.py that uses ActorLoader to validate all examples via business logic checks. Add integration test Validate All Actor Examples Import Without Errors to actor_examples.robot that confirms every example in examples/actors/ can be imported without errors. ISSUES CLOSED: #1504
199 lines
11 KiB
Plaintext
199 lines
11 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration tests for actor YAML examples
|
|
... Validates that all actor example files load correctly
|
|
... and demonstrate expected patterns (strategist, executor,
|
|
... reviewer, tool-only, validation, graphs, hierarchical)
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_actor_examples.py
|
|
${EXAMPLES_DIR} ${CURDIR}/../examples/actors
|
|
${DOCS_DIR} ${CURDIR}/../docs/reference
|
|
|
|
*** Test Cases ***
|
|
Load All Example YAML Files
|
|
[Documentation] Verify all actor example YAML files can be loaded successfully
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} count-examples ${EXAMPLES_DIR} cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} example-count:
|
|
|
|
Validate Simple LLM Example
|
|
[Documentation] Load simple_llm.yaml and verify basic LLM actor structure
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/simple_llm.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:llm
|
|
Should Contain ${result.stdout} name:assistants/code_reviewer
|
|
|
|
Validate LLM With Tools Example
|
|
[Documentation] Load llm_with_tools.yaml and verify tools configuration
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/llm_with_tools.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:llm
|
|
Should Contain ${result.stdout} name:local/assistants-file_analyzer
|
|
Should Contain ${result.stdout} has-tools
|
|
|
|
Validate Estimator Example
|
|
[Documentation] Load estimator.yaml and verify estimation actor configuration
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/estimator.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:llm
|
|
Should Contain ${result.stdout} name:local/estimator
|
|
Should Contain ${result.stdout} has-tools
|
|
Should Contain ${result.stdout} has-skills
|
|
|
|
Validate Tool Collection Example
|
|
[Documentation] Load tool_collection.yaml and verify tool-only actor
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/tool_collection.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:tool
|
|
Should Contain ${result.stdout} has-tools
|
|
|
|
Validate Simple Graph Example
|
|
[Documentation] Load simple_graph.yaml and verify linear graph structure
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/simple_graph.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:graph
|
|
Should Contain ${result.stdout} has-route
|
|
|
|
Validate Complex Graph Workflow Example
|
|
[Documentation] Load graph_workflow.yaml and verify complex graph with conditionals
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/graph_workflow.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:graph
|
|
Should Contain ${result.stdout} name:workflows/test_driven_dev
|
|
Should Contain ${result.stdout} has-route
|
|
Should Contain ${result.stdout} has-env-vars
|
|
|
|
Verify Strategist Pattern
|
|
[Documentation] Verify simple_llm.yaml follows strategist pattern (reviewer context)
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/simple_llm.yaml strategist cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} has-memory
|
|
Should Contain ${result.stdout} context-view:reviewer
|
|
|
|
Verify Tool-Only Pattern
|
|
[Documentation] Verify tool_collection.yaml follows tool-only pattern (no LLM)
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/tool_collection.yaml tool-only cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} pattern-ok:tool-only
|
|
Should Contain ${result.stdout} no-model
|
|
|
|
Verify Graph Pattern
|
|
[Documentation] Verify simple_graph.yaml follows graph workflow pattern
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/simple_graph.yaml graph cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} pattern-ok:graph
|
|
Should Contain ${result.stdout} has-entry-node
|
|
Should Contain ${result.stdout} has-exit-nodes
|
|
|
|
Verify Hierarchical Graph Pattern
|
|
[Documentation] Verify graph_workflow.yaml has hierarchical structure
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/graph_workflow.yaml hierarchical cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} pattern-ok:hierarchical
|
|
Should Contain ${result.stdout} has-subgraph
|
|
|
|
Verify Documentation References All Examples
|
|
[Documentation] Verify docs/reference/actors_examples.md references all example files
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-docs ${DOCS_DIR}/actors_examples.md cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} doc-references:simple_llm.yaml
|
|
Should Contain ${result.stdout} doc-references:llm_with_tools.yaml
|
|
Should Contain ${result.stdout} doc-references:estimator.yaml
|
|
Should Contain ${result.stdout} doc-references:tool_collection.yaml
|
|
Should Contain ${result.stdout} doc-references:simple_graph.yaml
|
|
Should Contain ${result.stdout} doc-references:graph_workflow.yaml
|
|
|
|
Verify Documentation Has Pattern Examples
|
|
[Documentation] Verify documentation includes all required actor patterns
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-doc-patterns ${DOCS_DIR}/actors_examples.md cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} has-pattern:strategist
|
|
Should Contain ${result.stdout} has-pattern:executor
|
|
Should Contain ${result.stdout} has-pattern:reviewer
|
|
Should Contain ${result.stdout} has-pattern:tool-only
|
|
Should Contain ${result.stdout} has-pattern:validation
|
|
Should Contain ${result.stdout} has-pattern:hierarchical
|
|
|
|
Reject Invalid Actor Name
|
|
[Documentation] Verify that actor with invalid name (no namespace) is rejected
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_no_namespace.yaml
|
|
Create File ${invalid_yaml} name: InvalidName\ntype: llm\nmodel: gpt-4\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} expected-fail
|
|
Should Contain ${result.stdout} namespace
|
|
|
|
Reject LLM Actor Without Model
|
|
[Documentation] Verify that LLM actor without model field is rejected
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_llm_no_model.yaml
|
|
Create File ${invalid_yaml} name: test/actor\ntype: llm\ndescription: Missing model\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} expected-fail
|
|
Should Contain ${result.stdout} model
|
|
|
|
Reject TOOL Actor Without Tools
|
|
[Documentation] Verify that TOOL actor without tools list is rejected
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_tool_no_tools.yaml
|
|
Create File ${invalid_yaml} name: test/tools\ntype: tool\ndescription: Missing tools\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} expected-fail
|
|
Should Contain ${result.stdout} tool
|
|
|
|
Reject GRAPH Actor Without Route
|
|
[Documentation] Verify that GRAPH actor without route definition is rejected
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_graph_no_route.yaml
|
|
Create File ${invalid_yaml} name: test/workflow\ntype: graph\nmodel: gpt-4\ndescription: Missing route\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} expected-fail
|
|
Should Contain ${result.stdout} route
|
|
|
|
Validate All Actor Examples Import Without Errors
|
|
[Documentation] Validate that all actor examples in examples/actors/ can be
|
|
... imported without errors using ActorLoader business logic checks.
|
|
... This is the integration test required by issue #1504 Definition of Done.
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-all ${EXAMPLES_DIR} cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} all-examples-ok:
|
|
Should Contain ${result.stdout} actor-imported:local/assistants-file_analyzer
|
|
Should Contain ${result.stdout} actor-imported:local/strategy_with_subplan
|
|
Should Contain ${result.stdout} actor-imported:local/estimator
|
|
Should Contain ${result.stdout} actor-imported:local/code-review
|
|
Should Contain ${result.stdout} actor-imported:utilities/file_operations
|
|
Should Contain ${result.stdout} actor-imported:workflows/document_processor
|
|
Should Contain ${result.stdout} actor-imported:workflows/test_driven_dev
|
|
Should Contain ${result.stdout} actor-imported:workflows/dev-workflow
|