aditya
2764fcef5c
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 16s
CI / build (pull_request) Successful in 25s
CI / quality (pull_request) Successful in 34s
CI / typecheck (pull_request) Successful in 39s
CI / security (pull_request) Successful in 52s
CI / unit_tests (pull_request) Successful in 2m51s
CI / integration_tests (pull_request) Successful in 3m41s
CI / docker (pull_request) Successful in 56s
CI / e2e_tests (pull_request) Successful in 3m56s
CI / coverage (pull_request) Successful in 6m28s
CI / benchmark-regression (pull_request) Successful in 38m39s
Address review-driven fixes across actor schema, preflight guardrails, docs/examples, and Behave/Robot coverage: unify preflight warning behavior with shared role-warning logic, resolve actor-name to config payloads in production preflight flow, harden response_format validation/coercion edge cases, extract duplicated helper logic, and expand negative-path test coverage. Also fix cross-scenario patcher leakage in step modules to eliminate full-run-only coverage failures.
179 lines
9.5 KiB
Plaintext
179 lines
9.5 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration tests for actor YAML examples
|
|
... Validates that all actor example files load correctly
|
|
... and demonstrate expected patterns (strategist, executor,
|
|
... reviewer, tool-only, validation, graphs, hierarchical)
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_actor_examples.py
|
|
${EXAMPLES_DIR} ${CURDIR}/../examples/actors
|
|
${DOCS_DIR} ${CURDIR}/../docs/reference
|
|
|
|
*** Test Cases ***
|
|
Load All Example YAML Files
|
|
[Documentation] Verify all actor example YAML files can be loaded successfully
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} count-examples ${EXAMPLES_DIR} cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} example-count:
|
|
|
|
Validate Simple LLM Example
|
|
[Documentation] Load simple_llm.yaml and verify basic LLM actor structure
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/simple_llm.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:llm
|
|
Should Contain ${result.stdout} name:assistants/code_reviewer
|
|
|
|
Validate LLM With Tools Example
|
|
[Documentation] Load llm_with_tools.yaml and verify tools configuration
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/llm_with_tools.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:llm
|
|
Should Contain ${result.stdout} has-tools
|
|
|
|
Validate Estimator Example
|
|
[Documentation] Load estimator.yaml and verify estimation actor configuration
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/estimator.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:llm
|
|
Should Contain ${result.stdout} name:local/estimator
|
|
Should Contain ${result.stdout} has-tools
|
|
Should Contain ${result.stdout} has-skills
|
|
|
|
Validate Tool Collection Example
|
|
[Documentation] Load tool_collection.yaml and verify tool-only actor
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/tool_collection.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:tool
|
|
Should Contain ${result.stdout} has-tools
|
|
|
|
Validate Simple Graph Example
|
|
[Documentation] Load simple_graph.yaml and verify linear graph structure
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/simple_graph.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:graph
|
|
Should Contain ${result.stdout} has-route
|
|
|
|
Validate Complex Graph Workflow Example
|
|
[Documentation] Load graph_workflow.yaml and verify complex graph with conditionals
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${EXAMPLES_DIR}/graph_workflow.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-ok
|
|
Should Contain ${result.stdout} type:graph
|
|
Should Contain ${result.stdout} name:workflows/test_driven_dev
|
|
Should Contain ${result.stdout} has-route
|
|
Should Contain ${result.stdout} has-env-vars
|
|
|
|
Verify Strategist Pattern
|
|
[Documentation] Verify simple_llm.yaml follows strategist pattern (reviewer context)
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/simple_llm.yaml strategist cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} has-memory
|
|
Should Contain ${result.stdout} context-view:reviewer
|
|
|
|
Verify Tool-Only Pattern
|
|
[Documentation] Verify tool_collection.yaml follows tool-only pattern (no LLM)
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/tool_collection.yaml tool-only cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} pattern-ok:tool-only
|
|
Should Contain ${result.stdout} no-model
|
|
|
|
Verify Graph Pattern
|
|
[Documentation] Verify simple_graph.yaml follows graph workflow pattern
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/simple_graph.yaml graph cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} pattern-ok:graph
|
|
Should Contain ${result.stdout} has-entry-node
|
|
Should Contain ${result.stdout} has-exit-nodes
|
|
|
|
Verify Hierarchical Graph Pattern
|
|
[Documentation] Verify graph_workflow.yaml has hierarchical structure
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-pattern ${EXAMPLES_DIR}/graph_workflow.yaml hierarchical cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} pattern-ok:hierarchical
|
|
Should Contain ${result.stdout} has-subgraph
|
|
|
|
Verify Documentation References All Examples
|
|
[Documentation] Verify docs/reference/actors_examples.md references all example files
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-docs ${DOCS_DIR}/actors_examples.md cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} doc-references:simple_llm.yaml
|
|
Should Contain ${result.stdout} doc-references:llm_with_tools.yaml
|
|
Should Contain ${result.stdout} doc-references:estimator.yaml
|
|
Should Contain ${result.stdout} doc-references:tool_collection.yaml
|
|
Should Contain ${result.stdout} doc-references:simple_graph.yaml
|
|
Should Contain ${result.stdout} doc-references:graph_workflow.yaml
|
|
|
|
Verify Documentation Has Pattern Examples
|
|
[Documentation] Verify documentation includes all required actor patterns
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-doc-patterns ${DOCS_DIR}/actors_examples.md cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} has-pattern:strategist
|
|
Should Contain ${result.stdout} has-pattern:executor
|
|
Should Contain ${result.stdout} has-pattern:reviewer
|
|
Should Contain ${result.stdout} has-pattern:tool-only
|
|
Should Contain ${result.stdout} has-pattern:validation
|
|
Should Contain ${result.stdout} has-pattern:hierarchical
|
|
|
|
Reject Invalid Actor Name
|
|
[Documentation] Verify that actor with invalid name (no namespace) is rejected
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_no_namespace.yaml
|
|
Create File ${invalid_yaml} name: InvalidName\ntype: llm\nmodel: gpt-4\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} expected-fail
|
|
Should Contain ${result.stdout} namespace
|
|
|
|
Reject LLM Actor Without Model
|
|
[Documentation] Verify that LLM actor without model field is rejected
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_llm_no_model.yaml
|
|
Create File ${invalid_yaml} name: test/actor\ntype: llm\ndescription: Missing model\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} expected-fail
|
|
Should Contain ${result.stdout} model
|
|
|
|
Reject TOOL Actor Without Tools
|
|
[Documentation] Verify that TOOL actor without tools list is rejected
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_tool_no_tools.yaml
|
|
Create File ${invalid_yaml} name: test/tools\ntype: tool\ndescription: Missing tools\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} expected-fail
|
|
Should Contain ${result.stdout} tool
|
|
|
|
Reject GRAPH Actor Without Route
|
|
[Documentation] Verify that GRAPH actor without route definition is rejected
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_graph_no_route.yaml
|
|
Create File ${invalid_yaml} name: test/workflow\ntype: graph\nmodel: gpt-4\ndescription: Missing route\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} expected-fail
|
|
Should Contain ${result.stdout} route
|