aditya
2764fcef5c
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 16s
CI / build (pull_request) Successful in 25s
CI / quality (pull_request) Successful in 34s
CI / typecheck (pull_request) Successful in 39s
CI / security (pull_request) Successful in 52s
CI / unit_tests (pull_request) Successful in 2m51s
CI / integration_tests (pull_request) Successful in 3m41s
CI / docker (pull_request) Successful in 56s
CI / e2e_tests (pull_request) Successful in 3m56s
CI / coverage (pull_request) Successful in 6m28s
CI / benchmark-regression (pull_request) Successful in 38m39s
Address review-driven fixes across actor schema, preflight guardrails, docs/examples, and Behave/Robot coverage: unify preflight warning behavior with shared role-warning logic, resolve actor-name to config payloads in production preflight flow, harden response_format validation/coercion edge cases, extract duplicated helper logic, and expand negative-path test coverage. Also fix cross-scenario patcher leakage in step modules to eliminate full-run-only coverage failures.
148 lines
8.2 KiB
Plaintext
148 lines
8.2 KiB
Plaintext
*** Settings ***
|
|
Documentation Smoke tests for Actor YAML schema validation
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_actor_schema.py
|
|
|
|
*** Test Cases ***
|
|
Validate Simple LLM Actor YAML
|
|
[Documentation] Parse the simple_llm example actor YAML and assert success
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${CURDIR}/../examples/actors/simple_llm.yaml cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-ok
|
|
|
|
Validate LLM With Tools Actor YAML
|
|
[Documentation] Parse the llm_with_tools example actor YAML and assert success
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${CURDIR}/../examples/actors/llm_with_tools.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-ok
|
|
|
|
Validate Tool Collection Actor YAML
|
|
[Documentation] Parse the tool_collection example actor YAML and assert success
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${CURDIR}/../examples/actors/tool_collection.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-ok
|
|
|
|
Validate Simple Graph Actor YAML
|
|
[Documentation] Parse the simple_graph example actor YAML and assert success
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${CURDIR}/../examples/actors/simple_graph.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-ok
|
|
|
|
Validate Complex Graph Workflow Actor YAML
|
|
[Documentation] Parse the graph_workflow example actor YAML and assert success
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate ${CURDIR}/../examples/actors/graph_workflow.yaml cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-ok
|
|
|
|
Reject Actor YAML Without Namespace
|
|
[Documentation] Attempt to parse actor YAML with invalid name (no namespace)
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_actor_no_namespace.yaml
|
|
Create File ${invalid_yaml} name: simple_actor\ntype: llm\ndescription: Test\nmodel: gpt-4\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-expected-fail
|
|
Should Contain ${result.stdout} namespaced
|
|
|
|
Reject LLM Actor Without Model
|
|
[Documentation] Attempt to parse LLM actor without required model field
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_llm_no_model.yaml
|
|
Create File ${invalid_yaml} name: test/actor\ntype: llm\ndescription: Missing model\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-expected-fail
|
|
Should Contain ${result.stdout} model
|
|
|
|
Reject TOOL Actor Without Tools
|
|
[Documentation] Attempt to parse TOOL actor without tools list
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_tool_no_tools.yaml
|
|
Create File ${invalid_yaml} name: test/tools\ntype: tool\ndescription: Missing tools\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-expected-fail
|
|
Should Contain ${result.stdout} tool
|
|
|
|
Reject GRAPH Actor Without Route
|
|
[Documentation] Attempt to parse GRAPH actor without route definition
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_graph_no_route.yaml
|
|
Create File ${invalid_yaml} name: test/workflow\ntype: graph\ndescription: Missing route\nmodel: gpt-4\n
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-expected-fail
|
|
Should Contain ${result.stdout} route
|
|
|
|
Reject Graph With Duplicate Node IDs
|
|
[Documentation] Attempt to parse graph with duplicate node IDs
|
|
[Tags] slow
|
|
${invalid_yaml}= Set Variable ${TEMPDIR}${/}invalid_graph_duplicate_nodes.yaml
|
|
${yaml_content}= Catenate SEPARATOR=\n
|
|
... name: test/duplicate
|
|
... type: graph
|
|
... description: Duplicate nodes
|
|
... model: gpt-4
|
|
... route:
|
|
... ${SPACE}${SPACE}nodes:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}- id: node1
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}type: agent
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}name: First
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}description: First node
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}config:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}prompt: Test
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}- id: node1
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}type: agent
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}name: Duplicate
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}description: Duplicate ID
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}config:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}prompt: Test
|
|
... ${SPACE}${SPACE}edges: []
|
|
... ${SPACE}${SPACE}entry_node: node1
|
|
... ${SPACE}${SPACE}exit_nodes: [node1]
|
|
Create File ${invalid_yaml} ${yaml_content}
|
|
${result}= Run Process ${PYTHON} ${HELPER} validate-invalid ${invalid_yaml} cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-schema-expected-fail
|
|
Should Contain ${result.stdout} Duplicate
|
|
|
|
Role Warnings For ActorConfigSchema Missing Response Format
|
|
[Documentation] Verify actor_role_warnings emits response_format warning for ActorConfigSchema estimation actor without response_format.
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} role-warning-model dummy cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-role-warning-model-ok
|
|
|
|
Role Warnings For Payload Unrecognized Context View
|
|
[Documentation] Verify actor_role_warnings emits context_view warning when estimation payload uses an unrecognized context_view value.
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} role-warning-payload dummy cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-role-warning-payload-ok
|
|
|
|
Role Warnings Accept Uppercase Estimation Role Hint
|
|
[Documentation] Verify actor_role_warnings treats uppercase role_hint as estimation and still emits response_format warning when missing.
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} role-warning-uppercase dummy cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-role-warning-uppercase-ok
|
|
|
|
Role Warnings Log Unrecognized Role Hint Typo
|
|
[Documentation] Verify invalid role_hint typo emits schema warning log while actor_role_warnings remains non-fatal and returns no estimation warnings.
|
|
[Tags] slow
|
|
${result}= Run Process ${PYTHON} ${HELPER} role-warning-invalid-role-log dummy cwd=${WORKSPACE}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} actor-role-warning-invalid-role-log-ok
|