forked from cleveragents/cleveragents-core
brent.edwards
51a6e76e9e
Three fixes targeting CI integration_tests failures (18 failures on e8aa5ac):
1. Merge duplicate *** Settings *** blocks in 14 robot files into single
blocks. Multiple Settings sections are non-standard RF practice and
may cause resource import failures in certain Robot Framework versions
or CI environments.
2. Replace bare 'python' with ${PYTHON} variable in all Run Process
calls (14 files). Noxfile now passes --variable PYTHON:<venv-path>
to robot so tests use the venv interpreter regardless of PATH. This
fixes '/usr/local/bin/python: No module named cleveragents' on CI.
3. Add comprehensive CI debug output in noxfile.py: file existence
checks for .resource files, PATH/Python resolution, fixture dir
checks, and RF version. This will diagnose any remaining resource
import issues.
Also: remove hardcoded '/app/src' sys.path.insert in
system_prompt_template_rendering.robot (not portable to CI), and
add trailing newline to common.resource.
All 204 tests pass locally (4 excluded: 2 slow, 2 discovery).
352 lines
12 KiB
Plaintext
352 lines
12 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration tests for RxPY route validation in run command
|
|
Library Process
|
|
Library OperatingSystem
|
|
Library String
|
|
Library Collections
|
|
Resource ${CURDIR}/v2_paths.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${PYTHON} python
|
|
${TEST_DIR} ${TEMPDIR}/rxpy_validation_test
|
|
${RXPY_CONFIG} ${TEST_DIR}/rxpy_config.yaml
|
|
${LANGGRAPH_CONFIG} ${TEST_DIR}/langgraph_config.yaml
|
|
${CONTEXT_DIR} ${TEST_DIR}/test_contexts
|
|
${UNIQUE_ID} ${EMPTY}
|
|
${RXPY_RUN_MODE_ERROR} RxPY stream routes are only supported in run mode unless allowed. Use --allow-rxpy-in-run-mode to bypass.
|
|
|
|
*** Test Cases ***
|
|
Test Run Command Rejects RxPY Routes
|
|
[Documentation] Verify run command rejects RxPY stream routes
|
|
[Tags] rxpy validation run
|
|
|
|
Create RxPY Config File
|
|
|
|
# Try to run with RxPY routes - should fail
|
|
${result} = Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${RXPY_CONFIG}
|
|
... -p test
|
|
... --unsafe
|
|
... stderr=STDOUT timeout=30s
|
|
|
|
Should Not Be Equal As Integers ${result.rc} 0 Command should have failed
|
|
Should Contain ${result.stdout} ${RXPY_RUN_MODE_ERROR}
|
|
Should Contain ${result.stdout} allow-rxpy-in-run-mode
|
|
|
|
Test Run Command With Context Does Not Create Context
|
|
[Documentation] Verify that no context is created when RxPY validation fails
|
|
[Tags] rxpy validation context
|
|
|
|
${context_name} = Set Variable rxpy_test_ctx_${UNIQUE_ID}
|
|
|
|
Create RxPY Config File
|
|
|
|
# Ensure context doesn't exist
|
|
Delete Context If Exists ${context_name}
|
|
|
|
# Try to run with --context flag
|
|
${result} = Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${RXPY_CONFIG}
|
|
... -p test
|
|
... --unsafe
|
|
... --context ${context_name}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... stderr=STDOUT timeout=30s
|
|
|
|
Should Not Be Equal As Integers ${result.rc} 0 Command should have failed
|
|
Should Contain ${result.stdout} ${RXPY_RUN_MODE_ERROR}
|
|
|
|
# Verify context was NOT created
|
|
${context_exists} = Context Exists ${context_name}
|
|
Should Not Be True ${context_exists} Context should not have been created
|
|
|
|
Test Run Command With LangGraph Routes Succeeds
|
|
[Documentation] Verify run command works with LangGraph routes
|
|
[Tags] langgraph validation run slow
|
|
|
|
Create LangGraph Config File
|
|
|
|
# Run with LangGraph routes - should succeed
|
|
${context_name} = Set Variable rxpy_test_ctx_${UNIQUE_ID}
|
|
${result} = Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${LANGGRAPH_CONFIG}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... -p test
|
|
... stderr=STDOUT timeout=30s
|
|
|
|
# For now, might fail on actual execution but shouldn't have route validation error
|
|
${output} = Set Variable ${result.stdout}
|
|
Should Not Contain ${output} ${RXPY_RUN_MODE_ERROR}
|
|
|
|
Test Run Command Accepts RxPY Routes When Allowed
|
|
[Documentation] Verify actor run works with RxPY routes when allowed
|
|
[Tags] rxpy run slow
|
|
|
|
Create RxPY Config File
|
|
|
|
${context_name} = Set Variable rxpy_test_ctx_${UNIQUE_ID}
|
|
${result} = Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${RXPY_CONFIG}
|
|
... -p test
|
|
... --unsafe
|
|
... --allow-rxpy-in-run-mode
|
|
... --context ${context_name}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... stderr=STDOUT
|
|
... timeout=30s
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Not Contain ${result.stdout} ${RXPY_RUN_MODE_ERROR}
|
|
|
|
Test Error Message Content
|
|
[Documentation] Verify error message contains helpful information
|
|
[Tags] rxpy validation error-message
|
|
|
|
Create RxPY Config File
|
|
|
|
${result} = Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${RXPY_CONFIG}
|
|
... -p test
|
|
... --unsafe
|
|
... stderr=STDOUT timeout=30s
|
|
|
|
Should Not Be Equal As Integers ${result.rc} 0
|
|
|
|
# Check error message contains all important information
|
|
${output} = Convert To Lower Case ${result.stdout}
|
|
Should Contain ${output} nested
|
|
Should Contain ${output} completion detection
|
|
Should Contain ${output} run mode
|
|
Should Contain ${output} langgraph
|
|
|
|
Test Run With Nested Switch Operators
|
|
[Documentation] Verify run command rejects configs with nested switch operators
|
|
[Tags] rxpy nested validation
|
|
|
|
Create Nested Switch Config File
|
|
|
|
${result} = Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${TEST_DIR}/nested_switch_config.yaml
|
|
... -p test
|
|
... --unsafe
|
|
... stderr=STDOUT timeout=30s
|
|
|
|
Should Not Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} ${RXPY_RUN_MODE_ERROR}
|
|
|
|
Test Mixed Route Types Detection
|
|
[Documentation] Verify detection works with mix of RxPY and LangGraph routes
|
|
[Tags] rxpy mixed validation
|
|
|
|
Create Mixed Routes Config File
|
|
|
|
# Should detect RxPY routes even if LangGraph routes also present
|
|
${result} = Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${TEST_DIR}/mixed_config.yaml
|
|
... -p test
|
|
... --unsafe
|
|
... stderr=STDOUT timeout=30s
|
|
|
|
Should Not Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} ${RXPY_RUN_MODE_ERROR}
|
|
|
|
Test Context File Not Created On Multiple Runs
|
|
[Documentation] Verify repeated failed runs do not create context
|
|
[Tags] rxpy context validation
|
|
|
|
${context_name} = Set Variable rxpy_multi_test_${UNIQUE_ID}
|
|
|
|
Create RxPY Config File
|
|
Delete Context If Exists ${context_name}
|
|
|
|
# Run multiple times
|
|
FOR ${i} IN RANGE 3
|
|
${result} = Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${RXPY_CONFIG}
|
|
... -p test
|
|
... --unsafe
|
|
... --context ${context_name}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... stderr=STDOUT timeout=30s
|
|
|
|
Should Not Be Equal As Integers ${result.rc} 0
|
|
END
|
|
|
|
# Verify context still doesn't exist after multiple attempts
|
|
${context_exists} = Context Exists ${context_name}
|
|
Should Not Be True ${context_exists}
|
|
|
|
*** Keywords ***
|
|
Setup Test Environment
|
|
[Documentation] Setup test environment
|
|
Create Directory ${TEST_DIR}
|
|
Create Directory ${CONTEXT_DIR}
|
|
|
|
# Generate unique ID for this test run
|
|
${timestamp} = Get Time epoch
|
|
Set Suite Variable ${UNIQUE_ID} ${timestamp}
|
|
|
|
Cleanup Test Environment
|
|
[Documentation] Clean up test environment
|
|
Remove Directory ${TEST_DIR} recursive=True
|
|
|
|
Create RxPY Config File
|
|
[Documentation] Create a test config with RxPY stream routes
|
|
${config} = Catenate SEPARATOR=\n
|
|
... actors:
|
|
... ${SPACE*2}echo_actor:
|
|
... ${SPACE*4}type: tool
|
|
... ${SPACE*4}config:
|
|
... ${SPACE*6}tools:
|
|
... ${SPACE*8}- echo
|
|
... ${SPACE*6}safe_mode: true
|
|
... ${EMPTY}
|
|
... routes:
|
|
... ${SPACE*2}echo_stream:
|
|
... ${SPACE*4}type: stream
|
|
... ${SPACE*4}stream_type: cold
|
|
... ${SPACE*4}operators:
|
|
... ${SPACE*6}- type: map
|
|
... ${SPACE*8}params:
|
|
... ${SPACE*10}actor: echo_actor
|
|
... ${SPACE*4}publications:
|
|
... ${SPACE*6}- __output__
|
|
... ${EMPTY}
|
|
... merges:
|
|
... ${SPACE*2}- sources: [__input__]
|
|
... ${SPACE*4}target: echo_stream
|
|
|
|
Create File ${RXPY_CONFIG} ${config}
|
|
|
|
Create LangGraph Config File
|
|
[Documentation] Create a test config with LangGraph routes
|
|
${config} = Catenate SEPARATOR=\n
|
|
... actors:
|
|
... ${SPACE*2}echo_actor:
|
|
... ${SPACE*4}type: tool
|
|
... ${SPACE*4}config:
|
|
... ${SPACE*6}tools:
|
|
... ${SPACE*8}- echo
|
|
... ${SPACE*6}safe_mode: true
|
|
... ${EMPTY}
|
|
... routes:
|
|
... ${SPACE*2}main:
|
|
... ${SPACE*4}type: graph
|
|
... ${SPACE*4}nodes:
|
|
... ${SPACE*6}- id: start
|
|
... ${SPACE*8}actor: echo_actor
|
|
... ${SPACE*4}edges:
|
|
... ${SPACE*6}- from: __input__
|
|
... ${SPACE*8}to: start
|
|
... ${SPACE*6}- from: start
|
|
... ${SPACE*8}to: __output__
|
|
... ${EMPTY}
|
|
... merges:
|
|
... ${SPACE*2}- sources: [__input__]
|
|
... ${SPACE*4}target: main
|
|
|
|
Create File ${LANGGRAPH_CONFIG} ${config}
|
|
|
|
Create Nested Switch Config File
|
|
[Documentation] Create a config with nested switch operators
|
|
${config} = Catenate SEPARATOR=\n
|
|
... actors:
|
|
... ${SPACE*2}test_actor:
|
|
... ${SPACE*4}type: tool
|
|
... ${SPACE*4}config:
|
|
... ${SPACE*6}tools:
|
|
... ${SPACE*8}- echo
|
|
... ${SPACE*6}safe_mode: true
|
|
... ${EMPTY}
|
|
... routes:
|
|
... ${SPACE*2}main:
|
|
... ${SPACE*4}type: stream
|
|
... ${SPACE*4}stream_type: cold
|
|
... ${SPACE*4}operators:
|
|
... ${SPACE*6}- type: switch
|
|
... ${SPACE*8}params:
|
|
... ${SPACE*10}cases:
|
|
... ${SPACE*12}- condition: "test"
|
|
... ${SPACE*14}route: nested_route
|
|
... ${SPACE*2}nested_route:
|
|
... ${SPACE*4}type: stream
|
|
... ${SPACE*4}stream_type: cold
|
|
... ${SPACE*4}operators:
|
|
... ${SPACE*6}- type: map
|
|
... ${SPACE*8}params:
|
|
... ${SPACE*10}actor: test_actor
|
|
... ${SPACE*4}publications:
|
|
... ${SPACE*6}- __output__
|
|
... ${EMPTY}
|
|
... merges:
|
|
... ${SPACE*2}- sources: [__input__]
|
|
... ${SPACE*4}target: main
|
|
|
|
Create File ${TEST_DIR}/nested_switch_config.yaml ${config}
|
|
|
|
Create Mixed Routes Config File
|
|
[Documentation] Create a config with both RxPY and LangGraph routes
|
|
${config} = Catenate SEPARATOR=\n
|
|
... actors:
|
|
... ${SPACE*2}test_actor:
|
|
... ${SPACE*4}type: tool
|
|
... ${SPACE*4}config:
|
|
... ${SPACE*6}tools:
|
|
... ${SPACE*8}- echo
|
|
... ${SPACE*6}safe_mode: true
|
|
... ${EMPTY}
|
|
... routes:
|
|
... ${SPACE*2}stream_route:
|
|
... ${SPACE*4}type: stream
|
|
... ${SPACE*4}stream_type: cold
|
|
... ${SPACE*4}operators:
|
|
... ${SPACE*6}- type: map
|
|
... ${SPACE*8}params:
|
|
... ${SPACE*10}actor: test_actor
|
|
... ${SPACE*4}publications:
|
|
... ${SPACE*6}- __output__
|
|
... ${SPACE*2}graph_route:
|
|
... ${SPACE*4}type: graph
|
|
... ${SPACE*4}entry_point: start
|
|
... ${SPACE*4}nodes:
|
|
... ${SPACE*6}start:
|
|
... ${SPACE*8}type: actor
|
|
... ${SPACE*8}actor: test_actor
|
|
... ${SPACE*4}edges:
|
|
... ${SPACE*6}- source: start
|
|
... ${SPACE*8}target: end
|
|
... ${EMPTY}
|
|
... merges:
|
|
... ${SPACE*2}- sources: [__input__]
|
|
... ${SPACE*4}target: stream_route
|
|
|
|
Create File ${TEST_DIR}/mixed_config.yaml ${config}
|
|
|
|
Delete Context If Exists
|
|
[Documentation] Delete a context if it exists
|
|
[Arguments] ${context_name}
|
|
|
|
${result} = Run Process ${PYTHON} -m cleveragents context delete
|
|
... ${context_name}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... --yes
|
|
... stderr=STDOUT
|
|
|
|
# Don't fail if context doesn't exist
|
|
Log Deleted context ${context_name} (if it existed)
|
|
|
|
Context Exists
|
|
[Documentation] Check if a context exists
|
|
[Arguments] ${context_name}
|
|
|
|
${result} = Run Process ${PYTHON} -m cleveragents context show
|
|
... ${context_name}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... stderr=STDOUT
|
|
|
|
# If return code is 0, context exists
|
|
${exists} = Evaluate ${result.rc} == 0
|
|
RETURN ${exists}
|