freemo
b122ec7ed5
CI / lint (pull_request) Successful in 23s
CI / quality (pull_request) Successful in 51s
CI / typecheck (pull_request) Successful in 1m0s
CI / security (pull_request) Successful in 55s
CI / build (pull_request) Successful in 25s
CI / helm (pull_request) Successful in 32s
CI / push-validation (pull_request) Successful in 26s
CI / e2e_tests (pull_request) Successful in 3m38s
CI / integration_tests (pull_request) Successful in 6m42s
CI / unit_tests (pull_request) Successful in 8m19s
CI / docker (pull_request) Successful in 13s
CI / coverage (pull_request) Successful in 15m23s
CI / status-check (pull_request) Successful in 2s
Remove the local ${PYTHON} python (and python3) variable definitions from
the *** Variables *** sections of all affected robot files. These local
definitions were overriding the pabot-injected venv Python path passed via
--variable PYTHON:/path/to/venv/python, causing tests to use the system
Python (which lacks required packages like structlog, sqlalchemy, etc.)
instead of the nox venv Python.
The correct ${PYTHON} value is already set by Setup Test Environment in
common.resource via sys.executable, and pabot passes it via --variable.
The local fallback definitions are redundant and harmful in parallel runs.
Audit found 56 robot files with the pattern (more than the 9 originally
identified in the issue). All occurrences have been removed.
ISSUES CLOSED: #1309
200 lines
10 KiB
Plaintext
200 lines
10 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration tests for LLM system prompt template rendering with runtime context
|
|
Library OperatingSystem
|
|
Library String
|
|
Library Collections
|
|
Library Process
|
|
Resource ${CURDIR}/v2_paths.resource
|
|
|
|
*** Variables ***
|
|
${TEMP_DIR} /tmp/cleveragents_test
|
|
${TEST_CONTEXT} system_prompt_template_test
|
|
|
|
*** Test Cases ***
|
|
System Prompt Template Variables Are Rendered With Runtime Context
|
|
[Documentation] Verify that JINJA2 templates in system prompts are rendered with actual runtime context
|
|
[Tags] integration llm template system_prompt
|
|
|
|
# Create a simple config with templated system prompt
|
|
${config_content}= Catenate SEPARATOR=
|
|
... cleveragents:\n
|
|
... ${SPACE}${SPACE}version: "3.0"\n
|
|
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
|
|
...
|
|
|
|
... actors:\n
|
|
... ${SPACE}${SPACE}filter_actor:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Topic: {{ context.topic | tojson }}, List: {{ context.items | length }}"\n
|
|
...
|
|
|
|
... routes:\n
|
|
... ${SPACE}${SPACE}main:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
|
|
... \n
|
|
... merges:\n
|
|
... ${SPACE}${SPACE}- sources: [__input__]\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}target: main\n
|
|
|
|
Create File ${TEMP_DIR}/template_test.yaml ${config_content}
|
|
|
|
# Create context with test data
|
|
${context_data}= Set Variable {"topic": "Artificial Intelligence", "audience": "researchers"}
|
|
Create File ${TEMP_DIR}/test_context.json ${context_data}
|
|
|
|
# Run the agent with --load-context - the LLM should receive a system prompt with rendered values
|
|
# Since we can't easily intercept the actual LLM call, we verify the config was loaded correctly
|
|
${result}= Run Process ${PYTHON} -m cleveragents actor run -c ${TEMP_DIR}/template_test.yaml --unsafe --allow-rxpy-in-run-mode --load-context ${TEMP_DIR}/test_context.json test-actor "Analyze this"
|
|
... shell=False timeout=120s on_timeout=kill
|
|
|
|
# The agent should have processed successfully (even if it times out, the config should load)
|
|
Should Not Contain ${result.stderr} TemplateError
|
|
Should Not Contain ${result.stderr} Failed to render
|
|
Remove File ${TEMP_DIR}/template_test.yaml
|
|
Remove File ${TEMP_DIR}/test_context.json
|
|
|
|
Config Parser Preserves Template Syntax During YAML Loading
|
|
[Documentation] Verify that template markers are not prematurely rendered during config parsing
|
|
[Tags] integration config template
|
|
|
|
# Create config with templates
|
|
${config_content}= Catenate SEPARATOR=
|
|
... cleveragents:\n
|
|
... ${SPACE}${SPACE}version: "3.0"\n
|
|
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
|
|
... \n
|
|
... actors:\n
|
|
... ${SPACE}${SPACE}actor1:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Topic: {{ context.topic }}"\n
|
|
... ${SPACE}${SPACE}actor2:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Data: {{ context.data }}"\n
|
|
... \n
|
|
... routes:\n
|
|
... ${SPACE}${SPACE}main:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
|
|
|
|
Create File ${TEMP_DIR}/preserve_test.yaml ${config_content}
|
|
|
|
# The config should load without errors and preserve templates
|
|
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/preserve_test.yaml')]); agent1_prompt = config.agents['actor1'].config.get('system_prompt', ''); print('PROMPT1:', agent1_prompt); assert '{{' in agent1_prompt and '}}' in agent1_prompt, f'Templates not preserved: {agent1_prompt}'
|
|
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} PROMPT1:
|
|
Should Contain ${result.stdout} {{ context.topic }}
|
|
|
|
# Clean up
|
|
Remove File ${TEMP_DIR}/preserve_test.yaml
|
|
|
|
Nested Context Variables In System Prompts
|
|
[Documentation] Verify that nested context variables like context.paper_details.topic work correctly
|
|
[Tags] integration llm template nested
|
|
|
|
# Create config with nested template variables
|
|
${config_content}= Catenate SEPARATOR=
|
|
... cleveragents:\n
|
|
... ${SPACE}${SPACE}version: "3.0"\n
|
|
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
|
|
... ${SPACE}${SPACE}unsafe: true\n
|
|
... \n
|
|
... actors:\n
|
|
... ${SPACE}${SPACE}nested_actor:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: |\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}Paper Details:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- Topic: {{ context.paper_details.topic }}\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- Length: {{ context.paper_details.length }}\n
|
|
... \n
|
|
... routes:\n
|
|
... ${SPACE}${SPACE}main:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}operators:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- type: map\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}params:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}actor: nested_actor\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
|
|
... \n
|
|
... merges:\n
|
|
... ${SPACE}${SPACE}- sources: [__input__]\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}target: main\n
|
|
|
|
Create File ${TEMP_DIR}/nested_test.yaml ${config_content}
|
|
|
|
# Create nested context
|
|
${context_data}= Set Variable {"paper_details": {"topic": "Machine Learning", "length": "5000 words"}}
|
|
Create File ${TEMP_DIR}/nested_context.json ${context_data}
|
|
|
|
# Verify the config loads and preserves templates
|
|
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/nested_test.yaml')]); prompt = config.agents['nested_actor'].config.get('system_prompt', ''); print('NESTED_PROMPT:', prompt); assert '{{ context.paper_details.topic }}' in prompt, f'Nested template not preserved: {prompt}'
|
|
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} {{ context.paper_details.topic }}
|
|
|
|
# Clean up
|
|
Remove File ${TEMP_DIR}/nested_test.yaml
|
|
Remove File ${TEMP_DIR}/nested_context.json
|
|
|
|
JINJA2 Filters In System Prompts Are Preserved
|
|
[Documentation] Verify that JINJA2 filters like tojson are preserved during config loading
|
|
[Tags] integration template jinja2 filters
|
|
|
|
# Create config with JINJA2 filters
|
|
${config_content}= Catenate SEPARATOR=
|
|
... cleveragents:\n
|
|
... ${SPACE}${SPACE}version: "3.0"\n
|
|
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
|
|
... \n
|
|
... actors:\n
|
|
... ${SPACE}${SPACE}filter_actor:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Topic: {{ context.topic | tojson }}, List: {{ context.items | length }}"\n
|
|
... \n
|
|
... routes:\n
|
|
... ${SPACE}${SPACE}main:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
|
|
|
|
Create File ${TEMP_DIR}/filter_test.yaml ${config_content}
|
|
|
|
# Verify filters are preserved
|
|
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/filter_test.yaml')]); prompt = config.agents['filter_actor'].config.get('system_prompt', ''); print('FILTER_PROMPT:', prompt); assert '| tojson' in prompt and '| length' in prompt, f'Filters not preserved: {prompt}'
|
|
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=120s on_timeout=kill
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} | tojson
|
|
Should Contain ${result.stdout} | length
|
|
|
|
# Clean up
|
|
Remove File ${TEMP_DIR}/filter_test.yaml
|
|
|
|
*** Keywords ***
|
|
Suite Setup
|
|
Create Directory ${TEMP_DIR}
|
|
|
|
Suite Teardown
|
|
Remove Directory ${TEMP_DIR} recursive=True
|