brent.edwards
51a6e76e9e
CI / lint (pull_request) Successful in 16s
CI / typecheck (pull_request) Successful in 28s
CI / security (pull_request) Successful in 21s
CI / quality (pull_request) Successful in 14s
CI / integration_tests (pull_request) Failing after 6m34s
CI / unit_tests (pull_request) Successful in 6m46s
CI / build (pull_request) Successful in 15s
CI / docker (pull_request) Successful in 8s
CI / coverage (pull_request) Successful in 5m8s
Three fixes targeting CI integration_tests failures (18 failures on e8aa5ac):
1. Merge duplicate *** Settings *** blocks in 14 robot files into single
blocks. Multiple Settings sections are non-standard RF practice and
may cause resource import failures in certain Robot Framework versions
or CI environments.
2. Replace bare 'python' with ${PYTHON} variable in all Run Process
calls (14 files). Noxfile now passes --variable PYTHON:<venv-path>
to robot so tests use the venv interpreter regardless of PATH. This
fixes '/usr/local/bin/python: No module named cleveragents' on CI.
3. Add comprehensive CI debug output in noxfile.py: file existence
checks for .resource files, PATH/Python resolution, fixture dir
checks, and RF version. This will diagnose any remaining resource
import issues.
Also: remove hardcoded '/app/src' sys.path.insert in
system_prompt_template_rendering.robot (not portable to CI), and
add trailing newline to common.resource.
All 204 tests pass locally (4 excluded: 2 slow, 2 discovery).
201 lines
10 KiB
Plaintext
201 lines
10 KiB
Plaintext
*** Settings ***
|
|
Documentation Integration tests for LLM system prompt template rendering with runtime context
|
|
Library OperatingSystem
|
|
Library String
|
|
Library Collections
|
|
Library Process
|
|
Resource ${CURDIR}/v2_paths.resource
|
|
|
|
*** Variables ***
|
|
${PYTHON} python
|
|
${TEMP_DIR} /tmp/cleveragents_test
|
|
${TEST_CONTEXT} system_prompt_template_test
|
|
|
|
*** Test Cases ***
|
|
System Prompt Template Variables Are Rendered With Runtime Context
|
|
[Documentation] Verify that JINJA2 templates in system prompts are rendered with actual runtime context
|
|
[Tags] integration llm template system_prompt
|
|
|
|
# Create a simple config with templated system prompt
|
|
${config_content}= Catenate SEPARATOR=
|
|
... cleveragents:\n
|
|
... ${SPACE}${SPACE}version: "3.0"\n
|
|
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
|
|
...
|
|
|
|
... actors:\n
|
|
... ${SPACE}${SPACE}filter_actor:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Topic: {{ context.topic | tojson }}, List: {{ context.items | length }}"\n
|
|
...
|
|
|
|
... routes:\n
|
|
... ${SPACE}${SPACE}main:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
|
|
... \n
|
|
... merges:\n
|
|
... ${SPACE}${SPACE}- sources: [__input__]\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}target: main\n
|
|
|
|
Create File ${TEMP_DIR}/template_test.yaml ${config_content}
|
|
|
|
# Create context with test data
|
|
${context_data}= Set Variable {"topic": "Artificial Intelligence", "audience": "researchers"}
|
|
Create File ${TEMP_DIR}/test_context.json ${context_data}
|
|
|
|
# Run the agent with --load-context - the LLM should receive a system prompt with rendered values
|
|
# Since we can't easily intercept the actual LLM call, we verify the config was loaded correctly
|
|
${result}= Run Process ${PYTHON} -m cleveragents actor run -c ${TEMP_DIR}/template_test.yaml --unsafe --allow-rxpy-in-run-mode --load-context ${TEMP_DIR}/test_context.json -p "Analyze this"
|
|
... shell=False timeout=30s
|
|
|
|
# The agent should have processed successfully (even if it times out, the config should load)
|
|
Should Not Contain ${result.stderr} TemplateError
|
|
Should Not Contain ${result.stderr} Failed to render
|
|
Remove File ${TEMP_DIR}/template_test.yaml
|
|
Remove File ${TEMP_DIR}/test_context.json
|
|
|
|
Config Parser Preserves Template Syntax During YAML Loading
|
|
[Documentation] Verify that template markers are not prematurely rendered during config parsing
|
|
[Tags] integration config template
|
|
|
|
# Create config with templates
|
|
${config_content}= Catenate SEPARATOR=
|
|
... cleveragents:\n
|
|
... ${SPACE}${SPACE}version: "3.0"\n
|
|
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
|
|
... \n
|
|
... actors:\n
|
|
... ${SPACE}${SPACE}actor1:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Topic: {{ context.topic }}"\n
|
|
... ${SPACE}${SPACE}actor2:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Data: {{ context.data }}"\n
|
|
... \n
|
|
... routes:\n
|
|
... ${SPACE}${SPACE}main:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
|
|
|
|
Create File ${TEMP_DIR}/preserve_test.yaml ${config_content}
|
|
|
|
# The config should load without errors and preserve templates
|
|
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/preserve_test.yaml')]); agent1_prompt = config.agents['actor1'].config.get('system_prompt', ''); print('PROMPT1:', agent1_prompt); assert '{{' in agent1_prompt and '}}' in agent1_prompt, f'Templates not preserved: {agent1_prompt}'
|
|
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=10s
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} PROMPT1:
|
|
Should Contain ${result.stdout} {{ context.topic }}
|
|
|
|
# Clean up
|
|
Remove File ${TEMP_DIR}/preserve_test.yaml
|
|
|
|
Nested Context Variables In System Prompts
|
|
[Documentation] Verify that nested context variables like context.paper_details.topic work correctly
|
|
[Tags] integration llm template nested
|
|
|
|
# Create config with nested template variables
|
|
${config_content}= Catenate SEPARATOR=
|
|
... cleveragents:\n
|
|
... ${SPACE}${SPACE}version: "3.0"\n
|
|
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
|
|
... ${SPACE}${SPACE}unsafe: true\n
|
|
... \n
|
|
... actors:\n
|
|
... ${SPACE}${SPACE}nested_actor:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: |\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}Paper Details:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- Topic: {{ context.paper_details.topic }}\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- Length: {{ context.paper_details.length }}\n
|
|
... \n
|
|
... routes:\n
|
|
... ${SPACE}${SPACE}main:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}operators:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- type: map\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}params:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}actor: nested_actor\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
|
|
... \n
|
|
... merges:\n
|
|
... ${SPACE}${SPACE}- sources: [__input__]\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}target: main\n
|
|
|
|
Create File ${TEMP_DIR}/nested_test.yaml ${config_content}
|
|
|
|
# Create nested context
|
|
${context_data}= Set Variable {"paper_details": {"topic": "Machine Learning", "length": "5000 words"}}
|
|
Create File ${TEMP_DIR}/nested_context.json ${context_data}
|
|
|
|
# Verify the config loads and preserves templates
|
|
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/nested_test.yaml')]); prompt = config.agents['nested_actor'].config.get('system_prompt', ''); print('NESTED_PROMPT:', prompt); assert '{{ context.paper_details.topic }}' in prompt, f'Nested template not preserved: {prompt}'
|
|
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=10s
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} {{ context.paper_details.topic }}
|
|
|
|
# Clean up
|
|
Remove File ${TEMP_DIR}/nested_test.yaml
|
|
Remove File ${TEMP_DIR}/nested_context.json
|
|
|
|
JINJA2 Filters In System Prompts Are Preserved
|
|
[Documentation] Verify that JINJA2 filters like tojson are preserved during config loading
|
|
[Tags] integration template jinja2 filters
|
|
|
|
# Create config with JINJA2 filters
|
|
${config_content}= Catenate SEPARATOR=
|
|
... cleveragents:\n
|
|
... ${SPACE}${SPACE}version: "3.0"\n
|
|
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
|
|
... \n
|
|
... actors:\n
|
|
... ${SPACE}${SPACE}filter_actor:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Topic: {{ context.topic | tojson }}, List: {{ context.items | length }}"\n
|
|
... \n
|
|
... routes:\n
|
|
... ${SPACE}${SPACE}main:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
|
|
|
|
Create File ${TEMP_DIR}/filter_test.yaml ${config_content}
|
|
|
|
# Verify filters are preserved
|
|
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/filter_test.yaml')]); prompt = config.agents['filter_actor'].config.get('system_prompt', ''); print('FILTER_PROMPT:', prompt); assert '| tojson' in prompt and '| length' in prompt, f'Filters not preserved: {prompt}'
|
|
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=10s
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} | tojson
|
|
Should Contain ${result.stdout} | length
|
|
|
|
# Clean up
|
|
Remove File ${TEMP_DIR}/filter_test.yaml
|
|
|
|
*** Keywords ***
|
|
Suite Setup
|
|
Create Directory ${TEMP_DIR}
|
|
|
|
Suite Teardown
|
|
Remove Directory ${TEMP_DIR} recursive=True
|