HAL9000/temp
0
0
Fork 0
forked from HAL9000/cleveragents-core
Code Pull Requests Activity
Files
master
temp/robot/system_prompt_template_rendering.robot
T
CoreRasurae e4c01492d5 refactor(cli): align actor run signature with spec positional args 
Aligned the `agents actor run` command signature with the
specification by introducing positional NAME and PROMPT arguments.
The --config/-c option is preserved as an optional fallback for
direct YAML invocation. When NAME is provided without --config,
the actor is resolved from the Actor Registry.

Updated both actor_run.py and actor.py run commands. Added backward
compatibility: if --config is provided, it takes precedence over
name-based resolution.

Review fixes applied (code review round 1):
- P1-1: Narrowed bare `except Exception` to `except NotFoundError`
  in _resolve_config_files to avoid masking infrastructure errors.
- P1-2: Moved _resolve_config_files call inside the try block in
  run() so container/registry init errors get user-friendly messages.
  Added `except click.exceptions.Exit: raise` to let typer.Exit
  propagate through the broadened try scope.
- P1-3: Added atexit.register cleanup for temp files created by
  _resolve_config_files (resource leak fix).
- P1-4: Added CHANGELOG.md entry for the breaking CLI change.
- P2-1: Extracted duplicated _resolve_config_files to shared module
  `_resolve_actor.py`; both actor.py and actor_run.py now import it.
- P2-2: Added guard for actors with no configuration data
  (config_blob=None) to produce a clear error instead of invalid YAML.
- P2-3/P2-4: Added 5 BDD scenarios exercising the real
  resolve_config_files function (registry path, yaml_text path,
  config_blob fallback, no-config-data error, not-found error).
- P2-5: Added @coverage tags to all new BDD scenarios.
- P3-1: Added timeout=120s and on_timeout=kill to Robot tests.
- Fixed rxpy_route_validation.robot tests that used the removed
  --prompt/-p option (replaced with positional NAME + PROMPT args).

Review fixes applied (code review round 2):
- P2-1: Aligned actor_run.py exception handler from
  `CleverAgentsException` to `CleverAgentsError`, matching actor.py
  so infrastructure errors from registry resolution get user-friendly
  messages instead of falling through to the generic handler.
- P2-2: Changed `yaml.dump` to `yaml.safe_dump` in _resolve_actor.py
  for fail-fast behavior on unexpected types, consistent with the
  codebase's dominant pattern.
- P3-6: Replaced defensive `getattr(actor, ...)` calls with direct
  Pydantic model attribute access (`actor.yaml_text`, `actor.config_blob`)
  for type-checker coverage.
- P3-1: Switched BDD temp file cleanup from post-assertion
  `unlink()` to `context.add_cleanup()` for leak-proof teardown.
- P2-3/P3-2/P3-3/P3-4: Added 3 BDD edge-case scenarios
  (empty config_blob dict, infrastructure error propagation, empty
  string name) and 1 Robot test case (actor_app registry resolution).

Review fixes applied (code review round 3):
- P1-1: Migrated 48 remaining `-p` invocations across 9 Robot test
  files to the new positional `NAME PROMPT` pattern (context_delete_all_yes,
  load_context_test, scientific_paper_e2e_test, routing_prefix_stripping,
  scientific_paper_basic, scientific_paper_writer_test,
  context_management_test, initial_next_command_test,
  system_prompt_template_rendering).
- P2-1: Documented `--config/-c` as a spec deviation in
  `_resolve_actor.py` module docstring (spec lines 4562-4566 define
  `actor run` with no --config option; issue #901 AC accepts keeping
  it as optional).
- P2-2: Corrected `--config` help text from "fallback" to "overrides
  registry-based name resolution" — the option takes precedence, not
  the other way around.
- P3-1: Added `.strip()` to `yaml_text` emptiness check in
  `_resolve_actor.py` to handle whitespace-only values that would
  otherwise bypass the `config_blob` fallback.
- P3-2: Added `from None` to the no-configuration-data
  `typer.Exit(code=2)` for consistency with the not-found path.
- P3-3: Added BDD scenario testing `--config` precedence for
  `actor_run_app` (was only tested for `actor_app`).
- P3-4: Strengthened config_blob BDD scenario to verify generated
  YAML is parseable via `yaml.safe_load` round-trip.
- P3-7: Replaced hardcoded `/tmp/dummy.yaml` with
  `tempfile.gettempdir()` for portability.
- P3-8: Moved 5 inline imports to module level per CONTRIBUTING.md
  §1292-1294 (3x `import click`, 1x InfrastructureError in steps;
  1x `import typer` in robot helper).
- P3-9: Added `encoding="utf-8"` to `_write_yaml` in Robot helper
  for consistency with production code.

Review fixes applied (code review round 4):
- P2-1: Wrapped `yaml.safe_dump` in `_resolve_actor.py` with
  `try/except yaml.YAMLError` so non-serialisable config_blob values
  produce a user-friendly error message instead of a raw traceback.
- P2-2: Moved remaining inline `import yaml` to module level in
  `actor_run_signature_steps.py` per CONTRIBUTING.md §1292-1294.
- P2-3: Replaced 3 bare `assert` statements in Robot helper
  `helper_actor_run_signature.py` with diagnostic `if/print/sys.exit`
  pattern matching the rest of the file, improving failure diagnostics.

Review fixes applied (code review round 5):
- P3-4: Replaced per-call `atexit.register(lambda)` in
  `_resolve_actor.py` with a module-level `_temp_files` set and a
  single `atexit` handler (`_cleanup_temp_files`) to prevent
  unbounded handler accumulation in same-process usage (e.g. test
  suites running multiple CliRunner invocations).
- P3-1/P3-2/P3-3: Added 4 BDD scenarios: whitespace-only
  `yaml_text` fallback to config_blob, config-precedence
  registry-not-consulted assertion for both `actor_app` and
  `actor_run_app`, multiple `--config` files with positional NAME.
- P4-1: Strengthened error-path BDD assertions to verify error
  message content (not-found, no-config-data, serialisation-error)
  alongside exit codes via captured stderr.
- P4-2: Added Robot test case for `actor_app` unknown-name error
  path and corresponding helper function.

ISSUES CLOSED: #901
2026-03-26 13:41:31 +00:00

201 lines
10 KiB
Plaintext
Raw Permalink Blame History

*** Settings ***
Documentation Integration tests for LLM system prompt template rendering with runtime context
Library OperatingSystem
Library String
Library Collections
Library Process
Resource ${CURDIR}/v2_paths.resource
*** Variables ***
${PYTHON} python
${TEMP_DIR} /tmp/cleveragents_test
${TEST_CONTEXT} system_prompt_template_test
*** Test Cases ***
System Prompt Template Variables Are Rendered With Runtime Context
[Documentation] Verify that JINJA2 templates in system prompts are rendered with actual runtime context
[Tags] integration llm template system_prompt
# Create a simple config with templated system prompt
${config_content}= Catenate SEPARATOR=
... cleveragents:\n
... ${SPACE}${SPACE}version: "3.0"\n
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
...
... actors:\n
... ${SPACE}${SPACE}filter_actor:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Topic: {{ context.topic | tojson }}, List: {{ context.items | length }}"\n
...
... routes:\n
... ${SPACE}${SPACE}main:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
... \n
... merges:\n
... ${SPACE}${SPACE}- sources: [__input__]\n
... ${SPACE}${SPACE}${SPACE}${SPACE}target: main\n
Create File ${TEMP_DIR}/template_test.yaml ${config_content}
# Create context with test data
${context_data}= Set Variable {"topic": "Artificial Intelligence", "audience": "researchers"}
Create File ${TEMP_DIR}/test_context.json ${context_data}
# Run the agent with --load-context - the LLM should receive a system prompt with rendered values
# Since we can't easily intercept the actual LLM call, we verify the config was loaded correctly
${result}= Run Process ${PYTHON} -m cleveragents actor run -c ${TEMP_DIR}/template_test.yaml --unsafe --allow-rxpy-in-run-mode --load-context ${TEMP_DIR}/test_context.json test-actor "Analyze this"
... shell=False timeout=120s on_timeout=kill
# The agent should have processed successfully (even if it times out, the config should load)
Should Not Contain ${result.stderr} TemplateError
Should Not Contain ${result.stderr} Failed to render
Remove File ${TEMP_DIR}/template_test.yaml
Remove File ${TEMP_DIR}/test_context.json
Config Parser Preserves Template Syntax During YAML Loading
[Documentation] Verify that template markers are not prematurely rendered during config parsing
[Tags] integration config template
# Create config with templates
${config_content}= Catenate SEPARATOR=
... cleveragents:\n
... ${SPACE}${SPACE}version: "3.0"\n
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
... \n
... actors:\n
... ${SPACE}${SPACE}actor1:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Topic: {{ context.topic }}"\n
... ${SPACE}${SPACE}actor2:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Data: {{ context.data }}"\n
... \n
... routes:\n
... ${SPACE}${SPACE}main:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
Create File ${TEMP_DIR}/preserve_test.yaml ${config_content}
# The config should load without errors and preserve templates
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/preserve_test.yaml')]); agent1_prompt = config.agents['actor1'].config.get('system_prompt', ''); print('PROMPT1:', agent1_prompt); assert '{{' in agent1_prompt and '}}' in agent1_prompt, f'Templates not preserved: {agent1_prompt}'
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=120s on_timeout=kill
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} PROMPT1:
Should Contain ${result.stdout} {{ context.topic }}
# Clean up
Remove File ${TEMP_DIR}/preserve_test.yaml
Nested Context Variables In System Prompts
[Documentation] Verify that nested context variables like context.paper_details.topic work correctly
[Tags] integration llm template nested
# Create config with nested template variables
${config_content}= Catenate SEPARATOR=
... cleveragents:\n
... ${SPACE}${SPACE}version: "3.0"\n
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
... ${SPACE}${SPACE}unsafe: true\n
... \n
... actors:\n
... ${SPACE}${SPACE}nested_actor:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: |\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}Paper Details:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- Topic: {{ context.paper_details.topic }}\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- Length: {{ context.paper_details.length }}\n
... \n
... routes:\n
... ${SPACE}${SPACE}main:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
... ${SPACE}${SPACE}${SPACE}${SPACE}operators:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- type: map\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}params:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}actor: nested_actor\n
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
... \n
... merges:\n
... ${SPACE}${SPACE}- sources: [__input__]\n
... ${SPACE}${SPACE}${SPACE}${SPACE}target: main\n
Create File ${TEMP_DIR}/nested_test.yaml ${config_content}
# Create nested context
${context_data}= Set Variable {"paper_details": {"topic": "Machine Learning", "length": "5000 words"}}
Create File ${TEMP_DIR}/nested_context.json ${context_data}
# Verify the config loads and preserves templates
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/nested_test.yaml')]); prompt = config.agents['nested_actor'].config.get('system_prompt', ''); print('NESTED_PROMPT:', prompt); assert '{{ context.paper_details.topic }}' in prompt, f'Nested template not preserved: {prompt}'
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=120s on_timeout=kill
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} {{ context.paper_details.topic }}
# Clean up
Remove File ${TEMP_DIR}/nested_test.yaml
Remove File ${TEMP_DIR}/nested_context.json
JINJA2 Filters In System Prompts Are Preserved
[Documentation] Verify that JINJA2 filters like tojson are preserved during config loading
[Tags] integration template jinja2 filters
# Create config with JINJA2 filters
${config_content}= Catenate SEPARATOR=
... cleveragents:\n
... ${SPACE}${SPACE}version: "3.0"\n
... ${SPACE}${SPACE}template_engine: "JINJA2"\n
... \n
... actors:\n
... ${SPACE}${SPACE}filter_actor:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}type: llm\n
... ${SPACE}${SPACE}${SPACE}${SPACE}config:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}provider: openai\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}model: gpt-3.5-turbo\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}system_prompt: "Topic: {{ context.topic | tojson }}, List: {{ context.items | length }}"\n
... \n
... routes:\n
... ${SPACE}${SPACE}main:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}type: stream\n
... ${SPACE}${SPACE}${SPACE}${SPACE}publications:\n
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}- __output__\n
Create File ${TEMP_DIR}/filter_test.yaml ${config_content}
# Verify filters are preserved
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/filter_test.yaml')]); prompt = config.agents['filter_actor'].config.get('system_prompt', ''); print('FILTER_PROMPT:', prompt); assert '| tojson' in prompt and '| length' in prompt, f'Filters not preserved: {prompt}'
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=120s on_timeout=kill
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} | tojson
Should Contain ${result.stdout} | length
# Clean up
Remove File ${TEMP_DIR}/filter_test.yaml
*** Keywords ***
Suite Setup
Create Directory ${TEMP_DIR}
Suite Teardown
Remove Directory ${TEMP_DIR} recursive=True
View Git Blame Copy Permalink