Files
cleveragents-core/robot/load_context_test.robot
T
CoreRasurae e4c01492d5
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 15s
CI / lint (pull_request) Successful in 3m18s
CI / quality (pull_request) Successful in 3m47s
CI / typecheck (pull_request) Successful in 3m52s
CI / security (pull_request) Successful in 4m1s
CI / integration_tests (pull_request) Successful in 6m53s
CI / unit_tests (pull_request) Successful in 7m9s
CI / docker (pull_request) Successful in 1m9s
CI / e2e_tests (pull_request) Successful in 11m5s
CI / coverage (pull_request) Successful in 11m15s
CI / status-check (pull_request) Successful in 1s
CI / lint (push) Successful in 3m16s
CI / build (push) Successful in 31s
CI / typecheck (push) Successful in 4m0s
CI / security (push) Successful in 4m1s
CI / benchmark-regression (push) Has been skipped
CI / quality (push) Successful in 4m8s
CI / integration_tests (push) Successful in 6m10s
CI / unit_tests (push) Successful in 7m35s
CI / docker (push) Successful in 1m9s
CI / e2e_tests (push) Successful in 9m58s
CI / coverage (push) Successful in 11m7s
CI / status-check (push) Successful in 1s
CI / benchmark-publish (push) Failing after 17m22s
CI / benchmark-regression (pull_request) Successful in 56m7s
refactor(cli): align actor run signature with spec positional args
Aligned the `agents actor run` command signature with the
specification by introducing positional NAME and PROMPT arguments.
The --config/-c option is preserved as an optional fallback for
direct YAML invocation. When NAME is provided without --config,
the actor is resolved from the Actor Registry.

Updated both actor_run.py and actor.py run commands. Added backward
compatibility: if --config is provided, it takes precedence over
name-based resolution.

Review fixes applied (code review round 1):
- P1-1: Narrowed bare `except Exception` to `except NotFoundError`
  in _resolve_config_files to avoid masking infrastructure errors.
- P1-2: Moved _resolve_config_files call inside the try block in
  run() so container/registry init errors get user-friendly messages.
  Added `except click.exceptions.Exit: raise` to let typer.Exit
  propagate through the broadened try scope.
- P1-3: Added atexit.register cleanup for temp files created by
  _resolve_config_files (resource leak fix).
- P1-4: Added CHANGELOG.md entry for the breaking CLI change.
- P2-1: Extracted duplicated _resolve_config_files to shared module
  `_resolve_actor.py`; both actor.py and actor_run.py now import it.
- P2-2: Added guard for actors with no configuration data
  (config_blob=None) to produce a clear error instead of invalid YAML.
- P2-3/P2-4: Added 5 BDD scenarios exercising the real
  resolve_config_files function (registry path, yaml_text path,
  config_blob fallback, no-config-data error, not-found error).
- P2-5: Added @coverage tags to all new BDD scenarios.
- P3-1: Added timeout=120s and on_timeout=kill to Robot tests.
- Fixed rxpy_route_validation.robot tests that used the removed
  --prompt/-p option (replaced with positional NAME + PROMPT args).

Review fixes applied (code review round 2):
- P2-1: Aligned actor_run.py exception handler from
  `CleverAgentsException` to `CleverAgentsError`, matching actor.py
  so infrastructure errors from registry resolution get user-friendly
  messages instead of falling through to the generic handler.
- P2-2: Changed `yaml.dump` to `yaml.safe_dump` in _resolve_actor.py
  for fail-fast behavior on unexpected types, consistent with the
  codebase's dominant pattern.
- P3-6: Replaced defensive `getattr(actor, ...)` calls with direct
  Pydantic model attribute access (`actor.yaml_text`, `actor.config_blob`)
  for type-checker coverage.
- P3-1: Switched BDD temp file cleanup from post-assertion
  `unlink()` to `context.add_cleanup()` for leak-proof teardown.
- P2-3/P3-2/P3-3/P3-4: Added 3 BDD edge-case scenarios
  (empty config_blob dict, infrastructure error propagation, empty
  string name) and 1 Robot test case (actor_app registry resolution).

Review fixes applied (code review round 3):
- P1-1: Migrated 48 remaining `-p` invocations across 9 Robot test
  files to the new positional `NAME PROMPT` pattern (context_delete_all_yes,
  load_context_test, scientific_paper_e2e_test, routing_prefix_stripping,
  scientific_paper_basic, scientific_paper_writer_test,
  context_management_test, initial_next_command_test,
  system_prompt_template_rendering).
- P2-1: Documented `--config/-c` as a spec deviation in
  `_resolve_actor.py` module docstring (spec lines 4562-4566 define
  `actor run` with no --config option; issue #901 AC accepts keeping
  it as optional).
- P2-2: Corrected `--config` help text from "fallback" to "overrides
  registry-based name resolution" — the option takes precedence, not
  the other way around.
- P3-1: Added `.strip()` to `yaml_text` emptiness check in
  `_resolve_actor.py` to handle whitespace-only values that would
  otherwise bypass the `config_blob` fallback.
- P3-2: Added `from None` to the no-configuration-data
  `typer.Exit(code=2)` for consistency with the not-found path.
- P3-3: Added BDD scenario testing `--config` precedence for
  `actor_run_app` (was only tested for `actor_app`).
- P3-4: Strengthened config_blob BDD scenario to verify generated
  YAML is parseable via `yaml.safe_load` round-trip.
- P3-7: Replaced hardcoded `/tmp/dummy.yaml` with
  `tempfile.gettempdir()` for portability.
- P3-8: Moved 5 inline imports to module level per CONTRIBUTING.md
  §1292-1294 (3x `import click`, 1x InfrastructureError in steps;
  1x `import typer` in robot helper).
- P3-9: Added `encoding="utf-8"` to `_write_yaml` in Robot helper
  for consistency with production code.

Review fixes applied (code review round 4):
- P2-1: Wrapped `yaml.safe_dump` in `_resolve_actor.py` with
  `try/except yaml.YAMLError` so non-serialisable config_blob values
  produce a user-friendly error message instead of a raw traceback.
- P2-2: Moved remaining inline `import yaml` to module level in
  `actor_run_signature_steps.py` per CONTRIBUTING.md §1292-1294.
- P2-3: Replaced 3 bare `assert` statements in Robot helper
  `helper_actor_run_signature.py` with diagnostic `if/print/sys.exit`
  pattern matching the rest of the file, improving failure diagnostics.

Review fixes applied (code review round 5):
- P3-4: Replaced per-call `atexit.register(lambda)` in
  `_resolve_actor.py` with a module-level `_temp_files` set and a
  single `atexit` handler (`_cleanup_temp_files`) to prevent
  unbounded handler accumulation in same-process usage (e.g. test
  suites running multiple CliRunner invocations).
- P3-1/P3-2/P3-3: Added 4 BDD scenarios: whitespace-only
  `yaml_text` fallback to config_blob, config-precedence
  registry-not-consulted assertion for both `actor_app` and
  `actor_run_app`, multiple `--config` files with positional NAME.
- P4-1: Strengthened error-path BDD assertions to verify error
  message content (not-found, no-config-data, serialisation-error)
  alongside exit codes via captured stderr.
- P4-2: Added Robot test case for `actor_app` unknown-name error
  path and corresponding helper function.

ISSUES CLOSED: #901
2026-03-26 13:41:31 +00:00

311 lines
12 KiB
Plaintext

*** Settings ***
Documentation Integration tests for --load-context CLI feature
... SEC1: Uses simple_echo_config.yaml which requires code blocks.
Library Process
Library OperatingSystem
Library String
Library DateTime
Library Collections
Resource ${CURDIR}/v2_paths.resource
Force Tags code_blocks
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
*** Variables ***
${PYTHON} python
${SIMPLE_CONFIG} ${V2_CONFIG_DIR}/simple_echo_config.yaml
${CONTEXT_DIR} ${TEMPDIR}/test_load_contexts
${UNIQUE_ID} ${EMPTY}
${TEMP} ${EMPTY}
*** Test Cases ***
Test Load Context Transiently With Run Command
[Documentation] Verify --load-context loads context without persisting
${context_file} = Create Sample Context JSON File
${result} = Run Process ${PYTHON} -m cleveragents actor run
... -c ${SIMPLE_CONFIG}
... --load-context ${context_file}
... --unsafe
... --allow-rxpy-in-run-mode
... test-actor "test message"
Should Be Equal As Integers ${result.rc} 0
# Verify no context directory was created (transient)
${home} = Get Environment Variable HOME
Directory Should Not Exist ${home}/.cleveragents/context/_temp_*
Test Load Context Into Named Context
[Documentation] Verify --load-context with --context imports and persists
${context_name} = Set Variable load_named_${UNIQUE_ID}
${context_file} = Create Sample Context JSON File
# Load context into named context
${result} = Run Process ${PYTHON} -m cleveragents actor run
... -c ${SIMPLE_CONFIG}
... --load-context ${context_file}
... --context ${context_name}
... --context-dir ${CONTEXT_DIR}
... --unsafe
... --allow-rxpy-in-run-mode
... test-actor "test message"
Should Be Equal As Integers ${result.rc} 0
# Verify context was created and persisted
Directory Should Exist ${CONTEXT_DIR}/${context_name}
File Should Exist ${CONTEXT_DIR}/${context_name}/messages.json
File Should Exist ${CONTEXT_DIR}/${context_name}/global_context.json
# Verify global context was loaded
${global_ctx} = Get File ${CONTEXT_DIR}/${context_name}/global_context.json
Should Contain ${global_ctx} test_key
Should Contain ${global_ctx} test_value
Test Load Context Replaces Existing Named Context
[Documentation] Verify --load-context overwrites existing context
${context_name} = Set Variable replace_${UNIQUE_ID}
# Create initial context with different data
${result1} = Run Process ${PYTHON} -m cleveragents actor run
... -c ${SIMPLE_CONFIG}
... --context ${context_name}
... --context-dir ${CONTEXT_DIR}
... --unsafe
... --allow-rxpy-in-run-mode
... test-actor "original message"
Should Be Equal As Integers ${result1.rc} 0
# Create new context file with different data
${new_context_file} = Create Context JSON File With Different Data
# Load new context, replacing the old one
${result2} = Run Process ${PYTHON} -m cleveragents actor run
... -c ${SIMPLE_CONFIG}
... --load-context ${new_context_file}
... --context ${context_name}
... --context-dir ${CONTEXT_DIR}
... --unsafe
... --allow-rxpy-in-run-mode
... test-actor "new message"
Should Be Equal As Integers ${result2.rc} 0
# Verify context was replaced with new data
${global_ctx} = Get File ${CONTEXT_DIR}/${context_name}/global_context.json
Should Contain ${global_ctx} replaced_key
Should Contain ${global_ctx} replaced_value
Test Load Context From Export Format
[Documentation] Verify loading context from exported file works correctly
${context_name} = Set Variable export_test_${UNIQUE_ID}
${export_file} = Set Variable ${TEMP}/exported_context.json
# Create a context with some data
${result1} = Run Process ${PYTHON} -m cleveragents actor run
... -c ${SIMPLE_CONFIG}
... --context ${context_name}
... --context-dir ${CONTEXT_DIR}
... --unsafe
... --allow-rxpy-in-run-mode
... test-actor "test data for export"
Should Be Equal As Integers ${result1.rc} 0
# Export the context
${result2} = Run Process ${PYTHON} -m cleveragents context export
... ${context_name}
... ${export_file}
... --context-dir ${CONTEXT_DIR}
Should Be Equal As Integers ${result2.rc} 0
File Should Exist ${export_file}
# Delete the original context
${result3} = Run Process ${PYTHON} -m cleveragents context delete
... ${context_name}
... --context-dir ${CONTEXT_DIR}
... --yes
Should Be Equal As Integers ${result3.rc} 0
# Load the exported file into a new context
${new_context_name} = Set Variable import_test_${UNIQUE_ID}
${result4} = Run Process ${PYTHON} -m cleveragents actor run
... -c ${SIMPLE_CONFIG}
... --load-context ${export_file}
... --context ${new_context_name}
... --context-dir ${CONTEXT_DIR}
... --unsafe
... --allow-rxpy-in-run-mode
... test-actor "verify import"
Should Be Equal As Integers ${result4.rc} 0
# Verify the imported context has the original data
Directory Should Exist ${CONTEXT_DIR}/${new_context_name}
${messages} = Get File ${CONTEXT_DIR}/${new_context_name}/messages.json
Should Contain ${messages} test data for export
Test Load Context With Non-Existent File
[Documentation] Verify appropriate error when context file doesn't exist
${result} = Run Process ${PYTHON} -m cleveragents actor run
... -c ${SIMPLE_CONFIG}
... --load-context /nonexistent/file.json
... --unsafe
... --allow-rxpy-in-run-mode
... test-actor "test"
Should Not Be Equal As Integers ${result.rc} 0
Should Contain Any ${result.stderr} does not exist No such file not found
Test Load Context With Invalid JSON
[Documentation] Verify appropriate error when JSON is malformed
${bad_json_file} = Set Variable ${TEMP}/bad_context.json
Create File ${bad_json_file} { invalid json content
${result} = Run Process ${PYTHON} -m cleveragents actor run
... -c ${SIMPLE_CONFIG}
... --load-context ${bad_json_file}
... --unsafe
... --allow-rxpy-in-run-mode
... test-actor "test"
Should Not Be Equal As Integers ${result.rc} 0
Test Load Context Help Text
[Documentation] Verify --load-context appears in help text
${result} = Run Process ${PYTHON} -m cleveragents actor run --help stderr=STDOUT env:NO_COLOR=1 env:TERM=dumb
Should Be Equal As Integers ${result.rc} 0
# Debug: show repr of bytes around 'load' to reveal any ANSI escape codes
${idx} = Evaluate $result.stdout.find('load')
${snippet} = Evaluate repr($result.stdout[max(0,$idx-30):$idx+40]) if $idx >= 0 else 'NOT FOUND; tail=' + repr($result.stdout[-200:])
Log DEBUG stdout repr around 'load': ${snippet} console=True
Should Contain ${result.stdout} --load-context
Test Load Context Interactive Help Text
[Documentation] Verify --load-context appears in actor run help text
${result} = Run Process ${PYTHON} -m cleveragents actor run --help stderr=STDOUT env:NO_COLOR=1 env:TERM=dumb
Should Be Equal As Integers ${result.rc} 0
# Debug: show repr of bytes around 'load' to reveal any ANSI escape codes
${idx} = Evaluate $result.stdout.find('load')
${snippet} = Evaluate repr($result.stdout[max(0,$idx-30):$idx+40]) if $idx >= 0 else 'NOT FOUND; tail=' + repr($result.stdout[-200:])
Log DEBUG stdout repr around 'load': ${snippet} console=True
Should Contain ${result.stdout} --load-context
Test Load Context With All Components
[Documentation] Verify all context components (messages, state, metadata, global_context) are loaded
${context_name} = Set Variable full_load_${UNIQUE_ID}
${context_file} = Create Full Context JSON File
${result} = Run Process ${PYTHON} -m cleveragents actor run
... -c ${SIMPLE_CONFIG}
... --load-context ${context_file}
... --context ${context_name}
... --context-dir ${CONTEXT_DIR}
... --unsafe
... --allow-rxpy-in-run-mode
... test-actor "verify all components"
Should Be Equal As Integers ${result.rc} 0
# Verify all components were loaded
File Should Exist ${CONTEXT_DIR}/${context_name}/messages.json
File Should Exist ${CONTEXT_DIR}/${context_name}/metadata.json
File Should Exist ${CONTEXT_DIR}/${context_name}/state.json
File Should Exist ${CONTEXT_DIR}/${context_name}/global_context.json
# Verify content of each component
${messages} = Get File ${CONTEXT_DIR}/${context_name}/messages.json
Should Contain ${messages} initial message
${state} = Get File ${CONTEXT_DIR}/${context_name}/state.json
Should Contain ${state} test_state_key
${global_ctx} = Get File ${CONTEXT_DIR}/${context_name}/global_context.json
Should Contain ${global_ctx} session_id
*** Keywords ***
Setup Test Environment
${timestamp} = Get Current Date result_format=%Y%m%d_%H%M%S
${random} = Evaluate random.randint(1000, 9999) modules=random
Set Suite Variable ${UNIQUE_ID} ${timestamp}_${random}
Set Suite Variable ${TEMP} ${TEMPDIR}/ca_loadctx_${UNIQUE_ID}
Set Suite Variable ${CONTEXT_DIR} ${TEMP}/contexts
Create Directory ${TEMP}
Create Directory ${CONTEXT_DIR}
Cleanup Test Environment
Run Keyword And Ignore Error Remove Directory ${TEMP} recursive=True
Create Sample Context JSON File
[Documentation] Create a simple context JSON file for testing
${context_file} = Set Variable ${TEMP}/sample_context.json
${content} = Catenate SEPARATOR=\n
... {
... "context_name": "sample",
... "messages": [
... {
... "role": "user",
... "content": "Hello",
... "timestamp": "2025-01-01T00:00:00",
... "metadata": {}
... }
... ],
... "metadata": {
... "created_at": "2025-01-01T00:00:00"
... },
... "state": {},
... "global_context": {
... "test_key": "test_value"
... }
... }
Create File ${context_file} ${content}
RETURN ${context_file}
Create Context JSON File With Different Data
[Documentation] Create a context JSON file with different data for replacement test
${context_file} = Set Variable ${TEMP}/different_context.json
${content} = Catenate SEPARATOR=\n
... {
... "context_name": "different",
... "messages": [],
... "metadata": {},
... "state": {},
... "global_context": {
... "replaced_key": "replaced_value"
... }
... }
Create File ${context_file} ${content}
RETURN ${context_file}
Create Full Context JSON File
[Documentation] Create a context JSON file with all components
${context_file} = Set Variable ${TEMP}/full_context.json
${content} = Catenate SEPARATOR=\n
... {
... "context_name": "full",
... "messages": [
... {
... "role": "user",
... "content": "initial message",
... "timestamp": "2025-01-01T00:00:00",
... "metadata": {}
... }
... ],
... "metadata": {
... "created_at": "2025-01-01T00:00:00",
... "message_count": 1
... },
... "state": {
... "test_state_key": "test_state_value"
... },
... "global_context": {
... "session_id": "12345",
... "user_preferences": {"theme": "dark"}
... }
... }
Create File ${context_file} ${content}
RETURN ${context_file}