HAL9000
f808abff86
Fix JSON syntax errors in .devcontainer/devcontainer.json (removed
invalid JS-style // comments) and .devcontainer/opencode.json (removed
90+ trailing commas). Apply auto-fixes for end-of-file and trailing
whitespace issues across 100+ files. Fix SIM105 ruff violations in
benchmarks/core_circuit_breaker_bench.py (use contextlib.suppress).
Note: The security fix from issue #7478 (validate_path startswith bypass)
was already delivered to master in commit e18ac5f2. This PR as currently
structured is non-atomic (35 commits across 10+ issues) and needs
significant restructure before merge. This commit only addresses the
CI/pre-commit failures.
ISSUES CLOSED: #7478
260 lines
9.9 KiB
Plaintext
260 lines
9.9 KiB
Plaintext
*** Settings ***
|
|
Documentation End-to-end integration test for Scientific Paper Writer
|
|
Library Process
|
|
Library OperatingSystem
|
|
Library String
|
|
Resource ${CURDIR}/v2_paths.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${CONFIG_FILE} ${CURDIR}/../examples/scientific_paper_writer.yaml
|
|
${CONTEXT_DIR} ${TEMPDIR}/paper_e2e_contexts
|
|
${CONTEXT_NAME} paper_e2e_${TEST_ID}
|
|
${TEST_ID} ${EMPTY}
|
|
|
|
|
|
*** Test Cases ***
|
|
Scientific Paper Writer Full Workflow
|
|
[Documentation] Test the complete workflow from intro through multiple stages
|
|
[Tags] integration e2e slow
|
|
[Timeout] 10 minutes
|
|
|
|
# Test 1: Intro stage
|
|
Log Testing intro stage...
|
|
${result}= Run Paper Command Hello
|
|
Should Contain ${result.stdout} Paper Writer
|
|
|
|
# Test 2: Advance to discovery
|
|
Log Advancing to discovery stage...
|
|
${result}= Run Paper Command !next discovery
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
# Verify we're now in discovery stage
|
|
${stage_check}= Run Paper Command !stage
|
|
Should Contain ${stage_check.stdout} discovery
|
|
|
|
# Test 3: Commands work
|
|
Log Testing commands...
|
|
${result}= Run Paper Command !stages
|
|
Should Contain ${result.stdout} discovery
|
|
Should Contain ${result.stdout} brainstorming
|
|
|
|
# Test 4: Test help command
|
|
Log Testing help command...
|
|
${result}= Run Paper Command !help
|
|
Should Contain ${result.stdout} Available Commands
|
|
Should Contain ${result.stdout} !next
|
|
Should Contain ${result.stdout} !accept
|
|
|
|
# Test 5: Skip to brainstorming with mock data
|
|
Log Setting up mock paper details and advancing to brainstorming...
|
|
${result}= Run Paper Command !next brainstorming
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
# Verify we're now in brainstorming stage
|
|
${stage_check}= Run Paper Command !stage
|
|
Should Contain ${stage_check.stdout} brainstorming
|
|
|
|
# Test 6: Test brainstorming stage
|
|
Log Testing brainstorming stage...
|
|
${result}= Run Paper Command I want to write about the impact of machine learning on healthcare, focusing on diagnostic imaging and patient outcomes timeout=90s
|
|
Length Should Be Greater Than ${result.stdout} 50
|
|
|
|
# Test 7: Verify stage may have advanced (brainstorming stage might auto-advance after LLM response)
|
|
# So we don't check for specific stage here anymore
|
|
|
|
# Test 8: Advance toward later stages (using structure or latex_generation as targets)
|
|
Log Advancing toward later stages...
|
|
${result}= Run Paper Command !next structure timeout=120s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
|
|
# Test 9: Verify stage list command works
|
|
Log Verifying stage list command...
|
|
${result}= Run Paper Command !stages
|
|
Should Contain ${result.stdout} structure
|
|
Should Contain ${result.stdout} latex_generation
|
|
|
|
Log End-to-end test completed successfully
|
|
|
|
Scientific Paper Writer LaTeX Generation
|
|
[Documentation] Test LaTeX generation using pre-populated context
|
|
[Tags] integration e2e latex code_blocks
|
|
[Timeout] 5 minutes
|
|
|
|
${ctx}= Set Variable latex_test_${TEST_ID}
|
|
|
|
# Import the brainstorming context fixture
|
|
Log Importing brainstorming context fixture...
|
|
${result}= Run Process ${PYTHON} -m cleveragents context import
|
|
... ${ctx}
|
|
... ${V2_PAPER_CONTEXTS_DIR}/03_brainstorming.json
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... stderr=STDOUT
|
|
... timeout=30s
|
|
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Log Context imported: ${result.stdout}
|
|
|
|
# Advance to latex_generation stage
|
|
Log Advancing to latex_generation stage...
|
|
${result}= Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${CONFIG_FILE}
|
|
... --context ${ctx}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... --unsafe
|
|
... --allow-rxpy-in-run-mode
|
|
... test-actor !next latex_generation
|
|
... stderr=STDOUT
|
|
... timeout=180s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
# Note: Stage may auto-advance after !next is called
|
|
|
|
# Generate LaTeX document (send message that should trigger LaTeX generation)
|
|
# The exact stage doesn't matter - the system should generate LaTeX when asked
|
|
Log Generating LaTeX document...
|
|
${result}= Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${CONFIG_FILE}
|
|
... --context ${ctx}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... --unsafe
|
|
... --allow-rxpy-in-run-mode
|
|
... test-actor Generate the complete LaTeX document
|
|
... stderr=STDOUT
|
|
... timeout=180s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
# System should produce SOME output (not just echo the input)
|
|
Should Not Be Equal ${result.stdout} Generate the complete LaTeX document
|
|
|
|
# Verify LaTeX structure
|
|
Log Verifying LaTeX document structure...
|
|
Should Contain ${result.stdout} \\documentclass
|
|
Should Contain ${result.stdout} \\begin{document}
|
|
Should Contain ${result.stdout} \\end{document}
|
|
Should Contain ${result.stdout} \\section
|
|
|
|
# Verify context was updated
|
|
Log Verifying context is accessible...
|
|
${result}= Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${CONFIG_FILE}
|
|
... --context ${ctx}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... --unsafe
|
|
... --allow-rxpy-in-run-mode
|
|
... test-actor !context
|
|
... stderr=STDOUT
|
|
... timeout=30s
|
|
Should Contain ${result.stdout} latex_source
|
|
|
|
Log LaTeX generation test completed
|
|
|
|
Scientific Paper Writer Stage Navigation
|
|
[Documentation] Test stage navigation and command processing
|
|
[Tags] integration commands code_blocks
|
|
[Timeout] 3 minutes
|
|
|
|
${ctx}= Set Variable stage_nav_${TEST_ID}
|
|
|
|
# Start at intro
|
|
${result}= Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${CONFIG_FILE}
|
|
... --context ${ctx}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... --unsafe
|
|
... --allow-rxpy-in-run-mode
|
|
... test-actor !stages
|
|
... stderr=STDOUT
|
|
... timeout=20s
|
|
Should Contain ${result.stdout} intro
|
|
|
|
# Test !next command
|
|
${result}= Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${CONFIG_FILE}
|
|
... --context ${ctx}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... --unsafe
|
|
... --allow-rxpy-in-run-mode
|
|
... test-actor !next
|
|
... stderr=STDOUT
|
|
... timeout=20s
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
# Verify we advanced from intro (stage may auto-advance past discovery to brainstorming)
|
|
${stage_check}= Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${CONFIG_FILE}
|
|
... --context ${ctx}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... --unsafe
|
|
... --allow-rxpy-in-run-mode
|
|
... test-actor !stage
|
|
... stderr=STDOUT
|
|
... timeout=20s
|
|
Should Not Contain ${stage_check.stdout} intro
|
|
|
|
# Test !help command
|
|
${result}= Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${CONFIG_FILE}
|
|
... --context ${ctx}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... --unsafe
|
|
... --allow-rxpy-in-run-mode
|
|
... test-actor !help
|
|
... stderr=STDOUT
|
|
... timeout=20s
|
|
Should Contain ${result.stdout} Available Commands
|
|
Should Contain ${result.stdout} !next
|
|
Should Contain ${result.stdout} !accept
|
|
|
|
# Test !stage command
|
|
${result}= Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${CONFIG_FILE}
|
|
... --context ${ctx}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... --unsafe
|
|
... --allow-rxpy-in-run-mode
|
|
... test-actor !stage
|
|
... stderr=STDOUT
|
|
... timeout=20s
|
|
Should Contain ${result.stdout} Current Stage
|
|
Should Contain ${result.stdout} Purpose
|
|
|
|
*** Keywords ***
|
|
Setup Test Environment
|
|
${timestamp}= Get Time epoch
|
|
Set Suite Variable ${TEST_ID} ${timestamp}
|
|
Set Suite Variable ${CONTEXT_NAME} paper_e2e_${TEST_ID}
|
|
Create Directory ${CONTEXT_DIR}
|
|
Log Test environment setup complete with ID: ${TEST_ID}
|
|
|
|
Cleanup Test Environment
|
|
Run Keyword And Ignore Error Remove Directory ${CONTEXT_DIR} recursive=True
|
|
Log Test environment cleaned up
|
|
|
|
Run Paper Command
|
|
[Arguments] ${command} ${timeout}=30s
|
|
[Documentation] Run a command against the paper writer with the persistent context
|
|
${result}= Run Process ${PYTHON} -m cleveragents actor run
|
|
... -c ${CONFIG_FILE}
|
|
... --context ${CONTEXT_NAME}
|
|
... --context-dir ${CONTEXT_DIR}
|
|
... --unsafe
|
|
... --allow-rxpy-in-run-mode
|
|
... test-actor ${command}
|
|
... stderr=STDOUT
|
|
... timeout=${timeout}
|
|
Log Command: ${command}
|
|
Log Output: ${result.stdout}
|
|
RETURN ${result}
|
|
|
|
Should Contain Any
|
|
[Arguments] ${text} @{patterns}
|
|
[Documentation] Check if text contains at least one of the patterns
|
|
FOR ${pattern} IN @{patterns}
|
|
${status}= Run Keyword And Return Status Should Contain ${text} ${pattern} ignore_case=True
|
|
IF ${status} RETURN
|
|
END
|
|
Fail Text does not contain any of: @{patterns}
|
|
|
|
Length Should Be Greater Than
|
|
[Arguments] ${text} ${min_length}
|
|
[Documentation] Check if text length is greater than minimum
|
|
${length}= Get Length ${text}
|
|
Should Be True ${length} > ${min_length} Text length ${length} is not greater than ${min_length}
|