Compare commits

...

1 Commits

Author SHA1 Message Date
Dev f3b78c64b3 fix(test): increase robot test timeouts to 120s and fix DB migration hang
CI / lint (pull_request) Successful in 14s
CI / build (pull_request) Successful in 18s
CI / quality (pull_request) Successful in 22s
CI / typecheck (pull_request) Successful in 29s
CI / security (pull_request) Successful in 35s
CI / integration_tests (pull_request) Successful in 2m9s
CI / unit_tests (pull_request) Successful in 3m33s
CI / docker (pull_request) Successful in 1m17s
CI / coverage (pull_request) Successful in 10m32s
Process timeouts (SIGTERM/rc=-15) caused 19 integration test failures
across 6 suites. CLI cold-start imports the entire application stack
eagerly, making 10-30s timeouts insufficient.

- Raise all Run Process timeouts to 120s in 6 robot files
- Set CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true in database_integration
  test header to prevent typer.confirm() blocking on stdin
2026-02-19 00:11:20 +00:00
6 changed files with 58 additions and 57 deletions
+38 -38
View File
@@ -16,7 +16,7 @@ ${PROJECT_NAME} test-project
*** Test Cases ***
Test CLI Help Shows All Commands
[Documentation] Verify that CLI help displays all expected command groups
${result} = Run Process ${PYTHON} -m cleveragents --help timeout=30s
${result} = Run Process ${PYTHON} -m cleveragents --help timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} AI-powered development assistant
Should Contain ${result.stdout} project
@@ -31,7 +31,7 @@ Test Project Initialization
[Documentation] Test initializing a new CleverAgents project
Create Directory ${TEST_DIR}/project1
${result} = Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME}
... cwd=${TEST_DIR}/project1 timeout=30s
... cwd=${TEST_DIR}/project1 timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} Project '${PROJECT_NAME}' initialized successfully
Directory Should Exist ${TEST_DIR}/project1/.cleveragents
@@ -44,9 +44,9 @@ Test Project Initialization
Test Project Cannot Initialize Twice
[Documentation] Verify project cannot be initialized twice without force
Create Directory ${TEST_DIR}/project2
Run Process ${PYTHON} -m cleveragents init first-project cwd=${TEST_DIR}/project2 timeout=30s
Run Process ${PYTHON} -m cleveragents init first-project cwd=${TEST_DIR}/project2 timeout=120s
${result} = Run Process ${PYTHON} -m cleveragents init second-project
... cwd=${TEST_DIR}/project2 timeout=30s
... cwd=${TEST_DIR}/project2 timeout=120s
Should Not Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stderr} Project already initialized
Should Contain ${result.stderr} --force
@@ -54,19 +54,19 @@ Test Project Cannot Initialize Twice
Test Force Reinitialize Project
[Documentation] Test force reinitialization of a project
Create Directory ${TEST_DIR}/project3
Run Process ${PYTHON} -m cleveragents init first-project cwd=${TEST_DIR}/project3 timeout=30s
Run Process ${PYTHON} -m cleveragents init first-project cwd=${TEST_DIR}/project3 timeout=120s
${result} = Run Process ${PYTHON} -m cleveragents init second-project --force
... cwd=${TEST_DIR}/project3 timeout=30s
... cwd=${TEST_DIR}/project3 timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} Project 'second-project' initialized successfully
Test Context Add Files
[Documentation] Test adding files to context
Create Directory ${TEST_DIR}/project4
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project4 timeout=30s
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project4 timeout=120s
Create File ${TEST_DIR}/project4/test.py print('hello world')
${result} = Run Process ${PYTHON} -m cleveragents context add test.py
... cwd=${TEST_DIR}/project4 timeout=30s
... cwd=${TEST_DIR}/project4 timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} Added 1 file(s) to context
Should Contain ${result.stdout} test.py
@@ -74,13 +74,13 @@ Test Context Add Files
Test Context List Files
[Documentation] Test listing context files
Create Directory ${TEST_DIR}/project5
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project5 timeout=30s
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project5 timeout=120s
Create File ${TEST_DIR}/project5/file1.py # File 1
Create File ${TEST_DIR}/project5/file2.py # File 2
Run Process ${PYTHON} -m cleveragents context add file1.py file2.py
... cwd=${TEST_DIR}/project5 timeout=30s
... cwd=${TEST_DIR}/project5 timeout=120s
${result} = Run Process ${PYTHON} -m cleveragents context list
... cwd=${TEST_DIR}/project5 timeout=30s
... cwd=${TEST_DIR}/project5 timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} file1.py
Should Contain ${result.stdout} file2.py
@@ -88,20 +88,20 @@ Test Context List Files
Test Context Clear
[Documentation] Test clearing context files
Create Directory ${TEST_DIR}/project6
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project6 timeout=30s
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project6 timeout=120s
Create File ${TEST_DIR}/project6/test.py # Test file
Run Process ${PYTHON} -m cleveragents context add test.py cwd=${TEST_DIR}/project6 timeout=30s
Run Process ${PYTHON} -m cleveragents context add test.py cwd=${TEST_DIR}/project6 timeout=120s
${result} = Run Process ${PYTHON} -m cleveragents context clear --yes
... cwd=${TEST_DIR}/project6 timeout=30s
... cwd=${TEST_DIR}/project6 timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} Cleared all files from context
Test Plan Creation With Tell
[Documentation] Test creating a plan using tell command
Create Directory ${TEST_DIR}/project7
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project7 timeout=30s
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project7 timeout=120s
${result} = Run Process ${PYTHON} -m cleveragents tell Add error handling to main function
... cwd=${TEST_DIR}/project7 timeout=30s
... cwd=${TEST_DIR}/project7 timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} Plan created
Should Contain ${result.stdout} error handling
@@ -109,10 +109,10 @@ Test Plan Creation With Tell
Test Plan Build
[Documentation] Test building a plan
Create Directory ${TEST_DIR}/project8
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project8 timeout=30s
Run Process ${PYTHON} -m cleveragents tell Create example code cwd=${TEST_DIR}/project8 timeout=30s
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project8 timeout=120s
Run Process ${PYTHON} -m cleveragents tell Create example code cwd=${TEST_DIR}/project8 timeout=120s
${result} = Run Process ${PYTHON} -m cleveragents build cwd=${TEST_DIR}/project8
... env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=30s
... env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} Plan built successfully
Should Contain ${result.stdout} Generated
@@ -121,11 +121,11 @@ Test Plan Build
Test Plan Apply
[Documentation] Test applying plan changes
Create Directory ${TEST_DIR}/project9
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project9 timeout=30s
Run Process ${PYTHON} -m cleveragents tell Create example file cwd=${TEST_DIR}/project9 timeout=30s
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project9 timeout=120s
Run Process ${PYTHON} -m cleveragents tell Create example file cwd=${TEST_DIR}/project9 timeout=120s
Run Process ${PYTHON} -m cleveragents build cwd=${TEST_DIR}/project9
... env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=30s
${result} = Run Process ${PYTHON} -m cleveragents apply --yes cwd=${TEST_DIR}/project9 timeout=30s
... env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s
${result} = Run Process ${PYTHON} -m cleveragents apply --yes cwd=${TEST_DIR}/project9 timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} Successfully applied
File Should Exist ${TEST_DIR}/project9/example.py
@@ -133,21 +133,21 @@ Test Plan Apply
Test Plan List
[Documentation] Test listing plans
Create Directory ${TEST_DIR}/project10
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project10 timeout=30s
Run Process ${PYTHON} -m cleveragents tell First plan cwd=${TEST_DIR}/project10 timeout=30s
Run Process ${PYTHON} -m cleveragents plan new second-plan cwd=${TEST_DIR}/project10 timeout=30s
${result} = Run Process ${PYTHON} -m cleveragents plan list cwd=${TEST_DIR}/project10 timeout=30s
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project10 timeout=120s
Run Process ${PYTHON} -m cleveragents tell First plan cwd=${TEST_DIR}/project10 timeout=120s
Run Process ${PYTHON} -m cleveragents plan new second-plan cwd=${TEST_DIR}/project10 timeout=120s
${result} = Run Process ${PYTHON} -m cleveragents plan list cwd=${TEST_DIR}/project10 timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} Plans (3 total)
Test Shortcut Commands Work
[Documentation] Test that shortcut commands work properly
Create Directory ${TEST_DIR}/project11
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project11 timeout=30s
Run Process ${PYTHON} -m cleveragents init ${PROJECT_NAME} cwd=${TEST_DIR}/project11 timeout=120s
Create File ${TEST_DIR}/project11/test.txt Test content
# Test context-load shortcut
${result} = Run Process ${PYTHON} -m cleveragents context-load test.txt
... cwd=${TEST_DIR}/project11 timeout=30s
... cwd=${TEST_DIR}/project11 timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} Added 1 file(s) to context
@@ -156,30 +156,30 @@ Test End To End Workflow
Create Directory ${TEST_DIR}/project12
# Initialize project
${init} = Run Process ${PYTHON} -m cleveragents init workflow-project
... cwd=${TEST_DIR}/project12 timeout=30s
... cwd=${TEST_DIR}/project12 timeout=120s
Should Be Equal As Numbers ${init.rc} 0
# Add context
Create File ${TEST_DIR}/project12/input.py def main(): pass
${add} = Run Process ${PYTHON} -m cleveragents context add input.py
... cwd=${TEST_DIR}/project12 timeout=30s
... cwd=${TEST_DIR}/project12 timeout=120s
Should Be Equal As Numbers ${add.rc} 0
# Create plan
${tell} = Run Process ${PYTHON} -m cleveragents tell Add logging to main function
... cwd=${TEST_DIR}/project12 timeout=30s
... cwd=${TEST_DIR}/project12 timeout=120s
Should Be Equal As Numbers ${tell.rc} 0
# Build plan
${build} = Run Process ${PYTHON} -m cleveragents build cwd=${TEST_DIR}/project12
... env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=30s
... env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s
Should Be Equal As Numbers ${build.rc} 0
# Apply changes
${apply} = Run Process ${PYTHON} -m cleveragents apply --yes cwd=${TEST_DIR}/project12 timeout=30s
${apply} = Run Process ${PYTHON} -m cleveragents apply --yes cwd=${TEST_DIR}/project12 timeout=120s
Should Be Equal As Numbers ${apply.rc} 0
# Verify file created
File Should Exist ${TEST_DIR}/project12/example.py
Test Command Error Handling
[Documentation] Test error handling for invalid commands
${result} = Run Process ${PYTHON} -m cleveragents invalid-command timeout=30s
${result} = Run Process ${PYTHON} -m cleveragents invalid-command timeout=120s
Should Not Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stderr} Invalid command
@@ -187,16 +187,16 @@ Test Project Status Without Project
[Documentation] Test project status command without initialized project
Create Directory ${TEST_DIR}/no_project
${result} = Run Process ${PYTHON} -m cleveragents project status
... cwd=${TEST_DIR}/no_project timeout=30s
... cwd=${TEST_DIR}/no_project timeout=120s
Should Not Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stderr} No project found
Test Project Status With Project
[Documentation] Test project status command with initialized project
Create Directory ${TEST_DIR}/with_project
Run Process ${PYTHON} -m cleveragents init status-test cwd=${TEST_DIR}/with_project timeout=30s
Run Process ${PYTHON} -m cleveragents init status-test cwd=${TEST_DIR}/with_project timeout=120s
${result} = Run Process ${PYTHON} -m cleveragents project status
... cwd=${TEST_DIR}/with_project timeout=30s
... cwd=${TEST_DIR}/with_project timeout=120s
Should Be Equal As Numbers ${result.rc} 0
Should Contain ${result.stdout} Project: status-test
Should Contain ${result.stdout} Plans: 1
+2 -1
View File
@@ -729,6 +729,7 @@ Run Python Script
... sys.path.insert(0, '${CURDIR}/..')
... os.environ['PYTHONWARNINGS'] = 'ignore'
... os.environ['CLEVERAGENTS_TESTING'] = 'true'
... os.environ['CLEVERAGENTS_AUTO_APPLY_MIGRATIONS'] = 'true'
... # Disable all logging
... import logging
... logging.disable(logging.CRITICAL)
@@ -755,7 +756,7 @@ Run Python Script
# Write to a temporary file
${temp_file}= Evaluate __import__('tempfile').NamedTemporaryFile(mode='w', suffix='.py', delete=False, dir='/tmp').name
Create File ${temp_file} ${full_code}
${result}= Run Process ${PYTHON} ${temp_file} timeout=10s stderr=STDOUT env:PYTHONWARNINGS=ignore env:PYTHONDONTWRITEBYTECODE=1
${result}= Run Process ${PYTHON} ${temp_file} timeout=120s stderr=STDOUT env:PYTHONWARNINGS=ignore env:PYTHONDONTWRITEBYTECODE=1
Remove File ${temp_file}
# Check if process failed and log stderr if present
IF ${result.rc} != 0
+3 -3
View File
@@ -28,7 +28,7 @@ Scientific Paper Writer Basic Integration Test
... --allow-rxpy-in-run-mode
... -p Hello
... stderr=DEVNULL
... timeout=20s
... timeout=120s
Should Contain ${result.stdout} Paper Writer
Log Intro test passed
@@ -42,7 +42,7 @@ Scientific Paper Writer Basic Integration Test
... --allow-rxpy-in-run-mode
... -p !help
... stderr=DEVNULL
... timeout=10s
... timeout=120s
Should Contain ${result.stdout} Available Commands
Log Command handler test passed
@@ -54,7 +54,7 @@ Scientific Paper Writer Basic Integration Test
... --allow-rxpy-in-run-mode
... -p Test message
... stderr=DEVNULL
... timeout=10s
... timeout=120s
Should Contain ${result.stdout} Test message
Log Echo test passed
+8 -8
View File
@@ -95,7 +95,7 @@ Scientific Paper Writer LaTeX Generation
... ${V2_PAPER_CONTEXTS_DIR}/03_brainstorming.json
... --context-dir ${CONTEXT_DIR}
... stderr=STDOUT
... timeout=10s
... timeout=120s
Should Be Equal As Integers ${result.rc} 0
Log Context imported: ${result.stdout}
@@ -154,7 +154,7 @@ Scientific Paper Writer LaTeX Generation
... --allow-rxpy-in-run-mode
... -p !context
... stderr=STDOUT
... timeout=30s
... timeout=120s
${has_latex_context}= Run Keyword And Return Status Should Contain ${result.stdout} latex_source
IF ${has_latex_context}
Log Context includes latex_source field
@@ -180,7 +180,7 @@ Scientific Paper Writer Stage Navigation
... --allow-rxpy-in-run-mode
... -p !stages
... stderr=STDOUT
... timeout=20s
... timeout=120s
Should Contain ${result.stdout} intro
# Test !next command
@@ -192,7 +192,7 @@ Scientific Paper Writer Stage Navigation
... --allow-rxpy-in-run-mode
... -p !next
... stderr=STDOUT
... timeout=20s
... timeout=120s
Should Be Equal As Integers ${result.rc} 0
# Verify we advanced from intro (stage may auto-advance past discovery to brainstorming)
${stage_check}= Run Process ${PYTHON} -m cleveragents actor run
@@ -203,7 +203,7 @@ Scientific Paper Writer Stage Navigation
... --allow-rxpy-in-run-mode
... -p !stage
... stderr=STDOUT
... timeout=20s
... timeout=120s
Should Not Contain ${stage_check.stdout} intro
# Test !help command
@@ -215,7 +215,7 @@ Scientific Paper Writer Stage Navigation
... --allow-rxpy-in-run-mode
... -p !help
... stderr=STDOUT
... timeout=20s
... timeout=120s
Should Contain ${result.stdout} Available Commands
Should Contain ${result.stdout} !next
Should Contain ${result.stdout} !accept
@@ -229,7 +229,7 @@ Scientific Paper Writer Stage Navigation
... --allow-rxpy-in-run-mode
... -p !stage
... stderr=STDOUT
... timeout=20s
... timeout=120s
Should Contain ${result.stdout} Current Stage
Should Contain ${result.stdout} Purpose
@@ -247,7 +247,7 @@ Cleanup Test Environment
Log Test environment cleaned up
Run Paper Command
[Arguments] ${command} ${timeout}=30s
[Arguments] ${command} ${timeout}=120s
[Documentation] Run a command against the paper writer with the persistent context
${result}= Run Process ${PYTHON} -m cleveragents actor run
... -c ${CONFIG_FILE}
+3 -3
View File
@@ -30,7 +30,7 @@ Test Context Export Import Commands
... --unsafe
... --allow-rxpy-in-run-mode
... -p Hello world
... timeout=10s
... timeout=120s
# Export it
${result} = Run Process ${PYTHON} -m cleveragents context export
@@ -61,7 +61,7 @@ Test Context List Command
... --unsafe
... --allow-rxpy-in-run-mode
... -p First
... timeout=10s
... timeout=120s
Run Process ${PYTHON} -m cleveragents actor run
... -c ${SIMPLE_CONFIG}
@@ -70,7 +70,7 @@ Test Context List Command
... --unsafe
... --allow-rxpy-in-run-mode
... -p Second
... timeout=10s
... timeout=120s
# List them
${result} = Run Process ${PYTHON} -m cleveragents context list
+4 -4
View File
@@ -51,7 +51,7 @@ System Prompt Template Variables Are Rendered With Runtime Context
# Run the agent with --load-context - the LLM should receive a system prompt with rendered values
# Since we can't easily intercept the actual LLM call, we verify the config was loaded correctly
${result}= Run Process ${PYTHON} -m cleveragents actor run -c ${TEMP_DIR}/template_test.yaml --unsafe --allow-rxpy-in-run-mode --load-context ${TEMP_DIR}/test_context.json -p "Analyze this"
... shell=False timeout=30s
... shell=False timeout=120s
# The agent should have processed successfully (even if it times out, the config should load)
Should Not Contain ${result.stderr} TemplateError
@@ -93,7 +93,7 @@ Config Parser Preserves Template Syntax During YAML Loading
# The config should load without errors and preserve templates
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/preserve_test.yaml')]); agent1_prompt = config.agents['actor1'].config.get('system_prompt', ''); print('PROMPT1:', agent1_prompt); assert '{{' in agent1_prompt and '}}' in agent1_prompt, f'Templates not preserved: {agent1_prompt}'
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=10s
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=120s
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} PROMPT1:
@@ -146,7 +146,7 @@ Nested Context Variables In System Prompts
# Verify the config loads and preserves templates
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/nested_test.yaml')]); prompt = config.agents['nested_actor'].config.get('system_prompt', ''); print('NESTED_PROMPT:', prompt); assert '{{ context.paper_details.topic }}' in prompt, f'Nested template not preserved: {prompt}'
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=10s
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=120s
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} {{ context.paper_details.topic }}
@@ -183,7 +183,7 @@ JINJA2 Filters In System Prompts Are Preserved
# Verify filters are preserved
${python_code}= Set Variable from pathlib import Path; from cleveragents.reactive.config_parser import ReactiveConfigParser; parser = ReactiveConfigParser(); config = parser.parse_files([Path('${TEMP_DIR}/filter_test.yaml')]); prompt = config.agents['filter_actor'].config.get('system_prompt', ''); print('FILTER_PROMPT:', prompt); assert '| tojson' in prompt and '| length' in prompt, f'Filters not preserved: {prompt}'
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=10s
${result}= Run Process ${PYTHON} -c ${python_code} shell=False timeout=120s
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} | tojson