fix(e2e): add timeout, setup keyword, and deduplicate temp directory in workflow tests
CI / lint (pull_request) Successful in 39s
CI / helm (pull_request) Successful in 39s
CI / push-validation (pull_request) Successful in 38s
CI / typecheck (pull_request) Successful in 58s
CI / quality (pull_request) Successful in 1m1s
CI / unit_tests (pull_request) Successful in 5m41s
CI / integration_tests (pull_request) Failing after 9m57s
CI / build (pull_request) Failing after 15m5s
CI / security (pull_request) Failing after 15m6s
CI / coverage (pull_request) Has been cancelled
CI / docker (pull_request) Has been cancelled
CI / status-check (pull_request) Has been cancelled
CI / lint (pull_request) Successful in 39s
CI / helm (pull_request) Successful in 39s
CI / push-validation (pull_request) Successful in 38s
CI / typecheck (pull_request) Successful in 58s
CI / quality (pull_request) Successful in 1m1s
CI / unit_tests (pull_request) Successful in 5m41s
CI / integration_tests (pull_request) Failing after 9m57s
CI / build (pull_request) Failing after 15m5s
CI / security (pull_request) Failing after 15m6s
CI / coverage (pull_request) Has been cancelled
CI / docker (pull_request) Has been cancelled
CI / status-check (pull_request) Has been cancelled
Address PR #10614 review findings: - Added [Timeout] 30 minutes to all 9 test cases to prevent CI hangs - Removed duplicate Create Temp Directory keyword from each file (moved to common_e2e.resource) - Added Set Up E2E Project Test keyword for consistent temp dir creation via TEST NAME variable - Added msg= parameters to Should Not Be Empty assertions for better debugging Signed-off-by: HAL9000 <hal9000@noreply.git.cleverthis.com>
This commit is contained in:
@@ -292,6 +292,24 @@ Create Synthetic Codebase
|
||||
${large_content}= Evaluate "# auto-generated large file\\n" + ("x = 1\\n" * 250)
|
||||
Create File ${base_dir}${/}large_file.py ${large_content}
|
||||
|
||||
Create Temp Directory
|
||||
[Documentation] Create a temporary directory for testing.
|
||||
...
|
||||
... Uses Python's tempfile module to create an isolated
|
||||
... temp directory with the given name prefix. Ideal for
|
||||
... project creation tests that need their own workspace.
|
||||
[Arguments] ${name}=${EMPTY}
|
||||
${temp_dir}= Evaluate __import__("tempfile").mkdtemp(prefix="${name}-")
|
||||
RETURN ${temp_dir}
|
||||
|
||||
Remove Temp Directory
|
||||
[Documentation] Remove a temporary directory created for testing.
|
||||
...
|
||||
... Safely removes the directory and all contents recursively,
|
||||
... ignoring errors if the directory does not exist.
|
||||
[Arguments] ${dir}
|
||||
Run Keyword And Ignore Error Remove Directory ${dir} recursive=True
|
||||
|
||||
Create Temp Git Repo
|
||||
[Documentation] Create a temporary git repository for E2E testing.
|
||||
...
|
||||
|
||||
@@ -14,6 +14,7 @@ Suite Teardown E2E Suite Teardown
|
||||
Force Tags E2E
|
||||
|
||||
*** Test Cases ***
|
||||
|
||||
Correction Revert Mode Workflow
|
||||
[Documentation] Test plan correction in revert mode.
|
||||
...
|
||||
@@ -21,15 +22,14 @@ Correction Revert Mode Workflow
|
||||
... then corrects it using revert mode to restore
|
||||
... the plan to a previous state.
|
||||
[Tags] correction-revert
|
||||
[Timeout] 30 minutes
|
||||
[Setup] Set Up E2E Project Test
|
||||
Skip If No LLM Keys
|
||||
# Create a temporary directory for the project
|
||||
${project_dir}= Create Temp Directory correction-revert-test
|
||||
# Initialize a new project
|
||||
${create_result}= Run CleverAgents Command project create --name correction-revert-project --path ${project_dir}
|
||||
Should Not Be Empty ${create_result.stdout}
|
||||
# Verify project was created
|
||||
Should Not Be Empty ${create_result.stdout} msg=project create returned empty output
|
||||
${show_result}= Run CleverAgents Command project show --name correction-revert-project
|
||||
Should Not Be Empty ${show_result.stdout}
|
||||
Should Not Be Empty ${show_result.stdout} msg=project show returned empty output
|
||||
Output Should Contain ${show_result} correction-revert-project
|
||||
|
||||
Correction Append Mode Workflow
|
||||
@@ -39,15 +39,14 @@ Correction Append Mode Workflow
|
||||
... then corrects it using append mode to add corrections
|
||||
... to the existing plan.
|
||||
[Tags] correction-append
|
||||
[Timeout] 30 minutes
|
||||
[Setup] Set Up E2E Project Test
|
||||
Skip If No LLM Keys
|
||||
# Create a temporary directory for the project
|
||||
${project_dir}= Create Temp Directory correction-append-test
|
||||
# Initialize a new project
|
||||
${create_result}= Run CleverAgents Command project create --name correction-append-project --path ${project_dir}
|
||||
Should Not Be Empty ${create_result.stdout}
|
||||
# Verify project was created
|
||||
Should Not Be Empty ${create_result.stdout} msg=project create returned empty output
|
||||
${show_result}= Run CleverAgents Command project show --name correction-append-project
|
||||
Should Not Be Empty ${show_result.stdout}
|
||||
Should Not Be Empty ${show_result.stdout} msg=project show returned empty output
|
||||
Output Should Contain ${show_result} correction-append-project
|
||||
|
||||
Correction State Transition Validation
|
||||
@@ -56,20 +55,18 @@ Correction State Transition Validation
|
||||
... Verifies that correction workflows properly transition
|
||||
... plan state through correction phases.
|
||||
[Tags] correction-state
|
||||
[Timeout] 30 minutes
|
||||
[Setup] Set Up E2E Project Test
|
||||
Skip If No LLM Keys
|
||||
# Create a temporary directory for the project
|
||||
${project_dir}= Create Temp Directory correction-state-test
|
||||
# Initialize a new project
|
||||
${create_result}= Run CleverAgents Command project create --name correction-state-project --path ${project_dir}
|
||||
Should Not Be Empty ${create_result.stdout}
|
||||
# Verify project was created
|
||||
Should Not Be Empty ${create_result.stdout} msg=project create returned empty output
|
||||
${show_result}= Run CleverAgents Command project show --name correction-state-project
|
||||
Should Not Be Empty ${show_result.stdout}
|
||||
Should Not Be Empty ${show_result.stdout} msg=project show returned empty output
|
||||
Output Should Contain ${show_result} correction-state-project
|
||||
|
||||
*** Keywords ***
|
||||
Create Temp Directory
|
||||
[Documentation] Create a temporary directory for testing.
|
||||
[Arguments] ${name}
|
||||
${temp_dir}= Evaluate __import__("tempfile").mkdtemp(prefix="${name}-")
|
||||
RETURN ${temp_dir}
|
||||
Set Up E2E Project Test
|
||||
[Documentation] Set up a fresh temp directory for the test case.
|
||||
${project_dir}= Create Temp Directory ${TEST NAME}
|
||||
Set Suite Variable ${PROJECT_DIR} ${project_dir}
|
||||
|
||||
@@ -15,22 +15,22 @@ Suite Teardown E2E Suite Teardown
|
||||
Force Tags E2E
|
||||
|
||||
*** Test Cases ***
|
||||
|
||||
Project Creation Workflow
|
||||
[Documentation] Test complete project creation workflow.
|
||||
...
|
||||
... Creates a new project via CLI, verifies project
|
||||
... is created successfully, and validates output format.
|
||||
[Tags] project-creation
|
||||
[Timeout] 30 minutes
|
||||
[Setup] Set Up E2E Project Test
|
||||
Skip If No LLM Keys
|
||||
# Create a temporary directory for the project
|
||||
${project_dir}= Create Temp Directory project-creation-test
|
||||
# Initialize a new project
|
||||
${result}= Run CleverAgents Command project create --name test-project --path ${project_dir}
|
||||
Should Not Be Empty ${result.stdout}
|
||||
Should Not Be Empty ${result.stdout} msg=project create returned empty output
|
||||
Output Should Contain ${result} test-project
|
||||
# Verify project was created
|
||||
${list_result}= Run CleverAgents Command project list
|
||||
Should Not Be Empty ${list_result.stdout}
|
||||
Should Not Be Empty ${list_result.stdout} msg=project list returned empty output
|
||||
Output Should Contain ${list_result} test-project
|
||||
|
||||
Actor Setup Workflow
|
||||
@@ -39,15 +39,14 @@ Actor Setup Workflow
|
||||
... Creates a project, adds actors to it, and verifies
|
||||
... actors are properly registered.
|
||||
[Tags] actor-setup
|
||||
[Timeout] 30 minutes
|
||||
[Setup] Set Up E2E Project Test
|
||||
Skip If No LLM Keys
|
||||
# Create a temporary directory for the project
|
||||
${project_dir}= Create Temp Directory actor-setup-test
|
||||
# Initialize a new project
|
||||
${create_result}= Run CleverAgents Command project create --name actor-test-project --path ${project_dir}
|
||||
Should Not Be Empty ${create_result.stdout}
|
||||
# List actors in the project
|
||||
Should Not Be Empty ${create_result.stdout} msg=project create returned empty output
|
||||
${list_result}= Run CleverAgents Command actor list
|
||||
Should Not Be Empty ${list_result.stdout}
|
||||
Should Not Be Empty ${list_result.stdout} msg=actor list returned empty output
|
||||
|
||||
Plan Execution Workflow
|
||||
[Documentation] Test complete plan execution workflow.
|
||||
@@ -55,20 +54,18 @@ Plan Execution Workflow
|
||||
... Creates a project, generates a plan, and executes it
|
||||
... through the complete workflow.
|
||||
[Tags] plan-execution
|
||||
[Timeout] 30 minutes
|
||||
[Setup] Set Up E2E Project Test
|
||||
Skip If No LLM Keys
|
||||
# Create a temporary directory for the project
|
||||
${project_dir}= Create Temp Directory plan-execution-test
|
||||
# Initialize a new project
|
||||
${create_result}= Run CleverAgents Command project create --name plan-test-project --path ${project_dir}
|
||||
Should Not Be Empty ${create_result.stdout}
|
||||
# Verify project creation
|
||||
Should Not Be Empty ${create_result.stdout} msg=project create returned empty output
|
||||
${show_result}= Run CleverAgents Command project show --name plan-test-project
|
||||
Should Not Be Empty ${show_result.stdout}
|
||||
Should Not Be Empty ${show_result.stdout} msg=project show returned empty output
|
||||
Output Should Contain ${show_result} plan-test-project
|
||||
|
||||
*** Keywords ***
|
||||
Create Temp Directory
|
||||
[Documentation] Create a temporary directory for testing.
|
||||
[Arguments] ${name}
|
||||
${temp_dir}= Evaluate __import__("tempfile").mkdtemp(prefix="${name}-")
|
||||
RETURN ${temp_dir}
|
||||
Set Up E2E Project Test
|
||||
[Documentation] Set up a fresh temp directory for the test case.
|
||||
${project_dir}= Create Temp Directory ${TEST NAME}
|
||||
Set Suite Variable ${PROJECT_DIR} ${project_dir}
|
||||
|
||||
@@ -14,21 +14,21 @@ Suite Teardown E2E Suite Teardown
|
||||
Force Tags E2E
|
||||
|
||||
*** Test Cases ***
|
||||
|
||||
Subplan Spawning Workflow
|
||||
[Documentation] Test subplan spawning from parent plan.
|
||||
...
|
||||
... Creates a project, generates a plan, and spawns
|
||||
... subplans from the parent plan.
|
||||
[Tags] subplan-spawn
|
||||
[Timeout] 30 minutes
|
||||
[Setup] Set Up E2E Project Test
|
||||
Skip If No LLM Keys
|
||||
# Create a temporary directory for the project
|
||||
${project_dir}= Create Temp Directory subplan-spawn-test
|
||||
# Initialize a new project
|
||||
${create_result}= Run CleverAgents Command project create --name subplan-spawn-project --path ${project_dir}
|
||||
Should Not Be Empty ${create_result.stdout}
|
||||
# Verify project was created
|
||||
Should Not Be Empty ${create_result.stdout} msg=project create returned empty output
|
||||
${show_result}= Run CleverAgents Command project show --name subplan-spawn-project
|
||||
Should Not Be Empty ${show_result.stdout}
|
||||
Should Not Be Empty ${show_result.stdout} msg=project show returned empty output
|
||||
Output Should Contain ${show_result} subplan-spawn-project
|
||||
|
||||
Three-Way Merge Workflow
|
||||
@@ -37,15 +37,14 @@ Three-Way Merge Workflow
|
||||
... Creates a project, generates a plan with subplans,
|
||||
... executes them, and performs three-way merge.
|
||||
[Tags] three-way-merge
|
||||
[Timeout] 30 minutes
|
||||
[Setup] Set Up E2E Project Test
|
||||
Skip If No LLM Keys
|
||||
# Create a temporary directory for the project
|
||||
${project_dir}= Create Temp Directory three-way-merge-test
|
||||
# Initialize a new project
|
||||
${create_result}= Run CleverAgents Command project create --name three-way-merge-project --path ${project_dir}
|
||||
Should Not Be Empty ${create_result.stdout}
|
||||
# Verify project was created
|
||||
Should Not Be Empty ${create_result.stdout} msg=project create returned empty output
|
||||
${show_result}= Run CleverAgents Command project show --name three-way-merge-project
|
||||
Should Not Be Empty ${show_result.stdout}
|
||||
Should Not Be Empty ${show_result.stdout} msg=project show returned empty output
|
||||
Output Should Contain ${show_result} three-way-merge-project
|
||||
|
||||
Merge Result Validation
|
||||
@@ -54,20 +53,18 @@ Merge Result Validation
|
||||
... Verifies that merge results are properly validated
|
||||
... and conform to specification requirements.
|
||||
[Tags] merge-validation
|
||||
[Timeout] 30 minutes
|
||||
[Setup] Set Up E2E Project Test
|
||||
Skip If No LLM Keys
|
||||
# Create a temporary directory for the project
|
||||
${project_dir}= Create Temp Directory merge-validation-test
|
||||
# Initialize a new project
|
||||
${create_result}= Run CleverAgents Command project create --name merge-validation-project --path ${project_dir}
|
||||
Should Not Be Empty ${create_result.stdout}
|
||||
# Verify project was created
|
||||
Should Not Be Empty ${create_result.stdout} msg=project create returned empty output
|
||||
${show_result}= Run CleverAgents Command project show --name merge-validation-project
|
||||
Should Not Be Empty ${show_result.stdout}
|
||||
Should Not Be Empty ${show_result.stdout} msg=project show returned empty output
|
||||
Output Should Contain ${show_result} merge-validation-project
|
||||
|
||||
*** Keywords ***
|
||||
Create Temp Directory
|
||||
[Documentation] Create a temporary directory for testing.
|
||||
[Arguments] ${name}
|
||||
${temp_dir}= Evaluate __import__("tempfile").mkdtemp(prefix="${name}-")
|
||||
RETURN ${temp_dir}
|
||||
Set Up E2E Project Test
|
||||
[Documentation] Set up a fresh temp directory for the test case.
|
||||
${project_dir}= Create Temp Directory ${TEST NAME}
|
||||
Set Suite Variable ${PROJECT_DIR} ${project_dir}
|
||||
|
||||
Reference in New Issue
Block a user