HAL9000
926665eec8
Robot Framework requires ≥2 spaces (or a tab) to separate multiple values in a [Tags] setting. The WF18 Suite Setup keyword had a single space between tdd_issue and tdd_issue_4188, causing RF to parse them as a single tag with a space in its name rather than two distinct tags. The tdd_expected_fail_listener validates that tdd_issue_N requires a standalone tdd_issue tag — the single-space bug was silently breaking that validation. Fix: change `tdd_issue tdd_issue_4188` to `tdd_issue tdd_issue_4188` (four-space separator, matching the rest of the file's style). ISSUES CLOSED: #10815
224 lines
11 KiB
Plaintext
224 lines
11 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E test for Workflow Example 18: Container with Remote Repo
|
|
... Clone (trusted profile).
|
|
...
|
|
... Exercises resource registration with container-instance and
|
|
... --clone-into flag for remote repo clone, two-step project
|
|
... creation and linking, action with trusted automation profile,
|
|
... plan-level execution environment with fallback priority, and
|
|
... full plan lifecycle including apply with container commit/push.
|
|
...
|
|
... Zero mocking — real CLI, real LLM API keys.
|
|
...
|
|
... Tagged ``tdd_expected_fail`` because the container clone workflow
|
|
... is resource-intensive (real LLM + Docker container operations)
|
|
... and the CI environment kills the process with SIGKILL when
|
|
... memory limits are exceeded (rc=-9). The tag inverts CI result
|
|
... until the container execution environment is tuned to operate
|
|
... within CI memory constraints. See issue #10815.
|
|
Resource common_e2e.resource
|
|
Suite Setup WF18 Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
Force Tags E2E
|
|
|
|
*** Variables ***
|
|
${ACTION_PREFIX} local/wf18-clone-action
|
|
${RESOURCE_PREFIX} local/wf18-clone-res
|
|
${PROJECT_PREFIX} local/wf18-clone-proj
|
|
|
|
*** Keywords ***
|
|
WF18 Suite Setup
|
|
[Documentation] E2E Suite Setup plus unique suffix generation and dynamic
|
|
... actor selection for WF18 tests.
|
|
[Tags] tdd_issue tdd_issue_4188
|
|
E2E Suite Setup
|
|
# Initialise the workspace so CLI commands work.
|
|
${init}= Run CleverAgents Command init --force --yes
|
|
Should Be Equal As Integers ${init.rc} 0
|
|
# Generate a unique suffix for resource/project names to avoid UNIQUE
|
|
# constraint collisions on repeated E2E runs or parallel CI.
|
|
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
|
|
Set Suite Variable ${RUN_SUFFIX} ${suffix}
|
|
# Pick an actor that matches the available API key.
|
|
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
|
|
IF ${has_anthropic}
|
|
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
|
|
ELSE
|
|
${actor}= Set Variable openai/gpt-4o
|
|
END
|
|
Set Suite Variable ${LLM_ACTOR} ${actor}
|
|
# Compute unique action name using the run suffix for parallel CI safety.
|
|
${action_name}= Set Variable ${ACTION_PREFIX}-${RUN_SUFFIX}
|
|
Set Suite Variable ${ACTION_NAME} ${action_name}
|
|
|
|
Create Remote Clone Repo
|
|
[Documentation] Create a temp git repo simulating a project that would
|
|
... be cloned from a remote repository into a container.
|
|
${repo}= Create Temp Git Repo wf18-remote-clone-${RUN_SUFFIX}
|
|
Create Directory ${repo}${/}src
|
|
Create Directory ${repo}${/}deploy
|
|
${app_content}= Catenate SEPARATOR=\n
|
|
... """Application for remote deployment via container clone."""
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... class DeploymentManager:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Manages deployment lifecycle."""
|
|
... ${EMPTY}
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}def __init__(self, repo_url, branch="main"):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.repo_url = repo_url
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.branch = branch
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.deployed = False
|
|
... ${EMPTY}
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}def deploy(self):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}"""Execute deployment."""
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.deployed = True
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"status": "deployed", "branch": self.branch}
|
|
Create File ${repo}${/}src${/}deploy_manager.py ${app_content}
|
|
${deploy_config}= Catenate SEPARATOR=\n
|
|
... # Deployment configuration
|
|
... CONTAINER_IMAGE=python:3.12-slim
|
|
... CLONE_DEPTH=1
|
|
... DEPLOY_TIMEOUT=300
|
|
... HEALTH_CHECK_INTERVAL=10
|
|
Create File ${repo}${/}deploy${/}config.env ${deploy_config}
|
|
# NOTE: This Dockerfile is fixture data only — never built by this test.
|
|
# It simulates a project that would be cloned into a container.
|
|
${dockerfile}= Catenate SEPARATOR=\n
|
|
... FROM python:3.12-slim
|
|
... RUN apt-get update && apt-get install -y git
|
|
... WORKDIR /workspace
|
|
... ARG REPO_URL
|
|
... RUN git clone --depth 1 \${REPO_URL} .
|
|
... CMD ["python", "-m", "src.deploy_manager"]
|
|
Create File ${repo}${/}Dockerfile ${dockerfile}
|
|
Create File ${repo}${/}src${/}__init__.py \n
|
|
${git_add}= Run Process git add . cwd=${repo}
|
|
Should Be Equal As Integers ${git_add.rc} 0 msg=Fixture git add failed: ${git_add.stderr}
|
|
${git_commit}= Run Process git commit -m Initial remote-clone container project cwd=${repo}
|
|
Should Be Equal As Integers ${git_commit.rc} 0 msg=Fixture git commit failed: ${git_commit.stderr}
|
|
RETURN ${repo}
|
|
|
|
WF18 Test Teardown
|
|
[Documentation] Log diagnostic context on failure for debugging.
|
|
... Captures plan status when a plan ID is available.
|
|
${plan_id}= Get Variable Value ${WF18_PLAN_ID} ${EMPTY}
|
|
IF '${plan_id}' != ''
|
|
${status} ${result}= Run Keyword And Ignore Error
|
|
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
|
|
IF '${status}' == 'PASS'
|
|
Log Teardown plan status: ${result.stdout} WARN
|
|
END
|
|
END
|
|
|
|
*** Test Cases ***
|
|
WF18 Container With Remote Repo Clone Trusted Profile
|
|
[Documentation] Trusted-profile workflow: container-instance resource with
|
|
... --clone-into for remote repo clone, two-step project setup,
|
|
... plan-level execution environment with fallback priority, and
|
|
... full plan lifecycle including apply with container commit/push.
|
|
...
|
|
... Tagged ``tdd_expected_fail`` because the container clone workflow
|
|
... is resource-intensive (real LLM + Docker container operations)
|
|
... and the CI environment kills the process with SIGKILL (rc=-9)
|
|
... when memory limits are exceeded. The tag inverts the CI result
|
|
... until the container execution environment is tuned for CI.
|
|
... See issue #10815.
|
|
[Tags] tdd_issue tdd_issue_4188 tdd_expected_fail tdd_issue_10815
|
|
[Timeout] 20 minutes
|
|
[Teardown] WF18 Test Teardown
|
|
Skip If No LLM Keys
|
|
Set Test Variable ${WF18_PLAN_ID} ${EMPTY}
|
|
# Step 1: Create the fixture repo that simulates the remote project to clone.
|
|
${repo}= Create Remote Clone Repo
|
|
# Step 2: Register the container-instance resource with --clone-into flag.
|
|
# The --clone-into value format is <repo-url>:<container-path>.
|
|
# The fixture repo is a local path — use file:// URI so git can clone it.
|
|
${res_name}= Set Variable ${RESOURCE_PREFIX}-${RUN_SUFFIX}
|
|
${clone_url}= Set Variable file://${repo}
|
|
${add_res}= Run CleverAgents Command
|
|
... resource add container-instance ${res_name}
|
|
... --image python:3.12-slim
|
|
... --clone-into ${clone_url}:/workspace
|
|
... --format json
|
|
... expected_rc=None
|
|
... timeout=60s
|
|
Log resource add container-instance stdout: ${add_res.stdout} level=DEBUG
|
|
Log resource add container-instance stderr: ${add_res.stderr} level=DEBUG
|
|
# AC1: container-instance resource registered with --clone-into.
|
|
Should Be Equal As Integers ${add_res.rc} 0
|
|
... msg=resource add container-instance failed (rc=${add_res.rc}). Check DEBUG logs above.
|
|
Output Should Contain ${add_res} ${res_name}
|
|
# Step 3: Create the project and link the container resource.
|
|
${proj_name}= Set Variable ${PROJECT_PREFIX}-${RUN_SUFFIX}
|
|
${create_proj}= Run CleverAgents Command
|
|
... project create ${proj_name}
|
|
... --description WF18 container clone deployment project
|
|
... expected_rc=None
|
|
... timeout=30s
|
|
Should Be Equal As Integers ${create_proj.rc} 0
|
|
... msg=project create failed (rc=${create_proj.rc}). Check DEBUG logs above.
|
|
${link_res}= Run CleverAgents Command
|
|
... project link-resource ${proj_name} ${res_name}
|
|
... expected_rc=None
|
|
... timeout=30s
|
|
Should Be Equal As Integers ${link_res.rc} 0
|
|
... msg=project link-resource failed (rc=${link_res.rc}). Check DEBUG logs above.
|
|
# AC2: Project created and linked to the container resource.
|
|
${proj_show}= Run CleverAgents Command project show ${proj_name} --format plain timeout=30s
|
|
Output Should Contain ${proj_show} wf18
|
|
# Step 4: Create the action YAML and register it.
|
|
${actor_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${ACTION_NAME}
|
|
... description: WF18 container clone trusted action
|
|
... automation_profile: trusted
|
|
... actor: ${LLM_ACTOR}
|
|
${action_yaml_path}= Set Variable ${SUITE_HOME}${/}wf18-action-${RUN_SUFFIX}.yaml
|
|
Create File ${action_yaml_path} ${actor_yaml}
|
|
${add_action}= Run CleverAgents Command
|
|
... action create --config ${action_yaml_path}
|
|
... expected_rc=None
|
|
... timeout=30s
|
|
Log action create stdout: ${add_action.stdout} level=DEBUG
|
|
Should Be Equal As Integers ${add_action.rc} 0
|
|
... msg=action create failed (rc=${add_action.rc}). Check DEBUG logs above.
|
|
# Step 5: Launch plan with trusted automation profile.
|
|
# Use plan use with the trusted action. Expected to be resource-intensive
|
|
# and may be killed by OOM in CI — tdd_expected_fail inverts the result.
|
|
${plan_result}= Run CleverAgents Command
|
|
... plan use ${ACTION_NAME} ${proj_name}
|
|
... --automation-profile trusted
|
|
... --format json
|
|
... expected_rc=None
|
|
... timeout=180s
|
|
Log plan use stdout: ${plan_result.stdout} level=DEBUG
|
|
Log plan use stderr: ${plan_result.stderr} level=DEBUG
|
|
Should Be Equal As Integers ${plan_result.rc} 0
|
|
... msg=plan use failed (rc=${plan_result.rc}). Check DEBUG logs above.
|
|
# Extract plan_id from JSON output for status polling and teardown.
|
|
${plan_data}= Extract JSON From Stdout ${plan_result.stdout}
|
|
${plan_id}= Evaluate $plan_data.get('plan_id', $plan_data.get('id', ''))
|
|
Set Test Variable ${WF18_PLAN_ID} ${plan_id}
|
|
Should Not Be Empty ${plan_id} msg=Could not extract plan_id from plan use output.
|
|
# AC3: Plan launched successfully; plan_id is available.
|
|
Log WF18 plan_id: ${plan_id}
|
|
# Step 6: Execute the plan (drives strategize + execute phases).
|
|
${exec_result}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... expected_rc=None
|
|
... timeout=600s
|
|
Log plan execute stdout: ${exec_result.stdout} level=DEBUG
|
|
Log plan execute stderr: ${exec_result.stderr} level=DEBUG
|
|
Should Be Equal As Integers ${exec_result.rc} 0
|
|
... msg=plan execute failed (rc=${exec_result.rc}). Check DEBUG logs above.
|
|
# Step 7: Apply the plan (writes results back, container commit/push).
|
|
${apply_result}= Run CleverAgents Command
|
|
... plan apply ${plan_id} --yes
|
|
... expected_rc=None
|
|
... timeout=120s
|
|
Log plan apply stdout: ${apply_result.stdout} level=DEBUG
|
|
Log plan apply stderr: ${apply_result.stderr} level=DEBUG
|
|
Should Be Equal As Integers ${apply_result.rc} 0
|
|
... msg=plan apply failed (rc=${apply_result.rc}). Check DEBUG logs above.
|
|
# AC4: Full plan lifecycle (use → execute → apply) completed without OOM.
|
|
Log WF18 Container Clone test passed: full plan lifecycle completed.
|