forked from cleveragents/cleveragents-core
freemo
48cff5cfe0
Renames `plan lifecycle-list` to `plan list` and `plan lifecycle-apply` to `plan apply` to align with the specification's canonical command names. Removes legacy V2 plan commands that occupied those names. - Renamed CLI command registrations from lifecycle-list/lifecycle-apply to list/apply - Removed legacy V2 apply and list commands (~200 lines) - Updated apply shortcut in main.py to delegate to v3 lifecycle - Added defensive null check for plan existence in apply command - Updated 63+ test, doc, and benchmark files for consistency Closes #881 Co-authored-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me> Co-committed-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me>
273 lines
15 KiB
Plaintext
273 lines
15 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E test for Workflow Example 18: Container with Remote Repo
|
|
... Clone (trusted profile).
|
|
...
|
|
... Exercises resource registration with container-instance and
|
|
... --clone-into flag for remote repo clone, two-step project
|
|
... creation and linking, action with trusted automation profile,
|
|
... plan-level execution environment with fallback priority, and
|
|
... full plan lifecycle including apply with container commit/push.
|
|
...
|
|
... Zero mocking — real CLI, real LLM API keys.
|
|
Resource common_e2e.resource
|
|
Suite Setup WF18 Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
Force Tags E2E
|
|
|
|
*** Variables ***
|
|
${ACTION_PREFIX} local/wf18-clone-action
|
|
${RESOURCE_PREFIX} local/wf18-clone-res
|
|
${PROJECT_PREFIX} local/wf18-clone-proj
|
|
|
|
*** Keywords ***
|
|
WF18 Suite Setup
|
|
[Documentation] E2E Suite Setup plus unique suffix generation and dynamic
|
|
... actor selection for WF18 tests.
|
|
E2E Suite Setup
|
|
# Initialise the workspace so CLI commands work.
|
|
${init}= Run CleverAgents Command init --force --yes
|
|
Should Be Equal As Integers ${init.rc} 0
|
|
# Generate a unique suffix for resource/project names to avoid UNIQUE
|
|
# constraint collisions on repeated E2E runs or parallel CI.
|
|
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
|
|
Set Suite Variable ${RUN_SUFFIX} ${suffix}
|
|
# Pick an actor that matches the available API key.
|
|
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
|
|
IF ${has_anthropic}
|
|
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
|
|
ELSE
|
|
${actor}= Set Variable openai/gpt-4o
|
|
END
|
|
Set Suite Variable ${LLM_ACTOR} ${actor}
|
|
# Compute unique action name using the run suffix for parallel CI safety.
|
|
${action_name}= Set Variable ${ACTION_PREFIX}-${RUN_SUFFIX}
|
|
Set Suite Variable ${ACTION_NAME} ${action_name}
|
|
|
|
Create Remote Clone Repo
|
|
[Documentation] Create a temp git repo simulating a project that would
|
|
... be cloned from a remote repository into a container.
|
|
${repo}= Create Temp Git Repo wf18-remote-clone-${RUN_SUFFIX}
|
|
Create Directory ${repo}${/}src
|
|
Create Directory ${repo}${/}deploy
|
|
${app_content}= Catenate SEPARATOR=\n
|
|
... """Application for remote deployment via container clone."""
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... class DeploymentManager:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Manages deployment lifecycle."""
|
|
... ${EMPTY}
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}def __init__(self, repo_url, branch="main"):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.repo_url = repo_url
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.branch = branch
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.deployed = False
|
|
... ${EMPTY}
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}def deploy(self):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}"""Execute deployment."""
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.deployed = True
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"status": "deployed", "branch": self.branch}
|
|
Create File ${repo}${/}src${/}deploy_manager.py ${app_content}
|
|
${deploy_config}= Catenate SEPARATOR=\n
|
|
... # Deployment configuration
|
|
... CONTAINER_IMAGE=python:3.12-slim
|
|
... CLONE_DEPTH=1
|
|
... DEPLOY_TIMEOUT=300
|
|
... HEALTH_CHECK_INTERVAL=10
|
|
Create File ${repo}${/}deploy${/}config.env ${deploy_config}
|
|
# NOTE: This Dockerfile is fixture data only — never built by this test.
|
|
# It simulates a project that would be cloned into a container.
|
|
${dockerfile}= Catenate SEPARATOR=\n
|
|
... FROM python:3.12-slim
|
|
... RUN apt-get update && apt-get install -y git
|
|
... WORKDIR /workspace
|
|
... ARG REPO_URL
|
|
... RUN git clone --depth 1 \${REPO_URL} .
|
|
... CMD ["python", "-m", "src.deploy_manager"]
|
|
Create File ${repo}${/}Dockerfile ${dockerfile}
|
|
Create File ${repo}${/}src${/}__init__.py \n
|
|
${git_add}= Run Process git add . cwd=${repo}
|
|
Should Be Equal As Integers ${git_add.rc} 0
|
|
... Fixture git add failed: ${git_add.stderr}
|
|
${git_commit}= Run Process git commit -m Initial remote-clone container project cwd=${repo}
|
|
Should Be Equal As Integers ${git_commit.rc} 0
|
|
... Fixture git commit failed: ${git_commit.stderr}
|
|
RETURN ${repo}
|
|
|
|
*** Test Cases ***
|
|
WF18 Container With Remote Repo Clone Trusted Profile
|
|
[Documentation] Trusted-profile workflow: container-instance resource with
|
|
... --clone-into for remote repo clone, two-step project setup,
|
|
... plan-level execution environment with fallback priority, and
|
|
... full plan lifecycle including apply with container commit/push.
|
|
[Timeout] 20 minutes
|
|
[Teardown] Log WF18 Container Clone test completed.
|
|
Skip If No LLM Keys
|
|
|
|
# ---- Create fixture repo ----
|
|
${repo}= Create Remote Clone Repo
|
|
|
|
# ---- Register container-instance resource with --clone-into ----
|
|
${resource_name}= Set Variable ${RESOURCE_PREFIX}-${RUN_SUFFIX}
|
|
${r_resource}= Run CleverAgents Command
|
|
... resource add container-instance ${resource_name}
|
|
... --clone-into ${repo}:/workspace
|
|
... --format plain
|
|
Should Not Contain ${r_resource.stdout}${r_resource.stderr} Traceback
|
|
Should Not Contain ${r_resource.stdout}${r_resource.stderr} INTERNAL
|
|
Output Should Contain ${r_resource} ${resource_name}
|
|
Output Should Contain ${r_resource} container-instance
|
|
|
|
# ---- Create project (two-step: create then link-resource per spec) ----
|
|
${project_name}= Set Variable ${PROJECT_PREFIX}-${RUN_SUFFIX}
|
|
${r_project_create}= Run CleverAgents Command
|
|
... project create ${project_name}
|
|
... --format plain
|
|
Should Not Contain ${r_project_create.stdout}${r_project_create.stderr} Traceback
|
|
Should Not Contain ${r_project_create.stdout}${r_project_create.stderr} INTERNAL
|
|
Output Should Contain ${r_project_create} ${project_name}
|
|
|
|
${r_link}= Run CleverAgents Command
|
|
... project link-resource ${project_name} ${resource_name}
|
|
... --format plain
|
|
Should Not Contain ${r_link.stdout}${r_link.stderr} Traceback
|
|
Should Not Contain ${r_link.stdout}${r_link.stderr} INTERNAL
|
|
Output Should Contain ${r_link} ${resource_name}
|
|
|
|
# ---- Create action with dynamic actor selection ----
|
|
${action_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${ACTION_NAME}
|
|
... description: Implement container-based remote deployment with repo cloning
|
|
... definition_of_done: Deployment pipeline configured and validated
|
|
... strategy_actor: ${LLM_ACTOR}
|
|
... execution_actor: ${LLM_ACTOR}
|
|
${action_path}= Set Variable ${SUITE_HOME}${/}wf18_action.yaml
|
|
Create File ${action_path} ${action_yaml}
|
|
${r_action}= Run CleverAgents Command
|
|
... action create --config ${action_path}
|
|
... --format plain
|
|
Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback
|
|
Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL
|
|
Output Should Contain ${r_action} ${ACTION_NAME}
|
|
|
|
# ---- Plan use with trusted profile, execution environment, and fallback priority ----
|
|
# NOTE: The spec Example 18 shows --execution-environment cloud/build-env
|
|
# (a resource name). The current CLI implementation validates against the
|
|
# ExecutionEnvironment enum which only accepts "host" or "container".
|
|
# Resource-name-based resolution is future work; we use "container" here to
|
|
# exercise the plan-level fallback routing path (precedence level 4).
|
|
${r_use}= Run CleverAgents Command
|
|
... plan use ${ACTION_NAME} ${project_name}
|
|
... --automation-profile trusted
|
|
... --execution-environment container
|
|
... --execution-env-priority fallback
|
|
... --format plain
|
|
... timeout=120s
|
|
Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback
|
|
Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL
|
|
Should Not Be Empty ${r_use.stdout} Plan use produced no output
|
|
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-HJKMNP-TV-Z]{26}
|
|
Should Not Be Empty ${plan_ids} msg=Expected ULID plan ID in plan use output
|
|
${plan_id}= Set Variable ${plan_ids}[0]
|
|
Log Plan ID: ${plan_id}
|
|
|
|
# ---- Strategize ----
|
|
# The plan lifecycle uses two sequential ``plan execute`` calls:
|
|
# the first advances the plan through the Strategize phase, and
|
|
# the second advances it through the Execute phase.
|
|
${r_strat}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format plain
|
|
... timeout=180s
|
|
Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback
|
|
Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL
|
|
Should Not Be Empty ${r_strat.stdout} Strategize produced no output
|
|
Output Should Contain ${r_strat} ${plan_id}
|
|
|
|
# ---- Execute ----
|
|
${r_exec}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format plain
|
|
... timeout=300s
|
|
Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback
|
|
Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL
|
|
Should Not Be Empty ${r_exec.stdout} Execute produced no output
|
|
Output Should Contain ${r_exec} ${plan_id}
|
|
# Verify container start and remote repo clone evidence (AC-4).
|
|
# Soft assertion: LLM output is non-deterministic so we warn rather than fail.
|
|
${exec_combined}= Set Variable ${r_exec.stdout}\n${r_exec.stderr}
|
|
${exec_lower}= Evaluate ($exec_combined).lower()
|
|
${has_container_evidence}= Evaluate 'container' in $exec_lower or 'clone' in $exec_lower or 'workspace' in $exec_lower or 'starting' in $exec_lower
|
|
Log Container/clone evidence in execute output: ${has_container_evidence}
|
|
Run Keyword And Warn On Failure Should Be True ${has_container_evidence}
|
|
... Execute output should contain evidence of container start or clone operation
|
|
# Verify execution environment resolution — plan fallback, precedence level 4 (AC-5).
|
|
${has_env_resolution}= Evaluate 'fallback' in $exec_lower or 'execution' in $exec_lower or 'environment' in $exec_lower or 'precedence' in $exec_lower or 'resolved' in $exec_lower
|
|
Log Execution environment resolution evidence: ${has_env_resolution}
|
|
Run Keyword And Warn On Failure Should Be True ${has_env_resolution}
|
|
... Execute output should contain execution environment resolution evidence (fallback/precedence)
|
|
|
|
# ---- Diff ----
|
|
${r_diff}= Run CleverAgents Command
|
|
... plan diff ${plan_id}
|
|
... --format plain
|
|
... timeout=60s
|
|
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
|
|
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
|
|
Should Not Be Empty ${r_diff.stdout} Plan diff produced no output
|
|
|
|
# ---- Capture pre-apply commit count ----
|
|
${pre_log}= Run Process git log --oneline cwd=${repo}
|
|
Should Be Equal As Integers ${pre_log.rc} 0
|
|
... Pre-apply git log failed: ${pre_log.stderr}
|
|
${pre_commit_count}= Get Line Count ${pre_log.stdout}
|
|
Log Pre-apply commit count: ${pre_commit_count}
|
|
|
|
# ---- Apply ----
|
|
${r_apply}= Run CleverAgents Command
|
|
... plan apply --yes ${plan_id}
|
|
... --format plain
|
|
... timeout=120s
|
|
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
|
|
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
|
|
Should Not Be Empty ${r_apply.stdout} Apply produced no output
|
|
Output Should Contain ${r_apply} ${plan_id}
|
|
# Verify apply output references container commit and push (AC-6).
|
|
# Soft assertion: LLM output is non-deterministic so we warn rather than fail.
|
|
${apply_combined}= Set Variable ${r_apply.stdout}\n${r_apply.stderr}
|
|
${apply_lower}= Evaluate ($apply_combined).lower()
|
|
${has_apply_evidence}= Evaluate 'commit' in $apply_lower or 'push' in $apply_lower or 'applied' in $apply_lower or 'container' in $apply_lower or 'origin' in $apply_lower
|
|
Log Apply container commit/push evidence: ${has_apply_evidence}
|
|
Run Keyword And Warn On Failure Should Be True ${has_apply_evidence}
|
|
... Apply output should contain evidence of commit/push from container
|
|
|
|
# ---- Status — verify terminal state ----
|
|
${r_status}= Run CleverAgents Command
|
|
... plan status ${plan_id}
|
|
... --format plain
|
|
... timeout=60s
|
|
Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback
|
|
Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL
|
|
Should Not Be Empty ${r_status.stdout} Plan status produced no output
|
|
Output Should Contain ${r_status} ${plan_id}
|
|
# Verify plan reached a terminal state (applied phase)
|
|
${status_lower}= Evaluate ($r_status.stdout).lower()
|
|
${is_terminal}= Evaluate 'applied' in $status_lower or 'complete' in $status_lower or 'done' in $status_lower or 'finished' in $status_lower or 'apply' in $status_lower
|
|
Should Be True ${is_terminal}
|
|
... Plan status should indicate a terminal/applied state after apply
|
|
|
|
# ---- Verify repo state after apply ----
|
|
${post_log}= Run Process git log --oneline cwd=${repo}
|
|
Should Be Equal As Integers ${post_log.rc} 0
|
|
... Post-apply git log failed: ${post_log.stderr}
|
|
Log Git log after apply: ${post_log.stdout}
|
|
${post_commit_count}= Get Line Count ${post_log.stdout}
|
|
# Apply may or may not create new commits depending on whether the LLM
|
|
# generated file changes. Log the delta for diagnostics but only
|
|
# hard-assert that the repo is still valid (at least the fixture commits).
|
|
${delta}= Evaluate ${post_commit_count} - ${pre_commit_count}
|
|
Log Commit count delta after apply: ${delta} (pre=${pre_commit_count}, post=${post_commit_count})
|
|
Should Be True ${post_commit_count} >= ${pre_commit_count}
|
|
... Repo commit count should not decrease after apply (pre=${pre_commit_count}, post=${post_commit_count})
|
|
# Verify fixture files are still present in the repo after the full lifecycle.
|
|
File Should Exist ${repo}${/}src${/}deploy_manager.py
|
|
File Should Exist ${repo}${/}Dockerfile
|