*** Settings *** Documentation E2E test for Workflow Example 17: Explicit Container with ... Directory Mount (trusted profile). ... ... Exercises resource registration (git-checkout and ... container-instance), project creation with linked container ... resource, execution environment override via project context ... set, action creation with trusted automation profile, and ... full plan lifecycle. ... ... Zero mocking — real CLI, real LLM API keys. Resource common_e2e.resource Suite Setup WF17 Suite Setup Suite Teardown E2E Suite Teardown Force Tags E2E *** Variables *** ${ACTION_BASE} local/wf17-container-mount ${RESOURCE_BASE} local/wf17-mount-repo ${CONTAINER_BASE} local/wf17-container ${PROJECT_BASE} local/wf17-mount-project *** Keywords *** WF17 Suite Setup [Documentation] E2E Suite Setup plus workspace initialisation and ... dynamic actor selection for WF17 container tests. E2E Suite Setup # Initialise the workspace so all CLI commands work. ${init}= Run CleverAgents Command init --force --yes Should Be Equal As Integers ${init.rc} 0 # Generate a unique suffix for resource/project names to avoid UNIQUE # constraint collisions on repeated or parallel E2E runs. ${suffix}= Evaluate __import__('uuid').uuid4().hex[:12] Set Suite Variable ${RUN_SUFFIX} ${suffix} # Compute run-unique names from base names + suffix. Set Suite Variable ${ACTION_NAME} ${ACTION_BASE}-${suffix} Set Suite Variable ${RESOURCE_NAME} ${RESOURCE_BASE}-${suffix} Set Suite Variable ${CONTAINER_NAME} ${CONTAINER_BASE}-${suffix} Set Suite Variable ${PROJECT_NAME} ${PROJECT_BASE}-${suffix} # Dynamically select actor based on available API key. ${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', '')) IF ${has_anthropic} ${actor}= Set Variable anthropic/claude-sonnet-4-20250514 ELSE ${actor}= Set Variable openai/gpt-4o END Set Suite Variable ${ACTOR} ${actor} Create Mountable Repo [Documentation] Create a temp git repo simulating a project that would ... be mounted into a container. Kept minimal (README + ... initial commit) — the fixture's purpose is to provide ... a valid git-checkout resource, not specific file content. ... ``Create Temp Git Repo`` already creates README.md and ... commits it, so no additional git operations are needed. ${repo}= Create Temp Git Repo wf17-mount-project-${RUN_SUFFIX} RETURN ${repo} *** Test Cases *** WF17 Explicit Container With Directory Mount Trusted Profile [Documentation] Trusted-profile workflow: register git-checkout and ... container-instance resources, create project, link ... container resource, set execution environment with ... plan-level override (precedence level 1), and run full ... plan lifecycle with trusted automation. ... ... The ``trusted`` automation profile grants auto-execution ... of tool invocations without user confirmation. This is ... intentional for E2E testing of the WF17 scenario where ... tool invocations should route to the container. [Timeout] 20 minutes Skip If No LLM Keys # ---- Create fixture repo ---- ${repo}= Create Mountable Repo ${branch_result}= Run Process git rev-parse --abbrev-ref HEAD ... cwd=${repo} timeout=60s on_timeout=kill Should Be Equal As Integers ${branch_result.rc} 0 git rev-parse failed: ${branch_result.stderr} ${branch}= Strip String ${branch_result.stdout} # ---- Register git-checkout resource ---- ${r_res}= Run CleverAgents Command ... resource add git-checkout ${RESOURCE_NAME} ... --path ${repo} --branch ${branch} --format json Should Not Contain ${r_res.stdout}${r_res.stderr} Traceback Should Not Contain ${r_res.stdout}${r_res.stderr} INTERNAL Output Should Contain ${r_res} ${RESOURCE_NAME} # ---- Register container-instance resource (WF17 Step 1) ---- # NOTE: --mount flags are not yet available on the CLI. The spec's WF17 # shows dual mounts (resource-ref rw + host-path ro), but the current # 'resource add container-instance' command only supports --image. # Dual-mount testing is deferred until CLI support is implemented. # Tracked in #1078. Log WARN: --mount flags not yet available on CLI — dual-mount test deferred (AC #2, tracked in #1078) WARN ${r_container}= Run CleverAgents Command ... resource add container-instance ${CONTAINER_NAME} ... --image python:3.12-slim --format json Should Not Contain ${r_container.stdout}${r_container.stderr} Traceback Should Not Contain ${r_container.stdout}${r_container.stderr} INTERNAL Output Should Contain ${r_container} ${CONTAINER_NAME} # ---- Create project and link resources (WF17 Step 2) ---- # Following the spec's prescribed sequence: project create without --resource, # then separate link-resource calls for each resource. ${r_proj}= Run CleverAgents Command ... project create ${PROJECT_NAME} --format json Should Not Contain ${r_proj.stdout}${r_proj.stderr} Traceback Should Not Contain ${r_proj.stdout}${r_proj.stderr} INTERNAL Output Should Contain ${r_proj} ${PROJECT_NAME} # Link git-checkout resource to the project ${r_link_repo}= Run CleverAgents Command ... project link-resource ${PROJECT_NAME} ${RESOURCE_NAME} ... --format json Should Not Contain ${r_link_repo.stdout}${r_link_repo.stderr} Traceback Should Not Contain ${r_link_repo.stdout}${r_link_repo.stderr} INTERNAL Output Should Contain ${r_link_repo} ${RESOURCE_NAME} # Link container resource to the project ${r_link}= Run CleverAgents Command ... project link-resource ${PROJECT_NAME} ${CONTAINER_NAME} ... --format json Should Not Contain ${r_link.stdout}${r_link.stderr} Traceback Should Not Contain ${r_link.stdout}${r_link.stderr} INTERNAL Output Should Contain ${r_link} ${CONTAINER_NAME} # Set execution environment to container via project context set (WF17 Step 2) # NOTE: --execution-environment accepts enum values (host/container), not resource # names. The spec shows resource names, but the CLI's ExecutionEnvironment enum only # validates enum values. This is a known spec divergence that is not yet tracked in # a dedicated issue (#1079 covers --execution-env-priority, not this enum issue). ${r_ctx}= Run CleverAgents Command ... project context set ... --execution-environment container ... ${PROJECT_NAME} --format json Should Not Contain ${r_ctx.stdout}${r_ctx.stderr} Traceback Should Not Contain ${r_ctx.stdout}${r_ctx.stderr} INTERNAL # Verify execution environment setting persisted via project context show. # NOTE: 'project context show' returns views/ACMS config but may not expose # the execution_environment field directly. We attempt field extraction and # fall back to logging the output if the field is absent. ${r_ctx_show}= Run CleverAgents Command ... project context show ${PROJECT_NAME} --format json ... expected_rc=None IF ${r_ctx_show.rc} == 0 ${persisted_env}= Safe Parse Json Field ${r_ctx_show.stdout} execution_environment IF "${persisted_env}" != "" Should Be Equal As Strings ${persisted_env.lower()} container ... Execution environment should be persisted as 'container' but got '${persisted_env}' Log Execution environment persisted: ${persisted_env} ELSE Log execution_environment field not present in project context show output — persistence verification deferred (field not exposed by current CLI) WARN END Log project context show output: ${r_ctx_show.stdout} ELSE Log project context show not available or failed (rc=${r_ctx_show.rc}) — verification skipped WARN END # ---- Create action with trusted profile (dynamic actor) ---- ${action_yaml}= Catenate SEPARATOR=\n ... name: ${ACTION_NAME} ... description: "Implement container-based deployment with directory mounts" ... definition_of_done: "Container config and mount paths validated" ... strategy_actor: ${ACTOR} ... execution_actor: ${ACTOR} ${action_path}= Set Variable ${SUITE_HOME}${/}wf17_action_${RUN_SUFFIX}.yaml Create File ${action_path} ${action_yaml} ${r_action}= Run CleverAgents Command ... action create --config ${action_path} --format json Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL Output Should Contain ${r_action} ${ACTION_NAME} # ---- Plan use with execution-env-priority override (WF17 Step 3) ---- # NOTE: Uses plan-level override (precedence level 1) as a workaround. # The spec describes project-level override (precedence level 2) via # 'project context set --execution-env-priority override', but the CLI # only implements this flag on 'plan use'. Project-level override testing # is deferred until CLI support is added. Tracked in #1079 and #1080. # Security note: trusted profile grants auto-execution of tool invocations # without user confirmation. This is intentional for E2E validation. ${r_use}= Run CleverAgents Command ... plan use ${ACTION_NAME} ${PROJECT_NAME} ... --automation-profile trusted ... --execution-environment container ... --execution-env-priority override ... --format json Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL ${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id Should Not Be Empty ${plan_id} msg=Expected plan_id in plan use JSON output Should Match Regexp ${plan_id} ^[0-9A-HJKMNP-TV-Z]{26}$ ... Plan ID should be a valid Crockford Base32 ULID: ${plan_id} Log Plan ID: ${plan_id} # ---- Strategize ---- ${r_strat}= Run CleverAgents Command ... plan execute ${plan_id} ... --format json ... expected_rc=None ... timeout=180s IF ${r_strat.rc} != 0 Fail plan execute (strategize) failed (rc=${r_strat.rc}) stdout=${r_strat.stdout} stderr=${r_strat.stderr} END Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL Output Should Contain ${r_strat} ${plan_id} # ---- Execute ---- # NOTE: 300s timeout (vs M6's 180s) accounts for potential container image # pull overhead on first execution in a clean CI environment. ${r_exec}= Run CleverAgents Command ... plan execute ${plan_id} ... --format json ... expected_rc=None ... timeout=300s IF ${r_exec.rc} != 0 Fail plan execute (execute) failed (rc=${r_exec.rc}) stdout=${r_exec.stdout} stderr=${r_exec.stderr} END Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL Output Should Contain ${r_exec} ${plan_id} # ---- Diff ---- ${r_diff}= Run CleverAgents Command ... plan diff ${plan_id} ... --format json ... timeout=120s Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL Should Not Be Empty ${r_diff.stdout} Plan diff produced no output # Capture HEAD SHA before apply so we can verify the repository changed. ${pre_apply_head}= Run Process git rev-parse HEAD ... cwd=${repo} timeout=60s on_timeout=kill ${pre_apply_sha}= Strip String ${pre_apply_head.stdout} # ---- Apply ---- ${r_apply}= Run CleverAgents Command ... plan apply --yes ${plan_id} ... --format json ... timeout=120s Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL Output Should Contain ${r_apply} ${plan_id} # ---- Post-apply repository observation (informational) ---- # Check whether apply produced new commits in the fixture # repository. Because the plan's file modifications are LLM-driven, # new commits are likely but not guaranteed on every run. We assert # that HEAD changed when we can read it; if HEAD did not move we log # a warning rather than fail, since the absence of new commits does # not indicate a bug in the container workflow itself. ${post_apply_head}= Run Process git rev-parse HEAD ... cwd=${repo} timeout=60s on_timeout=kill IF ${post_apply_head.rc} == 0 ${post_apply_sha}= Strip String ${post_apply_head.stdout} IF "${post_apply_sha}" == "${pre_apply_sha}" Log Post-apply HEAD unchanged (${pre_apply_sha}) — apply may not have produced commits this run WARN ELSE Log Post-apply HEAD moved: ${pre_apply_sha} → ${post_apply_sha} END ELSE Log Post-apply git rev-parse failed (rc=${post_apply_head.rc}) — observation skipped WARN END # ---- Status ---- ${r_status}= Run CleverAgents Command ... plan status ${plan_id} ... --format json ... timeout=120s Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL Should Not Be Empty ${r_status.stdout} Output Should Contain ${r_status} ${plan_id} # ---- Terminal state assertion ---- # After full lifecycle (strategize → execute → diff → apply → status), # the plan should have reached an apply-related phase, matching M6's # Full Flow Apply Step pattern. ${final_phase}= Safe Parse Json Field ${r_status.stdout} phase ${final_state}= Safe Parse Json Field ${r_status.stdout} processing_state Should Not Be Empty ${final_phase} Plan phase should be populated after full lifecycle Should Contain ${final_phase.lower()} apply ... Plan should reach apply phase after apply (got: ${final_phase}) Log Final phase=${final_phase} processing_state=${final_state} # ---- Container routing verification (best-effort) ---- # Attempt to verify that the plan resolved with execution_environment=container # by inspecting the plan status JSON. If the field is present, assert its value; # if absent, log the gap — direct container routing verification is not yet fully # feasible at the E2E level because the CLI does not emit routing-specific # indicators beyond echoed arguments. # NOTE: Container routing verification gap is not yet tracked in a dedicated # issue. A tracking issue should be created to cover CLI-level routing # indicators for container execution. ${exec_env}= Safe Parse Json Field ${r_status.stdout} execution_environment IF "${exec_env}" != "" Should Be Equal As Strings ${exec_env.lower()} container ... Execution environment should be 'container' but got '${exec_env}' Log Container routing verified: execution_environment=${exec_env} ELSE Log execution_environment field not present in plan status JSON — container routing verification gap (plan completed full lifecycle with execution-environment=container and execution-env-priority=override) WARN END WF17 TDD Dual Mount Registration [Documentation] Regression guard for AC #2: Register container-instance ... resource with dual mounts (resource-ref rw + host-path ro). ... Bug #1078 is now fixed — --mount flag is implemented on ... ``resource add container-instance``. ... Tracked in #1078. [Tags] tdd_issue tdd_issue_1078 [Timeout] 5 minutes # No LLM keys needed — this exercises pure CLI flag parsing. # Regression guard: --mount flag is now implemented on the CLI. ${r_container}= Run CleverAgents Command ... resource add container-instance ${CONTAINER_BASE}-mount-${RUN_SUFFIX} ... --image python:3.12-slim ... --mount ${RESOURCE_BASE}-${RUN_SUFFIX}:/workspace ... --mount /var/shared/config:/config:ro ... --format json ... expected_rc=None Should Be Equal As Integers ${r_container.rc} 0 ... resource add container-instance with --mount should succeed (bug #1078) Output Should Contain ${r_container} ${CONTAINER_BASE}-mount-${RUN_SUFFIX} # Verify mount information is present in the output ${combined}= Set Variable ${r_container.stdout}${r_container.stderr} Should Contain ${combined} /workspace ... Mount path /workspace should appear in resource output WF17 TDD Project Level Execution Env Priority Override [Documentation] TDD test for AC #3: Set execution environment override priority ... at the project level via ``project context set ... --execution-env-priority override``. ... Regression test for #1079: --execution-env-priority flag ... is now implemented on ``project context set``. [Tags] tdd_issue tdd_issue_1079 [Timeout] 5 minutes # Create a lightweight project for this TDD test so it does not depend on # the main test's side effects. ${tdd_proj}= Set Variable ${PROJECT_BASE}-tdd-ctx-${RUN_SUFFIX} ${r_proj}= Run CleverAgents Command ... project create ${tdd_proj} --format json ... expected_rc=None IF ${r_proj.rc} != 0 Fail TDD setup: project create failed (rc=${r_proj.rc}): ${r_proj.stderr} END # Attempt to set execution-env-priority at the project level. # This should succeed once #1079 is fixed. ${r_ctx}= Run CleverAgents Command ... project context set ... --execution-environment container ... --execution-env-priority override ... ${tdd_proj} --format json ... expected_rc=None Should Be Equal As Integers ${r_ctx.rc} 0 ... project context set --execution-env-priority should succeed (bug #1079) # Verify the setting persisted via project context show ${r_show}= Run CleverAgents Command ... project context show ${tdd_proj} --format json Should Be Equal As Integers ${r_show.rc} 0 ${show_combined}= Set Variable ${r_show.stdout}${r_show.stderr} Should Contain ${show_combined.lower()} override ... Project context should show execution-env-priority override WF17 TDD Precedence Level 2 Project Override Resolution [Documentation] TDD test for AC #5: Verify execution environment resolves via ... project-level override (precedence level 2) when no plan-level ... override is present. ... Regression test for #1079 and #1080: project-level ... execution-env-priority is now implemented and the ... resolution logic honours it. [Tags] tdd_issue tdd_issue_1080 [Timeout] 10 minutes # Create lightweight project and action for this TDD test so it does not # depend on the main test's side effects. ${tdd_proj}= Set Variable ${PROJECT_BASE}-tdd-prec-${RUN_SUFFIX} ${r_proj}= Run CleverAgents Command ... project create ${tdd_proj} --format json ... expected_rc=None IF ${r_proj.rc} != 0 Fail TDD setup: project create failed (rc=${r_proj.rc}): ${r_proj.stderr} END # Dynamically select actor based on available API key. ${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', '')) IF ${has_anthropic} ${tdd_actor}= Set Variable anthropic/claude-sonnet-4-20250514 ELSE ${tdd_actor}= Set Variable openai/gpt-4o END ${tdd_action_name}= Set Variable ${ACTION_BASE}-tdd-prec-${RUN_SUFFIX} ${tdd_action_yaml}= Catenate SEPARATOR=\n ... name: ${tdd_action_name} ... description: "TDD precedence level 2 test action" ... definition_of_done: "Verify project-level override resolution" ... strategy_actor: ${tdd_actor} ... execution_actor: ${tdd_actor} ${tdd_action_path}= Set Variable ${SUITE_HOME}${/}wf17_tdd_prec_action_${RUN_SUFFIX}.yaml Create File ${tdd_action_path} ${tdd_action_yaml} ${r_action}= Run CleverAgents Command ... action create --config ${tdd_action_path} --format json ... expected_rc=None IF ${r_action.rc} != 0 Fail TDD setup: action create failed (rc=${r_action.rc}): ${r_action.stderr} END # Set project-level execution environment with override priority. # This depends on #1079 being fixed first. ${r_ctx}= Run CleverAgents Command ... project context set ... --execution-environment container ... --execution-env-priority override ... ${tdd_proj} --format json ... expected_rc=None Should Be Equal As Integers ${r_ctx.rc} 0 ... project context set --execution-env-priority should succeed (depends on #1079) # Create a plan WITHOUT plan-level --execution-env-priority override. # The execution environment should still resolve to container via # project-level override (precedence level 2). ${r_use}= Run CleverAgents Command ... plan use ${tdd_action_name} ${tdd_proj} ... --automation-profile trusted ... --format json ... expected_rc=None IF ${r_use.rc} != 0 Fail plan use failed (rc=${r_use.rc}) — cannot verify precedence level 2: ${r_use.stderr} END ${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id Should Not Be Empty ${plan_id} msg=Expected plan_id in plan use output # Check plan status for execution environment resolution. ${r_status}= Run CleverAgents Command ... plan status ${plan_id} --format json ... expected_rc=None timeout=120s Should Be Equal As Integers ${r_status.rc} 0 # The plan should show execution_environment=container resolved via # project-level override (precedence level 2), not plan-level. ${exec_env}= Safe Parse Json Field ${r_status.stdout} execution_environment Should Be Equal As Strings ${exec_env.lower()} container ... Execution environment should resolve to container via project-level override (precedence level 2, bug #1080)