cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

test(e2e): workflow example 17 — explicit container with directory mount (trusted profile) #819

Merged
hurui200320 merged 1 commits from test/e2e-wf17-explicit-container into master 2026-03-30 08:25:36 +00:00
Conversation 5 Commits 1 Files Changed 2
+459
2 changed files with 459 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+9
View File
@@ -12,6 +12,15 @@
Known limitations: `plan prompt` not yet implemented as CLI subcommand,
action `--arg` omitted due to UNIQUE constraint bug, validation registration
omitted pending independent validation. (#758)
- Added E2E test for Workflow Example 17: explicit container with directory
mount using trusted automation profile. Exercises container-instance resource
registration, project link-resource, execution environment setting via
project context set, plan-level execution-env-priority override via plan use,
and full plan lifecycle with dynamic actor selection. Includes TDD
bug-capture tests for deferred acceptance criteria: dual mount registration
(#1078), project-level execution-env-priority (#1079), and precedence
level 2 resolution (#1080).
(`robot/e2e/wf17_explicit_container.robot`) (#763)
- Added `correction_attempts` table per specification DDL with
`CorrectionAttemptModel` ORM, `CorrectionAttemptRecord` domain model,
`CorrectionAttemptRepository` CRUD layer, Alembic migration, and
robot/e2e/wf17_explicit_container.robot
+450
View File
@@ -0,0 +1,450 @@
*** Settings ***
Documentation E2E test for Workflow Example 17: Explicit Container with
... Directory Mount (trusted profile).
...
... Exercises resource registration (git-checkout and
... container-instance), project creation with linked container
... resource, execution environment override via project context
... set, action creation with trusted automation profile, and
... full plan lifecycle.
...
... Zero mocking — real CLI, real LLM API keys.
Resource common_e2e.resource
Suite Setup WF17 Suite Setup
Suite Teardown E2E Suite Teardown
Force Tags E2E
*** Variables ***
${ACTION_BASE} local/wf17-container-mount
${RESOURCE_BASE} local/wf17-mount-repo
${CONTAINER_BASE} local/wf17-container
${PROJECT_BASE} local/wf17-mount-project
*** Keywords ***
WF17 Suite Setup
[Documentation] E2E Suite Setup plus workspace initialisation and
... dynamic actor selection for WF17 container tests.
E2E Suite Setup
# Initialise the workspace so all CLI commands work.
${init}= Run CleverAgents Command init --force --yes
Should Be Equal As Integers ${init.rc} 0
# Generate a unique suffix for resource/project names to avoid UNIQUE
# constraint collisions on repeated or parallel E2E runs.
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
# Compute run-unique names from base names + suffix.
Set Suite Variable ${ACTION_NAME} ${ACTION_BASE}-${suffix}
Set Suite Variable ${RESOURCE_NAME} ${RESOURCE_BASE}-${suffix}
Set Suite Variable ${CONTAINER_NAME} ${CONTAINER_BASE}-${suffix}
Set Suite Variable ${PROJECT_NAME} ${PROJECT_BASE}-${suffix}
# Dynamically select actor based on available API key.
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
IF ${has_anthropic}
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
ELSE
${actor}= Set Variable openai/gpt-4o
END
Set Suite Variable ${ACTOR} ${actor}
Create Mountable Repo
[Documentation] Create a temp git repo simulating a project that would
... be mounted into a container. Kept minimal (README +
... initial commit) — the fixture's purpose is to provide
... a valid git-checkout resource, not specific file content.
... ``Create Temp Git Repo`` already creates README.md and
... commits it, so no additional git operations are needed.
${repo}= Create Temp Git Repo wf17-mount-project-${RUN_SUFFIX}
RETURN ${repo}
*** Test Cases ***
WF17 Explicit Container With Directory Mount Trusted Profile
[Documentation] Trusted-profile workflow: register git-checkout and
... container-instance resources, create project, link
... container resource, set execution environment with
... plan-level override (precedence level 1), and run full
... plan lifecycle with trusted automation.
...
... The ``trusted`` automation profile grants auto-execution
... of tool invocations without user confirmation. This is
... intentional for E2E testing of the WF17 scenario where
... tool invocations should route to the container.
[Timeout] 20 minutes
Skip If No LLM Keys
# ---- Create fixture repo ----
${repo}= Create Mountable Repo
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD
... cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${branch_result.rc} 0 git rev-parse failed: ${branch_result.stderr}
${branch}= Strip String ${branch_result.stdout}
# ---- Register git-checkout resource ----
${r_res}= Run CleverAgents Command
... resource add git-checkout ${RESOURCE_NAME}
... --path ${repo} --branch ${branch} --format json
Should Not Contain ${r_res.stdout}${r_res.stderr} Traceback
Should Not Contain ${r_res.stdout}${r_res.stderr} INTERNAL
Output Should Contain ${r_res} ${RESOURCE_NAME}
# ---- Register container-instance resource (WF17 Step 1) ----
# NOTE: --mount flags are not yet available on the CLI. The spec's WF17
# shows dual mounts (resource-ref rw + host-path ro), but the current
# 'resource add container-instance' command only supports --image.
# Dual-mount testing is deferred until CLI support is implemented.
# Tracked in #1078.
Log WARN: --mount flags not yet available on CLI — dual-mount test deferred (AC #2, tracked in #1078) WARN
${r_container}= Run CleverAgents Command
... resource add container-instance ${CONTAINER_NAME}
... --image python:3.12-slim --format json
Should Not Contain ${r_container.stdout}${r_container.stderr} Traceback
Should Not Contain ${r_container.stdout}${r_container.stderr} INTERNAL
Output Should Contain ${r_container} ${CONTAINER_NAME}
# ---- Create project and link resources (WF17 Step 2) ----
# Following the spec's prescribed sequence: project create without --resource,
# then separate link-resource calls for each resource.
${r_proj}= Run CleverAgents Command
... project create ${PROJECT_NAME} --format json
Should Not Contain ${r_proj.stdout}${r_proj.stderr} Traceback
Should Not Contain ${r_proj.stdout}${r_proj.stderr} INTERNAL
Output Should Contain ${r_proj} ${PROJECT_NAME}
# Link git-checkout resource to the project
${r_link_repo}= Run CleverAgents Command
... project link-resource ${PROJECT_NAME} ${RESOURCE_NAME}
... --format json
Should Not Contain ${r_link_repo.stdout}${r_link_repo.stderr} Traceback
Should Not Contain ${r_link_repo.stdout}${r_link_repo.stderr} INTERNAL
Output Should Contain ${r_link_repo} ${RESOURCE_NAME}
# Link container resource to the project
${r_link}= Run CleverAgents Command
... project link-resource ${PROJECT_NAME} ${CONTAINER_NAME}
... --format json
Should Not Contain ${r_link.stdout}${r_link.stderr} Traceback
Should Not Contain ${r_link.stdout}${r_link.stderr} INTERNAL
Output Should Contain ${r_link} ${CONTAINER_NAME}
# Set execution environment to container via project context set (WF17 Step 2)
# NOTE: --execution-environment accepts enum values (host/container), not resource
# names. The spec shows resource names, but the CLI's ExecutionEnvironment enum only
# validates enum values. This is a known spec divergence that is not yet tracked in
# a dedicated issue (#1079 covers --execution-env-priority, not this enum issue).
${r_ctx}= Run CleverAgents Command
... project context set
... --execution-environment container
... ${PROJECT_NAME} --format json
Should Not Contain ${r_ctx.stdout}${r_ctx.stderr} Traceback
Should Not Contain ${r_ctx.stdout}${r_ctx.stderr} INTERNAL
# Verify execution environment setting persisted via project context show.
# NOTE: 'project context show' returns views/ACMS config but may not expose
# the execution_environment field directly. We attempt field extraction and
# fall back to logging the output if the field is absent.
${r_ctx_show}= Run CleverAgents Command
... project context show ${PROJECT_NAME} --format json
... expected_rc=None
IF ${r_ctx_show.rc} == 0
${persisted_env}= Safe Parse Json Field ${r_ctx_show.stdout} execution_environment
IF "${persisted_env}" != ""
Should Be Equal As Strings ${persisted_env.lower()} container
... Execution environment should be persisted as 'container' but got '${persisted_env}'
Log Execution environment persisted: ${persisted_env}
ELSE
Log execution_environment field not present in project context show output — persistence verification deferred (field not exposed by current CLI) WARN
END
Log project context show output: ${r_ctx_show.stdout}
ELSE
Log project context show not available or failed (rc=${r_ctx_show.rc}) — verification skipped WARN
END
# ---- Create action with trusted profile (dynamic actor) ----
${action_yaml}= Catenate SEPARATOR=\n
... name: ${ACTION_NAME}
... description: "Implement container-based deployment with directory mounts"
... definition_of_done: "Container config and mount paths validated"
... strategy_actor: ${ACTOR}
... execution_actor: ${ACTOR}
${action_path}= Set Variable ${SUITE_HOME}${/}wf17_action_${RUN_SUFFIX}.yaml
Create File ${action_path} ${action_yaml}
${r_action}= Run CleverAgents Command
... action create --config ${action_path} --format json
Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback
Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL
Output Should Contain ${r_action} ${ACTION_NAME}
# ---- Plan use with execution-env-priority override (WF17 Step 3) ----
# NOTE: Uses plan-level override (precedence level 1) as a workaround.
# The spec describes project-level override (precedence level 2) via
# 'project context set --execution-env-priority override', but the CLI
# only implements this flag on 'plan use'. Project-level override testing
# is deferred until CLI support is added. Tracked in #1079 and #1080.
# Security note: trusted profile grants auto-execution of tool invocations
# without user confirmation. This is intentional for E2E validation.
${r_use}= Run CleverAgents Command
... plan use ${ACTION_NAME} ${PROJECT_NAME}
... --automation-profile trusted
... --execution-environment container
... --execution-env-priority override
... --format json
Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback
Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL
${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id
Should Not Be Empty ${plan_id} msg=Expected plan_id in plan use JSON output
Should Match Regexp ${plan_id} ^[0-9A-HJKMNP-TV-Z]{26}$
... Plan ID should be a valid Crockford Base32 ULID: ${plan_id}
Log Plan ID: ${plan_id}
# ---- Strategize ----
${r_strat}= Run CleverAgents Command
... plan execute ${plan_id}
... --format json
... expected_rc=None
... timeout=180s
IF ${r_strat.rc} != 0
Fail plan execute (strategize) failed (rc=${r_strat.rc}) stdout=${r_strat.stdout} stderr=${r_strat.stderr}
END
Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback
Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL
Output Should Contain ${r_strat} ${plan_id}
# ---- Execute ----
# NOTE: 300s timeout (vs M6's 180s) accounts for potential container image
# pull overhead on first execution in a clean CI environment.
${r_exec}= Run CleverAgents Command
... plan execute ${plan_id}
... --format json
... expected_rc=None
... timeout=300s
IF ${r_exec.rc} != 0
Fail plan execute (execute) failed (rc=${r_exec.rc}) stdout=${r_exec.stdout} stderr=${r_exec.stderr}
END
Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback
Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL
Output Should Contain ${r_exec} ${plan_id}
# ---- Diff ----
${r_diff}= Run CleverAgents Command
... plan diff ${plan_id}
... --format json
... timeout=120s
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
Should Not Be Empty ${r_diff.stdout} Plan diff produced no output
# Capture HEAD SHA before apply so we can verify the repository changed.
${pre_apply_head}= Run Process git rev-parse HEAD
... cwd=${repo} timeout=60s on_timeout=kill
${pre_apply_sha}= Strip String ${pre_apply_head.stdout}
# ---- Apply ----
${r_apply}= Run CleverAgents Command
... plan lifecycle-apply --yes ${plan_id}
... --format json
... timeout=120s
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
Output Should Contain ${r_apply} ${plan_id}
# ---- Post-apply repository observation (informational) ----
# Check whether lifecycle-apply produced new commits in the fixture
# repository. Because the plan's file modifications are LLM-driven,
# new commits are likely but not guaranteed on every run. We assert
# that HEAD changed when we can read it; if HEAD did not move we log
# a warning rather than fail, since the absence of new commits does
# not indicate a bug in the container workflow itself.
${post_apply_head}= Run Process git rev-parse HEAD
... cwd=${repo} timeout=60s on_timeout=kill
IF ${post_apply_head.rc} == 0
${post_apply_sha}= Strip String ${post_apply_head.stdout}
IF "${post_apply_sha}" == "${pre_apply_sha}"
Log Post-apply HEAD unchanged (${pre_apply_sha}) — lifecycle-apply may not have produced commits this run WARN
ELSE
Log Post-apply HEAD moved: ${pre_apply_sha} → ${post_apply_sha}
END
ELSE
Log Post-apply git rev-parse failed (rc=${post_apply_head.rc}) — observation skipped WARN
END
# ---- Status ----
${r_status}= Run CleverAgents Command
... plan status ${plan_id}
... --format json
... timeout=120s
Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback
Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL
Should Not Be Empty ${r_status.stdout}
Output Should Contain ${r_status} ${plan_id}
# ---- Terminal state assertion ----
# After full lifecycle (strategize → execute → diff → apply → status),
# the plan should have reached an apply-related phase, matching M6's
# Full Flow Apply Step pattern.
${final_phase}= Safe Parse Json Field ${r_status.stdout} phase
${final_state}= Safe Parse Json Field ${r_status.stdout} processing_state
Should Not Be Empty ${final_phase} Plan phase should be populated after full lifecycle
Should Contain ${final_phase.lower()} apply
... Plan should reach apply phase after lifecycle-apply (got: ${final_phase})
Log Final phase=${final_phase} processing_state=${final_state}
# ---- Container routing verification (best-effort) ----
# Attempt to verify that the plan resolved with execution_environment=container
# by inspecting the plan status JSON. If the field is present, assert its value;
# if absent, log the gap — direct container routing verification is not yet fully
# feasible at the E2E level because the CLI does not emit routing-specific
# indicators beyond echoed arguments.
# NOTE: Container routing verification gap is not yet tracked in a dedicated
# issue. A tracking issue should be created to cover CLI-level routing
# indicators for container execution.
${exec_env}= Safe Parse Json Field ${r_status.stdout} execution_environment
IF "${exec_env}" != ""
Should Be Equal As Strings ${exec_env.lower()} container
... Execution environment should be 'container' but got '${exec_env}'
Log Container routing verified: execution_environment=${exec_env}
ELSE
Log execution_environment field not present in plan status JSON — container routing verification gap (plan completed full lifecycle with execution-environment=container and execution-env-priority=override) WARN
END
WF17 TDD Dual Mount Registration
[Documentation] Regression guard for AC #2: Register container-instance
... resource with dual mounts (resource-ref rw + host-path ro).
... Bug #1078 is now fixed — --mount flag is implemented on
... ``resource add container-instance``.
... Tracked in #1078.
[Tags] tdd_issue tdd_issue_1078
[Timeout] 5 minutes
# No LLM keys needed — this exercises pure CLI flag parsing.
# Regression guard: --mount flag is now implemented on the CLI.
${r_container}= Run CleverAgents Command
... resource add container-instance ${CONTAINER_BASE}-mount-${RUN_SUFFIX}
... --image python:3.12-slim
... --mount ${RESOURCE_BASE}-${RUN_SUFFIX}:/workspace
... --mount /var/shared/config:/config:ro
... --format json
... expected_rc=None
Should Be Equal As Integers ${r_container.rc} 0
... resource add container-instance with --mount should succeed (bug #1078)
Output Should Contain ${r_container} ${CONTAINER_BASE}-mount-${RUN_SUFFIX}
# Verify mount information is present in the output
${combined}= Set Variable ${r_container.stdout}${r_container.stderr}
Should Contain ${combined} /workspace
... Mount path /workspace should appear in resource output
WF17 TDD Project Level Execution Env Priority Override
[Documentation] TDD test for AC #3: Set execution environment override priority
... at the project level via ``project context set
... --execution-env-priority override``.
... Expected to fail until --execution-env-priority flag is
... implemented on ``project context set``.
... Tracked in #1079.
[Tags] tdd_expected_fail tdd_issue tdd_issue_1079
[Timeout] 5 minutes
# Create a lightweight project for this TDD test so it does not depend on
# the main test's side effects.
${tdd_proj}= Set Variable ${PROJECT_BASE}-tdd-ctx-${RUN_SUFFIX}
${r_proj}= Run CleverAgents Command
... project create ${tdd_proj} --format json
... expected_rc=None
IF ${r_proj.rc} != 0
Fail TDD setup: project create failed (rc=${r_proj.rc}): ${r_proj.stderr}
END
# Attempt to set execution-env-priority at the project level.
# This should succeed once #1079 is fixed.
${r_ctx}= Run CleverAgents Command
... project context set
... --execution-environment container
... --execution-env-priority override
... ${tdd_proj} --format json
... expected_rc=None
Should Be Equal As Integers ${r_ctx.rc} 0
... project context set --execution-env-priority should succeed (bug #1079)
# Verify the setting persisted via project context show
${r_show}= Run CleverAgents Command
... project context show ${tdd_proj} --format json
Should Be Equal As Integers ${r_show.rc} 0
${show_combined}= Set Variable ${r_show.stdout}${r_show.stderr}
Should Contain ${show_combined.lower()} override
... Project context should show execution-env-priority override
WF17 TDD Precedence Level 2 Project Override Resolution
[Documentation] TDD test for AC #5: Verify execution environment resolves via
... project-level override (precedence level 2) when no plan-level
... override is present.
... Expected to fail until project-level execution-env-priority is
... implemented and the resolution logic honours it.
... Tracked in #1079 and #1080.
[Tags] tdd_expected_fail tdd_issue tdd_issue_1080
[Timeout] 10 minutes
# Create lightweight project and action for this TDD test so it does not
# depend on the main test's side effects.
${tdd_proj}= Set Variable ${PROJECT_BASE}-tdd-prec-${RUN_SUFFIX}
${r_proj}= Run CleverAgents Command
... project create ${tdd_proj} --format json
... expected_rc=None
IF ${r_proj.rc} != 0
Fail TDD setup: project create failed (rc=${r_proj.rc}): ${r_proj.stderr}
END
# Dynamically select actor based on available API key.
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
IF ${has_anthropic}
${tdd_actor}= Set Variable anthropic/claude-sonnet-4-20250514
ELSE
${tdd_actor}= Set Variable openai/gpt-4o
END
${tdd_action_name}= Set Variable ${ACTION_BASE}-tdd-prec-${RUN_SUFFIX}
${tdd_action_yaml}= Catenate SEPARATOR=\n
... name: ${tdd_action_name}
... description: "TDD precedence level 2 test action"
... definition_of_done: "Verify project-level override resolution"
... strategy_actor: ${tdd_actor}
... execution_actor: ${tdd_actor}
${tdd_action_path}= Set Variable ${SUITE_HOME}${/}wf17_tdd_prec_action_${RUN_SUFFIX}.yaml
Create File ${tdd_action_path} ${tdd_action_yaml}
${r_action}= Run CleverAgents Command
... action create --config ${tdd_action_path} --format json
... expected_rc=None
IF ${r_action.rc} != 0
Fail TDD setup: action create failed (rc=${r_action.rc}): ${r_action.stderr}
END
# Set project-level execution environment with override priority.
# This depends on #1079 being fixed first.
${r_ctx}= Run CleverAgents Command
... project context set
... --execution-environment container
... --execution-env-priority override
... ${tdd_proj} --format json
... expected_rc=None
Should Be Equal As Integers ${r_ctx.rc} 0
... project context set --execution-env-priority should succeed (depends on #1079)
# Create a plan WITHOUT plan-level --execution-env-priority override.
# The execution environment should still resolve to container via
# project-level override (precedence level 2).
${r_use}= Run CleverAgents Command
... plan use ${tdd_action_name} ${tdd_proj}
... --automation-profile trusted
... --format json
... expected_rc=None
IF ${r_use.rc} != 0
Fail plan use failed (rc=${r_use.rc}) — cannot verify precedence level 2: ${r_use.stderr}
END
${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id
Should Not Be Empty ${plan_id} msg=Expected plan_id in plan use output
# Check plan status for execution environment resolution.
${r_status}= Run CleverAgents Command
... plan status ${plan_id} --format json
... expected_rc=None timeout=120s
Should Be Equal As Integers ${r_status.rc} 0
# The plan should show execution_environment=container resolved via
# project-level override (precedence level 2), not plan-level.
${exec_env}= Safe Parse Json Field ${r_status.stdout} execution_environment
Should Be Equal As Strings ${exec_env.lower()} container
... Execution environment should resolve to container via project-level override (precedence level 2, bug #1080)