forked from cleveragents/cleveragents-core
freemo
48cff5cfe0
Renames `plan lifecycle-list` to `plan list` and `plan lifecycle-apply` to `plan apply` to align with the specification's canonical command names. Removes legacy V2 plan commands that occupied those names. - Renamed CLI command registrations from lifecycle-list/lifecycle-apply to list/apply - Removed legacy V2 apply and list commands (~200 lines) - Updated apply shortcut in main.py to delegate to v3 lifecycle - Added defensive null check for plan existence in apply command - Updated 63+ test, doc, and benchmark files for consistency Closes #881 Co-authored-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me> Co-committed-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me>
474 lines
27 KiB
Plaintext
474 lines
27 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E test for Workflow Example 16: Devcontainer-Driven Development
|
|
... (supervised automation profile).
|
|
...
|
|
... Exercises the devcontainer-specific plan lifecycle:
|
|
... devcontainer auto-detection during resource registration,
|
|
... lazy container build during plan execution, tool invocation
|
|
... routing to the container workspace, and apply writing changes
|
|
... back to the host filesystem via bind mount.
|
|
...
|
|
... **Devcontainer-specific assertions** are strict and contribute
|
|
... to AC validation. If one or more AC indicators are missing,
|
|
... the test records each unmet AC and fails explicitly rather
|
|
... than skipping or passing silently.
|
|
...
|
|
... The test is tagged ``tdd_expected_fail`` because devcontainer
|
|
... features are not yet fully wired. The
|
|
... ``tdd_expected_fail_listener`` inverts the failure to a pass
|
|
... in CI until all AC indicators are present.
|
|
...
|
|
... Zero mocking — real CLI, real LLM API keys.
|
|
Resource common_e2e.resource
|
|
Suite Setup WF16 Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
Force Tags E2E
|
|
|
|
*** Variables ***
|
|
${ACTION_PREFIX} local/wf16-devcontainer-action
|
|
${RESOURCE_PREFIX} local/wf16-devcontainer-repo
|
|
${PROJECT_PREFIX} local/wf16-devcontainer-project
|
|
|
|
*** Keywords ***
|
|
WF16 Suite Setup
|
|
[Documentation] E2E Suite Setup plus database initialisation, unique suffix
|
|
... generation, and dynamic actor selection for WF16 tests.
|
|
E2E Suite Setup
|
|
# Initialise the database so CLI commands work in all tests.
|
|
# Use --force because the workspace may already contain an initialised project.
|
|
${init}= Run CleverAgents Command init --force --yes
|
|
Should Be Equal As Integers ${init.rc} 0
|
|
# Generate a unique suffix for resource/project names to avoid UNIQUE
|
|
# constraint collisions when tests run in parallel or are re-run
|
|
# against the same database (uuid4 provides ~4 billion possibilities).
|
|
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
|
|
Set Suite Variable ${RUN_SUFFIX} ${suffix}
|
|
Set Suite Variable ${ACTION_NAME} ${ACTION_PREFIX}-${suffix}
|
|
Set Suite Variable ${RESOURCE_NAME} ${RESOURCE_PREFIX}-${suffix}
|
|
Set Suite Variable ${PROJECT_NAME} ${PROJECT_PREFIX}-${suffix}
|
|
# Pick an actor that matches available API keys.
|
|
# Prefer OpenAI first to reduce Anthropic credit-quota flakiness.
|
|
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
|
|
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
|
|
IF ${has_openai}
|
|
# Use gpt-4o-mini for cost optimization — WF16 exercises plan lifecycle
|
|
# mechanics (not LLM quality), so a smaller model suffices.
|
|
${actor}= Set Variable openai/gpt-4o-mini
|
|
ELSE IF ${has_anthropic}
|
|
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
|
|
ELSE
|
|
${actor}= Set Variable openai/gpt-4o-mini
|
|
END
|
|
Set Suite Variable ${SELECTED_ACTOR} ${actor}
|
|
|
|
Create Devcontainer Repo
|
|
[Documentation] Create a temp git repo with devcontainer configuration.
|
|
... Returns the path to the created repository.
|
|
${repo}= Create Temp Git Repo wf16-devcontainer-${RUN_SUFFIX}
|
|
Create Directory ${repo}${/}.devcontainer
|
|
Create Directory ${repo}${/}src
|
|
${devcontainer_json}= Catenate SEPARATOR=\n
|
|
... {
|
|
... ${SPACE}${SPACE}"name": "wf16-dev",
|
|
... ${SPACE}${SPACE}"image": "mcr.microsoft.com/devcontainers/python:3.12@sha256:3de8f04c3748897ff3aa32b11ee18306d6c56556f4d548a276d0ff397b28b9da",
|
|
... ${SPACE}${SPACE}"customizations": {
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"vscode": {
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}"extensions": ["ms-python.python"]
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}}
|
|
... ${SPACE}${SPACE}}
|
|
... }
|
|
Create File ${repo}${/}.devcontainer${/}devcontainer.json ${devcontainer_json}
|
|
${app_content}= Catenate SEPARATOR=\n
|
|
... """Main application module."""
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def main():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Entry point."""
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("Hello from devcontainer app")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return 0
|
|
Create File ${repo}${/}src${/}app.py ${app_content}
|
|
${test_content}= Catenate SEPARATOR=\n
|
|
... """Tests for the application."""
|
|
... from src.app import main
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def test_main():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}assert main() == 0
|
|
Create File ${repo}${/}src${/}test_app.py ${test_content}
|
|
Create File ${repo}${/}src${/}__init__.py \n
|
|
Create File ${repo}${/}requirements.txt pytest>=7.0\n
|
|
${r_add}= Run Process git add . cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc})
|
|
${r_commit}= Run Process git commit -m Initial devcontainer project cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc})
|
|
RETURN ${repo}
|
|
|
|
WF16 Test Teardown
|
|
[Documentation] Log diagnostic context on failure for debugging.
|
|
... Captures plan status when a plan ID is available.
|
|
${plan_id}= Get Variable Value ${WF16_PLAN_ID} ${EMPTY}
|
|
IF '${plan_id}' != ''
|
|
${status} ${result}= Run Keyword And Ignore Error
|
|
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
|
|
IF '${status}' == 'PASS'
|
|
Log Teardown plan status: ${result.stdout} WARN
|
|
END
|
|
END
|
|
|
|
*** Test Cases ***
|
|
WF16 Devcontainer Driven Development Supervised Profile
|
|
[Documentation] Supervised-profile workflow exercising devcontainer-specific
|
|
... behaviors: auto-detection during resource registration,
|
|
... lazy container build during execution, tool invocation
|
|
... routing to the container workspace, and apply writing
|
|
... changes back to the host filesystem via bind mount.
|
|
...
|
|
... Devcontainer-specific assertions are enforced via
|
|
... explicit AC checks. Missing AC indicators are collected
|
|
... and reported as explicit test failures.
|
|
...
|
|
... Tagged ``tdd_expected_fail`` because devcontainer features
|
|
... are not yet fully wired; the listener inverts the failure
|
|
... to a CI pass until all AC indicators are present.
|
|
[Tags] tdd_expected_fail tdd_issue tdd_issue_1208
|
|
# Timeout budget: 35 minutes test-level timeout. Per-step timeouts sum
|
|
# higher in theory, but retries and conditional paths are mutually
|
|
# exclusive — realistic worst-case is well under 35 minutes.
|
|
[Timeout] 35 minutes
|
|
[Teardown] WF16 Test Teardown
|
|
Skip If No LLM Keys
|
|
${ac_checks_missing}= Evaluate []
|
|
Set Test Variable ${WF16_PLAN_ID} ${EMPTY}
|
|
|
|
# ---- Create fixture repo with devcontainer ----
|
|
${repo}= Create Devcontainer Repo
|
|
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${branch_result.rc} 0 msg=git rev-parse failed (rc=${branch_result.rc})
|
|
${branch}= Strip String ${branch_result.stdout}
|
|
Log Fixture repo created at ${repo} on branch ${branch}
|
|
|
|
# ---- Register resource (AC-3: devcontainer auto-detection) ----
|
|
${r_res}= Run CleverAgents Command
|
|
... resource add git-checkout ${RESOURCE_NAME}
|
|
... --path ${repo} --branch ${branch}
|
|
Should Not Contain ${r_res.stdout} Traceback
|
|
Should Not Contain ${r_res.stderr} Traceback
|
|
Should Not Contain ${r_res.stdout} INTERNAL
|
|
Should Not Contain ${r_res.stderr} INTERNAL
|
|
Output Should Contain ${r_res} ${RESOURCE_NAME}
|
|
# AC-3: Verify devcontainer auto-detection indicators from resource add.
|
|
# Spec Example 16 shows "devcontainer" plus "detected (not built)".
|
|
${res_combined}= Set Variable ${r_res.stdout}\n${r_res.stderr}
|
|
${res_lower}= Evaluate ($res_combined).lower()
|
|
${has_devcontainer}= Evaluate 'devcontainer' in $res_lower
|
|
${has_detected}= Evaluate 'detected' in $res_lower
|
|
${has_not_built}= Evaluate 'not built' in $res_lower
|
|
${has_ac3}= Evaluate $has_devcontainer and $has_detected and $has_not_built
|
|
IF ${has_ac3}
|
|
Log AC-3 verified: resource output shows devcontainer detected (not built) state
|
|
ELSE
|
|
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-3: resource add output missing explicit detected (not built) devcontainer state']
|
|
Log AC-3 unmet: missing explicit detected (not built) devcontainer state in resource output. AC-4, AC-5, and AC-6 are dependent on AC-3 and are likely to also fail. WARN
|
|
END
|
|
Log Resource registered: ${RESOURCE_NAME}
|
|
|
|
# ---- Create project ----
|
|
${r_proj}= Run CleverAgents Command
|
|
... project create ${PROJECT_NAME}
|
|
... --resource ${RESOURCE_NAME}
|
|
Should Not Contain ${r_proj.stdout} Traceback
|
|
Should Not Contain ${r_proj.stderr} Traceback
|
|
Should Not Contain ${r_proj.stdout} INTERNAL
|
|
Should Not Contain ${r_proj.stderr} INTERNAL
|
|
Output Should Contain ${r_proj} ${PROJECT_NAME}
|
|
Log Project created: ${PROJECT_NAME}
|
|
|
|
# ---- Create action (dynamic actor selection) ----
|
|
${action_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${ACTION_NAME}
|
|
... description: Implement feature in devcontainer-enabled project
|
|
... definition_of_done: Feature implemented with tests passing
|
|
... strategy_actor: ${SELECTED_ACTOR}
|
|
... execution_actor: ${SELECTED_ACTOR}
|
|
... reusable: true
|
|
... read_only: false
|
|
${action_path}= Set Variable ${SUITE_HOME}${/}wf16_action.yaml
|
|
Create File ${action_path} ${action_yaml}
|
|
${r_action}= Run CleverAgents Command
|
|
... action create --config ${action_path}
|
|
Should Not Contain ${r_action.stdout} Traceback
|
|
Should Not Contain ${r_action.stderr} Traceback
|
|
Should Not Contain ${r_action.stdout} INTERNAL
|
|
Should Not Contain ${r_action.stderr} INTERNAL
|
|
Output Should Contain ${r_action} ${ACTION_NAME}
|
|
Log Action created: ${ACTION_NAME} with actor ${SELECTED_ACTOR}
|
|
|
|
# ---- Plan use with supervised automation profile ----
|
|
${r_use}= Run CleverAgents Command
|
|
... plan use ${ACTION_NAME} ${PROJECT_NAME}
|
|
... --automation-profile supervised
|
|
... --format json
|
|
... expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${r_use.rc} 0
|
|
... plan use failed (rc=${r_use.rc}); stderr redacted from assertion message
|
|
Should Not Contain ${r_use.stdout} Traceback
|
|
Should Not Contain ${r_use.stderr} Traceback
|
|
Should Not Contain ${r_use.stdout} INTERNAL
|
|
Should Not Contain ${r_use.stderr} INTERNAL
|
|
# Parse plan ID from structured JSON first to avoid selecting the wrong
|
|
# ULID when output contains multiple identifiers. Only use regex fallback
|
|
# if JSON parsing fails and the fallback result is unambiguous.
|
|
Output Should Contain ${r_use} plan_id
|
|
${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id
|
|
${plan_id}= Set Variable If $plan_id is None ${EMPTY} ${plan_id}
|
|
IF '${plan_id}' != ''
|
|
${plan_id}= Strip String ${plan_id}
|
|
Should Match Regexp ${plan_id} (?i)^[0-9A-HJKMNP-TV-Z]{26}$
|
|
ELSE
|
|
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-HJKMNP-TV-Z]{26} flags=IGNORECASE
|
|
Should Not Be Empty ${plan_ids} msg=Expected plan ID in plan use output
|
|
${unique_plan_ids}= Evaluate list(dict.fromkeys($plan_ids))
|
|
${plan_id_count}= Evaluate len($unique_plan_ids)
|
|
Should Be Equal As Integers ${plan_id_count} 1
|
|
... msg=Ambiguous ULID fallback while extracting plan ID: ${unique_plan_ids}
|
|
${plan_id}= Set Variable ${unique_plan_ids}[0]
|
|
END
|
|
# Verify automation profile was accepted via JSON output
|
|
${resolved_profile}= Safe Parse Json Field ${r_use.stdout} automation_profile
|
|
${resolved_profile}= Set Variable If $resolved_profile is None ${EMPTY} ${resolved_profile}
|
|
IF '${resolved_profile}' != ''
|
|
Should Be Equal As Strings ${resolved_profile} supervised
|
|
ELSE
|
|
# Fallback: check combined output for "supervised" profile name
|
|
Output Should Contain ${r_use} supervised
|
|
END
|
|
Set Test Variable ${WF16_PLAN_ID} ${plan_id}
|
|
Log Plan created: ${plan_id} with supervised profile
|
|
|
|
# ---- Strategize + Execute ----
|
|
# The first plan execute call advances through all pending phases
|
|
# (Strategize → Execute) in a single invocation.
|
|
${r_strat_first}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=180s
|
|
${r_strat}= Set Variable ${r_strat_first}
|
|
IF ${r_strat_first.rc} != 0
|
|
Log First execute returned rc=${r_strat_first.rc}; retrying once WARN
|
|
${r_strat_retry}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=300s
|
|
IF ${r_strat_retry.rc} != 0
|
|
Fail WF16 execute instability after retry (first pass): rc1=${r_strat_first.rc}, rc2=${r_strat_retry.rc} (stderr redacted)
|
|
ELSE
|
|
${r_strat}= Set Variable ${r_strat_retry}
|
|
END
|
|
END
|
|
Should Not Contain ${r_strat.stdout} Traceback
|
|
Should Not Contain ${r_strat.stderr} Traceback
|
|
Should Not Contain ${r_strat.stdout} INTERNAL
|
|
Should Not Contain ${r_strat.stderr} INTERNAL
|
|
${strat_first_combined}= Set Variable ${r_strat_first.stdout}\n${r_strat_first.stderr}
|
|
${strat_first_lower}= Evaluate ($strat_first_combined).lower()
|
|
Log First execute completed: rc=${r_strat.rc}
|
|
|
|
# AC-4: Verify lazy devcontainer build indicators are present in the
|
|
# first execute call (proof of lazy build trigger on first execution).
|
|
${has_build_first}= Evaluate 'building' in $strat_first_lower and 'devcontainer' in $strat_first_lower
|
|
IF ${has_build_first}
|
|
Log AC-4 verified: first execute output shows lazy devcontainer build
|
|
ELSE
|
|
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-4: first execute output missing lazy devcontainer build indicator']
|
|
Log AC-4 unmet: first execute output missing lazy devcontainer build indicator WARN
|
|
END
|
|
|
|
# Determine whether a second execute call is needed; only re-execute when
|
|
# the first execute did not reach a ready-for-apply state.
|
|
${r_status_after_first}= Run CleverAgents Command
|
|
... plan status ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${r_status_after_first.rc} 0
|
|
... plan status after first execute failed (rc=${r_status_after_first.rc}); stderr redacted from assertion message
|
|
${phase_after_first}= Safe Parse Json Field ${r_status_after_first.stdout} phase
|
|
${phase_after_first}= Set Variable If $phase_after_first is None ${EMPTY} ${phase_after_first}
|
|
${state_after_first}= Safe Parse Json Field ${r_status_after_first.stdout} processing_state
|
|
${state_after_first}= Set Variable If $state_after_first is None ${EMPTY} ${state_after_first}
|
|
${phase_after_first_norm}= Evaluate $phase_after_first.strip().lower()
|
|
${state_after_first_norm}= Evaluate $state_after_first.strip().lower()
|
|
${is_apply_phase}= Evaluate $phase_after_first_norm == 'apply'
|
|
${is_execute_complete}= Evaluate $phase_after_first_norm == 'execute' and $state_after_first_norm in ['complete', 'completed']
|
|
${is_already_applied}= Evaluate $state_after_first_norm == 'applied'
|
|
${ready_for_apply}= Evaluate $is_apply_phase or $is_execute_complete or $is_already_applied
|
|
${needs_second_execute}= Evaluate not $ready_for_apply
|
|
|
|
# ---- Defensive re-execute (AC-4: lazy container build, AC-5: container routing) ----
|
|
# The first call typically completes both Strategize and Execute phases.
|
|
# This second call is a defensive re-check — it is a no-op if both
|
|
# phases already completed, but ensures execution finishes if the first
|
|
# call only advanced through Strategize.
|
|
${r_exec}= Set Variable ${NONE}
|
|
IF ${needs_second_execute}
|
|
${r_exec}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=300s
|
|
IF ${r_exec.rc} != 0
|
|
Log Conditional re-check execute returned rc=${r_exec.rc}; retrying once WARN
|
|
${r_exec_retry}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=300s
|
|
IF ${r_exec_retry.rc} != 0
|
|
Fail WF16 execute instability after retry (conditional re-check): rc1=${r_exec.rc}, rc2=${r_exec_retry.rc} (stderr redacted)
|
|
ELSE
|
|
${r_exec}= Set Variable ${r_exec_retry}
|
|
END
|
|
END
|
|
Should Not Contain ${r_exec.stdout} Traceback
|
|
Should Not Contain ${r_exec.stderr} Traceback
|
|
Should Not Contain ${r_exec.stdout} INTERNAL
|
|
Should Not Contain ${r_exec.stderr} INTERNAL
|
|
Log Conditional second execute completed: rc=${r_exec.rc}
|
|
ELSE
|
|
Log Second execute skipped: first execute already reached ready-for-apply state
|
|
END
|
|
# Combine output from both execute calls — devcontainer indicators may
|
|
# appear in either invocation depending on which call runs each phase.
|
|
IF ${needs_second_execute}
|
|
${exec_combined}= Set Variable ${r_strat.stdout}\n${r_strat.stderr}\n${r_exec.stdout}\n${r_exec.stderr}
|
|
ELSE
|
|
${exec_combined}= Set Variable ${r_strat.stdout}\n${r_strat.stderr}
|
|
END
|
|
# AC-5: Require concrete routing evidence with both explicit devcontainer
|
|
# identity and explicit container workspace path evidence.
|
|
${has_dc_resource}= Evaluate bool(__import__('re').search(r'(?im)resource\s*:\s*[^\n]*\(devcontainer-instance\)', $exec_combined))
|
|
${has_dc_resolution}= Evaluate bool(__import__('re').search(r'(?im)resolved via\s*:\s*nearest-ancestor devcontainer', $exec_combined))
|
|
${has_dc_environment}= Evaluate bool(__import__('re').search(r'(?im)environment\s*:\s*devcontainer\b', $exec_combined))
|
|
${has_dc_identity}= Evaluate $has_dc_resource or ($has_dc_resolution and $has_dc_environment)
|
|
${has_workspace_header}= Evaluate bool(__import__('re').search(r'(?im)workspace\s*:\s*/workspaces?/', $exec_combined))
|
|
${has_workspace_tool_path}= Evaluate bool(__import__('re').search(r'(?im)in\s+container\s+/workspaces?/', $exec_combined))
|
|
${has_workspace_context}= Evaluate $has_workspace_header or $has_workspace_tool_path
|
|
${has_routing}= Evaluate $has_dc_identity and $has_workspace_context
|
|
IF ${has_routing}
|
|
Log AC-5 verified: devcontainer-specific routing indicators detected in execute output
|
|
ELSE
|
|
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-5: execute output missing devcontainer-specific routing indicator']
|
|
Log AC-5 unmet: missing devcontainer-specific routing indicator in execute output WARN
|
|
END
|
|
IF ${needs_second_execute}
|
|
Log Execute completed: rc=${r_exec.rc}
|
|
ELSE
|
|
Log Execute completed in first pass (no second execute needed)
|
|
END
|
|
|
|
# ---- Diff (verify non-empty changeset) ----
|
|
${r_diff}= Run CleverAgents Command
|
|
... plan diff ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${r_diff.rc} 0
|
|
... plan diff failed (rc=${r_diff.rc}); stderr redacted from assertion message
|
|
Should Not Contain ${r_diff.stdout} Traceback
|
|
Should Not Contain ${r_diff.stderr} Traceback
|
|
Should Not Contain ${r_diff.stdout} INTERNAL
|
|
Should Not Contain ${r_diff.stderr} INTERNAL
|
|
Should Not Be Empty ${r_diff.stdout} msg=Plan diff produced no output — expected a changeset
|
|
Log Diff completed with non-empty output
|
|
|
|
# ---- Apply (AC-6: host filesystem write verification) ----
|
|
# Capture HEAD SHA before apply to detect new commits written by apply
|
|
${pre_apply_head}= Run Process git rev-parse HEAD cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${pre_apply_head.rc} 0 msg=git rev-parse HEAD failed before apply (rc=${pre_apply_head.rc})
|
|
${head_before}= Strip String ${pre_apply_head.stdout}
|
|
${pre_apply_status}= Run Process git status --porcelain cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${pre_apply_status.rc} 0 msg=git status --porcelain failed before apply (rc=${pre_apply_status.rc})
|
|
${worktree_before}= Strip String ${pre_apply_status.stdout}
|
|
Should Be Empty ${worktree_before} msg=Fixture repository must be clean before apply
|
|
Log HEAD before apply: ${head_before}
|
|
# Use ``plan apply`` so the plan drives through all three Apply
|
|
# sub-transitions synchronously:
|
|
# Execute/complete → Apply/queued → Apply/processing → Apply/applied.
|
|
# ``plan apply`` with a plan ID calls ``_lifecycle_apply_with_id``
|
|
# which completes the full transition to the terminal Apply/applied
|
|
# state.
|
|
${r_apply}= Run CleverAgents Command
|
|
... plan apply --yes --format json ${plan_id}
|
|
... expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${r_apply.rc} 0
|
|
... plan apply failed (rc=${r_apply.rc}); stderr redacted from assertion message
|
|
Should Not Contain ${r_apply.stdout} Traceback
|
|
Should Not Contain ${r_apply.stderr} Traceback
|
|
Should Not Contain ${r_apply.stdout} INTERNAL
|
|
Should Not Contain ${r_apply.stderr} INTERNAL
|
|
# Verify apply output contains the plan ID (meaningful verification
|
|
# that apply processed the correct plan).
|
|
Output Should Contain ${r_apply} ${plan_id}
|
|
# Verify the apply phase in JSON output (parse success + phase presence
|
|
# are mandatory before phase assertion).
|
|
${apply_phase}= Safe Parse Json Field ${r_apply.stdout} phase
|
|
${apply_phase}= Set Variable If $apply_phase is None ${EMPTY} ${apply_phase}
|
|
Should Not Be Empty ${apply_phase} msg=Could not parse non-empty phase from plan apply JSON
|
|
Should Contain ${apply_phase.lower()} apply Plan phase should indicate apply after plan apply
|
|
${post_apply_head}= Run Process git rev-parse HEAD cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${post_apply_head.rc} 0 msg=git rev-parse HEAD failed after apply (rc=${post_apply_head.rc})
|
|
${head_after}= Strip String ${post_apply_head.stdout}
|
|
${post_apply_status}= Run Process git status --porcelain cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${post_apply_status.rc} 0 msg=git status --porcelain failed after apply (rc=${post_apply_status.rc})
|
|
${worktree_after}= Strip String ${post_apply_status.stdout}
|
|
Log HEAD after apply: ${head_after}
|
|
${apply_combined}= Set Variable ${r_apply.stdout}\n${r_apply.stderr}
|
|
${apply_lower}= Evaluate ($apply_combined).lower()
|
|
${has_bind_mount_mechanism}= Evaluate 'bind mount' in $apply_lower or 'bind-mount' in $apply_lower or 'bind_mount' in $apply_lower
|
|
${has_head_advance}= Evaluate $head_before != $head_after
|
|
${has_worktree_changes}= Evaluate bool($worktree_after.strip())
|
|
${has_host_mutation}= Evaluate $has_head_advance or $has_worktree_changes
|
|
${has_ac6}= Evaluate $has_bind_mount_mechanism and $has_host_mutation
|
|
IF ${has_ac6}
|
|
IF ${has_worktree_changes}
|
|
Log AC-6 host mutation evidence (git status --porcelain): ${worktree_after}
|
|
END
|
|
Log AC-6 verified: bind-mount mechanism signal and concrete host mutation both detected
|
|
ELSE
|
|
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-6: missing bind-mount mechanism signal and/or host mutation evidence after apply']
|
|
Log AC-6 unmet: missing bind-mount mechanism signal and/or host mutation evidence after apply WARN
|
|
END
|
|
Log Apply completed: plan transitioned to apply phase
|
|
|
|
# ---- Status (terminal state verification) ----
|
|
${r_status}= Run CleverAgents Command
|
|
... plan status ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${r_status.rc} 0
|
|
... plan status failed (rc=${r_status.rc}); stderr redacted from assertion message
|
|
Should Not Contain ${r_status.stdout} Traceback
|
|
Should Not Contain ${r_status.stderr} Traceback
|
|
Should Not Contain ${r_status.stdout} INTERNAL
|
|
Should Not Contain ${r_status.stderr} INTERNAL
|
|
Should Not Be Empty ${r_status.stdout}
|
|
Output Should Contain ${r_status} ${plan_id}
|
|
# Verify success-only terminal state after apply.
|
|
${status_phase}= Safe Parse Json Field ${r_status.stdout} phase
|
|
${status_phase}= Set Variable If $status_phase is None ${EMPTY} ${status_phase}
|
|
Should Not Be Empty ${status_phase} msg=Could not parse non-empty phase from plan status JSON
|
|
Should Contain ${status_phase.lower()} apply
|
|
... Plan phase should remain apply after successful plan apply
|
|
${status_state}= Safe Parse Json Field ${r_status.stdout} processing_state
|
|
${status_state}= Set Variable If $status_state is None ${EMPTY} ${status_state}
|
|
IF '${status_state}' != ''
|
|
Log Final processing_state: ${status_state}
|
|
${state_lower}= Evaluate ($status_state).lower()
|
|
Should Be Equal As Strings ${state_lower} applied
|
|
... Expected success-only terminal state 'applied' but found: ${status_state}
|
|
ELSE
|
|
Fail Could not parse processing_state from plan status JSON
|
|
END
|
|
${missing_ac_count}= Evaluate len($ac_checks_missing)
|
|
IF ${missing_ac_count} > 0
|
|
${missing_ac_summary}= Evaluate '; '.join($ac_checks_missing)
|
|
Fail WF16 AC verification incomplete: ${missing_ac_summary}
|
|
END
|
|
Log Plan status verified: ${plan_id} in state ${status_state}
|