forked from cleveragents/cleveragents-core
hurui200320
b6c3169634
Implements E2E test for devcontainer-driven development workflow.
Tests project setup with .devcontainer config, resource registration,
and full plan lifecycle with supervised automation profile.
Changes from Cycle 5 review:
1. Replaced custom WF16_ALLOW_XFAIL_SKIP escape hatch with the project
standard tdd_expected_fail tag system. Test is now tagged
[Tags] tdd_expected_fail tdd_bug tdd_bug_762 and the listener
inverts failures in CI until devcontainer features are fully wired.
2. Added diagnostic WF16 Test Teardown keyword that captures plan
status JSON on failure, mirroring the WF05 teardown pattern.
3. Added stderr checks to all Traceback/INTERNAL guard assertions
so tracebacks appearing only in stderr are also detected.
4. Rebased branch onto latest master (e4c01492).
5. Documented timeout budget: added comment explaining theoretical
worst-case (~46min) vs realistic execution under the 35min limit.
6. Added fail-fast warning in AC-3 check noting that AC-4, AC-5, and
AC-6 are dependent on AC-3 and are likely to also fail.
7. Reordered plan apply arguments to --yes --format json <plan_id>
for spec consistency.
8. Decomposed long ready_for_apply Evaluate expression into three
intermediate booleans (is_apply_phase, is_execute_complete,
is_already_applied) for readability.
9. Added shared_session.close() calls in resource_dag.robot before
each final print() to ensure explicit session cleanup.
10. Cleaned up stale xfail/skip classification references in log
messages.
Changes from earlier cycles (3-4) retained in this commit:
- Terminal state validation, AC output combination, JSON format flags,
suite-level init, Force Tags, automation profile fallback, ULID regex
IGNORECASE, AC-5 Evaluate decomposition, defensive re-check pattern.
ISSUES CLOSED: #762
474 lines
27 KiB
Plaintext
474 lines
27 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E test for Workflow Example 16: Devcontainer-Driven Development
|
|
... (supervised automation profile).
|
|
...
|
|
... Exercises the devcontainer-specific plan lifecycle:
|
|
... devcontainer auto-detection during resource registration,
|
|
... lazy container build during plan execution, tool invocation
|
|
... routing to the container workspace, and apply writing changes
|
|
... back to the host filesystem via bind mount.
|
|
...
|
|
... **Devcontainer-specific assertions** are strict and contribute
|
|
... to AC validation. If one or more AC indicators are missing,
|
|
... the test records each unmet AC and fails explicitly rather
|
|
... than skipping or passing silently.
|
|
...
|
|
... The test is tagged ``tdd_expected_fail`` because devcontainer
|
|
... features are not yet fully wired. The
|
|
... ``tdd_expected_fail_listener`` inverts the failure to a pass
|
|
... in CI until all AC indicators are present.
|
|
...
|
|
... Zero mocking — real CLI, real LLM API keys.
|
|
Resource common_e2e.resource
|
|
Suite Setup WF16 Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
Force Tags E2E
|
|
|
|
*** Variables ***
|
|
${ACTION_PREFIX} local/wf16-devcontainer-action
|
|
${RESOURCE_PREFIX} local/wf16-devcontainer-repo
|
|
${PROJECT_PREFIX} local/wf16-devcontainer-project
|
|
|
|
*** Keywords ***
|
|
WF16 Suite Setup
|
|
[Documentation] E2E Suite Setup plus database initialisation, unique suffix
|
|
... generation, and dynamic actor selection for WF16 tests.
|
|
E2E Suite Setup
|
|
# Initialise the database so CLI commands work in all tests.
|
|
# Use --force because the workspace may already contain an initialised project.
|
|
${init}= Run CleverAgents Command init --force --yes
|
|
Should Be Equal As Integers ${init.rc} 0
|
|
# Generate a unique suffix for resource/project names to avoid UNIQUE
|
|
# constraint collisions when tests run in parallel or are re-run
|
|
# against the same database (uuid4 provides ~4 billion possibilities).
|
|
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
|
|
Set Suite Variable ${RUN_SUFFIX} ${suffix}
|
|
Set Suite Variable ${ACTION_NAME} ${ACTION_PREFIX}-${suffix}
|
|
Set Suite Variable ${RESOURCE_NAME} ${RESOURCE_PREFIX}-${suffix}
|
|
Set Suite Variable ${PROJECT_NAME} ${PROJECT_PREFIX}-${suffix}
|
|
# Pick an actor that matches available API keys.
|
|
# Prefer OpenAI first to reduce Anthropic credit-quota flakiness.
|
|
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
|
|
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
|
|
IF ${has_openai}
|
|
# Use gpt-4o-mini for cost optimization — WF16 exercises plan lifecycle
|
|
# mechanics (not LLM quality), so a smaller model suffices.
|
|
${actor}= Set Variable openai/gpt-4o-mini
|
|
ELSE IF ${has_anthropic}
|
|
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
|
|
ELSE
|
|
${actor}= Set Variable openai/gpt-4o-mini
|
|
END
|
|
Set Suite Variable ${SELECTED_ACTOR} ${actor}
|
|
|
|
Create Devcontainer Repo
|
|
[Documentation] Create a temp git repo with devcontainer configuration.
|
|
... Returns the path to the created repository.
|
|
${repo}= Create Temp Git Repo wf16-devcontainer-${RUN_SUFFIX}
|
|
Create Directory ${repo}${/}.devcontainer
|
|
Create Directory ${repo}${/}src
|
|
${devcontainer_json}= Catenate SEPARATOR=\n
|
|
... {
|
|
... ${SPACE}${SPACE}"name": "wf16-dev",
|
|
... ${SPACE}${SPACE}"image": "mcr.microsoft.com/devcontainers/python:3.12@sha256:3de8f04c3748897ff3aa32b11ee18306d6c56556f4d548a276d0ff397b28b9da",
|
|
... ${SPACE}${SPACE}"customizations": {
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"vscode": {
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}"extensions": ["ms-python.python"]
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}}
|
|
... ${SPACE}${SPACE}}
|
|
... }
|
|
Create File ${repo}${/}.devcontainer${/}devcontainer.json ${devcontainer_json}
|
|
${app_content}= Catenate SEPARATOR=\n
|
|
... """Main application module."""
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def main():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Entry point."""
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}print("Hello from devcontainer app")
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return 0
|
|
Create File ${repo}${/}src${/}app.py ${app_content}
|
|
${test_content}= Catenate SEPARATOR=\n
|
|
... """Tests for the application."""
|
|
... from src.app import main
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def test_main():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}assert main() == 0
|
|
Create File ${repo}${/}src${/}test_app.py ${test_content}
|
|
Create File ${repo}${/}src${/}__init__.py \n
|
|
Create File ${repo}${/}requirements.txt pytest>=7.0\n
|
|
${r_add}= Run Process git add . cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc})
|
|
${r_commit}= Run Process git commit -m Initial devcontainer project cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc})
|
|
RETURN ${repo}
|
|
|
|
WF16 Test Teardown
|
|
[Documentation] Log diagnostic context on failure for debugging.
|
|
... Captures plan status when a plan ID is available.
|
|
${plan_id}= Get Variable Value ${WF16_PLAN_ID} ${EMPTY}
|
|
IF '${plan_id}' != ''
|
|
${status} ${result}= Run Keyword And Ignore Error
|
|
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
|
|
IF '${status}' == 'PASS'
|
|
Log Teardown plan status: ${result.stdout} WARN
|
|
END
|
|
END
|
|
|
|
*** Test Cases ***
|
|
WF16 Devcontainer Driven Development Supervised Profile
|
|
[Documentation] Supervised-profile workflow exercising devcontainer-specific
|
|
... behaviors: auto-detection during resource registration,
|
|
... lazy container build during execution, tool invocation
|
|
... routing to the container workspace, and apply writing
|
|
... changes back to the host filesystem via bind mount.
|
|
...
|
|
... Devcontainer-specific assertions are enforced via
|
|
... explicit AC checks. Missing AC indicators are collected
|
|
... and reported as explicit test failures.
|
|
...
|
|
... Tagged ``tdd_expected_fail`` because devcontainer features
|
|
... are not yet fully wired; the listener inverts the failure
|
|
... to a CI pass until all AC indicators are present.
|
|
[Tags] tdd_expected_fail tdd_issue tdd_issue_1208
|
|
# Timeout budget: 35 minutes test-level timeout. Per-step timeouts sum
|
|
# higher in theory, but retries and conditional paths are mutually
|
|
# exclusive — realistic worst-case is well under 35 minutes.
|
|
[Timeout] 35 minutes
|
|
[Teardown] WF16 Test Teardown
|
|
Skip If No LLM Keys
|
|
${ac_checks_missing}= Evaluate []
|
|
Set Test Variable ${WF16_PLAN_ID} ${EMPTY}
|
|
|
|
# ---- Create fixture repo with devcontainer ----
|
|
${repo}= Create Devcontainer Repo
|
|
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${branch_result.rc} 0 msg=git rev-parse failed (rc=${branch_result.rc})
|
|
${branch}= Strip String ${branch_result.stdout}
|
|
Log Fixture repo created at ${repo} on branch ${branch}
|
|
|
|
# ---- Register resource (AC-3: devcontainer auto-detection) ----
|
|
${r_res}= Run CleverAgents Command
|
|
... resource add git-checkout ${RESOURCE_NAME}
|
|
... --path ${repo} --branch ${branch}
|
|
Should Not Contain ${r_res.stdout} Traceback
|
|
Should Not Contain ${r_res.stderr} Traceback
|
|
Should Not Contain ${r_res.stdout} INTERNAL
|
|
Should Not Contain ${r_res.stderr} INTERNAL
|
|
Output Should Contain ${r_res} ${RESOURCE_NAME}
|
|
# AC-3: Verify devcontainer auto-detection indicators from resource add.
|
|
# Spec Example 16 shows "devcontainer" plus "detected (not built)".
|
|
${res_combined}= Set Variable ${r_res.stdout}\n${r_res.stderr}
|
|
${res_lower}= Evaluate ($res_combined).lower()
|
|
${has_devcontainer}= Evaluate 'devcontainer' in $res_lower
|
|
${has_detected}= Evaluate 'detected' in $res_lower
|
|
${has_not_built}= Evaluate 'not built' in $res_lower
|
|
${has_ac3}= Evaluate $has_devcontainer and $has_detected and $has_not_built
|
|
IF ${has_ac3}
|
|
Log AC-3 verified: resource output shows devcontainer detected (not built) state
|
|
ELSE
|
|
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-3: resource add output missing explicit detected (not built) devcontainer state']
|
|
Log AC-3 unmet: missing explicit detected (not built) devcontainer state in resource output. AC-4, AC-5, and AC-6 are dependent on AC-3 and are likely to also fail. WARN
|
|
END
|
|
Log Resource registered: ${RESOURCE_NAME}
|
|
|
|
# ---- Create project ----
|
|
${r_proj}= Run CleverAgents Command
|
|
... project create ${PROJECT_NAME}
|
|
... --resource ${RESOURCE_NAME}
|
|
Should Not Contain ${r_proj.stdout} Traceback
|
|
Should Not Contain ${r_proj.stderr} Traceback
|
|
Should Not Contain ${r_proj.stdout} INTERNAL
|
|
Should Not Contain ${r_proj.stderr} INTERNAL
|
|
Output Should Contain ${r_proj} ${PROJECT_NAME}
|
|
Log Project created: ${PROJECT_NAME}
|
|
|
|
# ---- Create action (dynamic actor selection) ----
|
|
${action_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${ACTION_NAME}
|
|
... description: Implement feature in devcontainer-enabled project
|
|
... definition_of_done: Feature implemented with tests passing
|
|
... strategy_actor: ${SELECTED_ACTOR}
|
|
... execution_actor: ${SELECTED_ACTOR}
|
|
... reusable: true
|
|
... read_only: false
|
|
${action_path}= Set Variable ${SUITE_HOME}${/}wf16_action.yaml
|
|
Create File ${action_path} ${action_yaml}
|
|
${r_action}= Run CleverAgents Command
|
|
... action create --config ${action_path}
|
|
Should Not Contain ${r_action.stdout} Traceback
|
|
Should Not Contain ${r_action.stderr} Traceback
|
|
Should Not Contain ${r_action.stdout} INTERNAL
|
|
Should Not Contain ${r_action.stderr} INTERNAL
|
|
Output Should Contain ${r_action} ${ACTION_NAME}
|
|
Log Action created: ${ACTION_NAME} with actor ${SELECTED_ACTOR}
|
|
|
|
# ---- Plan use with supervised automation profile ----
|
|
${r_use}= Run CleverAgents Command
|
|
... plan use ${ACTION_NAME} ${PROJECT_NAME}
|
|
... --automation-profile supervised
|
|
... --format json
|
|
... expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${r_use.rc} 0
|
|
... plan use failed (rc=${r_use.rc}); stderr redacted from assertion message
|
|
Should Not Contain ${r_use.stdout} Traceback
|
|
Should Not Contain ${r_use.stderr} Traceback
|
|
Should Not Contain ${r_use.stdout} INTERNAL
|
|
Should Not Contain ${r_use.stderr} INTERNAL
|
|
# Parse plan ID from structured JSON first to avoid selecting the wrong
|
|
# ULID when output contains multiple identifiers. Only use regex fallback
|
|
# if JSON parsing fails and the fallback result is unambiguous.
|
|
Output Should Contain ${r_use} plan_id
|
|
${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id
|
|
${plan_id}= Set Variable If $plan_id is None ${EMPTY} ${plan_id}
|
|
IF '${plan_id}' != ''
|
|
${plan_id}= Strip String ${plan_id}
|
|
Should Match Regexp ${plan_id} (?i)^[0-9A-HJKMNP-TV-Z]{26}$
|
|
ELSE
|
|
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-HJKMNP-TV-Z]{26} flags=IGNORECASE
|
|
Should Not Be Empty ${plan_ids} msg=Expected plan ID in plan use output
|
|
${unique_plan_ids}= Evaluate list(dict.fromkeys($plan_ids))
|
|
${plan_id_count}= Evaluate len($unique_plan_ids)
|
|
Should Be Equal As Integers ${plan_id_count} 1
|
|
... msg=Ambiguous ULID fallback while extracting plan ID: ${unique_plan_ids}
|
|
${plan_id}= Set Variable ${unique_plan_ids}[0]
|
|
END
|
|
# Verify automation profile was accepted via JSON output
|
|
${resolved_profile}= Safe Parse Json Field ${r_use.stdout} automation_profile
|
|
${resolved_profile}= Set Variable If $resolved_profile is None ${EMPTY} ${resolved_profile}
|
|
IF '${resolved_profile}' != ''
|
|
Should Be Equal As Strings ${resolved_profile} supervised
|
|
ELSE
|
|
# Fallback: check combined output for "supervised" profile name
|
|
Output Should Contain ${r_use} supervised
|
|
END
|
|
Set Test Variable ${WF16_PLAN_ID} ${plan_id}
|
|
Log Plan created: ${plan_id} with supervised profile
|
|
|
|
# ---- Strategize + Execute ----
|
|
# The first plan execute call advances through all pending phases
|
|
# (Strategize → Execute) in a single invocation.
|
|
${r_strat_first}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=180s
|
|
${r_strat}= Set Variable ${r_strat_first}
|
|
IF ${r_strat_first.rc} != 0
|
|
Log First execute returned rc=${r_strat_first.rc}; retrying once WARN
|
|
${r_strat_retry}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=300s
|
|
IF ${r_strat_retry.rc} != 0
|
|
Fail WF16 execute instability after retry (first pass): rc1=${r_strat_first.rc}, rc2=${r_strat_retry.rc} (stderr redacted)
|
|
ELSE
|
|
${r_strat}= Set Variable ${r_strat_retry}
|
|
END
|
|
END
|
|
Should Not Contain ${r_strat.stdout} Traceback
|
|
Should Not Contain ${r_strat.stderr} Traceback
|
|
Should Not Contain ${r_strat.stdout} INTERNAL
|
|
Should Not Contain ${r_strat.stderr} INTERNAL
|
|
${strat_first_combined}= Set Variable ${r_strat_first.stdout}\n${r_strat_first.stderr}
|
|
${strat_first_lower}= Evaluate ($strat_first_combined).lower()
|
|
Log First execute completed: rc=${r_strat.rc}
|
|
|
|
# AC-4: Verify lazy devcontainer build indicators are present in the
|
|
# first execute call (proof of lazy build trigger on first execution).
|
|
${has_build_first}= Evaluate 'building' in $strat_first_lower and 'devcontainer' in $strat_first_lower
|
|
IF ${has_build_first}
|
|
Log AC-4 verified: first execute output shows lazy devcontainer build
|
|
ELSE
|
|
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-4: first execute output missing lazy devcontainer build indicator']
|
|
Log AC-4 unmet: first execute output missing lazy devcontainer build indicator WARN
|
|
END
|
|
|
|
# Determine whether a second execute call is needed; only re-execute when
|
|
# the first execute did not reach a ready-for-apply state.
|
|
${r_status_after_first}= Run CleverAgents Command
|
|
... plan status ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${r_status_after_first.rc} 0
|
|
... plan status after first execute failed (rc=${r_status_after_first.rc}); stderr redacted from assertion message
|
|
${phase_after_first}= Safe Parse Json Field ${r_status_after_first.stdout} phase
|
|
${phase_after_first}= Set Variable If $phase_after_first is None ${EMPTY} ${phase_after_first}
|
|
${state_after_first}= Safe Parse Json Field ${r_status_after_first.stdout} processing_state
|
|
${state_after_first}= Set Variable If $state_after_first is None ${EMPTY} ${state_after_first}
|
|
${phase_after_first_norm}= Evaluate $phase_after_first.strip().lower()
|
|
${state_after_first_norm}= Evaluate $state_after_first.strip().lower()
|
|
${is_apply_phase}= Evaluate $phase_after_first_norm == 'apply'
|
|
${is_execute_complete}= Evaluate $phase_after_first_norm == 'execute' and $state_after_first_norm in ['complete', 'completed']
|
|
${is_already_applied}= Evaluate $state_after_first_norm == 'applied'
|
|
${ready_for_apply}= Evaluate $is_apply_phase or $is_execute_complete or $is_already_applied
|
|
${needs_second_execute}= Evaluate not $ready_for_apply
|
|
|
|
# ---- Defensive re-execute (AC-4: lazy container build, AC-5: container routing) ----
|
|
# The first call typically completes both Strategize and Execute phases.
|
|
# This second call is a defensive re-check — it is a no-op if both
|
|
# phases already completed, but ensures execution finishes if the first
|
|
# call only advanced through Strategize.
|
|
${r_exec}= Set Variable ${NONE}
|
|
IF ${needs_second_execute}
|
|
${r_exec}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=300s
|
|
IF ${r_exec.rc} != 0
|
|
Log Conditional re-check execute returned rc=${r_exec.rc}; retrying once WARN
|
|
${r_exec_retry}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=300s
|
|
IF ${r_exec_retry.rc} != 0
|
|
Fail WF16 execute instability after retry (conditional re-check): rc1=${r_exec.rc}, rc2=${r_exec_retry.rc} (stderr redacted)
|
|
ELSE
|
|
${r_exec}= Set Variable ${r_exec_retry}
|
|
END
|
|
END
|
|
Should Not Contain ${r_exec.stdout} Traceback
|
|
Should Not Contain ${r_exec.stderr} Traceback
|
|
Should Not Contain ${r_exec.stdout} INTERNAL
|
|
Should Not Contain ${r_exec.stderr} INTERNAL
|
|
Log Conditional second execute completed: rc=${r_exec.rc}
|
|
ELSE
|
|
Log Second execute skipped: first execute already reached ready-for-apply state
|
|
END
|
|
# Combine output from both execute calls — devcontainer indicators may
|
|
# appear in either invocation depending on which call runs each phase.
|
|
IF ${needs_second_execute}
|
|
${exec_combined}= Set Variable ${r_strat.stdout}\n${r_strat.stderr}\n${r_exec.stdout}\n${r_exec.stderr}
|
|
ELSE
|
|
${exec_combined}= Set Variable ${r_strat.stdout}\n${r_strat.stderr}
|
|
END
|
|
# AC-5: Require concrete routing evidence with both explicit devcontainer
|
|
# identity and explicit container workspace path evidence.
|
|
${has_dc_resource}= Evaluate bool(__import__('re').search(r'(?im)resource\s*:\s*[^\n]*\(devcontainer-instance\)', $exec_combined))
|
|
${has_dc_resolution}= Evaluate bool(__import__('re').search(r'(?im)resolved via\s*:\s*nearest-ancestor devcontainer', $exec_combined))
|
|
${has_dc_environment}= Evaluate bool(__import__('re').search(r'(?im)environment\s*:\s*devcontainer\b', $exec_combined))
|
|
${has_dc_identity}= Evaluate $has_dc_resource or ($has_dc_resolution and $has_dc_environment)
|
|
${has_workspace_header}= Evaluate bool(__import__('re').search(r'(?im)workspace\s*:\s*/workspaces?/', $exec_combined))
|
|
${has_workspace_tool_path}= Evaluate bool(__import__('re').search(r'(?im)in\s+container\s+/workspaces?/', $exec_combined))
|
|
${has_workspace_context}= Evaluate $has_workspace_header or $has_workspace_tool_path
|
|
${has_routing}= Evaluate $has_dc_identity and $has_workspace_context
|
|
IF ${has_routing}
|
|
Log AC-5 verified: devcontainer-specific routing indicators detected in execute output
|
|
ELSE
|
|
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-5: execute output missing devcontainer-specific routing indicator']
|
|
Log AC-5 unmet: missing devcontainer-specific routing indicator in execute output WARN
|
|
END
|
|
IF ${needs_second_execute}
|
|
Log Execute completed: rc=${r_exec.rc}
|
|
ELSE
|
|
Log Execute completed in first pass (no second execute needed)
|
|
END
|
|
|
|
# ---- Diff (verify non-empty changeset) ----
|
|
${r_diff}= Run CleverAgents Command
|
|
... plan diff ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${r_diff.rc} 0
|
|
... plan diff failed (rc=${r_diff.rc}); stderr redacted from assertion message
|
|
Should Not Contain ${r_diff.stdout} Traceback
|
|
Should Not Contain ${r_diff.stderr} Traceback
|
|
Should Not Contain ${r_diff.stdout} INTERNAL
|
|
Should Not Contain ${r_diff.stderr} INTERNAL
|
|
Should Not Be Empty ${r_diff.stdout} msg=Plan diff produced no output — expected a changeset
|
|
Log Diff completed with non-empty output
|
|
|
|
# ---- Apply (AC-6: host filesystem write verification) ----
|
|
# Capture HEAD SHA before apply to detect new commits written by apply
|
|
${pre_apply_head}= Run Process git rev-parse HEAD cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${pre_apply_head.rc} 0 msg=git rev-parse HEAD failed before apply (rc=${pre_apply_head.rc})
|
|
${head_before}= Strip String ${pre_apply_head.stdout}
|
|
${pre_apply_status}= Run Process git status --porcelain cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${pre_apply_status.rc} 0 msg=git status --porcelain failed before apply (rc=${pre_apply_status.rc})
|
|
${worktree_before}= Strip String ${pre_apply_status.stdout}
|
|
Should Be Empty ${worktree_before} msg=Fixture repository must be clean before apply
|
|
Log HEAD before apply: ${head_before}
|
|
# Use ``plan apply`` (not ``lifecycle-apply``) so the plan drives
|
|
# through all three Apply sub-transitions synchronously:
|
|
# Execute/complete → Apply/queued → Apply/processing → Apply/applied.
|
|
# ``lifecycle-apply`` only transitions to Apply/queued; ``plan apply``
|
|
# with a plan ID calls ``_lifecycle_apply_with_id`` which completes
|
|
# the full transition to the terminal Apply/applied state.
|
|
${r_apply}= Run CleverAgents Command
|
|
... plan apply --yes --format json ${plan_id}
|
|
... expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${r_apply.rc} 0
|
|
... plan apply failed (rc=${r_apply.rc}); stderr redacted from assertion message
|
|
Should Not Contain ${r_apply.stdout} Traceback
|
|
Should Not Contain ${r_apply.stderr} Traceback
|
|
Should Not Contain ${r_apply.stdout} INTERNAL
|
|
Should Not Contain ${r_apply.stderr} INTERNAL
|
|
# Verify apply output contains the plan ID (meaningful verification
|
|
# that apply processed the correct plan).
|
|
Output Should Contain ${r_apply} ${plan_id}
|
|
# Verify the apply phase in JSON output (parse success + phase presence
|
|
# are mandatory before phase assertion).
|
|
${apply_phase}= Safe Parse Json Field ${r_apply.stdout} phase
|
|
${apply_phase}= Set Variable If $apply_phase is None ${EMPTY} ${apply_phase}
|
|
Should Not Be Empty ${apply_phase} msg=Could not parse non-empty phase from plan apply JSON
|
|
Should Contain ${apply_phase.lower()} apply Plan phase should indicate apply after plan apply
|
|
${post_apply_head}= Run Process git rev-parse HEAD cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${post_apply_head.rc} 0 msg=git rev-parse HEAD failed after apply (rc=${post_apply_head.rc})
|
|
${head_after}= Strip String ${post_apply_head.stdout}
|
|
${post_apply_status}= Run Process git status --porcelain cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${post_apply_status.rc} 0 msg=git status --porcelain failed after apply (rc=${post_apply_status.rc})
|
|
${worktree_after}= Strip String ${post_apply_status.stdout}
|
|
Log HEAD after apply: ${head_after}
|
|
${apply_combined}= Set Variable ${r_apply.stdout}\n${r_apply.stderr}
|
|
${apply_lower}= Evaluate ($apply_combined).lower()
|
|
${has_bind_mount_mechanism}= Evaluate 'bind mount' in $apply_lower or 'bind-mount' in $apply_lower or 'bind_mount' in $apply_lower
|
|
${has_head_advance}= Evaluate $head_before != $head_after
|
|
${has_worktree_changes}= Evaluate bool($worktree_after.strip())
|
|
${has_host_mutation}= Evaluate $has_head_advance or $has_worktree_changes
|
|
${has_ac6}= Evaluate $has_bind_mount_mechanism and $has_host_mutation
|
|
IF ${has_ac6}
|
|
IF ${has_worktree_changes}
|
|
Log AC-6 host mutation evidence (git status --porcelain): ${worktree_after}
|
|
END
|
|
Log AC-6 verified: bind-mount mechanism signal and concrete host mutation both detected
|
|
ELSE
|
|
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-6: missing bind-mount mechanism signal and/or host mutation evidence after apply']
|
|
Log AC-6 unmet: missing bind-mount mechanism signal and/or host mutation evidence after apply WARN
|
|
END
|
|
Log Apply completed: plan transitioned to apply phase
|
|
|
|
# ---- Status (terminal state verification) ----
|
|
${r_status}= Run CleverAgents Command
|
|
... plan status ${plan_id}
|
|
... --format json
|
|
... expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${r_status.rc} 0
|
|
... plan status failed (rc=${r_status.rc}); stderr redacted from assertion message
|
|
Should Not Contain ${r_status.stdout} Traceback
|
|
Should Not Contain ${r_status.stderr} Traceback
|
|
Should Not Contain ${r_status.stdout} INTERNAL
|
|
Should Not Contain ${r_status.stderr} INTERNAL
|
|
Should Not Be Empty ${r_status.stdout}
|
|
Output Should Contain ${r_status} ${plan_id}
|
|
# Verify success-only terminal state after apply.
|
|
${status_phase}= Safe Parse Json Field ${r_status.stdout} phase
|
|
${status_phase}= Set Variable If $status_phase is None ${EMPTY} ${status_phase}
|
|
Should Not Be Empty ${status_phase} msg=Could not parse non-empty phase from plan status JSON
|
|
Should Contain ${status_phase.lower()} apply
|
|
... Plan phase should remain apply after successful plan apply
|
|
${status_state}= Safe Parse Json Field ${r_status.stdout} processing_state
|
|
${status_state}= Set Variable If $status_state is None ${EMPTY} ${status_state}
|
|
IF '${status_state}' != ''
|
|
Log Final processing_state: ${status_state}
|
|
${state_lower}= Evaluate ($status_state).lower()
|
|
Should Be Equal As Strings ${state_lower} applied
|
|
... Expected success-only terminal state 'applied' but found: ${status_state}
|
|
ELSE
|
|
Fail Could not parse processing_state from plan status JSON
|
|
END
|
|
${missing_ac_count}= Evaluate len($ac_checks_missing)
|
|
IF ${missing_ac_count} > 0
|
|
${missing_ac_summary}= Evaluate '; '.join($ac_checks_missing)
|
|
Fail WF16 AC verification incomplete: ${missing_ac_summary}
|
|
END
|
|
Log Plan status verified: ${plan_id} in state ${status_state}
|