*** Settings *** Documentation E2E test for Workflow Example 4: Multi-Project Dependency Update. ... ... Advanced scenario using the supervised automation profile. ... Three microservices share a common library with a breaking ... change (v1 to v2). CleverAgents creates a parent plan ... targeting all 4 projects, spawns child subplans per project, ... executes in dependency order (common-lib first, then services), ... validates per project, and applies in dependency order. ... ... Zero mocking — real CLI, real LLM API keys. Resource common_e2e.resource Suite Setup WF04 Suite Setup Suite Teardown E2E Suite Teardown Force Tags E2E *** Variables *** ${ACTION_BASE} local/wf04-dep-update ${LIB_BASE} wf04-common-lib ${SVC1_BASE} wf04-svc-auth ${SVC2_BASE} wf04-svc-billing ${SVC3_BASE} wf04-svc-gateway ${WF04_SNAPSHOT_HELPER} ${CURDIR}${/}wf04_snapshot_helper.py *** Keywords *** WF04 Suite Setup [Documentation] E2E Suite Setup plus unique suffix generation and actor selection. E2E Suite Setup # Initialise the database so commands work in all tests. ${init}= Run CleverAgents Command init --force --yes Should Be Equal As Integers ${init.rc} 0 Should Not Contain ${init.stdout}${init.stderr} Traceback Should Not Contain ${init.stdout}${init.stderr} INTERNAL # Generate a unique suffix for resource/project names to avoid # UNIQUE constraint collisions on repeated or parallel CI runs. ${suffix}= Evaluate __import__('uuid').uuid4().hex[:12] Set Suite Variable ${RUN_SUFFIX} ${suffix} # Derive unique names from base + suffix Set Suite Variable ${ACTION_NAME} ${ACTION_BASE}-${suffix} Set Suite Variable ${LIB_RESOURCE} ${LIB_BASE}-res-${suffix} Set Suite Variable ${SVC1_RESOURCE} ${SVC1_BASE}-res-${suffix} Set Suite Variable ${SVC2_RESOURCE} ${SVC2_BASE}-res-${suffix} Set Suite Variable ${SVC3_RESOURCE} ${SVC3_BASE}-res-${suffix} Set Suite Variable ${LIB_PROJECT} ${LIB_BASE}-proj-${suffix} Set Suite Variable ${SVC1_PROJECT} ${SVC1_BASE}-proj-${suffix} Set Suite Variable ${SVC2_PROJECT} ${SVC2_BASE}-proj-${suffix} Set Suite Variable ${SVC3_PROJECT} ${SVC3_BASE}-proj-${suffix} # Pick an actor that matches the available API key. ${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', '')) ${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', '')) IF ${has_openai} ${actor}= Set Variable openai/gpt-4o ELSE IF ${has_anthropic} ${actor}= Set Variable anthropic/claude-sonnet-4-20250514 ELSE ${actor}= Set Variable openai/gpt-4o END Set Suite Variable ${LLM_ACTOR} ${actor} Create Library Repo [Documentation] Create temp git repo for the common library. ${repo}= Create Temp Git Repo ${LIB_BASE}-${RUN_SUFFIX} Create Directory ${repo}${/}src ${lib_content}= Catenate SEPARATOR=\n ... """Common library v1 — shared utilities.""" ... ${EMPTY} ... ${EMPTY} ... __version__ = "1.0.0" ... ${EMPTY} ... ${EMPTY} ... def connect(host, port): ... ${SPACE}${SPACE}${SPACE}${SPACE}"""Connect to a service (v1 API).""" ... ${SPACE}${SPACE}${SPACE}${SPACE}return {"host": host, "port": port, "status": "connected"} Create File ${repo}${/}src${/}client.py ${lib_content} ${git_add}= Run Process git add . cwd=${repo} timeout=30s on_timeout=kill Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr} ${git_commit}= Run Process git commit -m Initial common library v1 cwd=${repo} timeout=30s on_timeout=kill Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr} RETURN ${repo} Create Service Repo [Documentation] Create temp git repo for a microservice. [Arguments] ${name} ${import_line} ${repo}= Create Temp Git Repo ${name}-${RUN_SUFFIX} Create Directory ${repo}${/}src ${svc_content}= Catenate SEPARATOR=\n ... """${name} service — depends on common-lib v1.""" ... ${import_line} ... ${EMPTY} ... ${EMPTY} ... def start(): ... ${SPACE}${SPACE}${SPACE}${SPACE}conn = connect("localhost", 8080) ... ${SPACE}${SPACE}${SPACE}${SPACE}return conn Create File ${repo}${/}src${/}app.py ${svc_content} Create File ${repo}${/}requirements.txt common-lib==1.0.0\n ${git_add}= Run Process git add . cwd=${repo} timeout=30s on_timeout=kill Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr} ${git_commit}= Run Process git commit -m Initial ${name} service cwd=${repo} timeout=30s on_timeout=kill Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr} RETURN ${repo} Register Resource And Project [Documentation] Register a git-checkout resource and create a project. [Arguments] ${resource_name} ${project_name} ${repo_dir} ${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=30s on_timeout=kill Should Be Equal As Integers ${branch_result.rc} 0 git rev-parse failed: ${branch_result.stderr} ${branch}= Strip String ${branch_result.stdout} ${r_res}= Run CleverAgents Command ... resource add git-checkout ${resource_name} ... --path ${repo_dir} --branch ${branch} Should Be Equal As Integers ${r_res.rc} 0 resource add failed: ${r_res.stderr} Should Not Contain ${r_res.stdout}${r_res.stderr} Traceback Should Not Contain ${r_res.stdout}${r_res.stderr} INTERNAL ${r_proj}= Run CleverAgents Command ... project create ${project_name} ... --resource ${resource_name} Should Be Equal As Integers ${r_proj.rc} 0 project create failed: ${r_proj.stderr} Should Not Contain ${r_proj.stdout}${r_proj.stderr} Traceback Should Not Contain ${r_proj.stdout}${r_proj.stderr} INTERNAL WF04 Test Teardown [Documentation] Log diagnostic context on failure for debugging. ... Captures plan status and decision tree so CI failures ... in this 25-minute LLM-dependent test have actionable data. ${plan_id}= Get Variable Value ${WF04_PLAN_ID} ${EMPTY} IF '${plan_id}' != '' ${status} ${result}= Run Keyword And Ignore Error ... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s IF '${status}' == 'PASS' Log Teardown plan status: ${result.stdout} WARN END ${tree_status} ${tree_result}= Run Keyword And Ignore Error ... Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=30s IF '${tree_status}' == 'PASS' Log Teardown plan tree: ${tree_result.stdout} WARN END END Register Validation For Project [Documentation] Create a validation YAML, register it, and attach it to a project's resource. [Arguments] ${validation_name} ${resource_name} ${project_name} ${val_yaml}= Catenate SEPARATOR=\n ... name: ${validation_name} ... description: "Simple pass-through validation for E2E testing" ... source: custom ... mode: required ... code: | ... ${SPACE}${SPACE}return {"passed": True, "data": {}, "message": "Validation passed"} ... input_schema: ... ${SPACE}${SPACE}type: object ... ${SPACE}${SPACE}properties: {} ... timeout: 30 ${val_path}= Set Variable ${SUITE_HOME}${/}${validation_name}.yaml Create File ${val_path} ${val_yaml} ${r_add}= Run CleverAgents Command ... validation add --config ${val_path} expected_rc=None Log Validation add rc=${r_add.rc} stdout=${r_add.stdout} stderr=${r_add.stderr} Should Be Equal As Integers ${r_add.rc} 0 ... validation add failed (rc=${r_add.rc}): ${r_add.stderr} Should Not Contain ${r_add.stdout}${r_add.stderr} Traceback Should Not Contain ${r_add.stdout}${r_add.stderr} INTERNAL # Positional args: (resource first, validation second) ${r_attach}= Run CleverAgents Command ... validation attach --project ${project_name} ... ${resource_name} ${validation_name} expected_rc=None Log Validation attach rc=${r_attach.rc} stdout=${r_attach.stdout} stderr=${r_attach.stderr} Should Be Equal As Integers ${r_attach.rc} 0 ... validation attach failed for ${project_name} (rc=${r_attach.rc}): ${r_attach.stderr} Should Not Contain ${r_attach.stdout}${r_attach.stderr} Traceback Should Not Contain ${r_attach.stdout}${r_attach.stderr} INTERNAL Attach Validation To Project [Documentation] Attach an already-registered validation to a project's resource. [Arguments] ${validation_name} ${resource_name} ${project_name} # Positional args: (resource first, validation second) ${r_attach}= Run CleverAgents Command ... validation attach --project ${project_name} ... ${resource_name} ${validation_name} expected_rc=None Log Validation attach rc=${r_attach.rc} stdout=${r_attach.stdout} stderr=${r_attach.stderr} Should Be Equal As Integers ${r_attach.rc} 0 ... validation attach failed for ${project_name} (rc=${r_attach.rc}): ${r_attach.stderr} Should Not Contain ${r_attach.stdout}${r_attach.stderr} Traceback Should Not Contain ${r_attach.stdout}${r_attach.stderr} INTERNAL Parse Json Payload [Documentation] Parse JSON object/array from stdout with optional log preamble. ... Delegates to ``Extract JSON From Stdout`` which uses ... ``json.JSONDecoder().raw_decode()`` for robustness against ... trailing non-JSON output. [Arguments] ${stdout} ${parsed}= Extract JSON From Stdout ${stdout} RETURN ${parsed} Get WF04 Plan Snapshot [Documentation] Read parent/subplan metadata for deterministic WF04 assertions. [Arguments] ${plan_id} ${snapshot_result}= Run Process ... ${PYTHON} ${WF04_SNAPSHOT_HELPER} ${plan_id} ... cwd=${SUITE_HOME} timeout=120s on_timeout=kill ... env:CLEVERAGENTS_HOME=${SUITE_HOME} ... env:CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true ... env:NO_COLOR=1 ... env:PYTHONPATH=${WORKSPACE}${/}src Should Be Equal As Integers ${snapshot_result.rc} 0 ... Snapshot helper failed (rc=${snapshot_result.rc}): ${snapshot_result.stderr} Should Not Be Empty ${snapshot_result.stdout} ${snapshot}= Parse Json Payload ${snapshot_result.stdout} RETURN ${snapshot} Verify WF04 Child Plan Spawning [Documentation] AC-4: verify exactly 4 child plans mapped one-per-project. [Arguments] ${snapshot} ${subplans}= Evaluate $snapshot.get('subplans', []) ${subplan_count}= Evaluate len($subplans) IF ${subplan_count} == 0 Skip LLM produced no child plans — AC-4 child-plan mapping assertions were not exercised END Should Be Equal As Integers ${subplan_count} 4 ... Expected exactly 4 child plans (common-lib + 3 services), found ${subplan_count} ${mapped_projects}= Evaluate sorted({p for sp in $subplans for p in sp.get('mapped_projects', []) if p}) ${expected_projects}= Create List ${LIB_PROJECT} ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT} ${expected_sorted}= Evaluate sorted($expected_projects) ${all_projects_covered}= Evaluate $mapped_projects == $expected_sorted Should Be True ${all_projects_covered} ... Child plans should cover all 4 projects. expected=${expected_sorted} actual=${mapped_projects} ${single_mapping}= Evaluate all(len(sp.get('mapped_projects', [])) == 1 for sp in $subplans) Should Be True ${single_mapping} ... Each child plan should map to exactly one project scope Verify WF04 Execution Order [Documentation] AC-5: common-lib executes before service subplans. [Arguments] ${snapshot} ${subplans}= Evaluate $snapshot.get('subplans', []) ${subplan_count}= Evaluate len($subplans) IF ${subplan_count} == 0 Skip LLM produced no child plans — AC-5 execution-order assertions were not exercised END ${lib_subplans}= Evaluate [sp for sp in $subplans if '${LIB_PROJECT}' in sp.get('mapped_projects', [])] ${lib_count}= Evaluate len($lib_subplans) Should Be Equal As Integers ${lib_count} 1 ... Expected exactly one common-lib child plan, found ${lib_count} ${lib_completed}= Evaluate $lib_subplans[0].get('completed_at') or $lib_subplans[0].get('execute_completed_at') or '' Should Not Be Empty ${lib_completed} ... common-lib child plan completion timestamp is required for ordering checks ${service_projects}= Create List ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT} ${svc_subplans}= Evaluate [sp for sp in $subplans if any(p in $service_projects for p in sp.get('mapped_projects', []))] ${svc_count}= Evaluate len($svc_subplans) Should Be Equal As Integers ${svc_count} 3 ... Expected exactly three service child plans, found ${svc_count} ${svc_starts_present}= Evaluate all((sp.get('started_at') or sp.get('execute_started_at') or '') != '' for sp in $svc_subplans) Should Be True ${svc_starts_present} ... Service child plans must expose start timestamps for execution-order checks ${services_after_lib}= Evaluate all((sp.get('started_at') or sp.get('execute_started_at')) >= $lib_completed for sp in $svc_subplans) Should Be True ${services_after_lib} ... Service execution must start only after common-lib execution completes Verify WF04 Validation Outcomes [Documentation] AC-6: each child plan must report validation pass results. [Arguments] ${snapshot} ${subplans}= Evaluate $snapshot.get('subplans', []) ${subplan_count}= Evaluate len($subplans) IF ${subplan_count} == 0 Skip LLM produced no child plans — AC-6 per-project validation assertions were not exercised END ${validation_present}= Evaluate all(isinstance(sp.get('child_validation_summary'), dict) and len(sp.get('child_validation_summary')) > 0 for sp in $subplans) Should Be True ${validation_present} ... Each child plan must expose a non-empty validation summary ${validation_passed}= Evaluate all(int((sp.get('child_validation_summary') or {}).get('required_passed', 0) or 0) >= 1 and int((sp.get('child_validation_summary') or {}).get('required_failed', 0) or 0) == 0 for sp in $subplans) Should Be True ${validation_passed} ... Each child plan must pass required validations (required_failed == 0) Verify WF04 Apply Order [Documentation] AC-7: common-lib apply must complete before service applies. [Arguments] ${snapshot} ${subplans}= Evaluate $snapshot.get('subplans', []) ${subplan_count}= Evaluate len($subplans) IF ${subplan_count} == 0 Skip LLM produced no child plans — AC-7 apply-order assertions were not exercised END ${lib_subplans}= Evaluate [sp for sp in $subplans if '${LIB_PROJECT}' in sp.get('mapped_projects', [])] ${lib_count}= Evaluate len($lib_subplans) Should Be Equal As Integers ${lib_count} 1 ... Expected exactly one common-lib child plan, found ${lib_count} ${lib_applied}= Evaluate $lib_subplans[0].get('applied_at') or $lib_subplans[0].get('child_updated_at') or '' Should Not Be Empty ${lib_applied} ... common-lib child plan apply timestamp is required for apply-order checks ${service_projects}= Create List ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT} ${svc_subplans}= Evaluate [sp for sp in $subplans if any(p in $service_projects for p in sp.get('mapped_projects', []))] ${svc_count}= Evaluate len($svc_subplans) Should Be Equal As Integers ${svc_count} 3 ... Expected exactly three service child plans, found ${svc_count} ${svc_apply_present}= Evaluate all((sp.get('applied_at') or sp.get('child_updated_at') or '') != '' for sp in $svc_subplans) Should Be True ${svc_apply_present} ... Service child plans must expose apply/updated timestamps for apply-order checks ${services_after_lib_apply}= Evaluate all((sp.get('applied_at') or sp.get('child_updated_at')) >= $lib_applied for sp in $svc_subplans) Should Be True ${services_after_lib_apply} ... Service applies must occur after common-lib apply Count Decision Nodes [Documentation] Recursively count decision nodes in a plan tree JSON structure. ... Invokes ``wf04_snapshot_helper.py --count-nodes`` as a subprocess ... to avoid importing the application DI container into the Robot ... test runner process. [Arguments] ${tree_payload} ${tree_json}= Evaluate __import__('json').dumps($tree_payload) ${tmp_path}= Evaluate __import__('tempfile').NamedTemporaryFile(mode='w', suffix='.json', delete=False).name Evaluate __import__('pathlib').Path(r'${tmp_path}').write_text($tree_json, encoding='utf-8') ${result}= Run Process ... ${PYTHON} ${WF04_SNAPSHOT_HELPER} --count-nodes ${tmp_path} ... timeout=30s on_timeout=kill Evaluate __import__('os').unlink(r'${tmp_path}') Should Be Equal As Integers ${result.rc} 0 ... count-nodes failed (rc=${result.rc}): ${result.stderr} ${count}= Convert To Integer ${result.stdout.strip()} RETURN ${count} Verify Plan In List [Documentation] Verify a plan appears in list output. [Arguments] ${plan_id} ${list_result}= Run CleverAgents Command ... plan list --format json expected_rc=None timeout=120s Should Be Equal As Integers ${list_result.rc} 0 ... list failed (rc=${list_result.rc}): ${list_result.stderr} Output Should Contain ${list_result} ${plan_id} *** Test Cases *** WF04 Multi Project Dependency Update Supervised Profile [Documentation] Full supervised-profile workflow: register 4 repos, ... create multi-project action with invariants, plan use ... targeting all 4 projects with --automation-profile supervised, ... execute with child plan spawning in dependency order, ... verify per-project validation, and apply in dependency order. [Timeout] 25 minutes [Teardown] WF04 Test Teardown Skip If No LLM Keys # Initialise test variable for teardown access. Set Test Variable ${WF04_PLAN_ID} ${EMPTY} # ---- Create fixture repos ---- ${lib_repo}= Create Library Repo ${svc1_repo}= Create Service Repo svc-auth from common_lib.client import connect ${svc2_repo}= Create Service Repo svc-billing from common_lib.client import connect ${svc3_repo}= Create Service Repo svc-gateway from common_lib.client import connect # ---- Register resources and projects ---- Register Resource And Project ${LIB_RESOURCE} ${LIB_PROJECT} ${lib_repo} Register Resource And Project ${SVC1_RESOURCE} ${SVC1_PROJECT} ${svc1_repo} Register Resource And Project ${SVC2_RESOURCE} ${SVC2_PROJECT} ${svc2_repo} Register Resource And Project ${SVC3_RESOURCE} ${SVC3_PROJECT} ${svc3_repo} # ---- Register and attach validations for all 4 projects (AC-6) ---- ${val_name}= Set Variable local/wf04-val-${RUN_SUFFIX} Register Validation For Project ${val_name} ${LIB_RESOURCE} ${LIB_PROJECT} # Validation already registered; attach to remaining 3 projects Attach Validation To Project ${val_name} ${SVC1_RESOURCE} ${SVC1_PROJECT} Attach Validation To Project ${val_name} ${SVC2_RESOURCE} ${SVC2_PROJECT} Attach Validation To Project ${val_name} ${SVC3_RESOURCE} ${SVC3_PROJECT} # ---- Create action with supervised profile and invariants ---- ${action_yaml}= Catenate SEPARATOR=\n ... name: ${ACTION_NAME} ... description: Update common-lib from v1 to v2 across all dependent services ... definition_of_done: All services updated to use common-lib v2 API ... strategy_actor: ${LLM_ACTOR} ... execution_actor: ${LLM_ACTOR} ... automation_profile: supervised ... reusable: true ... state: available ... invariants: ... ${SPACE}${SPACE}- "Each dependent project must be updated in its own child plan" ... ${SPACE}${SPACE}- "All child plans must pass validation before any can be applied" ... ${SPACE}${SPACE}- "The library update in common-lib must be applied first" ${action_path}= Set Variable ${SUITE_HOME}${/}wf04_action.yaml Create File ${action_path} ${action_yaml} ${r_action}= Run CleverAgents Command ... action create --config ${action_path} Should Be Equal As Integers ${r_action.rc} 0 ... action create failed (rc=${r_action.rc}): ${r_action.stderr} Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL Output Should Contain ${r_action} ${ACTION_NAME} # ---- Plan use targeting ALL 4 projects with supervised profile (AC-3) ---- ${r_use}= Run CleverAgents Command ... plan use ${ACTION_NAME} ... ${LIB_PROJECT} ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT} ... --automation-profile supervised ... --format json ... timeout=120s Should Be Equal As Integers ${r_use.rc} 0 ... plan use failed (rc=${r_use.rc}): ${r_use.stderr} Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL ${use_payload}= Parse Json Payload ${r_use.stdout} ${plan_id}= Evaluate str($use_payload.get('plan_id', '')) Should Not Be Empty ${plan_id} msg=Expected plan_id in plan use JSON output Log Plan ID: ${plan_id} Set Test Variable ${WF04_PLAN_ID} ${plan_id} # Verify plan targets all 4 projects exactly (AC-3) ${use_projects}= Evaluate sorted([link.get('project_name', '') for link in $use_payload.get('project_links', []) if isinstance(link, dict)]) ${expected_projects}= Create List ${LIB_PROJECT} ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT} ${expected_sorted}= Evaluate sorted($expected_projects) ${projects_match}= Evaluate $use_projects == $expected_sorted Should Be True ${projects_match} ... plan use should target all 4 projects. expected=${expected_sorted} actual=${use_projects} # ---- Strategize ---- # Supervised profile requires two explicit ``plan execute`` calls: # the first advances the plan through strategize, the second runs # actual execution. Both use the same CLI command. ${r_strat}= Run CleverAgents Command ... plan execute ${plan_id} ... --format json expected_rc=None timeout=180s Log Strategize rc=${r_strat.rc} stdout=${r_strat.stdout} stderr=${r_strat.stderr} Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL IF ${r_strat.rc} != 0 Fail plan execute (strategize) failed (rc=${r_strat.rc}): ${r_strat.stderr} END # ---- Decision tree — verify child plan spawning (AC-4) ---- ${r_tree}= Run CleverAgents Command ... plan tree ${plan_id} --format json ... expected_rc=None timeout=60s Should Be Equal As Integers ${r_tree.rc} 0 ... plan tree failed (rc=${r_tree.rc}): ${r_tree.stderr} Should Not Contain ${r_tree.stdout}${r_tree.stderr} Traceback Should Not Contain ${r_tree.stdout}${r_tree.stderr} INTERNAL Should Not Be Empty ${r_tree.stdout} Plan tree output should not be empty Log Decision tree: ${r_tree.stdout} # Parse tree for child plan / decision structure ${tree_payload}= Parse Json Payload ${r_tree.stdout} ${decision_count}= Count Decision Nodes ${tree_payload} Log Decision tree contains ${decision_count} decision node(s) Should Be True ${decision_count} >= 1 ... Plan tree should contain at least one decision node after strategize (found ${decision_count}) # Some providers/runs expose a minimal strategize tree before execute. # Child spawning is asserted deterministically after execute via snapshot. # ---- Execute — dependency-ordered execution (AC-5) ---- ${r_exec}= Run CleverAgents Command ... plan execute ${plan_id} ... --format json expected_rc=None timeout=300s Log Execute rc=${r_exec.rc} stdout=${r_exec.stdout} stderr=${r_exec.stderr} Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL IF ${r_exec.rc} != 0 Fail plan execute failed (rc=${r_exec.rc}): ${r_exec.stderr} END # Deterministic WF04 assertions after execute (AC-4/AC-5/AC-6) ${exec_snapshot}= Get WF04 Plan Snapshot ${plan_id} # Guard: if the snapshot contains zero subplans, skip the entire test rather # than letting individual verification keywords silently skip all ACs. # This ensures CI reports show SKIPPED (visible) rather than PASSED (misleading). ${exec_subplan_count}= Evaluate int($exec_snapshot.get('subplan_count', 0)) IF ${exec_subplan_count} == 0 Skip LLM produced 0 subplans — AC-4/5/6/7 verification cannot be exercised (entire test skipped) END Verify WF04 Child Plan Spawning ${exec_snapshot} Verify WF04 Execution Order ${exec_snapshot} Verify WF04 Validation Outcomes ${exec_snapshot} # ---- Post-execute decision tree — verify child plan count (AC-4) ---- ${r_tree_post}= Run CleverAgents Command ... plan tree ${plan_id} --format json ... expected_rc=None timeout=60s Should Be Equal As Integers ${r_tree_post.rc} 0 ... plan tree (post-execute) failed (rc=${r_tree_post.rc}): ${r_tree_post.stderr} Should Not Contain ${r_tree_post.stdout}${r_tree_post.stderr} Traceback Should Not Contain ${r_tree_post.stdout}${r_tree_post.stderr} INTERNAL ${tree_post_payload}= Parse Json Payload ${r_tree_post.stdout} ${post_exec_decision_count}= Count Decision Nodes ${tree_post_payload} Log Post-execute decision tree contains ${post_exec_decision_count} decision node(s) # Require non-trivial tree depth after execute. Subplan spawning is # asserted deterministically via internal snapshot checks above. Should Be True ${post_exec_decision_count} >= 2 ... Plan tree should contain at least 2 decision nodes after execute (found ${post_exec_decision_count}) # Execute should preserve or grow the tree — never shrink it. Should Be True ${post_exec_decision_count} >= ${decision_count} ... Post-execute decision count (${post_exec_decision_count}) should not be less than post-strategize count (${decision_count}) # ---- AC-6 verified above via per-child validation summaries ---- # ---- Plan list — verify plan exists ---- Verify Plan In List ${plan_id} # ---- Verify plan status shows multi-project state ---- ${r_status_mid}= Run CleverAgents Command ... plan status ${plan_id} --format json ... expected_rc=None timeout=60s Should Be Equal As Integers ${r_status_mid.rc} 0 ... plan status failed (rc=${r_status_mid.rc}): ${r_status_mid.stderr} Should Not Contain ${r_status_mid.stdout}${r_status_mid.stderr} Traceback Should Not Contain ${r_status_mid.stdout}${r_status_mid.stderr} INTERNAL Output Should Contain ${r_status_mid} ${plan_id} # Check for automation profile reference in status ${status_combined}= Set Variable ${r_status_mid.stdout}${r_status_mid.stderr} ${status_lower}= Evaluate ($status_combined).lower() ${has_profile_ref}= Evaluate 'supervised' in $status_lower or 'automation' in $status_lower or 'profile' in $status_lower Should Be True ${has_profile_ref} ... Plan status should reference automation profile (supervised) # Parse automation_profile field for precise assertion ${mid_profile}= Safe Parse Json Field ${r_status_mid.stdout} automation_profile Should Not Be Empty ${mid_profile} ... automation_profile field should be present in plan status JSON Should Be Equal As Strings ${mid_profile} supervised ... Plan automation profile should be supervised (found ${mid_profile}) # ---- Diff ---- ${r_diff}= Run CleverAgents Command ... plan diff ${plan_id} --format plain ... expected_rc=None timeout=60s Log Diff rc=${r_diff.rc} stdout=${r_diff.stdout} stderr=${r_diff.stderr} Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL Should Be Equal As Integers ${r_diff.rc} 0 ... plan diff failed (rc=${r_diff.rc}): ${r_diff.stderr} # ---- Apply — dependency-ordered apply (AC-7) ---- ${r_apply}= Run CleverAgents Command ... plan apply ${plan_id} --yes --format json ... expected_rc=None timeout=180s Log Apply rc=${r_apply.rc} stdout=${r_apply.stdout} stderr=${r_apply.stderr} Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL IF ${r_apply.rc} == 0 Output Should Contain ${r_apply} ${plan_id} # Verify the plan transitioned — check for apply-phase indicators (AC-7) ${apply_phase}= Safe Parse Json Field ${r_apply.stdout} phase Should Not Be Empty ${apply_phase} ... phase field should be present in apply JSON output ${apply_phase_lower}= Evaluate ($apply_phase).lower() Should Contain ${apply_phase_lower} apply ... Plan phase should indicate apply after apply (found ${apply_phase}) ${apply_snapshot}= Get WF04 Plan Snapshot ${plan_id} # Guard: apply snapshot must also contain subplans for AC-7 verification. # The exec guard above already skips the whole test if 0 subplans, so # reaching this point implies subplans existed post-execute. If they # disappeared post-apply, that would be a real regression worth flagging. ${apply_subplan_count}= Evaluate int($apply_snapshot.get('subplan_count', 0)) Should Be True ${apply_subplan_count} >= 1 ... Snapshot reported 0 subplans after apply — subplans existed post-execute but vanished post-apply Verify WF04 Apply Order ${apply_snapshot} Verify WF04 Validation Outcomes ${apply_snapshot} ELSE Fail apply failed (rc=${r_apply.rc}) stdout=${r_apply.stdout} stderr=${r_apply.stderr} END # ---- Verify final status ---- ${r_status}= Run CleverAgents Command ... plan status ${plan_id} --format json ... expected_rc=None timeout=60s Should Be Equal As Integers ${r_status.rc} 0 ... plan status (final) failed (rc=${r_status.rc}): ${r_status.stderr} Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL Should Not Be Empty ${r_status.stdout} Final plan status output should not be empty Output Should Contain ${r_status} ${plan_id} # Parse phase from final status and assert non-empty (proves lifecycle events processed) ${final_phase}= Safe Parse Json Field ${r_status.stdout} phase ${final_state}= Safe Parse Json Field ${r_status.stdout} processing_state Log Final phase=${final_phase} processing_state=${final_state} ${final_state_populated}= Evaluate $final_phase != '' or $final_state != '' Should Be True ${final_state_populated} ... Final plan status should have non-empty phase or processing_state # After a full lifecycle (execute + apply), phase or state should reflect completion ${final_lower}= Evaluate ($final_phase).lower() if $final_phase else '' ${state_lower}= Evaluate ($final_state).lower() if $final_state else '' ${is_terminal}= Evaluate 'apply' in $final_lower or 'complete' in $final_lower or 'done' in $final_lower or 'complete' in $state_lower or 'done' in $state_lower or 'applied' in $state_lower Should Be True ${is_terminal} ... Final status should indicate a terminal/applied state (phase=${final_phase}, state=${final_state}) # ---- Final: plan should still appear in list ---- Verify Plan In List ${plan_id}