*** Settings *** Documentation E2E acceptance criteria for M6 (v3.5.0) — autonomy hardening. ... ... Validates the full CleverAgents CLI with **zero mocking**: ... session lifecycle, automation profiles, project setup, ... plan lifecycle via A2A facade, guard enforcement, and ... a full autonomy acceptance flow. LLM-dependent tests ... use ``Skip If No LLM Keys`` for graceful degradation. Resource common_e2e.resource Suite Setup M6 Suite Setup Suite Teardown E2E Suite Teardown Force Tags E2E *** Keywords *** M6 Suite Setup [Documentation] E2E Suite Setup plus unique run suffix and action registration. E2E Suite Setup # Generate a unique suffix for resource/project names to avoid UNIQUE # constraint collisions on repeated E2E runs against the same database. # uuid4 provides ~4 billion possibilities vs randint's 9000 for parallel CI safety. ${suffix}= Evaluate __import__('uuid').uuid4().hex[:12] Set Suite Variable ${RUN_SUFFIX} ${suffix} # Register the local/code-review action needed by plan lifecycle tests. # Pick an actor that matches the available API key. ${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', '')) IF ${has_openai} ${actor}= Set Variable openai/gpt-4o ELSE ${actor}= Set Variable anthropic/claude-sonnet-4-20250514 END ${action_yaml}= Catenate SEPARATOR=\n ... name: local/code-review ... description: "Perform a code review on project sources" ... strategy_actor: ${actor} ... execution_actor: ${actor} ... definition_of_done: "Code review completed: issues identified and documented." ... reusable: true ... read_only: false ... state: available ${yaml_path}= Set Variable ${SUITE_HOME}${/}code-review-action.yaml Create File ${yaml_path} ${action_yaml} ${action_create}= Run CleverAgents Command action create --config ${yaml_path} expected_rc=None IF ${action_create.rc} != 0 Log Could not register local/code-review action (rc=${action_create.rc}): ${action_create.stderr} WARN END Setup Plan Test Resources [Documentation] Create temp git repo, register as resource, and create a project. ... Returns the project name. Shared across LLM-dependent plan tests ... to eliminate boilerplate (P2-22: keep file under 500 lines). [Arguments] ${prefix} ${repo_path}= Create Temp Git Repo ${prefix}-repo-${RUN_SUFFIX} ${res_name}= Set Variable ${prefix}-res-${RUN_SUFFIX} ${add_res}= Run CleverAgents Command resource add git-checkout ${res_name} --path ${repo_path} Should Be Equal As Integers ${add_res.rc} 0 ${proj_name}= Set Variable ${prefix}-proj-${RUN_SUFFIX} ${create_proj}= Run CleverAgents Command project create --resource ${res_name} ${proj_name} Should Be Equal As Integers ${create_proj.rc} 0 RETURN ${proj_name} Plan Lifecycle Assertions [Documentation] Assert plan lifecycle output after successful plan use. [Arguments] ${plan_use} Output Should Contain ${plan_use} plan_id ${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output Verify Plan In List ${plan_id} Verify Plan In List [Documentation] Verify a plan appears in list output. [Arguments] ${plan_id} ${list_result}= Run CleverAgents Command plan list --format json expected_rc=None timeout=120s Should Be Equal As Integers ${list_result.rc} 0 ... list failed (rc=${list_result.rc}): ${list_result.stderr} Output Should Contain ${list_result} ${plan_id} Guard Enforcement Assertions [Documentation] Assert guard/profile information is present in plan output. ... ... Verifies the resolved automation profile matches the expected ... profile name and checks for guard-specific fields in output. [Arguments] ${plan_use} ${expected_profile}=manual Output Should Contain ${plan_use} plan_id # Verify the resolved automation profile matches expectation (P2-24) ${resolved_profile}= Safe Parse Json Field ${plan_use.stdout} automation_profile IF '${resolved_profile}' != '' Should Be Equal As Strings ${resolved_profile} ${expected_profile} END # Verify automation-profile-specific fields appear in combined output ${combined}= Set Variable ${plan_use.stdout} ${plan_use.stderr} ${combined_lower}= Evaluate ($combined).lower() ${has_profile}= Evaluate 'automation_profile' in $combined_lower or 'automation-profile' in $combined_lower or 'require_sandbox' in $combined_lower or 'decompose_task' in $combined_lower or 'create_tool' in $combined_lower or 'safety_profile' in $combined_lower or '"${expected_profile}"' in $combined_lower Should Be True ${has_profile} Plan output should reference automation profile fields (automation_profile, require_sandbox, decompose_task, etc.) Full Flow Apply Step [Documentation] Attempt apply after execute succeeds and verify state transition. ... Apply may fail if the plan is not in the correct state after ... LLM execution; hard assertions on the success path only. [Arguments] ${plan_id} ${apply}= Run CleverAgents Command plan apply --yes ${plan_id} --format json expected_rc=None timeout=180s Log Apply result rc=${apply.rc} stdout=${apply.stdout} stderr=${apply.stderr} IF ${apply.rc} == 0 Output Should Contain ${apply} ${plan_id} # Verify the plan actually transitioned — check for apply-phase indicators ${apply_phase}= Safe Parse Json Field ${apply.stdout} phase IF '${apply_phase}' != '' Should Contain ${apply_phase.lower()} apply Plan phase should indicate apply after apply END ELSE Fail apply failed (rc=${apply.rc}) stdout=${apply.stdout} stderr=${apply.stderr} END *** Test Cases *** M6 E2E Session Lifecycle [Documentation] Create, list, show, and delete a session via the CLI. [Teardown] Run Keyword And Ignore Error Run CleverAgents Command session delete ${session_id} --yes expected_rc=None # Initialise variable so teardown never references an undefined name. Set Test Variable ${session_id} ${EMPTY} # Create a session and capture its ID via JSON output ${create}= Run CleverAgents Command session create --format json Should Not Be Empty ${create.stdout} Output Should Contain ${create} session_id # Parse session_id from JSON output (handle potential leading non-JSON output) ${session_id}= Safe Parse Json Field ${create.stdout} session_id Set Test Variable ${session_id} Should Not Be Empty ${session_id} # List sessions — should include the new one ${list_result}= Run CleverAgents Command session list --format json Output Should Contain ${list_result} ${session_id} # Show session details ${show_result}= Run CleverAgents Command session show ${session_id} --format json Output Should Contain ${show_result} ${session_id} # Delete session ${delete_result}= Run CleverAgents Command session delete ${session_id} --yes Should Be Equal As Integers ${delete_result.rc} 0 Output Should Contain ${delete_result} deleted # Confirm deletion by re-listing — the session should no longer appear. ${post_delete_list}= Run CleverAgents Command session list --format json ${post_combined}= Set Variable ${post_delete_list.stdout}\n${post_delete_list.stderr} Should Not Contain ${post_combined} ${session_id} Session ${session_id} still appears after deletion M6 E2E Automation Profile List [Documentation] Verify all eight built-in automation profiles are listed. ${result}= Run CleverAgents Command automation-profile list --format json Should Be Equal As Integers ${result.rc} 0 Should Not Be Empty ${result.stdout} # Verify all 8 built-in profiles are present (spec: manual through full-auto). Output Should Contain ${result} manual Output Should Contain ${result} review Output Should Contain ${result} supervised Output Should Contain ${result} cautious Output Should Contain ${result} trusted # Use case-sensitive match with JSON-quoted forms for short profile names # to avoid false-positive substring matches (e.g. "ci" in "deficit"). Output Should Contain ${result} "auto" case_insensitive=${FALSE} Output Should Contain ${result} "ci" case_insensitive=${FALSE} Output Should Contain ${result} full-auto M6 E2E Automation Profile Show [Documentation] Verify showing a single automation profile returns threshold info. ${result}= Run CleverAgents Command automation-profile show manual Should Be Equal As Integers ${result.rc} 0 Should Not Be Empty ${result.stdout} Output Should Contain ${result} manual # Threshold fields should be present Output Should Contain ${result} decompose_task Output Should Contain ${result} create_tool Output Should Contain ${result} select_tool M6 E2E Config Automation Profile [Documentation] Set and get the automation profile configuration key. [Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None # Set the profile to ci ${set_result}= Run CleverAgents Command config set core.automation-profile ci --format json Should Be Equal As Integers ${set_result.rc} 0 # Use case-sensitive JSON-quoted match for "ci" to avoid false positives on short strings Output Should Contain ${set_result} "ci" case_insensitive=${FALSE} # Get and verify via JSON — parse the value field for precise assertion ${get_result}= Run CleverAgents Command config get core.automation-profile --format json Should Be Equal As Integers ${get_result.rc} 0 ${config_value}= Safe Parse Json Field ${get_result.stdout} value Should Be Equal As Strings ${config_value} ci # Reset to default (manual) is handled by [Teardown]. M6 E2E Init And Project Setup [Documentation] Initialize, add a git resource, create a project, and verify. [Teardown] Log Init And Project Setup test completed (cleanup via suite teardown) # Create a temp git repo for the resource ${repo_path}= Create Temp Git Repo m6-setup-repo-${RUN_SUFFIX} # Add a git-checkout resource with a run-unique name ${res_name}= Set Variable m6-setup-res-${RUN_SUFFIX} ${add_res}= Run CleverAgents Command resource add git-checkout ${res_name} --path ${repo_path} Should Be Equal As Integers ${add_res.rc} 0 Output Should Contain ${add_res} ${res_name} # Create a project linked to the resource ${proj_name}= Set Variable m6-setup-proj-${RUN_SUFFIX} ${create_proj}= Run CleverAgents Command project create --resource ${res_name} ${proj_name} Should Be Equal As Integers ${create_proj.rc} 0 Output Should Contain ${create_proj} ${proj_name} # Verify project appears in the list (use --format json to avoid Rich table truncation) ${list_proj}= Run CleverAgents Command project list --format json Should Be Equal As Integers ${list_proj.rc} 0 Output Should Contain ${list_proj} ${proj_name} M6 E2E Guard Enforcement Denylist Budget Limits [Documentation] Register a custom profile with explicit guards (denylist, budget, ... tool-call limits) and verify the guard fields are stored correctly. ... Covers AC-4: "Test verifies guard enforcement (denylist, budget ... caps, tool call limits)". [Teardown] Run Keyword And Ignore Error Run CleverAgents Command automation-profile remove ${guard_profile_name} --yes expected_rc=None Set Test Variable ${guard_profile_name} ${EMPTY} ${guard_profile_name}= Set Variable e2e-guard-${RUN_SUFFIX} Set Test Variable ${guard_profile_name} # Build YAML config for a custom profile with all three guard categories ${yaml}= Catenate SEPARATOR=\n ... name: ${guard_profile_name} ... description: E2E test profile with denylist budget and tool-call limits ... schema_version: "1.0" ... decompose_task: 0.5 ... create_tool: 0.5 ... select_tool: 1.0 ... safety: ... ${SPACE}${SPACE}require_sandbox: true ... ${SPACE}${SPACE}require_checkpoints: true ... ${SPACE}${SPACE}max_cost_per_plan: 50.0 ... ${SPACE}${SPACE}max_retries_per_step: 3 ... guards: ... ${SPACE}${SPACE}max_tool_calls_per_step: 10 ... ${SPACE}${SPACE}max_total_cost: 25.0 ... ${SPACE}${SPACE}tool_denylist: ... ${SPACE}${SPACE}${SPACE}${SPACE}- shell_exec ... ${SPACE}${SPACE}${SPACE}${SPACE}- rm_rf ... ${SPACE}${SPACE}${SPACE}${SPACE}- drop_database ... ${SPACE}${SPACE}require_approval_for_writes: true ... ${SPACE}${SPACE}require_approval_for_apply: true ${yaml_path}= Set Variable ${SUITE_HOME}${/}guard-profile.yaml Create File ${yaml_path} ${yaml} # Register the custom profile ${add_result}= Run CleverAgents Command automation-profile add --config ${yaml_path} --format json Should Be Equal As Integers ${add_result.rc} 0 Output Should Contain ${add_result} ${guard_profile_name} # Show the profile and verify guard fields ${show_result}= Run CleverAgents Command automation-profile show ${guard_profile_name} --format json Should Be Equal As Integers ${show_result.rc} 0 # Verify denylist entries Output Should Contain ${show_result} tool_denylist Output Should Contain ${show_result} shell_exec Output Should Contain ${show_result} rm_rf Output Should Contain ${show_result} drop_database # Verify budget cap Output Should Contain ${show_result} max_total_cost # Verify tool call limit Output Should Contain ${show_result} max_tool_calls_per_step # Verify approval requirements Output Should Contain ${show_result} require_approval_for_writes Output Should Contain ${show_result} require_approval_for_apply M6 E2E Plan Lifecycle Via CLI [Documentation] Test A2A facade plan lifecycle: create plan via action, list, and status. [Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None Skip If No LLM Keys ${proj_name}= Setup Plan Test Resources plan-lc # Use an action to create a plan with automation profile ${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile ci --format json expected_rc=None timeout=180s # P0-2: Fail (not Skip) when API keys are present but plan use fails. Should Be Equal As Integers ${plan_use.rc} 0 ... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr} Plan Lifecycle Assertions ${plan_use} M6 E2E Guard Enforcement Via Profile [Documentation] Verify guard enforcement with a strict automation profile. [Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None Skip If No LLM Keys ${proj_name}= Setup Plan Test Resources guard # Attempt to use an action; with manual profile guards should constrain autonomy ${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile manual --format json expected_rc=None timeout=180s # P0-2: Fail (not Skip) when API keys are present but plan use fails. Should Be Equal As Integers ${plan_use.rc} 0 ... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr} Guard Enforcement Assertions ${plan_use} expected_profile=manual M6 E2E Profile Precedence Plan Overrides Global [Documentation] Verify plan-level automation profile overrides global config. ... Sets global profile to "review", then creates a plan with ... explicit --automation-profile trusted. The plan output must ... show "trusted" (not "review"), proving plan > global precedence. ... Covers AC-5: "Test verifies automation profile resolution ... precedence (plan > action > global)". ... Note: action > global precedence requires the action's ... automation_profile to propagate to the Plan during plan-use, ... which is not yet wired in PlanLifecycleService.use_action; ... this test covers plan > global. [Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None Skip If No LLM Keys # Set global profile to "review" ${set_global}= Run CleverAgents Command config set core.automation-profile review --format json Should Be Equal As Integers ${set_global.rc} 0 ${get_global}= Run CleverAgents Command config get core.automation-profile --format json ${global_value}= Safe Parse Json Field ${get_global.stdout} value Should Be Equal As Strings ${global_value} review # Setup resources and project ${proj_name}= Setup Plan Test Resources prec # Create plan with EXPLICIT "trusted" profile (should override "review" global) ${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile trusted --format json expected_rc=None timeout=180s # P0-2: Fail (not Skip) when API keys are present but plan use fails. Should Be Equal As Integers ${plan_use.rc} 0 ... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr} # Parse automation_profile from plan output: must be "trusted", NOT "review" ${resolved_profile}= Safe Parse Json Field ${plan_use.stdout} automation_profile IF '${resolved_profile}' != '' Should Be Equal As Strings ${resolved_profile} trusted Should Not Be Equal As Strings ${resolved_profile} review Log Precedence confirmed: plan-level "trusted" overrides global "review" ELSE # Fallback: check combined output for "trusted" profile name Output Should Contain ${plan_use} trusted END M6 E2E Event Queue Via Plan Lifecycle Transitions [Documentation] Exercise event queue publish/subscribe through plan lifecycle. ... Each plan operation (create, execute) publishes domain events ... (PLAN_CREATED, PHASE_TRANSITION) consumed by event handlers that ... update plan state. Verifying state transitions via CLI proves ... the event bus delivered and processed the published events. ... Covers AC-3: "Test exercises event queue publish/subscribe via ... real CLI". [Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None Skip If No LLM Keys ${proj_name}= Setup Plan Test Resources evq # 1. Create plan — publishes PLAN_CREATED on the domain event bus ${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile ci --format json expected_rc=None timeout=180s # P0-2: Fail (not Skip) when API keys are present but plan use fails. Should Be Equal As Integers ${plan_use.rc} 0 ... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr} ${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output # 2. Verify plan appears in list (proves creation event was processed) Verify Plan In List ${plan_id} # 3. Capture initial plan state via status ${status1}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s Should Be Equal As Integers ${status1.rc} 0 ... plan status failed (rc=${status1.rc}): ${status1.stderr} ${initial_phase}= Safe Parse Json Field ${status1.stdout} phase ${initial_state}= Safe Parse Json Field ${status1.stdout} processing_state Log Initial phase=${initial_phase} processing_state=${initial_state} # 4. Execute plan — triggers PHASE_TRANSITION and execution events ${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s # P0-3: Hard assertions on SUCCESS path; LLM execution may legitimately fail # (e.g. strategize phase not yet complete — no background worker in E2E). IF ${execute.rc} == 0 Output Should Contain ${execute} ${plan_id} # 5. Re-check status — should reflect post-execute state ${status2}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s Should Be Equal As Integers ${status2.rc} 0 ... plan status failed after execute (rc=${status2.rc}): ${status2.stderr} ${post_phase}= Safe Parse Json Field ${status2.stdout} phase ${post_state}= Safe Parse Json Field ${status2.stdout} processing_state Log Post-execute phase=${post_phase} processing_state=${post_state} # P0-3: Hard assertion — at least one state field must be populated, # proving the event bus delivered and processed lifecycle events. ${state_populated}= Evaluate '${post_phase}' != '' or '${post_state}' != '' Should Be True ${state_populated} ... Event queue should process lifecycle events — expected non-empty phase or processing_state after execute ELSE Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr} END # 6. Final: plan should still appear in list Verify Plan In List ${plan_id} M6 E2E Hierarchical Decomposition Via Plan Tree [Documentation] Verify hierarchical plan decomposition via the plan tree command. ... After plan creation and execution, ``plan tree`` should return ... a decision tree with at least root-level decisions. If the LLM ... decomposes the task into subplans the tree will show nested ... children with hierarchy indicators. ... Covers AC-6: "Test exercises a full autonomy acceptance flow ... with hierarchical decomposition". [Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None Skip If No LLM Keys ${proj_name}= Setup Plan Test Resources tree # Create plan with full-auto profile (enables install_dependency for decomposition) ${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile full-auto --format json expected_rc=None timeout=180s # P0-2: Fail (not Skip) when API keys are present but plan use fails. Should Be Equal As Integers ${plan_use.rc} 0 ... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr} ${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output # Execute plan to populate the decision tree with strategy/execution decisions ${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s IF ${execute.rc} != 0 Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr} END # View decision tree — must contain at least root-level decisions ${tree}= Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=120s # P0-4: Hard assertion — tree command must succeed. Should Be Equal As Integers ${tree.rc} 0 ... plan tree failed (rc=${tree.rc}): ${tree.stderr} Should Not Be Empty ${tree.stdout} Plan tree output should not be empty # P0-4: Hard assertion — at least one decision node must exist after execution. ${decision_count}= Evaluate $tree.stdout.count('"decision_id"') Log Decision tree contains ${decision_count} decision node(s) Should Be True ${decision_count} >= 1 ... Plan tree should contain at least one decision node after execution (found ${decision_count}) # Check for hierarchical children (decomposition infrastructure indicator) ${has_children_key}= Evaluate '"children"' in $tree.stdout IF ${has_children_key} Log Tree structure has children field (decomposition infrastructure functional) END M6 E2E Full Autonomy Acceptance Flow [Documentation] End-to-end: init, resource, project, action, plan use, execute, status, apply. [Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None Skip If No LLM Keys ${proj_name}= Setup Plan Test Resources flow # Use an action to create a plan ${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile full-auto --format json expected_rc=None timeout=180s # P0-2: Fail (not Skip) when API keys are present but plan use fails. Should Be Equal As Integers ${plan_use.rc} 0 ... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr} # Parse plan_id from JSON output ${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output # Check plan status ${status}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s Should Be Equal As Integers ${status.rc} 0 ... plan status failed (rc=${status.rc}): ${status.stderr} Output Should Contain ${status} ${plan_id} # Execute ${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s IF ${execute.rc} == 0 Output Should Contain ${execute} ${plan_id} # Verify execute output contains phase-transition indicators ${exec_phase}= Safe Parse Json Field ${execute.stdout} phase Log Execute phase: ${exec_phase} # Apply Full Flow Apply Step ${plan_id} ELSE Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr} END # Final verification: list should show the plan Verify Plan In List ${plan_id}