forked from cleveragents/cleveragents-core
998aaf25f8
Centralize E2E initialization in common suite setup so DB-dependent CLI commands no longer rely on per-suite or per-test init workarounds. This also sanitizes suite-home names to prevent invalid path-derived initialization failures in isolated Robot runs. ISSUES CLOSED: #1023
442 lines
27 KiB
Plaintext
442 lines
27 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E acceptance criteria for M6 (v3.5.0) — autonomy hardening.
|
|
...
|
|
... Validates the full CleverAgents CLI with **zero mocking**:
|
|
... session lifecycle, automation profiles, project setup,
|
|
... plan lifecycle via A2A facade, guard enforcement, and
|
|
... a full autonomy acceptance flow. LLM-dependent tests
|
|
... use ``Skip If No LLM Keys`` for graceful degradation.
|
|
Resource common_e2e.resource
|
|
Suite Setup M6 Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
Force Tags E2E
|
|
|
|
*** Keywords ***
|
|
M6 Suite Setup
|
|
[Documentation] E2E Suite Setup plus unique run suffix and action registration.
|
|
E2E Suite Setup
|
|
# Generate a unique suffix for resource/project names to avoid UNIQUE
|
|
# constraint collisions on repeated E2E runs against the same database.
|
|
# uuid4 provides ~4 billion possibilities vs randint's 9000 for parallel CI safety.
|
|
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
|
|
Set Suite Variable ${RUN_SUFFIX} ${suffix}
|
|
# Register the local/code-review action needed by plan lifecycle tests.
|
|
# Pick an actor that matches the available API key.
|
|
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
|
|
IF ${has_openai}
|
|
${actor}= Set Variable openai/gpt-4o
|
|
ELSE
|
|
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
|
|
END
|
|
${action_yaml}= Catenate SEPARATOR=\n
|
|
... name: local/code-review
|
|
... description: "Perform a code review on project sources"
|
|
... strategy_actor: ${actor}
|
|
... execution_actor: ${actor}
|
|
... definition_of_done: "Code review completed: issues identified and documented."
|
|
... reusable: true
|
|
... read_only: false
|
|
... state: available
|
|
${yaml_path}= Set Variable ${SUITE_HOME}${/}code-review-action.yaml
|
|
Create File ${yaml_path} ${action_yaml}
|
|
${action_create}= Run CleverAgents Command action create --config ${yaml_path} expected_rc=None
|
|
IF ${action_create.rc} != 0
|
|
Log Could not register local/code-review action (rc=${action_create.rc}): ${action_create.stderr} WARN
|
|
END
|
|
|
|
Setup Plan Test Resources
|
|
[Documentation] Create temp git repo, register as resource, and create a project.
|
|
... Returns the project name. Shared across LLM-dependent plan tests
|
|
... to eliminate boilerplate (P2-22: keep file under 500 lines).
|
|
[Arguments] ${prefix}
|
|
${repo_path}= Create Temp Git Repo ${prefix}-repo-${RUN_SUFFIX}
|
|
${res_name}= Set Variable ${prefix}-res-${RUN_SUFFIX}
|
|
${add_res}= Run CleverAgents Command resource add git-checkout ${res_name} --path ${repo_path}
|
|
Should Be Equal As Integers ${add_res.rc} 0
|
|
${proj_name}= Set Variable ${prefix}-proj-${RUN_SUFFIX}
|
|
${create_proj}= Run CleverAgents Command project create --resource ${res_name} ${proj_name}
|
|
Should Be Equal As Integers ${create_proj.rc} 0
|
|
RETURN ${proj_name}
|
|
|
|
Plan Lifecycle Assertions
|
|
[Documentation] Assert plan lifecycle output after successful plan use.
|
|
[Arguments] ${plan_use}
|
|
Output Should Contain ${plan_use} plan_id
|
|
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
|
Verify Plan In List ${plan_id}
|
|
|
|
Verify Plan In List
|
|
[Documentation] Verify a plan appears in list output.
|
|
[Arguments] ${plan_id}
|
|
${list_result}= Run CleverAgents Command plan list --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${list_result.rc} 0
|
|
... list failed (rc=${list_result.rc}): ${list_result.stderr}
|
|
Output Should Contain ${list_result} ${plan_id}
|
|
|
|
Guard Enforcement Assertions
|
|
[Documentation] Assert guard/profile information is present in plan output.
|
|
...
|
|
... Verifies the resolved automation profile matches the expected
|
|
... profile name and checks for guard-specific fields in output.
|
|
[Arguments] ${plan_use} ${expected_profile}=manual
|
|
Output Should Contain ${plan_use} plan_id
|
|
# Verify the resolved automation profile matches expectation (P2-24)
|
|
${resolved_profile}= Safe Parse Json Field ${plan_use.stdout} automation_profile
|
|
IF '${resolved_profile}' != ''
|
|
Should Be Equal As Strings ${resolved_profile} ${expected_profile}
|
|
END
|
|
# Verify automation-profile-specific fields appear in combined output
|
|
${combined}= Set Variable ${plan_use.stdout} ${plan_use.stderr}
|
|
${combined_lower}= Evaluate ($combined).lower()
|
|
${has_profile}= Evaluate 'automation_profile' in $combined_lower or 'automation-profile' in $combined_lower or 'require_sandbox' in $combined_lower or 'decompose_task' in $combined_lower or 'create_tool' in $combined_lower or 'safety_profile' in $combined_lower or '"${expected_profile}"' in $combined_lower
|
|
Should Be True ${has_profile} Plan output should reference automation profile fields (automation_profile, require_sandbox, decompose_task, etc.)
|
|
|
|
Full Flow Apply Step
|
|
[Documentation] Attempt apply after execute succeeds and verify state transition.
|
|
... Apply may fail if the plan is not in the correct state after
|
|
... LLM execution; hard assertions on the success path only.
|
|
[Arguments] ${plan_id}
|
|
${apply}= Run CleverAgents Command plan apply --yes ${plan_id} --format json expected_rc=None timeout=180s
|
|
Log Apply result rc=${apply.rc} stdout=${apply.stdout} stderr=${apply.stderr}
|
|
IF ${apply.rc} == 0
|
|
Output Should Contain ${apply} ${plan_id}
|
|
# Verify the plan actually transitioned — check for apply-phase indicators
|
|
${apply_phase}= Safe Parse Json Field ${apply.stdout} phase
|
|
IF '${apply_phase}' != ''
|
|
Should Contain ${apply_phase.lower()} apply Plan phase should indicate apply after apply
|
|
END
|
|
ELSE
|
|
Fail apply failed (rc=${apply.rc}) stdout=${apply.stdout} stderr=${apply.stderr}
|
|
END
|
|
|
|
*** Test Cases ***
|
|
M6 E2E Session Lifecycle
|
|
[Documentation] Create, list, show, and delete a session via the CLI.
|
|
[Teardown] Run Keyword And Ignore Error Run CleverAgents Command session delete ${session_id} --yes expected_rc=None
|
|
# Initialise variable so teardown never references an undefined name.
|
|
Set Test Variable ${session_id} ${EMPTY}
|
|
# Create a session and capture its ID via JSON output
|
|
${create}= Run CleverAgents Command session create --format json
|
|
Should Not Be Empty ${create.stdout}
|
|
Output Should Contain ${create} session_id
|
|
# Parse session_id from JSON output (handle potential leading non-JSON output)
|
|
${session_id}= Safe Parse Json Field ${create.stdout} session_id
|
|
Set Test Variable ${session_id}
|
|
Should Not Be Empty ${session_id}
|
|
# List sessions — should include the new one
|
|
${list_result}= Run CleverAgents Command session list --format json
|
|
Output Should Contain ${list_result} ${session_id}
|
|
# Show session details
|
|
${show_result}= Run CleverAgents Command session show ${session_id} --format json
|
|
Output Should Contain ${show_result} ${session_id}
|
|
# Delete session
|
|
${delete_result}= Run CleverAgents Command session delete ${session_id} --yes
|
|
Should Be Equal As Integers ${delete_result.rc} 0
|
|
Output Should Contain ${delete_result} deleted
|
|
# Confirm deletion by re-listing — the session should no longer appear.
|
|
${post_delete_list}= Run CleverAgents Command session list --format json
|
|
${post_combined}= Set Variable ${post_delete_list.stdout}\n${post_delete_list.stderr}
|
|
Should Not Contain ${post_combined} ${session_id} Session ${session_id} still appears after deletion
|
|
|
|
M6 E2E Automation Profile List
|
|
[Documentation] Verify all eight built-in automation profiles are listed.
|
|
${result}= Run CleverAgents Command automation-profile list --format json
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Not Be Empty ${result.stdout}
|
|
# Verify all 8 built-in profiles are present (spec: manual through full-auto).
|
|
Output Should Contain ${result} manual
|
|
Output Should Contain ${result} review
|
|
Output Should Contain ${result} supervised
|
|
Output Should Contain ${result} cautious
|
|
Output Should Contain ${result} trusted
|
|
# Use case-sensitive match with JSON-quoted forms for short profile names
|
|
# to avoid false-positive substring matches (e.g. "ci" in "deficit").
|
|
Output Should Contain ${result} "auto" case_insensitive=${FALSE}
|
|
Output Should Contain ${result} "ci" case_insensitive=${FALSE}
|
|
Output Should Contain ${result} full-auto
|
|
|
|
M6 E2E Automation Profile Show
|
|
[Documentation] Verify showing a single automation profile returns threshold info.
|
|
${result}= Run CleverAgents Command automation-profile show manual
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Not Be Empty ${result.stdout}
|
|
Output Should Contain ${result} manual
|
|
# Threshold fields should be present
|
|
Output Should Contain ${result} decompose_task
|
|
Output Should Contain ${result} create_tool
|
|
Output Should Contain ${result} select_tool
|
|
|
|
M6 E2E Config Automation Profile
|
|
[Documentation] Set and get the automation profile configuration key.
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
# Set the profile to ci
|
|
${set_result}= Run CleverAgents Command config set core.automation-profile ci --format json
|
|
Should Be Equal As Integers ${set_result.rc} 0
|
|
# Use case-sensitive JSON-quoted match for "ci" to avoid false positives on short strings
|
|
Output Should Contain ${set_result} "ci" case_insensitive=${FALSE}
|
|
# Get and verify via JSON — parse the value field for precise assertion
|
|
${get_result}= Run CleverAgents Command config get core.automation-profile --format json
|
|
Should Be Equal As Integers ${get_result.rc} 0
|
|
${config_value}= Safe Parse Json Field ${get_result.stdout} value
|
|
Should Be Equal As Strings ${config_value} ci
|
|
# Reset to default (manual) is handled by [Teardown].
|
|
|
|
M6 E2E Init And Project Setup
|
|
[Documentation] Initialize, add a git resource, create a project, and verify.
|
|
[Teardown] Log Init And Project Setup test completed (cleanup via suite teardown)
|
|
# Create a temp git repo for the resource
|
|
${repo_path}= Create Temp Git Repo m6-setup-repo-${RUN_SUFFIX}
|
|
# Add a git-checkout resource with a run-unique name
|
|
${res_name}= Set Variable m6-setup-res-${RUN_SUFFIX}
|
|
${add_res}= Run CleverAgents Command resource add git-checkout ${res_name} --path ${repo_path}
|
|
Should Be Equal As Integers ${add_res.rc} 0
|
|
Output Should Contain ${add_res} ${res_name}
|
|
# Create a project linked to the resource
|
|
${proj_name}= Set Variable m6-setup-proj-${RUN_SUFFIX}
|
|
${create_proj}= Run CleverAgents Command project create --resource ${res_name} ${proj_name}
|
|
Should Be Equal As Integers ${create_proj.rc} 0
|
|
Output Should Contain ${create_proj} ${proj_name}
|
|
# Verify project appears in the list (use --format json to avoid Rich table truncation)
|
|
${list_proj}= Run CleverAgents Command project list --format json
|
|
Should Be Equal As Integers ${list_proj.rc} 0
|
|
Output Should Contain ${list_proj} ${proj_name}
|
|
|
|
M6 E2E Guard Enforcement Denylist Budget Limits
|
|
[Documentation] Register a custom profile with explicit guards (denylist, budget,
|
|
... tool-call limits) and verify the guard fields are stored correctly.
|
|
... Covers AC-4: "Test verifies guard enforcement (denylist, budget
|
|
... caps, tool call limits)".
|
|
[Teardown] Run Keyword And Ignore Error Run CleverAgents Command automation-profile remove ${guard_profile_name} --yes expected_rc=None
|
|
Set Test Variable ${guard_profile_name} ${EMPTY}
|
|
${guard_profile_name}= Set Variable e2e-guard-${RUN_SUFFIX}
|
|
Set Test Variable ${guard_profile_name}
|
|
# Build YAML config for a custom profile with all three guard categories
|
|
${yaml}= Catenate SEPARATOR=\n
|
|
... name: ${guard_profile_name}
|
|
... description: E2E test profile with denylist budget and tool-call limits
|
|
... schema_version: "1.0"
|
|
... decompose_task: 0.5
|
|
... create_tool: 0.5
|
|
... select_tool: 1.0
|
|
... safety:
|
|
... ${SPACE}${SPACE}require_sandbox: true
|
|
... ${SPACE}${SPACE}require_checkpoints: true
|
|
... ${SPACE}${SPACE}max_cost_per_plan: 50.0
|
|
... ${SPACE}${SPACE}max_retries_per_step: 3
|
|
... guards:
|
|
... ${SPACE}${SPACE}max_tool_calls_per_step: 10
|
|
... ${SPACE}${SPACE}max_total_cost: 25.0
|
|
... ${SPACE}${SPACE}tool_denylist:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}- shell_exec
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}- rm_rf
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}- drop_database
|
|
... ${SPACE}${SPACE}require_approval_for_writes: true
|
|
... ${SPACE}${SPACE}require_approval_for_apply: true
|
|
${yaml_path}= Set Variable ${SUITE_HOME}${/}guard-profile.yaml
|
|
Create File ${yaml_path} ${yaml}
|
|
# Register the custom profile
|
|
${add_result}= Run CleverAgents Command automation-profile add --config ${yaml_path} --format json
|
|
Should Be Equal As Integers ${add_result.rc} 0
|
|
Output Should Contain ${add_result} ${guard_profile_name}
|
|
# Show the profile and verify guard fields
|
|
${show_result}= Run CleverAgents Command automation-profile show ${guard_profile_name} --format json
|
|
Should Be Equal As Integers ${show_result.rc} 0
|
|
# Verify denylist entries
|
|
Output Should Contain ${show_result} tool_denylist
|
|
Output Should Contain ${show_result} shell_exec
|
|
Output Should Contain ${show_result} rm_rf
|
|
Output Should Contain ${show_result} drop_database
|
|
# Verify budget cap
|
|
Output Should Contain ${show_result} max_total_cost
|
|
# Verify tool call limit
|
|
Output Should Contain ${show_result} max_tool_calls_per_step
|
|
# Verify approval requirements
|
|
Output Should Contain ${show_result} require_approval_for_writes
|
|
Output Should Contain ${show_result} require_approval_for_apply
|
|
|
|
M6 E2E Plan Lifecycle Via CLI
|
|
[Documentation] Test A2A facade plan lifecycle: create plan via action, list, and status.
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj_name}= Setup Plan Test Resources plan-lc
|
|
# Use an action to create a plan with automation profile
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile ci --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0
|
|
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
Plan Lifecycle Assertions ${plan_use}
|
|
|
|
M6 E2E Guard Enforcement Via Profile
|
|
[Documentation] Verify guard enforcement with a strict automation profile.
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj_name}= Setup Plan Test Resources guard
|
|
# Attempt to use an action; with manual profile guards should constrain autonomy
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile manual --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0
|
|
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
Guard Enforcement Assertions ${plan_use} expected_profile=manual
|
|
|
|
M6 E2E Profile Precedence Plan Overrides Global
|
|
[Documentation] Verify plan-level automation profile overrides global config.
|
|
... Sets global profile to "review", then creates a plan with
|
|
... explicit --automation-profile trusted. The plan output must
|
|
... show "trusted" (not "review"), proving plan > global precedence.
|
|
... Covers AC-5: "Test verifies automation profile resolution
|
|
... precedence (plan > action > global)".
|
|
... Note: action > global precedence requires the action's
|
|
... automation_profile to propagate to the Plan during plan-use,
|
|
... which is not yet wired in PlanLifecycleService.use_action;
|
|
... this test covers plan > global.
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
# Set global profile to "review"
|
|
${set_global}= Run CleverAgents Command config set core.automation-profile review --format json
|
|
Should Be Equal As Integers ${set_global.rc} 0
|
|
${get_global}= Run CleverAgents Command config get core.automation-profile --format json
|
|
${global_value}= Safe Parse Json Field ${get_global.stdout} value
|
|
Should Be Equal As Strings ${global_value} review
|
|
# Setup resources and project
|
|
${proj_name}= Setup Plan Test Resources prec
|
|
# Create plan with EXPLICIT "trusted" profile (should override "review" global)
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile trusted --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0
|
|
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
# Parse automation_profile from plan output: must be "trusted", NOT "review"
|
|
${resolved_profile}= Safe Parse Json Field ${plan_use.stdout} automation_profile
|
|
IF '${resolved_profile}' != ''
|
|
Should Be Equal As Strings ${resolved_profile} trusted
|
|
Should Not Be Equal As Strings ${resolved_profile} review
|
|
Log Precedence confirmed: plan-level "trusted" overrides global "review"
|
|
ELSE
|
|
# Fallback: check combined output for "trusted" profile name
|
|
Output Should Contain ${plan_use} trusted
|
|
END
|
|
|
|
M6 E2E Event Queue Via Plan Lifecycle Transitions
|
|
[Documentation] Exercise event queue publish/subscribe through plan lifecycle.
|
|
... Each plan operation (create, execute) publishes domain events
|
|
... (PLAN_CREATED, PHASE_TRANSITION) consumed by event handlers that
|
|
... update plan state. Verifying state transitions via CLI proves
|
|
... the event bus delivered and processed the published events.
|
|
... Covers AC-3: "Test exercises event queue publish/subscribe via
|
|
... real CLI".
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj_name}= Setup Plan Test Resources evq
|
|
# 1. Create plan — publishes PLAN_CREATED on the domain event bus
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile ci --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0
|
|
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
|
# 2. Verify plan appears in list (proves creation event was processed)
|
|
Verify Plan In List ${plan_id}
|
|
# 3. Capture initial plan state via status
|
|
${status1}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${status1.rc} 0
|
|
... plan status failed (rc=${status1.rc}): ${status1.stderr}
|
|
${initial_phase}= Safe Parse Json Field ${status1.stdout} phase
|
|
${initial_state}= Safe Parse Json Field ${status1.stdout} processing_state
|
|
Log Initial phase=${initial_phase} processing_state=${initial_state}
|
|
# 4. Execute plan — triggers PHASE_TRANSITION and execution events
|
|
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
|
|
# P0-3: Hard assertions on SUCCESS path; LLM execution may legitimately fail
|
|
# (e.g. strategize phase not yet complete — no background worker in E2E).
|
|
IF ${execute.rc} == 0
|
|
Output Should Contain ${execute} ${plan_id}
|
|
# 5. Re-check status — should reflect post-execute state
|
|
${status2}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${status2.rc} 0
|
|
... plan status failed after execute (rc=${status2.rc}): ${status2.stderr}
|
|
${post_phase}= Safe Parse Json Field ${status2.stdout} phase
|
|
${post_state}= Safe Parse Json Field ${status2.stdout} processing_state
|
|
Log Post-execute phase=${post_phase} processing_state=${post_state}
|
|
# P0-3: Hard assertion — at least one state field must be populated,
|
|
# proving the event bus delivered and processed lifecycle events.
|
|
${state_populated}= Evaluate '${post_phase}' != '' or '${post_state}' != ''
|
|
Should Be True ${state_populated}
|
|
... Event queue should process lifecycle events — expected non-empty phase or processing_state after execute
|
|
ELSE
|
|
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
|
|
END
|
|
# 6. Final: plan should still appear in list
|
|
Verify Plan In List ${plan_id}
|
|
|
|
M6 E2E Hierarchical Decomposition Via Plan Tree
|
|
[Documentation] Verify hierarchical plan decomposition via the plan tree command.
|
|
... After plan creation and execution, ``plan tree`` should return
|
|
... a decision tree with at least root-level decisions. If the LLM
|
|
... decomposes the task into subplans the tree will show nested
|
|
... children with hierarchy indicators.
|
|
... Covers AC-6: "Test exercises a full autonomy acceptance flow
|
|
... with hierarchical decomposition".
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj_name}= Setup Plan Test Resources tree
|
|
# Create plan with full-auto profile (enables install_dependency for decomposition)
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile full-auto --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0
|
|
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
|
# Execute plan to populate the decision tree with strategy/execution decisions
|
|
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
|
|
IF ${execute.rc} != 0
|
|
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
|
|
END
|
|
# View decision tree — must contain at least root-level decisions
|
|
${tree}= Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=120s
|
|
# P0-4: Hard assertion — tree command must succeed.
|
|
Should Be Equal As Integers ${tree.rc} 0
|
|
... plan tree failed (rc=${tree.rc}): ${tree.stderr}
|
|
Should Not Be Empty ${tree.stdout} Plan tree output should not be empty
|
|
# P0-4: Hard assertion — at least one decision node must exist after execution.
|
|
${decision_count}= Evaluate $tree.stdout.count('"decision_id"')
|
|
Log Decision tree contains ${decision_count} decision node(s)
|
|
Should Be True ${decision_count} >= 1
|
|
... Plan tree should contain at least one decision node after execution (found ${decision_count})
|
|
# Check for hierarchical children (decomposition infrastructure indicator)
|
|
${has_children_key}= Evaluate '"children"' in $tree.stdout
|
|
IF ${has_children_key}
|
|
Log Tree structure has children field (decomposition infrastructure functional)
|
|
END
|
|
|
|
M6 E2E Full Autonomy Acceptance Flow
|
|
[Documentation] End-to-end: init, resource, project, action, plan use, execute, status, apply.
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj_name}= Setup Plan Test Resources flow
|
|
# Use an action to create a plan
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile full-auto --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0
|
|
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
# Parse plan_id from JSON output
|
|
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
|
# Check plan status
|
|
${status}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${status.rc} 0
|
|
... plan status failed (rc=${status.rc}): ${status.stderr}
|
|
Output Should Contain ${status} ${plan_id}
|
|
# Execute
|
|
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
|
|
IF ${execute.rc} == 0
|
|
Output Should Contain ${execute} ${plan_id}
|
|
# Verify execute output contains phase-transition indicators
|
|
${exec_phase}= Safe Parse Json Field ${execute.stdout} phase
|
|
Log Execute phase: ${exec_phase}
|
|
# Apply
|
|
Full Flow Apply Step ${plan_id}
|
|
ELSE
|
|
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
|
|
END
|
|
# Final verification: list should show the plan
|
|
Verify Plan In List ${plan_id}
|