hurui200320
e2b127b7e5
CI / lint (pull_request) Successful in 37s
CI / typecheck (pull_request) Successful in 1m18s
CI / security (pull_request) Successful in 56s
CI / quality (pull_request) Successful in 48s
CI / build (pull_request) Successful in 30s
CI / helm (pull_request) Successful in 41s
CI / push-validation (pull_request) Successful in 28s
CI / integration_tests (pull_request) Successful in 4m32s
CI / e2e_tests (pull_request) Successful in 4m42s
CI / coverage (pull_request) Successful in 13m24s
CI / unit_tests (pull_request) Successful in 3m13s
CI / docker (pull_request) Successful in 1m36s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-regression (push) Failing after 0s
CI / benchmark-publish (push) Failing after 0s
CI / push-validation (push) Successful in 12s
CI / build (push) Successful in 15s
CI / helm (push) Successful in 16s
CI / lint (push) Successful in 43s
CI / typecheck (push) Successful in 51s
CI / security (push) Successful in 51s
CI / e2e_tests (push) Successful in 2m14s
CI / quality (push) Successful in 3m44s
CI / integration_tests (push) Successful in 7m0s
CI / unit_tests (push) Successful in 8m33s
CI / coverage (push) Successful in 6m21s
CI / docker (push) Successful in 1m31s
CI / status-check (push) Successful in 2s
The existing actor-selection logic in several E2E suite setups checked only
whether OPENAI_API_KEY was present (non-empty). A valid key that has hit its
quota limit passes that check but fails at runtime with HTTP 429, causing the
test to fail even though Anthropic credits are available.
Changes:
- Add robot/e2e/check_openai_key.py: stdlib-only (urllib.request) script that
sends a minimal chat-completion request ('Hi', max_tokens=1, gpt-4o-mini) to
the OpenAI API. Exits 0 on HTTP 200; exits 1 for quota (429), auth (401),
network errors, or any other failure.
- Add 'Resolve LLM Actor' keyword to robot/e2e/common_e2e.resource: runs the
probe script via ${PYTHON} and returns the openai_model argument (default
openai/gpt-4o) on success, or the anthropic_model argument (default
anthropic/claude-sonnet-4-20250514) on failure. Skips the probe entirely when
OPENAI_API_KEY is not set.
- Update m6_acceptance.robot, wf04_multi_project.robot, wf05_db_migration.robot,
wf07_cicd.robot, and wf16_devcontainer.robot to use 'Resolve LLM Actor'
instead of the inline has_openai boolean check.
No production source code (src/) is modified. The decision to fall back to
Anthropic is made once per suite setup, before any test runs.
Closes #10198
431 lines
25 KiB
Plaintext
431 lines
25 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E acceptance criteria for M6 (v3.5.0) — autonomy hardening.
|
|
...
|
|
... Validates the full CleverAgents CLI with **zero mocking**:
|
|
... session lifecycle, automation profiles, project setup,
|
|
... plan lifecycle via A2A facade, guard enforcement, and
|
|
... a full autonomy acceptance flow. LLM-dependent tests
|
|
... use ``Skip If No LLM Keys`` for graceful degradation.
|
|
Resource common_e2e.resource
|
|
Suite Setup M6 Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
Force Tags E2E
|
|
|
|
*** Keywords ***
|
|
M6 Suite Setup
|
|
[Documentation] E2E Suite Setup plus unique run suffix and action registration.
|
|
E2E Suite Setup
|
|
# Generate a unique suffix for resource/project names to avoid UNIQUE
|
|
# constraint collisions on repeated E2E runs against the same database.
|
|
# uuid4 provides ~4 billion possibilities vs randint's 9000 for parallel CI safety.
|
|
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
|
|
Set Suite Variable ${RUN_SUFFIX} ${suffix}
|
|
# Register the local/code-review action needed by plan lifecycle tests.
|
|
# Probe the OpenAI API to confirm the key is usable before selecting actor.
|
|
# Falls back to Anthropic when the key is absent or quota-exhausted.
|
|
${actor}= Resolve LLM Actor
|
|
${action_yaml}= Catenate SEPARATOR=\n
|
|
... name: local/code-review
|
|
... description: "Perform a code review on project sources"
|
|
... strategy_actor: ${actor}
|
|
... execution_actor: ${actor}
|
|
... definition_of_done: "Code review completed: issues identified and documented."
|
|
... reusable: true
|
|
... read_only: false
|
|
... state: available
|
|
${yaml_path}= Set Variable ${SUITE_HOME}${/}code-review-action.yaml
|
|
Create File ${yaml_path} ${action_yaml}
|
|
${action_create}= Run CleverAgents Command action create --config ${yaml_path} expected_rc=None
|
|
IF ${action_create.rc} != 0
|
|
Log Could not register local/code-review action (rc=${action_create.rc}): ${action_create.stderr} WARN
|
|
END
|
|
|
|
Setup Plan Test Resources
|
|
[Documentation] Create temp git repo, register as resource, and create a project.
|
|
... Returns the project name. Shared across LLM-dependent plan tests
|
|
... to eliminate boilerplate (P2-22: keep file under 500 lines).
|
|
[Arguments] ${prefix}
|
|
${repo_path}= Create Temp Git Repo ${prefix}-repo-${RUN_SUFFIX}
|
|
${res_name}= Set Variable ${prefix}-res-${RUN_SUFFIX}
|
|
${add_res}= Run CleverAgents Command resource add git-checkout ${res_name} --path ${repo_path}
|
|
Should Be Equal As Integers ${add_res.rc} 0
|
|
${proj_name}= Set Variable ${prefix}-proj-${RUN_SUFFIX}
|
|
${create_proj}= Run CleverAgents Command project create --resource ${res_name} ${proj_name}
|
|
Should Be Equal As Integers ${create_proj.rc} 0
|
|
RETURN ${proj_name}
|
|
|
|
Plan Lifecycle Assertions
|
|
[Documentation] Assert plan lifecycle output after successful plan use.
|
|
[Arguments] ${plan_use}
|
|
Output Should Contain ${plan_use} plan_id
|
|
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
|
Verify Plan In List ${plan_id}
|
|
|
|
Verify Plan In List
|
|
[Documentation] Verify a plan appears in list output.
|
|
[Arguments] ${plan_id}
|
|
${list_result}= Run CleverAgents Command plan list --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${list_result.rc} 0 msg=list failed (rc=${list_result.rc}): ${list_result.stderr}
|
|
Output Should Contain ${list_result} ${plan_id}
|
|
|
|
Guard Enforcement Assertions
|
|
[Documentation] Assert guard/profile information is present in plan output.
|
|
...
|
|
... Verifies the resolved automation profile matches the expected
|
|
... profile name and checks for guard-specific fields in output.
|
|
[Arguments] ${plan_use} ${expected_profile}=manual
|
|
Output Should Contain ${plan_use} plan_id
|
|
# Verify the resolved automation profile matches expectation (P2-24)
|
|
${resolved_profile}= Safe Parse Json Field ${plan_use.stdout} automation_profile
|
|
IF '${resolved_profile}' != ''
|
|
Should Be Equal As Strings ${resolved_profile} ${expected_profile}
|
|
END
|
|
# Verify automation-profile-specific fields appear in combined output
|
|
${combined}= Set Variable ${plan_use.stdout} ${plan_use.stderr}
|
|
${combined_lower}= Evaluate ($combined).lower()
|
|
${has_profile}= Evaluate 'automation_profile' in $combined_lower or 'automation-profile' in $combined_lower or 'require_sandbox' in $combined_lower or 'decompose_task' in $combined_lower or 'create_tool' in $combined_lower or 'safety_profile' in $combined_lower or '"${expected_profile}"' in $combined_lower
|
|
Should Be True ${has_profile} Plan output should reference automation profile fields (automation_profile, require_sandbox, decompose_task, etc.)
|
|
|
|
Full Flow Apply Step
|
|
[Documentation] Attempt apply after execute succeeds and verify state transition.
|
|
... Apply may fail if the plan is not in the correct state after
|
|
... LLM execution; hard assertions on the success path only.
|
|
[Arguments] ${plan_id}
|
|
${apply}= Run CleverAgents Command plan apply --yes ${plan_id} --format json expected_rc=None timeout=180s
|
|
Log Apply result rc=${apply.rc} stdout=${apply.stdout} stderr=${apply.stderr}
|
|
IF ${apply.rc} == 0
|
|
Output Should Contain ${apply} ${plan_id}
|
|
# Verify the plan actually transitioned — check for apply-phase indicators
|
|
${apply_phase}= Safe Parse Json Field ${apply.stdout} phase
|
|
IF '${apply_phase}' != ''
|
|
Should Contain ${apply_phase.lower()} apply Plan phase should indicate apply after apply
|
|
END
|
|
ELSE
|
|
Fail apply failed (rc=${apply.rc}) stdout=${apply.stdout} stderr=${apply.stderr}
|
|
END
|
|
|
|
*** Test Cases ***
|
|
M6 E2E Session Lifecycle
|
|
[Documentation] Create, list, show, and delete a session via the CLI.
|
|
[Teardown] Run Keyword And Ignore Error Run CleverAgents Command session delete ${session_id} --yes expected_rc=None
|
|
# Initialise variable so teardown never references an undefined name.
|
|
Set Test Variable ${session_id} ${EMPTY}
|
|
# Create a session and capture its ID via JSON output
|
|
${create}= Run CleverAgents Command session create --format json
|
|
Should Not Be Empty ${create.stdout}
|
|
Output Should Contain ${create} session_id
|
|
# Parse session_id from JSON output (handle potential leading non-JSON output)
|
|
${session_id}= Safe Parse Json Field ${create.stdout} session_id
|
|
Set Test Variable ${session_id}
|
|
Should Not Be Empty ${session_id}
|
|
# List sessions — should include the new one
|
|
${list_result}= Run CleverAgents Command session list --format json
|
|
Output Should Contain ${list_result} ${session_id}
|
|
# Show session details
|
|
${show_result}= Run CleverAgents Command session show ${session_id} --format json
|
|
Output Should Contain ${show_result} ${session_id}
|
|
# Delete session
|
|
${delete_result}= Run CleverAgents Command session delete ${session_id} --yes
|
|
Should Be Equal As Integers ${delete_result.rc} 0
|
|
Output Should Contain ${delete_result} deleted
|
|
# Confirm deletion by re-listing — the session should no longer appear.
|
|
${post_delete_list}= Run CleverAgents Command session list --format json
|
|
${post_combined}= Set Variable ${post_delete_list.stdout}\n${post_delete_list.stderr}
|
|
Should Not Contain ${post_combined} ${session_id} Session ${session_id} still appears after deletion
|
|
|
|
M6 E2E Automation Profile List
|
|
[Documentation] Verify all eight built-in automation profiles are listed.
|
|
${result}= Run CleverAgents Command automation-profile list --format json
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Not Be Empty ${result.stdout}
|
|
# Verify all 8 built-in profiles are present (spec: manual through full-auto).
|
|
Output Should Contain ${result} manual
|
|
Output Should Contain ${result} review
|
|
Output Should Contain ${result} supervised
|
|
Output Should Contain ${result} cautious
|
|
Output Should Contain ${result} trusted
|
|
# Use case-sensitive match with JSON-quoted forms for short profile names
|
|
# to avoid false-positive substring matches (e.g. "ci" in "deficit").
|
|
Output Should Contain ${result} "auto" case_insensitive=${FALSE}
|
|
Output Should Contain ${result} "ci" case_insensitive=${FALSE}
|
|
Output Should Contain ${result} full-auto
|
|
|
|
M6 E2E Automation Profile Show
|
|
[Documentation] Verify showing a single automation profile returns threshold info.
|
|
${result}= Run CleverAgents Command automation-profile show manual
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Not Be Empty ${result.stdout}
|
|
Output Should Contain ${result} manual
|
|
# Threshold fields should be present
|
|
Output Should Contain ${result} decompose_task
|
|
Output Should Contain ${result} create_tool
|
|
Output Should Contain ${result} select_tool
|
|
|
|
M6 E2E Config Automation Profile
|
|
[Documentation] Set and get the automation profile configuration key.
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
# Set the profile to ci
|
|
${set_result}= Run CleverAgents Command config set core.automation-profile ci --format json
|
|
Should Be Equal As Integers ${set_result.rc} 0
|
|
# Use case-sensitive JSON-quoted match for "ci" to avoid false positives on short strings
|
|
Output Should Contain ${set_result} "ci" case_insensitive=${FALSE}
|
|
# Get and verify via JSON — parse the value field for precise assertion
|
|
${get_result}= Run CleverAgents Command config get core.automation-profile --format json
|
|
Should Be Equal As Integers ${get_result.rc} 0
|
|
${config_value}= Safe Parse Json Field ${get_result.stdout} value
|
|
Should Be Equal As Strings ${config_value} ci
|
|
# Reset to default (manual) is handled by [Teardown].
|
|
|
|
M6 E2E Init And Project Setup
|
|
[Documentation] Initialize, add a git resource, create a project, and verify.
|
|
[Teardown] Log Init And Project Setup test completed (cleanup via suite teardown)
|
|
# Create a temp git repo for the resource
|
|
${repo_path}= Create Temp Git Repo m6-setup-repo-${RUN_SUFFIX}
|
|
# Add a git-checkout resource with a run-unique name
|
|
${res_name}= Set Variable m6-setup-res-${RUN_SUFFIX}
|
|
${add_res}= Run CleverAgents Command resource add git-checkout ${res_name} --path ${repo_path}
|
|
Should Be Equal As Integers ${add_res.rc} 0
|
|
Output Should Contain ${add_res} ${res_name}
|
|
# Create a project linked to the resource
|
|
${proj_name}= Set Variable m6-setup-proj-${RUN_SUFFIX}
|
|
${create_proj}= Run CleverAgents Command project create --resource ${res_name} ${proj_name}
|
|
Should Be Equal As Integers ${create_proj.rc} 0
|
|
Output Should Contain ${create_proj} ${proj_name}
|
|
# Verify project appears in the list (use --format json to avoid Rich table truncation)
|
|
${list_proj}= Run CleverAgents Command project list --format json
|
|
Should Be Equal As Integers ${list_proj.rc} 0
|
|
Output Should Contain ${list_proj} ${proj_name}
|
|
|
|
M6 E2E Guard Enforcement Denylist Budget Limits
|
|
[Documentation] Register a custom profile with explicit guards (denylist, budget,
|
|
... tool-call limits) and verify the guard fields are stored correctly.
|
|
... Covers AC-4: "Test verifies guard enforcement (denylist, budget
|
|
... caps, tool call limits)".
|
|
[Teardown] Run Keyword And Ignore Error Run CleverAgents Command automation-profile remove ${guard_profile_name} --yes expected_rc=None
|
|
Set Test Variable ${guard_profile_name} ${EMPTY}
|
|
${guard_profile_name}= Set Variable e2e-guard-${RUN_SUFFIX}
|
|
Set Test Variable ${guard_profile_name}
|
|
# Build YAML config for a custom profile with all three guard categories
|
|
${yaml}= Catenate SEPARATOR=\n
|
|
... name: ${guard_profile_name}
|
|
... description: E2E test profile with denylist budget and tool-call limits
|
|
... schema_version: "1.0"
|
|
... decompose_task: 0.5
|
|
... create_tool: 0.5
|
|
... select_tool: 1.0
|
|
... safety:
|
|
... ${SPACE}${SPACE}require_sandbox: true
|
|
... ${SPACE}${SPACE}require_checkpoints: true
|
|
... ${SPACE}${SPACE}max_cost_per_plan: 50.0
|
|
... ${SPACE}${SPACE}max_retries_per_step: 3
|
|
... guards:
|
|
... ${SPACE}${SPACE}max_tool_calls_per_step: 10
|
|
... ${SPACE}${SPACE}max_total_cost: 25.0
|
|
... ${SPACE}${SPACE}tool_denylist:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}- shell_exec
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}- rm_rf
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}- drop_database
|
|
... ${SPACE}${SPACE}require_approval_for_writes: true
|
|
... ${SPACE}${SPACE}require_approval_for_apply: true
|
|
${yaml_path}= Set Variable ${SUITE_HOME}${/}guard-profile.yaml
|
|
Create File ${yaml_path} ${yaml}
|
|
# Register the custom profile
|
|
${add_result}= Run CleverAgents Command automation-profile add --config ${yaml_path} --format json
|
|
Should Be Equal As Integers ${add_result.rc} 0
|
|
Output Should Contain ${add_result} ${guard_profile_name}
|
|
# Show the profile and verify guard fields
|
|
${show_result}= Run CleverAgents Command automation-profile show ${guard_profile_name} --format json
|
|
Should Be Equal As Integers ${show_result.rc} 0
|
|
# Verify denylist entries
|
|
Output Should Contain ${show_result} tool_denylist
|
|
Output Should Contain ${show_result} shell_exec
|
|
Output Should Contain ${show_result} rm_rf
|
|
Output Should Contain ${show_result} drop_database
|
|
# Verify budget cap
|
|
Output Should Contain ${show_result} max_total_cost
|
|
# Verify tool call limit
|
|
Output Should Contain ${show_result} max_tool_calls_per_step
|
|
# Verify approval requirements
|
|
Output Should Contain ${show_result} require_approval_for_writes
|
|
Output Should Contain ${show_result} require_approval_for_apply
|
|
|
|
M6 E2E Plan Lifecycle Via CLI
|
|
[Documentation] Test A2A facade plan lifecycle: create plan via action, list, and status.
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj_name}= Setup Plan Test Resources plan-lc
|
|
# Use an action to create a plan with automation profile
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile ci --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
Plan Lifecycle Assertions ${plan_use}
|
|
|
|
M6 E2E Guard Enforcement Via Profile
|
|
[Documentation] Verify guard enforcement with a strict automation profile.
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj_name}= Setup Plan Test Resources guard
|
|
# Attempt to use an action; with manual profile guards should constrain autonomy
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile manual --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
Guard Enforcement Assertions ${plan_use} expected_profile=manual
|
|
|
|
M6 E2E Profile Precedence Plan Overrides Global
|
|
[Documentation] Verify plan-level automation profile overrides global config.
|
|
[Tags] tdd_issue tdd_issue_4196
|
|
... Sets global profile to "review", then creates a plan with
|
|
... explicit --automation-profile trusted. The plan output must
|
|
... show "trusted" (not "review"), proving plan > global precedence.
|
|
... Covers AC-5: "Test verifies automation profile resolution
|
|
... precedence (plan > action > global)".
|
|
... Note: action > global precedence requires the action's
|
|
... automation_profile to propagate to the Plan during plan-use,
|
|
... which is not yet wired in PlanLifecycleService.use_action;
|
|
... this test covers plan > global.
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
# Set global profile to "review"
|
|
${set_global}= Run CleverAgents Command config set core.automation-profile review --format json
|
|
Should Be Equal As Integers ${set_global.rc} 0
|
|
${get_global}= Run CleverAgents Command config get core.automation-profile --format json
|
|
${global_value}= Safe Parse Json Field ${get_global.stdout} value
|
|
Should Be Equal As Strings ${global_value} review
|
|
# Setup resources and project
|
|
${proj_name}= Setup Plan Test Resources prec
|
|
# Create plan with EXPLICIT "trusted" profile (should override "review" global)
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile trusted --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
# Parse automation_profile from plan output: must be "trusted", NOT "review"
|
|
${resolved_profile}= Safe Parse Json Field ${plan_use.stdout} automation_profile
|
|
IF '${resolved_profile}' != ''
|
|
Should Be Equal As Strings ${resolved_profile} trusted
|
|
Should Not Be Equal As Strings ${resolved_profile} review
|
|
Log Precedence confirmed: plan-level "trusted" overrides global "review"
|
|
ELSE
|
|
# Fallback: check combined output for "trusted" profile name
|
|
Output Should Contain ${plan_use} trusted
|
|
END
|
|
|
|
M6 E2E Event Queue Via Plan Lifecycle Transitions
|
|
[Documentation] Exercise event queue publish/subscribe through plan lifecycle.
|
|
[Tags] tdd_issue tdd_issue_4189
|
|
... Each plan operation (create, execute) publishes domain events
|
|
... (PLAN_CREATED, PHASE_TRANSITION) consumed by event handlers that
|
|
... update plan state. Verifying state transitions via CLI proves
|
|
... the event bus delivered and processed the published events.
|
|
... Covers AC-3: "Test exercises event queue publish/subscribe via
|
|
... real CLI".
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj_name}= Setup Plan Test Resources evq
|
|
# 1. Create plan — publishes PLAN_CREATED on the domain event bus
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile ci --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
|
# 2. Verify plan appears in list (proves creation event was processed)
|
|
Verify Plan In List ${plan_id}
|
|
# 3. Capture initial plan state via status
|
|
${status1}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${status1.rc} 0 msg=plan status failed (rc=${status1.rc}): ${status1.stderr}
|
|
${initial_phase}= Safe Parse Json Field ${status1.stdout} phase
|
|
${initial_state}= Safe Parse Json Field ${status1.stdout} processing_state
|
|
Log Initial phase=${initial_phase} processing_state=${initial_state}
|
|
# 4. Execute plan — triggers PHASE_TRANSITION and execution events
|
|
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
|
|
# P0-3: Hard assertions on SUCCESS path; LLM execution may legitimately fail
|
|
# (e.g. strategize phase not yet complete — no background worker in E2E).
|
|
IF ${execute.rc} == 0
|
|
Output Should Contain ${execute} ${plan_id}
|
|
# 5. Re-check status — should reflect post-execute state
|
|
${status2}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${status2.rc} 0 msg=plan status failed after execute (rc=${status2.rc}): ${status2.stderr}
|
|
${post_phase}= Safe Parse Json Field ${status2.stdout} phase
|
|
${post_state}= Safe Parse Json Field ${status2.stdout} processing_state
|
|
Log Post-execute phase=${post_phase} processing_state=${post_state}
|
|
# P0-3: Hard assertion — at least one state field must be populated,
|
|
# proving the event bus delivered and processed lifecycle events.
|
|
${state_populated}= Evaluate '${post_phase}' != '' or '${post_state}' != ''
|
|
Should Be True ${state_populated}
|
|
... Event queue should process lifecycle events — expected non-empty phase or processing_state after execute
|
|
ELSE
|
|
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
|
|
END
|
|
# 6. Final: plan should still appear in list
|
|
Verify Plan In List ${plan_id}
|
|
|
|
M6 E2E Hierarchical Decomposition Via Plan Tree
|
|
[Documentation] Verify hierarchical plan decomposition via the plan tree command.
|
|
[Tags] tdd_issue tdd_issue_4189
|
|
... After plan creation and execution, ``plan tree`` should return
|
|
... a decision tree with at least root-level decisions. If the LLM
|
|
... decomposes the task into subplans the tree will show nested
|
|
... children with hierarchy indicators.
|
|
... Covers AC-6: "Test exercises a full autonomy acceptance flow
|
|
... with hierarchical decomposition".
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj_name}= Setup Plan Test Resources tree
|
|
# Create plan with full-auto profile (enables install_dependency for decomposition)
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile full-auto --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
|
# Execute plan to populate the decision tree with strategy/execution decisions
|
|
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
|
|
IF ${execute.rc} != 0
|
|
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
|
|
END
|
|
# View decision tree — must contain at least root-level decisions
|
|
${tree}= Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=120s
|
|
# P0-4: Hard assertion — tree command must succeed.
|
|
Should Be Equal As Integers ${tree.rc} 0 msg=plan tree failed (rc=${tree.rc}): ${tree.stderr}
|
|
Should Not Be Empty ${tree.stdout} Plan tree output should not be empty
|
|
# P0-4: Hard assertion — at least one decision node must exist after execution.
|
|
${decision_count}= Evaluate $tree.stdout.count('"decision_id"')
|
|
Log Decision tree contains ${decision_count} decision node(s)
|
|
Should Be True ${decision_count} >= 1
|
|
... Plan tree should contain at least one decision node after execution (found ${decision_count})
|
|
# Check for hierarchical children (decomposition infrastructure indicator)
|
|
${has_children_key}= Evaluate '"children"' in $tree.stdout
|
|
IF ${has_children_key}
|
|
Log Tree structure has children field (decomposition infrastructure functional)
|
|
END
|
|
|
|
M6 E2E Full Autonomy Acceptance Flow
|
|
[Documentation] End-to-end: init, resource, project, action, plan use, execute, status, apply.
|
|
[Tags] tdd_issue tdd_issue_4189
|
|
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj_name}= Setup Plan Test Resources flow
|
|
# Use an action to create a plan
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile full-auto --format json expected_rc=None timeout=180s
|
|
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
|
|
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
# Parse plan_id from JSON output
|
|
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
|
# Check plan status
|
|
${status}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${status.rc} 0 msg=plan status failed (rc=${status.rc}): ${status.stderr}
|
|
Output Should Contain ${status} ${plan_id}
|
|
# Execute
|
|
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
|
|
IF ${execute.rc} == 0
|
|
Output Should Contain ${execute} ${plan_id}
|
|
# Verify execute output contains phase-transition indicators
|
|
${exec_phase}= Safe Parse Json Field ${execute.stdout} phase
|
|
Log Execute phase: ${exec_phase}
|
|
# Apply
|
|
Full Flow Apply Step ${plan_id}
|
|
ELSE
|
|
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
|
|
END
|
|
# Final verification: list should show the plan
|
|
Verify Plan In List ${plan_id}
|