HAL9000
380d0a737a
All 9 test cases previously only called `project create` + `project show` despite documentation claiming to test correction, plan execution, and subplan spawning workflows. This commit fixes that. Changes per reviewer blockers: Blocker 1 (hollow stubs): Each test now invokes the workflow its name and documentation describe: - test_correction_workflow.robot: calls `plan correct --mode revert/append` (with --dry-run for mode tests; actual correction for state transition test) - test_project_plan_workflow.robot: Plan Execution Workflow now calls `plan use` + `plan execute` + `plan status`; Project Creation Workflow uses resource-based project creation and verifies list output - test_subplan_workflow.robot: Subplan Spawning and Merge Result Validation inspect `plan tree` for spec-required child_plans/decision_ids fields; Three-Way Merge Workflow exercises `plan apply` Blocker 2 (temp dir leaks): All tests now use `Create Temp Git Repo` (creates inside SUITE_HOME, cleaned by E2E Suite Teardown) instead of `Create Temp Directory` (tempfile.mkdtemp outside SUITE_HOME, leaked). The unused `Set Up E2E Project Test` local keyword (which created an orphaned suite-variable temp dir) is removed from all three files. Each file gains a proper suite setup keyword that registers the local/code-review action needed by plan lifecycle tests, following the pattern established in m6_acceptance.robot.
145 lines
8.5 KiB
Plaintext
145 lines
8.5 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E workflow test for subplan spawning and merge infrastructure.
|
|
...
|
|
... Exercises subplan and merge workflows via plan tree inspection:
|
|
... 1. Subplan Spawning - verifies plan tree contains child_plans field
|
|
... after plan execution, confirming subplan spawning infrastructure
|
|
... 2. Three-Way Merge - verifies plan apply runs after execution,
|
|
... exercising the plan finalisation / merge path
|
|
... 3. Merge Result Validation - verifies plan tree output conforms to
|
|
... the spec-required envelope (command, data, child_plans, decision_ids)
|
|
...
|
|
... Requires real LLM API keys (ANTHROPIC_API_KEY or OPENAI_API_KEY).
|
|
... Tests are skipped automatically when no keys are available.
|
|
Resource common_e2e.resource
|
|
Suite Setup Subplan Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
Force Tags E2E
|
|
|
|
*** Keywords ***
|
|
Subplan Suite Setup
|
|
[Documentation] E2E Suite Setup plus action registration.
|
|
E2E Suite Setup
|
|
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
|
|
Set Suite Variable ${RUN_SUFFIX} ${suffix}
|
|
${actor}= Resolve LLM Actor
|
|
${yaml}= Catenate SEPARATOR=\n
|
|
... name: local/code-review
|
|
... description: "Perform a code review on project sources"
|
|
... strategy_actor: ${actor}
|
|
... execution_actor: ${actor}
|
|
... definition_of_done: "Code review completed."
|
|
... reusable: true
|
|
... read_only: false
|
|
... state: available
|
|
${yaml_path}= Set Variable ${SUITE_HOME}${/}code-review-action.yaml
|
|
Create File ${yaml_path} ${yaml}
|
|
${reg}= Run CleverAgents Command action create --config ${yaml_path} expected_rc=None
|
|
IF ${reg.rc} != 0
|
|
Log Could not register local/code-review action: ${reg.stderr} WARN
|
|
END
|
|
|
|
Setup Subplan Resources
|
|
[Documentation] Create git repo resource and project for subplan tests.
|
|
... Resources are created inside SUITE_HOME and cleaned up
|
|
... by E2E Suite Teardown automatically.
|
|
[Arguments] ${prefix}
|
|
${repo}= Create Temp Git Repo ${prefix}-repo-${RUN_SUFFIX}
|
|
${res}= Set Variable ${prefix}-res-${RUN_SUFFIX}
|
|
${add}= Run CleverAgents Command resource add git-checkout ${res} --path ${repo}
|
|
Should Be Equal As Integers ${add.rc} 0
|
|
${proj}= Set Variable ${prefix}-proj-${RUN_SUFFIX}
|
|
${create}= Run CleverAgents Command project create --resource ${res} ${proj}
|
|
Should Be Equal As Integers ${create.rc} 0
|
|
RETURN ${proj}
|
|
|
|
*** Test Cases ***
|
|
|
|
Subplan Spawning Workflow
|
|
[Documentation] Test subplan spawning infrastructure via plan tree.
|
|
...
|
|
... Creates a project, generates a plan via action, executes it, then
|
|
... inspects plan tree to confirm the spec-required child_plans field
|
|
... is present. This verifies the subplan spawning infrastructure is
|
|
... functional: the field is always present whether or not the LLM
|
|
... chose to decompose the task into child subplans.
|
|
[Tags] subplan-spawn
|
|
[Timeout] 30 minutes
|
|
[Teardown] Run Keyword And Ignore Error Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj}= Setup Subplan Resources sp-spawn
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj} --automation-profile full-auto --format json expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
|
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
|
|
IF ${execute.rc} != 0
|
|
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
|
|
END
|
|
${tree}= Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${tree.rc} 0 msg=plan tree failed: ${tree.stderr}
|
|
Should Not Be Empty ${tree.stdout} msg=plan tree returned empty output
|
|
${has_child_plans}= Evaluate '"child_plans"' in $tree.stdout
|
|
Should Be True ${has_child_plans} Plan tree output should contain spec-required child_plans field
|
|
|
|
Three-Way Merge Workflow
|
|
[Documentation] Test plan apply path which exercises merge logic.
|
|
...
|
|
... Creates a project, generates a plan, executes it, then attempts
|
|
... plan apply. The apply step exercises the plan finalisation and
|
|
... merge path, verifying that completed execution can transition to
|
|
... the apply phase without error.
|
|
[Tags] three-way-merge
|
|
[Timeout] 30 minutes
|
|
[Teardown] Run Keyword And Ignore Error Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj}= Setup Subplan Resources sp-merge
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj} --automation-profile full-auto --format json expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
|
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
|
|
IF ${execute.rc} != 0
|
|
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
|
|
END
|
|
${apply}= Run CleverAgents Command plan apply --yes ${plan_id} --format json expected_rc=None timeout=180s
|
|
IF ${apply.rc} == 0
|
|
Output Should Contain ${apply} ${plan_id}
|
|
ELSE
|
|
Fail plan apply failed (rc=${apply.rc}) stdout=${apply.stdout} stderr=${apply.stderr}
|
|
END
|
|
|
|
Merge Result Validation
|
|
[Documentation] Test plan tree output conforms to spec-required envelope.
|
|
...
|
|
... Creates a project, generates a plan, executes it, then verifies
|
|
... plan tree returns the spec-required envelope fields: command,
|
|
... data, plan_id, child_plans, and decision_ids. These fields
|
|
... provide the structural evidence that merge result data is
|
|
... accessible via the CLI.
|
|
[Tags] merge-validation
|
|
[Timeout] 30 minutes
|
|
[Teardown] Run Keyword And Ignore Error Run CleverAgents Command config set core.automation-profile manual expected_rc=None
|
|
Skip If No LLM Keys
|
|
${proj}= Setup Subplan Resources sp-result
|
|
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj} --automation-profile full-auto --format json expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
|
|
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
|
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
|
|
IF ${execute.rc} != 0
|
|
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
|
|
END
|
|
${tree}= Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${tree.rc} 0 msg=plan tree failed: ${tree.stderr}
|
|
${has_command}= Evaluate '"command"' in $tree.stdout
|
|
Should Be True ${has_command} Plan tree output should contain spec-required "command" envelope key
|
|
${has_data}= Evaluate '"data"' in $tree.stdout
|
|
Should Be True ${has_data} Plan tree output should contain spec-required "data" envelope key
|
|
${has_plan_id}= Evaluate '"plan_id"' in $tree.stdout
|
|
Should Be True ${has_plan_id} Plan tree output should contain "plan_id" in envelope data
|
|
${has_child_plans}= Evaluate '"child_plans"' in $tree.stdout
|
|
Should Be True ${has_child_plans} Plan tree output should contain spec-required "child_plans" field
|
|
${has_decision_ids}= Evaluate '"decision_ids"' in $tree.stdout
|
|
Should Be True ${has_decision_ids} Plan tree output should contain "decision_ids" mapping after execution
|