From 90215d2a6413cb6494d4d4d3c836c93279742b6e Mon Sep 17 00:00:00 2001 From: "Brent E. Edwards" Date: Thu, 2 Apr 2026 16:58:37 +0000 Subject: [PATCH] test(e2e): update m1_acceptance.robot (#1260) Co-authored-by: Brent E. Edwards Co-committed-by: Brent E. Edwards --- robot/e2e/m1_acceptance.robot | 122 ++++++++++++++++++++++++++++------ 1 file changed, 101 insertions(+), 21 deletions(-) diff --git a/robot/e2e/m1_acceptance.robot b/robot/e2e/m1_acceptance.robot index 014c41baa..fbf1beae3 100644 --- a/robot/e2e/m1_acceptance.robot +++ b/robot/e2e/m1_acceptance.robot @@ -8,6 +8,12 @@ Documentation E2E acceptance test for M1 — minimal plan execution flow. ... Flow: action create → resource add → project create → plan use ... → plan execute (strategize) → plan execute (execute) → plan diff ... → plan apply. +... +... Tagged ``tdd_expected_fail`` because ``_get_plan_executor()`` +... does not pass ``sandbox_root`` to ``PlanExecutor``, so +... ``LLMExecuteActor._write_to_sandbox()`` is never called and +... LLM-generated files (HELLO.md) are silently discarded. +... See `#1313 `_. Resource common_e2e.resource Suite Setup E2E Suite Setup Suite Teardown E2E Suite Teardown @@ -19,7 +25,7 @@ M1 Full Plan Lifecycle ... Creates an action from YAML, registers a git-checkout ... resource, creates a project, and runs the full plan ... lifecycle through apply with post-apply commit verification. - [Tags] E2E + [Tags] E2E tdd_expected_fail tdd_issue tdd_issue_1313 # ── 1. Create a temporary git repo for isolation ────────────── ${repo_path}= Create Temp Git Repo m1-acceptance-repo @@ -40,85 +46,159 @@ M1 Full Plan Lifecycle # ── 3. Create the action ───────────────────────────────────── ${action_result}= Run CleverAgents Command ... action create --config ${action_yaml} + ... --format plain + Should Be Equal As Integers ${action_result.rc} 0 Output Should Contain ${action_result} local/test-action + Output Should Contain ${action_result} M1 E2E acceptance test action + Output Should Contain ${action_result} State: available # ── 4. Register git-checkout resource ──────────────────────── ${resource_result}= Run CleverAgents Command ... resource add git-checkout local/test-repo - ... --path ${repo_path} --branch master + ... --path ${repo_path} --branch master + ... --format plain Should Be Equal As Integers ${resource_result.rc} 0 ... Resource add failed: ${resource_result.stderr} + Output Should Contain ${resource_result} name: local/test-repo + Output Should Contain ${resource_result} type: git-checkout # ── 5. Create project linked to resource ───────────────────── ${project_result}= Run CleverAgents Command ... project create --resource local/test-repo local/test-project + ... --format plain Should Be Equal As Integers ${project_result.rc} 0 ... Project create failed: ${project_result.stderr} + Output Should Contain ${project_result} namespaced_name: local/test-project + Output Should Contain ${project_result} linked_resources: # ── 6. Plan use — create plan from action + project ────────── ${plan_use_result}= Run CleverAgents Command ... plan use local/test-action local/test-project ... --format plain expected_rc=${0} - Should Be Equal As Integers ${plan_use_result.rc} 0 - ... Plan use failed: ${plan_use_result.stderr} + # Extract plan ID from output (ULID pattern: 26 alphanumeric chars) ${plan_id}= Extract Plan Id ${plan_use_result.stdout} Should Not Be Empty ${plan_id} Could not extract plan ID from plan use output + Should Be Equal As Integers ${plan_use_result.rc} 0 + ... Plan use failed: ${plan_use_result.stderr} + Output Should Contain ${plan_use_result} phase: strategize + Output Should Contain ${plan_use_result} action_name: local/test-action + Output Should Contain ${plan_use_result} automation_profile: manual + Output Should Contain ${plan_use_result} description: M1 E2E acceptance test action + Output Should Contain ${plan_use_result} definition_of_done: Create a file called HELLO.md with a short greeting. + Output Should Contain ${plan_use_result} strategy_actor: openai/gpt-4o-mini + Output Should Contain ${plan_use_result} execution_actor: openai/gpt-4o-mini + Output Should Contain ${plan_use_result} estimation_actor: None + Output Should Contain ${plan_use_result} invariant_actor: None + # ── 7. Plan execute — strategize phase ─────────────────────── ${exec1_result}= Run CleverAgents Command - ... plan execute ${plan_id} + ... plan execute ${plan_id} --format plain ... timeout=300s + Log Strategize execute rc=${exec1_result.rc} - # Strategize may succeed or the plan may need processing first + # Strategize must succeed (rc=0 enforced) ${exec1_combined}= Set Variable ${exec1_result.stdout}\n${exec1_result.stderr} Log Strategize output: ${exec1_combined} + Should Be Equal As Integers ${exec1_result.rc} 0 + Output Should Contain ${exec1_result} action_name: local/test-action + Output Should Contain ${exec1_result} phase: execute + Output Should Contain ${exec1_result} processing_state: complete + Output Should Contain ${exec1_result} project_links: + Output Should Contain ${exec1_result} local/test-project + Output Should Contain ${exec1_result} description: M1 E2E acceptance test action + Output Should Contain ${exec1_result} strategy_actor: openai/gpt-4o-mini + Output Should Contain ${exec1_result} execution_actor: openai/gpt-4o-mini + Output Should Contain ${exec1_result} definition_of_done: Create a file called HELLO.md with a short greeting. + # ── 8. Plan execute — advance to execute phase ─────────────── ${exec2_result}= Run CleverAgents Command - ... plan execute ${plan_id} + ... plan execute ${plan_id} --format plain ... timeout=300s + Log Execute phase rc=${exec2_result.rc} ${exec2_combined}= Set Variable ${exec2_result.stdout}\n${exec2_result.stderr} Log Execute output: ${exec2_combined} - # ── 9. Plan diff — verify changeset exists ─────────────────── + # Output matches step 7: both report the plan's current state after transition + Should Be Equal As Integers ${exec2_result.rc} 0 + Output Should Contain ${exec2_result} action_name: local/test-action + Output Should Contain ${exec2_result} phase: execute + Output Should Contain ${exec2_result} processing_state: complete + Output Should Contain ${exec2_result} project_links: + Output Should Contain ${exec2_result} local/test-project + Output Should Contain ${exec2_result} description: M1 E2E acceptance test action + Output Should Contain ${exec2_result} strategy_actor: openai/gpt-4o-mini + Output Should Contain ${exec2_result} execution_actor: openai/gpt-4o-mini + Output Should Contain ${exec2_result} definition_of_done: Create a file called HELLO.md with a short greeting. + + # ── 9. Plan diff — verify changeset contains HELLO.md ────────── ${diff_result}= Run CleverAgents Command - ... plan diff ${plan_id} + ... plan diff ${plan_id} --format plain ... timeout=120s + Log Diff rc=${diff_result.rc} ${diff_combined}= Set Variable ${diff_result.stdout}\n${diff_result.stderr} Log Diff output: ${diff_combined} - # Diff succeeded (rc=0 enforced above); verify it produced output + + Should Be Equal As Integers ${diff_result.rc} 0 Should Not Be Empty ${diff_result.stdout} ... Plan diff produced no output + # The LLM was asked to create HELLO.md — the diff must show it. + # This assertion will fail until #1313 is resolved (sandbox_root + # is not wired into the execute pipeline). + Output Should Contain ${diff_result} HELLO # ── 10. Plan apply — apply changes to the repo ─────────────── ${apply_result}= Run CleverAgents Command - ... plan apply --yes ${plan_id} + ... plan apply --yes --format plain ${plan_id} ... timeout=300s Log Apply rc=${apply_result.rc} ${apply_combined}= Set Variable ${apply_result.stdout}\n${apply_result.stderr} Log Apply output: ${apply_combined} - # ── 11. Verify post-apply commit in target repo ────────────── - ${git_log}= Run Process git log -1 --oneline + Should Be Equal As Integers ${apply_result.rc} 0 + Output Should Contain ${apply_result} phase: apply + Output Should Contain ${apply_result} processing_state: applied + Output Should Contain ${apply_result} state: applied + Output Should Contain ${apply_result} project_links: + Output Should Contain ${apply_result} - {"project_name": "local/test-project"} + Output Should Contain ${apply_result} arguments: + Output Should Contain ${apply_result} automation_profile: manual + Output Should Contain ${apply_result} action_name: local/test-action + Output Should Contain ${apply_result} description: M1 E2E acceptance test action + Output Should Contain ${apply_result} definition_of_done: Create a file called HELLO.md with a short greeting. + Output Should Contain ${apply_result} strategy_actor: openai/gpt-4o-mini + Output Should Contain ${apply_result} execution_actor: openai/gpt-4o-mini + Output Should Contain ${apply_result} estimation_actor: None + Output Should Contain ${apply_result} invariant_actor: None + + # ── 11. Verify post-apply commit and HELLO.md in target repo ── + ${git_log}= Run Process git log --oneline ... cwd=${repo_path} timeout=60s on_timeout=kill Log Git log after apply: ${git_log.stdout} - # The repo should have at least the initial commit; if apply worked - # there will be a second commit from CleverAgents - Should Not Be Empty ${git_log.stdout} - ... No commits found in target repo after apply + Should Be Equal As Integers ${git_log.rc} 0 + # After apply there must be at least 2 commits: the initial commit + # from Create Temp Git Repo plus the CleverAgents apply commit. + ${commit_lines}= Get Line Count ${git_log.stdout} + Should Be True ${commit_lines} >= 2 + ... Expected at least 2 commits after apply, found ${commit_lines} + + # HELLO.md must exist in the repo after apply. + # This assertion will fail until #1313 is resolved. + File Should Exist ${repo_path}${/}HELLO.md # ── 12. Structural validation summary ──────────────────────── # Action create succeeded (rc=0 verified above) # Resource add succeeded (rc=0 verified above) # Project create succeeded (rc=0 verified above) # Plan use created a plan with a valid ID (verified above) - # Execute steps were attempted (logged above) - # Diff was attempted (logged above) - # Apply was attempted (logged above) - # Git repo still has commits (verified above) + # Execute phases completed (rc=0, output validated above) + # Diff showed HELLO.md changes (validated above) + # Apply succeeded and produced a commit (validated above) + # HELLO.md exists in target repo (validated above) Log M1 Full Plan Lifecycle E2E test completed successfully *** Keywords ***