test(e2e): update m1_acceptance.robot
CI / build (pull_request) Successful in 16s
CI / helm (pull_request) Successful in 20s
CI / lint (pull_request) Successful in 3m19s
CI / quality (pull_request) Successful in 3m44s
CI / typecheck (pull_request) Successful in 3m56s
CI / security (pull_request) Successful in 4m6s
CI / unit_tests (pull_request) Successful in 8m53s
CI / docker (pull_request) Successful in 1m23s
CI / coverage (pull_request) Successful in 12m40s
CI / e2e_tests (pull_request) Successful in 19m11s
CI / integration_tests (pull_request) Successful in 24m38s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 55m0s

Added return code checks and output validation for all CLI steps
in the M1 acceptance test. Each step now verifies rc=0 and checks
expected output strings against stdout/stderr.

Steps updated: action create, resource add git-checkout, project create,
plan use, plan execute (strategize), plan execute (execute), plan diff,
plan apply, and post-apply git log verification.

ISSUES CLOSED: #1249
This commit is contained in:
2026-04-02 04:19:58 +00:00
parent c937abdc09
commit bfb3286a35
+101 -21
View File
@@ -8,6 +8,12 @@ Documentation E2E acceptance test for M1 — minimal plan execution flow.
... Flow: action create → resource add → project create → plan use
... → plan execute (strategize) → plan execute (execute) → plan diff
... → plan apply.
...
... Tagged ``tdd_expected_fail`` because ``_get_plan_executor()``
... does not pass ``sandbox_root`` to ``PlanExecutor``, so
... ``LLMExecuteActor._write_to_sandbox()`` is never called and
... LLM-generated files (HELLO.md) are silently discarded.
... See `#1313 <https://git.cleverthis.com/cleveragents/cleveragents-core/issues/1313>`_.
Resource common_e2e.resource
Suite Setup E2E Suite Setup
Suite Teardown E2E Suite Teardown
@@ -19,7 +25,7 @@ M1 Full Plan Lifecycle
... Creates an action from YAML, registers a git-checkout
... resource, creates a project, and runs the full plan
... lifecycle through apply with post-apply commit verification.
[Tags] E2E
[Tags] E2E tdd_expected_fail tdd_issue tdd_issue_1313
# ── 1. Create a temporary git repo for isolation ──────────────
${repo_path}= Create Temp Git Repo m1-acceptance-repo
@@ -40,85 +46,159 @@ M1 Full Plan Lifecycle
# ── 3. Create the action ─────────────────────────────────────
${action_result}= Run CleverAgents Command
... action create --config ${action_yaml}
... --format plain
Should Be Equal As Integers ${action_result.rc} 0
Output Should Contain ${action_result} local/test-action
Output Should Contain ${action_result} M1 E2E acceptance test action
Output Should Contain ${action_result} State: available
# ── 4. Register git-checkout resource ────────────────────────
${resource_result}= Run CleverAgents Command
... resource add git-checkout local/test-repo
... --path ${repo_path} --branch master
... --path ${repo_path} --branch master
... --format plain
Should Be Equal As Integers ${resource_result.rc} 0
... Resource add failed: ${resource_result.stderr}
Output Should Contain ${resource_result} name: local/test-repo
Output Should Contain ${resource_result} type: git-checkout
# ── 5. Create project linked to resource ─────────────────────
${project_result}= Run CleverAgents Command
... project create --resource local/test-repo local/test-project
... --format plain
Should Be Equal As Integers ${project_result.rc} 0
... Project create failed: ${project_result.stderr}
Output Should Contain ${project_result} namespaced_name: local/test-project
Output Should Contain ${project_result} linked_resources:
# ── 6. Plan use — create plan from action + project ──────────
${plan_use_result}= Run CleverAgents Command
... plan use local/test-action local/test-project
... --format plain expected_rc=${0}
Should Be Equal As Integers ${plan_use_result.rc} 0
... Plan use failed: ${plan_use_result.stderr}
# Extract plan ID from output (ULID pattern: 26 alphanumeric chars)
${plan_id}= Extract Plan Id ${plan_use_result.stdout}
Should Not Be Empty ${plan_id} Could not extract plan ID from plan use output
Should Be Equal As Integers ${plan_use_result.rc} 0
... Plan use failed: ${plan_use_result.stderr}
Output Should Contain ${plan_use_result} phase: strategize
Output Should Contain ${plan_use_result} action_name: local/test-action
Output Should Contain ${plan_use_result} automation_profile: manual
Output Should Contain ${plan_use_result} description: M1 E2E acceptance test action
Output Should Contain ${plan_use_result} definition_of_done: Create a file called HELLO.md with a short greeting.
Output Should Contain ${plan_use_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${plan_use_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${plan_use_result} estimation_actor: None
Output Should Contain ${plan_use_result} invariant_actor: None
# ── 7. Plan execute — strategize phase ───────────────────────
${exec1_result}= Run CleverAgents Command
... plan execute ${plan_id}
... plan execute ${plan_id} --format plain
... timeout=300s
Log Strategize execute rc=${exec1_result.rc}
# Strategize may succeed or the plan may need processing first
# Strategize must succeed (rc=0 enforced)
${exec1_combined}= Set Variable ${exec1_result.stdout}\n${exec1_result.stderr}
Log Strategize output: ${exec1_combined}
Should Be Equal As Integers ${exec1_result.rc} 0
Output Should Contain ${exec1_result} action_name: local/test-action
Output Should Contain ${exec1_result} phase: execute
Output Should Contain ${exec1_result} processing_state: complete
Output Should Contain ${exec1_result} project_links:
Output Should Contain ${exec1_result} local/test-project
Output Should Contain ${exec1_result} description: M1 E2E acceptance test action
Output Should Contain ${exec1_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${exec1_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${exec1_result} definition_of_done: Create a file called HELLO.md with a short greeting.
# ── 8. Plan execute — advance to execute phase ───────────────
${exec2_result}= Run CleverAgents Command
... plan execute ${plan_id}
... plan execute ${plan_id} --format plain
... timeout=300s
Log Execute phase rc=${exec2_result.rc}
${exec2_combined}= Set Variable ${exec2_result.stdout}\n${exec2_result.stderr}
Log Execute output: ${exec2_combined}
# ── 9. Plan diff — verify changeset exists ───────────────────
# Output matches step 7: both report the plan's current state after transition
Should Be Equal As Integers ${exec2_result.rc} 0
Output Should Contain ${exec2_result} action_name: local/test-action
Output Should Contain ${exec2_result} phase: execute
Output Should Contain ${exec2_result} processing_state: complete
Output Should Contain ${exec2_result} project_links:
Output Should Contain ${exec2_result} local/test-project
Output Should Contain ${exec2_result} description: M1 E2E acceptance test action
Output Should Contain ${exec2_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${exec2_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${exec2_result} definition_of_done: Create a file called HELLO.md with a short greeting.
# ── 9. Plan diff — verify changeset contains HELLO.md ──────────
${diff_result}= Run CleverAgents Command
... plan diff ${plan_id}
... plan diff ${plan_id} --format plain
... timeout=120s
Log Diff rc=${diff_result.rc}
${diff_combined}= Set Variable ${diff_result.stdout}\n${diff_result.stderr}
Log Diff output: ${diff_combined}
# Diff succeeded (rc=0 enforced above); verify it produced output
Should Be Equal As Integers ${diff_result.rc} 0
Should Not Be Empty ${diff_result.stdout}
... Plan diff produced no output
# The LLM was asked to create HELLO.md — the diff must show it.
# This assertion will fail until #1313 is resolved (sandbox_root
# is not wired into the execute pipeline).
Output Should Contain ${diff_result} HELLO
# ── 10. Plan apply — apply changes to the repo ───────────────
${apply_result}= Run CleverAgents Command
... plan apply --yes ${plan_id}
... plan apply --yes --format plain ${plan_id}
... timeout=300s
Log Apply rc=${apply_result.rc}
${apply_combined}= Set Variable ${apply_result.stdout}\n${apply_result.stderr}
Log Apply output: ${apply_combined}
# ── 11. Verify post-apply commit in target repo ──────────────
${git_log}= Run Process git log -1 --oneline
Should Be Equal As Integers ${apply_result.rc} 0
Output Should Contain ${apply_result} phase: apply
Output Should Contain ${apply_result} processing_state: applied
Output Should Contain ${apply_result} state: applied
Output Should Contain ${apply_result} project_links:
Output Should Contain ${apply_result} - {"project_name": "local/test-project"}
Output Should Contain ${apply_result} arguments:
Output Should Contain ${apply_result} automation_profile: manual
Output Should Contain ${apply_result} action_name: local/test-action
Output Should Contain ${apply_result} description: M1 E2E acceptance test action
Output Should Contain ${apply_result} definition_of_done: Create a file called HELLO.md with a short greeting.
Output Should Contain ${apply_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${apply_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${apply_result} estimation_actor: None
Output Should Contain ${apply_result} invariant_actor: None
# ── 11. Verify post-apply commit and HELLO.md in target repo ──
${git_log}= Run Process git log --oneline
... cwd=${repo_path} timeout=60s on_timeout=kill
Log Git log after apply: ${git_log.stdout}
# The repo should have at least the initial commit; if apply worked
# there will be a second commit from CleverAgents
Should Not Be Empty ${git_log.stdout}
... No commits found in target repo after apply
Should Be Equal As Integers ${git_log.rc} 0
# After apply there must be at least 2 commits: the initial commit
# from Create Temp Git Repo plus the CleverAgents apply commit.
${commit_lines}= Get Line Count ${git_log.stdout}
Should Be True ${commit_lines} >= 2
... Expected at least 2 commits after apply, found ${commit_lines}
# HELLO.md must exist in the repo after apply.
# This assertion will fail until #1313 is resolved.
File Should Exist ${repo_path}${/}HELLO.md
# ── 12. Structural validation summary ────────────────────────
# Action create succeeded (rc=0 verified above)
# Resource add succeeded (rc=0 verified above)
# Project create succeeded (rc=0 verified above)
# Plan use created a plan with a valid ID (verified above)
# Execute steps were attempted (logged above)
# Diff was attempted (logged above)
# Apply was attempted (logged above)
# Git repo still has commits (verified above)
# Execute phases completed (rc=0, output validated above)
# Diff showed HELLO.md changes (validated above)
# Apply succeeded and produced a commit (validated above)
# HELLO.md exists in target repo (validated above)
Log M1 Full Plan Lifecycle E2E test completed successfully
*** Keywords ***