cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

test(e2e): update m1_acceptance.robot #1260

Merged
freemo merged 1 commits from test/e2e-update-m1-acceptance into master 2026-04-02 16:58:38 +00:00
Conversation 6 Commits 1 Files Changed 1
+101 -21
1 changed files with 101 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
robot/e2e/m1_acceptance.robot
+101 -21
View File
@@ -8,6 +8,12 @@ Documentation E2E acceptance test for M1 — minimal plan execution flow.
... Flow: action create → resource add → project create → plan use
... → plan execute (strategize) → plan execute (execute) → plan diff
... → plan apply.
...
... Tagged ``tdd_expected_fail`` because ``_get_plan_executor()``
... does not pass ``sandbox_root`` to ``PlanExecutor``, so
... ``LLMExecuteActor._write_to_sandbox()`` is never called and
... LLM-generated files (HELLO.md) are silently discarded.
... See `#1313 <https://git.cleverthis.com/cleveragents/cleveragents-core/issues/1313>`_.
Resource common_e2e.resource
Suite Setup E2E Suite Setup
Suite Teardown E2E Suite Teardown
@@ -19,7 +25,7 @@ M1 Full Plan Lifecycle
... Creates an action from YAML, registers a git-checkout
... resource, creates a project, and runs the full plan
... lifecycle through apply with post-apply commit verification.
[Tags] E2E
[Tags] E2E tdd_expected_fail tdd_issue tdd_issue_1313
# ── 1. Create a temporary git repo for isolation ──────────────
${repo_path}= Create Temp Git Repo m1-acceptance-repo
@@ -40,85 +46,159 @@ M1 Full Plan Lifecycle
# ── 3. Create the action ─────────────────────────────────────
${action_result}= Run CleverAgents Command
... action create --config ${action_yaml}
... --format plain
Should Be Equal As Integers ${action_result.rc} 0
Output Should Contain ${action_result} local/test-action
Output Should Contain ${action_result} M1 E2E acceptance test action
Output Should Contain ${action_result} State: available
# ── 4. Register git-checkout resource ────────────────────────
${resource_result}= Run CleverAgents Command
... resource add git-checkout local/test-repo
... --path ${repo_path} --branch master
... --path ${repo_path} --branch master
... --format plain
Should Be Equal As Integers ${resource_result.rc} 0
... Resource add failed: ${resource_result.stderr}
Output Should Contain ${resource_result} name: local/test-repo
Output Should Contain ${resource_result} type: git-checkout
# ── 5. Create project linked to resource ─────────────────────
${project_result}= Run CleverAgents Command
... project create --resource local/test-repo local/test-project
... --format plain
Should Be Equal As Integers ${project_result.rc} 0
... Project create failed: ${project_result.stderr}
Output Should Contain ${project_result} namespaced_name: local/test-project
Output Should Contain ${project_result} linked_resources:
# ── 6. Plan use — create plan from action + project ──────────
${plan_use_result}= Run CleverAgents Command
... plan use local/test-action local/test-project
... --format plain expected_rc=${0}
Should Be Equal As Integers ${plan_use_result.rc} 0
... Plan use failed: ${plan_use_result.stderr}
# Extract plan ID from output (ULID pattern: 26 alphanumeric chars)
${plan_id}= Extract Plan Id ${plan_use_result.stdout}
Should Not Be Empty ${plan_id} Could not extract plan ID from plan use output
freemo commented 2026-04-02 04:45:39 +00:00
Outdated
Review
Copy Link
Copy Source

BUG: Wrong variable reference. This checks ${exec2_result.rc} (the previous execute step's return code) instead of ${diff_result.rc} (the diff step's return code). While it won't crash (exec2_result exists), it's validating the wrong command.

Should be:

Should Be Equal As Integers    ${diff_result.rc}    0
**BUG: Wrong variable reference.** This checks `${exec2_result.rc}` (the previous execute step's return code) instead of `${diff_result.rc}` (the diff step's return code). While it won't crash (exec2_result exists), it's validating the wrong command. Should be: ``` Should Be Equal As Integers ${diff_result.rc} 0 ```
Should Be Equal As Integers ${plan_use_result.rc} 0
... Plan use failed: ${plan_use_result.stderr}
Output Should Contain ${plan_use_result} phase: strategize
Output Should Contain ${plan_use_result} action_name: local/test-action
Output Should Contain ${plan_use_result} automation_profile: manual
Output Should Contain ${plan_use_result} description: M1 E2E acceptance test action
Output Should Contain ${plan_use_result} definition_of_done: Create a file called HELLO.md with a short greeting.
Output Should Contain ${plan_use_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${plan_use_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${plan_use_result} estimation_actor: None
Output Should Contain ${plan_use_result} invariant_actor: None
# ── 7. Plan execute — strategize phase ───────────────────────
${exec1_result}= Run CleverAgents Command
... plan execute ${plan_id}
... plan execute ${plan_id} --format plain
... timeout=300s
Log Strategize execute rc=${exec1_result.rc}
# Strategize may succeed or the plan may need processing first
# Strategize must succeed (rc=0 enforced)
${exec1_combined}= Set Variable ${exec1_result.stdout}\n${exec1_result.stderr}
Log Strategize output: ${exec1_combined}
Should Be Equal As Integers ${exec1_result.rc} 0
Output Should Contain ${exec1_result} action_name: local/test-action
Output Should Contain ${exec1_result} phase: execute
Output Should Contain ${exec1_result} processing_state: complete
Output Should Contain ${exec1_result} project_links:
freemo commented 2026-04-02 04:45:39 +00:00
Outdated
Review
Copy Link
Copy Source

Questionable assertion + debug comments. The ### IMPORTANT QUESTION block should not be in production test code. More importantly, if the LLM successfully creates HELLO.md during execution, then No changes in changeset. would be the wrong expected output — there should be changes. Please resolve this question and either:

  1. Assert that changes ARE present (if LLM execution is expected to produce them), or
  2. Document clearly why no changes are expected at this point in the flow.

Also remove the ### IMPORTANT QUESTION debug comments.

**Questionable assertion + debug comments.** The `### IMPORTANT QUESTION` block should not be in production test code. More importantly, if the LLM successfully creates HELLO.md during execution, then `No changes in changeset.` would be the wrong expected output — there *should* be changes. Please resolve this question and either: 1. Assert that changes ARE present (if LLM execution is expected to produce them), or 2. Document clearly why no changes are expected at this point in the flow. Also remove the `### IMPORTANT QUESTION` debug comments.
Output Should Contain ${exec1_result} local/test-project
Output Should Contain ${exec1_result} description: M1 E2E acceptance test action
Output Should Contain ${exec1_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${exec1_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${exec1_result} definition_of_done: Create a file called HELLO.md with a short greeting.
# ── 8. Plan execute — advance to execute phase ───────────────
${exec2_result}= Run CleverAgents Command
... plan execute ${plan_id}
... plan execute ${plan_id} --format plain
... timeout=300s
Log Execute phase rc=${exec2_result.rc}
${exec2_combined}= Set Variable ${exec2_result.stdout}\n${exec2_result.stderr}
Log Execute output: ${exec2_combined}
# ── 9. Plan diff — verify changeset exists ───────────────────
# Output matches step 7: both report the plan's current state after transition
Should Be Equal As Integers ${exec2_result.rc} 0
Output Should Contain ${exec2_result} action_name: local/test-action
Output Should Contain ${exec2_result} phase: execute
Output Should Contain ${exec2_result} processing_state: complete
Output Should Contain ${exec2_result} project_links:
Output Should Contain ${exec2_result} local/test-project
freemo commented 2026-04-02 04:45:39 +00:00
Outdated
Review
Copy Link
Copy Source

BUG: Wrong variable reference. This checks ${git_log.rc} but ${git_log} is not defined until step 11 (the Run Process git log call below). This will cause a Robot Framework runtime error.

Should be:

Should Be Equal As Integers    ${apply_result.rc}    0
**BUG: Wrong variable reference.** This checks `${git_log.rc}` but `${git_log}` is not defined until step 11 (the `Run Process git log` call below). This will cause a Robot Framework runtime error. Should be: ``` Should Be Equal As Integers ${apply_result.rc} 0 ```
Output Should Contain ${exec2_result} description: M1 E2E acceptance test action
Output Should Contain ${exec2_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${exec2_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${exec2_result} definition_of_done: Create a file called HELLO.md with a short greeting.
# ── 9. Plan diff — verify changeset contains HELLO.md ──────────
${diff_result}= Run CleverAgents Command
... plan diff ${plan_id}
... plan diff ${plan_id} --format plain
... timeout=120s
Log Diff rc=${diff_result.rc}
${diff_combined}= Set Variable ${diff_result.stdout}\n${diff_result.stderr}
Log Diff output: ${diff_combined}
# Diff succeeded (rc=0 enforced above); verify it produced output
Should Be Equal As Integers ${diff_result.rc} 0
Should Not Be Empty ${diff_result.stdout}
... Plan diff produced no output
# The LLM was asked to create HELLO.md — the diff must show it.
# This assertion will fail until #1313 is resolved (sandbox_root
# is not wired into the execute pipeline).
Output Should Contain ${diff_result} HELLO
# ── 10. Plan apply — apply changes to the repo ───────────────
${apply_result}= Run CleverAgents Command
... plan apply --yes ${plan_id}
... plan apply --yes --format plain ${plan_id}
... timeout=300s
Log Apply rc=${apply_result.rc}
${apply_combined}= Set Variable ${apply_result.stdout}\n${apply_result.stderr}
Log Apply output: ${apply_combined}
# ── 11. Verify post-apply commit in target repo ──────────────
${git_log}= Run Process git log -1 --oneline
Should Be Equal As Integers ${apply_result.rc} 0
Output Should Contain ${apply_result} phase: apply
Output Should Contain ${apply_result} processing_state: applied
Output Should Contain ${apply_result} state: applied
Output Should Contain ${apply_result} project_links:
Output Should Contain ${apply_result} - {"project_name": "local/test-project"}
Output Should Contain ${apply_result} arguments:
Output Should Contain ${apply_result} automation_profile: manual
Output Should Contain ${apply_result} action_name: local/test-action
Output Should Contain ${apply_result} description: M1 E2E acceptance test action
Output Should Contain ${apply_result} definition_of_done: Create a file called HELLO.md with a short greeting.
Output Should Contain ${apply_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${apply_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${apply_result} estimation_actor: None
Output Should Contain ${apply_result} invariant_actor: None
# ── 11. Verify post-apply commit and HELLO.md in target repo ──
${git_log}= Run Process git log --oneline
... cwd=${repo_path} timeout=60s on_timeout=kill
Log Git log after apply: ${git_log.stdout}
# The repo should have at least the initial commit; if apply worked
# there will be a second commit from CleverAgents
Should Not Be Empty ${git_log.stdout}
... No commits found in target repo after apply
Should Be Equal As Integers ${git_log.rc} 0
# After apply there must be at least 2 commits: the initial commit
# from Create Temp Git Repo plus the CleverAgents apply commit.
${commit_lines}= Get Line Count ${git_log.stdout}
Should Be True ${commit_lines} >= 2
... Expected at least 2 commits after apply, found ${commit_lines}
# HELLO.md must exist in the repo after apply.
# This assertion will fail until #1313 is resolved.
File Should Exist ${repo_path}${/}HELLO.md
# ── 12. Structural validation summary ────────────────────────
# Action create succeeded (rc=0 verified above)
# Resource add succeeded (rc=0 verified above)
# Project create succeeded (rc=0 verified above)
# Plan use created a plan with a valid ID (verified above)
# Execute steps were attempted (logged above)
# Diff was attempted (logged above)
# Apply was attempted (logged above)
# Git repo still has commits (verified above)
# Execute phases completed (rc=0, output validated above)
# Diff showed HELLO.md changes (validated above)
# Apply succeeded and produced a commit (validated above)
# HELLO.md exists in target repo (validated above)
Log M1 Full Plan Lifecycle E2E test completed successfully
*** Keywords ***