From c3ed77a95c0579d384171d932bfbce5f5691058b Mon Sep 17 00:00:00 2001 From: Jeffrey Phillips Freeman Date: Wed, 13 May 2026 06:15:41 +0000 Subject: [PATCH 1/2] test: restore complete M2 acceptance e2e test The M2 acceptance test was incomplete (truncated at line 48). Restore the full test case that exercises the complete M2 milestone: - Actor YAML compilation and registration - Resource registration and project creation - Action creation with custom actor configuration - Full plan lifecycle: use, strategize, execute, diff, apply - Verification of plan status and integrity Exercises real LLM API integration using OpenAI GPT-4 model. ISSUES CLOSED: #10812 --- robot/e2e/m2_acceptance.robot | 146 ++++++++++++++++++++++++++-------- 1 file changed, 115 insertions(+), 31 deletions(-) diff --git a/robot/e2e/m2_acceptance.robot b/robot/e2e/m2_acceptance.robot index 55bbdba24..278584522 100644 --- a/robot/e2e/m2_acceptance.robot +++ b/robot/e2e/m2_acceptance.robot @@ -1,48 +1,132 @@ *** Settings *** -Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration. +Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration. ... -... Exercises actor YAML compilation into functional graphs, skill registry, -... tool lifecycle, and plan execution with a custom actor using real LLM keys. -... Zero mocking — all CLI invocations hit the real CleverAgents binary with -... real provider keys. -Resource common_e2e.resource -Suite Setup E2E Suite Setup +... Exercises actor YAML compilation into functional graphs, skill registry, +... tool lifecycle, and plan execution with a custom actor using real LLM keys. +... Zero mocking — all CLI invocations hit the real CleverAgents binary with +... real provider keys. +Resource common_e2e.resource +Suite Setup E2E Suite Setup Suite Teardown E2E Suite Teardown *** Variables *** -${ACTOR_NAME} local/m2-e2e-actor -${ACTION_NAME} local/m2-e2e-action -${RESOURCE_NAME} local/m2-e2e-repo -${PROJECT_NAME} local/m2-e2e-project +${ACTOR_NAME} local/m2-e2e-actor +${ACTION_NAME} local/m2-e2e-action +${RESOURCE_NAME} local/m2-e2e-repo +${PROJECT_NAME} local/m2-e2e-project *** Test Cases *** M2 Full Actor Compiler And LLM Integration - [Documentation] End-to-end acceptance test for the M2 milestone. - ... - ... Exercises actor YAML compilation, registration via CLI, - ... resource and project setup, action creation referencing a - ... custom actor, and the full plan lifecycle (use, execute - ... strategize, execute, diff, apply) with real LLM API keys. - [Tags] E2E tdd_issue tdd_issue_4189 tdd_expected_fail - Skip If No LLM Keys - # ---- Step 1: Create temp git repo with sample project files ---- - ${repo_dir}= Create Temp Git Repo m2-e2e-repo - Create Directory ${repo_dir}${/}src - Create File ${repo_dir}${/}src${/}main.py print("hello world")\n - ${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill - Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above. - ${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill - Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above. - # Detect the default branch name created by git init - ${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill - ${branch}= Strip String ${branch_result.stdout} - Log Detected branch: ${branch} + [Documentation] End-to-end acceptance test for the M2 milestone. + ... + ... Exercises actor YAML compilation, registration via CLI, + ... resource and project setup, action creation referencing a + ... custom actor, and the full plan lifecycle (use, execute + ... strategize, execute, diff, apply) with real LLM API keys. + [Tags] E2E + Skip If No LLM Keys + # ---- Step 1: Create temp git repo with sample project files ---- + ${repo_dir}= Create Temp Git Repo m2-e2e-repo + Create Directory ${repo_dir}${/}src + Create File ${repo_dir}${/}src${/}main.py print("hello world")\n + ${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill + Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above. + ${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill + Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above. + # Detect the default branch name created by git init + ${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill + ${branch}= Strip String ${branch_result.stdout} + Log Detected branch: ${branch} + # ---- Step 2: Create and register custom actor YAML ---- + ${actor_yaml}= Catenate SEPARATOR=\n + ... name: ${ACTOR_NAME} + ... type: llm + ... description: M2 E2E acceptance actor for compiler and LLM integration + ... version: "1.0" + ... model: gpt-4 + ${actor_yaml_path}= Set Variable ${SUITE_HOME}${/}m2_actor.yaml + Create File ${actor_yaml_path} ${actor_yaml} + ${actor_config}= Catenate SEPARATOR=\n + ... { + ... "name": "${ACTOR_NAME}", + ... "provider": "openai", + ... "model": "gpt-4", + ... "options": {"temperature": 0.2} + ... } + ${config_path}= Set Variable ${SUITE_HOME}${/}actor_config.json + Create File ${config_path} ${actor_config} + ${r_actor}= Run CleverAgents Command + ... actor add --config ${config_path} --format plain + Should Not Contain ${r_actor.stdout}${r_actor.stderr} Traceback + Log Actor registration: ${r_actor.stdout} + # ---- Step 3: Register resource and create project ---- + ${r_resource}= Run CleverAgents Command + ... resource add git-checkout ${RESOURCE_NAME} + ... --path ${repo_dir} --branch ${branch} --format plain + Output Should Contain ${r_resource} ${RESOURCE_NAME} + ${r_project}= Run CleverAgents Command + ... project create ${PROJECT_NAME} + ... --description M2 E2E acceptance project + ... --resource ${RESOURCE_NAME} --format plain + Output Should Contain ${r_project} ${PROJECT_NAME} + # ---- Step 4: Create action referencing the custom actor ---- + ${action_yaml}= Catenate SEPARATOR=\n + ... name: ${ACTION_NAME} + ... description: M2 acceptance test action for actor compiler and LLM integration + ... definition_of_done: Generate or modify at least one source file + ... strategy_actor: openai/gpt-4 + ... execution_actor: openai/gpt-4 + ${action_yaml_path}= Set Variable ${SUITE_HOME}${/}action.yaml + Create File ${action_yaml_path} ${action_yaml} + ${r_action}= Run CleverAgents Command + ... action create --config ${action_yaml_path} --format plain + Output Should Contain ${r_action} ${ACTION_NAME} + # ---- Step 5: Plan use ---- + ${r_use}= Run CleverAgents Command + ... plan use ${ACTION_NAME} ${PROJECT_NAME} --format plain + Should Not Be Empty ${r_use.stdout} + ${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-Z]{26} + Should Not Be Empty ${plan_ids} msg=Expected a ULID plan ID in plan use output + ${plan_id}= Set Variable ${plan_ids}[0] + Log Extracted plan_id: ${plan_id} + # ---- Step 6: Plan execute — strategize phase ---- + ${r_strategize}= Run CleverAgents Command + ... plan execute ${plan_id} --format plain + ... timeout=180s + Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} INTERNAL + Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} Traceback + Log Strategize phase output: ${r_strategize.stdout} + # ---- Step 7: Plan execute — execute phase ---- + ${r_execute}= Run CleverAgents Command + ... plan execute ${plan_id} --format plain + ... timeout=180s + Should Not Contain ${r_execute.stdout}${r_execute.stderr} INTERNAL + Should Not Contain ${r_execute.stdout}${r_execute.stderr} Traceback + Log Execute phase output: ${r_execute.stdout} + # ---- Step 8: Plan diff ---- + ${r_diff}= Run CleverAgents Command + ... plan diff ${plan_id} --format plain + Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL + Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback + Log Diff output: ${r_diff.stdout} + # ---- Step 9: Plan apply ---- + ${r_apply}= Run CleverAgents Command + ... plan apply --yes ${plan_id} --format plain + Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL + Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback + Log Apply output: ${r_apply.stdout} + # ---- Step 10: Verify actor compilation and plan integrity ---- + ${r_status}= Run CleverAgents Command + ... plan status ${plan_id} --format plain + Should Not Be Empty ${r_status.stdout} + Output Should Contain ${r_status} ${plan_id} + Log Final plan status: ${r_status.stdout} -- 2.52.0 From 09bc5222a5b87d1b577b0fabc213a36d72c555dc Mon Sep 17 00:00:00 2001 From: CleverThis Date: Thu, 4 Jun 2026 00:04:13 -0400 Subject: [PATCH 2/2] test(e2e): fix M2 acceptance test LLM provider and actor validation - Replace hardcoded gpt-4/openai/gpt-4 references with Resolve LLM Actor keyword so the test falls back to Anthropic when OpenAI is unavailable - Add explicit return-code check (rc==0) for actor registration in Step 2 - Update CHANGELOG.md with entry for PR #11191 --- CHANGELOG.md | 1 + robot/e2e/m2_acceptance.robot | 17 ++++++++++++----- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c427b892e..60f3e1565 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ Changed `wf10_batch.robot` to be less likely to create files, and `plan_generation_graph.robot` to give more test answers. ## [Unreleased] +- **test(e2e): restore complete M2 acceptance test** (#11191): Restored the truncated M2 full actor compiler and LLM integration e2e acceptance test to its complete 10-step form. Added dynamic LLM provider selection via `Resolve LLM Actor` (falls back to Anthropic when OpenAI is unavailable or quota-exhausted), replacing hardcoded `gpt-4` / `openai/gpt-4` references in the actor config and action YAML. Added explicit return-code validation (`Should Be Equal As Integers ${r_actor.rc} 0`) for the actor registration step. - **docs(a2a): ACP to A2A migration guide** (#10230): Added migration guide documenting how to upgrade from the ACP module to the A2A module introduced in v3.6.0, including symbol renames, field renames, operation-name mappings, and YAML configuration updates. - **Plan Prompt JSON Timing Field** (#9353): `agents plan prompt --format json` now includes `timing.started` as an ISO 8601 UTC timestamp in the JSON envelope, diff --git a/robot/e2e/m2_acceptance.robot b/robot/e2e/m2_acceptance.robot index 278584522..1c10b841a 100644 --- a/robot/e2e/m2_acceptance.robot +++ b/robot/e2e/m2_acceptance.robot @@ -38,26 +38,33 @@ M2 Full Actor Compiler And LLM Integration ${branch}= Strip String ${branch_result.stdout} Log Detected branch: ${branch} + # ---- Resolve LLM actor (OpenAI if available, Anthropic fallback) ---- + ${llm_actor}= Resolve LLM Actor + ${llm_parts}= Evaluate $llm_actor.split('/', 1) + ${llm_provider}= Set Variable ${llm_parts}[0] + ${llm_model}= Set Variable ${llm_parts}[1] + # ---- Step 2: Create and register custom actor YAML ---- ${actor_yaml}= Catenate SEPARATOR=\n ... name: ${ACTOR_NAME} ... type: llm ... description: M2 E2E acceptance actor for compiler and LLM integration ... version: "1.0" - ... model: gpt-4 + ... model: ${llm_model} ${actor_yaml_path}= Set Variable ${SUITE_HOME}${/}m2_actor.yaml Create File ${actor_yaml_path} ${actor_yaml} ${actor_config}= Catenate SEPARATOR=\n ... { ... "name": "${ACTOR_NAME}", - ... "provider": "openai", - ... "model": "gpt-4", + ... "provider": "${llm_provider}", + ... "model": "${llm_model}", ... "options": {"temperature": 0.2} ... } ${config_path}= Set Variable ${SUITE_HOME}${/}actor_config.json Create File ${config_path} ${actor_config} ${r_actor}= Run CleverAgents Command ... actor add --config ${config_path} --format plain + Should Be Equal As Integers ${r_actor.rc} 0 msg=Actor add failed (rc=${r_actor.rc}). Check DEBUG logs above. Should Not Contain ${r_actor.stdout}${r_actor.stderr} Traceback Log Actor registration: ${r_actor.stdout} @@ -77,8 +84,8 @@ M2 Full Actor Compiler And LLM Integration ... name: ${ACTION_NAME} ... description: M2 acceptance test action for actor compiler and LLM integration ... definition_of_done: Generate or modify at least one source file - ... strategy_actor: openai/gpt-4 - ... execution_actor: openai/gpt-4 + ... strategy_actor: ${llm_actor} + ... execution_actor: ${llm_actor} ${action_yaml_path}= Set Variable ${SUITE_HOME}${/}action.yaml Create File ${action_yaml_path} ${action_yaml} ${r_action}= Run CleverAgents Command -- 2.52.0