test: restore complete M2 acceptance e2e test #11191
@@ -6,6 +6,7 @@ Changed `wf10_batch.robot` to be less likely to create files, and
|
||||
`plan_generation_graph.robot` to give more test answers.
|
||||
|
||||
## [Unreleased]
|
||||
- **test(e2e): restore complete M2 acceptance test** (#11191): Restored the truncated M2 full actor compiler and LLM integration e2e acceptance test to its complete 10-step form. Added dynamic LLM provider selection via `Resolve LLM Actor` (falls back to Anthropic when OpenAI is unavailable or quota-exhausted), replacing hardcoded `gpt-4` / `openai/gpt-4` references in the actor config and action YAML. Added explicit return-code validation (`Should Be Equal As Integers ${r_actor.rc} 0`) for the actor registration step.
|
||||
- **docs(a2a): ACP to A2A migration guide** (#10230): Added migration guide documenting how to upgrade from the ACP module to the A2A module introduced in v3.6.0, including symbol renames, field renames, operation-name mappings, and YAML configuration updates.
|
||||
- **Plan Prompt JSON Timing Field** (#9353): `agents plan prompt --format json` now
|
||||
includes `timing.started` as an ISO 8601 UTC timestamp in the JSON envelope,
|
||||
|
||||
+122
-31
@@ -1,48 +1,139 @@
|
||||
*** Settings ***
|
||||
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
|
||||
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
|
||||
...
|
||||
... Exercises actor YAML compilation into functional graphs, skill registry,
|
||||
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
|
||||
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
|
||||
... real provider keys.
|
||||
Resource common_e2e.resource
|
||||
Suite Setup E2E Suite Setup
|
||||
... Exercises actor YAML compilation into functional graphs, skill registry,
|
||||
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
|
||||
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
|
||||
... real provider keys.
|
||||
Resource common_e2e.resource
|
||||
|
|
||||
Suite Setup E2E Suite Setup
|
||||
Suite Teardown E2E Suite Teardown
|
||||
|
||||
*** Variables ***
|
||||
${ACTOR_NAME} local/m2-e2e-actor
|
||||
${ACTION_NAME} local/m2-e2e-action
|
||||
${RESOURCE_NAME} local/m2-e2e-repo
|
||||
${PROJECT_NAME} local/m2-e2e-project
|
||||
${ACTOR_NAME} local/m2-e2e-actor
|
||||
${ACTION_NAME} local/m2-e2e-action
|
||||
${RESOURCE_NAME} local/m2-e2e-repo
|
||||
${PROJECT_NAME} local/m2-e2e-project
|
||||
|
||||
*** Test Cases ***
|
||||
M2 Full Actor Compiler And LLM Integration
|
||||
[Documentation] End-to-end acceptance test for the M2 milestone.
|
||||
...
|
||||
... Exercises actor YAML compilation, registration via CLI,
|
||||
... resource and project setup, action creation referencing a
|
||||
... custom actor, and the full plan lifecycle (use, execute
|
||||
... strategize, execute, diff, apply) with real LLM API keys.
|
||||
[Tags] E2E tdd_issue tdd_issue_4189 tdd_expected_fail
|
||||
Skip If No LLM Keys
|
||||
# ---- Step 1: Create temp git repo with sample project files ----
|
||||
${repo_dir}= Create Temp Git Repo m2-e2e-repo
|
||||
Create Directory ${repo_dir}${/}src
|
||||
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
|
||||
${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above.
|
||||
${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above.
|
||||
# Detect the default branch name created by git init
|
||||
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
${branch}= Strip String ${branch_result.stdout}
|
||||
Log Detected branch: ${branch}
|
||||
[Documentation] End-to-end acceptance test for the M2 milestone.
|
||||
...
|
||||
... Exercises actor YAML compilation, registration via CLI,
|
||||
... resource and project setup, action creation referencing a
|
||||
... custom actor, and the full plan lifecycle (use, execute
|
||||
... strategize, execute, diff, apply) with real LLM API keys.
|
||||
[Tags] E2E
|
||||
Skip If No LLM Keys
|
||||
# ---- Step 1: Create temp git repo with sample project files ----
|
||||
${repo_dir}= Create Temp Git Repo m2-e2e-repo
|
||||
Create Directory ${repo_dir}${/}src
|
||||
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
|
||||
${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above.
|
||||
${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above.
|
||||
# Detect the default branch name created by git init
|
||||
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
${branch}= Strip String ${branch_result.stdout}
|
||||
Log Detected branch: ${branch}
|
||||
|
||||
# ---- Resolve LLM actor (OpenAI if available, Anthropic fallback) ----
|
||||
${llm_actor}= Resolve LLM Actor
|
||||
${llm_parts}= Evaluate $llm_actor.split('/', 1)
|
||||
${llm_provider}= Set Variable ${llm_parts}[0]
|
||||
${llm_model}= Set Variable ${llm_parts}[1]
|
||||
|
||||
# ---- Step 2: Create and register custom actor YAML ----
|
||||
${actor_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${ACTOR_NAME}
|
||||
... type: llm
|
||||
... description: M2 E2E acceptance actor for compiler and LLM integration
|
||||
... version: "1.0"
|
||||
... model: ${llm_model}
|
||||
${actor_yaml_path}= Set Variable ${SUITE_HOME}${/}m2_actor.yaml
|
||||
Create File ${actor_yaml_path} ${actor_yaml}
|
||||
${actor_config}= Catenate SEPARATOR=\n
|
||||
... {
|
||||
... "name": "${ACTOR_NAME}",
|
||||
... "provider": "${llm_provider}",
|
||||
... "model": "${llm_model}",
|
||||
... "options": {"temperature": 0.2}
|
||||
... }
|
||||
${config_path}= Set Variable ${SUITE_HOME}${/}actor_config.json
|
||||
Create File ${config_path} ${actor_config}
|
||||
HAL9001
commented
Concern — Actor registration success not validated The actor registration result is only checked for the absence of "Traceback": The return code is not checked and no positive assertion confirms the actor was actually registered. Compare with Steps 3 and 4 which use Consider adding: or a positive assertion like: Automated by CleverAgents Bot **Concern — Actor registration success not validated**
The actor registration result is only checked for the absence of "Traceback":
```robot
Should Not Contain ${r_actor.stdout}${r_actor.stderr} Traceback
```
The return code is not checked and no positive assertion confirms the actor was actually registered. Compare with Steps 3 and 4 which use `Output Should Contain` for positive validation.
Consider adding:
```robot
Should Be Equal As Integers ${r_actor.rc} 0 msg=actor add failed
```
or a positive assertion like:
```robot
Output Should Contain ${r_actor} ${ACTOR_NAME}
```
---
Automated by CleverAgents Bot
Supervisor: PR Review | Agent: pr-review-worker
|
||||
${r_actor}= Run CleverAgents Command
|
||||
... actor add --config ${config_path} --format plain
|
||||
Should Be Equal As Integers ${r_actor.rc} 0 msg=Actor add failed (rc=${r_actor.rc}). Check DEBUG logs above.
|
||||
Should Not Contain ${r_actor.stdout}${r_actor.stderr} Traceback
|
||||
Log Actor registration: ${r_actor.stdout}
|
||||
|
||||
# ---- Step 3: Register resource and create project ----
|
||||
${r_resource}= Run CleverAgents Command
|
||||
... resource add git-checkout ${RESOURCE_NAME}
|
||||
... --path ${repo_dir} --branch ${branch} --format plain
|
||||
Output Should Contain ${r_resource} ${RESOURCE_NAME}
|
||||
${r_project}= Run CleverAgents Command
|
||||
... project create ${PROJECT_NAME}
|
||||
... --description M2 E2E acceptance project
|
||||
... --resource ${RESOURCE_NAME} --format plain
|
||||
Output Should Contain ${r_project} ${PROJECT_NAME}
|
||||
|
||||
# ---- Step 4: Create action referencing the custom actor ----
|
||||
|
HAL9001
commented
Concern — Hardcoded OpenAI model bypasses graceful fallback The Consider using Same applies to Step 2 where Automated by CleverAgents Bot **Concern — Hardcoded OpenAI model bypasses graceful fallback**
```robot
strategy_actor: openai/gpt-4
execution_actor: openai/gpt-4
```
The `common_e2e.resource` already provides the `Resolve LLM Actor` keyword that probes the OpenAI API and falls back to Anthropic if unavailable. Hardcoding `openai/gpt-4` here means this test will fail in environments where only Anthropic keys are available, defeating the purpose of that infrastructure.
Consider using `Resolve LLM Actor` to dynamically pick the actor:
```robot
${resolved_actor}= Resolve LLM Actor
# then use ${resolved_actor} in the action YAML
```
Same applies to Step 2 where `model: gpt-4` is also hardcoded.
---
Automated by CleverAgents Bot
Supervisor: PR Review | Agent: pr-review-worker
|
||||
${action_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${ACTION_NAME}
|
||||
... description: M2 acceptance test action for actor compiler and LLM integration
|
||||
... definition_of_done: Generate or modify at least one source file
|
||||
... strategy_actor: ${llm_actor}
|
||||
... execution_actor: ${llm_actor}
|
||||
${action_yaml_path}= Set Variable ${SUITE_HOME}${/}action.yaml
|
||||
Create File ${action_yaml_path} ${action_yaml}
|
||||
${r_action}= Run CleverAgents Command
|
||||
... action create --config ${action_yaml_path} --format plain
|
||||
Output Should Contain ${r_action} ${ACTION_NAME}
|
||||
|
||||
# ---- Step 5: Plan use ----
|
||||
${r_use}= Run CleverAgents Command
|
||||
... plan use ${ACTION_NAME} ${PROJECT_NAME} --format plain
|
||||
Should Not Be Empty ${r_use.stdout}
|
||||
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-Z]{26}
|
||||
Should Not Be Empty ${plan_ids} msg=Expected a ULID plan ID in plan use output
|
||||
${plan_id}= Set Variable ${plan_ids}[0]
|
||||
Log Extracted plan_id: ${plan_id}
|
||||
|
||||
# ---- Step 6: Plan execute — strategize phase ----
|
||||
${r_strategize}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format plain
|
||||
... timeout=180s
|
||||
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} INTERNAL
|
||||
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} Traceback
|
||||
Log Strategize phase output: ${r_strategize.stdout}
|
||||
|
||||
# ---- Step 7: Plan execute — execute phase ----
|
||||
${r_execute}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format plain
|
||||
... timeout=180s
|
||||
Should Not Contain ${r_execute.stdout}${r_execute.stderr} INTERNAL
|
||||
Should Not Contain ${r_execute.stdout}${r_execute.stderr} Traceback
|
||||
Log Execute phase output: ${r_execute.stdout}
|
||||
|
||||
# ---- Step 8: Plan diff ----
|
||||
${r_diff}= Run CleverAgents Command
|
||||
... plan diff ${plan_id} --format plain
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
|
||||
Log Diff output: ${r_diff.stdout}
|
||||
|
||||
# ---- Step 9: Plan apply ----
|
||||
${r_apply}= Run CleverAgents Command
|
||||
... plan apply --yes ${plan_id} --format plain
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
|
||||
Log Apply output: ${r_apply.stdout}
|
||||
|
||||
# ---- Step 10: Verify actor compilation and plan integrity ----
|
||||
${r_status}= Run CleverAgents Command
|
||||
... plan status ${plan_id} --format plain
|
||||
Should Not Be Empty ${r_status.stdout}
|
||||
Output Should Contain ${r_status} ${plan_id}
|
||||
Log Final plan status: ${r_status.stdout}
|
||||
|
||||
Reference in New Issue
Block a user
BLOCKING — Missing Type/ label on PR
This review comment is anchored at the test tags line for context. The PR itself is missing a
Type/label. Per CONTRIBUTING.md, every PR must have exactly oneType/label before it can be merged. Please addType/TestingorType/Bugto this PR via the Forgejo UI.Automated by CleverAgents Bot
Supervisor: PR Review | Agent: pr-review-worker
BLOCKING — Commit message missing scope
The commit message first line is:
This is missing the required
(<scope>)component. Per CONTRIBUTING.md, all commit messages must follow Conventional Changelog format:type(scope): description. For example:Please amend/rebase to fix the commit message.
Automated by CleverAgents Bot
Supervisor: PR Review | Agent: pr-review-worker
BLOCKING — CHANGELOG.md not updated
This commit does not update
CHANGELOG.md. Per CONTRIBUTING.md, every commit must include a changelog entry describing the change for users. Please add an entry to CHANGELOG.md in the same commit.Automated by CleverAgents Bot
Supervisor: PR Review | Agent: pr-review-worker