HAL9000
09bc5222a5
CI / lint (pull_request) Successful in 39s
CI / helm (pull_request) Successful in 33s
CI / build (pull_request) Successful in 50s
CI / quality (pull_request) Successful in 1m6s
CI / typecheck (pull_request) Successful in 1m10s
CI / push-validation (pull_request) Successful in 26s
CI / security (pull_request) Successful in 1m18s
CI / unit_tests (pull_request) Successful in 6m2s
CI / docker (pull_request) Successful in 1m28s
CI / integration_tests (pull_request) Successful in 10m12s
CI / coverage (pull_request) Successful in 10m56s
CI / status-check (pull_request) Successful in 3s
- Replace hardcoded gpt-4/openai/gpt-4 references with Resolve LLM Actor keyword so the test falls back to Anthropic when OpenAI is unavailable - Add explicit return-code check (rc==0) for actor registration in Step 2 - Update CHANGELOG.md with entry for PR #11191
140 lines
7.1 KiB
Plaintext
140 lines
7.1 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
|
|
...
|
|
... Exercises actor YAML compilation into functional graphs, skill registry,
|
|
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
|
|
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
|
|
... real provider keys.
|
|
Resource common_e2e.resource
|
|
Suite Setup E2E Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
|
|
*** Variables ***
|
|
${ACTOR_NAME} local/m2-e2e-actor
|
|
${ACTION_NAME} local/m2-e2e-action
|
|
${RESOURCE_NAME} local/m2-e2e-repo
|
|
${PROJECT_NAME} local/m2-e2e-project
|
|
|
|
*** Test Cases ***
|
|
M2 Full Actor Compiler And LLM Integration
|
|
[Documentation] End-to-end acceptance test for the M2 milestone.
|
|
...
|
|
... Exercises actor YAML compilation, registration via CLI,
|
|
... resource and project setup, action creation referencing a
|
|
... custom actor, and the full plan lifecycle (use, execute
|
|
... strategize, execute, diff, apply) with real LLM API keys.
|
|
[Tags] E2E
|
|
Skip If No LLM Keys
|
|
# ---- Step 1: Create temp git repo with sample project files ----
|
|
${repo_dir}= Create Temp Git Repo m2-e2e-repo
|
|
Create Directory ${repo_dir}${/}src
|
|
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
|
|
${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above.
|
|
${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above.
|
|
# Detect the default branch name created by git init
|
|
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
|
|
${branch}= Strip String ${branch_result.stdout}
|
|
Log Detected branch: ${branch}
|
|
|
|
# ---- Resolve LLM actor (OpenAI if available, Anthropic fallback) ----
|
|
${llm_actor}= Resolve LLM Actor
|
|
${llm_parts}= Evaluate $llm_actor.split('/', 1)
|
|
${llm_provider}= Set Variable ${llm_parts}[0]
|
|
${llm_model}= Set Variable ${llm_parts}[1]
|
|
|
|
# ---- Step 2: Create and register custom actor YAML ----
|
|
${actor_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${ACTOR_NAME}
|
|
... type: llm
|
|
... description: M2 E2E acceptance actor for compiler and LLM integration
|
|
... version: "1.0"
|
|
... model: ${llm_model}
|
|
${actor_yaml_path}= Set Variable ${SUITE_HOME}${/}m2_actor.yaml
|
|
Create File ${actor_yaml_path} ${actor_yaml}
|
|
${actor_config}= Catenate SEPARATOR=\n
|
|
... {
|
|
... "name": "${ACTOR_NAME}",
|
|
... "provider": "${llm_provider}",
|
|
... "model": "${llm_model}",
|
|
... "options": {"temperature": 0.2}
|
|
... }
|
|
${config_path}= Set Variable ${SUITE_HOME}${/}actor_config.json
|
|
Create File ${config_path} ${actor_config}
|
|
${r_actor}= Run CleverAgents Command
|
|
... actor add --config ${config_path} --format plain
|
|
Should Be Equal As Integers ${r_actor.rc} 0 msg=Actor add failed (rc=${r_actor.rc}). Check DEBUG logs above.
|
|
Should Not Contain ${r_actor.stdout}${r_actor.stderr} Traceback
|
|
Log Actor registration: ${r_actor.stdout}
|
|
|
|
# ---- Step 3: Register resource and create project ----
|
|
${r_resource}= Run CleverAgents Command
|
|
... resource add git-checkout ${RESOURCE_NAME}
|
|
... --path ${repo_dir} --branch ${branch} --format plain
|
|
Output Should Contain ${r_resource} ${RESOURCE_NAME}
|
|
${r_project}= Run CleverAgents Command
|
|
... project create ${PROJECT_NAME}
|
|
... --description M2 E2E acceptance project
|
|
... --resource ${RESOURCE_NAME} --format plain
|
|
Output Should Contain ${r_project} ${PROJECT_NAME}
|
|
|
|
# ---- Step 4: Create action referencing the custom actor ----
|
|
${action_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${ACTION_NAME}
|
|
... description: M2 acceptance test action for actor compiler and LLM integration
|
|
... definition_of_done: Generate or modify at least one source file
|
|
... strategy_actor: ${llm_actor}
|
|
... execution_actor: ${llm_actor}
|
|
${action_yaml_path}= Set Variable ${SUITE_HOME}${/}action.yaml
|
|
Create File ${action_yaml_path} ${action_yaml}
|
|
${r_action}= Run CleverAgents Command
|
|
... action create --config ${action_yaml_path} --format plain
|
|
Output Should Contain ${r_action} ${ACTION_NAME}
|
|
|
|
# ---- Step 5: Plan use ----
|
|
${r_use}= Run CleverAgents Command
|
|
... plan use ${ACTION_NAME} ${PROJECT_NAME} --format plain
|
|
Should Not Be Empty ${r_use.stdout}
|
|
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-Z]{26}
|
|
Should Not Be Empty ${plan_ids} msg=Expected a ULID plan ID in plan use output
|
|
${plan_id}= Set Variable ${plan_ids}[0]
|
|
Log Extracted plan_id: ${plan_id}
|
|
|
|
# ---- Step 6: Plan execute — strategize phase ----
|
|
${r_strategize}= Run CleverAgents Command
|
|
... plan execute ${plan_id} --format plain
|
|
... timeout=180s
|
|
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} INTERNAL
|
|
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} Traceback
|
|
Log Strategize phase output: ${r_strategize.stdout}
|
|
|
|
# ---- Step 7: Plan execute — execute phase ----
|
|
${r_execute}= Run CleverAgents Command
|
|
... plan execute ${plan_id} --format plain
|
|
... timeout=180s
|
|
Should Not Contain ${r_execute.stdout}${r_execute.stderr} INTERNAL
|
|
Should Not Contain ${r_execute.stdout}${r_execute.stderr} Traceback
|
|
Log Execute phase output: ${r_execute.stdout}
|
|
|
|
# ---- Step 8: Plan diff ----
|
|
${r_diff}= Run CleverAgents Command
|
|
... plan diff ${plan_id} --format plain
|
|
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
|
|
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
|
|
Log Diff output: ${r_diff.stdout}
|
|
|
|
# ---- Step 9: Plan apply ----
|
|
${r_apply}= Run CleverAgents Command
|
|
... plan apply --yes ${plan_id} --format plain
|
|
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
|
|
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
|
|
Log Apply output: ${r_apply.stdout}
|
|
|
|
# ---- Step 10: Verify actor compilation and plan integrity ----
|
|
${r_status}= Run CleverAgents Command
|
|
... plan status ${plan_id} --format plain
|
|
Should Not Be Empty ${r_status.stdout}
|
|
Output Should Contain ${r_status} ${plan_id}
|
|
Log Final plan status: ${r_status.stdout}
|