forked from cleveragents/cleveragents-core
freemo
48cff5cfe0
Renames `plan lifecycle-list` to `plan list` and `plan lifecycle-apply` to `plan apply` to align with the specification's canonical command names. Removes legacy V2 plan commands that occupied those names. - Renamed CLI command registrations from lifecycle-list/lifecycle-apply to list/apply - Removed legacy V2 apply and list commands (~200 lines) - Updated apply shortcut in main.py to delegate to v3 lifecycle - Added defensive null check for plan existence in apply command - Updated 63+ test, doc, and benchmark files for consistency Closes #881 Co-authored-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me> Co-committed-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me>
578 lines
34 KiB
Plaintext
578 lines
34 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E test for Workflow Example 4: Multi-Project Dependency Update.
|
|
...
|
|
... Advanced scenario using the supervised automation profile.
|
|
... Three microservices share a common library with a breaking
|
|
... change (v1 to v2). CleverAgents creates a parent plan
|
|
... targeting all 4 projects, spawns child subplans per project,
|
|
... executes in dependency order (common-lib first, then services),
|
|
... validates per project, and applies in dependency order.
|
|
...
|
|
... Zero mocking — real CLI, real LLM API keys.
|
|
Resource common_e2e.resource
|
|
Suite Setup WF04 Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
Force Tags E2E
|
|
|
|
*** Variables ***
|
|
${ACTION_BASE} local/wf04-dep-update
|
|
${LIB_BASE} wf04-common-lib
|
|
${SVC1_BASE} wf04-svc-auth
|
|
${SVC2_BASE} wf04-svc-billing
|
|
${SVC3_BASE} wf04-svc-gateway
|
|
${WF04_SNAPSHOT_HELPER} ${CURDIR}${/}wf04_snapshot_helper.py
|
|
|
|
*** Keywords ***
|
|
WF04 Suite Setup
|
|
[Documentation] E2E Suite Setup plus unique suffix generation and actor selection.
|
|
E2E Suite Setup
|
|
# Initialise the database so commands work in all tests.
|
|
${init}= Run CleverAgents Command init --force --yes
|
|
Should Be Equal As Integers ${init.rc} 0
|
|
Should Not Contain ${init.stdout}${init.stderr} Traceback
|
|
Should Not Contain ${init.stdout}${init.stderr} INTERNAL
|
|
# Generate a unique suffix for resource/project names to avoid
|
|
# UNIQUE constraint collisions on repeated or parallel CI runs.
|
|
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
|
|
Set Suite Variable ${RUN_SUFFIX} ${suffix}
|
|
# Derive unique names from base + suffix
|
|
Set Suite Variable ${ACTION_NAME} ${ACTION_BASE}-${suffix}
|
|
Set Suite Variable ${LIB_RESOURCE} ${LIB_BASE}-res-${suffix}
|
|
Set Suite Variable ${SVC1_RESOURCE} ${SVC1_BASE}-res-${suffix}
|
|
Set Suite Variable ${SVC2_RESOURCE} ${SVC2_BASE}-res-${suffix}
|
|
Set Suite Variable ${SVC3_RESOURCE} ${SVC3_BASE}-res-${suffix}
|
|
Set Suite Variable ${LIB_PROJECT} ${LIB_BASE}-proj-${suffix}
|
|
Set Suite Variable ${SVC1_PROJECT} ${SVC1_BASE}-proj-${suffix}
|
|
Set Suite Variable ${SVC2_PROJECT} ${SVC2_BASE}-proj-${suffix}
|
|
Set Suite Variable ${SVC3_PROJECT} ${SVC3_BASE}-proj-${suffix}
|
|
# Pick an actor that matches the available API key.
|
|
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
|
|
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
|
|
IF ${has_openai}
|
|
${actor}= Set Variable openai/gpt-4o
|
|
ELSE IF ${has_anthropic}
|
|
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
|
|
ELSE
|
|
${actor}= Set Variable openai/gpt-4o
|
|
END
|
|
Set Suite Variable ${LLM_ACTOR} ${actor}
|
|
|
|
Create Library Repo
|
|
[Documentation] Create temp git repo for the common library.
|
|
${repo}= Create Temp Git Repo ${LIB_BASE}-${RUN_SUFFIX}
|
|
Create Directory ${repo}${/}src
|
|
${lib_content}= Catenate SEPARATOR=\n
|
|
... """Common library v1 — shared utilities."""
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... __version__ = "1.0.0"
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def connect(host, port):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Connect to a service (v1 API)."""
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"host": host, "port": port, "status": "connected"}
|
|
Create File ${repo}${/}src${/}client.py ${lib_content}
|
|
${git_add}= Run Process git add . cwd=${repo} timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
|
|
${git_commit}= Run Process git commit -m Initial common library v1 cwd=${repo} timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
|
|
RETURN ${repo}
|
|
|
|
Create Service Repo
|
|
[Documentation] Create temp git repo for a microservice.
|
|
[Arguments] ${name} ${import_line}
|
|
${repo}= Create Temp Git Repo ${name}-${RUN_SUFFIX}
|
|
Create Directory ${repo}${/}src
|
|
${svc_content}= Catenate SEPARATOR=\n
|
|
... """${name} service — depends on common-lib v1."""
|
|
... ${import_line}
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def start():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}conn = connect("localhost", 8080)
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return conn
|
|
Create File ${repo}${/}src${/}app.py ${svc_content}
|
|
Create File ${repo}${/}requirements.txt common-lib==1.0.0\n
|
|
${git_add}= Run Process git add . cwd=${repo} timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
|
|
${git_commit}= Run Process git commit -m Initial ${name} service cwd=${repo} timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
|
|
RETURN ${repo}
|
|
|
|
Register Resource And Project
|
|
[Documentation] Register a git-checkout resource and create a project.
|
|
[Arguments] ${resource_name} ${project_name} ${repo_dir}
|
|
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${branch_result.rc} 0 git rev-parse failed: ${branch_result.stderr}
|
|
${branch}= Strip String ${branch_result.stdout}
|
|
${r_res}= Run CleverAgents Command
|
|
... resource add git-checkout ${resource_name}
|
|
... --path ${repo_dir} --branch ${branch}
|
|
Should Be Equal As Integers ${r_res.rc} 0 resource add failed: ${r_res.stderr}
|
|
Should Not Contain ${r_res.stdout}${r_res.stderr} Traceback
|
|
Should Not Contain ${r_res.stdout}${r_res.stderr} INTERNAL
|
|
${r_proj}= Run CleverAgents Command
|
|
... project create ${project_name}
|
|
... --resource ${resource_name}
|
|
Should Be Equal As Integers ${r_proj.rc} 0 project create failed: ${r_proj.stderr}
|
|
Should Not Contain ${r_proj.stdout}${r_proj.stderr} Traceback
|
|
Should Not Contain ${r_proj.stdout}${r_proj.stderr} INTERNAL
|
|
|
|
WF04 Test Teardown
|
|
[Documentation] Log diagnostic context on failure for debugging.
|
|
... Captures plan status and decision tree so CI failures
|
|
... in this 25-minute LLM-dependent test have actionable data.
|
|
${plan_id}= Get Variable Value ${WF04_PLAN_ID} ${EMPTY}
|
|
IF '${plan_id}' != ''
|
|
${status} ${result}= Run Keyword And Ignore Error
|
|
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
|
|
IF '${status}' == 'PASS'
|
|
Log Teardown plan status: ${result.stdout} WARN
|
|
END
|
|
${tree_status} ${tree_result}= Run Keyword And Ignore Error
|
|
... Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=30s
|
|
IF '${tree_status}' == 'PASS'
|
|
Log Teardown plan tree: ${tree_result.stdout} WARN
|
|
END
|
|
END
|
|
|
|
Register Validation For Project
|
|
[Documentation] Create a validation YAML, register it, and attach it to a project's resource.
|
|
[Arguments] ${validation_name} ${resource_name} ${project_name}
|
|
${val_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${validation_name}
|
|
... description: "Simple pass-through validation for E2E testing"
|
|
... source: custom
|
|
... mode: required
|
|
... code: |
|
|
... ${SPACE}${SPACE}return {"passed": True, "data": {}, "message": "Validation passed"}
|
|
... input_schema:
|
|
... ${SPACE}${SPACE}type: object
|
|
... ${SPACE}${SPACE}properties: {}
|
|
... timeout: 30
|
|
${val_path}= Set Variable ${SUITE_HOME}${/}${validation_name}.yaml
|
|
Create File ${val_path} ${val_yaml}
|
|
${r_add}= Run CleverAgents Command
|
|
... validation add --config ${val_path} expected_rc=None
|
|
Log Validation add rc=${r_add.rc} stdout=${r_add.stdout} stderr=${r_add.stderr}
|
|
Should Be Equal As Integers ${r_add.rc} 0
|
|
... validation add failed (rc=${r_add.rc}): ${r_add.stderr}
|
|
Should Not Contain ${r_add.stdout}${r_add.stderr} Traceback
|
|
Should Not Contain ${r_add.stdout}${r_add.stderr} INTERNAL
|
|
# Positional args: <resource_name> <validation_name> (resource first, validation second)
|
|
${r_attach}= Run CleverAgents Command
|
|
... validation attach --project ${project_name}
|
|
... ${resource_name} ${validation_name} expected_rc=None
|
|
Log Validation attach rc=${r_attach.rc} stdout=${r_attach.stdout} stderr=${r_attach.stderr}
|
|
Should Be Equal As Integers ${r_attach.rc} 0
|
|
... validation attach failed for ${project_name} (rc=${r_attach.rc}): ${r_attach.stderr}
|
|
Should Not Contain ${r_attach.stdout}${r_attach.stderr} Traceback
|
|
Should Not Contain ${r_attach.stdout}${r_attach.stderr} INTERNAL
|
|
|
|
Attach Validation To Project
|
|
[Documentation] Attach an already-registered validation to a project's resource.
|
|
[Arguments] ${validation_name} ${resource_name} ${project_name}
|
|
# Positional args: <resource_name> <validation_name> (resource first, validation second)
|
|
${r_attach}= Run CleverAgents Command
|
|
... validation attach --project ${project_name}
|
|
... ${resource_name} ${validation_name} expected_rc=None
|
|
Log Validation attach rc=${r_attach.rc} stdout=${r_attach.stdout} stderr=${r_attach.stderr}
|
|
Should Be Equal As Integers ${r_attach.rc} 0
|
|
... validation attach failed for ${project_name} (rc=${r_attach.rc}): ${r_attach.stderr}
|
|
Should Not Contain ${r_attach.stdout}${r_attach.stderr} Traceback
|
|
Should Not Contain ${r_attach.stdout}${r_attach.stderr} INTERNAL
|
|
|
|
Parse Json Payload
|
|
[Documentation] Parse JSON object/array from stdout with optional log preamble.
|
|
... Delegates to ``Extract JSON From Stdout`` which uses
|
|
... ``json.JSONDecoder().raw_decode()`` for robustness against
|
|
... trailing non-JSON output.
|
|
[Arguments] ${stdout}
|
|
${parsed}= Extract JSON From Stdout ${stdout}
|
|
RETURN ${parsed}
|
|
|
|
Get WF04 Plan Snapshot
|
|
[Documentation] Read parent/subplan metadata for deterministic WF04 assertions.
|
|
[Arguments] ${plan_id}
|
|
${snapshot_result}= Run Process
|
|
... ${PYTHON} ${WF04_SNAPSHOT_HELPER} ${plan_id}
|
|
... cwd=${SUITE_HOME} timeout=120s on_timeout=kill
|
|
... env:CLEVERAGENTS_HOME=${SUITE_HOME}
|
|
... env:CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true
|
|
... env:NO_COLOR=1
|
|
... env:PYTHONPATH=${WORKSPACE}${/}src
|
|
Should Be Equal As Integers ${snapshot_result.rc} 0
|
|
... Snapshot helper failed (rc=${snapshot_result.rc}): ${snapshot_result.stderr}
|
|
Should Not Be Empty ${snapshot_result.stdout}
|
|
${snapshot}= Parse Json Payload ${snapshot_result.stdout}
|
|
RETURN ${snapshot}
|
|
|
|
Verify WF04 Child Plan Spawning
|
|
[Documentation] AC-4: verify exactly 4 child plans mapped one-per-project.
|
|
[Arguments] ${snapshot}
|
|
${subplans}= Evaluate $snapshot.get('subplans', [])
|
|
${subplan_count}= Evaluate len($subplans)
|
|
IF ${subplan_count} == 0
|
|
Skip LLM produced no child plans — AC-4 child-plan mapping assertions were not exercised
|
|
END
|
|
Should Be Equal As Integers ${subplan_count} 4
|
|
... Expected exactly 4 child plans (common-lib + 3 services), found ${subplan_count}
|
|
${mapped_projects}= Evaluate sorted({p for sp in $subplans for p in sp.get('mapped_projects', []) if p})
|
|
${expected_projects}= Create List ${LIB_PROJECT} ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
|
|
${expected_sorted}= Evaluate sorted($expected_projects)
|
|
${all_projects_covered}= Evaluate $mapped_projects == $expected_sorted
|
|
Should Be True ${all_projects_covered}
|
|
... Child plans should cover all 4 projects. expected=${expected_sorted} actual=${mapped_projects}
|
|
${single_mapping}= Evaluate all(len(sp.get('mapped_projects', [])) == 1 for sp in $subplans)
|
|
Should Be True ${single_mapping}
|
|
... Each child plan should map to exactly one project scope
|
|
|
|
Verify WF04 Execution Order
|
|
[Documentation] AC-5: common-lib executes before service subplans.
|
|
[Arguments] ${snapshot}
|
|
${subplans}= Evaluate $snapshot.get('subplans', [])
|
|
${subplan_count}= Evaluate len($subplans)
|
|
IF ${subplan_count} == 0
|
|
Skip LLM produced no child plans — AC-5 execution-order assertions were not exercised
|
|
END
|
|
${lib_subplans}= Evaluate [sp for sp in $subplans if '${LIB_PROJECT}' in sp.get('mapped_projects', [])]
|
|
${lib_count}= Evaluate len($lib_subplans)
|
|
Should Be Equal As Integers ${lib_count} 1
|
|
... Expected exactly one common-lib child plan, found ${lib_count}
|
|
${lib_completed}= Evaluate $lib_subplans[0].get('completed_at') or $lib_subplans[0].get('execute_completed_at') or ''
|
|
Should Not Be Empty ${lib_completed}
|
|
... common-lib child plan completion timestamp is required for ordering checks
|
|
${service_projects}= Create List ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
|
|
${svc_subplans}= Evaluate [sp for sp in $subplans if any(p in $service_projects for p in sp.get('mapped_projects', []))]
|
|
${svc_count}= Evaluate len($svc_subplans)
|
|
Should Be Equal As Integers ${svc_count} 3
|
|
... Expected exactly three service child plans, found ${svc_count}
|
|
${svc_starts_present}= Evaluate all((sp.get('started_at') or sp.get('execute_started_at') or '') != '' for sp in $svc_subplans)
|
|
Should Be True ${svc_starts_present}
|
|
... Service child plans must expose start timestamps for execution-order checks
|
|
${services_after_lib}= Evaluate all((sp.get('started_at') or sp.get('execute_started_at')) >= $lib_completed for sp in $svc_subplans)
|
|
Should Be True ${services_after_lib}
|
|
... Service execution must start only after common-lib execution completes
|
|
|
|
Verify WF04 Validation Outcomes
|
|
[Documentation] AC-6: each child plan must report validation pass results.
|
|
[Arguments] ${snapshot}
|
|
${subplans}= Evaluate $snapshot.get('subplans', [])
|
|
${subplan_count}= Evaluate len($subplans)
|
|
IF ${subplan_count} == 0
|
|
Skip LLM produced no child plans — AC-6 per-project validation assertions were not exercised
|
|
END
|
|
${validation_present}= Evaluate all(isinstance(sp.get('child_validation_summary'), dict) and len(sp.get('child_validation_summary')) > 0 for sp in $subplans)
|
|
Should Be True ${validation_present}
|
|
... Each child plan must expose a non-empty validation summary
|
|
${validation_passed}= Evaluate all(int((sp.get('child_validation_summary') or {}).get('required_passed', 0) or 0) >= 1 and int((sp.get('child_validation_summary') or {}).get('required_failed', 0) or 0) == 0 for sp in $subplans)
|
|
Should Be True ${validation_passed}
|
|
... Each child plan must pass required validations (required_failed == 0)
|
|
|
|
Verify WF04 Apply Order
|
|
[Documentation] AC-7: common-lib apply must complete before service applies.
|
|
[Arguments] ${snapshot}
|
|
${subplans}= Evaluate $snapshot.get('subplans', [])
|
|
${subplan_count}= Evaluate len($subplans)
|
|
IF ${subplan_count} == 0
|
|
Skip LLM produced no child plans — AC-7 apply-order assertions were not exercised
|
|
END
|
|
${lib_subplans}= Evaluate [sp for sp in $subplans if '${LIB_PROJECT}' in sp.get('mapped_projects', [])]
|
|
${lib_count}= Evaluate len($lib_subplans)
|
|
Should Be Equal As Integers ${lib_count} 1
|
|
... Expected exactly one common-lib child plan, found ${lib_count}
|
|
${lib_applied}= Evaluate $lib_subplans[0].get('applied_at') or $lib_subplans[0].get('child_updated_at') or ''
|
|
Should Not Be Empty ${lib_applied}
|
|
... common-lib child plan apply timestamp is required for apply-order checks
|
|
${service_projects}= Create List ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
|
|
${svc_subplans}= Evaluate [sp for sp in $subplans if any(p in $service_projects for p in sp.get('mapped_projects', []))]
|
|
${svc_count}= Evaluate len($svc_subplans)
|
|
Should Be Equal As Integers ${svc_count} 3
|
|
... Expected exactly three service child plans, found ${svc_count}
|
|
${svc_apply_present}= Evaluate all((sp.get('applied_at') or sp.get('child_updated_at') or '') != '' for sp in $svc_subplans)
|
|
Should Be True ${svc_apply_present}
|
|
... Service child plans must expose apply/updated timestamps for apply-order checks
|
|
${services_after_lib_apply}= Evaluate all((sp.get('applied_at') or sp.get('child_updated_at')) >= $lib_applied for sp in $svc_subplans)
|
|
Should Be True ${services_after_lib_apply}
|
|
... Service applies must occur after common-lib apply
|
|
|
|
Count Decision Nodes
|
|
[Documentation] Recursively count decision nodes in a plan tree JSON structure.
|
|
... Invokes ``wf04_snapshot_helper.py --count-nodes`` as a subprocess
|
|
... to avoid importing the application DI container into the Robot
|
|
... test runner process.
|
|
[Arguments] ${tree_payload}
|
|
${tree_json}= Evaluate __import__('json').dumps($tree_payload)
|
|
${tmp_path}= Evaluate __import__('tempfile').NamedTemporaryFile(mode='w', suffix='.json', delete=False).name
|
|
Evaluate __import__('pathlib').Path(r'${tmp_path}').write_text($tree_json, encoding='utf-8')
|
|
${result}= Run Process
|
|
... ${PYTHON} ${WF04_SNAPSHOT_HELPER} --count-nodes ${tmp_path}
|
|
... timeout=30s on_timeout=kill
|
|
Evaluate __import__('os').unlink(r'${tmp_path}')
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
... count-nodes failed (rc=${result.rc}): ${result.stderr}
|
|
${count}= Convert To Integer ${result.stdout.strip()}
|
|
RETURN ${count}
|
|
|
|
Verify Plan In List
|
|
[Documentation] Verify a plan appears in list output.
|
|
[Arguments] ${plan_id}
|
|
${list_result}= Run CleverAgents Command
|
|
... plan list --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${list_result.rc} 0
|
|
... list failed (rc=${list_result.rc}): ${list_result.stderr}
|
|
Output Should Contain ${list_result} ${plan_id}
|
|
|
|
*** Test Cases ***
|
|
WF04 Multi Project Dependency Update Supervised Profile
|
|
[Documentation] Full supervised-profile workflow: register 4 repos,
|
|
... create multi-project action with invariants, plan use
|
|
... targeting all 4 projects with --automation-profile supervised,
|
|
... execute with child plan spawning in dependency order,
|
|
... verify per-project validation, and apply in dependency order.
|
|
[Timeout] 25 minutes
|
|
[Teardown] WF04 Test Teardown
|
|
Skip If No LLM Keys
|
|
# Initialise test variable for teardown access.
|
|
Set Test Variable ${WF04_PLAN_ID} ${EMPTY}
|
|
|
|
# ---- Create fixture repos ----
|
|
${lib_repo}= Create Library Repo
|
|
${svc1_repo}= Create Service Repo svc-auth from common_lib.client import connect
|
|
${svc2_repo}= Create Service Repo svc-billing from common_lib.client import connect
|
|
${svc3_repo}= Create Service Repo svc-gateway from common_lib.client import connect
|
|
|
|
# ---- Register resources and projects ----
|
|
Register Resource And Project ${LIB_RESOURCE} ${LIB_PROJECT} ${lib_repo}
|
|
Register Resource And Project ${SVC1_RESOURCE} ${SVC1_PROJECT} ${svc1_repo}
|
|
Register Resource And Project ${SVC2_RESOURCE} ${SVC2_PROJECT} ${svc2_repo}
|
|
Register Resource And Project ${SVC3_RESOURCE} ${SVC3_PROJECT} ${svc3_repo}
|
|
|
|
# ---- Register and attach validations for all 4 projects (AC-6) ----
|
|
${val_name}= Set Variable local/wf04-val-${RUN_SUFFIX}
|
|
Register Validation For Project ${val_name} ${LIB_RESOURCE} ${LIB_PROJECT}
|
|
# Validation already registered; attach to remaining 3 projects
|
|
Attach Validation To Project ${val_name} ${SVC1_RESOURCE} ${SVC1_PROJECT}
|
|
Attach Validation To Project ${val_name} ${SVC2_RESOURCE} ${SVC2_PROJECT}
|
|
Attach Validation To Project ${val_name} ${SVC3_RESOURCE} ${SVC3_PROJECT}
|
|
|
|
# ---- Create action with supervised profile and invariants ----
|
|
${action_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${ACTION_NAME}
|
|
... description: Update common-lib from v1 to v2 across all dependent services
|
|
... definition_of_done: All services updated to use common-lib v2 API
|
|
... strategy_actor: ${LLM_ACTOR}
|
|
... execution_actor: ${LLM_ACTOR}
|
|
... automation_profile: supervised
|
|
... reusable: true
|
|
... state: available
|
|
... invariants:
|
|
... ${SPACE}${SPACE}- "Each dependent project must be updated in its own child plan"
|
|
... ${SPACE}${SPACE}- "All child plans must pass validation before any can be applied"
|
|
... ${SPACE}${SPACE}- "The library update in common-lib must be applied first"
|
|
${action_path}= Set Variable ${SUITE_HOME}${/}wf04_action.yaml
|
|
Create File ${action_path} ${action_yaml}
|
|
${r_action}= Run CleverAgents Command
|
|
... action create --config ${action_path}
|
|
Should Be Equal As Integers ${r_action.rc} 0
|
|
... action create failed (rc=${r_action.rc}): ${r_action.stderr}
|
|
Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback
|
|
Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL
|
|
Output Should Contain ${r_action} ${ACTION_NAME}
|
|
|
|
# ---- Plan use targeting ALL 4 projects with supervised profile (AC-3) ----
|
|
${r_use}= Run CleverAgents Command
|
|
... plan use ${ACTION_NAME}
|
|
... ${LIB_PROJECT} ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
|
|
... --automation-profile supervised
|
|
... --format json
|
|
... timeout=120s
|
|
Should Be Equal As Integers ${r_use.rc} 0
|
|
... plan use failed (rc=${r_use.rc}): ${r_use.stderr}
|
|
Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback
|
|
Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL
|
|
${use_payload}= Parse Json Payload ${r_use.stdout}
|
|
${plan_id}= Evaluate str($use_payload.get('plan_id', ''))
|
|
Should Not Be Empty ${plan_id} msg=Expected plan_id in plan use JSON output
|
|
Log Plan ID: ${plan_id}
|
|
Set Test Variable ${WF04_PLAN_ID} ${plan_id}
|
|
|
|
# Verify plan targets all 4 projects exactly (AC-3)
|
|
${use_projects}= Evaluate sorted([link.get('project_name', '') for link in $use_payload.get('project_links', []) if isinstance(link, dict)])
|
|
${expected_projects}= Create List ${LIB_PROJECT} ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
|
|
${expected_sorted}= Evaluate sorted($expected_projects)
|
|
${projects_match}= Evaluate $use_projects == $expected_sorted
|
|
Should Be True ${projects_match}
|
|
... plan use should target all 4 projects. expected=${expected_sorted} actual=${use_projects}
|
|
|
|
# ---- Strategize ----
|
|
# Supervised profile requires two explicit ``plan execute`` calls:
|
|
# the first advances the plan through strategize, the second runs
|
|
# actual execution. Both use the same CLI command.
|
|
${r_strat}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format json expected_rc=None timeout=180s
|
|
Log Strategize rc=${r_strat.rc} stdout=${r_strat.stdout} stderr=${r_strat.stderr}
|
|
Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback
|
|
Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL
|
|
IF ${r_strat.rc} != 0
|
|
Fail plan execute (strategize) failed (rc=${r_strat.rc}): ${r_strat.stderr}
|
|
END
|
|
|
|
# ---- Decision tree — verify child plan spawning (AC-4) ----
|
|
${r_tree}= Run CleverAgents Command
|
|
... plan tree ${plan_id} --format json
|
|
... expected_rc=None timeout=60s
|
|
Should Be Equal As Integers ${r_tree.rc} 0
|
|
... plan tree failed (rc=${r_tree.rc}): ${r_tree.stderr}
|
|
Should Not Contain ${r_tree.stdout}${r_tree.stderr} Traceback
|
|
Should Not Contain ${r_tree.stdout}${r_tree.stderr} INTERNAL
|
|
Should Not Be Empty ${r_tree.stdout} Plan tree output should not be empty
|
|
Log Decision tree: ${r_tree.stdout}
|
|
|
|
# Parse tree for child plan / decision structure
|
|
${tree_payload}= Parse Json Payload ${r_tree.stdout}
|
|
${decision_count}= Count Decision Nodes ${tree_payload}
|
|
Log Decision tree contains ${decision_count} decision node(s)
|
|
Should Be True ${decision_count} >= 1
|
|
... Plan tree should contain at least one decision node after strategize (found ${decision_count})
|
|
# Some providers/runs expose a minimal strategize tree before execute.
|
|
# Child spawning is asserted deterministically after execute via snapshot.
|
|
|
|
# ---- Execute — dependency-ordered execution (AC-5) ----
|
|
${r_exec}= Run CleverAgents Command
|
|
... plan execute ${plan_id}
|
|
... --format json expected_rc=None timeout=300s
|
|
Log Execute rc=${r_exec.rc} stdout=${r_exec.stdout} stderr=${r_exec.stderr}
|
|
Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback
|
|
Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL
|
|
IF ${r_exec.rc} != 0
|
|
Fail plan execute failed (rc=${r_exec.rc}): ${r_exec.stderr}
|
|
END
|
|
|
|
# Deterministic WF04 assertions after execute (AC-4/AC-5/AC-6)
|
|
${exec_snapshot}= Get WF04 Plan Snapshot ${plan_id}
|
|
# Guard: if the snapshot contains zero subplans, skip the entire test rather
|
|
# than letting individual verification keywords silently skip all ACs.
|
|
# This ensures CI reports show SKIPPED (visible) rather than PASSED (misleading).
|
|
${exec_subplan_count}= Evaluate int($exec_snapshot.get('subplan_count', 0))
|
|
IF ${exec_subplan_count} == 0
|
|
Skip LLM produced 0 subplans — AC-4/5/6/7 verification cannot be exercised (entire test skipped)
|
|
END
|
|
Verify WF04 Child Plan Spawning ${exec_snapshot}
|
|
Verify WF04 Execution Order ${exec_snapshot}
|
|
Verify WF04 Validation Outcomes ${exec_snapshot}
|
|
|
|
# ---- Post-execute decision tree — verify child plan count (AC-4) ----
|
|
${r_tree_post}= Run CleverAgents Command
|
|
... plan tree ${plan_id} --format json
|
|
... expected_rc=None timeout=60s
|
|
Should Be Equal As Integers ${r_tree_post.rc} 0
|
|
... plan tree (post-execute) failed (rc=${r_tree_post.rc}): ${r_tree_post.stderr}
|
|
Should Not Contain ${r_tree_post.stdout}${r_tree_post.stderr} Traceback
|
|
Should Not Contain ${r_tree_post.stdout}${r_tree_post.stderr} INTERNAL
|
|
${tree_post_payload}= Parse Json Payload ${r_tree_post.stdout}
|
|
${post_exec_decision_count}= Count Decision Nodes ${tree_post_payload}
|
|
Log Post-execute decision tree contains ${post_exec_decision_count} decision node(s)
|
|
# Require non-trivial tree depth after execute. Subplan spawning is
|
|
# asserted deterministically via internal snapshot checks above.
|
|
Should Be True ${post_exec_decision_count} >= 2
|
|
... Plan tree should contain at least 2 decision nodes after execute (found ${post_exec_decision_count})
|
|
# Execute should preserve or grow the tree — never shrink it.
|
|
Should Be True ${post_exec_decision_count} >= ${decision_count}
|
|
... Post-execute decision count (${post_exec_decision_count}) should not be less than post-strategize count (${decision_count})
|
|
|
|
# ---- AC-6 verified above via per-child validation summaries ----
|
|
|
|
# ---- Plan list — verify plan exists ----
|
|
Verify Plan In List ${plan_id}
|
|
|
|
# ---- Verify plan status shows multi-project state ----
|
|
${r_status_mid}= Run CleverAgents Command
|
|
... plan status ${plan_id} --format json
|
|
... expected_rc=None timeout=60s
|
|
Should Be Equal As Integers ${r_status_mid.rc} 0
|
|
... plan status failed (rc=${r_status_mid.rc}): ${r_status_mid.stderr}
|
|
Should Not Contain ${r_status_mid.stdout}${r_status_mid.stderr} Traceback
|
|
Should Not Contain ${r_status_mid.stdout}${r_status_mid.stderr} INTERNAL
|
|
Output Should Contain ${r_status_mid} ${plan_id}
|
|
# Check for automation profile reference in status
|
|
${status_combined}= Set Variable ${r_status_mid.stdout}${r_status_mid.stderr}
|
|
${status_lower}= Evaluate ($status_combined).lower()
|
|
${has_profile_ref}= Evaluate 'supervised' in $status_lower or 'automation' in $status_lower or 'profile' in $status_lower
|
|
Should Be True ${has_profile_ref}
|
|
... Plan status should reference automation profile (supervised)
|
|
# Parse automation_profile field for precise assertion
|
|
${mid_profile}= Safe Parse Json Field ${r_status_mid.stdout} automation_profile
|
|
Should Not Be Empty ${mid_profile}
|
|
... automation_profile field should be present in plan status JSON
|
|
Should Be Equal As Strings ${mid_profile} supervised
|
|
... Plan automation profile should be supervised (found ${mid_profile})
|
|
|
|
# ---- Diff ----
|
|
${r_diff}= Run CleverAgents Command
|
|
... plan diff ${plan_id} --format plain
|
|
... expected_rc=None timeout=60s
|
|
Log Diff rc=${r_diff.rc} stdout=${r_diff.stdout} stderr=${r_diff.stderr}
|
|
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
|
|
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
|
|
Should Be Equal As Integers ${r_diff.rc} 0
|
|
... plan diff failed (rc=${r_diff.rc}): ${r_diff.stderr}
|
|
|
|
# ---- Apply — dependency-ordered apply (AC-7) ----
|
|
${r_apply}= Run CleverAgents Command
|
|
... plan apply ${plan_id} --yes --format json
|
|
... expected_rc=None timeout=180s
|
|
Log Apply rc=${r_apply.rc} stdout=${r_apply.stdout} stderr=${r_apply.stderr}
|
|
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
|
|
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
|
|
IF ${r_apply.rc} == 0
|
|
Output Should Contain ${r_apply} ${plan_id}
|
|
# Verify the plan transitioned — check for apply-phase indicators (AC-7)
|
|
${apply_phase}= Safe Parse Json Field ${r_apply.stdout} phase
|
|
Should Not Be Empty ${apply_phase}
|
|
... phase field should be present in apply JSON output
|
|
${apply_phase_lower}= Evaluate ($apply_phase).lower()
|
|
Should Contain ${apply_phase_lower} apply
|
|
... Plan phase should indicate apply after apply (found ${apply_phase})
|
|
${apply_snapshot}= Get WF04 Plan Snapshot ${plan_id}
|
|
# Guard: apply snapshot must also contain subplans for AC-7 verification.
|
|
# The exec guard above already skips the whole test if 0 subplans, so
|
|
# reaching this point implies subplans existed post-execute. If they
|
|
# disappeared post-apply, that would be a real regression worth flagging.
|
|
${apply_subplan_count}= Evaluate int($apply_snapshot.get('subplan_count', 0))
|
|
Should Be True ${apply_subplan_count} >= 1
|
|
... Snapshot reported 0 subplans after apply — subplans existed post-execute but vanished post-apply
|
|
Verify WF04 Apply Order ${apply_snapshot}
|
|
Verify WF04 Validation Outcomes ${apply_snapshot}
|
|
ELSE
|
|
Fail apply failed (rc=${r_apply.rc}) stdout=${r_apply.stdout} stderr=${r_apply.stderr}
|
|
END
|
|
|
|
# ---- Verify final status ----
|
|
${r_status}= Run CleverAgents Command
|
|
... plan status ${plan_id} --format json
|
|
... expected_rc=None timeout=60s
|
|
Should Be Equal As Integers ${r_status.rc} 0
|
|
... plan status (final) failed (rc=${r_status.rc}): ${r_status.stderr}
|
|
Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback
|
|
Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL
|
|
Should Not Be Empty ${r_status.stdout} Final plan status output should not be empty
|
|
Output Should Contain ${r_status} ${plan_id}
|
|
# Parse phase from final status and assert non-empty (proves lifecycle events processed)
|
|
${final_phase}= Safe Parse Json Field ${r_status.stdout} phase
|
|
${final_state}= Safe Parse Json Field ${r_status.stdout} processing_state
|
|
Log Final phase=${final_phase} processing_state=${final_state}
|
|
${final_state_populated}= Evaluate $final_phase != '' or $final_state != ''
|
|
Should Be True ${final_state_populated}
|
|
... Final plan status should have non-empty phase or processing_state
|
|
# After a full lifecycle (execute + apply), phase or state should reflect completion
|
|
${final_lower}= Evaluate ($final_phase).lower() if $final_phase else ''
|
|
${state_lower}= Evaluate ($final_state).lower() if $final_state else ''
|
|
${is_terminal}= Evaluate 'apply' in $final_lower or 'complete' in $final_lower or 'done' in $final_lower or 'complete' in $state_lower or 'done' in $state_lower or 'applied' in $state_lower
|
|
Should Be True ${is_terminal}
|
|
... Final status should indicate a terminal/applied state (phase=${final_phase}, state=${final_state})
|
|
|
|
# ---- Final: plan should still appear in list ----
|
|
Verify Plan In List ${plan_id}
|