HAL9000
f808abff86
Fix JSON syntax errors in .devcontainer/devcontainer.json (removed
invalid JS-style // comments) and .devcontainer/opencode.json (removed
90+ trailing commas). Apply auto-fixes for end-of-file and trailing
whitespace issues across 100+ files. Fix SIM105 ruff violations in
benchmarks/core_circuit_breaker_bench.py (use contextlib.suppress).
Note: The security fix from issue #7478 (validate_path startswith bypass)
was already delivered to master in commit e18ac5f2. This PR as currently
structured is non-atomic (35 commits across 10+ issues) and needs
significant restructure before merge. This commit only addresses the
CI/pre-commit failures.
ISSUES CLOSED: #7478
321 lines
18 KiB
Plaintext
321 lines
18 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E test for Workflow Example 4: Multi-Project Dependency Update.
|
|
...
|
|
... Advanced scenario using the supervised automation profile.
|
|
... Three microservices share a common library with a breaking
|
|
... change (v1 to v2). CleverAgents creates a parent plan
|
|
... targeting all 4 projects, spawns child subplans per project,
|
|
... executes in dependency order (common-lib first, then services),
|
|
... validates per project, and applies in dependency order.
|
|
...
|
|
... Zero mocking — real CLI, real LLM API keys.
|
|
Resource common_e2e.resource
|
|
Suite Setup WF04 Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
Force Tags E2E
|
|
|
|
*** Variables ***
|
|
${ACTION_BASE} local/wf04-dep-update
|
|
${LIB_BASE} wf04-common-lib
|
|
${SVC1_BASE} wf04-svc-auth
|
|
${SVC2_BASE} wf04-svc-billing
|
|
${SVC3_BASE} wf04-svc-gateway
|
|
${WF04_SNAPSHOT_HELPER} ${CURDIR}${/}wf04_snapshot_helper.py
|
|
|
|
*** Keywords ***
|
|
WF04 Suite Setup
|
|
[Documentation] E2E Suite Setup plus unique suffix generation and actor selection.
|
|
E2E Suite Setup
|
|
# Initialise the database so commands work in all tests.
|
|
${init}= Run CleverAgents Command init --force --yes
|
|
Should Be Equal As Integers ${init.rc} 0
|
|
Should Not Contain ${init.stdout}${init.stderr} Traceback
|
|
Should Not Contain ${init.stdout}${init.stderr} INTERNAL
|
|
# Generate a unique suffix for resource/project names to avoid
|
|
# UNIQUE constraint collisions on repeated or parallel CI runs.
|
|
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
|
|
Set Suite Variable ${RUN_SUFFIX} ${suffix}
|
|
# Derive unique names from base + suffix
|
|
Set Suite Variable ${ACTION_NAME} ${ACTION_BASE}-${suffix}
|
|
Set Suite Variable ${LIB_RESOURCE} ${LIB_BASE}-res-${suffix}
|
|
Set Suite Variable ${SVC1_RESOURCE} ${SVC1_BASE}-res-${suffix}
|
|
Set Suite Variable ${SVC2_RESOURCE} ${SVC2_BASE}-res-${suffix}
|
|
Set Suite Variable ${SVC3_RESOURCE} ${SVC3_BASE}-res-${suffix}
|
|
Set Suite Variable ${LIB_PROJECT} ${LIB_BASE}-proj-${suffix}
|
|
Set Suite Variable ${SVC1_PROJECT} ${SVC1_BASE}-proj-${suffix}
|
|
Set Suite Variable ${SVC2_PROJECT} ${SVC2_BASE}-proj-${suffix}
|
|
Set Suite Variable ${SVC3_PROJECT} ${SVC3_BASE}-proj-${suffix}
|
|
# Probe the OpenAI API to confirm the key is usable before selecting actor.
|
|
# Falls back to Anthropic when the key is absent or quota-exhausted.
|
|
${actor}= Resolve LLM Actor
|
|
Set Suite Variable ${LLM_ACTOR} ${actor}
|
|
|
|
Create Library Repo
|
|
[Documentation] Create temp git repo for the common library.
|
|
${repo}= Create Temp Git Repo ${LIB_BASE}-${RUN_SUFFIX}
|
|
Create Directory ${repo}${/}src
|
|
${lib_content}= Catenate SEPARATOR=\n
|
|
... """Common library v1 — shared utilities."""
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... __version__ = "1.0.0"
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def connect(host, port):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Connect to a service (v1 API)."""
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"host": host, "port": port, "status": "connected"}
|
|
Create File ${repo}${/}src${/}client.py ${lib_content}
|
|
${git_add}= Run Process git add . cwd=${repo} timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
|
|
${git_commit}= Run Process git commit -m Initial common library v1 cwd=${repo} timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
|
|
RETURN ${repo}
|
|
|
|
Create Service Repo
|
|
[Documentation] Create temp git repo for a microservice.
|
|
[Arguments] ${name} ${import_line}
|
|
${repo}= Create Temp Git Repo ${name}-${RUN_SUFFIX}
|
|
Create Directory ${repo}${/}src
|
|
${svc_content}= Catenate SEPARATOR=\n
|
|
... """${name} service — depends on common-lib v1."""
|
|
... ${import_line}
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def start():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}conn = connect("localhost", 8080)
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return conn
|
|
Create File ${repo}${/}src${/}app.py ${svc_content}
|
|
Create File ${repo}${/}requirements.txt common-lib==1.0.0\n
|
|
${git_add}= Run Process git add . cwd=${repo} timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
|
|
${git_commit}= Run Process git commit -m Initial ${name} service cwd=${repo} timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
|
|
RETURN ${repo}
|
|
|
|
Register Resource And Project
|
|
[Documentation] Register a git-checkout resource and create a project.
|
|
[Arguments] ${resource_name} ${project_name} ${repo_dir}
|
|
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${branch_result.rc} 0 msg=git rev-parse failed: ${branch_result.stderr}
|
|
${branch}= Strip String ${branch_result.stdout}
|
|
${r_res}= Run CleverAgents Command
|
|
... resource add git-checkout ${resource_name}
|
|
... --path ${repo_dir} --branch ${branch}
|
|
Should Be Equal As Integers ${r_res.rc} 0 resource add failed: ${r_res.stderr}
|
|
Should Not Contain ${r_res.stdout}${r_res.stderr} Traceback
|
|
Should Not Contain ${r_res.stdout}${r_res.stderr} INTERNAL
|
|
${r_proj}= Run CleverAgents Command
|
|
... project create ${project_name}
|
|
... --resource ${resource_name}
|
|
Should Be Equal As Integers ${r_proj.rc} 0 project create failed: ${r_proj.stderr}
|
|
Should Not Contain ${r_proj.stdout}${r_proj.stderr} Traceback
|
|
Should Not Contain ${r_proj.stdout}${r_proj.stderr} INTERNAL
|
|
|
|
WF04 Test Teardown
|
|
[Documentation] Log diagnostic context on failure for debugging.
|
|
... Captures plan status and decision tree so CI failures
|
|
... in this 25-minute LLM-dependent test have actionable data.
|
|
${plan_id}= Get Variable Value ${WF04_PLAN_ID} ${EMPTY}
|
|
IF '${plan_id}' != ''
|
|
${status} ${result}= Run Keyword And Ignore Error
|
|
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
|
|
IF '${status}' == 'PASS'
|
|
Log Teardown plan status: ${result.stdout} WARN
|
|
END
|
|
${tree_status} ${tree_result}= Run Keyword And Ignore Error
|
|
... Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=30s
|
|
IF '${tree_status}' == 'PASS'
|
|
Log Teardown plan tree: ${tree_result.stdout} WARN
|
|
END
|
|
END
|
|
|
|
Register Validation For Project
|
|
[Documentation] Create a validation YAML, register it, and attach it to a project's resource.
|
|
[Arguments] ${validation_name} ${resource_name} ${project_name}
|
|
${val_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${validation_name}
|
|
... description: "Simple pass-through validation for E2E testing"
|
|
... source: custom
|
|
... mode: required
|
|
... code: |
|
|
... ${SPACE}${SPACE}return {"passed": True, "data": {}, "message": "Validation passed"}
|
|
... input_schema:
|
|
... ${SPACE}${SPACE}type: object
|
|
... ${SPACE}${SPACE}properties: {}
|
|
... timeout: 30
|
|
${val_path}= Set Variable ${SUITE_HOME}${/}${validation_name}.yaml
|
|
Create File ${val_path} ${val_yaml}
|
|
${r_add}= Run CleverAgents Command
|
|
... validation add --config ${val_path} expected_rc=None
|
|
Log Validation add rc=${r_add.rc} stdout=${r_add.stdout} stderr=${r_add.stderr}
|
|
Should Be Equal As Integers ${r_add.rc} 0 msg=validation add failed (rc=${r_add.rc}): ${r_add.stderr}
|
|
Should Not Contain ${r_add.stdout}${r_add.stderr} Traceback
|
|
Should Not Contain ${r_add.stdout}${r_add.stderr} INTERNAL
|
|
# Positional args: <resource_name> <validation_name> (resource first, validation second)
|
|
${r_attach}= Run CleverAgents Command
|
|
... validation attach --project ${project_name}
|
|
... ${resource_name} ${validation_name} expected_rc=None
|
|
Log Validation attach rc=${r_attach.rc} stdout=${r_attach.stdout} stderr=${r_attach.stderr}
|
|
Should Be Equal As Integers ${r_attach.rc} 0 msg=validation attach failed for ${project_name} (rc=${r_attach.rc}): ${r_attach.stderr}
|
|
Should Not Contain ${r_attach.stdout}${r_attach.stderr} Traceback
|
|
Should Not Contain ${r_attach.stdout}${r_attach.stderr} INTERNAL
|
|
|
|
Attach Validation To Project
|
|
[Documentation] Attach an already-registered validation to a project's resource.
|
|
[Arguments] ${validation_name} ${resource_name} ${project_name}
|
|
# Positional args: <resource_name> <validation_name> (resource first, validation second)
|
|
${r_attach}= Run CleverAgents Command
|
|
... validation attach --project ${project_name}
|
|
... ${resource_name} ${validation_name} expected_rc=None
|
|
Log Validation attach rc=${r_attach.rc} stdout=${r_attach.stdout} stderr=${r_attach.stderr}
|
|
Should Be Equal As Integers ${r_attach.rc} 0 msg=validation attach failed for ${project_name} (rc=${r_attach.rc}): ${r_attach.stderr}
|
|
Should Not Contain ${r_attach.stdout}${r_attach.stderr} Traceback
|
|
Should Not Contain ${r_attach.stdout}${r_attach.stderr} INTERNAL
|
|
|
|
Parse Json Payload
|
|
[Documentation] Parse JSON object/array from stdout with optional log preamble.
|
|
... Delegates to ``Extract JSON From Stdout`` which uses
|
|
... ``json.JSONDecoder().raw_decode()`` for robustness against
|
|
... trailing non-JSON output.
|
|
[Arguments] ${stdout}
|
|
${parsed}= Extract JSON From Stdout ${stdout}
|
|
RETURN ${parsed}
|
|
|
|
Get WF04 Plan Snapshot
|
|
[Documentation] Read parent/subplan metadata for deterministic WF04 assertions.
|
|
[Arguments] ${plan_id}
|
|
${snapshot_result}= Run Process
|
|
... ${PYTHON} ${WF04_SNAPSHOT_HELPER} ${plan_id}
|
|
... cwd=${SUITE_HOME} timeout=120s on_timeout=kill
|
|
... env:CLEVERAGENTS_HOME=${SUITE_HOME}
|
|
... env:CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true
|
|
... env:NO_COLOR=1
|
|
... env:PYTHONPATH=${WORKSPACE}${/}src
|
|
Should Be Equal As Integers ${snapshot_result.rc} 0 msg=Snapshot helper failed (rc=${snapshot_result.rc}): ${snapshot_result.stderr}
|
|
Should Not Be Empty ${snapshot_result.stdout}
|
|
${snapshot}= Parse Json Payload ${snapshot_result.stdout}
|
|
RETURN ${snapshot}
|
|
|
|
Verify WF04 Child Plan Spawning
|
|
[Documentation] AC-4: verify exactly 4 child plans mapped one-per-project.
|
|
[Arguments] ${snapshot}
|
|
${subplans}= Evaluate $snapshot.get('subplans', [])
|
|
${subplan_count}= Evaluate len($subplans)
|
|
IF ${subplan_count} == 0
|
|
Skip LLM produced no child plans - AC-4 child-plan mapping assertions were not exercised
|
|
END
|
|
Should Be Equal As Integers ${subplan_count} 4 msg=Expected exactly 4 child plans (common-lib + 3 services), found ${subplan_count}
|
|
${mapped_projects}= Evaluate sorted({p for sp in $subplans for p in sp.get('mapped_projects', []) if p})
|
|
${expected_projects}= Create List ${LIB_PROJECT} ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
|
|
${expected_sorted}= Evaluate sorted($expected_projects)
|
|
${all_projects_covered}= Evaluate $mapped_projects == $expected_sorted
|
|
Should Be True ${all_projects_covered}
|
|
... Child plans should cover all 4 projects. expected=${expected_sorted} actual=${mapped_projects}
|
|
${single_mapping}= Evaluate all(len(sp.get('mapped_projects', [])) == 1 for sp in $subplans)
|
|
Should Be True ${single_mapping}
|
|
... Each child plan should map to exactly one project scope
|
|
|
|
Verify WF04 Execution Order
|
|
[Documentation] AC-5: common-lib executes before service subplans.
|
|
[Arguments] ${snapshot}
|
|
${subplans}= Evaluate $snapshot.get('subplans', [])
|
|
${subplan_count}= Evaluate len($subplans)
|
|
IF ${subplan_count} == 0
|
|
Skip LLM produced no child plans - AC-5 execution-order assertions were not exercised
|
|
END
|
|
${lib_subplans}= Evaluate [sp for sp in $subplans if '${LIB_PROJECT}' in sp.get('mapped_projects', [])]
|
|
${lib_count}= Evaluate len($lib_subplans)
|
|
Should Be Equal As Integers ${lib_count} 1 msg=Expected exactly one common-lib child plan, found ${lib_count}
|
|
${lib_completed}= Evaluate $lib_subplans[0].get('completed_at') or $lib_subplans[0].get('execute_completed_at') or ''
|
|
Should Not Be Empty ${lib_completed}
|
|
... common-lib child plan completion timestamp is required for ordering checks
|
|
${service_projects}= Create List ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
|
|
${svc_subplans}= Evaluate [sp for sp in $subplans if any(p in $service_projects for p in sp.get('mapped_projects', []))]
|
|
${svc_count}= Evaluate len($svc_subplans)
|
|
Should Be Equal As Integers ${svc_count} 3 msg=Expected exactly three service child plans, found ${svc_count}
|
|
${svc_starts_present}= Evaluate all((sp.get('started_at') or sp.get('execute_started_at') or '') != '' for sp in $svc_subplans)
|
|
Should Be True ${svc_starts_present}
|
|
... Service child plans must expose start timestamps for execution-order checks
|
|
${services_after_lib}= Evaluate all((sp.get('started_at') or sp.get('execute_started_at')) >= $lib_completed for sp in $svc_subplans)
|
|
Should Be True ${services_after_lib}
|
|
... Service execution must start only after common-lib execution completes
|
|
|
|
Verify WF04 Validation Outcomes
|
|
[Documentation] AC-6: each child plan must report validation pass results.
|
|
[Arguments] ${snapshot}
|
|
${subplans}= Evaluate $snapshot.get('subplans', [])
|
|
${subplan_count}= Evaluate len($subplans)
|
|
IF ${subplan_count} == 0
|
|
Skip LLM produced no child plans - AC-6 per-project validation assertions were not exercised
|
|
END
|
|
${validation_present}= Evaluate all(isinstance(sp.get('child_validation_summary'), dict) and len(sp.get('child_validation_summary')) > 0 for sp in $subplans)
|
|
Should Be True ${validation_present}
|
|
... Each child plan must expose a non-empty validation summary
|
|
${validation_passed}= Evaluate all(int((sp.get('child_validation_summary') or {}).get('required_passed', 0) or 0) >= 1 and int((sp.get('child_validation_summary') or {}).get('required_failed', 0) or 0) == 0 for sp in $subplans)
|
|
Should Be True ${validation_passed}
|
|
... Each child plan must pass required validations (required_failed == 0)
|
|
|
|
Verify WF04 Apply Order
|
|
[Documentation] AC-7: common-lib apply must complete before service applies.
|
|
[Arguments] ${snapshot}
|
|
${subplans}= Evaluate $snapshot.get('subplans', [])
|
|
${subplan_count}= Evaluate len($subplans)
|
|
IF ${subplan_count} == 0
|
|
Skip LLM produced no child plans - AC-7 apply-order assertions were not exercised
|
|
END
|
|
${lib_subplans}= Evaluate [sp for sp in $subplans if '${LIB_PROJECT}' in sp.get('mapped_projects', [])]
|
|
${lib_count}= Evaluate len($lib_subplans)
|
|
Should Be Equal As Integers ${lib_count} 1 msg=Expected exactly one common-lib child plan, found ${lib_count}
|
|
${lib_applied}= Evaluate $lib_subplans[0].get('applied_at') or $lib_subplans[0].get('child_updated_at') or ''
|
|
Should Not Be Empty ${lib_applied}
|
|
... common-lib child plan apply timestamp is required for apply-order checks
|
|
${service_projects}= Create List ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
|
|
${svc_subplans}= Evaluate [sp for sp in $subplans if any(p in $service_projects for p in sp.get('mapped_projects', []))]
|
|
${svc_count}= Evaluate len($svc_subplans)
|
|
Should Be Equal As Integers ${svc_count} 3 msg=Expected exactly three service child plans, found ${svc_count}
|
|
${svc_apply_present}= Evaluate all((sp.get('applied_at') or sp.get('child_updated_at') or '') != '' for sp in $svc_subplans)
|
|
Should Be True ${svc_apply_present}
|
|
... Service child plans must expose apply/updated timestamps for apply-order checks
|
|
${services_after_lib_apply}= Evaluate all((sp.get('applied_at') or sp.get('child_updated_at')) >= $lib_applied for sp in $svc_subplans)
|
|
Should Be True ${services_after_lib_apply}
|
|
... Service applies must occur after common-lib apply
|
|
|
|
Count Decision Nodes
|
|
[Documentation] Recursively count decision nodes in a plan tree JSON structure.
|
|
... Invokes ``wf04_snapshot_helper.py --count-nodes`` as a subprocess
|
|
... to avoid importing the application DI container into the Robot
|
|
... test runner process.
|
|
[Arguments] ${tree_payload}
|
|
${tree_json}= Evaluate __import__('json').dumps($tree_payload)
|
|
${tmp_path}= Evaluate __import__('tempfile').NamedTemporaryFile(mode='w', suffix='.json', delete=False).name
|
|
Evaluate __import__('pathlib').Path(r'${tmp_path}').write_text($tree_json, encoding='utf-8')
|
|
${result}= Run Process
|
|
... ${PYTHON} ${WF04_SNAPSHOT_HELPER} --count-nodes ${tmp_path}
|
|
... timeout=30s on_timeout=kill
|
|
Evaluate __import__('os').unlink(r'${tmp_path}')
|
|
Should Be Equal As Integers ${result.rc} 0 msg=count-nodes failed (rc=${result.rc}): ${result.stderr}
|
|
${count}= Convert To Integer ${result.stdout.strip()}
|
|
RETURN ${count}
|
|
|
|
Verify Plan In List
|
|
[Documentation] Verify a plan appears in list output.
|
|
[Arguments] ${plan_id}
|
|
${list_result}= Run CleverAgents Command
|
|
... plan list --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${list_result.rc} 0 msg=list failed (rc=${list_result.rc}): ${list_result.stderr}
|
|
Output Should Contain ${list_result} ${plan_id}
|
|
|
|
*** Test Cases ***
|
|
WF04 Multi Project Dependency Update Supervised Profile
|
|
[Documentation] Full supervised-profile workflow: register 4 repos,
|
|
... create multi-project action with invariants, plan use
|
|
... targeting all 4 projects with --automation-profile supervised,
|
|
... execute with child plan spawning in dependency order,
|
|
... verify per-project validation, and apply in dependency order.
|
|
[Tags] tdd_issue tdd_issue_4189
|
|
[Timeout] 25 minutes
|
|
[Teardown] WF04 Test Teardown
|
|
Skip If No LLM Keys
|
|
# Initialise test variable for teardown access.
|
|
Set Test Variable ${WF04_PLAN_ID} ${EMPTY}
|