forked from cleveragents/cleveragents-core
brent.edwards
998aaf25f8
Centralize E2E initialization in common suite setup so DB-dependent CLI commands no longer rely on per-suite or per-test init workarounds. This also sanitizes suite-home names to prevent invalid path-derived initialization failures in isolated Robot runs. ISSUES CLOSED: #1023
570 lines
34 KiB
Plaintext
570 lines
34 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E test for Workflow Example 5: Database Schema Migration with Safety Nets.
|
|
...
|
|
... Advanced scenario using the **review** automation profile.
|
|
... Registers a custom resource type (postgres-db), creates custom
|
|
... skills with spec-aligned database tools (query_db, execute_migration,
|
|
... backfill_column), exercises phased child plan execution via
|
|
... ``plan tree``, attempts checkpoint-based rollback via
|
|
... ``plan rollback``, and verifies migration changes after apply.
|
|
...
|
|
... Zero mocking — real CLI, real LLM API keys.
|
|
Resource common_e2e.resource
|
|
Suite Setup WF05 Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
Force Tags E2E
|
|
|
|
*** Variables ***
|
|
${ACTION_NAME} local/wf05-db-migration
|
|
${SKILL_NAME} local/wf05-db-tools
|
|
${RESOURCE_TYPE_NAME} local/wf05-postgres-db
|
|
|
|
*** Keywords ***
|
|
WF05 Suite Setup
|
|
[Documentation] E2E Suite Setup plus dynamic actor selection.
|
|
E2E Suite Setup
|
|
# Generate unique suffix for resource/project names to avoid UNIQUE
|
|
# constraint collisions on repeated E2E runs against the same database.
|
|
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
|
|
Set Suite Variable ${RUN_SUFFIX} ${suffix}
|
|
# Pick an actor that matches available API keys.
|
|
# Prefer OpenAI first to reduce Anthropic credit-quota flakiness.
|
|
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
|
|
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
|
|
IF ${has_openai}
|
|
${actor}= Set Variable openai/gpt-4o
|
|
ELSE IF ${has_anthropic}
|
|
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
|
|
ELSE
|
|
${actor}= Set Variable openai/gpt-4o
|
|
END
|
|
Set Suite Variable ${WF05_ACTOR} ${actor}
|
|
|
|
Create DB App Repo
|
|
[Documentation] Create temp git repo with a Python app simulating
|
|
... a database schema and application code.
|
|
${repo}= Create Temp Git Repo wf05-db-app-${RUN_SUFFIX}
|
|
Create Directory ${repo}${/}src
|
|
Create Directory ${repo}${/}migrations
|
|
Create Directory ${repo}${/}tests
|
|
${schema_content}= Catenate SEPARATOR=\n
|
|
... """Database schema — users table (missing last_login_at)."""
|
|
... ${EMPTY}
|
|
... USERS_SCHEMA = {
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"table": "users",
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"columns": [
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "id", "type": "INTEGER", "primary_key": True},
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "email", "type": "VARCHAR(255)"},
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "created_at", "type": "TIMESTAMP"},
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}],
|
|
... }
|
|
Create File ${repo}${/}src${/}schema.py ${schema_content}
|
|
${app_content}= Catenate SEPARATOR=\n
|
|
... """Application code — user service."""
|
|
... from src.schema import USERS_SCHEMA
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def get_user(user_id):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Fetch a user by ID."""
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"id": user_id, "email": "user@example.com"}
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def get_user_activity(user_id):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Get user activity — needs last_login_at but column is missing."""
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}user = get_user(user_id)
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}# BUG: no last_login_at field available
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"user": user, "last_login": None}
|
|
Create File ${repo}${/}src${/}app.py ${app_content}
|
|
${test_content}= Catenate SEPARATOR=\n
|
|
... """Tests for user service."""
|
|
... from src.app import get_user, get_user_activity
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def test_get_user():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}user = get_user(1)
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}assert user["id"] == 1
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def test_get_user_activity():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}result = get_user_activity(1)
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}assert result["last_login"] is None
|
|
Create File ${repo}${/}tests${/}test_app.py ${test_content}
|
|
Create File ${repo}${/}src${/}__init__.py \n
|
|
Create File ${repo}${/}tests${/}__init__.py \n
|
|
Create File ${repo}${/}migrations${/}__init__.py \n
|
|
Create File ${repo}${/}requirements.txt pytest>=7.0\n
|
|
${git_add}= Run Process git add . cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
|
|
${git_commit}= Run Process git commit -m Initial DB app with missing last_login_at cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
|
|
RETURN ${repo}
|
|
|
|
WF05 Test Teardown
|
|
[Documentation] Log diagnostic context on failure for debugging.
|
|
${plan_id}= Get Variable Value ${WF05_PLAN_ID} ${EMPTY}
|
|
IF '${plan_id}' != ''
|
|
${status} ${result}= Run Keyword And Ignore Error
|
|
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
|
|
IF '${status}' == 'PASS'
|
|
Log Teardown plan status: ${result.stdout} WARN
|
|
END
|
|
${tree_status} ${tree_result}= Run Keyword And Ignore Error
|
|
... Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=30s
|
|
IF '${tree_status}' == 'PASS'
|
|
Log Teardown plan tree: ${tree_result.stdout} WARN
|
|
END
|
|
END
|
|
|
|
*** Test Cases ***
|
|
WF05 Database Schema Migration With Safety Nets Review Profile
|
|
[Documentation] Full review-profile workflow: register custom resource type,
|
|
... register git-checkout resource, create custom skill with
|
|
... spec-aligned DB tools (query_db, execute_migration,
|
|
... backfill_column), create migration action with review
|
|
... automation profile, exercise phased child plan execution,
|
|
... attempt checkpoint-based rollback, verify migration changes.
|
|
[Timeout] 30 minutes
|
|
[Teardown] WF05 Test Teardown
|
|
Skip If No LLM Keys
|
|
# Initialise test variable for teardown access.
|
|
Set Test Variable ${WF05_PLAN_ID} ${EMPTY}
|
|
|
|
# ── 1. Fixture: create temp repo ──
|
|
${repo}= Create DB App Repo
|
|
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${branch_result.rc} 0 git rev-parse failed: ${branch_result.stderr}
|
|
${branch}= Strip String ${branch_result.stdout}
|
|
|
|
# ── 2. Register custom resource type (AC #2: custom resource type via CLI) ──
|
|
${res_type_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${RESOURCE_TYPE_NAME}
|
|
... description: PostgreSQL database resource for WF05 E2E test
|
|
... resource_kind: physical
|
|
... sandbox_strategy: transaction_rollback
|
|
... user_addable: true
|
|
... cli_args:
|
|
... ${SPACE}${SPACE}- name: host
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Database hostname
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
|
... ${SPACE}${SPACE}- name: port
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}required: false
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Port (default 5432)
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: integer
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}default: 5432
|
|
... ${SPACE}${SPACE}- name: database
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Database name
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
|
... ${SPACE}${SPACE}- name: schema
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}required: false
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Schema (default public)
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}default: public
|
|
... capabilities:
|
|
... ${SPACE}${SPACE}read: true
|
|
... ${SPACE}${SPACE}write: true
|
|
... ${SPACE}${SPACE}sandbox: true
|
|
... ${SPACE}${SPACE}checkpoint: true
|
|
${res_type_path}= Set Variable ${SUITE_HOME}${/}wf05_resource_type.yaml
|
|
Create File ${res_type_path} ${res_type_yaml}
|
|
${r_type}= Run CleverAgents Command
|
|
... resource type add --config ${res_type_path} --format json
|
|
... expected_rc=None
|
|
Should Be Equal As Integers ${r_type.rc} 0
|
|
... resource type add failed (rc=${r_type.rc}): ${r_type.stderr}
|
|
Should Not Contain ${r_type.stdout}${r_type.stderr} Traceback
|
|
Should Not Contain ${r_type.stdout}${r_type.stderr} INTERNAL
|
|
Output Should Contain ${r_type} ${RESOURCE_TYPE_NAME}
|
|
|
|
# ── 3. Register git-checkout resource and create project ──
|
|
${res_name}= Set Variable wf05-res-${RUN_SUFFIX}
|
|
${r_res}= Run CleverAgents Command
|
|
... resource add git-checkout ${res_name}
|
|
... --path ${repo} --branch ${branch}
|
|
Should Be Equal As Integers ${r_res.rc} 0
|
|
... resource add failed (rc=${r_res.rc}): ${r_res.stderr}
|
|
Should Not Contain ${r_res.stdout}${r_res.stderr} Traceback
|
|
Should Not Contain ${r_res.stdout}${r_res.stderr} INTERNAL
|
|
Output Should Contain ${r_res} ${res_name}
|
|
|
|
${proj_name}= Set Variable wf05-proj-${RUN_SUFFIX}
|
|
${r_proj}= Run CleverAgents Command
|
|
... project create --resource ${res_name} ${proj_name}
|
|
Should Be Equal As Integers ${r_proj.rc} 0
|
|
... project create failed (rc=${r_proj.rc}): ${r_proj.stderr}
|
|
Should Not Contain ${r_proj.stdout}${r_proj.stderr} Traceback
|
|
Should Not Contain ${r_proj.stdout}${r_proj.stderr} INTERNAL
|
|
Output Should Contain ${r_proj} ${proj_name}
|
|
|
|
# ── 4. Instantiate custom postgres-db resource and link to project ──
|
|
${db_res_name}= Set Variable wf05-db-${RUN_SUFFIX}
|
|
${r_db_res}= Run CleverAgents Command
|
|
... resource add ${RESOURCE_TYPE_NAME} ${db_res_name}
|
|
... --host localhost --database wf05_test_db
|
|
... expected_rc=None
|
|
IF ${r_db_res.rc} == 0
|
|
Output Should Contain ${r_db_res} ${db_res_name}
|
|
Log Custom postgres-db resource instance created: ${db_res_name}
|
|
# Link custom DB resource to project (spec Step 1: project link-resource)
|
|
${r_link_db}= Run CleverAgents Command
|
|
... project link-resource ${proj_name} ${db_res_name}
|
|
... expected_rc=None
|
|
IF ${r_link_db.rc} == 0
|
|
Log Custom DB resource linked to project: ${db_res_name}
|
|
ELSE
|
|
${link_err}= Set Variable ${r_link_db.stdout}${r_link_db.stderr}
|
|
Should Not Contain ${link_err} Traceback
|
|
${link_has_no_such_option}= Evaluate 'No such option' in $link_err or 'NoSuchOption' in $link_err
|
|
IF not ${link_has_no_such_option}
|
|
Should Not Contain ${link_err} INTERNAL
|
|
END
|
|
Log project link-resource returned rc=${r_link_db.rc} (may require additional CLI support) WARN
|
|
END
|
|
ELSE
|
|
${db_err}= Set Variable ${r_db_res.stdout}${r_db_res.stderr}
|
|
Should Not Contain ${db_err} Traceback
|
|
${db_has_no_such_option}= Evaluate 'No such option' in $db_err or 'NoSuchOption' in $db_err
|
|
IF not ${db_has_no_such_option}
|
|
Should Not Contain ${db_err} INTERNAL
|
|
END
|
|
Log Custom resource instantiation returned rc=${r_db_res.rc} (may require additional CLI support) WARN
|
|
END
|
|
|
|
# ── 5. Create custom skill with spec-aligned DB tools (AC #3) ──
|
|
# Note: writes/checkpointable are schema-validated at registration time.
|
|
# The persisted Skill model currently stores tool refs by namespaced name.
|
|
${skill_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${SKILL_NAME}
|
|
... description: Safe database operations with transaction support
|
|
... tools:
|
|
... ${SPACE}${SPACE}- name: local/query_db
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Execute a read-only SQL query and return results
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}writes: false
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}checkpointable: false
|
|
... ${SPACE}${SPACE}- name: local/execute_migration
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Execute a DDL migration within a transaction
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}writes: true
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}checkpointable: true
|
|
... ${SPACE}${SPACE}- name: local/backfill_column
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Batch-update a column using a source query
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}writes: true
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}checkpointable: true
|
|
${skill_path}= Set Variable ${SUITE_HOME}${/}wf05_skill.yaml
|
|
Create File ${skill_path} ${skill_yaml}
|
|
${r_skill}= Run CleverAgents Command
|
|
... skill add --config ${skill_path} --format json
|
|
... expected_rc=None
|
|
Should Be Equal As Integers ${r_skill.rc} 0
|
|
... skill add failed (rc=${r_skill.rc}): ${r_skill.stderr}
|
|
Should Not Contain ${r_skill.stdout}${r_skill.stderr} Traceback
|
|
Should Not Contain ${r_skill.stdout}${r_skill.stderr} INTERNAL
|
|
Output Should Contain ${r_skill} ${SKILL_NAME}
|
|
# Verify individual tool names are registered
|
|
Output Should Contain ${r_skill} local/query_db
|
|
Output Should Contain ${r_skill} local/execute_migration
|
|
Output Should Contain ${r_skill} local/backfill_column
|
|
|
|
# ── 6. Create action with review automation profile ──
|
|
${action_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${ACTION_NAME}
|
|
... description: Add last_login_at column to users table with backfill and app update
|
|
... definition_of_done: |
|
|
... ${SPACE}${SPACE}Database migration adds the column with a sensible default.
|
|
... ${SPACE}${SPACE}Backfill completes for all rows using the specified source.
|
|
... ${SPACE}${SPACE}Application code reads/writes the new column.
|
|
... ${SPACE}${SPACE}All tests pass with the new schema.
|
|
... ${SPACE}${SPACE}A rollback migration is available and tested.
|
|
... strategy_actor: ${WF05_ACTOR}
|
|
... execution_actor: ${WF05_ACTOR}
|
|
... automation_profile: review
|
|
... reusable: true
|
|
... state: available
|
|
... invariants:
|
|
... ${SPACE}${SPACE}- Migration must be backward-compatible (add column, don't rename or drop)
|
|
... ${SPACE}${SPACE}- Backfill must be batched to avoid locking the table
|
|
... ${SPACE}${SPACE}- Rollback migration must be provided and tested
|
|
... ${SPACE}${SPACE}- Application code must handle both old (null) and new values gracefully
|
|
... arguments:
|
|
... ${SPACE}${SPACE}- name: table_name
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Target table for schema migration
|
|
... ${SPACE}${SPACE}- name: column_name
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: New column to add
|
|
... ${SPACE}${SPACE}- name: column_type
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: SQL type for the new column
|
|
... ${SPACE}${SPACE}- name: backfill_source
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Where to source backfill data
|
|
${action_path}= Set Variable ${SUITE_HOME}${/}wf05_action.yaml
|
|
Create File ${action_path} ${action_yaml}
|
|
${r_action}= Run CleverAgents Command
|
|
... action create --config ${action_path} --format json
|
|
... expected_rc=None
|
|
Should Be Equal As Integers ${r_action.rc} 0
|
|
... action create failed (rc=${r_action.rc}): ${r_action.stderr}
|
|
Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback
|
|
Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL
|
|
Output Should Contain ${r_action} ${ACTION_NAME}
|
|
|
|
# ── 7. Plan use — create plan from action + project with args ──
|
|
${r_use}= Run CleverAgents Command
|
|
... plan use ${ACTION_NAME} ${proj_name}
|
|
... --arg table_name=users --arg column_name=last_login_at
|
|
... --arg column_type=TIMESTAMP --arg backfill_source=audit_log
|
|
... --automation-profile review
|
|
... --format json expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${r_use.rc} 0
|
|
... plan use failed (rc=${r_use.rc}): ${r_use.stderr}
|
|
Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback
|
|
Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL
|
|
${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id
|
|
Should Not Be Empty ${plan_id} Could not extract plan_id from plan use output
|
|
Set Test Variable ${WF05_PLAN_ID} ${plan_id}
|
|
Log Plan ID: ${plan_id}
|
|
# Verify review automation profile was applied.
|
|
# If plan use output omits automation_profile, fall back to plan status.
|
|
${resolved_profile}= Safe Parse Json Field ${r_use.stdout} automation_profile
|
|
${resolved_profile}= Set Variable If $resolved_profile is None ${EMPTY} ${resolved_profile}
|
|
${resolved_profile}= Set Variable If '${resolved_profile}' == 'None' ${EMPTY} ${resolved_profile}
|
|
IF '${resolved_profile}' == ''
|
|
${r_profile_status}= Run CleverAgents Command
|
|
... plan status ${plan_id} --format json
|
|
... expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${r_profile_status.rc} 0
|
|
... plan status for profile verification failed (rc=${r_profile_status.rc}): ${r_profile_status.stderr}
|
|
Should Not Contain ${r_profile_status.stdout}${r_profile_status.stderr} Traceback
|
|
Should Not Contain ${r_profile_status.stdout}${r_profile_status.stderr} INTERNAL
|
|
${resolved_profile}= Safe Parse Json Field ${r_profile_status.stdout} automation_profile
|
|
${resolved_profile}= Set Variable If $resolved_profile is None ${EMPTY} ${resolved_profile}
|
|
${resolved_profile}= Set Variable If '${resolved_profile}' == 'None' ${EMPTY} ${resolved_profile}
|
|
END
|
|
Should Be Equal As Strings ${resolved_profile} review
|
|
... Expected automation_profile 'review' but got '${resolved_profile}'
|
|
|
|
# ── 8. Strategize ──
|
|
${r_strat}= Run CleverAgents Command
|
|
... plan execute ${plan_id} --format json
|
|
... expected_rc=None timeout=300s
|
|
Should Be Equal As Integers ${r_strat.rc} 0
|
|
... plan execute (strategize) failed (rc=${r_strat.rc}): ${r_strat.stderr}
|
|
Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback
|
|
Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL
|
|
Output Should Contain ${r_strat} ${plan_id}
|
|
Log Strategize stdout: ${r_strat.stdout}
|
|
|
|
# ── 9. Decision tree — verify phased child plans (AC #4) ──
|
|
${r_tree}= Run CleverAgents Command
|
|
... plan tree ${plan_id} --format json
|
|
... expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${r_tree.rc} 0
|
|
... plan tree failed (rc=${r_tree.rc}): ${r_tree.stderr}
|
|
Should Not Contain ${r_tree.stdout}${r_tree.stderr} Traceback
|
|
Should Not Contain ${r_tree.stdout}${r_tree.stderr} INTERNAL
|
|
Should Not Be Empty ${r_tree.stdout} Plan tree output should not be empty
|
|
# Parse JSON and assert structural evidence of phased decomposition.
|
|
# plan tree output may include a log line prefix before JSON payload.
|
|
${tree_json_str}= Evaluate
|
|
... (lambda s: (lambda lines: (lambda idx: ('\\n'.join(lines[idx:]).strip() if idx is not None else s.strip()))(next((i for i,l in enumerate(lines) if l.lstrip().startswith('[') or l.lstrip().startswith('{')), None)))([ln for ln in s.splitlines() if ln.strip()]))($r_tree.stdout)
|
|
${tree_parse_status} ${tree_json}= Run Keyword And Ignore Error
|
|
... Evaluate __import__('json').loads($tree_json_str)
|
|
Should Be Equal As Strings ${tree_parse_status} PASS
|
|
... plan tree output must be parseable JSON (error: ${tree_json})
|
|
${decision_count}= Evaluate
|
|
... (lambda root: (lambda walk: walk(walk, root))(lambda self, node: (1 if isinstance(node, dict) and str(node.get('decision_id', '')).strip() != '' else 0) + sum(self(self, child) for child in ((node.get('children') if isinstance(node, dict) else []) or [])) + (sum(self(self, item) for item in node) if isinstance(node, list) else 0)))($tree_json)
|
|
Log Decision tree contains ${decision_count} decision node(s)
|
|
Should Be True ${decision_count} >= 2
|
|
... Plan tree should contain at least 2 decision nodes (found ${decision_count})
|
|
${children_key_count}= Evaluate __import__('json').dumps($tree_json).count('"children"')
|
|
Should Be True ${children_key_count} >= 1
|
|
... Plan tree should include structured children fields
|
|
${child_link_count}= Evaluate
|
|
... (lambda root: (lambda walk: walk(walk, root))(lambda self, node: ((len((node.get('children') if isinstance(node, dict) else []) or [])) if isinstance(node, dict) else 0) + sum(self(self, child) for child in ((node.get('children') if isinstance(node, dict) else []) or [])) + (sum(self(self, item) for item in node) if isinstance(node, list) else 0)))($tree_json)
|
|
IF ${decision_count} < 3
|
|
Log Decision count (${decision_count}) is below 3 — decomposition may be minimal in this run WARN
|
|
END
|
|
IF ${child_link_count} < 1
|
|
Log Decision tree has no child links in this run — phased decomposition may be minimal WARN
|
|
END
|
|
|
|
# ── 10. Execute ──
|
|
${r_exec}= Run CleverAgents Command
|
|
... plan execute ${plan_id} --format json
|
|
... expected_rc=None timeout=300s
|
|
IF ${r_exec.rc} != 0
|
|
Fail plan execute failed (rc=${r_exec.rc}) stdout=${r_exec.stdout} stderr=${r_exec.stderr}
|
|
END
|
|
Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback
|
|
Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL
|
|
Output Should Contain ${r_exec} ${plan_id}
|
|
Log Execute stdout: ${r_exec.stdout}
|
|
|
|
# ── 11. Plan status — verify phase and check for checkpoints ──
|
|
${r_status}= Run CleverAgents Command
|
|
... plan status ${plan_id} --format json
|
|
... expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${r_status.rc} 0
|
|
... plan status failed (rc=${r_status.rc}): ${r_status.stderr}
|
|
Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback
|
|
Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL
|
|
Should Not Be Empty ${r_status.stdout}
|
|
Output Should Contain ${r_status} ${plan_id}
|
|
# Assert phase field is populated after execute
|
|
${status_phase}= Safe Parse Json Field ${r_status.stdout} phase
|
|
Should Not Be Empty ${status_phase} Plan phase should be populated after execute
|
|
Log Plan phase after execute: ${status_phase}
|
|
# Extract checkpoint ID if available (for rollback attempt)
|
|
${checkpoint_id}= Safe Parse Json Field ${r_status.stdout} last_checkpoint_id
|
|
# Guard against Python None being returned as string "None" from JSON null
|
|
${checkpoint_id}= Set Variable If $checkpoint_id is None ${EMPTY} ${checkpoint_id}
|
|
${checkpoint_id}= Set Variable If '${checkpoint_id}' == 'None' ${EMPTY} ${checkpoint_id}
|
|
Log Checkpoint ID from status: ${checkpoint_id}
|
|
|
|
# ── 12. Checkpoint rollback attempt (AC #5) ──
|
|
IF '${checkpoint_id}' != ''
|
|
Log Attempting plan rollback to checkpoint ${checkpoint_id}
|
|
${r_rollback}= Run CleverAgents Command
|
|
... plan rollback --yes ${plan_id} ${checkpoint_id}
|
|
... --format json expected_rc=None timeout=120s
|
|
Log Rollback rc=${r_rollback.rc} stdout=${r_rollback.stdout} stderr=${r_rollback.stderr}
|
|
Should Be Equal As Integers ${r_rollback.rc} 0
|
|
... plan rollback to checkpoint ${checkpoint_id} failed (rc=${r_rollback.rc}): ${r_rollback.stderr}
|
|
Should Not Contain ${r_rollback.stdout}${r_rollback.stderr} Traceback
|
|
Should Not Contain ${r_rollback.stdout}${r_rollback.stderr} INTERNAL
|
|
Output Should Contain ${r_rollback} ${plan_id}
|
|
Log Rollback succeeded — re-executing plan to continue lifecycle
|
|
# Re-execute after rollback so we can proceed to diff/apply
|
|
${r_reexec}= Run CleverAgents Command
|
|
... plan execute ${plan_id} --format json
|
|
... expected_rc=None timeout=300s
|
|
Log Re-execute after rollback rc=${r_reexec.rc}
|
|
IF ${r_reexec.rc} != 0
|
|
# Re-execute failure is non-fatal: the plan may still be in an applicable
|
|
# state from the original execute, so we continue to diff/apply rather
|
|
# than aborting. If those steps also fail the test will still surface a
|
|
# clear failure at the relevant assertion.
|
|
Log Re-execute after rollback failed (rc=${r_reexec.rc}); continuing to diff/apply WARN
|
|
ELSE
|
|
Should Not Contain ${r_reexec.stdout}${r_reexec.stderr} Traceback
|
|
Should Not Contain ${r_reexec.stdout}${r_reexec.stderr} INTERNAL
|
|
END
|
|
ELSE
|
|
Log AC #5 visibility: no checkpoint_id present, so real rollback path was not executed in this run WARN
|
|
END
|
|
|
|
# Also validate graceful failure path on a fake checkpoint ID.
|
|
${r_rollback_noop}= Run CleverAgents Command
|
|
... plan rollback --yes ${plan_id} no-such-checkpoint
|
|
... --format json expected_rc=None timeout=60s
|
|
Log Rollback (no checkpoint) rc=${r_rollback_noop.rc} stderr=${r_rollback_noop.stderr}
|
|
Should Not Be Equal As Integers ${r_rollback_noop.rc} 0
|
|
... plan rollback with fake checkpoint should fail gracefully (got rc=0)
|
|
Should Not Contain ${r_rollback_noop.stdout}${r_rollback_noop.stderr} Traceback
|
|
Should Not Contain ${r_rollback_noop.stdout}${r_rollback_noop.stderr} INTERNAL
|
|
|
|
# ── 13. Diff — verify changeset is non-empty (AC #6: migration verification) ──
|
|
${r_diff}= Run CleverAgents Command
|
|
... plan diff ${plan_id} --format json
|
|
... expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${r_diff.rc} 0
|
|
... plan diff failed (rc=${r_diff.rc}): ${r_diff.stderr}
|
|
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
|
|
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
|
|
Should Not Be Empty ${r_diff.stdout} Plan diff produced no output
|
|
${diff_preview_lower}= Evaluate ($r_diff.stdout).lower()
|
|
${has_diff_signal}= Evaluate 'migration' in $diff_preview_lower or 'backfill' in $diff_preview_lower or 'last_login' in $diff_preview_lower or 'column' in $diff_preview_lower or 'audit_log' in $diff_preview_lower or 'diff' in $diff_preview_lower or 'change' in $diff_preview_lower or 'file' in $diff_preview_lower
|
|
Should Be True ${has_diff_signal}
|
|
... plan diff output should include meaningful diff/change indicators before apply
|
|
|
|
# ── 14. Apply ──
|
|
# Save baseline SHA before apply for accurate diff comparison
|
|
${baseline_sha_result}= Run Process git rev-parse HEAD cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${baseline_sha_result.rc} 0
|
|
... git rev-parse HEAD failed: ${baseline_sha_result.stderr}
|
|
${baseline_sha}= Strip String ${baseline_sha_result.stdout}
|
|
Log Baseline SHA before apply: ${baseline_sha}
|
|
|
|
${r_apply}= Run CleverAgents Command
|
|
... plan apply --yes ${plan_id} --format json
|
|
... expected_rc=None timeout=180s
|
|
Log Apply rc=${r_apply.rc} stdout=${r_apply.stdout}
|
|
IF ${r_apply.rc} == 0
|
|
Output Should Contain ${r_apply} ${plan_id}
|
|
ELSE
|
|
Fail apply failed (rc=${r_apply.rc}) stdout=${r_apply.stdout} stderr=${r_apply.stderr}
|
|
END
|
|
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
|
|
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
|
|
|
|
# Verify terminal state after apply.
|
|
${r_status_after_apply}= Run CleverAgents Command
|
|
... plan status ${plan_id} --format json
|
|
... expected_rc=None timeout=120s
|
|
Should Be Equal As Integers ${r_status_after_apply.rc} 0
|
|
... plan status after apply failed (rc=${r_status_after_apply.rc}): ${r_status_after_apply.stderr}
|
|
Should Not Contain ${r_status_after_apply.stdout}${r_status_after_apply.stderr} Traceback
|
|
Should Not Contain ${r_status_after_apply.stdout}${r_status_after_apply.stderr} INTERNAL
|
|
${apply_phase}= Safe Parse Json Field ${r_status_after_apply.stdout} phase
|
|
${apply_state}= Safe Parse Json Field ${r_status_after_apply.stdout} processing_state
|
|
Should Not Be Empty ${apply_phase} plan status after apply should include phase
|
|
Should Not Be Empty ${apply_state} plan status after apply should include processing_state
|
|
${apply_phase_lower}= Evaluate ($apply_phase).lower()
|
|
IF 'apply' not in $apply_phase_lower
|
|
Log Post-apply phase is '${apply_phase}' instead of apply; treating terminal state as authoritative WARN
|
|
END
|
|
${is_terminal_state}= Evaluate ($apply_state.lower() in ['applied', 'constrained', 'errored', 'cancelled', 'complete'])
|
|
${is_apply_progress_state}= Evaluate ('apply' in $apply_phase_lower)
|
|
Should Be True ${is_terminal_state} or ${is_apply_progress_state}
|
|
... expected apply to produce terminal state or apply-phase progress (phase=${apply_phase}, state=${apply_state})
|
|
IF not ${is_terminal_state}
|
|
Log Post-apply state '${apply_state}' is non-terminal; apply may complete asynchronously WARN
|
|
END
|
|
|
|
# ── 15. Verify repo state and migration content (AC #6) ──
|
|
${log_result}= Run Process git log --oneline -10 cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${log_result.rc} 0
|
|
... git log failed (rc=${log_result.rc}): ${log_result.stderr}
|
|
Log Git log: ${log_result.stdout}
|
|
${line_count}= Get Line Count ${log_result.stdout}
|
|
# Hard assert minimum fixture commits exist; LLM-generated commits are
|
|
# non-deterministic, so we only hard-assert the fixture baseline (2 commits:
|
|
# Create Temp Git Repo initial + DB app fixture).
|
|
Should Be True ${line_count} >= 2
|
|
... Expected at least 2 commits (fixture baseline), got ${line_count}
|
|
IF ${line_count} < 3
|
|
Log No new commits from apply (${line_count} total) — LLM may not have produced file changes WARN
|
|
END
|
|
|
|
# Diff against baseline SHA to capture only changes from apply
|
|
${diff_all}= Run Process git diff ${baseline_sha} HEAD --name-only cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${diff_all.rc} 0
|
|
... git diff --name-only failed (rc=${diff_all.rc}): ${diff_all.stderr}
|
|
Log Files changed since apply: ${diff_all.stdout}
|
|
${full_diff}= Run Process git diff ${baseline_sha} HEAD cwd=${repo} timeout=60s on_timeout=kill
|
|
Should Be Equal As Integers ${full_diff.rc} 0
|
|
... git diff failed (rc=${full_diff.rc}): ${full_diff.stderr}
|
|
${migration_evidence_text}= Evaluate ($full_diff.stdout + ' ' + $r_diff.stdout).lower()
|
|
${has_migration_content}= Evaluate 'last_login' in $migration_evidence_text or 'schema' in $migration_evidence_text or 'migration' in $migration_evidence_text or 'column' in $migration_evidence_text or 'alter' in $migration_evidence_text
|
|
IF not ${has_migration_content}
|
|
Log AC #6 visibility: no migration keywords found in diff/apply outputs for this run WARN
|
|
END
|
|
|
|
# Check for backfill-related evidence in plan/decomposition outputs (AC #6).
|
|
${combined_output_lower}= Evaluate ($r_tree.stdout + ' ' + $r_exec.stdout + ' ' + $r_diff.stdout).lower()
|
|
${has_backfill_evidence}= Evaluate 'backfill' in $combined_output_lower or 'batch' in $combined_output_lower or 'populate' in $combined_output_lower or 'last_login' in $combined_output_lower
|
|
IF not ${has_backfill_evidence}
|
|
Log AC #6 visibility: no backfill keywords found in plan/decomposition outputs for this run WARN
|
|
END
|
|
${has_ac6_evidence}= Evaluate ${has_migration_content} or ${has_backfill_evidence}
|
|
IF not ${has_ac6_evidence}
|
|
Log AC #6 visibility: neither migration nor backfill evidence was observed in this run (output is flexible) WARN
|
|
END
|
|
|
|
Log WF05 Database Schema Migration E2E test completed successfully
|