forked from cleveragents/cleveragents-core
test(e2e): workflow example 5 — database schema migration with safety nets (review profile) (#816)
## Summary E2E test for Workflow Example 5 — database schema migration with safety nets using the **review** automation profile. Exercises the full spec-aligned workflow: - **Custom resource type registration** via `resource type add --config` (postgres-db type with `transaction_rollback` sandbox strategy, `--host`, `--port`, `--database`, `--schema` CLI args with flat `type`/`default` fields per `ResourceTypeArgument` schema) - **Custom resource instantiation** — attempts `resource add` with the custom type to exercise mixed resource types, followed by `project link-resource` to link DB resource to the project - **Custom skill creation** with spec-aligned database tools: `local/query_db` (read-only), `local/execute_migration` (writes, checkpointable), `local/backfill_column` (writes, checkpointable) — registered via `skill add --config`, with namespaced tool reference names per `SkillToolRefSchema` validation - **Action creation** with `automation_profile: review`, `reusable: true`, `state: available`, spec invariants, and typed `arguments` section (`table_name`, `column_name`, `column_type`, `backfill_source` — all required per spec, using `arguments` field per `ActionConfigSchema`) - **Plan use** with `--arg` flags exercising parameterized action invocation including `backfill_source=audit_log`, plus **explicit `--automation-profile review`** flag (action-to-plan profile propagation is not yet wired in `PlanLifecycleService.use_action`) - **Phased child plan verification** via `plan tree --format json` with `decision_count >= 2` hard assertion on framework decisions plus WARN tiers for LLM decomposition quality (`< 3`, `< 5`) - **Plan phase assertion** — hard assertion that phase is populated after execute - **Checkpoint-based rollback** with hard assertions: `rc=0` on rollback success, `rc!=0` on fake checkpoint, None guard for JSON null checkpoint IDs, re-execute with Traceback/INTERNAL checks on success and explanatory comment on failure path - **Plan diff** with hard `rc=0` assertion and content-signal verification - **Migration content verification** — baseline SHA saved before apply, diff against baseline (not `HEAD~1`), WARN-level check on migration keywords (`last_login`, `schema`, `migration`, `column`, `alter`) — flexible per LLM non-determinism - **Commit count** assertion `>= 2` (fixture baseline: Create Temp Git Repo + DB fixture commit), WARN if no additional commits from lifecycle-apply - **Backfill evidence** WARN-level check in plan tree/execution output (`backfill`, `batch`, `populate`, `last_login`) with explanatory comment noting tree covers decomposition plan - **Combined AC #6 gate** — if *both* migration content *and* backfill evidence are absent, explicit WARN visibility for CI debugging - **Terminal state assertion** after `lifecycle-apply` — `plan status` call verifies phase/processing_state reflects terminal or apply-progress outcome - **Automation profile fallback verification** — if `plan use` output omits `automation_profile`, falls back to `plan status` for secondary verification (hard assertion always runs) - **Traceback and INTERNAL checks** on all CLI commands (resource add, project create, resource type add, skill add, action create, plan use, strategize, execute, plan tree, plan status, plan diff, plan rollback, re-execute after rollback, lifecycle-apply) including custom resource error paths - **Dynamic actor selection** — detects available API keys (Anthropic/OpenAI) at suite setup - **Skip If No LLM Keys** guard for graceful CI degradation - **Test-level teardown** with diagnostic logging for both plan status and plan tree on failure - **30-minute timeout** covering worst-case rollback+re-execute path - **Force Tags** for consistency with `m6_acceptance.robot` - **Timeout parameters** (`timeout=60s on_timeout=kill`) on all local `Run Process` git commands - **Sequential section numbering** (1 through 15) for readability Closes #751 ISSUES CLOSED: #751 ## Approach Follows the patterns established by `m6_acceptance.robot` and `m2_acceptance.robot`: - `WF05 Suite Setup` initialises the workspace, generates a unique run suffix, and detects available LLM API keys - `Safe Parse Json Field` from `common_e2e.resource` for JSON field extraction with None guards for JSON null values - All CLI commands use `--format json` for predictable, parseable output - `expected_rc=None` with explicit `Should Be Equal As Integers` for detailed failure messages - Hard assertions on infrastructure/framework behavior (CLI commands, phase transitions, tool registration) - WARN-level assertions on LLM-dependent output (decision decomposition, migration content, backfill evidence, commit count) — per ticket requirement "output validation is flexible" - Traceback and INTERNAL checks on all CLI commands following `m2_acceptance.robot` pattern - Baseline SHA approach for post-apply diff verification eliminates false positives from fixture commits ## Bug Fix: LifecyclePlanRepository.update() UNIQUE Constraint Violation **Root cause**: `LifecyclePlanRepository.update()` called `clear()` on child relationship collections (project_links, arguments, invariants) followed by `append()` with new items, but only flushed at the end. SQLAlchemy's default operation ordering can emit INSERTs before DELETEs within the same flush, causing `UNIQUE constraint failed: plan_arguments.plan_id, plan_arguments.name` when plans have arguments. **Fix**: Group all three `clear()` calls together and flush them before appending new rows. This ensures the DELETEs are committed before any INSERTs, preventing the UNIQUE constraint violation. **Impact**: This was a latent bug affecting ALL plans with arguments when `update()` is called. Previously undetected because existing E2E tests (M1, M2, M5, M6) create plans without `--arg` flags. ## Review Fixes (addressing medium findings from @CoreRasurae review) | # | Finding | Fix | |---|---------|-----| | **BUG-1** | No regression test for UNIQUE constraint fix | Added targeted BDD scenario in `repositories_coverage_boost.feature` — creates plan with argument `x=v1`, updates to `x=v2`, asserts no `IntegrityError` | | **TEST-1** | AC #4 weakened — fragile string counting | Replaced raw `count('"decision_id"')` with proper JSON parsing via `json.loads()`, recursive tree walking for decision counting, structural `children_key_count` and `child_link_count` verification | | **TEST-2** | AC #5 conditionally tested | Added explicit WARN log when no checkpoint_id is present ("AC #5 visibility"); fake checkpoint test now runs unconditionally (moved outside IF/ELSE) with Traceback/INTERNAL checks | | **TEST-3** | No terminal state assertion after lifecycle-apply | Added `plan status` call after apply with phase/processing_state extraction; hard assertion on terminal state or apply-phase progress | | **TEST-4** | AC #6 migration/backfill WARN-only | Added combined gate (`has_ac6_evidence`): if *both* migration and backfill evidence are absent, explicit WARN for CI visibility. WARN-only is intentional per ticket AC "output validation is flexible" | | **TEST-5** | Automation profile silently skipped | Added fallback to `plan status --format json` when `plan use` output omits `automation_profile`; hard assertion (`Should Be Equal As Strings review`) now always executes | | **TEST-8** | Missing Traceback/INTERNAL on custom resource error paths | Added Traceback/INTERNAL checks inside both `resource add` and `project link-resource` ELSE branches with `NoSuchOption` guard | ## Quality Gates - `nox -e lint` ✅ - `nox -e typecheck` ✅ (0 errors) - `nox -e unit_tests` ✅ (471 features, 12,422 scenarios, 0 failures) - `nox -e integration_tests` ✅ (1,727 tests, 0 failures) - `nox -e e2e_tests` ✅ (42 tests, 42 passed, 0 failed) - `nox -e coverage_report` ✅ (98%, meets threshold) ## Manual Verification ### Prerequisites - `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` environment variable set ### Commands ```bash nox -e e2e_tests # Or run just this suite: python -m robot --outputdir build/reports/robot --include E2E robot/e2e/wf05_db_migration.robot ``` Reviewed-on: cleveragents/cleveragents-core#816 Reviewed-by: Luis Mendes <luis.mendes@cleverthis.com> Co-authored-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me> Co-committed-by: Jeffrey Phillips Freeman <the@jeffreyfreeman.me>
This commit is contained in:
Forgejo
+8
-1
@@ -113,7 +113,7 @@
|
||||
`EXECUTE_TYPES`. Code relying on `is_strategize_type` or `is_execute_type`
|
||||
returning `False` for `resource_selection` will see different results.
|
||||
Reclassification aligns with ADR-007 L72 and ADR-033 L74 which permit
|
||||
resource selection during planning. (#931)
|
||||
resource selection during planning. (#931)
|
||||
- Added ResourceHandler CRUD and discovery methods: read, write, delete,
|
||||
list_children, diff, and discover_children. Frozen dataclass result types
|
||||
(Content, WriteResult, DeleteResult, DiffResult) added to the handler
|
||||
@@ -122,6 +122,13 @@
|
||||
pathlib/os/difflib. DevcontainerHandler implements read, write, and
|
||||
discover_children via `devcontainer exec`. DatabaseResourceHandler
|
||||
inherits NotImplementedError stubs pending connection management. (#827)
|
||||
- Added E2E test for Workflow Example 5: Database Schema Migration with Safety
|
||||
Nets (review automation profile). Exercises custom resource type registration
|
||||
(`resource type add`), custom skill creation with spec-aligned database tools
|
||||
(`query_db`, `execute_migration`, `backfill_column`), phased child plan
|
||||
execution verification via `plan tree`, checkpoint-based rollback via
|
||||
`plan rollback`, and post-apply migration content verification.
|
||||
(`robot/e2e/wf05_db_migration.robot`) (#751)
|
||||
- Added built-in deferred virtual resource types: `remote`, `submodule`, and
|
||||
`symlink` with equivalence metadata rules for cross-repo and cross-layer
|
||||
identity tracking. Registry bootstrap includes deferred virtual types but
|
||||
|
||||
@@ -80,6 +80,17 @@ Feature: Repository coverage boost for actions, plans, and resources
|
||||
And retrieving the plan should show the new plan arguments
|
||||
And retrieving the plan should show the new plan invariants
|
||||
|
||||
@plan_update
|
||||
Scenario: Updating a plan argument with the same name does not hit UNIQUE constraints
|
||||
Given a valid action object named "local/plan-action-arg-regression"
|
||||
And the action has been persisted in the database
|
||||
And a lifecycle plan domain object linked to "local/plan-action-arg-regression"
|
||||
And the lifecycle plan has argument "x" set to "v1"
|
||||
And the lifecycle plan has been persisted in the database
|
||||
When the lifecycle plan argument "x" is updated to "v2"
|
||||
Then the plan update should succeed without error
|
||||
And retrieving the plan should show argument "x" value "v2"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LifecyclePlanRepository.list_plans filtered by phase
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -454,6 +454,67 @@ def step_verify_plan_invariants(context: Context) -> None:
|
||||
assert "All integration tests must pass" in texts
|
||||
|
||||
|
||||
@given('the lifecycle plan has argument "{arg_name}" set to "{arg_value}"')
|
||||
def step_set_initial_plan_argument(
|
||||
context: Context, arg_name: str, arg_value: str
|
||||
) -> None:
|
||||
"""Set an initial argument on the in-memory plan before persisting.
|
||||
|
||||
This enables a regression scenario where update() replaces a child
|
||||
argument row with the same composite key (plan_id, name).
|
||||
"""
|
||||
context.plan = context.plan.model_copy(
|
||||
update={
|
||||
"arguments": {arg_name: arg_value},
|
||||
"arguments_order": [arg_name],
|
||||
"timestamps": PlanTimestamps(
|
||||
created_at=context.plan.timestamps.created_at,
|
||||
updated_at=datetime.now(),
|
||||
),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@when('the lifecycle plan argument "{arg_name}" is updated to "{new_value}"')
|
||||
def step_update_plan_argument_same_name(
|
||||
context: Context, arg_name: str, new_value: str
|
||||
) -> None:
|
||||
"""Update an existing argument name to a new value via repository.update()."""
|
||||
updated = context.plan.model_copy(
|
||||
update={
|
||||
"arguments": {arg_name: new_value},
|
||||
"arguments_order": [arg_name],
|
||||
"timestamps": PlanTimestamps(
|
||||
created_at=context.plan.timestamps.created_at,
|
||||
updated_at=datetime.now(),
|
||||
),
|
||||
}
|
||||
)
|
||||
context.plan = updated
|
||||
try:
|
||||
context.result_plan = context.plan_repo.update(updated)
|
||||
context.db_session.commit()
|
||||
context.error = None
|
||||
except Exception as exc:
|
||||
context.error = exc
|
||||
|
||||
|
||||
@then('retrieving the plan should show argument "{arg_name}" value "{expected_value}"')
|
||||
def step_verify_plan_argument_value(
|
||||
context: Context, arg_name: str, expected_value: str
|
||||
) -> None:
|
||||
"""Verify the replacement argument row persisted with new value."""
|
||||
fetched = context.plan_repo.get(context.plan.identity.plan_id)
|
||||
assert fetched is not None, "Plan not found after update"
|
||||
assert arg_name in fetched.arguments, (
|
||||
f"Expected argument '{arg_name}' in plan args, got {fetched.arguments}"
|
||||
)
|
||||
assert fetched.arguments[arg_name] == expected_value, (
|
||||
f"Expected argument '{arg_name}' value '{expected_value}', "
|
||||
f"got '{fetched.arguments[arg_name]}'"
|
||||
)
|
||||
|
||||
|
||||
# ========================================================================
|
||||
# LifecyclePlanRepository.list_plans filtered by phase (lines 1269-1271)
|
||||
# ========================================================================
|
||||
|
||||
@@ -0,0 +1,571 @@
|
||||
*** Settings ***
|
||||
Documentation E2E test for Workflow Example 5: Database Schema Migration with Safety Nets.
|
||||
...
|
||||
... Advanced scenario using the **review** automation profile.
|
||||
... Registers a custom resource type (postgres-db), creates custom
|
||||
... skills with spec-aligned database tools (query_db, execute_migration,
|
||||
... backfill_column), exercises phased child plan execution via
|
||||
... ``plan tree``, attempts checkpoint-based rollback via
|
||||
... ``plan rollback``, and verifies migration changes after apply.
|
||||
...
|
||||
... Zero mocking — real CLI, real LLM API keys.
|
||||
Resource common_e2e.resource
|
||||
Suite Setup WF05 Suite Setup
|
||||
Suite Teardown E2E Suite Teardown
|
||||
Force Tags E2E
|
||||
|
||||
*** Variables ***
|
||||
${ACTION_NAME} local/wf05-db-migration
|
||||
${SKILL_NAME} local/wf05-db-tools
|
||||
${RESOURCE_TYPE_NAME} local/wf05-postgres-db
|
||||
|
||||
*** Keywords ***
|
||||
WF05 Suite Setup
|
||||
[Documentation] E2E Suite Setup plus database init and dynamic actor selection.
|
||||
E2E Suite Setup
|
||||
${init}= Run CleverAgents Command init --force --yes
|
||||
Should Be Equal As Integers ${init.rc} 0
|
||||
# Generate unique suffix for resource/project names to avoid UNIQUE
|
||||
# constraint collisions on repeated E2E runs against the same database.
|
||||
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
|
||||
Set Suite Variable ${RUN_SUFFIX} ${suffix}
|
||||
# Pick an actor that matches available API keys.
|
||||
# Prefer OpenAI first to reduce Anthropic credit-quota flakiness.
|
||||
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
|
||||
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
|
||||
IF ${has_openai}
|
||||
${actor}= Set Variable openai/gpt-4o
|
||||
ELSE IF ${has_anthropic}
|
||||
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
|
||||
ELSE
|
||||
${actor}= Set Variable openai/gpt-4o
|
||||
END
|
||||
Set Suite Variable ${WF05_ACTOR} ${actor}
|
||||
|
||||
Create DB App Repo
|
||||
[Documentation] Create temp git repo with a Python app simulating
|
||||
... a database schema and application code.
|
||||
${repo}= Create Temp Git Repo wf05-db-app-${RUN_SUFFIX}
|
||||
Create Directory ${repo}${/}src
|
||||
Create Directory ${repo}${/}migrations
|
||||
Create Directory ${repo}${/}tests
|
||||
${schema_content}= Catenate SEPARATOR=\n
|
||||
... """Database schema — users table (missing last_login_at)."""
|
||||
... ${EMPTY}
|
||||
... USERS_SCHEMA = {
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}"table": "users",
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}"columns": [
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "id", "type": "INTEGER", "primary_key": True},
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "email", "type": "VARCHAR(255)"},
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "created_at", "type": "TIMESTAMP"},
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}],
|
||||
... }
|
||||
Create File ${repo}${/}src${/}schema.py ${schema_content}
|
||||
${app_content}= Catenate SEPARATOR=\n
|
||||
... """Application code — user service."""
|
||||
... from src.schema import USERS_SCHEMA
|
||||
... ${EMPTY}
|
||||
... ${EMPTY}
|
||||
... def get_user(user_id):
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Fetch a user by ID."""
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"id": user_id, "email": "user@example.com"}
|
||||
... ${EMPTY}
|
||||
... ${EMPTY}
|
||||
... def get_user_activity(user_id):
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Get user activity — needs last_login_at but column is missing."""
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}user = get_user(user_id)
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}# BUG: no last_login_at field available
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"user": user, "last_login": None}
|
||||
Create File ${repo}${/}src${/}app.py ${app_content}
|
||||
${test_content}= Catenate SEPARATOR=\n
|
||||
... """Tests for user service."""
|
||||
... from src.app import get_user, get_user_activity
|
||||
... ${EMPTY}
|
||||
... ${EMPTY}
|
||||
... def test_get_user():
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}user = get_user(1)
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}assert user["id"] == 1
|
||||
... ${EMPTY}
|
||||
... ${EMPTY}
|
||||
... def test_get_user_activity():
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}result = get_user_activity(1)
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}assert result["last_login"] is None
|
||||
Create File ${repo}${/}tests${/}test_app.py ${test_content}
|
||||
Create File ${repo}${/}src${/}__init__.py \n
|
||||
Create File ${repo}${/}tests${/}__init__.py \n
|
||||
Create File ${repo}${/}migrations${/}__init__.py \n
|
||||
Create File ${repo}${/}requirements.txt pytest>=7.0\n
|
||||
${git_add}= Run Process git add . cwd=${repo} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
|
||||
${git_commit}= Run Process git commit -m Initial DB app with missing last_login_at cwd=${repo} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
|
||||
RETURN ${repo}
|
||||
|
||||
WF05 Test Teardown
|
||||
[Documentation] Log diagnostic context on failure for debugging.
|
||||
${plan_id}= Get Variable Value ${WF05_PLAN_ID} ${EMPTY}
|
||||
IF '${plan_id}' != ''
|
||||
${status} ${result}= Run Keyword And Ignore Error
|
||||
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
|
||||
IF '${status}' == 'PASS'
|
||||
Log Teardown plan status: ${result.stdout} WARN
|
||||
END
|
||||
${tree_status} ${tree_result}= Run Keyword And Ignore Error
|
||||
... Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=30s
|
||||
IF '${tree_status}' == 'PASS'
|
||||
Log Teardown plan tree: ${tree_result.stdout} WARN
|
||||
END
|
||||
END
|
||||
|
||||
*** Test Cases ***
|
||||
WF05 Database Schema Migration With Safety Nets Review Profile
|
||||
[Documentation] Full review-profile workflow: register custom resource type,
|
||||
... register git-checkout resource, create custom skill with
|
||||
... spec-aligned DB tools (query_db, execute_migration,
|
||||
... backfill_column), create migration action with review
|
||||
... automation profile, exercise phased child plan execution,
|
||||
... attempt checkpoint-based rollback, verify migration changes.
|
||||
[Timeout] 30 minutes
|
||||
[Teardown] WF05 Test Teardown
|
||||
Skip If No LLM Keys
|
||||
# Initialise test variable for teardown access.
|
||||
Set Test Variable ${WF05_PLAN_ID} ${EMPTY}
|
||||
|
||||
# ── 1. Fixture: create temp repo ──
|
||||
${repo}= Create DB App Repo
|
||||
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${branch_result.rc} 0 git rev-parse failed: ${branch_result.stderr}
|
||||
${branch}= Strip String ${branch_result.stdout}
|
||||
|
||||
# ── 2. Register custom resource type (AC #2: custom resource type via CLI) ──
|
||||
${res_type_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${RESOURCE_TYPE_NAME}
|
||||
... description: PostgreSQL database resource for WF05 E2E test
|
||||
... resource_kind: physical
|
||||
... sandbox_strategy: transaction_rollback
|
||||
... user_addable: true
|
||||
... cli_args:
|
||||
... ${SPACE}${SPACE}- name: host
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Database hostname
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
||||
... ${SPACE}${SPACE}- name: port
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}required: false
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Port (default 5432)
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type: integer
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}default: 5432
|
||||
... ${SPACE}${SPACE}- name: database
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Database name
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
||||
... ${SPACE}${SPACE}- name: schema
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}required: false
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Schema (default public)
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}default: public
|
||||
... capabilities:
|
||||
... ${SPACE}${SPACE}read: true
|
||||
... ${SPACE}${SPACE}write: true
|
||||
... ${SPACE}${SPACE}sandbox: true
|
||||
... ${SPACE}${SPACE}checkpoint: true
|
||||
${res_type_path}= Set Variable ${SUITE_HOME}${/}wf05_resource_type.yaml
|
||||
Create File ${res_type_path} ${res_type_yaml}
|
||||
${r_type}= Run CleverAgents Command
|
||||
... resource type add --config ${res_type_path} --format json
|
||||
... expected_rc=None
|
||||
Should Be Equal As Integers ${r_type.rc} 0
|
||||
... resource type add failed (rc=${r_type.rc}): ${r_type.stderr}
|
||||
Should Not Contain ${r_type.stdout}${r_type.stderr} Traceback
|
||||
Should Not Contain ${r_type.stdout}${r_type.stderr} INTERNAL
|
||||
Output Should Contain ${r_type} ${RESOURCE_TYPE_NAME}
|
||||
|
||||
# ── 3. Register git-checkout resource and create project ──
|
||||
${res_name}= Set Variable wf05-res-${RUN_SUFFIX}
|
||||
${r_res}= Run CleverAgents Command
|
||||
... resource add git-checkout ${res_name}
|
||||
... --path ${repo} --branch ${branch}
|
||||
Should Be Equal As Integers ${r_res.rc} 0
|
||||
... resource add failed (rc=${r_res.rc}): ${r_res.stderr}
|
||||
Should Not Contain ${r_res.stdout}${r_res.stderr} Traceback
|
||||
Should Not Contain ${r_res.stdout}${r_res.stderr} INTERNAL
|
||||
Output Should Contain ${r_res} ${res_name}
|
||||
|
||||
${proj_name}= Set Variable wf05-proj-${RUN_SUFFIX}
|
||||
${r_proj}= Run CleverAgents Command
|
||||
... project create --resource ${res_name} ${proj_name}
|
||||
Should Be Equal As Integers ${r_proj.rc} 0
|
||||
... project create failed (rc=${r_proj.rc}): ${r_proj.stderr}
|
||||
Should Not Contain ${r_proj.stdout}${r_proj.stderr} Traceback
|
||||
Should Not Contain ${r_proj.stdout}${r_proj.stderr} INTERNAL
|
||||
Output Should Contain ${r_proj} ${proj_name}
|
||||
|
||||
# ── 4. Instantiate custom postgres-db resource and link to project ──
|
||||
${db_res_name}= Set Variable wf05-db-${RUN_SUFFIX}
|
||||
${r_db_res}= Run CleverAgents Command
|
||||
... resource add ${RESOURCE_TYPE_NAME} ${db_res_name}
|
||||
... --host localhost --database wf05_test_db
|
||||
... expected_rc=None
|
||||
IF ${r_db_res.rc} == 0
|
||||
Output Should Contain ${r_db_res} ${db_res_name}
|
||||
Log Custom postgres-db resource instance created: ${db_res_name}
|
||||
# Link custom DB resource to project (spec Step 1: project link-resource)
|
||||
${r_link_db}= Run CleverAgents Command
|
||||
... project link-resource ${proj_name} ${db_res_name}
|
||||
... expected_rc=None
|
||||
IF ${r_link_db.rc} == 0
|
||||
Log Custom DB resource linked to project: ${db_res_name}
|
||||
ELSE
|
||||
${link_err}= Set Variable ${r_link_db.stdout}${r_link_db.stderr}
|
||||
Should Not Contain ${link_err} Traceback
|
||||
${link_has_no_such_option}= Evaluate 'No such option' in $link_err or 'NoSuchOption' in $link_err
|
||||
IF not ${link_has_no_such_option}
|
||||
Should Not Contain ${link_err} INTERNAL
|
||||
END
|
||||
Log project link-resource returned rc=${r_link_db.rc} (may require additional CLI support) WARN
|
||||
END
|
||||
ELSE
|
||||
${db_err}= Set Variable ${r_db_res.stdout}${r_db_res.stderr}
|
||||
Should Not Contain ${db_err} Traceback
|
||||
${db_has_no_such_option}= Evaluate 'No such option' in $db_err or 'NoSuchOption' in $db_err
|
||||
IF not ${db_has_no_such_option}
|
||||
Should Not Contain ${db_err} INTERNAL
|
||||
END
|
||||
Log Custom resource instantiation returned rc=${r_db_res.rc} (may require additional CLI support) WARN
|
||||
END
|
||||
|
||||
# ── 5. Create custom skill with spec-aligned DB tools (AC #3) ──
|
||||
# Note: writes/checkpointable are schema-validated at registration time.
|
||||
# The persisted Skill model currently stores tool refs by namespaced name.
|
||||
${skill_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${SKILL_NAME}
|
||||
... description: Safe database operations with transaction support
|
||||
... tools:
|
||||
... ${SPACE}${SPACE}- name: local/query_db
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Execute a read-only SQL query and return results
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}writes: false
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}checkpointable: false
|
||||
... ${SPACE}${SPACE}- name: local/execute_migration
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Execute a DDL migration within a transaction
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}writes: true
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}checkpointable: true
|
||||
... ${SPACE}${SPACE}- name: local/backfill_column
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Batch-update a column using a source query
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}writes: true
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}checkpointable: true
|
||||
${skill_path}= Set Variable ${SUITE_HOME}${/}wf05_skill.yaml
|
||||
Create File ${skill_path} ${skill_yaml}
|
||||
${r_skill}= Run CleverAgents Command
|
||||
... skill add --config ${skill_path} --format json
|
||||
... expected_rc=None
|
||||
Should Be Equal As Integers ${r_skill.rc} 0
|
||||
... skill add failed (rc=${r_skill.rc}): ${r_skill.stderr}
|
||||
Should Not Contain ${r_skill.stdout}${r_skill.stderr} Traceback
|
||||
Should Not Contain ${r_skill.stdout}${r_skill.stderr} INTERNAL
|
||||
Output Should Contain ${r_skill} ${SKILL_NAME}
|
||||
# Verify individual tool names are registered
|
||||
Output Should Contain ${r_skill} local/query_db
|
||||
Output Should Contain ${r_skill} local/execute_migration
|
||||
Output Should Contain ${r_skill} local/backfill_column
|
||||
|
||||
# ── 6. Create action with review automation profile ──
|
||||
${action_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${ACTION_NAME}
|
||||
... description: Add last_login_at column to users table with backfill and app update
|
||||
... definition_of_done: |
|
||||
... ${SPACE}${SPACE}Database migration adds the column with a sensible default.
|
||||
... ${SPACE}${SPACE}Backfill completes for all rows using the specified source.
|
||||
... ${SPACE}${SPACE}Application code reads/writes the new column.
|
||||
... ${SPACE}${SPACE}All tests pass with the new schema.
|
||||
... ${SPACE}${SPACE}A rollback migration is available and tested.
|
||||
... strategy_actor: ${WF05_ACTOR}
|
||||
... execution_actor: ${WF05_ACTOR}
|
||||
... automation_profile: review
|
||||
... reusable: true
|
||||
... state: available
|
||||
... invariants:
|
||||
... ${SPACE}${SPACE}- Migration must be backward-compatible (add column, don't rename or drop)
|
||||
... ${SPACE}${SPACE}- Backfill must be batched to avoid locking the table
|
||||
... ${SPACE}${SPACE}- Rollback migration must be provided and tested
|
||||
... ${SPACE}${SPACE}- Application code must handle both old (null) and new values gracefully
|
||||
... arguments:
|
||||
... ${SPACE}${SPACE}- name: table_name
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Target table for schema migration
|
||||
... ${SPACE}${SPACE}- name: column_name
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: New column to add
|
||||
... ${SPACE}${SPACE}- name: column_type
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: SQL type for the new column
|
||||
... ${SPACE}${SPACE}- name: backfill_source
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Where to source backfill data
|
||||
${action_path}= Set Variable ${SUITE_HOME}${/}wf05_action.yaml
|
||||
Create File ${action_path} ${action_yaml}
|
||||
${r_action}= Run CleverAgents Command
|
||||
... action create --config ${action_path} --format json
|
||||
... expected_rc=None
|
||||
Should Be Equal As Integers ${r_action.rc} 0
|
||||
... action create failed (rc=${r_action.rc}): ${r_action.stderr}
|
||||
Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback
|
||||
Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL
|
||||
Output Should Contain ${r_action} ${ACTION_NAME}
|
||||
|
||||
# ── 7. Plan use — create plan from action + project with args ──
|
||||
${r_use}= Run CleverAgents Command
|
||||
... plan use ${ACTION_NAME} ${proj_name}
|
||||
... --arg table_name=users --arg column_name=last_login_at
|
||||
... --arg column_type=TIMESTAMP --arg backfill_source=audit_log
|
||||
... --automation-profile review
|
||||
... --format json expected_rc=None timeout=120s
|
||||
Should Be Equal As Integers ${r_use.rc} 0
|
||||
... plan use failed (rc=${r_use.rc}): ${r_use.stderr}
|
||||
Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback
|
||||
Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL
|
||||
${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id
|
||||
Should Not Be Empty ${plan_id} Could not extract plan_id from plan use output
|
||||
Set Test Variable ${WF05_PLAN_ID} ${plan_id}
|
||||
Log Plan ID: ${plan_id}
|
||||
# Verify review automation profile was applied.
|
||||
# If plan use output omits automation_profile, fall back to plan status.
|
||||
${resolved_profile}= Safe Parse Json Field ${r_use.stdout} automation_profile
|
||||
${resolved_profile}= Set Variable If $resolved_profile is None ${EMPTY} ${resolved_profile}
|
||||
${resolved_profile}= Set Variable If '${resolved_profile}' == 'None' ${EMPTY} ${resolved_profile}
|
||||
IF '${resolved_profile}' == ''
|
||||
${r_profile_status}= Run CleverAgents Command
|
||||
... plan status ${plan_id} --format json
|
||||
... expected_rc=None timeout=120s
|
||||
Should Be Equal As Integers ${r_profile_status.rc} 0
|
||||
... plan status for profile verification failed (rc=${r_profile_status.rc}): ${r_profile_status.stderr}
|
||||
Should Not Contain ${r_profile_status.stdout}${r_profile_status.stderr} Traceback
|
||||
Should Not Contain ${r_profile_status.stdout}${r_profile_status.stderr} INTERNAL
|
||||
${resolved_profile}= Safe Parse Json Field ${r_profile_status.stdout} automation_profile
|
||||
${resolved_profile}= Set Variable If $resolved_profile is None ${EMPTY} ${resolved_profile}
|
||||
${resolved_profile}= Set Variable If '${resolved_profile}' == 'None' ${EMPTY} ${resolved_profile}
|
||||
END
|
||||
Should Be Equal As Strings ${resolved_profile} review
|
||||
... Expected automation_profile 'review' but got '${resolved_profile}'
|
||||
|
||||
# ── 8. Strategize ──
|
||||
${r_strat}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format json
|
||||
... expected_rc=None timeout=300s
|
||||
Should Be Equal As Integers ${r_strat.rc} 0
|
||||
... plan execute (strategize) failed (rc=${r_strat.rc}): ${r_strat.stderr}
|
||||
Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback
|
||||
Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL
|
||||
Output Should Contain ${r_strat} ${plan_id}
|
||||
Log Strategize stdout: ${r_strat.stdout}
|
||||
|
||||
# ── 9. Decision tree — verify phased child plans (AC #4) ──
|
||||
${r_tree}= Run CleverAgents Command
|
||||
... plan tree ${plan_id} --format json
|
||||
... expected_rc=None timeout=120s
|
||||
Should Be Equal As Integers ${r_tree.rc} 0
|
||||
... plan tree failed (rc=${r_tree.rc}): ${r_tree.stderr}
|
||||
Should Not Contain ${r_tree.stdout}${r_tree.stderr} Traceback
|
||||
Should Not Contain ${r_tree.stdout}${r_tree.stderr} INTERNAL
|
||||
Should Not Be Empty ${r_tree.stdout} Plan tree output should not be empty
|
||||
# Parse JSON and assert structural evidence of phased decomposition.
|
||||
# plan tree output may include a log line prefix before JSON payload.
|
||||
${tree_json_str}= Evaluate
|
||||
... (lambda s: (lambda lines: (lambda idx: ('\\n'.join(lines[idx:]).strip() if idx is not None else s.strip()))(next((i for i,l in enumerate(lines) if l.lstrip().startswith('[') or l.lstrip().startswith('{')), None)))([ln for ln in s.splitlines() if ln.strip()]))($r_tree.stdout)
|
||||
${tree_parse_status} ${tree_json}= Run Keyword And Ignore Error
|
||||
... Evaluate __import__('json').loads($tree_json_str)
|
||||
Should Be Equal As Strings ${tree_parse_status} PASS
|
||||
... plan tree output must be parseable JSON (error: ${tree_json})
|
||||
${decision_count}= Evaluate
|
||||
... (lambda root: (lambda walk: walk(walk, root))(lambda self, node: (1 if isinstance(node, dict) and str(node.get('decision_id', '')).strip() != '' else 0) + sum(self(self, child) for child in ((node.get('children') if isinstance(node, dict) else []) or [])) + (sum(self(self, item) for item in node) if isinstance(node, list) else 0)))($tree_json)
|
||||
Log Decision tree contains ${decision_count} decision node(s)
|
||||
Should Be True ${decision_count} >= 2
|
||||
... Plan tree should contain at least 2 decision nodes (found ${decision_count})
|
||||
${children_key_count}= Evaluate __import__('json').dumps($tree_json).count('"children"')
|
||||
Should Be True ${children_key_count} >= 1
|
||||
... Plan tree should include structured children fields
|
||||
${child_link_count}= Evaluate
|
||||
... (lambda root: (lambda walk: walk(walk, root))(lambda self, node: ((len((node.get('children') if isinstance(node, dict) else []) or [])) if isinstance(node, dict) else 0) + sum(self(self, child) for child in ((node.get('children') if isinstance(node, dict) else []) or [])) + (sum(self(self, item) for item in node) if isinstance(node, list) else 0)))($tree_json)
|
||||
IF ${decision_count} < 3
|
||||
Log Decision count (${decision_count}) is below 3 — decomposition may be minimal in this run WARN
|
||||
END
|
||||
IF ${child_link_count} < 1
|
||||
Log Decision tree has no child links in this run — phased decomposition may be minimal WARN
|
||||
END
|
||||
|
||||
# ── 10. Execute ──
|
||||
${r_exec}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format json
|
||||
... expected_rc=None timeout=300s
|
||||
IF ${r_exec.rc} != 0
|
||||
Fail plan execute failed (rc=${r_exec.rc}) stdout=${r_exec.stdout} stderr=${r_exec.stderr}
|
||||
END
|
||||
Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback
|
||||
Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL
|
||||
Output Should Contain ${r_exec} ${plan_id}
|
||||
Log Execute stdout: ${r_exec.stdout}
|
||||
|
||||
# ── 11. Plan status — verify phase and check for checkpoints ──
|
||||
${r_status}= Run CleverAgents Command
|
||||
... plan status ${plan_id} --format json
|
||||
... expected_rc=None timeout=120s
|
||||
Should Be Equal As Integers ${r_status.rc} 0
|
||||
... plan status failed (rc=${r_status.rc}): ${r_status.stderr}
|
||||
Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback
|
||||
Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL
|
||||
Should Not Be Empty ${r_status.stdout}
|
||||
Output Should Contain ${r_status} ${plan_id}
|
||||
# Assert phase field is populated after execute
|
||||
${status_phase}= Safe Parse Json Field ${r_status.stdout} phase
|
||||
Should Not Be Empty ${status_phase} Plan phase should be populated after execute
|
||||
Log Plan phase after execute: ${status_phase}
|
||||
# Extract checkpoint ID if available (for rollback attempt)
|
||||
${checkpoint_id}= Safe Parse Json Field ${r_status.stdout} last_checkpoint_id
|
||||
# Guard against Python None being returned as string "None" from JSON null
|
||||
${checkpoint_id}= Set Variable If $checkpoint_id is None ${EMPTY} ${checkpoint_id}
|
||||
${checkpoint_id}= Set Variable If '${checkpoint_id}' == 'None' ${EMPTY} ${checkpoint_id}
|
||||
Log Checkpoint ID from status: ${checkpoint_id}
|
||||
|
||||
# ── 12. Checkpoint rollback attempt (AC #5) ──
|
||||
IF '${checkpoint_id}' != ''
|
||||
Log Attempting plan rollback to checkpoint ${checkpoint_id}
|
||||
${r_rollback}= Run CleverAgents Command
|
||||
... plan rollback --yes ${plan_id} ${checkpoint_id}
|
||||
... --format json expected_rc=None timeout=120s
|
||||
Log Rollback rc=${r_rollback.rc} stdout=${r_rollback.stdout} stderr=${r_rollback.stderr}
|
||||
Should Be Equal As Integers ${r_rollback.rc} 0
|
||||
... plan rollback to checkpoint ${checkpoint_id} failed (rc=${r_rollback.rc}): ${r_rollback.stderr}
|
||||
Should Not Contain ${r_rollback.stdout}${r_rollback.stderr} Traceback
|
||||
Should Not Contain ${r_rollback.stdout}${r_rollback.stderr} INTERNAL
|
||||
Output Should Contain ${r_rollback} ${plan_id}
|
||||
Log Rollback succeeded — re-executing plan to continue lifecycle
|
||||
# Re-execute after rollback so we can proceed to diff/apply
|
||||
${r_reexec}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format json
|
||||
... expected_rc=None timeout=300s
|
||||
Log Re-execute after rollback rc=${r_reexec.rc}
|
||||
IF ${r_reexec.rc} != 0
|
||||
# Re-execute failure is non-fatal: the plan may still be in an applicable
|
||||
# state from the original execute, so we continue to diff/apply rather
|
||||
# than aborting. If those steps also fail the test will still surface a
|
||||
# clear failure at the relevant assertion.
|
||||
Log Re-execute after rollback failed (rc=${r_reexec.rc}); continuing to diff/apply WARN
|
||||
ELSE
|
||||
Should Not Contain ${r_reexec.stdout}${r_reexec.stderr} Traceback
|
||||
Should Not Contain ${r_reexec.stdout}${r_reexec.stderr} INTERNAL
|
||||
END
|
||||
ELSE
|
||||
Log AC #5 visibility: no checkpoint_id present, so real rollback path was not executed in this run WARN
|
||||
END
|
||||
|
||||
# Also validate graceful failure path on a fake checkpoint ID.
|
||||
${r_rollback_noop}= Run CleverAgents Command
|
||||
... plan rollback --yes ${plan_id} no-such-checkpoint
|
||||
... --format json expected_rc=None timeout=60s
|
||||
Log Rollback (no checkpoint) rc=${r_rollback_noop.rc} stderr=${r_rollback_noop.stderr}
|
||||
Should Not Be Equal As Integers ${r_rollback_noop.rc} 0
|
||||
... plan rollback with fake checkpoint should fail gracefully (got rc=0)
|
||||
Should Not Contain ${r_rollback_noop.stdout}${r_rollback_noop.stderr} Traceback
|
||||
Should Not Contain ${r_rollback_noop.stdout}${r_rollback_noop.stderr} INTERNAL
|
||||
|
||||
# ── 13. Diff — verify changeset is non-empty (AC #6: migration verification) ──
|
||||
${r_diff}= Run CleverAgents Command
|
||||
... plan diff ${plan_id} --format json
|
||||
... expected_rc=None timeout=120s
|
||||
Should Be Equal As Integers ${r_diff.rc} 0
|
||||
... plan diff failed (rc=${r_diff.rc}): ${r_diff.stderr}
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
|
||||
Should Not Be Empty ${r_diff.stdout} Plan diff produced no output
|
||||
${diff_preview_lower}= Evaluate ($r_diff.stdout).lower()
|
||||
${has_diff_signal}= Evaluate 'migration' in $diff_preview_lower or 'backfill' in $diff_preview_lower or 'last_login' in $diff_preview_lower or 'column' in $diff_preview_lower or 'audit_log' in $diff_preview_lower or 'diff' in $diff_preview_lower or 'change' in $diff_preview_lower or 'file' in $diff_preview_lower
|
||||
Should Be True ${has_diff_signal}
|
||||
... plan diff output should include meaningful diff/change indicators before apply
|
||||
|
||||
# ── 14. Apply ──
|
||||
# Save baseline SHA before apply for accurate diff comparison
|
||||
${baseline_sha_result}= Run Process git rev-parse HEAD cwd=${repo} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${baseline_sha_result.rc} 0
|
||||
... git rev-parse HEAD failed: ${baseline_sha_result.stderr}
|
||||
${baseline_sha}= Strip String ${baseline_sha_result.stdout}
|
||||
Log Baseline SHA before apply: ${baseline_sha}
|
||||
|
||||
${r_apply}= Run CleverAgents Command
|
||||
... plan lifecycle-apply ${plan_id} --format json
|
||||
... expected_rc=None timeout=180s
|
||||
Log Apply rc=${r_apply.rc} stdout=${r_apply.stdout}
|
||||
IF ${r_apply.rc} == 0
|
||||
Output Should Contain ${r_apply} ${plan_id}
|
||||
ELSE
|
||||
Fail lifecycle-apply failed (rc=${r_apply.rc}) stdout=${r_apply.stdout} stderr=${r_apply.stderr}
|
||||
END
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
|
||||
|
||||
# Verify terminal state after lifecycle-apply.
|
||||
${r_status_after_apply}= Run CleverAgents Command
|
||||
... plan status ${plan_id} --format json
|
||||
... expected_rc=None timeout=120s
|
||||
Should Be Equal As Integers ${r_status_after_apply.rc} 0
|
||||
... plan status after lifecycle-apply failed (rc=${r_status_after_apply.rc}): ${r_status_after_apply.stderr}
|
||||
Should Not Contain ${r_status_after_apply.stdout}${r_status_after_apply.stderr} Traceback
|
||||
Should Not Contain ${r_status_after_apply.stdout}${r_status_after_apply.stderr} INTERNAL
|
||||
${apply_phase}= Safe Parse Json Field ${r_status_after_apply.stdout} phase
|
||||
${apply_state}= Safe Parse Json Field ${r_status_after_apply.stdout} processing_state
|
||||
Should Not Be Empty ${apply_phase} plan status after apply should include phase
|
||||
Should Not Be Empty ${apply_state} plan status after apply should include processing_state
|
||||
${apply_phase_lower}= Evaluate ($apply_phase).lower()
|
||||
IF 'apply' not in $apply_phase_lower
|
||||
Log Post-apply phase is '${apply_phase}' instead of apply; treating terminal state as authoritative WARN
|
||||
END
|
||||
${is_terminal_state}= Evaluate ($apply_state.lower() in ['applied', 'constrained', 'errored', 'cancelled', 'complete'])
|
||||
${is_apply_progress_state}= Evaluate ('apply' in $apply_phase_lower)
|
||||
Should Be True ${is_terminal_state} or ${is_apply_progress_state}
|
||||
... expected lifecycle-apply to produce terminal state or apply-phase progress (phase=${apply_phase}, state=${apply_state})
|
||||
IF not ${is_terminal_state}
|
||||
Log Post-apply state '${apply_state}' is non-terminal; apply may complete asynchronously WARN
|
||||
END
|
||||
|
||||
# ── 15. Verify repo state and migration content (AC #6) ──
|
||||
${log_result}= Run Process git log --oneline -10 cwd=${repo} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${log_result.rc} 0
|
||||
... git log failed (rc=${log_result.rc}): ${log_result.stderr}
|
||||
Log Git log: ${log_result.stdout}
|
||||
${line_count}= Get Line Count ${log_result.stdout}
|
||||
# Hard assert minimum fixture commits exist; LLM-generated commits are
|
||||
# non-deterministic, so we only hard-assert the fixture baseline (2 commits:
|
||||
# Create Temp Git Repo initial + DB app fixture).
|
||||
Should Be True ${line_count} >= 2
|
||||
... Expected at least 2 commits (fixture baseline), got ${line_count}
|
||||
IF ${line_count} < 3
|
||||
Log No new commits from lifecycle-apply (${line_count} total) — LLM may not have produced file changes WARN
|
||||
END
|
||||
|
||||
# Diff against baseline SHA to capture only changes from lifecycle-apply
|
||||
${diff_all}= Run Process git diff ${baseline_sha} HEAD --name-only cwd=${repo} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${diff_all.rc} 0
|
||||
... git diff --name-only failed (rc=${diff_all.rc}): ${diff_all.stderr}
|
||||
Log Files changed since apply: ${diff_all.stdout}
|
||||
${full_diff}= Run Process git diff ${baseline_sha} HEAD cwd=${repo} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${full_diff.rc} 0
|
||||
... git diff failed (rc=${full_diff.rc}): ${full_diff.stderr}
|
||||
${migration_evidence_text}= Evaluate ($full_diff.stdout + ' ' + $r_diff.stdout).lower()
|
||||
${has_migration_content}= Evaluate 'last_login' in $migration_evidence_text or 'schema' in $migration_evidence_text or 'migration' in $migration_evidence_text or 'column' in $migration_evidence_text or 'alter' in $migration_evidence_text
|
||||
IF not ${has_migration_content}
|
||||
Log AC #6 visibility: no migration keywords found in diff/apply outputs for this run WARN
|
||||
END
|
||||
|
||||
# Check for backfill-related evidence in plan/decomposition outputs (AC #6).
|
||||
${combined_output_lower}= Evaluate ($r_tree.stdout + ' ' + $r_exec.stdout + ' ' + $r_diff.stdout).lower()
|
||||
${has_backfill_evidence}= Evaluate 'backfill' in $combined_output_lower or 'batch' in $combined_output_lower or 'populate' in $combined_output_lower or 'last_login' in $combined_output_lower
|
||||
IF not ${has_backfill_evidence}
|
||||
Log AC #6 visibility: no backfill keywords found in plan/decomposition outputs for this run WARN
|
||||
END
|
||||
${has_ac6_evidence}= Evaluate ${has_migration_content} or ${has_backfill_evidence}
|
||||
IF not ${has_ac6_evidence}
|
||||
Log AC #6 visibility: neither migration nor backfill evidence was observed in this run (output is flexible) WARN
|
||||
END
|
||||
|
||||
Log WF05 Database Schema Migration E2E test completed successfully
|
||||
@@ -1395,8 +1395,16 @@ class LifecyclePlanRepository:
|
||||
plan.timestamps.applied_at
|
||||
)
|
||||
|
||||
# Replace child project links
|
||||
# Replace child collections. We must flush the DELETEs
|
||||
# from clear() before appending new rows, otherwise
|
||||
# SQLAlchemy may emit INSERTs before DELETEs, which
|
||||
# triggers UNIQUE constraint violations on composite keys
|
||||
# like (plan_id, name) in plan_arguments.
|
||||
row.project_links_rel.clear() # type: ignore[union-attr]
|
||||
row.arguments_rel.clear() # type: ignore[union-attr]
|
||||
row.invariants_rel.clear() # type: ignore[union-attr]
|
||||
session.flush()
|
||||
|
||||
now_iso = plan.timestamps.updated_at.isoformat()
|
||||
for pl in getattr(plan, "project_links", []) or []:
|
||||
row.project_links_rel.append( # type: ignore[union-attr]
|
||||
@@ -1408,8 +1416,6 @@ class LifecyclePlanRepository:
|
||||
)
|
||||
)
|
||||
|
||||
# Replace child arguments
|
||||
row.arguments_rel.clear() # type: ignore[union-attr]
|
||||
arguments_dict: dict[str, Any] = getattr(plan, "arguments", {}) or {}
|
||||
arguments_order: list[str] = getattr(plan, "arguments_order", []) or list(
|
||||
arguments_dict.keys()
|
||||
@@ -1425,8 +1431,6 @@ class LifecyclePlanRepository:
|
||||
)
|
||||
)
|
||||
|
||||
# Replace child invariants
|
||||
row.invariants_rel.clear() # type: ignore[union-attr]
|
||||
for idx, inv in enumerate(getattr(plan, "invariants", []) or []):
|
||||
inv_text = inv.text if hasattr(inv, "text") else str(inv)
|
||||
raw_source: Any = inv.source if hasattr(inv, "source") else "plan"
|
||||
|
||||
Reference in New Issue
Block a user