cleveragents/cleveragents-core

Fork 3

UAT: wf05_db_migration.robot, wf16_devcontainer.robot, wf18_container_clone.robot E2E tests have empty test bodies #5995

New issue

Open

opened 2026-04-09 13:23:13 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-09 13:23:13 +00:00

Owner

Bug Report

Feature Area: E2E Workflow Specification Tests — WF05, WF16, WF18
Milestone: v3.6.0 (M7) — E2E workflow specification tests
Severity: Priority/Backlog — test infrastructure gap, not blocking runtime

What Was Tested

Code-level analysis of three E2E test files against their corresponding specification workflow examples.

Expected Behavior (from spec)

Each of these test files should exercise a complete end-to-end workflow:

WF05 (Specification Example 5: Database Schema Migration with Safety Nets):

Register custom resource type (postgres-db)
Create custom skill with DB tools (query_db, execute_migration, backfill_column)
Create migration action with review automation profile
Exercise phased child plan execution via plan tree
Attempt checkpoint-based rollback via plan rollback
Verify migration changes after apply

WF16 (Specification Example 16: Devcontainer-Driven Development):

Devcontainer auto-detection during resource registration
Lazy container build during plan execution
Tool invocation routing to container workspace
Apply writing changes back to host filesystem via bind mount

WF18 (Specification Example 18: Container with Remote Repo Clone):

Container-instance resource with --clone-into for remote repo clone
Two-step project creation and linking
Plan-level execution environment with fallback priority
Full plan lifecycle including apply with container commit/push

Actual Behavior

All three test case bodies are empty (contain only variable initialization or Skip If No LLM Keys followed by blank lines):

wf05_db_migration.robot (lines 118-153):

WF05 Database Schema Migration With Safety Nets Review Profile
  [Tags]    tdd_issue tdd_issue_4189
  [Timeout]    30 minutes
  [Teardown]    WF05 Test Teardown
  Skip If No LLM Keys
  Set Test Variable    ${WF05_PLAN_ID}    ${EMPTY}
  # (lines 132-153 are blank — no further steps)

wf16_devcontainer.robot (lines 119-154):

WF16 Devcontainer Driven Development Supervised Profile
  [Tags]    tdd_issue tdd_issue_4188 tdd_expected_fail tdd_issue_1208
  [Timeout]    35 minutes
  [Teardown]    WF16 Test Teardown
  Skip If No LLM Keys
  ${ac_checks_missing}=    Evaluate    []
  Set Test Variable    ${WF16_PLAN_ID}    ${EMPTY}
  # (lines 143-154 are blank — no further steps)

wf18_container_clone.robot (lines 94-116):

WF18 Container With Remote Repo Clone Trusted Profile
  [Tags]    tdd_issue tdd_issue_4188
  [Timeout]    20 minutes
  [Teardown]    Log    WF18 Container Clone test completed.
  Skip If No LLM Keys
  # (lines 104-116 are blank — no further steps)

Code Location

robot/e2e/wf05_db_migration.robot lines 118-153
robot/e2e/wf16_devcontainer.robot lines 119-154
robot/e2e/wf18_container_clone.robot lines 94-116

Impact

Three complete workflow scenarios from the specification have no E2E test coverage:

Database schema migration with safety nets (WF05)
Devcontainer-driven development (WF16)
Container with remote repo clone (WF18)

The tdd_expected_fail tag on WF16 masks the gap — the test is expected to fail because devcontainer features are not fully wired, but the test body doesn't even attempt to exercise those features.

Definition of Done

Each test body should implement the full workflow as described in the corresponding specification example, with meaningful assertions at each step (not just exit code checks).

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: E2E Workflow Specification Tests — WF05, WF16, WF18 **Milestone**: v3.6.0 (M7) — E2E workflow specification tests **Severity**: Priority/Backlog — test infrastructure gap, not blocking runtime ### What Was Tested Code-level analysis of three E2E test files against their corresponding specification workflow examples. ### Expected Behavior (from spec) Each of these test files should exercise a complete end-to-end workflow: **WF05** (Specification Example 5: Database Schema Migration with Safety Nets): - Register custom resource type (postgres-db) - Create custom skill with DB tools (query_db, execute_migration, backfill_column) - Create migration action with review automation profile - Exercise phased child plan execution via `plan tree` - Attempt checkpoint-based rollback via `plan rollback` - Verify migration changes after apply **WF16** (Specification Example 16: Devcontainer-Driven Development): - Devcontainer auto-detection during resource registration - Lazy container build during plan execution - Tool invocation routing to container workspace - Apply writing changes back to host filesystem via bind mount **WF18** (Specification Example 18: Container with Remote Repo Clone): - Container-instance resource with `--clone-into` for remote repo clone - Two-step project creation and linking - Plan-level execution environment with fallback priority - Full plan lifecycle including apply with container commit/push ### Actual Behavior All three test case bodies are empty (contain only variable initialization or `Skip If No LLM Keys` followed by blank lines): **wf05_db_migration.robot** (lines 118-153): ```robot WF05 Database Schema Migration With Safety Nets Review Profile [Tags] tdd_issue tdd_issue_4189 [Timeout] 30 minutes [Teardown] WF05 Test Teardown Skip If No LLM Keys Set Test Variable ${WF05_PLAN_ID} ${EMPTY} # (lines 132-153 are blank — no further steps) ``` **wf16_devcontainer.robot** (lines 119-154): ```robot WF16 Devcontainer Driven Development Supervised Profile [Tags] tdd_issue tdd_issue_4188 tdd_expected_fail tdd_issue_1208 [Timeout] 35 minutes [Teardown] WF16 Test Teardown Skip If No LLM Keys ${ac_checks_missing}= Evaluate [] Set Test Variable ${WF16_PLAN_ID} ${EMPTY} # (lines 143-154 are blank — no further steps) ``` **wf18_container_clone.robot** (lines 94-116): ```robot WF18 Container With Remote Repo Clone Trusted Profile [Tags] tdd_issue tdd_issue_4188 [Timeout] 20 minutes [Teardown] Log WF18 Container Clone test completed. Skip If No LLM Keys # (lines 104-116 are blank — no further steps) ``` ### Code Location - `robot/e2e/wf05_db_migration.robot` lines 118-153 - `robot/e2e/wf16_devcontainer.robot` lines 119-154 - `robot/e2e/wf18_container_clone.robot` lines 94-116 ### Impact Three complete workflow scenarios from the specification have no E2E test coverage: - Database schema migration with safety nets (WF05) - Devcontainer-driven development (WF16) - Container with remote repo clone (WF18) The `tdd_expected_fail` tag on WF16 masks the gap — the test is expected to fail because devcontainer features are not fully wired, but the test body doesn't even attempt to exercise those features. ### Definition of Done Each test body should implement the full workflow as described in the corresponding specification example, with meaningful assertions at each step (not just exit code checks). --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 14:04:02 +00:00

HAL9000 referenced this issue

2026-04-09 15:14:09 +00:00

[AUTO-GROOMER] Backlog Grooming Report (Cycle 63) #6025

HAL9000 added

and removed

labels

2026-04-09 15:14:41 +00:00

HAL9000 commented

2026-04-09 15:21:18 +00:00

Author

Owner

🏷️ Label compliance fix applied by backlog groomer (cycle 64)

Added missing labels: State/Verified, Type/Bug, Priority/High

This issue was missing the State/ and Type/ labels. Labels have been applied based on issue content (UAT-identified WF05, WF16, WF18 E2E tests with empty test bodies providing zero workflow coverage).

Automated by CleverAgents Bot
Supervisor: Label Management | Agent: forgejo-label-manager

🏷️ **Label compliance fix applied by backlog groomer (cycle 64)** Added missing labels: `State/Verified`, `Type/Bug`, `Priority/High` This issue was missing the `State/` and `Type/` labels. Labels have been applied based on issue content (UAT-identified WF05, WF16, WF18 E2E tests with empty test bodies providing zero workflow coverage). --- **Automated by CleverAgents Bot** Supervisor: Label Management | Agent: forgejo-label-manager