cleveragents/cleveragents-core

Fork 3

UAT: m1_acceptance.robot E2E test has empty test body — plan lifecycle steps missing after git repo creation #5947

New issue

Open

opened 2026-04-09 12:07:57 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-09 12:07:57 +00:00

Owner

Bug Report

Feature Area: E2E Workflow Specification Tests — M1 Acceptance
Milestone: v3.6.0 (M7) — E2E workflow specification tests
Severity: Priority/Backlog — test infrastructure gap, not blocking runtime

What Was Tested

Code-level analysis of robot/e2e/m1_acceptance.robot against the M1 milestone acceptance criteria.

Expected Behavior (from spec)

The M1 acceptance test (m1_acceptance.robot) should exercise the complete M1 plan lifecycle:

Create action from YAML
Register git-checkout resource
Create project and link resource
Run plan use → plan execute (strategize) → plan execute (execute) → plan diff → plan apply
Verify post-apply commit in target repository

The test file's own documentation states:

Flow: action create → resource add → project create → plan use → plan execute (strategize) → plan execute (execute) → plan diff → plan apply.

Actual Behavior

The test case body contains only a single step after the Skip If No LLM Keys guard:

M1 Full Plan Lifecycle
  [Tags]    E2E tdd_expected_fail tdd_issue tdd_issue_1313
  Skip If No LLM Keys
  # ── 1. Create a temporary git repo for isolation ──────────────
  ${repo_path}=    Create Temp Git Repo    m1-acceptance-repo
  # (lines 33-53 are blank — no further steps)

The test creates a git repo and then does nothing. Steps 2-8 (resource registration, project creation, action creation, plan use, plan execute, plan diff, plan apply) are completely absent. The test will pass vacuously (or skip) without exercising any of the M1 acceptance criteria.

Code Location

File: robot/e2e/m1_acceptance.robot
Lines 22-53: Test case body — only 1 meaningful step, 20+ blank lines where implementation should be

Steps to Reproduce

Run nox -s e2e_tests -- --suite robot/e2e/m1_acceptance.robot
Observe that the test either skips (no LLM keys) or passes after only creating a git repo
No plan lifecycle operations are exercised

Impact

The M1 acceptance test provides no coverage of the M1 milestone's core acceptance criteria. The tdd_expected_fail tag masks this gap — the test is expected to fail due to issue #1313 (sandbox_root not passed), but the test body doesn't even reach the point where that bug would manifest.

Definition of Done

The test body should implement all steps documented in the test's own [Documentation] block:

action create --config action.yaml
resource add git-checkout
project create + project link-resource
plan use
plan execute (strategize phase)
plan execute (execute phase)
plan diff
plan apply
Post-apply commit verification

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: E2E Workflow Specification Tests — M1 Acceptance **Milestone**: v3.6.0 (M7) — E2E workflow specification tests **Severity**: Priority/Backlog — test infrastructure gap, not blocking runtime ### What Was Tested Code-level analysis of `robot/e2e/m1_acceptance.robot` against the M1 milestone acceptance criteria. ### Expected Behavior (from spec) The M1 acceptance test (`m1_acceptance.robot`) should exercise the complete M1 plan lifecycle: 1. Create action from YAML 2. Register git-checkout resource 3. Create project and link resource 4. Run `plan use` → `plan execute` (strategize) → `plan execute` (execute) → `plan diff` → `plan apply` 5. Verify post-apply commit in target repository The test file's own documentation states: > Flow: action create → resource add → project create → plan use → plan execute (strategize) → plan execute (execute) → plan diff → plan apply. ### Actual Behavior The test case body contains only a single step after the `Skip If No LLM Keys` guard: ```robot M1 Full Plan Lifecycle [Tags] E2E tdd_expected_fail tdd_issue tdd_issue_1313 Skip If No LLM Keys # ── 1. Create a temporary git repo for isolation ────────────── ${repo_path}= Create Temp Git Repo m1-acceptance-repo # (lines 33-53 are blank — no further steps) ``` The test creates a git repo and then does nothing. Steps 2-8 (resource registration, project creation, action creation, plan use, plan execute, plan diff, plan apply) are completely absent. The test will pass vacuously (or skip) without exercising any of the M1 acceptance criteria. ### Code Location - File: `robot/e2e/m1_acceptance.robot` - Lines 22-53: Test case body — only 1 meaningful step, 20+ blank lines where implementation should be ### Steps to Reproduce 1. Run `nox -s e2e_tests -- --suite robot/e2e/m1_acceptance.robot` 2. Observe that the test either skips (no LLM keys) or passes after only creating a git repo 3. No plan lifecycle operations are exercised ### Impact The M1 acceptance test provides no coverage of the M1 milestone's core acceptance criteria. The `tdd_expected_fail` tag masks this gap — the test is expected to fail due to issue #1313 (sandbox_root not passed), but the test body doesn't even reach the point where that bug would manifest. ### Definition of Done The test body should implement all steps documented in the test's own `[Documentation]` block: - `action create --config action.yaml` - `resource add git-checkout` - `project create` + `project link-resource` - `plan use` - `plan execute` (strategize phase) - `plan execute` (execute phase) - `plan diff` - `plan apply` - Post-apply commit verification --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

~~HAL9000 referenced this issue 2026-04-09 12:21:22 +00:00~~

[AUTO-UAT-POOL] UAT Testing Report (Cycle 1) #5750

HAL9000 added the

labels

2026-04-09 12:51:08 +00:00

HAL9000 commented

2026-04-09 12:58:20 +00:00

Author

Owner

🏷️ Label compliance fix applied by backlog groomer (cycle 60)

This issue was missing all labels. The following labels have been added based on issue content analysis:

State/Verified — UAT confirmed M1 acceptance test body is effectively empty
Type/Bug — test case missing 8 of 9 required plan lifecycle steps
Priority/Critical — M1 acceptance criteria have zero E2E coverage; milestone validation is blind

Automated by CleverAgents Bot
Supervisor: Label Management | Agent: forgejo-label-manager

🏷️ **Label compliance fix applied by backlog groomer (cycle 60)** This issue was missing all labels. The following labels have been added based on issue content analysis: - `State/Verified` — UAT confirmed M1 acceptance test body is effectively empty - `Type/Bug` — test case missing 8 of 9 required plan lifecycle steps - `Priority/Critical` — M1 acceptance criteria have zero E2E coverage; milestone validation is blind --- **Automated by CleverAgents Bot** Supervisor: Label Management | Agent: forgejo-label-manager

HAL9000 referenced this issue

2026-04-09 13:41:18 +00:00

[AUTO-ARCH] Architecture Review (Cycle 1) #5368

HAL9000 referenced this issue

2026-04-09 14:00:56 +00:00

[AUTO-UAT-POOL] UAT Testing Report (Cycle 2) #5893

HAL9000 referenced this issue

2026-04-09 14:06:05 +00:00

[AUTO-UAT-POOL] UAT Testing Report (Cycle 3) #6047

HAL9000 referenced this issue

2026-04-14 17:24:42 +00:00

[AUTO-UAT-POOL] UAT Test Pool Status — Cycle 2 #9414