E2E acceptance tests m1_acceptance.robot and m2_acceptance.robot have empty test bodies #8487

New issue

Open

opened 2026-04-13 20:08:33 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-13 20:08:33 +00:00

Owner

Background and Context

During UAT monitoring of issue #8459 (E2E test regression), it was discovered that while robot/e2e/m1_acceptance.robot and robot/e2e/m2_acceptance.robot are physically present in the repository (unlike the 16 files removed by commit 8ea00f518582d28c1653d76db1223d2af5fa88c7), their test case bodies are functionally empty — the test case keyword sections contain only blank lines after the initial setup/skip guard, with no actual test steps exercising the M1/M2 acceptance criteria.

This means the files exist on disk but provide zero test coverage of the milestone acceptance scenarios they are supposed to validate.

Current Behavior

robot/e2e/m1_acceptance.robot — Test case M1 Full Plan Lifecycle:

Sets up a temp git repo
Calls Skip If No LLM Keys
Then has 20+ blank lines with no test steps
No assertions, no CLI invocations, no plan lifecycle exercise

robot/e2e/m2_acceptance.robot — Test case M2 Full Actor Compiler And LLM Integration:

Creates a temp git repo and commits files
Then has 8+ blank lines with no test steps
No assertions, no CLI invocations, no actor compiler exercise

Both files have [Tags] E2E tdd_issue tdd_issue_4189 tdd_expected_fail — the tdd_expected_fail tag means the TDD listener will invert the result, but with an empty body the test will pass vacuously (no assertions = no failures = no inversions), giving a false green signal.

Expected Behavior

Both acceptance test files must contain complete test step sequences that:

Exercise the full M1/M2 plan lifecycle (action create → resource add → project create → plan use → plan execute → plan diff → plan apply)
Include meaningful assertions beyond return codes (output validation, state verification, artifact inspection)
Verify the acceptance criteria documented in the test's [Documentation] block
Either pass (if the underlying bug is fixed) or fail with tdd_expected_fail inversion (if the bug is still open)

Acceptance Criteria

robot/e2e/m1_acceptance.robot test body is complete with all M1 lifecycle steps and assertions
robot/e2e/m2_acceptance.robot test body is complete with all M2 actor compiler steps and assertions
Tests include output validation (not just return code checks)
Tests run under nox -s e2e_tests without vacuous pass
tdd_expected_fail tag is retained if the underlying bugs (#1313 for M1, #4189 for M2) are still open

Supporting Information

Discovered at HEAD commit: c5820266fd5b63e1dbfce833159e6d9791d530f4
Related to: #8459 (E2E test regression — broader context)
Test framework: Robot Framework
Session: nox -s e2e_tests

Metadata

Commit Message: test: restore m1 and m2 acceptance test bodies
Branch: test/restore-m1-m2-acceptance-bodies

Subtasks

Restore complete test body for robot/e2e/m1_acceptance.robot (M1 Full Plan Lifecycle)
Restore complete test body for robot/e2e/m2_acceptance.robot (M2 Full Actor Compiler And LLM Integration)
Verify tests include meaningful assertions (output validation, state checks)
Verify nox -s e2e_tests runs both tests without vacuous pass
Confirm tdd_expected_fail tag handling is correct for each test

Definition of Done

This issue is complete when:

Both m1_acceptance.robot and m2_acceptance.robot have complete, non-empty test bodies
Each test exercises the full CLI lifecycle documented in its [Documentation] block
Tests include at least 3 meaningful assertions beyond return code checks
nox -s e2e_tests executes both tests (they may fail with tdd_expected_fail inversion if underlying bugs are open, but must not vacuously pass)
A single Git commit is created where the first line is exactly:
```
test: restore m1 and m2 acceptance test bodies
```
The commit is pushed to test/restore-m1-m2-acceptance-bodies and submitted as a PR to master

Automated by CleverAgents Bot
Supervisor: UAT Test Pool | Agent: uat-test-pool-supervisor

## Background and Context During UAT monitoring of issue #8459 (E2E test regression), it was discovered that while `robot/e2e/m1_acceptance.robot` and `robot/e2e/m2_acceptance.robot` are physically present in the repository (unlike the 16 files removed by commit `8ea00f518582d28c1653d76db1223d2af5fa88c7`), their test case bodies are **functionally empty** — the test case keyword sections contain only blank lines after the initial setup/skip guard, with no actual test steps exercising the M1/M2 acceptance criteria. This means the files exist on disk but provide **zero test coverage** of the milestone acceptance scenarios they are supposed to validate. ## Current Behavior `robot/e2e/m1_acceptance.robot` — Test case `M1 Full Plan Lifecycle`: - Sets up a temp git repo - Calls `Skip If No LLM Keys` - Then has **20+ blank lines** with no test steps - No assertions, no CLI invocations, no plan lifecycle exercise `robot/e2e/m2_acceptance.robot` — Test case `M2 Full Actor Compiler And LLM Integration`: - Creates a temp git repo and commits files - Then has **8+ blank lines** with no test steps - No assertions, no CLI invocations, no actor compiler exercise Both files have `[Tags] E2E tdd_issue tdd_issue_4189 tdd_expected_fail` — the `tdd_expected_fail` tag means the TDD listener will invert the result, but with an empty body the test will pass vacuously (no assertions = no failures = no inversions), giving a false green signal. ## Expected Behavior Both acceptance test files must contain complete test step sequences that: 1. Exercise the full M1/M2 plan lifecycle (action create → resource add → project create → plan use → plan execute → plan diff → plan apply) 2. Include meaningful assertions beyond return codes (output validation, state verification, artifact inspection) 3. Verify the acceptance criteria documented in the test's `[Documentation]` block 4. Either pass (if the underlying bug is fixed) or fail with `tdd_expected_fail` inversion (if the bug is still open) ## Acceptance Criteria - [ ] `robot/e2e/m1_acceptance.robot` test body is complete with all M1 lifecycle steps and assertions - [ ] `robot/e2e/m2_acceptance.robot` test body is complete with all M2 actor compiler steps and assertions - [ ] Tests include output validation (not just return code checks) - [ ] Tests run under `nox -s e2e_tests` without vacuous pass - [ ] `tdd_expected_fail` tag is retained if the underlying bugs (#1313 for M1, #4189 for M2) are still open ## Supporting Information - Discovered at HEAD commit: `c5820266fd5b63e1dbfce833159e6d9791d530f4` - Related to: #8459 (E2E test regression — broader context) - Test framework: Robot Framework - Session: `nox -s e2e_tests` ## Metadata - **Commit Message**: `test: restore m1 and m2 acceptance test bodies` - **Branch**: `test/restore-m1-m2-acceptance-bodies` ## Subtasks - [ ] Restore complete test body for `robot/e2e/m1_acceptance.robot` (M1 Full Plan Lifecycle) - [ ] Restore complete test body for `robot/e2e/m2_acceptance.robot` (M2 Full Actor Compiler And LLM Integration) - [ ] Verify tests include meaningful assertions (output validation, state checks) - [ ] Verify `nox -s e2e_tests` runs both tests without vacuous pass - [ ] Confirm `tdd_expected_fail` tag handling is correct for each test ## Definition of Done This issue is complete when: - Both `m1_acceptance.robot` and `m2_acceptance.robot` have complete, non-empty test bodies - Each test exercises the full CLI lifecycle documented in its `[Documentation]` block - Tests include at least 3 meaningful assertions beyond return code checks - `nox -s e2e_tests` executes both tests (they may fail with `tdd_expected_fail` inversion if underlying bugs are open, but must not vacuously pass) - A single Git commit is created where the **first line** is exactly: ``` test: restore m1 and m2 acceptance test bodies ``` - The commit is pushed to `test/restore-m1-m2-acceptance-bodies` and submitted as a PR to `master` --- **Automated by CleverAgents Bot** Supervisor: UAT Test Pool | Agent: uat-test-pool-supervisor

HAL9000 added the

labels

2026-04-13 20:09:55 +00:00