test(e2e): E2E acceptance criteria for M3 (v3.2.0) — decisions, validations, and invariants #743

Open
opened 2026-03-12 19:33:44 +00:00 by freemo · 1 comment
Owner

Metadata

  • Commit Message: test(e2e): E2E acceptance criteria for M3 (v3.2.0) — decisions, validations, and invariants
  • Branch: test/e2e-m3-acceptance

Background

True end-to-end acceptance test for the M3 (v3.2.0) milestone: Decisions + Validations + Invariants. This test exercises the complete M3 success criteria with zero mocking — real CLI invocations, real LLM API keys, real subprocess execution. The test validates that decisions are recorded during Strategize with full context snapshots, the decision tree persists and renders correctly, corrections in revert mode re-execute from decision points, and invariants are enforced during strategize.

This is a Robot Framework test tagged with @E2E, running in the dedicated nox -s e2e_tests session.

Expected Behavior

The E2E test exercises decision recording, decision tree visualization (plan tree), decision explanation (plan explain), invariant management (invariant add/list), and decision correction (plan correct --mode=revert) through real CLI commands with real LLM API keys.

Acceptance Criteria

  • Robot Framework test suite tagged with [Tags] E2E in robot/e2e/ directory
  • Test executes a plan and verifies decisions are recorded with context snapshots
  • Test runs plan tree and verifies decision tree renders correctly
  • Test runs plan explain and verifies decision details are shown
  • Test adds invariants via invariant add and verifies enforcement during strategize
  • Test runs plan correct --mode=revert and verifies re-execution from decision point
  • All CLI invocations use real LLM API keys (no mocking, stubbing, or test doubles)
  • Output validation is flexible — checks structural components, not exact character matching
  • Test passes via nox -s e2e_tests
  • Coverage >=97% maintained

Subtasks

  • Write Robot Framework E2E test suite robot/e2e/m3_acceptance.robot with [Tags] E2E
  • Implement decision tree and explanation verification steps
  • Implement invariant management and enforcement verification steps
  • Implement correction/revert verification steps
  • Add flexible output assertions
  • Verify test passes with real LLM API keys via nox -s e2e_tests
  • Tests (Behave): N/A (this is an E2E test issue)
  • Tests (Robot): The E2E Robot test suite IS this issue's deliverable
  • Verify coverage >=97% via nox -s coverage_report
  • Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
  • The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
## Metadata - **Commit Message**: `test(e2e): E2E acceptance criteria for M3 (v3.2.0) — decisions, validations, and invariants` - **Branch**: `test/e2e-m3-acceptance` ## Background True end-to-end acceptance test for the M3 (v3.2.0) milestone: Decisions + Validations + Invariants. This test exercises the complete M3 success criteria with **zero mocking** — real CLI invocations, real LLM API keys, real subprocess execution. The test validates that decisions are recorded during Strategize with full context snapshots, the decision tree persists and renders correctly, corrections in revert mode re-execute from decision points, and invariants are enforced during strategize. This is a Robot Framework test tagged with `@E2E`, running in the dedicated `nox -s e2e_tests` session. ## Expected Behavior The E2E test exercises decision recording, decision tree visualization (`plan tree`), decision explanation (`plan explain`), invariant management (`invariant add/list`), and decision correction (`plan correct --mode=revert`) through real CLI commands with real LLM API keys. ## Acceptance Criteria - [ ] Robot Framework test suite tagged with `[Tags] E2E` in `robot/e2e/` directory - [ ] Test executes a plan and verifies decisions are recorded with context snapshots - [ ] Test runs `plan tree` and verifies decision tree renders correctly - [ ] Test runs `plan explain` and verifies decision details are shown - [ ] Test adds invariants via `invariant add` and verifies enforcement during strategize - [ ] Test runs `plan correct --mode=revert` and verifies re-execution from decision point - [ ] All CLI invocations use real LLM API keys (no mocking, stubbing, or test doubles) - [ ] Output validation is flexible — checks structural components, not exact character matching - [ ] Test passes via `nox -s e2e_tests` - [ ] Coverage >=97% maintained ## Subtasks - [ ] Write Robot Framework E2E test suite `robot/e2e/m3_acceptance.robot` with `[Tags] E2E` - [ ] Implement decision tree and explanation verification steps - [ ] Implement invariant management and enforcement verification steps - [ ] Implement correction/revert verification steps - [ ] Add flexible output assertions - [ ] Verify test passes with real LLM API keys via `nox -s e2e_tests` - [ ] Tests (Behave): N/A (this is an E2E test issue) - [ ] Tests (Robot): The E2E Robot test suite IS this issue's deliverable - [ ] Verify coverage >=97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.
freemo self-assigned this 2026-03-12 19:33:44 +00:00
freemo added this to the v3.2.0 milestone 2026-03-12 19:33:44 +00:00
Author
Owner

Implementation complete. PR #799 (test/e2e-m3-acceptancemaster).

Deliverable: robot/e2e/m3_acceptance.robot — 14-step E2E acceptance test covering decision recording, plan tree, plan explain, invariant add/invariant list, plan correct --mode=revert, plan diff, plan lifecycle-apply, and plan status.

Quality gates: nox lint, format, typecheck all passed. Coverage >= 97% maintained (E2E tests excluded from Behave coverage).

Implementation complete. PR #799 (`test/e2e-m3-acceptance` → `master`). **Deliverable**: `robot/e2e/m3_acceptance.robot` — 14-step E2E acceptance test covering decision recording, `plan tree`, `plan explain`, `invariant add`/`invariant list`, `plan correct --mode=revert`, `plan diff`, `plan lifecycle-apply`, and `plan status`. **Quality gates**: nox lint, format, typecheck all passed. Coverage >= 97% maintained (E2E tests excluded from Behave coverage).
freemo removed their assignment 2026-03-22 23:45:13 +00:00
freemo self-assigned this 2026-04-02 06:13:50 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#743
No description provided.