test(e2e): implement E2E acceptance criteria test suites for v3.2.0 milestone #8970

Open
opened 2026-04-14 04:34:25 +00:00 by HAL9000 · 1 comment
Owner

Background and Context

Epic #739 requires a dedicated E2E acceptance criteria test suite for each milestone. The v3.2.0 milestone (M3: Decisions + Validations + Invariants) has 9 acceptance criteria that must be validated end-to-end using real LLM API keys with zero mocking. These tests exercise the real CleverAgents CLI — agents plan tree, agents plan explain, agents invariant add/list, and agents plan correct --mode=revert|append — exactly as a user would. Without these E2E suites, there is no automated verification that the milestone's acceptance criteria are satisfied in a real-world scenario.

Expected Behavior

When this issue is complete:

  • A Robot Framework E2E suite (tests/e2e/milestones/v3_2_0_acceptance.robot) exists, tagged @E2E
  • The suite validates all 9 v3.2.0 acceptance criteria using real LLM API calls
  • Output validation is flexible — checks major structural components without strict character-by-character comparison
  • The suite runs successfully under nox -s e2e_tests with real API keys

Acceptance Criteria

  • Robot Framework suite tests/e2e/milestones/v3_2_0_acceptance.robot exists and is tagged @E2E
  • Suite validates: decisions are recorded during Strategize with full context snapshots
  • Suite validates: agents plan tree renders the decision tree correctly
  • Suite validates: agents plan explain shows decision details including alternatives considered
  • Suite validates: agents invariant add creates invariants; agents invariant list displays them
  • Suite validates: invariants are enforced during strategize
  • Suite validates: agents plan correct --mode=revert re-executes from the targeted decision point
  • Suite validates: agents plan correct --mode=append adds guidance without recomputing
  • Output validation uses flexible matching (structural checks, not exact character matching)
  • Suite passes under nox -s e2e_tests with real LLM API keys
  • No mocking, stubbing, or test doubles of any kind in the suite

Subtasks

  • Create tests/e2e/milestones/ directory
  • Write v3_2_0_acceptance.robot with @E2E tag and test cases for each acceptance criterion
  • Implement flexible output validation keywords (e.g., Should Contain Structural Component)
  • Write test case for decision recording during Strategize
  • Write test case for agents plan tree rendering
  • Write test case for agents plan explain output
  • Write test cases for agents invariant add and agents invariant list
  • Write test case for invariant enforcement during strategize
  • Write test cases for agents plan correct --mode=revert and --mode=append
  • Run suite under nox -s e2e_tests and verify all tests pass

Definition of Done

  • All acceptance criteria met
  • Tests written and passing (coverage >= 97%)
  • Code reviewed and approved
  • Documentation updated if needed
  • No regressions introduced

Metadata

  • Commit message: test(e2e): implement E2E acceptance criteria test suites for v3.2.0 milestone
  • Branch name: test/e2e-v3-2-0-milestone-acceptance

Automated by CleverAgents Bot
Supervisor: Epic Planning Pool | Agent: epic-planning-pool-supervisor

## Background and Context Epic #739 requires a dedicated E2E acceptance criteria test suite for each milestone. The v3.2.0 milestone (M3: Decisions + Validations + Invariants) has 9 acceptance criteria that must be validated end-to-end using real LLM API keys with zero mocking. These tests exercise the real CleverAgents CLI — `agents plan tree`, `agents plan explain`, `agents invariant add/list`, and `agents plan correct --mode=revert|append` — exactly as a user would. Without these E2E suites, there is no automated verification that the milestone's acceptance criteria are satisfied in a real-world scenario. ## Expected Behavior When this issue is complete: - A Robot Framework E2E suite (`tests/e2e/milestones/v3_2_0_acceptance.robot`) exists, tagged `@E2E` - The suite validates all 9 v3.2.0 acceptance criteria using real LLM API calls - Output validation is flexible — checks major structural components without strict character-by-character comparison - The suite runs successfully under `nox -s e2e_tests` with real API keys ## Acceptance Criteria - [ ] Robot Framework suite `tests/e2e/milestones/v3_2_0_acceptance.robot` exists and is tagged `@E2E` - [ ] Suite validates: decisions are recorded during Strategize with full context snapshots - [ ] Suite validates: `agents plan tree` renders the decision tree correctly - [ ] Suite validates: `agents plan explain` shows decision details including alternatives considered - [ ] Suite validates: `agents invariant add` creates invariants; `agents invariant list` displays them - [ ] Suite validates: invariants are enforced during strategize - [ ] Suite validates: `agents plan correct --mode=revert` re-executes from the targeted decision point - [ ] Suite validates: `agents plan correct --mode=append` adds guidance without recomputing - [ ] Output validation uses flexible matching (structural checks, not exact character matching) - [ ] Suite passes under `nox -s e2e_tests` with real LLM API keys - [ ] No mocking, stubbing, or test doubles of any kind in the suite ## Subtasks - [ ] Create `tests/e2e/milestones/` directory - [ ] Write `v3_2_0_acceptance.robot` with `@E2E` tag and test cases for each acceptance criterion - [ ] Implement flexible output validation keywords (e.g., `Should Contain Structural Component`) - [ ] Write test case for decision recording during Strategize - [ ] Write test case for `agents plan tree` rendering - [ ] Write test case for `agents plan explain` output - [ ] Write test cases for `agents invariant add` and `agents invariant list` - [ ] Write test case for invariant enforcement during strategize - [ ] Write test cases for `agents plan correct --mode=revert` and `--mode=append` - [ ] Run suite under `nox -s e2e_tests` and verify all tests pass ## Definition of Done - [ ] All acceptance criteria met - [ ] Tests written and passing (coverage >= 97%) - [ ] Code reviewed and approved - [ ] Documentation updated if needed - [ ] No regressions introduced ## Metadata - **Commit message:** `test(e2e): implement E2E acceptance criteria test suites for v3.2.0 milestone` - **Branch name:** `test/e2e-v3-2-0-milestone-acceptance` --- **Automated by CleverAgents Bot** Supervisor: Epic Planning Pool | Agent: epic-planning-pool-supervisor
HAL9000 added this to the v3.2.0 milestone 2026-04-14 04:45:21 +00:00
Author
Owner

Verified — E2E acceptance criteria test suite for v3.2.0 is required for milestone completion. MoSCoW: Must-have. Priority: Medium.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — E2E acceptance criteria test suite for v3.2.0 is required for milestone completion. MoSCoW: Must-have. Priority: Medium. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#8970
No description provided.