test(e2e): implement E2E workflow example test suites for specification examples 10–18 #8975

Open
opened 2026-04-14 04:34:46 +00:00 by HAL9000 · 1 comment
Owner

Background and Context

Epic #739 requires all 18 specification workflow examples to have dedicated E2E Robot Framework test suites. This issue covers the second half — workflow examples 10–18. Each test must use real LLM API keys with zero mocking, exercising the actual CLI subprocess exactly as a user would. These later workflow examples typically cover more advanced features (e.g., decision correction, invariant enforcement, multi-step plan workflows) and are critical for validating the full breadth of the specification. Without these tests, the E2E suite is incomplete and the Epic cannot be closed.

Expected Behavior

When this issue is complete:

  • Robot Framework E2E suites for workflow examples 10–18 exist under tests/e2e/workflow_examples/, each tagged @E2E
  • Each suite exercises the exact command sequence from the corresponding specification workflow example
  • Output validation is flexible — checks major structural components without strict character-by-character comparison
  • All 9 suites pass under nox -s e2e_tests with real LLM API keys
  • No mocking, stubbing, or test doubles of any kind are used

Acceptance Criteria

  • One .robot file per workflow example (examples 10–18) exists under tests/e2e/workflow_examples/, each tagged @E2E
  • Workflow example 10 E2E test passes with real LLM API keys
  • Workflow example 11 E2E test passes with real LLM API keys
  • Workflow example 12 E2E test passes with real LLM API keys
  • Workflow example 13 E2E test passes with real LLM API keys
  • Workflow example 14 E2E test passes with real LLM API keys
  • Workflow example 15 E2E test passes with real LLM API keys
  • Workflow example 16 E2E test passes with real LLM API keys
  • Workflow example 17 E2E test passes with real LLM API keys
  • Workflow example 18 E2E test passes with real LLM API keys
  • Output validation uses flexible matching (structural checks, not exact character matching)
  • No mocking, stubbing, or test doubles of any kind in any suite
  • All 18 workflow example suites (examples 1–18) pass together under nox -s e2e_tests

Subtasks

  • Write workflow_example_10.robot through workflow_example_18.robot, each tagged @E2E
  • Reuse shared E2E keywords library from tests/e2e/resources/e2e_keywords.resource
  • Map each workflow example to its specification section and document the reference in the suite header
  • Run all 9 suites under nox -s e2e_tests and verify they pass with real API keys
  • Run the full 18-suite E2E suite together and verify no cross-suite interference
  • Update tests/e2e/README.md to mark all 18 workflow example suites as complete

Definition of Done

  • All acceptance criteria met
  • Tests written and passing (coverage >= 97%)
  • Code reviewed and approved
  • Documentation updated if needed
  • No regressions introduced

Metadata

  • Commit message: test(e2e): implement E2E workflow example test suites for specification examples 10–18
  • Branch name: test/e2e-workflow-examples-10-18

Automated by CleverAgents Bot
Supervisor: Epic Planning Pool | Agent: epic-planning-pool-supervisor

## Background and Context Epic #739 requires all 18 specification workflow examples to have dedicated E2E Robot Framework test suites. This issue covers the second half — workflow examples 10–18. Each test must use real LLM API keys with zero mocking, exercising the actual CLI subprocess exactly as a user would. These later workflow examples typically cover more advanced features (e.g., decision correction, invariant enforcement, multi-step plan workflows) and are critical for validating the full breadth of the specification. Without these tests, the E2E suite is incomplete and the Epic cannot be closed. ## Expected Behavior When this issue is complete: - Robot Framework E2E suites for workflow examples 10–18 exist under `tests/e2e/workflow_examples/`, each tagged `@E2E` - Each suite exercises the exact command sequence from the corresponding specification workflow example - Output validation is flexible — checks major structural components without strict character-by-character comparison - All 9 suites pass under `nox -s e2e_tests` with real LLM API keys - No mocking, stubbing, or test doubles of any kind are used ## Acceptance Criteria - [ ] One `.robot` file per workflow example (examples 10–18) exists under `tests/e2e/workflow_examples/`, each tagged `@E2E` - [ ] Workflow example 10 E2E test passes with real LLM API keys - [ ] Workflow example 11 E2E test passes with real LLM API keys - [ ] Workflow example 12 E2E test passes with real LLM API keys - [ ] Workflow example 13 E2E test passes with real LLM API keys - [ ] Workflow example 14 E2E test passes with real LLM API keys - [ ] Workflow example 15 E2E test passes with real LLM API keys - [ ] Workflow example 16 E2E test passes with real LLM API keys - [ ] Workflow example 17 E2E test passes with real LLM API keys - [ ] Workflow example 18 E2E test passes with real LLM API keys - [ ] Output validation uses flexible matching (structural checks, not exact character matching) - [ ] No mocking, stubbing, or test doubles of any kind in any suite - [ ] All 18 workflow example suites (examples 1–18) pass together under `nox -s e2e_tests` ## Subtasks - [ ] Write `workflow_example_10.robot` through `workflow_example_18.robot`, each tagged `@E2E` - [ ] Reuse shared E2E keywords library from `tests/e2e/resources/e2e_keywords.resource` - [ ] Map each workflow example to its specification section and document the reference in the suite header - [ ] Run all 9 suites under `nox -s e2e_tests` and verify they pass with real API keys - [ ] Run the full 18-suite E2E suite together and verify no cross-suite interference - [ ] Update `tests/e2e/README.md` to mark all 18 workflow example suites as complete ## Definition of Done - [ ] All acceptance criteria met - [ ] Tests written and passing (coverage >= 97%) - [ ] Code reviewed and approved - [ ] Documentation updated if needed - [ ] No regressions introduced ## Metadata - **Commit message:** `test(e2e): implement E2E workflow example test suites for specification examples 10–18` - **Branch name:** `test/e2e-workflow-examples-10-18` --- **Automated by CleverAgents Bot** Supervisor: Epic Planning Pool | Agent: epic-planning-pool-supervisor
HAL9000 added this to the v3.2.0 milestone 2026-04-14 04:45:22 +00:00
Author
Owner

Verified — E2E test suites for workflow examples 10-18 are required for v3.2.0. MoSCoW: Must-have. Priority: Medium.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — E2E test suites for workflow examples 10-18 are required for v3.2.0. MoSCoW: Must-have. Priority: Medium. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#8975
No description provided.