UAT: 9 of 18 spec workflow examples have no E2E tests — Examples 1, 2, 3, 6, 9, 10, 11, 13, 15 missing from robot/e2e/ #6008

Open
opened 2026-04-09 13:35:39 +00:00 by HAL9000 · 1 comment
Owner

Bug Report

Feature Area: E2E Workflow Specification Tests — Missing Workflow Coverage
Milestone: v3.6.0 (M7) — E2E workflow specification tests
Severity: Priority/Backlog — test infrastructure gap, not blocking runtime

What Was Tested

Code-level analysis of robot/e2e/ directory against the 18 workflow examples in docs/specification.md (Section: "Workflow Examples").

Expected Behavior (from spec)

The v3.6.0 milestone scope explicitly includes "E2E workflow specification tests" (issue #679: "Translating 18 spec workflow examples"). The specification contains 18 complete workflow examples that should each have a corresponding E2E test in robot/e2e/.

Actual Behavior

Only 9 of the 18 workflow examples have E2E test files. The following 9 examples have no E2E test at all:

Spec Example Description E2E File Status
Example 1 Hello World — Fix a Single Bug None Missing
Example 2 Automated Test Generation for a Module None (wf02 is in robot/, not e2e/) Missing
Example 3 Multi-File Refactoring with Invariants None Missing
Example 6 Writing Technical Documentation from Codebase Analysis None Missing
Example 9 Session-Driven Interactive Exploration None Missing
Example 10 Full-Auto Batch Operations — Formatting and Linting None Missing
Example 11 Complex Graph Actor for Multi-Stage Code Review None Missing
Example 13 Custom Automation Profile with Semantic Escalation None Missing
Example 15 Disaster Recovery — Rollback a Failed Apply None Missing

The 9 examples that DO have E2E test files (though many have empty/stub bodies — see separate bugs):

  • Example 4 → wf04_multi_project.robot (empty body)
  • Example 5 → wf05_db_migration.robot (empty body)
  • Example 7 → wf07_cicd.robot ✓ (has real tests)
  • Example 12 → wf12_hierarchical.robot (TDD placeholder)
  • Example 14 → wf14_server_mode.robot ✓ (has real tests)
  • Example 16 → wf16_devcontainer.robot (empty body)
  • Example 17 → wf17_explicit_container.robot (all Skip)
  • Example 18 → wf18_container_clone.robot (empty body)

Code Location

  • robot/e2e/ directory — missing wf01, wf02 (e2e version), wf03, wf06, wf09, wf10, wf11, wf13, wf15 files

Impact

9 complete workflow scenarios from the specification have zero E2E test coverage:

  1. Example 1 (Hello World): The most basic workflow — single bug fix with manual profile. This is the entry-level scenario that all new users follow. No E2E test.

  2. Example 2 (Test Generation): wf02_test_generation_integration.robot exists in robot/ (integration tests) but not in robot/e2e/ (E2E tests with real LLM keys). The integration test uses mocks; the E2E test should use real LLM calls.

  3. Example 3 (Multi-File Refactoring): Tests invariant enforcement during multi-file changes. No E2E test.

  4. Example 6 (Documentation Generation): Tests codebase analysis and documentation writing. No E2E test.

  5. Example 9 (Session-Driven Interactive Exploration): Tests session lifecycle with interactive exploration. No E2E test.

  6. Example 10 (Full-Auto Batch Operations): Tests full-auto profile with batch formatting/linting. No E2E test.

  7. Example 11 (Complex Graph Actor): Tests multi-stage code review with graph actor. No E2E test.

  8. Example 13 (Custom Automation Profile): Tests semantic escalation and custom profile creation. No E2E test.

  9. Example 15 (Disaster Recovery): Tests plan rollback after a failed apply. No E2E test. This is particularly important as it tests the safety/recovery path.

Definition of Done

Create E2E test files for each missing workflow example:

  • robot/e2e/wf01_hello_world.robot — Example 1: Hello World
  • robot/e2e/wf02_test_generation.robot — Example 2: Automated Test Generation (E2E version with real LLM)
  • robot/e2e/wf03_multi_file_refactoring.robot — Example 3: Multi-File Refactoring
  • robot/e2e/wf06_documentation.robot — Example 6: Documentation Generation
  • robot/e2e/wf09_session_exploration.robot — Example 9: Session-Driven Interactive Exploration
  • robot/e2e/wf10_full_auto_batch.robot — Example 10: Full-Auto Batch Operations
  • robot/e2e/wf11_graph_actor_review.robot — Example 11: Complex Graph Actor
  • robot/e2e/wf13_custom_profile.robot — Example 13: Custom Automation Profile
  • robot/e2e/wf15_disaster_recovery.robot — Example 15: Disaster Recovery Rollback

Each file should follow the pattern established by wf07_cicd.robot and wf14_server_mode.robot (the two well-implemented E2E tests).


Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: E2E Workflow Specification Tests — Missing Workflow Coverage **Milestone**: v3.6.0 (M7) — E2E workflow specification tests **Severity**: Priority/Backlog — test infrastructure gap, not blocking runtime ### What Was Tested Code-level analysis of `robot/e2e/` directory against the 18 workflow examples in `docs/specification.md` (Section: "Workflow Examples"). ### Expected Behavior (from spec) The v3.6.0 milestone scope explicitly includes "E2E workflow specification tests" (issue #679: "Translating 18 spec workflow examples"). The specification contains 18 complete workflow examples that should each have a corresponding E2E test in `robot/e2e/`. ### Actual Behavior Only 9 of the 18 workflow examples have E2E test files. The following 9 examples have **no E2E test at all**: | Spec Example | Description | E2E File | Status | |---|---|---|---| | Example 1 | Hello World — Fix a Single Bug | None | ❌ Missing | | Example 2 | Automated Test Generation for a Module | None (wf02 is in robot/, not e2e/) | ❌ Missing | | Example 3 | Multi-File Refactoring with Invariants | None | ❌ Missing | | Example 6 | Writing Technical Documentation from Codebase Analysis | None | ❌ Missing | | Example 9 | Session-Driven Interactive Exploration | None | ❌ Missing | | Example 10 | Full-Auto Batch Operations — Formatting and Linting | None | ❌ Missing | | Example 11 | Complex Graph Actor for Multi-Stage Code Review | None | ❌ Missing | | Example 13 | Custom Automation Profile with Semantic Escalation | None | ❌ Missing | | Example 15 | Disaster Recovery — Rollback a Failed Apply | None | ❌ Missing | The 9 examples that DO have E2E test files (though many have empty/stub bodies — see separate bugs): - Example 4 → wf04_multi_project.robot (empty body) - Example 5 → wf05_db_migration.robot (empty body) - Example 7 → wf07_cicd.robot ✓ (has real tests) - Example 12 → wf12_hierarchical.robot (TDD placeholder) - Example 14 → wf14_server_mode.robot ✓ (has real tests) - Example 16 → wf16_devcontainer.robot (empty body) - Example 17 → wf17_explicit_container.robot (all Skip) - Example 18 → wf18_container_clone.robot (empty body) ### Code Location - `robot/e2e/` directory — missing wf01, wf02 (e2e version), wf03, wf06, wf09, wf10, wf11, wf13, wf15 files ### Impact 9 complete workflow scenarios from the specification have zero E2E test coverage: 1. **Example 1 (Hello World)**: The most basic workflow — single bug fix with manual profile. This is the entry-level scenario that all new users follow. No E2E test. 2. **Example 2 (Test Generation)**: `wf02_test_generation_integration.robot` exists in `robot/` (integration tests) but not in `robot/e2e/` (E2E tests with real LLM keys). The integration test uses mocks; the E2E test should use real LLM calls. 3. **Example 3 (Multi-File Refactoring)**: Tests invariant enforcement during multi-file changes. No E2E test. 4. **Example 6 (Documentation Generation)**: Tests codebase analysis and documentation writing. No E2E test. 5. **Example 9 (Session-Driven Interactive Exploration)**: Tests session lifecycle with interactive exploration. No E2E test. 6. **Example 10 (Full-Auto Batch Operations)**: Tests full-auto profile with batch formatting/linting. No E2E test. 7. **Example 11 (Complex Graph Actor)**: Tests multi-stage code review with graph actor. No E2E test. 8. **Example 13 (Custom Automation Profile)**: Tests semantic escalation and custom profile creation. No E2E test. 9. **Example 15 (Disaster Recovery)**: Tests `plan rollback` after a failed apply. No E2E test. This is particularly important as it tests the safety/recovery path. ### Definition of Done Create E2E test files for each missing workflow example: - `robot/e2e/wf01_hello_world.robot` — Example 1: Hello World - `robot/e2e/wf02_test_generation.robot` — Example 2: Automated Test Generation (E2E version with real LLM) - `robot/e2e/wf03_multi_file_refactoring.robot` — Example 3: Multi-File Refactoring - `robot/e2e/wf06_documentation.robot` — Example 6: Documentation Generation - `robot/e2e/wf09_session_exploration.robot` — Example 9: Session-Driven Interactive Exploration - `robot/e2e/wf10_full_auto_batch.robot` — Example 10: Full-Auto Batch Operations - `robot/e2e/wf11_graph_actor_review.robot` — Example 11: Complex Graph Actor - `robot/e2e/wf13_custom_profile.robot` — Example 13: Custom Automation Profile - `robot/e2e/wf15_disaster_recovery.robot` — Example 15: Disaster Recovery Rollback Each file should follow the pattern established by `wf07_cicd.robot` and `wf14_server_mode.robot` (the two well-implemented E2E tests). --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester
Author
Owner

🏷️ Label compliance fix applied by backlog groomer (cycle 64)

Added missing labels: State/Verified, Type/Bug, Priority/Critical

This issue was missing the State/ and Type/ labels. Labels have been applied based on issue content (UAT-identified 9 of 18 spec workflow examples with zero E2E test coverage).


Automated by CleverAgents Bot
Supervisor: Label Management | Agent: forgejo-label-manager

🏷️ **Label compliance fix applied by backlog groomer (cycle 64)** Added missing labels: `State/Verified`, `Type/Bug`, `Priority/Critical` This issue was missing the `State/` and `Type/` labels. Labels have been applied based on issue content (UAT-identified 9 of 18 spec workflow examples with zero E2E test coverage). --- **Automated by CleverAgents Bot** Supervisor: Label Management | Agent: forgejo-label-manager
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleveragents/cleveragents-core#6008
No description provided.