Epic: E2E Testing Suite for Acceptance Criteria and Workflow Examples #739
Labels
No labels
auto/needs-reevaluation
controller-managed
auto/blocked-by-deps
auto/ci-timeout
auto/claimed-implementer
auto/claimed-merge
auto/claimed-reviewer
auto/driver-down
auto/invariant-violation
auto/last-attempt-tier-0
auto/last-attempt-tier-1
auto/last-attempt-tier-2
auto/last-attempt-tier-min
Automation Tracking
auto/needs-conflict-resolution
auto/needs-implementer
auto/postmortem
auto/ready-to-merge
auto/restart-throttled
auto/revert
auto/sentinel
auto/stale-inactivity
auto/unstable
Blocked
Bounty
$100
Bounty
$1000
Bounty
$10000
Bounty
$20
Bounty
$2000
Bounty
$250
Bounty
$50
Bounty
$500
Bounty
$5000
Bounty
$750
MoSCoW
Could have
MoSCoW
Must have
MoSCoW
Should have
Needs Feedback
Points
1
Points
13
Points
2
Points
21
Points
3
Points
34
Points
5
Points
55
Points
8
Points
88
Priority
Backlog
Priority
CI Blocker
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Signed-off: Owner
Signed-off: Scrum Master
Signed-off: Tech Lead
Spike
State
Completed
State
Duplicate
State
In Progress
State
In Review
State
Paused
State
Unverified
State
Verified
State
Wont Do
Type
Automation
Type
Bug
Type
Discussion
Type
Documentation
Type
Epic
Type
Feature
Type
Legendary
Type
Refactor
Type
Support
Type
Task
Type
Testing
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Blocks
Depends on
#376 Legendary: Hardening, Testing & Security
cleveragents/cleveragents-core
#740 test(e2e): set up E2E test infrastructure — nox session, CI job, Robot Framework @E2E tag
cleveragents/cleveragents-core
#741 test(e2e): E2E acceptance criteria for M1 (v3.0.0) — minimal plan execution flow
cleveragents/cleveragents-core
#742 test(e2e): E2E acceptance criteria for M2 (v3.1.0) — actor compiler and LLM integration
cleveragents/cleveragents-core
#743 test(e2e): E2E acceptance criteria for M3 (v3.2.0) — decisions, validations, and invariants
cleveragents/cleveragents-core
#744 test(e2e): E2E acceptance criteria for M4 (v3.3.0) — corrections, subplans, and checkpoints
cleveragents/cleveragents-core
#745 test(e2e): E2E acceptance criteria for M5 (v3.4.0) — ACMS v1 and context scaling
cleveragents/cleveragents-core
#746 test(e2e): E2E acceptance criteria for M6 (v3.5.0) — autonomy hardening
cleveragents/cleveragents-core
#747 test(e2e): workflow example 1 — Hello World, fix a single bug (manual profile)
cleveragents/cleveragents-core
#748 test(e2e): workflow example 2 — automated test generation for a module (trusted profile)
cleveragents/cleveragents-core
#749 test(e2e): workflow example 3 — multi-file refactoring with invariants (cautious profile)
cleveragents/cleveragents-core
#750 test(e2e): workflow example 4 — multi-project dependency update (supervised profile)
cleveragents/cleveragents-core
#751 test(e2e): workflow example 5 — database schema migration with safety nets (review profile)
cleveragents/cleveragents-core
#752 test(e2e): workflow example 6 — documentation generation from codebase analysis (trusted profile)
cleveragents/cleveragents-core
#753 test(e2e): workflow example 7 — CI/CD integration, automated PR review and fix (ci profile)
cleveragents/cleveragents-core
#754 test(e2e): workflow example 8 — cloud infrastructure management (supervised profile)
cleveragents/cleveragents-core
#755 test(e2e): workflow example 9 — session-driven interactive exploration (review profile)
cleveragents/cleveragents-core
#756 test(e2e): workflow example 10 — full-auto batch formatting and linting (full-auto profile)
cleveragents/cleveragents-core
#757 test(e2e): workflow example 11 — complex graph actor for multi-stage code review (trusted profile)
cleveragents/cleveragents-core
#758 test(e2e): workflow example 12 — large-scale hierarchical feature implementation (supervised profile)
cleveragents/cleveragents-core
#759 test(e2e): workflow example 13 — custom automation profile with semantic escalation
cleveragents/cleveragents-core
#760 test(e2e): workflow example 14 — server mode team collaboration (supervised profile)
cleveragents/cleveragents-core
#761 test(e2e): workflow example 15 — disaster recovery, rollback a failed apply (trusted profile)
cleveragents/cleveragents-core
#762 test(e2e): workflow example 16 — devcontainer-driven development (supervised profile)
cleveragents/cleveragents-core
#764 test(e2e): workflow example 18 — container with remote repo clone (trusted profile)
cleveragents/cleveragents-core
#763 test(e2e): workflow example 17 — explicit container with directory mount (trusted profile)
cleveragents/cleveragents-core
#1601 TEST-INFRA: [ci-environment] Unable to clone repository due to TLS/SSL handshake failure
cleveragents/cleveragents-core
#1609 fix(tests): resolve 114 ambiguous Behave step definitions across features/steps/
cleveragents/cleveragents-core
#1681 TEST-INFRA: [missing-test-levels] Add Robot and ASV tests for application module
cleveragents/cleveragents-core
#1685 TEST-INFRA: [flaky-tests] Unable to connect to the Forgejo API
cleveragents/cleveragents-core
#1691 TEST-INFRA: [ci-execution-time] Unable to clone repository cleveragents/cleveragents-core
cleveragents/cleveragents-core
#1743 TEST-INFRA: [missing-test-levels] Add Behave, Robot, and ASV tests for the application module
cleveragents/cleveragents-core
#1791 UAT:
nox -e unit_tests crashes with AmbiguousStep — tui_thought_block_steps.py and tui_first_run_steps.py both define @then('the rendered text should contain "{text}"')
cleveragents/cleveragents-core
#1806 BUG: [test-architecture] AmbiguousStep errors — 553 duplicate Behave step patterns across features/steps/
cleveragents/cleveragents-core
#1847 TEST-INFRA: [test-architecture] Fix AmbiguousStep error in Behave steps
cleveragents/cleveragents-core
#1864 TEST-INFRA: [test-architecture] AmbiguousStep error in Behave step definitions
cleveragents/cleveragents-core
#1872 TEST-INFRA: [test-architecture] Split long feature files
cleveragents/cleveragents-core
#1882 TEST-INFRA: [coverage-gaps] Improve test coverage for src/cleveragents/application/services/uko_loader.py
cleveragents/cleveragents-core
#1902 TEST-INFRA: [test-architecture] Name step files after their associated feature
cleveragents/cleveragents-core
#1965 TEST-INFRA: [test-architecture] Establish a clear naming convention for helper files
cleveragents/cleveragents-core
#5965 fix(e2e): redesign e2e test chain in m5_acceptance.robot to eliminate cascading tdd_expected_fail dependencies
cleveragents/cleveragents-core
#6892 TEST-INFRA: [test-data-quality] Improve Test Data Variation in BDD Scenarios
cleveragents/cleveragents-core
#8969 test(e2e): implement nox e2e_tests session and CI job for E2E Robot Framework suite
cleveragents/cleveragents-core
#8970 test(e2e): implement E2E acceptance criteria test suites for v3.2.0 milestone
cleveragents/cleveragents-core
#8973 test(e2e): implement E2E workflow example test suites for specification examples 1–9
cleveragents/cleveragents-core
#8975 test(e2e): implement E2E workflow example test suites for specification examples 10–18
cleveragents/cleveragents-core
Reference
cleveragents/cleveragents-core#739
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Background
Covers all true end-to-end (E2E) tests for milestone acceptance criteria and specification workflow examples. Unlike integration tests, E2E tests use zero mocking, stubbing, or test doubles of any kind — they exercise the real CleverAgents CLI with real LLM API keys (Anthropic/OpenAI) against real subprocess invocations, exactly as a user would experience them. E2E tests run in their own dedicated nox session (
nox -s e2e_tests) and CI job, completely separate from the standard integration test suite.See Forgejo dependency links for child issues.
Expected Behavior
Each milestone has a dedicated E2E acceptance criteria test suite, and each of the 18 specification workflow examples has a dedicated E2E test. Tests are Robot Framework suites tagged with
@E2E. They validate real command sequences with real LLM responses, and output validation is flexible (checking major structural components without strict character-by-character comparison). All E2E tests are excluded fromnox -s integration_testsand run only vianox -s e2e_tests.Acceptance Criteria
nox -s e2e_testssession exists and runs only@E2E-tagged Robot Framework testsnox -s integration_testssessionDefinition of Done
This Epic is complete when all child issues (tracked via Forgejo dependency links) are closed and merged. All E2E milestone acceptance suites and workflow example E2E tests pass with real LLM API keys in the dedicated
nox -s e2e_testssession.🤖 Backlog Groomer (groomer-1): ⚠️ Stale In Progress — This issue has been in
State/In Progressfor 383 hours (~16 days) with no updates. Current state:State/In Progress.Is this blocked? Please update the status or add a comment explaining the current situation. Consider:
State/Verifiedif work is pausednox -e unit_testscrashes with AmbiguousStep —tui_thought_block_steps.pyandtui_first_run_steps.pyboth define@then('the rendered text should contain "{text}"')#1791nox -e unit_testscrashes with AmbiguousStep —tui_thought_block_steps.pyandtui_first_run_steps.pyboth define@then('the rendered text should contain "{text}"')Label compliance fix applied:
State/In Progresswith valid labelState/In progressState/In Progress(capital P) is an orphaned label. The correct label isState/In progress(lowercase p).Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: ca-backlog-groomer