cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

test(e2e): E2E acceptance criteria for M1 (v3.0.0) — minimal plan execution flow #789

Closed
freemo wants to merge 1 commits from test/e2e-m1-acceptance into master
pull from: test/e2e-m1-acceptance
merge into: cleveragents:master
Conversation 2 Commits 1 Files Changed 2
+139

1 Commits

Author SHA1 Message Date
freemo 0d7a06efbb test(e2e): E2E acceptance criteria for M1 (v3.0.0) — minimal plan execution flow
CI / benchmark-publish (pull_request) Has been skipped
Details
CI / lint (pull_request) Successful in 15s
Details
CI / build (pull_request) Successful in 17s
Details
CI / quality (pull_request) Successful in 18s
Details
CI / e2e_tests (pull_request) Failing after 25s
Details
CI / security (pull_request) Successful in 33s
Details
CI / typecheck (pull_request) Successful in 36s
Details
CI / integration_tests (pull_request) Successful in 2m53s
Details
CI / unit_tests (pull_request) Successful in 3m52s
Details
CI / docker (pull_request) Successful in 54s
Details
CI / coverage (pull_request) Successful in 6m53s
Details
CI / benchmark-regression (pull_request) Successful in 34m3s
Details 
Added Robot Framework E2E test suite for M1 milestone acceptance criteria.
Tests the complete plan lifecycle (action create → resource add → project
create → plan use → plan execute strategize → plan execute → plan diff →
plan apply) with real LLM API keys and no mocking.

Key implementation details:
- Uses openai/gpt-4o-mini as strategy/execution actor (cost-effective)
- Simple definition_of_done: "Create a file called HELLO.md"
- Creates isolated temp git repo via Create Temp Git Repo keyword
- Extracts plan ID via ULID regex from plain-text output
- Uses expected_rc=None for LLM-dependent steps (execute, diff, apply)
  to handle non-deterministic LLM behavior gracefully
- Flexible structural assertions: checks rc, output presence, git log
- Skips gracefully when no LLM API keys (ANTHROPIC/OPENAI) are set
- Tagged [E2E] so it runs only in nox -s e2e_tests session

ISSUES CLOSED: #741
 2026-03-13 23:10:53 +00:00