cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

test(e2e): workflow example 2 — automated test generation for a module (trusted profile) #795

Closed
freemo wants to merge 1 commits from test/e2e-wf02-test-generation into master
pull from: test/e2e-wf02-test-generation
merge into: cleveragents:master
Conversation 5 Commits 1 Files Changed 2
+546

1 Commits

Author SHA1 Message Date
freemo cd06fe58f6 test(e2e): workflow example 2 — automated test generation for a module (trusted profile)
CI / benchmark-publish (pull_request) Has been skipped
Details
CI / build (pull_request) Successful in 45s
Details
CI / lint (pull_request) Successful in 6m18s
Details
CI / quality (pull_request) Successful in 6m45s
Details
CI / security (pull_request) Successful in 6m49s
Details
CI / typecheck (pull_request) Successful in 6m50s
Details
CI / integration_tests (pull_request) Successful in 10m6s
Details
CI / unit_tests (pull_request) Successful in 10m15s
Details
CI / docker (pull_request) Successful in 1m8s
Details
CI / e2e_tests (pull_request) Failing after 13m36s
Details
CI / coverage (pull_request) Successful in 11m47s
Details
CI / status-check (pull_request) Successful in 1s
Details
CI / benchmark-regression (pull_request) Successful in 1h0m14s
Details 
E2E Robot Framework test for WF02: Automated Test Generation using the
trusted automation profile.  Exercises the full plan lifecycle (action
create, resource add, project create, validation add/attach, plan
use/execute/artifacts/diff/apply) against a temporary repository with
a low-coverage auth module.

Key design decisions:
- Registers coverage validation (pytest --cov=src/auth --cov-fail-under=80)
  and attaches it to the project, per spec WF02 Step 1.
- Verifies plan artifacts listing after execute (spec §36971).
- Verifies coverage improvement after apply — checks apply output for
  validation results, falls back to running pytest --cov directly.
- All LLM-dependent commands use expected_rc=None for flexible assertions.
- Plan ID extraction uses Safe Parse Json Field from common_e2e.resource
  (eliminates duplicated Extract Plan Id keyword).
- Invariant check uses regex matching any src/ path in diff output (not
  just src/auth.py), and fails hard on violation (not just warn).
- Post-apply verification detects both new test files and modifications
  to existing files; handles zero-artifact LLM responses gracefully.
- Commit Fixture Files checks git return codes consistently with
  common_e2e.resource patterns.
- Traceback/INTERNAL checks on all plan execute and apply outputs.
- Skip If No LLM Keys guard for graceful CI degradation.
- Actor selection based on available API key (Anthropic/OpenAI).
- CHANGELOG.md updated per CONTRIBUTING.md requirements.

ISSUES CLOSED: #748
 2026-03-27 09:59:47 +00:00