cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

test(e2e): E2E acceptance criteria for M3 (v3.2.0) — decisions, validations, and invariants #799

Closed
freemo wants to merge 8 commits from test/e2e-m3-acceptance into master
pull from: test/e2e-m3-acceptance
merge into: cleveragents:master
Conversation 11 Commits 8 Files Changed 5
+303 -15

8 Commits

Author SHA1 Message Date
hamza.khyari a4d543428c fix(test): restore tdd_expected_fail — bug #1022 fix not yet on master
CI / e2e_tests (pull_request) Failing after 5s
Details
CI / build (pull_request) Successful in 15s
Details
CI / helm (pull_request) Successful in 21s
Details
CI / lint (pull_request) Successful in 3m30s
Details
CI / quality (pull_request) Successful in 3m56s
Details
CI / typecheck (pull_request) Successful in 4m10s
Details
CI / security (pull_request) Successful in 4m22s
Details
CI / unit_tests (pull_request) Successful in 9m31s
Details
CI / docker (pull_request) Successful in 11s
Details
CI / integration_tests (pull_request) Successful in 22m2s
Details
CI / coverage (pull_request) Successful in 12m41s
Details
CI / status-check (pull_request) Failing after 1s
Details
CI / benchmark-publish (pull_request) Has been skipped
Details
CI / benchmark-regression (pull_request) Successful in 55m9s
Details 
Issue #1022 was closed but the actual invariant persistence fix has
not been merged to master.  invariant list still returns 'no
invariants found' between CLI invocations.  Restore tdd_expected_fail
with correct tdd_issue tags until the fix lands.

Ref: #743
 2026-04-02 06:21:01 +00:00
hamza.khyari e33558f2e5 fix(test): remove tdd_expected_fail, harden correction and status assertions 
P1-A: Remove tdd_expected_fail, tdd_bug, tdd_bug_1022 tags — bug
#1022 is closed, the test should pass on its own now.

P1-C: After plan correct, verify the decision tree contains
'superseded: true' proving the targeted decision was actually
corrected and marked superseded.

P1-D: Replace weak keyword-based final status check with structured
JSON assertion using plan status --format json.  Verify phase is
'apply' and processing_state is one of queued/processing/complete/applied.

P1-B (decision_id extraction): validated as not-a-bug — plan tree
--format json does not include plan_id, only decision_id fields.
First ULID match is always a decision_id.

Ref: #743
 2026-04-02 06:21:01 +00:00
hamza.khyari f15e0182af test(e2e): harden M3 acceptance assertions for snapshots and invariants 
- AC-2: assert context_snapshot.hot_context_hash via plan explain
  --format json --show-context --show-reasoning
- AC-5: add hard assertion that plan tree --format json contains an
  invariant_enforced decision for the added invariant

The AC-5 assertion is intentionally hard while the test remains
@tdd_expected_fail. This encodes the intended CLI behavior without
relying on non-deterministic LLM wording.

Ref: #743
 2026-04-02 06:21:01 +00:00
hamza.khyari 866186e254 fix(test): address CoreRasurae review findings for M3 E2E acceptance test 
C1 (Critical): Fix plan explain and plan correct to use Decision ID
extracted from plan tree --format json instead of Plan ID.  The spec
requires a decision_id, not a plan_id.

C2 (Critical): Dynamic actor selection based on available API key.
Uses anthropic/claude-sonnet-4 when ANTHROPIC_API_KEY is set,
falls back to openai/gpt-4.  Prevents misleading auth failures in
Anthropic-only environments.

H1: Add rc assertions on git add/commit after auth module creation.
H3: Add alternative/option to plan explain keyword check.
H4: Replace triple-quote Should Be True with $variable syntax across
all keyword assertions to prevent injection from LLM output.
H5: Tighten keyword assertions — use decision_id structural check
in tree JSON, specific terminal indicators in final status.
M1: Remove generic 'phase' from final status, use applied/complete/apply.
M5: Replace deprecated Return From Keyword If with IF/RETURN in
Extract Plan Id keyword (common_e2e.resource).

Ref: #743
 2026-04-02 06:21:01 +00:00
hamza.khyari 9784288e36 fix(test): add context snapshot and invariant enforcement assertions for M3 AC 
AC-2: Add plan tree --format json step (8b) to verify decisions are
recorded with context snapshots.  Checks for decision_id keys and
context_snapshot/snapshot/context presence in JSON output.

AC-5: Add invariant enforcement check (7b) after strategize phase.
Checks whether the invariant text or 'invariant' keyword appears in
strategize output.  Diagnostic-only (not hard assertion) because LLM
output is non-deterministic.

Ref: #743
 2026-04-02 06:21:01 +00:00
hamza.khyari ed9e9b1485 test(e2e): tag M3 acceptance test as tdd_expected_fail (#1022) 
InvariantService uses in-memory storage only — invariants added via
`invariant add` are lost when the CLI process exits. The M3 E2E test
correctly catches this as a persistence failure at the `invariant list`
step. Invariant persistence is a v3.4.0 deliverable (not M3 scope).

Tagged with tdd_bug, tdd_bug_1022, tdd_expected_fail per TDD bug-capture
workflow (CONTRIBUTING.md). The listener inverts the result so the test
passes CI while the bug is unfixed.

ISSUES CLOSED: #743
 2026-04-02 06:21:01 +00:00
hamza.khyari f3cf98440c fix(test): harden M3 E2E assertions and consolidate Extract Plan Id keyword 
Address review findings from PR #799:

- BUG-1: Extract Plan Id returns EMPTY instead of 'unknown-plan-id';
  callers now assert extraction succeeded with diagnostic messages
- TEST-1: Remove tautological len()>0 fallback from plan tree assertion;
  split into explicit emptiness check + semantic keyword validation
- TEST-2: Add content assertions on invariant add (verifies 'invariant')
  and invariant list (verifies persisted invariant text)
- TEST-3: Add 'Should Not Contain ... INTERNAL' checks on strategize,
  execute, correct, diff, and apply commands (matches M2 pattern)
- CODE-1: Consolidate Extract Plan Id into common_e2e.resource as a
  shared keyword with optional stderr arg; remove duplicate definitions
  from m1_acceptance.robot and m3_acceptance.robot; update m2 to use it
 2026-04-02 06:21:01 +00:00
freemo ecda78e1c8 test(e2e): E2E acceptance criteria for M3 (v3.2.0) — decisions, validations, and invariants 
Add robot/e2e/m3_acceptance.robot exercising the full M3 feature set:

  - Decision recording during plan execution
  - Decision tree visualization (plan tree)
  - Decision explanation (plan explain)
  - Invariant management (invariant add/list)
  - Decision correction (plan correct --mode=revert)

Zero mocking — all CLI invocations use real LLM API keys.
Follows existing E2E patterns (common_e2e.resource, Skip If No LLM Keys).

ISSUES CLOSED: #743
 2026-04-02 06:21:01 +00:00