cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

test(e2e): validate M3 acceptance criteria for v3.2.0 milestone closure #527

Closed
hurui200320 wants to merge 1 commits from test/m3-acceptance-gate into master
pull from: test/m3-acceptance-gate
merge into: cleveragents:master
Conversation 1 Commits 1 Files Changed 3
+54 -1
3 changed files with 54 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+10
View File
@@ -2,6 +2,16 @@
## Unreleased
- Validated M3 acceptance criteria for v3.2.0 milestone closure. All 10 E2E
verification tests pass against the final implementation: decision tree
recording and persistence, context snapshots, invariant enforcement during
strategize, dry-run and live revert corrections. Added acceptance criteria
tags and milestone documentation to the robot suite. (#494)
- Validated M4 acceptance criteria for v3.3.0 milestone closure. All 7 E2E
verification tests pass: subplan spawning, parallel execution with max_parallel,
plan tree viewing, three-way merge for non-conflicting changes, conflict
surfacing, and parent plan subplan status tracking. Added acceptance criteria
tags and milestone documentation. (#495)
- Added `SafetyProfile` domain model with configurable safety constraints (allowed skill
categories, sandbox/checkpoint requirements, human-approval flag, cost/retry limits) and
integrated it into the `Action` model via `from_config`/`as_cli_dict`. Persistence backed
CONTRIBUTORS.md
+1
View File
@@ -4,6 +4,7 @@
* Brent E. Edwards <brent.edwards@cleverthis.com>
* Hamza Khyari <hamza.khyari@cleverthis.com>
* Luis Mendes <luis.p.mendes@gmail.com>
* Rui Hu <rui.hu@cleverthis.com>
# Details
robot/m3_e2e_verification.robot
+43 -1
View File
@@ -1,13 +1,18 @@
*** Settings ***
Documentation End-to-end verification of M3 success criteria:
Documentation End-to-end verification of M3 (v3.2.0) acceptance criteria:
... decision tree recording, context snapshots,
... decision tree persistence and rendering,
... invariant enforcement during strategize,
... dry-run correction via impact analysis,
... and live revert correction re-execution.
...
... This suite is the final gate before closing milestone v3.2.0.
... All acceptance criteria from the v3.2.0 milestone description
... must pass before the milestone can be closed.
Resource ${CURDIR}/common.resource
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
Force Tags m3 acceptance_gate v3.2.0
*** Variables ***
${HELPER} ${CURDIR}/helper_m3_e2e_verification.py
@@ -18,6 +23,10 @@ Plan Execution Generates Decisions During Strategize
... service received the correct call, render plan
... status via CLI, and check decision tree structure
... with root (prompt_definition) and plan_id linkage.
...
... Validates: plan use + plan execute generate decisions
... during Strategize phase.
[Tags] success_criteria decision_recording
${result}= Run Process ${PYTHON} ${HELPER} plan-generates-decisions cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
@@ -29,6 +38,9 @@ Decision Tree View Via Plan Tree
... decision tree parent-child relationships, adjacency
... list, BFS traversal covering all nodes, and
... cli_dict rendering with all required keys.
...
... Validates: plan tree displays the decision tree correctly.
[Tags] success_criteria decision_tree
${result}= Run Process ${PYTHON} ${HELPER} decision-tree-view cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
@@ -41,6 +53,9 @@ Decision Explain Shows Full Context
... hot_context_ref, relevant_resources, and
... actor_state_ref. Also checks alternatives and
... rationale are populated.
...
... Validates: plan explain shows full decision context.
[Tags] success_criteria decision_explain
${result}= Run Process ${PYTHON} ${HELPER} decision-explain cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
@@ -52,6 +67,9 @@ Invariant Add And List Via CLI And Service
... InvariantService directly and the ``agents
... invariant add/list`` CLI commands. Verifies
... scope filtering and CLI mock integration.
...
... Validates: invariant add and invariant list CLI commands.
[Tags] success_criteria invariant_management
${result}= Run Process ${PYTHON} ${HELPER} invariant-add-list cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
@@ -63,6 +81,10 @@ Correction Dry Run Via Plan Correct
... analysis output. Tests both the CorrectionService
... directly and the ``agents plan correct --dry-run``
... CLI command with mocked services.
...
... Validates: plan correct with --dry-run performs
... impact analysis without modifying state.
[Tags] success_criteria correction
${result}= Run Process ${PYTHON} ${HELPER} correction-dry-run cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
@@ -75,6 +97,10 @@ Correction Live Revert Executes And Re-Creates Decisions
... is untouched, artifacts are archived, correction
... attempt is recorded, and a new corrected decision
... can be spawned at the reverted point.
...
... Validates: plan correct with --mode revert executes
... live correction.
[Tags] success_criteria correction
${result}= Run Process ${PYTHON} ${HELPER} correction-live-revert cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
@@ -85,6 +111,10 @@ Decisions Recorded With Full Context Snapshot
[Documentation] Verify every decision in the tree has a complete
... context snapshot and that model_dump / model_validate
... round-trips preserve all snapshot fields.
...
... Technical criterion: decisions recorded during
... Strategize with full context snapshot.
[Tags] technical_criteria context_snapshot
${result}= Run Process ${PYTHON} ${HELPER} decisions-context-snapshot cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
@@ -97,6 +127,10 @@ Decision Tree Persists To Database And Renders
... tree structure, sequence numbers, and plan_id
... are preserved, and that plan status CLI renders
... correctly after the round-trip.
...
... Technical criterion: decision tree persists to
... database and renders correctly.
[Tags] technical_criteria persistence
${result}= Run Process ${PYTHON} ${HELPER} decision-tree-persistence cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
@@ -108,6 +142,10 @@ Correction Revert Re-Executes From Decision Point
... target subtree (child + grandchild) and allows
... new decisions to be spawned from the corrected
... point, forming a new valid subtree.
...
... Technical criterion: correction in revert mode
... re-executes from decision point.
[Tags] technical_criteria correction_reexecution
${result}= Run Process ${PYTHON} ${HELPER} correction-revert-reexecutes cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
@@ -120,6 +158,10 @@ Invariants Enforced During Strategize
... de-duplicated case-insensitively, and enforcement
... records are created with decision IDs. Also
... verifies InvariantSet.merge produces correct output.
...
... Technical criterion: invariants are enforced
... during strategize.
[Tags] technical_criteria invariant_enforcement
${result}= Run Process ${PYTHON} ${HELPER} invariants-enforced-strategize cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}