cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

test(e2e): validate M3 acceptance criteria for v3.2.0 milestone closure #559

Merged
hurui200320 merged 1 commits from test/m3-acceptance-gate into master 2026-03-05 15:09:20 +00:00
Conversation 13 Commits 1 Files Changed 7
+682 -748
7 changed files with 682 additions and 748 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+7
View File
@@ -2,6 +2,13 @@
## Unreleased
- Validated M3 acceptance criteria for v3.2.0 milestone closure. All 10 E2E
hurui200320 marked this conversation as resolved Outdated
CoreRasurae commented 2026-03-05 10:28:29 +00:00
Outdated
Review
Copy Link
Copy Source

[HIGH] F3 — Two separate CHANGELOG entries both tagged (#494). This appears to be a squash artifact from combining the original commit + review-feedback commit. A single squashed commit should have one consolidated entry.

Recommendation: Merge these into a single entry covering the final state of the work.

**[HIGH] F3** — Two separate CHANGELOG entries both tagged `(#494)`. This appears to be a squash artifact from combining the original commit + review-feedback commit. A single squashed commit should have one consolidated entry. **Recommendation:** Merge these into a single entry covering the final state of the work.
verification tests pass against the final implementation, exercising real
CLI command paths (``plan use``, ``plan execute``, ``plan tree``,
``plan explain``, project-scoped ``invariant add/list``, dry-run and live
``plan correct``), database-backed persistence, context snapshots, and
invariant enforcement during strategize. Added acceptance criteria tags and
milestone documentation to the robot suite. (#494)
- Added `builtin/plan-subplan` tool for strategy actors to emit `SUBPLAN_SPAWN` or
`SUBPLAN_PARALLEL_SPAWN` decisions. Validates payload via `SubplanPayload` (Pydantic),
applies defaults (merge strategy, max_parallel, dependencies), generates rationale text,
CONTRIBUTORS.md
+1
View File
@@ -4,6 +4,7 @@
* Brent E. Edwards <brent.edwards@cleverthis.com>
* Hamza Khyari <hamza.khyari@cleverthis.com>
* Luis Mendes <luis.p.mendes@gmail.com>
* Rui Hu <rui.hu@cleverthis.com>
# Details
robot/actor_context_management.robot
+3 -1
View File
@@ -89,7 +89,9 @@ Test Actor-Based Workflow
# Build plan
${result} = Run Process ${PYTHON} -m cleveragents build
... cwd=${project_dir} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=30s
# Normal duration: ~10-15s. Timeout raised from 30s to 120s for pabot
# cold-start (16 parallel processes) + Alembic migration overhead.
... cwd=${project_dir} env:CLEVERAGENTS_TESTING_USE_MOCK_AI=true timeout=120s
Should Be Equal As Integers ${result.rc} 0
# Apply changes
robot/changeset_persistence.robot
+3 -1
View File
@@ -9,7 +9,9 @@ Force Tags changeset persistence
*** Variables ***
${PYTHON} python
${TIMEOUT} 30s
# Normal duration: ~5-10s per test. Timeout raised from 30s to 120s for
CoreRasurae commented 2026-03-05 12:58:43 +00:00
Outdated
Review
Copy Link
Copy Source

[L2 - LOW] Commit message says "Added explicit Robot process timeouts (60s)" but this file (and actor_context_management.robot, decision_di_wiring_smoke.robot) increases the timeout to 120s, not 60s. The M3 suite itself uses 60s. Minor documentation inconsistency.

**[L2 - LOW]** Commit message says "Added explicit Robot process timeouts (60s)" but this file (and `actor_context_management.robot`, `decision_di_wiring_smoke.robot`) increases the timeout to **120s**, not 60s. The M3 suite itself uses 60s. Minor documentation inconsistency.
# pabot cold-start (16 parallel processes) + Alembic migration overhead.
${TIMEOUT} 120s
*** Keywords ***
Set Suite Variables
robot/decision_di_wiring_smoke.robot
+4 -2
View File
@@ -11,13 +11,15 @@ ${HELPER_SCRIPT} robot/helper_decision_di.py
Verify Decision DI Resolution
[Documentation] Verify DecisionService can be resolved from the DI container
[Tags] di decision smoke
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} resolve-service cwd=${WORKSPACE} timeout=30s
# Normal duration: ~5-10s. Timeout raised from 30s to 120s for pabot cold-start.
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} resolve-service cwd=${WORKSPACE} timeout=120s
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} resolve-service-ok
Verify Decision Recording Integration
[Documentation] Verify DecisionService records decisions during lifecycle transitions
[Tags] di decision lifecycle smoke
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} record-integration cwd=${WORKSPACE} timeout=30s
# Normal duration: ~5-10s. Timeout raised from 30s to 120s for pabot cold-start.
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} record-integration cwd=${WORKSPACE} timeout=120s
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} record-integration-ok
robot/helper_m3_e2e_verification.py
+593 -706
View File
File diff suppressed because it is too large Load Diff
robot/m3_e2e_verification.robot
+71 -38
View File
@@ -1,58 +1,71 @@
*** Settings ***
Documentation End-to-end verification of M3 success criteria:
Documentation End-to-end verification of M3 (v3.2.0) acceptance criteria:
... decision tree recording, context snapshots,
... decision tree persistence and rendering,
... invariant enforcement during strategize,
... dry-run correction via impact analysis,
... and live revert correction re-execution.
...
... This suite is the final gate before closing milestone v3.2.0.
... All acceptance criteria from the v3.2.0 milestone description
... must pass before the milestone can be closed.
Resource ${CURDIR}/common.resource
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
Force Tags m3 acceptance_gate v3.2.0
*** Variables ***
${HELPER} ${CURDIR}/helper_m3_e2e_verification.py
*** Test Cases ***
Plan Execution Generates Decisions During Strategize
[Documentation] Execute a plan via CLI, verify the lifecycle
... service received the correct call, render plan
... status via CLI, and check decision tree structure
... with root (prompt_definition) and plan_id linkage.
${result}= Run Process ${PYTHON} ${HELPER} plan-generates-decisions cwd=${WORKSPACE}
[Documentation] Execute ``agents plan use`` and ``agents plan execute``
... through the CLI command path, then verify decisions
... were recorded during Strategize before execution.
...
... Validates: plan use + plan execute generate decisions
... during Strategize phase.
[Tags] success_criteria decision_recording
${result}= Run Process ${PYTHON} ${HELPER} plan-generates-decisions cwd=${WORKSPACE} timeout=60s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} m3-plan-generates-decisions-ok
Decision Tree View Via Plan Tree
[Documentation] Exercise plan status CLI rendering, then verify
... decision tree parent-child relationships, adjacency
... list, BFS traversal covering all nodes, and
... cli_dict rendering with all required keys.
${result}= Run Process ${PYTHON} ${HELPER} decision-tree-view cwd=${WORKSPACE}
[Documentation] Invoke ``agents plan tree <plan_id>`` and validate
... nested JSON output for root/child/grandchild
... relationships from the rendered tree.
...
... Validates: plan tree displays the decision tree correctly.
[Tags] success_criteria decision_tree
${result}= Run Process ${PYTHON} ${HELPER} decision-tree-view cwd=${WORKSPACE} timeout=60s
CoreRasurae commented 2026-03-05 12:58:43 +00:00
Outdated
Review
Copy Link
Copy Source

[L3 - LOW] No test exercises the --show-superseded flag on plan tree. The spec and CLI (plan.py:2928-2930) support this flag for rendering both current and historical branches. The correction_revert_reexecutes helper creates superseded decisions but never renders the tree with this flag.

**[L3 - LOW]** No test exercises the `--show-superseded` flag on `plan tree`. The spec and CLI (`plan.py:2928-2930`) support this flag for rendering both current and historical branches. The `correction_revert_reexecutes` helper creates superseded decisions but never renders the tree with this flag.
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} m3-decision-tree-view-ok
Decision Explain Shows Full Context
[Documentation] Exercise plan status CLI rendering, then verify
... context snapshot contains hot_context_hash,
... hot_context_ref, relevant_resources, and
... actor_state_ref. Also checks alternatives and
... rationale are populated.
${result}= Run Process ${PYTHON} ${HELPER} decision-explain cwd=${WORKSPACE}
[Documentation] Invoke ``agents plan explain <decision_id>`` with
... ``--show-context`` and ``--show-reasoning`` and
... verify full decision context fields are present.
...
... Validates: plan explain shows full decision context.
[Tags] success_criteria decision_explain
${result}= Run Process ${PYTHON} ${HELPER} decision-explain cwd=${WORKSPACE} timeout=60s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} m3-decision-explain-ok
Invariant Add And List Via CLI And Service
[Documentation] Add and list project invariants via both the
... InvariantService directly and the ``agents
... invariant add/list`` CLI commands. Verifies
... scope filtering and CLI mock integration.
${result}= Run Process ${PYTHON} ${HELPER} invariant-add-list cwd=${WORKSPACE}
[Documentation] Validate project-scoped invariant commands:
... ``agents invariant add --project local/large-project``
... and ``agents invariant list --project local/large-project``.
...
... Validates: invariant add and invariant list CLI commands.
[Tags] success_criteria invariant_management
${result}= Run Process ${PYTHON} ${HELPER} invariant-add-list cwd=${WORKSPACE} timeout=60s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
@@ -63,19 +76,25 @@ Correction Dry Run Via Plan Correct
... analysis output. Tests both the CorrectionService
... directly and the ``agents plan correct --dry-run``
... CLI command with mocked services.
${result}= Run Process ${PYTHON} ${HELPER} correction-dry-run cwd=${WORKSPACE}
...
... Validates: plan correct with --dry-run performs
... impact analysis without modifying state.
[Tags] success_criteria correction_dry_run
${result}= Run Process ${PYTHON} ${HELPER} correction-dry-run cwd=${WORKSPACE} timeout=60s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} m3-correction-dry-run-ok
Correction Live Revert Executes And Re-Creates Decisions
[Documentation] Execute a live correction in revert mode.
... Verifies affected decisions are reverted, root
... is untouched, artifacts are archived, correction
... attempt is recorded, and a new corrected decision
... can be spawned at the reverted point.
${result}= Run Process ${PYTHON} ${HELPER} correction-live-revert cwd=${WORKSPACE}
[Documentation] Execute ``agents plan correct --mode revert``
... through the CLI path (with ``--yes``) and verify
... reverted decisions and applied status in output.
...
... Validates: plan correct with --mode revert executes
... live correction.
[Tags] success_criteria correction_live_revert
${result}= Run Process ${PYTHON} ${HELPER} correction-live-revert cwd=${WORKSPACE} timeout=60s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
@@ -85,19 +104,25 @@ Decisions Recorded With Full Context Snapshot
[Documentation] Verify every decision in the tree has a complete
... context snapshot and that model_dump / model_validate
... round-trips preserve all snapshot fields.
${result}= Run Process ${PYTHON} ${HELPER} decisions-context-snapshot cwd=${WORKSPACE}
...
... Technical criterion: decisions recorded during
... Strategize with full context snapshot.
[Tags] technical_criteria context_snapshot
${result}= Run Process ${PYTHON} ${HELPER} decisions-context-snapshot cwd=${WORKSPACE} timeout=60s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} m3-decisions-context-snapshot-ok
Decision Tree Persists To Database And Renders
[Documentation] Verify the decision tree survives a persistence
... round-trip (model_dump -> model_validate), that
... tree structure, sequence numbers, and plan_id
... are preserved, and that plan status CLI renders
... correctly after the round-trip.
${result}= Run Process ${PYTHON} ${HELPER} decision-tree-persistence cwd=${WORKSPACE}
[Documentation] Persist decisions through ``DecisionService`` with
... ``UnitOfWork`` backing, then verify round-trip
... retrieval and ``agents plan tree`` rendering.
...
... Technical criterion: decision tree persists to
... database and renders correctly.
[Tags] technical_criteria persistence
${result}= Run Process ${PYTHON} ${HELPER} decision-tree-persistence cwd=${WORKSPACE} timeout=60s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
@@ -108,7 +133,11 @@ Correction Revert Re-Executes From Decision Point
... target subtree (child + grandchild) and allows
... new decisions to be spawned from the corrected
... point, forming a new valid subtree.
${result}= Run Process ${PYTHON} ${HELPER} correction-revert-reexecutes cwd=${WORKSPACE}
...
... Technical criterion: correction in revert mode
... re-executes from decision point.
[Tags] technical_criteria correction_reexecution
${result}= Run Process ${PYTHON} ${HELPER} correction-revert-reexecutes cwd=${WORKSPACE} timeout=60s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
@@ -120,7 +149,11 @@ Invariants Enforced During Strategize
... de-duplicated case-insensitively, and enforcement
... records are created with decision IDs. Also
... verifies InvariantSet.merge produces correct output.
${result}= Run Process ${PYTHON} ${HELPER} invariants-enforced-strategize cwd=${WORKSPACE}
...
... Technical criterion: invariants are enforced
... during strategize.
[Tags] technical_criteria invariant_enforcement
${result}= Run Process ${PYTHON} ${HELPER} invariants-enforced-strategize cwd=${WORKSPACE} timeout=60s
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0