forked from cleveragents/cleveragents-core
test(e2e): verify M3 success criteria — decision tree and correction
Robot Framework E2E test suite for M3 milestone verification covering: - Plan execution generating decisions during Strategize phase - Decision tree viewing with parent-child relationships and BFS traversal - Decision explanation with full context snapshot verification - Invariant add/list via CLI and InvariantService with scope filtering - Dry-run correction via CorrectionService with impact analysis - Live revert correction execution with decision re-creation - Context snapshot round-trip serialisation assertions - Decision tree persistence via model_dump/model_validate - Correction revert re-execution from decision point - Invariant enforcement during strategize with merge precedence ISSUES CLOSED: #404
This commit is contained in:
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,126 @@
|
||||
*** Settings ***
|
||||
Documentation End-to-end verification of M3 success criteria:
|
||||
... decision tree recording, context snapshots,
|
||||
... decision tree persistence and rendering,
|
||||
... invariant enforcement during strategize,
|
||||
... dry-run correction via impact analysis,
|
||||
... and live revert correction re-execution.
|
||||
Resource ${CURDIR}/common.resource
|
||||
Suite Setup Setup Test Environment
|
||||
Suite Teardown Cleanup Test Environment
|
||||
|
||||
*** Variables ***
|
||||
${HELPER} ${CURDIR}/helper_m3_e2e_verification.py
|
||||
|
||||
*** Test Cases ***
|
||||
Plan Execution Generates Decisions During Strategize
|
||||
[Documentation] Execute a plan that generates decisions during
|
||||
... Strategize phase. Verifies a decision tree is
|
||||
... built with root (prompt_definition), strategy
|
||||
... choices, and correct plan_id linkage.
|
||||
${result}= Run Process ${PYTHON} ${HELPER} plan-generates-decisions cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} m3-plan-generates-decisions-ok
|
||||
|
||||
Decision Tree View Via Plan Tree
|
||||
[Documentation] View the decision tree structure and verify
|
||||
... parent-child relationships, adjacency list,
|
||||
... BFS traversal covering all nodes, and cli_dict
|
||||
... rendering with all required keys.
|
||||
${result}= Run Process ${PYTHON} ${HELPER} decision-tree-view cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} m3-decision-tree-view-ok
|
||||
|
||||
Decision Explain Shows Full Context
|
||||
[Documentation] Explain a specific decision and verify the
|
||||
... context snapshot contains hot_context_hash,
|
||||
... hot_context_ref, relevant_resources, and
|
||||
... actor_state_ref. Also checks alternatives and
|
||||
... rationale are populated.
|
||||
${result}= Run Process ${PYTHON} ${HELPER} decision-explain cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} m3-decision-explain-ok
|
||||
|
||||
Invariant Add And List Via CLI And Service
|
||||
[Documentation] Add and list project invariants via both the
|
||||
... InvariantService directly and the ``agents
|
||||
... invariant add/list`` CLI commands. Verifies
|
||||
... scope filtering and CLI mock integration.
|
||||
${result}= Run Process ${PYTHON} ${HELPER} invariant-add-list cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} m3-invariant-add-list-ok
|
||||
|
||||
Correction Dry Run Via Plan Correct
|
||||
[Documentation] Perform a dry-run correction and verify impact
|
||||
... analysis output. Tests both the CorrectionService
|
||||
... directly and the ``agents plan correct --dry-run``
|
||||
... CLI command with mocked services.
|
||||
${result}= Run Process ${PYTHON} ${HELPER} correction-dry-run cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} m3-correction-dry-run-ok
|
||||
|
||||
Correction Live Revert Executes And Re-Creates Decisions
|
||||
[Documentation] Execute a live correction in revert mode.
|
||||
... Verifies affected decisions are reverted, root
|
||||
... is untouched, artifacts are archived, correction
|
||||
... attempt is recorded, and a new corrected decision
|
||||
... can be spawned at the reverted point.
|
||||
${result}= Run Process ${PYTHON} ${HELPER} correction-live-revert cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} m3-correction-live-revert-ok
|
||||
|
||||
Decisions Recorded With Full Context Snapshot
|
||||
[Documentation] Verify every decision in the tree has a complete
|
||||
... context snapshot and that model_dump / model_validate
|
||||
... round-trips preserve all snapshot fields.
|
||||
${result}= Run Process ${PYTHON} ${HELPER} decisions-context-snapshot cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} m3-decisions-context-snapshot-ok
|
||||
|
||||
Decision Tree Persists To Database And Renders
|
||||
[Documentation] Verify the decision tree survives a persistence
|
||||
... round-trip (model_dump -> model_validate) and
|
||||
... that tree structure, sequence numbers, plan_id
|
||||
... consistency, and cli_dict rendering are preserved.
|
||||
${result}= Run Process ${PYTHON} ${HELPER} decision-tree-persistence cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} m3-decision-tree-persistence-ok
|
||||
|
||||
Correction Revert Re-Executes From Decision Point
|
||||
[Documentation] Verify that a revert correction invalidates the
|
||||
... target subtree (child + grandchild) and allows
|
||||
... new decisions to be spawned from the corrected
|
||||
... point, forming a new valid subtree.
|
||||
${result}= Run Process ${PYTHON} ${HELPER} correction-revert-reexecutes cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} m3-correction-revert-reexecutes-ok
|
||||
|
||||
Invariants Enforced During Strategize
|
||||
[Documentation] Verify invariants at global, project, and plan
|
||||
... scopes are merged with correct precedence,
|
||||
... de-duplicated case-insensitively, and enforcement
|
||||
... records are created with decision IDs. Also
|
||||
... verifies InvariantSet.merge produces correct output.
|
||||
${result}= Run Process ${PYTHON} ${HELPER} invariants-enforced-strategize cwd=${WORKSPACE}
|
||||
Log ${result.stdout}
|
||||
Log ${result.stderr}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} m3-invariants-enforced-strategize-ok
|
||||
Reference in New Issue
Block a user