From ecda78e1c88078280a9d797dd5d4dd31c207cc07 Mon Sep 17 00:00:00 2001
From: Jeffrey Phillips Freeman <the@jeffreyfreeman.me>
Date: Thu, 12 Mar 2026 23:34:45 +0000
Subject: [PATCH 1/8] =?UTF-8?q?test(e2e):=20E2E=20acceptance=20criteria=20?=
 =?UTF-8?q?for=20M3=20(v3.2.0)=20=E2=80=94=20decisions,=20validations,=20a?=
 =?UTF-8?q?nd=20invariants?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Add robot/e2e/m3_acceptance.robot exercising the full M3 feature set:

  - Decision recording during plan execution
  - Decision tree visualization (plan tree)
  - Decision explanation (plan explain)
  - Invariant management (invariant add/list)
  - Decision correction (plan correct --mode=revert)

Zero mocking — all CLI invocations use real LLM API keys.
Follows existing E2E patterns (common_e2e.resource, Skip If No LLM Keys).

ISSUES CLOSED: #743
---
 CHANGELOG.md                  |   5 +
 robot/e2e/m3_acceptance.robot | 199 ++++++++++++++++++++++++++++++++++
 2 files changed, 204 insertions(+)
 create mode 100644 robot/e2e/m3_acceptance.robot

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 36aab2df1..c5d88c90f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1021,6 +1021,11 @@ attribute 'db'` after `agents init`. Root cause: `_get_session_service()`
   when API keys are absent, and `--exclude E2E` on the standard integration
   test session. Includes a minimal smoke test exercising `agents --version`
   and `agents --help`. (#740)
+- Added E2E acceptance test for M3 (v3.2.0) milestone: decisions, validations,
+  and invariants. Tests decision recording, decision tree visualization
+  (plan tree), decision explanation (plan explain), invariant management
+  (invariant add/list), and decision correction (plan correct --mode=revert)
+  with real LLM API keys and zero mocking. (#743)
 - Implemented `tdd_expected_fail` tag handling in Robot Framework via a Listener v3
   module (`robot/tdd_expected_fail_listener.py`). Tests tagged `tdd_expected_fail`
   that fail have their result inverted to pass (expected failure); tests that
diff --git a/robot/e2e/m3_acceptance.robot b/robot/e2e/m3_acceptance.robot
new file mode 100644
index 000000000..87fcbb83c
--- /dev/null
+++ b/robot/e2e/m3_acceptance.robot
@@ -0,0 +1,199 @@
+*** Settings ***
+Documentation     E2E acceptance test for M3 (v3.2.0): Decisions, Validations, and Invariants.
+...
+...               Exercises decision recording during plan execution, decision tree
+...               visualization (plan tree), decision explanation (plan explain),
+...               invariant management (invariant add/list), and decision correction
+...               (plan correct --mode=revert) with real LLM API keys.
+...
+...               Zero mocking — all CLI invocations use real providers.
+Resource          common_e2e.resource
+Suite Setup       E2E Suite Setup
+Suite Teardown    E2E Suite Teardown
+
+*** Variables ***
+${ACTION_NAME}        local/m3-e2e-action
+${RESOURCE_NAME}      local/m3-e2e-repo
+${PROJECT_NAME}       local/m3-e2e-project
+
+*** Test Cases ***
+M3 Decision Recording And Tree Visualization
+    [Documentation]    End-to-end test for M3 milestone: decision recording, tree
+    ...                visualization, explanation, invariant management, and correction.
+    ...
+    ...                Exercises the full M3 feature set through the CLI with real
+    ...                LLM API keys. Validates structural output, not exact text.
+    [Tags]    E2E
+
+    # ── Initialize database ──
+    ${r_init}=    Run CleverAgents Command
+    ...    init    --yes    --force
+    Should Not Contain    ${r_init.stdout}${r_init.stderr}    Traceback
+
+    # ── 1. Create temp git repo with sample project files ──
+    ${repo_dir}=    Create Temp Git Repo    m3-acceptance-repo
+    Create Directory    ${repo_dir}${/}src
+    ${auth_code}=    Catenate    SEPARATOR=\n
+    ...    """Authentication module with raw SQL queries."""
+    ...    import sqlite3
+    ...    ${EMPTY}
+    ...    ${EMPTY}
+    ...    def authenticate(username: str, password: str) -> bool:
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}"""Authenticate user against database."""
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}conn = sqlite3.connect("users.db")
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}cursor = conn.cursor()
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}cursor.execute(
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}"SELECT * FROM users WHERE name=? AND pass=?",
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}(username, password),
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE})
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}result = cursor.fetchone()
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}conn.close()
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}return result is not None
+    ...    ${EMPTY}
+    ...    ${EMPTY}
+    ...    def get_user(user_id: int) -> dict:
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}"""Fetch user by ID."""
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}conn = sqlite3.connect("users.db")
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}cursor = conn.cursor()
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}cursor.execute("SELECT * FROM users WHERE id=?", (user_id,))
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}row = cursor.fetchone()
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}conn.close()
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}return {"id": row[0], "name": row[1]} if row else {}
+    Create File    ${repo_dir}${/}src${/}auth.py    ${auth_code}
+    Run Process    git    add    .    cwd=${repo_dir}
+    Run Process    git    commit    -m    Add auth module    cwd=${repo_dir}
+    ${branch_result}=    Run Process    git    rev-parse    --abbrev-ref    HEAD    cwd=${repo_dir}
+    ${branch}=    Strip String    ${branch_result.stdout}
+
+    # ── 2. Register resource and create project ──
+    ${r_resource}=    Run CleverAgents Command
+    ...    resource    add    git-checkout    ${RESOURCE_NAME}
+    ...    --path    ${repo_dir}    --branch    ${branch}
+    Should Not Contain    ${r_resource.stdout}${r_resource.stderr}    Traceback
+    Output Should Contain    ${r_resource}    ${RESOURCE_NAME}
+
+    ${r_project}=    Run CleverAgents Command
+    ...    project    create    ${PROJECT_NAME}
+    ...    --description    M3 E2E acceptance project
+    ...    --resource    ${RESOURCE_NAME}
+    Should Not Contain    ${r_project.stdout}${r_project.stderr}    Traceback
+    Output Should Contain    ${r_project}    ${PROJECT_NAME}
+
+    # ── 3. Add invariants at project scope ──
+    ${inv_result}=    Run CleverAgents Command
+    ...    invariant    add    --project    ${PROJECT_NAME}
+    ...    API function signatures must not change
+    Log    Invariant add stdout: ${inv_result.stdout}
+    Log    Invariant add stderr: ${inv_result.stderr}
+    ${inv_output}=    Set Variable    ${inv_result.stdout}\n${inv_result.stderr}
+    Should Not Contain    ${inv_output}    Traceback
+
+    # ── 4. List invariants to verify persistence ──
+    ${inv_list}=    Run CleverAgents Command
+    ...    invariant    list    --project    ${PROJECT_NAME}
+    Log    Invariant list stdout: ${inv_list.stdout}
+    Log    Invariant list stderr: ${inv_list.stderr}
+    ${inv_list_output}=    Set Variable    ${inv_list.stdout}\n${inv_list.stderr}
+    Should Not Contain    ${inv_list_output}    Traceback
+
+    # ── 5. Create action YAML for refactoring task ──
+    ${action_yaml}=    Catenate    SEPARATOR=\n
+    ...    name: ${ACTION_NAME}
+    ...    description: Refactor auth module from raw SQL to ORM
+    ...    definition_of_done: Replace raw SQL queries with ORM while preserving API
+    ...    strategy_actor: openai/gpt-4
+    ...    execution_actor: openai/gpt-4
+    ${action_yaml_path}=    Set Variable    ${SUITE_HOME}${/}m3_action.yaml
+    Create File    ${action_yaml_path}    ${action_yaml}
+    ${r_action}=    Run CleverAgents Command
+    ...    action    create    --config    ${action_yaml_path}
+    Should Not Contain    ${r_action.stdout}${r_action.stderr}    Traceback
+    Output Should Contain    ${r_action}    ${ACTION_NAME}
+
+    # ── 6. Plan use — create plan ──
+    ${r_use}=    Run CleverAgents Command
+    ...    plan    use    ${ACTION_NAME}    ${PROJECT_NAME}    --format    plain
+    Should Not Be Empty    ${r_use.stdout}
+    ${plan_id}=    Extract Plan Id    ${r_use.stdout}    ${r_use.stderr}
+    Log    Extracted plan_id: ${plan_id}
+
+    # ── 7. Plan execute — strategize phase (decisions recorded) ──
+    ${r_strategize}=    Run CleverAgents Command
+    ...    plan    execute    ${plan_id}    --format    plain
+    ...    timeout=300s
+    Should Not Contain    ${r_strategize.stdout}${r_strategize.stderr}    Traceback
+    Log    Strategize output: ${r_strategize.stdout}
+
+    # ── 8. Plan tree — view decision tree ──
+    ${r_tree}=    Run CleverAgents Command
+    ...    plan    tree    ${plan_id}    --format    plain
+    Log    Plan tree stdout: ${r_tree.stdout}
+    Log    Plan tree stderr: ${r_tree.stderr}
+    ${tree_output}=    Set Variable    ${r_tree.stdout}\n${r_tree.stderr}
+    Should Not Contain    ${tree_output}    Traceback
+    # Tree output should reference the plan or contain tree-like structure
+    Should Be True    'plan' in '''${tree_output}'''.lower() or 'decision' in '''${tree_output}'''.lower() or 'tree' in '''${tree_output}'''.lower() or len('''${r_tree.stdout}''') > 0
+
+    # ── 9. Plan explain — inspect a decision (use plan_id as fallback) ──
+    ${r_explain}=    Run CleverAgents Command
+    ...    plan    explain    ${plan_id}    --format    plain
+    Log    Plan explain stdout: ${r_explain.stdout}
+    Log    Plan explain stderr: ${r_explain.stderr}
+    ${explain_output}=    Set Variable    ${r_explain.stdout}\n${r_explain.stderr}
+    Should Not Contain    ${explain_output}    Traceback
+
+    # ── 10. Plan execute — advance to execute phase ──
+    ${r_execute}=    Run CleverAgents Command
+    ...    plan    execute    ${plan_id}    --format    plain
+    ...    timeout=300s
+    Should Not Contain    ${r_execute.stdout}${r_execute.stderr}    Traceback
+    Log    Execute phase output: ${r_execute.stdout}
+
+    # ── 11. Plan correct — revert mode with guidance ──
+    ${r_correct}=    Run CleverAgents Command
+    ...    plan    correct    ${plan_id}    --mode    revert
+    ...    --guidance    Use SQLAlchemy ORM instead of raw SQL
+    ...    --yes    --format    plain
+    ...    timeout=300s
+    Log    Plan correct stdout: ${r_correct.stdout}
+    Log    Plan correct stderr: ${r_correct.stderr}
+    ${correct_output}=    Set Variable    ${r_correct.stdout}\n${r_correct.stderr}
+    Should Not Contain    ${correct_output}    Traceback
+
+    # ── 12. Plan diff — show changes ──
+    ${r_diff}=    Run CleverAgents Command
+    ...    plan    diff    ${plan_id}    --format    plain
+    Log    Plan diff stdout: ${r_diff.stdout}
+    Log    Plan diff stderr: ${r_diff.stderr}
+    Should Not Contain    ${r_diff.stdout}${r_diff.stderr}    Traceback
+
+    # ── 13. Plan apply ──
+    ${r_apply}=    Run CleverAgents Command
+    ...    plan    lifecycle-apply    ${plan_id}    --format    plain
+    ...    timeout=300s
+    Log    Plan apply stdout: ${r_apply.stdout}
+    Log    Plan apply stderr: ${r_apply.stderr}
+    Should Not Contain    ${r_apply.stdout}${r_apply.stderr}    Traceback
+
+    # ── 14. Final plan status ──
+    ${r_status}=    Run CleverAgents Command
+    ...    plan    status    ${plan_id}    --format    plain
+    Should Not Be Empty    ${r_status.stdout}
+    Log    Final plan status: ${r_status.stdout}
+    Log    M3 acceptance E2E test completed successfully
+
+*** Keywords ***
+Extract Plan Id
+    [Documentation]    Extract a plan ID (ULID/UUID) from combined CLI output.
+    [Arguments]    ${stdout}    ${stderr}
+    ${combined}=    Set Variable    ${stdout}\n${stderr}
+    ${status}    ${match}=    Run Keyword And Ignore Error
+    ...    Evaluate    __import__('re').search(r'[0-9A-Z]{26}', '''${combined}''').group()
+    Return From Keyword If    '${status}' == 'PASS'    ${match}
+    ${status}    ${match}=    Run Keyword And Ignore Error
+    ...    Evaluate    __import__('re').search(r'[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', '''${combined}''').group()
+    Return From Keyword If    '${status}' == 'PASS'    ${match}
+    ${status}    ${match}=    Run Keyword And Ignore Error
+    ...    Evaluate    __import__('re').search(r'plan_id[:\\s]+([\\w-]+)', '''${combined}''').group(1)
+    Return From Keyword If    '${status}' == 'PASS'    ${match}
+    RETURN    unknown-plan-id
-- 
2.52.0


From f3cf98440cf134744e8592f07e7775ab320db05f Mon Sep 17 00:00:00 2001
From: Hamza Khyari <hamza.khyari@cleverthis.com>
Date: Tue, 17 Mar 2026 04:52:35 +0100
Subject: [PATCH 2/8] fix(test): harden M3 E2E assertions and consolidate
 Extract Plan Id keyword

Address review findings from PR #799:

- BUG-1: Extract Plan Id returns EMPTY instead of 'unknown-plan-id';
  callers now assert extraction succeeded with diagnostic messages
- TEST-1: Remove tautological len()>0 fallback from plan tree assertion;
  split into explicit emptiness check + semantic keyword validation
- TEST-2: Add content assertions on invariant add (verifies 'invariant')
  and invariant list (verifies persisted invariant text)
- TEST-3: Add 'Should Not Contain ... INTERNAL' checks on strategize,
  execute, correct, diff, and apply commands (matches M2 pattern)
- CODE-1: Consolidate Extract Plan Id into common_e2e.resource as a
  shared keyword with optional stderr arg; remove duplicate definitions
  from m1_acceptance.robot and m3_acceptance.robot; update m2 to use it
---
 robot/e2e/common_e2e.resource | 19 +++++++++++++++++++
 robot/e2e/m1_acceptance.robot | 12 ------------
 robot/e2e/m2_acceptance.robot |  6 +++---
 robot/e2e/m3_acceptance.robot | 29 +++++++++++++----------------
 4 files changed, 35 insertions(+), 31 deletions(-)

diff --git a/robot/e2e/common_e2e.resource b/robot/e2e/common_e2e.resource
index 29bd6e3ff..db21ad68e 100644
--- a/robot/e2e/common_e2e.resource
+++ b/robot/e2e/common_e2e.resource
@@ -186,6 +186,25 @@ Extract JSON From Stdout
     END
     RETURN    ${json_obj}
 
+Extract Plan Id
+    [Documentation]    Extract a plan ID (ULID or UUID) from CLI output.
+    ...
+    ...                Searches stdout (and optionally stderr) for a 26-character
+    ...                ULID, a standard UUID, or a ``plan_id: <value>`` pattern.
+    ...                Returns the first match, or EMPTY if none found.
+    [Arguments]    ${stdout}    ${stderr}=${EMPTY}
+    ${combined}=    Set Variable    ${stdout}\n${stderr}
+    ${status}    ${match}=    Run Keyword And Ignore Error
+    ...    Evaluate    __import__('re').search(r'[0-9A-Z]{26}', '''${combined}''').group()
+    Return From Keyword If    '${status}' == 'PASS'    ${match}
+    ${status}    ${match}=    Run Keyword And Ignore Error
+    ...    Evaluate    __import__('re').search(r'[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', '''${combined}''').group()
+    Return From Keyword If    '${status}' == 'PASS'    ${match}
+    ${status}    ${match}=    Run Keyword And Ignore Error
+    ...    Evaluate    __import__('re').search(r'plan_id[:\\s]+([\\w-]+)', '''${combined}''').group(1)
+    Return From Keyword If    '${status}' == 'PASS'    ${match}
+    RETURN    ${EMPTY}
+
 Link Resource To Project
     [Documentation]    Link the suite-level workspace resource to a project.
     ...
diff --git a/robot/e2e/m1_acceptance.robot b/robot/e2e/m1_acceptance.robot
index 014c41baa..73e68f0e9 100644
--- a/robot/e2e/m1_acceptance.robot
+++ b/robot/e2e/m1_acceptance.robot
@@ -121,15 +121,3 @@ M1 Full Plan Lifecycle
     # Git repo still has commits (verified above)
     Log    M1 Full Plan Lifecycle E2E test completed successfully
 
-*** Keywords ***
-Extract Plan Id
-    [Documentation]    Extract a ULID-style plan ID from command output.
-    ...
-    ...                Searches for a 26-character alphanumeric ULID pattern
-    ...                in the output text. Returns the first match or EMPTY.
-    [Arguments]    ${text}
-    # ULID pattern: 26 uppercase alphanumeric characters (Crockford Base32)
-    ${matches}=    Get Regexp Matches    ${text}    [0-9A-HJ-NP-Z]{26}    flags=IGNORECASE
-    ${count}=    Get Length    ${matches}
-    ${plan_id}=    Set Variable If    ${count} > 0    ${matches}[0]    ${EMPTY}
-    RETURN    ${plan_id}
diff --git a/robot/e2e/m2_acceptance.robot b/robot/e2e/m2_acceptance.robot
index 104521bca..8a3d18df6 100644
--- a/robot/e2e/m2_acceptance.robot
+++ b/robot/e2e/m2_acceptance.robot
@@ -88,9 +88,9 @@ M2 Full Actor Compiler And LLM Integration
     ${r_use}=    Run CleverAgents Command
     ...    plan    use    ${ACTION_NAME}    ${PROJECT_NAME}    --format    plain
     Should Not Be Empty    ${r_use.stdout}
-    ${plan_ids}=    Get Regexp Matches    ${r_use.stdout}    [0-9A-Z]{26}
-    Should Not Be Empty    ${plan_ids}    msg=Expected a ULID plan ID in plan use output
-    ${plan_id}=    Set Variable    ${plan_ids}[0]
+    ${plan_id}=    Extract Plan Id    ${r_use.stdout}    ${r_use.stderr}
+    Should Not Be Empty    ${plan_id}
+    ...    Could not extract plan ID from plan use output.\nSTDOUT: ${r_use.stdout}\nSTDERR: ${r_use.stderr}
     Log    Extracted plan_id: ${plan_id}
 
     # ---- Step 6: Plan execute — strategize phase ----
diff --git a/robot/e2e/m3_acceptance.robot b/robot/e2e/m3_acceptance.robot
index 87fcbb83c..914005fa1 100644
--- a/robot/e2e/m3_acceptance.robot
+++ b/robot/e2e/m3_acceptance.robot
@@ -87,6 +87,7 @@ M3 Decision Recording And Tree Visualization
     Log    Invariant add stderr: ${inv_result.stderr}
     ${inv_output}=    Set Variable    ${inv_result.stdout}\n${inv_result.stderr}
     Should Not Contain    ${inv_output}    Traceback
+    Output Should Contain    ${inv_result}    invariant
 
     # ── 4. List invariants to verify persistence ──
     ${inv_list}=    Run CleverAgents Command
@@ -95,6 +96,7 @@ M3 Decision Recording And Tree Visualization
     Log    Invariant list stderr: ${inv_list.stderr}
     ${inv_list_output}=    Set Variable    ${inv_list.stdout}\n${inv_list.stderr}
     Should Not Contain    ${inv_list_output}    Traceback
+    Output Should Contain    ${inv_list}    API function signatures
 
     # ── 5. Create action YAML for refactoring task ──
     ${action_yaml}=    Catenate    SEPARATOR=\n
@@ -115,6 +117,8 @@ M3 Decision Recording And Tree Visualization
     ...    plan    use    ${ACTION_NAME}    ${PROJECT_NAME}    --format    plain
     Should Not Be Empty    ${r_use.stdout}
     ${plan_id}=    Extract Plan Id    ${r_use.stdout}    ${r_use.stderr}
+    Should Not Be Empty    ${plan_id}
+    ...    Could not extract plan ID from plan use output.\nSTDOUT: ${r_use.stdout}\nSTDERR: ${r_use.stderr}
     Log    Extracted plan_id: ${plan_id}
 
     # ── 7. Plan execute — strategize phase (decisions recorded) ──
@@ -122,6 +126,7 @@ M3 Decision Recording And Tree Visualization
     ...    plan    execute    ${plan_id}    --format    plain
     ...    timeout=300s
     Should Not Contain    ${r_strategize.stdout}${r_strategize.stderr}    Traceback
+    Should Not Contain    ${r_strategize.stdout}${r_strategize.stderr}    INTERNAL
     Log    Strategize output: ${r_strategize.stdout}
 
     # ── 8. Plan tree — view decision tree ──
@@ -132,7 +137,10 @@ M3 Decision Recording And Tree Visualization
     ${tree_output}=    Set Variable    ${r_tree.stdout}\n${r_tree.stderr}
     Should Not Contain    ${tree_output}    Traceback
     # Tree output should reference the plan or contain tree-like structure
-    Should Be True    'plan' in '''${tree_output}'''.lower() or 'decision' in '''${tree_output}'''.lower() or 'tree' in '''${tree_output}'''.lower() or len('''${r_tree.stdout}''') > 0
+    Should Not Be Empty    ${r_tree.stdout}
+    ...    Plan tree produced no output for plan ${plan_id}
+    Should Be True    'plan' in '''${tree_output}'''.lower() or 'decision' in '''${tree_output}'''.lower() or 'tree' in '''${tree_output}'''.lower()
+    ...    Plan tree output does not contain expected keywords (plan/decision/tree).\nOutput: ${tree_output}
 
     # ── 9. Plan explain — inspect a decision (use plan_id as fallback) ──
     ${r_explain}=    Run CleverAgents Command
@@ -147,6 +155,7 @@ M3 Decision Recording And Tree Visualization
     ...    plan    execute    ${plan_id}    --format    plain
     ...    timeout=300s
     Should Not Contain    ${r_execute.stdout}${r_execute.stderr}    Traceback
+    Should Not Contain    ${r_execute.stdout}${r_execute.stderr}    INTERNAL
     Log    Execute phase output: ${r_execute.stdout}
 
     # ── 11. Plan correct — revert mode with guidance ──
@@ -159,6 +168,7 @@ M3 Decision Recording And Tree Visualization
     Log    Plan correct stderr: ${r_correct.stderr}
     ${correct_output}=    Set Variable    ${r_correct.stdout}\n${r_correct.stderr}
     Should Not Contain    ${correct_output}    Traceback
+    Should Not Contain    ${correct_output}    INTERNAL
 
     # ── 12. Plan diff — show changes ──
     ${r_diff}=    Run CleverAgents Command
@@ -166,6 +176,7 @@ M3 Decision Recording And Tree Visualization
     Log    Plan diff stdout: ${r_diff.stdout}
     Log    Plan diff stderr: ${r_diff.stderr}
     Should Not Contain    ${r_diff.stdout}${r_diff.stderr}    Traceback
+    Should Not Contain    ${r_diff.stdout}${r_diff.stderr}    INTERNAL
 
     # ── 13. Plan apply ──
     ${r_apply}=    Run CleverAgents Command
@@ -174,6 +185,7 @@ M3 Decision Recording And Tree Visualization
     Log    Plan apply stdout: ${r_apply.stdout}
     Log    Plan apply stderr: ${r_apply.stderr}
     Should Not Contain    ${r_apply.stdout}${r_apply.stderr}    Traceback
+    Should Not Contain    ${r_apply.stdout}${r_apply.stderr}    INTERNAL
 
     # ── 14. Final plan status ──
     ${r_status}=    Run CleverAgents Command
@@ -182,18 +194,3 @@ M3 Decision Recording And Tree Visualization
     Log    Final plan status: ${r_status.stdout}
     Log    M3 acceptance E2E test completed successfully
 
-*** Keywords ***
-Extract Plan Id
-    [Documentation]    Extract a plan ID (ULID/UUID) from combined CLI output.
-    [Arguments]    ${stdout}    ${stderr}
-    ${combined}=    Set Variable    ${stdout}\n${stderr}
-    ${status}    ${match}=    Run Keyword And Ignore Error
-    ...    Evaluate    __import__('re').search(r'[0-9A-Z]{26}', '''${combined}''').group()
-    Return From Keyword If    '${status}' == 'PASS'    ${match}
-    ${status}    ${match}=    Run Keyword And Ignore Error
-    ...    Evaluate    __import__('re').search(r'[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', '''${combined}''').group()
-    Return From Keyword If    '${status}' == 'PASS'    ${match}
-    ${status}    ${match}=    Run Keyword And Ignore Error
-    ...    Evaluate    __import__('re').search(r'plan_id[:\\s]+([\\w-]+)', '''${combined}''').group(1)
-    Return From Keyword If    '${status}' == 'PASS'    ${match}
-    RETURN    unknown-plan-id
-- 
2.52.0


From ed9e9b1485ccf61607b9dc252905487754997a38 Mon Sep 17 00:00:00 2001
From: Hamza Khyari <hamza.khyari@cleverthis.com>
Date: Tue, 17 Mar 2026 05:17:52 +0100
Subject: [PATCH 3/8] test(e2e): tag M3 acceptance test as tdd_expected_fail
 (#1022)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

InvariantService uses in-memory storage only — invariants added via
`invariant add` are lost when the CLI process exits. The M3 E2E test
correctly catches this as a persistence failure at the `invariant list`
step. Invariant persistence is a v3.4.0 deliverable (not M3 scope).

Tagged with tdd_bug, tdd_bug_1022, tdd_expected_fail per TDD bug-capture
workflow (CONTRIBUTING.md). The listener inverts the result so the test
passes CI while the bug is unfixed.

ISSUES CLOSED: #743
---
 robot/e2e/common_e2e.resource | 24 ++++++++++++++----------
 robot/e2e/m3_acceptance.robot | 26 ++++++++++++++++++++++----
 2 files changed, 36 insertions(+), 14 deletions(-)

diff --git a/robot/e2e/common_e2e.resource b/robot/e2e/common_e2e.resource
index db21ad68e..b4ac21b7c 100644
--- a/robot/e2e/common_e2e.resource
+++ b/robot/e2e/common_e2e.resource
@@ -190,19 +190,23 @@ Extract Plan Id
     [Documentation]    Extract a plan ID (ULID or UUID) from CLI output.
     ...
     ...                Searches stdout (and optionally stderr) for a 26-character
-    ...                ULID, a standard UUID, or a ``plan_id: <value>`` pattern.
+    ...                ULID (Crockford Base32), a standard UUID, or a
+    ...                ``plan_id: <value>`` pattern.
     ...                Returns the first match, or EMPTY if none found.
     [Arguments]    ${stdout}    ${stderr}=${EMPTY}
     ${combined}=    Set Variable    ${stdout}\n${stderr}
-    ${status}    ${match}=    Run Keyword And Ignore Error
-    ...    Evaluate    __import__('re').search(r'[0-9A-Z]{26}', '''${combined}''').group()
-    Return From Keyword If    '${status}' == 'PASS'    ${match}
-    ${status}    ${match}=    Run Keyword And Ignore Error
-    ...    Evaluate    __import__('re').search(r'[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', '''${combined}''').group()
-    Return From Keyword If    '${status}' == 'PASS'    ${match}
-    ${status}    ${match}=    Run Keyword And Ignore Error
-    ...    Evaluate    __import__('re').search(r'plan_id[:\\s]+([\\w-]+)', '''${combined}''').group(1)
-    Return From Keyword If    '${status}' == 'PASS'    ${match}
+    # Try ULID (Crockford Base32: excludes I, L, O, U)
+    @{ulid_matches}=    Get Regexp Matches    ${combined}    [0-9A-HJKMNP-TV-Z]{26}
+    ${ulid_count}=    Get Length    ${ulid_matches}
+    Return From Keyword If    ${ulid_count} > 0    ${ulid_matches}[0]
+    # Try UUID
+    @{uuid_matches}=    Get Regexp Matches    ${combined}    [0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}
+    ${uuid_count}=    Get Length    ${uuid_matches}
+    Return From Keyword If    ${uuid_count} > 0    ${uuid_matches}[0]
+    # Try plan_id: <value> pattern
+    @{id_matches}=    Get Regexp Matches    ${combined}    plan_id[:\\\\s]+(\\w+)    1
+    ${id_count}=    Get Length    ${id_matches}
+    Return From Keyword If    ${id_count} > 0    ${id_matches}[0]
     RETURN    ${EMPTY}
 
 Link Resource To Project
diff --git a/robot/e2e/m3_acceptance.robot b/robot/e2e/m3_acceptance.robot
index 914005fa1..d3a6d82bb 100644
--- a/robot/e2e/m3_acceptance.robot
+++ b/robot/e2e/m3_acceptance.robot
@@ -23,7 +23,9 @@ M3 Decision Recording And Tree Visualization
     ...
     ...                Exercises the full M3 feature set through the CLI with real
     ...                LLM API keys. Validates structural output, not exact text.
-    [Tags]    E2E
+    [Tags]    E2E    tdd_bug    tdd_bug_1022    tdd_expected_fail
+    [Timeout]    15 minutes
+    Skip If No LLM Keys
 
     # ── Initialize database ──
     ${r_init}=    Run CleverAgents Command
@@ -60,9 +62,9 @@ M3 Decision Recording And Tree Visualization
     ...    ${SPACE}${SPACE}${SPACE}${SPACE}conn.close()
     ...    ${SPACE}${SPACE}${SPACE}${SPACE}return {"id": row[0], "name": row[1]} if row else {}
     Create File    ${repo_dir}${/}src${/}auth.py    ${auth_code}
-    Run Process    git    add    .    cwd=${repo_dir}
-    Run Process    git    commit    -m    Add auth module    cwd=${repo_dir}
-    ${branch_result}=    Run Process    git    rev-parse    --abbrev-ref    HEAD    cwd=${repo_dir}
+    Run Process    git    add    .    cwd=${repo_dir}    timeout=60s    on_timeout=kill
+    Run Process    git    commit    -m    Add auth module    cwd=${repo_dir}    timeout=60s    on_timeout=kill
+    ${branch_result}=    Run Process    git    rev-parse    --abbrev-ref    HEAD    cwd=${repo_dir}    timeout=60s    on_timeout=kill
     ${branch}=    Strip String    ${branch_result.stdout}
 
     # ── 2. Register resource and create project ──
@@ -88,6 +90,7 @@ M3 Decision Recording And Tree Visualization
     ${inv_output}=    Set Variable    ${inv_result.stdout}\n${inv_result.stderr}
     Should Not Contain    ${inv_output}    Traceback
     Output Should Contain    ${inv_result}    invariant
+    Output Should Contain    ${inv_result}    API function signatures
 
     # ── 4. List invariants to verify persistence ──
     ${inv_list}=    Run CleverAgents Command
@@ -149,6 +152,11 @@ M3 Decision Recording And Tree Visualization
     Log    Plan explain stderr: ${r_explain.stderr}
     ${explain_output}=    Set Variable    ${r_explain.stdout}\n${r_explain.stderr}
     Should Not Contain    ${explain_output}    Traceback
+    Should Not Be Empty    ${r_explain.stdout}
+    ...    Plan explain produced no output for plan ${plan_id}
+    # M3 AC: "plan explain shows decision details including alternatives"
+    Should Be True    'decision' in '''${explain_output}'''.lower() or 'rationale' in '''${explain_output}'''.lower() or 'reasoning' in '''${explain_output}'''.lower() or 'explain' in '''${explain_output}'''.lower()
+    ...    Plan explain output missing decision/rationale keywords.\nOutput: ${explain_output}
 
     # ── 10. Plan execute — advance to execute phase ──
     ${r_execute}=    Run CleverAgents Command
@@ -169,6 +177,11 @@ M3 Decision Recording And Tree Visualization
     ${correct_output}=    Set Variable    ${r_correct.stdout}\n${r_correct.stderr}
     Should Not Contain    ${correct_output}    Traceback
     Should Not Contain    ${correct_output}    INTERNAL
+    Should Not Be Empty    ${r_correct.stdout}
+    ...    Plan correct produced no output for plan ${plan_id}
+    # M3 AC: "plan correct --mode=revert re-executes from targeted decision point"
+    Should Be True    'revert' in '''${correct_output}'''.lower() or 'correct' in '''${correct_output}'''.lower() or 'decision' in '''${correct_output}'''.lower()
+    ...    Plan correct output missing revert/correct/decision keywords.\nOutput: ${correct_output}
 
     # ── 12. Plan diff — show changes ──
     ${r_diff}=    Run CleverAgents Command
@@ -191,6 +204,11 @@ M3 Decision Recording And Tree Visualization
     ${r_status}=    Run CleverAgents Command
     ...    plan    status    ${plan_id}    --format    plain
     Should Not Be Empty    ${r_status.stdout}
+    ${status_output}=    Set Variable    ${r_status.stdout}\n${r_status.stderr}
+    Should Not Contain    ${status_output}    Traceback
+    # Verify plan reached a terminal or advanced phase
+    Should Be True    'applied' in '''${status_output}'''.lower() or 'complete' in '''${status_output}'''.lower() or 'phase' in '''${status_output}'''.lower()
+    ...    Final status does not indicate terminal phase.\nOutput: ${status_output}
     Log    Final plan status: ${r_status.stdout}
     Log    M3 acceptance E2E test completed successfully
 
-- 
2.52.0


From 9784288e362230b0f15ddd0b043715d11888cbb7 Mon Sep 17 00:00:00 2001
From: Hamza Khyari <hamza.khyari@cleverthis.com>
Date: Wed, 1 Apr 2026 15:57:47 +0000
Subject: [PATCH 4/8] fix(test): add context snapshot and invariant enforcement
 assertions for M3 AC

AC-2: Add plan tree --format json step (8b) to verify decisions are
recorded with context snapshots.  Checks for decision_id keys and
context_snapshot/snapshot/context presence in JSON output.

AC-5: Add invariant enforcement check (7b) after strategize phase.
Checks whether the invariant text or 'invariant' keyword appears in
strategize output.  Diagnostic-only (not hard assertion) because LLM
output is non-deterministic.

Ref: #743
---
 robot/e2e/m3_acceptance.robot | 35 +++++++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)

diff --git a/robot/e2e/m3_acceptance.robot b/robot/e2e/m3_acceptance.robot
index d3a6d82bb..800e7fbbb 100644
--- a/robot/e2e/m3_acceptance.robot
+++ b/robot/e2e/m3_acceptance.robot
@@ -132,19 +132,50 @@ M3 Decision Recording And Tree Visualization
     Should Not Contain    ${r_strategize.stdout}${r_strategize.stderr}    INTERNAL
     Log    Strategize output: ${r_strategize.stdout}
 
-    # ── 8. Plan tree — view decision tree ──
+    # ── 7b. Verify invariant enforcement during strategize ──
+    # M3 AC #5: invariants are enforced during strategize.  The strategize
+    # output or the decision tree should reference the invariant text or
+    # indicate invariant-aware processing.
+    ${strat_combined}=    Set Variable    ${r_strategize.stdout}\n${r_strategize.stderr}
+    ${inv_mentioned}=    Evaluate
+    ...    'invariant' in '''${strat_combined}'''.lower() or 'api function signatures' in '''${strat_combined}'''.lower()
+    Log    Invariant mentioned in strategize output: ${inv_mentioned}
+    # NOTE: if the LLM does not explicitly mention invariants in its output,
+    # this is logged for diagnostics.  The invariant was registered (step 3)
+    # and the strategize phase has access to it.  Hard assertion is deferred
+    # because LLM output is non-deterministic.
+
+    # ── 8. Plan tree — view decision tree (plain) ──
     ${r_tree}=    Run CleverAgents Command
     ...    plan    tree    ${plan_id}    --format    plain
     Log    Plan tree stdout: ${r_tree.stdout}
     Log    Plan tree stderr: ${r_tree.stderr}
     ${tree_output}=    Set Variable    ${r_tree.stdout}\n${r_tree.stderr}
     Should Not Contain    ${tree_output}    Traceback
-    # Tree output should reference the plan or contain tree-like structure
     Should Not Be Empty    ${r_tree.stdout}
     ...    Plan tree produced no output for plan ${plan_id}
     Should Be True    'plan' in '''${tree_output}'''.lower() or 'decision' in '''${tree_output}'''.lower() or 'tree' in '''${tree_output}'''.lower()
     ...    Plan tree output does not contain expected keywords (plan/decision/tree).\nOutput: ${tree_output}
 
+    # ── 8b. Plan tree JSON — verify decisions recorded with context snapshots ──
+    # M3 AC #2: decisions are recorded with context snapshots.
+    ${r_tree_json}=    Run CleverAgents Command
+    ...    plan    tree    ${plan_id}    --format    json
+    Should Not Contain    ${r_tree_json.stdout}${r_tree_json.stderr}    Traceback
+    Should Not Be Empty    ${r_tree_json.stdout}
+    ...    Plan tree --format json produced no output
+    Log    Plan tree JSON: ${r_tree_json.stdout}
+    # Verify JSON contains decision_id keys (structural proof of recorded decisions)
+    Should Contain    ${r_tree_json.stdout}    decision_id
+    ...    Plan tree JSON missing 'decision_id' — decisions may not be recorded
+    # Verify context snapshot data is present (context_snapshot or snapshot or context)
+    ${has_snapshot}=    Evaluate
+    ...    'context_snapshot' in '''${r_tree_json.stdout}''' or 'snapshot' in '''${r_tree_json.stdout}''' or 'context' in '''${r_tree_json.stdout}'''
+    Log    Context snapshot found in tree JSON: ${has_snapshot}
+    # NOTE: context_snapshot presence depends on the tree JSON schema.
+    # If the schema doesn't include snapshots inline, the snapshot is stored
+    # separately and verified via plan explain.  Logged for diagnostics.
+
     # ── 9. Plan explain — inspect a decision (use plan_id as fallback) ──
     ${r_explain}=    Run CleverAgents Command
     ...    plan    explain    ${plan_id}    --format    plain
-- 
2.52.0


From 866186e25494f733c42d143feba14961f9b13ba6 Mon Sep 17 00:00:00 2001
From: Hamza Khyari <hamza.khyari@cleverthis.com>
Date: Wed, 1 Apr 2026 16:11:35 +0000
Subject: [PATCH 5/8] fix(test): address CoreRasurae review findings for M3 E2E
 acceptance test
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

C1 (Critical): Fix plan explain and plan correct to use Decision ID
extracted from plan tree --format json instead of Plan ID.  The spec
requires a decision_id, not a plan_id.

C2 (Critical): Dynamic actor selection based on available API key.
Uses anthropic/claude-sonnet-4 when ANTHROPIC_API_KEY is set,
falls back to openai/gpt-4.  Prevents misleading auth failures in
Anthropic-only environments.

H1: Add rc assertions on git add/commit after auth module creation.
H3: Add alternative/option to plan explain keyword check.
H4: Replace triple-quote Should Be True with $variable syntax across
all keyword assertions to prevent injection from LLM output.
H5: Tighten keyword assertions — use decision_id structural check
in tree JSON, specific terminal indicators in final status.
M1: Remove generic 'phase' from final status, use applied/complete/apply.
M5: Replace deprecated Return From Keyword If with IF/RETURN in
Extract Plan Id keyword (common_e2e.resource).

Ref: #743
---
 robot/e2e/common_e2e.resource |   7 +-
 robot/e2e/m3_acceptance.robot | 131 +++++++++++++++++-----------------
 2 files changed, 68 insertions(+), 70 deletions(-)

diff --git a/robot/e2e/common_e2e.resource b/robot/e2e/common_e2e.resource
index b4ac21b7c..2f4525050 100644
--- a/robot/e2e/common_e2e.resource
+++ b/robot/e2e/common_e2e.resource
@@ -195,18 +195,19 @@ Extract Plan Id
     ...                Returns the first match, or EMPTY if none found.
     [Arguments]    ${stdout}    ${stderr}=${EMPTY}
     ${combined}=    Set Variable    ${stdout}\n${stderr}
+    # M5: Use IF/RETURN instead of deprecated Return From Keyword If
     # Try ULID (Crockford Base32: excludes I, L, O, U)
     @{ulid_matches}=    Get Regexp Matches    ${combined}    [0-9A-HJKMNP-TV-Z]{26}
     ${ulid_count}=    Get Length    ${ulid_matches}
-    Return From Keyword If    ${ulid_count} > 0    ${ulid_matches}[0]
+    IF    ${ulid_count} > 0    RETURN    ${ulid_matches}[0]
     # Try UUID
     @{uuid_matches}=    Get Regexp Matches    ${combined}    [0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}
     ${uuid_count}=    Get Length    ${uuid_matches}
-    Return From Keyword If    ${uuid_count} > 0    ${uuid_matches}[0]
+    IF    ${uuid_count} > 0    RETURN    ${uuid_matches}[0]
     # Try plan_id: <value> pattern
     @{id_matches}=    Get Regexp Matches    ${combined}    plan_id[:\\\\s]+(\\w+)    1
     ${id_count}=    Get Length    ${id_matches}
-    Return From Keyword If    ${id_count} > 0    ${id_matches}[0]
+    IF    ${id_count} > 0    RETURN    ${id_matches}[0]
     RETURN    ${EMPTY}
 
 Link Resource To Project
diff --git a/robot/e2e/m3_acceptance.robot b/robot/e2e/m3_acceptance.robot
index 800e7fbbb..b13562bed 100644
--- a/robot/e2e/m3_acceptance.robot
+++ b/robot/e2e/m3_acceptance.robot
@@ -8,6 +8,7 @@ Documentation     E2E acceptance test for M3 (v3.2.0): Decisions, Validations, a
 ...
 ...               Zero mocking — all CLI invocations use real providers.
 Resource          common_e2e.resource
+Library           String
 Suite Setup       E2E Suite Setup
 Suite Teardown    E2E Suite Teardown
 
@@ -62,8 +63,11 @@ M3 Decision Recording And Tree Visualization
     ...    ${SPACE}${SPACE}${SPACE}${SPACE}conn.close()
     ...    ${SPACE}${SPACE}${SPACE}${SPACE}return {"id": row[0], "name": row[1]} if row else {}
     Create File    ${repo_dir}${/}src${/}auth.py    ${auth_code}
-    Run Process    git    add    .    cwd=${repo_dir}    timeout=60s    on_timeout=kill
-    Run Process    git    commit    -m    Add auth module    cwd=${repo_dir}    timeout=60s    on_timeout=kill
+    # H1: Assert rc on git add/commit
+    ${r_git_add}=    Run Process    git    add    .    cwd=${repo_dir}    timeout=60s    on_timeout=kill
+    Should Be Equal As Integers    ${r_git_add.rc}    0    msg=git add failed (rc=${r_git_add.rc})
+    ${r_git_commit}=    Run Process    git    commit    -m    Add auth module    cwd=${repo_dir}    timeout=60s    on_timeout=kill
+    Should Be Equal As Integers    ${r_git_commit.rc}    0    msg=git commit failed (rc=${r_git_commit.rc})
     ${branch_result}=    Run Process    git    rev-parse    --abbrev-ref    HEAD    cwd=${repo_dir}    timeout=60s    on_timeout=kill
     ${branch}=    Strip String    ${branch_result.stdout}
 
@@ -87,8 +91,7 @@ M3 Decision Recording And Tree Visualization
     ...    API function signatures must not change
     Log    Invariant add stdout: ${inv_result.stdout}
     Log    Invariant add stderr: ${inv_result.stderr}
-    ${inv_output}=    Set Variable    ${inv_result.stdout}\n${inv_result.stderr}
-    Should Not Contain    ${inv_output}    Traceback
+    Should Not Contain    ${inv_result.stdout}${inv_result.stderr}    Traceback
     Output Should Contain    ${inv_result}    invariant
     Output Should Contain    ${inv_result}    API function signatures
 
@@ -97,17 +100,20 @@ M3 Decision Recording And Tree Visualization
     ...    invariant    list    --project    ${PROJECT_NAME}
     Log    Invariant list stdout: ${inv_list.stdout}
     Log    Invariant list stderr: ${inv_list.stderr}
-    ${inv_list_output}=    Set Variable    ${inv_list.stdout}\n${inv_list.stderr}
-    Should Not Contain    ${inv_list_output}    Traceback
+    Should Not Contain    ${inv_list.stdout}${inv_list.stderr}    Traceback
     Output Should Contain    ${inv_list}    API function signatures
 
-    # ── 5. Create action YAML for refactoring task ──
+    # ── 5. Create action YAML — C2: dynamic actor selection ──
+    # Select actor based on available API key to avoid misleading failures
+    # when only one provider's key is available.
+    ${actor}=    Set Variable If    '%{ANTHROPIC_API_KEY}' != ''
+    ...    anthropic/claude-sonnet-4    openai/gpt-4
     ${action_yaml}=    Catenate    SEPARATOR=\n
     ...    name: ${ACTION_NAME}
     ...    description: Refactor auth module from raw SQL to ORM
     ...    definition_of_done: Replace raw SQL queries with ORM while preserving API
-    ...    strategy_actor: openai/gpt-4
-    ...    execution_actor: openai/gpt-4
+    ...    strategy_actor: ${actor}
+    ...    execution_actor: ${actor}
     ${action_yaml_path}=    Set Variable    ${SUITE_HOME}${/}m3_action.yaml
     Create File    ${action_yaml_path}    ${action_yaml}
     ${r_action}=    Run CleverAgents Command
@@ -121,7 +127,7 @@ M3 Decision Recording And Tree Visualization
     Should Not Be Empty    ${r_use.stdout}
     ${plan_id}=    Extract Plan Id    ${r_use.stdout}    ${r_use.stderr}
     Should Not Be Empty    ${plan_id}
-    ...    Could not extract plan ID from plan use output.\nSTDOUT: ${r_use.stdout}\nSTDERR: ${r_use.stderr}
+    ...    Could not extract plan ID from plan use output.
     Log    Extracted plan_id: ${plan_id}
 
     # ── 7. Plan execute — strategize phase (decisions recorded) ──
@@ -132,62 +138,54 @@ M3 Decision Recording And Tree Visualization
     Should Not Contain    ${r_strategize.stdout}${r_strategize.stderr}    INTERNAL
     Log    Strategize output: ${r_strategize.stdout}
 
-    # ── 7b. Verify invariant enforcement during strategize ──
-    # M3 AC #5: invariants are enforced during strategize.  The strategize
-    # output or the decision tree should reference the invariant text or
-    # indicate invariant-aware processing.
-    ${strat_combined}=    Set Variable    ${r_strategize.stdout}\n${r_strategize.stderr}
-    ${inv_mentioned}=    Evaluate
-    ...    'invariant' in '''${strat_combined}'''.lower() or 'api function signatures' in '''${strat_combined}'''.lower()
-    Log    Invariant mentioned in strategize output: ${inv_mentioned}
-    # NOTE: if the LLM does not explicitly mention invariants in its output,
-    # this is logged for diagnostics.  The invariant was registered (step 3)
-    # and the strategize phase has access to it.  Hard assertion is deferred
-    # because LLM output is non-deterministic.
+    # ── 7b. Check invariant enforcement in strategize output ──
+    ${inv_in_strat}=    Evaluate
+    ...    'invariant' in $r_strategize.stdout.lower() or 'api function signatures' in $r_strategize.stdout.lower()
+    Log    Invariant mentioned in strategize output: ${inv_in_strat}
 
-    # ── 8. Plan tree — view decision tree (plain) ──
-    ${r_tree}=    Run CleverAgents Command
-    ...    plan    tree    ${plan_id}    --format    plain
-    Log    Plan tree stdout: ${r_tree.stdout}
-    Log    Plan tree stderr: ${r_tree.stderr}
-    ${tree_output}=    Set Variable    ${r_tree.stdout}\n${r_tree.stderr}
-    Should Not Contain    ${tree_output}    Traceback
-    Should Not Be Empty    ${r_tree.stdout}
-    ...    Plan tree produced no output for plan ${plan_id}
-    Should Be True    'plan' in '''${tree_output}'''.lower() or 'decision' in '''${tree_output}'''.lower() or 'tree' in '''${tree_output}'''.lower()
-    ...    Plan tree output does not contain expected keywords (plan/decision/tree).\nOutput: ${tree_output}
-
-    # ── 8b. Plan tree JSON — verify decisions recorded with context snapshots ──
-    # M3 AC #2: decisions are recorded with context snapshots.
+    # ── 8. Plan tree JSON — extract decision IDs and verify snapshots ──
+    # C1: Use plan tree JSON to extract a real decision_id for explain/correct
     ${r_tree_json}=    Run CleverAgents Command
     ...    plan    tree    ${plan_id}    --format    json
     Should Not Contain    ${r_tree_json.stdout}${r_tree_json.stderr}    Traceback
     Should Not Be Empty    ${r_tree_json.stdout}
     ...    Plan tree --format json produced no output
     Log    Plan tree JSON: ${r_tree_json.stdout}
-    # Verify JSON contains decision_id keys (structural proof of recorded decisions)
+    # H5: Verify decision_id key exists (specific structural check)
     Should Contain    ${r_tree_json.stdout}    decision_id
-    ...    Plan tree JSON missing 'decision_id' — decisions may not be recorded
-    # Verify context snapshot data is present (context_snapshot or snapshot or context)
+    ...    Plan tree JSON missing 'decision_id' — decisions not recorded
+    # Extract first decision_id via regex
+    ${decision_ids}=    Get Regexp Matches    ${r_tree_json.stdout}    [0-9A-HJKMNP-TV-Z]{26}
+    ${has_decisions}=    Get Length    ${decision_ids}
+    Should Be True    ${has_decisions} > 0
+    ...    No decision IDs found in plan tree JSON output
+    ${decision_id}=    Set Variable    ${decision_ids}[0]
+    Log    Extracted decision_id: ${decision_id}
+    # Check context snapshot presence
     ${has_snapshot}=    Evaluate
-    ...    'context_snapshot' in '''${r_tree_json.stdout}''' or 'snapshot' in '''${r_tree_json.stdout}''' or 'context' in '''${r_tree_json.stdout}'''
+    ...    'context_snapshot' in $r_tree_json.stdout or 'snapshot' in $r_tree_json.stdout or 'context' in $r_tree_json.stdout
     Log    Context snapshot found in tree JSON: ${has_snapshot}
-    # NOTE: context_snapshot presence depends on the tree JSON schema.
-    # If the schema doesn't include snapshots inline, the snapshot is stored
-    # separately and verified via plan explain.  Logged for diagnostics.
 
-    # ── 9. Plan explain — inspect a decision (use plan_id as fallback) ──
+    # ── 8b. Plan tree plain — structural check ──
+    ${r_tree}=    Run CleverAgents Command
+    ...    plan    tree    ${plan_id}    --format    plain
+    Should Not Contain    ${r_tree.stdout}${r_tree.stderr}    Traceback
+    Should Not Be Empty    ${r_tree.stdout}
+    ...    Plan tree produced no output for plan ${plan_id}
+
+    # ── 9. Plan explain — C1: use decision_id, not plan_id ──
     ${r_explain}=    Run CleverAgents Command
-    ...    plan    explain    ${plan_id}    --format    plain
+    ...    plan    explain    ${decision_id}    --format    plain
     Log    Plan explain stdout: ${r_explain.stdout}
     Log    Plan explain stderr: ${r_explain.stderr}
-    ${explain_output}=    Set Variable    ${r_explain.stdout}\n${r_explain.stderr}
-    Should Not Contain    ${explain_output}    Traceback
+    Should Not Contain    ${r_explain.stdout}${r_explain.stderr}    Traceback
     Should Not Be Empty    ${r_explain.stdout}
-    ...    Plan explain produced no output for plan ${plan_id}
-    # M3 AC: "plan explain shows decision details including alternatives"
-    Should Be True    'decision' in '''${explain_output}'''.lower() or 'rationale' in '''${explain_output}'''.lower() or 'reasoning' in '''${explain_output}'''.lower() or 'explain' in '''${explain_output}'''.lower()
-    ...    Plan explain output missing decision/rationale keywords.\nOutput: ${explain_output}
+    ...    Plan explain produced no output for decision ${decision_id}
+    # H3: Check for alternatives/options in addition to decision/rationale
+    # H4: Use $variable syntax to avoid triple-quote injection
+    Should Be True
+    ...    'decision' in $r_explain.stdout.lower() or 'rationale' in $r_explain.stdout.lower() or 'reasoning' in $r_explain.stdout.lower() or 'alternative' in $r_explain.stdout.lower() or 'option' in $r_explain.stdout.lower()
+    ...    Plan explain output missing decision/rationale/alternatives keywords.
 
     # ── 10. Plan execute — advance to execute phase ──
     ${r_execute}=    Run CleverAgents Command
@@ -197,22 +195,22 @@ M3 Decision Recording And Tree Visualization
     Should Not Contain    ${r_execute.stdout}${r_execute.stderr}    INTERNAL
     Log    Execute phase output: ${r_execute.stdout}
 
-    # ── 11. Plan correct — revert mode with guidance ──
+    # ── 11. Plan correct — C1: use decision_id, not plan_id ──
     ${r_correct}=    Run CleverAgents Command
-    ...    plan    correct    ${plan_id}    --mode    revert
+    ...    plan    correct    ${decision_id}    --mode    revert
     ...    --guidance    Use SQLAlchemy ORM instead of raw SQL
     ...    --yes    --format    plain
     ...    timeout=300s
     Log    Plan correct stdout: ${r_correct.stdout}
     Log    Plan correct stderr: ${r_correct.stderr}
-    ${correct_output}=    Set Variable    ${r_correct.stdout}\n${r_correct.stderr}
-    Should Not Contain    ${correct_output}    Traceback
-    Should Not Contain    ${correct_output}    INTERNAL
+    Should Not Contain    ${r_correct.stdout}${r_correct.stderr}    Traceback
+    Should Not Contain    ${r_correct.stdout}${r_correct.stderr}    INTERNAL
     Should Not Be Empty    ${r_correct.stdout}
-    ...    Plan correct produced no output for plan ${plan_id}
-    # M3 AC: "plan correct --mode=revert re-executes from targeted decision point"
-    Should Be True    'revert' in '''${correct_output}'''.lower() or 'correct' in '''${correct_output}'''.lower() or 'decision' in '''${correct_output}'''.lower()
-    ...    Plan correct output missing revert/correct/decision keywords.\nOutput: ${correct_output}
+    ...    Plan correct produced no output for decision ${decision_id}
+    # H4: Use $variable syntax
+    Should Be True
+    ...    'revert' in $r_correct.stdout.lower() or 'correct' in $r_correct.stdout.lower() or 'decision' in $r_correct.stdout.lower()
+    ...    Plan correct output missing revert/correct/decision keywords.
 
     # ── 12. Plan diff — show changes ──
     ${r_diff}=    Run CleverAgents Command
@@ -231,15 +229,14 @@ M3 Decision Recording And Tree Visualization
     Should Not Contain    ${r_apply.stdout}${r_apply.stderr}    Traceback
     Should Not Contain    ${r_apply.stdout}${r_apply.stderr}    INTERNAL
 
-    # ── 14. Final plan status ──
+    # ── 14. Final plan status — M1: remove generic 'phase', use specific terms ──
     ${r_status}=    Run CleverAgents Command
     ...    plan    status    ${plan_id}    --format    plain
     Should Not Be Empty    ${r_status.stdout}
-    ${status_output}=    Set Variable    ${r_status.stdout}\n${r_status.stderr}
-    Should Not Contain    ${status_output}    Traceback
-    # Verify plan reached a terminal or advanced phase
-    Should Be True    'applied' in '''${status_output}'''.lower() or 'complete' in '''${status_output}'''.lower() or 'phase' in '''${status_output}'''.lower()
-    ...    Final status does not indicate terminal phase.\nOutput: ${status_output}
+    Should Not Contain    ${r_status.stdout}${r_status.stderr}    Traceback
+    # H4/H5/M1: Use $variable syntax, check specific terminal indicators
+    Should Be True
+    ...    'applied' in $r_status.stdout.lower() or 'complete' in $r_status.stdout.lower() or 'apply' in $r_status.stdout.lower()
+    ...    Final status does not indicate terminal phase (expected applied/complete/apply).
     Log    Final plan status: ${r_status.stdout}
     Log    M3 acceptance E2E test completed successfully
-
-- 
2.52.0


From f15e0182affb102fd3878d794c465de75eadfb3f Mon Sep 17 00:00:00 2001
From: Hamza Khyari <hamza.khyari@cleverthis.com>
Date: Wed, 1 Apr 2026 19:12:43 +0000
Subject: [PATCH 6/8] test(e2e): harden M3 acceptance assertions for snapshots
 and invariants

- AC-2: assert context_snapshot.hot_context_hash via plan explain
  --format json --show-context --show-reasoning
- AC-5: add hard assertion that plan tree --format json contains an
  invariant_enforced decision for the added invariant

The AC-5 assertion is intentionally hard while the test remains
@tdd_expected_fail. This encodes the intended CLI behavior without
relying on non-deterministic LLM wording.

Ref: #743
---
 robot/e2e/m3_acceptance.robot | 47 ++++++++++++++++++++++-------------
 1 file changed, 30 insertions(+), 17 deletions(-)

diff --git a/robot/e2e/m3_acceptance.robot b/robot/e2e/m3_acceptance.robot
index b13562bed..b2f074ff6 100644
--- a/robot/e2e/m3_acceptance.robot
+++ b/robot/e2e/m3_acceptance.robot
@@ -138,12 +138,7 @@ M3 Decision Recording And Tree Visualization
     Should Not Contain    ${r_strategize.stdout}${r_strategize.stderr}    INTERNAL
     Log    Strategize output: ${r_strategize.stdout}
 
-    # ── 7b. Check invariant enforcement in strategize output ──
-    ${inv_in_strat}=    Evaluate
-    ...    'invariant' in $r_strategize.stdout.lower() or 'api function signatures' in $r_strategize.stdout.lower()
-    Log    Invariant mentioned in strategize output: ${inv_in_strat}
-
-    # ── 8. Plan tree JSON — extract decision IDs and verify snapshots ──
+    # ── 8. Plan tree JSON — extract decision IDs ──
     # C1: Use plan tree JSON to extract a real decision_id for explain/correct
     ${r_tree_json}=    Run CleverAgents Command
     ...    plan    tree    ${plan_id}    --format    json
@@ -154,6 +149,12 @@ M3 Decision Recording And Tree Visualization
     # H5: Verify decision_id key exists (specific structural check)
     Should Contain    ${r_tree_json.stdout}    decision_id
     ...    Plan tree JSON missing 'decision_id' — decisions not recorded
+    # AC-5: invariant enforcement should be represented structurally in the
+    # decision tree as an invariant_enforced decision. This is intentionally
+    # a hard assertion; the test remains tdd_expected_fail until the CLI
+    # strategize path persists/surfaces invariant enforcement decisions.
+    Should Contain    ${r_tree_json.stdout}    invariant_enforced
+    ...    Plan tree JSON does not contain an invariant_enforced decision for the added invariant
     # Extract first decision_id via regex
     ${decision_ids}=    Get Regexp Matches    ${r_tree_json.stdout}    [0-9A-HJKMNP-TV-Z]{26}
     ${has_decisions}=    Get Length    ${decision_ids}
@@ -161,10 +162,6 @@ M3 Decision Recording And Tree Visualization
     ...    No decision IDs found in plan tree JSON output
     ${decision_id}=    Set Variable    ${decision_ids}[0]
     Log    Extracted decision_id: ${decision_id}
-    # Check context snapshot presence
-    ${has_snapshot}=    Evaluate
-    ...    'context_snapshot' in $r_tree_json.stdout or 'snapshot' in $r_tree_json.stdout or 'context' in $r_tree_json.stdout
-    Log    Context snapshot found in tree JSON: ${has_snapshot}
 
     # ── 8b. Plan tree plain — structural check ──
     ${r_tree}=    Run CleverAgents Command
@@ -173,19 +170,35 @@ M3 Decision Recording And Tree Visualization
     Should Not Be Empty    ${r_tree.stdout}
     ...    Plan tree produced no output for plan ${plan_id}
 
-    # ── 9. Plan explain — C1: use decision_id, not plan_id ──
+    # ── 9. Plan explain JSON — C1 + AC-2 + H3 ──
+    # Use decision_id (not plan_id) and request structured context/reasoning so
+    # the test can assert that the decision carries a context snapshot and that
+    # alternatives are represented in the explain output.
     ${r_explain}=    Run CleverAgents Command
-    ...    plan    explain    ${decision_id}    --format    plain
+    ...    plan    explain    ${decision_id}    --format    json    --show-context    --show-reasoning
     Log    Plan explain stdout: ${r_explain.stdout}
     Log    Plan explain stderr: ${r_explain.stderr}
     Should Not Contain    ${r_explain.stdout}${r_explain.stderr}    Traceback
     Should Not Be Empty    ${r_explain.stdout}
     ...    Plan explain produced no output for decision ${decision_id}
-    # H3: Check for alternatives/options in addition to decision/rationale
-    # H4: Use $variable syntax to avoid triple-quote injection
-    Should Be True
-    ...    'decision' in $r_explain.stdout.lower() or 'rationale' in $r_explain.stdout.lower() or 'reasoning' in $r_explain.stdout.lower() or 'alternative' in $r_explain.stdout.lower() or 'option' in $r_explain.stdout.lower()
-    ...    Plan explain output missing decision/rationale/alternatives keywords.
+    Should Contain    ${r_explain.stdout}    decision_id
+    ...    Explain JSON missing decision_id field
+    Should Contain    ${r_explain.stdout}    alternatives_considered
+    ...    Explain JSON missing alternatives_considered field
+    Should Contain    ${r_explain.stdout}    context_snapshot
+    ...    Explain JSON missing context_snapshot field
+    Should Contain    ${r_explain.stdout}    hot_context_hash
+    ...    Explain JSON missing hot_context_hash within context_snapshot
+    ${explain_json}=    Extract JSON From Stdout    ${r_explain.stdout}
+    ${explained_id}=    Evaluate    $explain_json.get('decision_id', '')
+    Should Be Equal    ${explained_id}    ${decision_id}
+    ...    plan explain returned a different decision_id than requested
+    ${hot_context_hash}=    Evaluate    $explain_json.get('context_snapshot', {}).get('hot_context_hash', '')
+    Should Not Be Empty    ${hot_context_hash}
+    ...    context_snapshot.hot_context_hash is empty — snapshot not recorded
+    ${has_alternatives_key}=    Evaluate    'alternatives_considered' in $explain_json
+    Should Be True    ${has_alternatives_key}
+    ...    Explain JSON does not include alternatives_considered key
 
     # ── 10. Plan execute — advance to execute phase ──
     ${r_execute}=    Run CleverAgents Command
-- 
2.52.0


From e33558f2e5a4b10894ec13e18be562ff9fbca25a Mon Sep 17 00:00:00 2001
From: Hamza Khyari <hamza.khyari@cleverthis.com>
Date: Wed, 1 Apr 2026 23:08:52 +0000
Subject: [PATCH 7/8] fix(test): remove tdd_expected_fail, harden correction
 and status assertions
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

P1-A: Remove tdd_expected_fail, tdd_bug, tdd_bug_1022 tags — bug
#1022 is closed, the test should pass on its own now.

P1-C: After plan correct, verify the decision tree contains
'superseded: true' proving the targeted decision was actually
corrected and marked superseded.

P1-D: Replace weak keyword-based final status check with structured
JSON assertion using plan status --format json.  Verify phase is
'apply' and processing_state is one of queued/processing/complete/applied.

P1-B (decision_id extraction): validated as not-a-bug — plan tree
--format json does not include plan_id, only decision_id fields.
First ULID match is always a decision_id.

Ref: #743
---
 robot/e2e/m3_acceptance.robot | 32 ++++++++++++++++++++++++--------
 1 file changed, 24 insertions(+), 8 deletions(-)

diff --git a/robot/e2e/m3_acceptance.robot b/robot/e2e/m3_acceptance.robot
index b2f074ff6..f46977de1 100644
--- a/robot/e2e/m3_acceptance.robot
+++ b/robot/e2e/m3_acceptance.robot
@@ -24,7 +24,7 @@ M3 Decision Recording And Tree Visualization
     ...
     ...                Exercises the full M3 feature set through the CLI with real
     ...                LLM API keys. Validates structural output, not exact text.
-    [Tags]    E2E    tdd_bug    tdd_bug_1022    tdd_expected_fail
+    [Tags]    E2E
     [Timeout]    15 minutes
     Skip If No LLM Keys
 
@@ -225,6 +225,18 @@ M3 Decision Recording And Tree Visualization
     ...    'revert' in $r_correct.stdout.lower() or 'correct' in $r_correct.stdout.lower() or 'decision' in $r_correct.stdout.lower()
     ...    Plan correct output missing revert/correct/decision keywords.
 
+    # ── 11b. Verify correction created a superseded decision ──
+    # P1-C: After plan correct, the targeted decision should be superseded
+    # and a new correction decision should exist in the tree.
+    ${r_tree_post}=    Run CleverAgents Command
+    ...    plan    tree    ${plan_id}    --format    json
+    Should Not Contain    ${r_tree_post.stdout}${r_tree_post.stderr}    Traceback
+    Should Not Be Empty    ${r_tree_post.stdout}
+    # The corrected decision should now be marked superseded in the tree
+    Should Contain    ${r_tree_post.stdout}    "superseded": true
+    ...    Plan tree after correction does not show any superseded decisions — correction may not have taken effect
+    Log    Post-correction tree JSON: ${r_tree_post.stdout}
+
     # ── 12. Plan diff — show changes ──
     ${r_diff}=    Run CleverAgents Command
     ...    plan    diff    ${plan_id}    --format    plain
@@ -242,14 +254,18 @@ M3 Decision Recording And Tree Visualization
     Should Not Contain    ${r_apply.stdout}${r_apply.stderr}    Traceback
     Should Not Contain    ${r_apply.stdout}${r_apply.stderr}    INTERNAL
 
-    # ── 14. Final plan status — M1: remove generic 'phase', use specific terms ──
+    # ── 14. Final plan status — P1-D: use JSON for structured terminal check ──
     ${r_status}=    Run CleverAgents Command
-    ...    plan    status    ${plan_id}    --format    plain
+    ...    plan    status    ${plan_id}    --format    json
     Should Not Be Empty    ${r_status.stdout}
     Should Not Contain    ${r_status.stdout}${r_status.stderr}    Traceback
-    # H4/H5/M1: Use $variable syntax, check specific terminal indicators
-    Should Be True
-    ...    'applied' in $r_status.stdout.lower() or 'complete' in $r_status.stdout.lower() or 'apply' in $r_status.stdout.lower()
-    ...    Final status does not indicate terminal phase (expected applied/complete/apply).
-    Log    Final plan status: ${r_status.stdout}
+    Log    Final plan status JSON: ${r_status.stdout}
+    # Verify plan reached the Apply phase
+    Should Match Regexp    ${r_status.stdout}
+    ...    "phase"\\s*:\\s*"apply"
+    ...    Plan did not reach Apply phase after lifecycle-apply
+    # Verify processing state is terminal or queued (lifecycle-apply is async)
+    Should Match Regexp    ${r_status.stdout}
+    ...    "processing_state"\\s*:\\s*"(queued|processing|complete|applied)"
+    ...    Plan is in unexpected processing state after lifecycle-apply
     Log    M3 acceptance E2E test completed successfully
-- 
2.52.0


From a4d543428c542ba5dca7e332e9b6eb8c5b2394ff Mon Sep 17 00:00:00 2001
From: Hamza Khyari <hamza.khyari@cleverthis.com>
Date: Thu, 2 Apr 2026 00:26:27 +0000
Subject: [PATCH 8/8] =?UTF-8?q?fix(test):=20restore=20tdd=5Fexpected=5Ffai?=
 =?UTF-8?q?l=20=E2=80=94=20bug=20#1022=20fix=20not=20yet=20on=20master?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Issue #1022 was closed but the actual invariant persistence fix has
not been merged to master.  invariant list still returns 'no
invariants found' between CLI invocations.  Restore tdd_expected_fail
with correct tdd_issue tags until the fix lands.

Ref: #743
---
 robot/e2e/m3_acceptance.robot | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/robot/e2e/m3_acceptance.robot b/robot/e2e/m3_acceptance.robot
index f46977de1..b00649c0b 100644
--- a/robot/e2e/m3_acceptance.robot
+++ b/robot/e2e/m3_acceptance.robot
@@ -24,7 +24,7 @@ M3 Decision Recording And Tree Visualization
     ...
     ...                Exercises the full M3 feature set through the CLI with real
     ...                LLM API keys. Validates structural output, not exact text.
-    [Tags]    E2E
+    [Tags]    E2E    tdd_issue    tdd_issue_1022    tdd_expected_fail
     [Timeout]    15 minutes
     Skip If No LLM Keys
 
-- 
2.52.0