cleveragents-core/robot/e2e/m1_acceptance.robot

*** Settings ***
Documentation     E2E acceptance test for M1 — minimal plan execution flow.
...
...               Exercises the complete M1 (v3.0.0) milestone success criteria
...               with **zero mocking**: real CLI invocations, real LLM API keys,
...               and real subprocess execution against a temporary git repository.
...
...               Flow: action create → resource add → project create → plan use
...               → plan execute (strategize) → plan execute (execute) → plan diff
...               → plan apply.
...
...               Tagged ``tdd_expected_fail`` because ``_get_plan_executor()``
...               does not pass ``sandbox_root`` to ``PlanExecutor``, so
...               ``LLMExecuteActor._write_to_sandbox()`` is never called and
...               LLM-generated files (HELLO.md) are silently discarded.
...               See `#1313 <https://git.cleverthis.com/cleveragents/cleveragents-core/issues/1313>`_.
Resource          common_e2e.resource
Suite Setup       E2E Suite Setup
Suite Teardown    E2E Suite Teardown

*** Test Cases ***
M1 Full Plan Lifecycle
    [Documentation]    Exercise the complete M1 plan lifecycle with real LLM.
    ...
    ...                Creates an action from YAML, registers a git-checkout
    ...                resource, creates a project, and runs the full plan
    ...                lifecycle through apply with post-apply commit verification.
    [Tags]    E2E    tdd_expected_fail    tdd_issue    tdd_issue_1313

    # ── 1. Create a temporary git repo for isolation ──────────────
    ${repo_path}=    Create Temp Git Repo    m1-acceptance-repo

    # ── 2. Write action YAML config ──────────────────────────────
    ${action_yaml}=    Set Variable    ${SUITE_HOME}${/}m1_test_action.yaml
    ${yaml_content}=    Catenate    SEPARATOR=\n
    ...    name: local/test-action
    ...    description: "M1 E2E acceptance test action"
    ...    strategy_actor: openai/gpt-4o-mini
    ...    execution_actor: openai/gpt-4o-mini
    ...    definition_of_done: |
    ...    ${SPACE}${SPACE}Create a file called HELLO.md with a short greeting.
    ...    reusable: true
    ...    read_only: false
    Create File    ${action_yaml}    ${yaml_content}

    # ── 3. Create the action ─────────────────────────────────────
    ${action_result}=    Run CleverAgents Command
    ...    action    create    --config    ${action_yaml}
    ...    --format    plain
    Should Be Equal As Integers    ${action_result.rc}    0
    Output Should Contain    ${action_result}    local/test-action
    Output Should Contain    ${action_result}    M1 E2E acceptance test action
    Output Should Contain    ${action_result}    State: available

    # ── 4. Register git-checkout resource ────────────────────────
    ${resource_result}=    Run CleverAgents Command
    ...    resource    add    git-checkout    local/test-repo
    ...    --path      ${repo_path}    --branch    master
    ...    --format    plain
    Should Be Equal As Integers    ${resource_result.rc}    0
    ...    Resource add failed: ${resource_result.stderr}
    Output Should Contain    ${resource_result}    name: local/test-repo
    Output Should Contain    ${resource_result}    type: git-checkout

    # ── 5. Create project linked to resource ─────────────────────
    ${project_result}=    Run CleverAgents Command
    ...    project    create    --resource    local/test-repo    local/test-project
    ...    --format    plain
    Should Be Equal As Integers    ${project_result.rc}    0
    ...    Project create failed: ${project_result.stderr}
    Output Should Contain    ${project_result}    namespaced_name: local/test-project
    Output Should Contain    ${project_result}    linked_resources:

    # ── 6. Plan use — create plan from action + project ──────────
    ${plan_use_result}=    Run CleverAgents Command
    ...    plan    use    local/test-action    local/test-project
    ...    --format    plain    expected_rc=${0}

    # Extract plan ID from output (ULID pattern: 26 alphanumeric chars)
    ${plan_id}=    Extract Plan Id    ${plan_use_result.stdout}
    Should Not Be Empty    ${plan_id}    Could not extract plan ID from plan use output

    Should Be Equal As Integers    ${plan_use_result.rc}    0
    ...    Plan use failed: ${plan_use_result.stderr}
    Output Should Contain    ${plan_use_result}    phase: strategize
    Output Should Contain    ${plan_use_result}    action_name: local/test-action
    Output Should Contain    ${plan_use_result}    automation_profile: manual
    Output Should Contain    ${plan_use_result}    description: M1 E2E acceptance test action
    Output Should Contain    ${plan_use_result}    definition_of_done: Create a file called HELLO.md with a short greeting.
    Output Should Contain    ${plan_use_result}    strategy_actor: openai/gpt-4o-mini
    Output Should Contain    ${plan_use_result}    execution_actor: openai/gpt-4o-mini
    Output Should Contain    ${plan_use_result}    estimation_actor: None
    Output Should Contain    ${plan_use_result}    invariant_actor: None

    # ── 7. Plan execute — strategize phase ───────────────────────
    ${exec1_result}=    Run CleverAgents Command
    ...    plan    execute    ${plan_id}    --format    plain
    ...    timeout=300s

    Log    Strategize execute rc=${exec1_result.rc}
    # Strategize must succeed (rc=0 enforced)
    ${exec1_combined}=    Set Variable    ${exec1_result.stdout}\n${exec1_result.stderr}
    Log    Strategize output: ${exec1_combined}

    Should Be Equal As Integers    ${exec1_result.rc}    0
    Output Should Contain    ${exec1_result}    action_name: local/test-action
    Output Should Contain    ${exec1_result}    phase: execute
    Output Should Contain    ${exec1_result}    processing_state: complete
    Output Should Contain    ${exec1_result}    project_links:
    Output Should Contain    ${exec1_result}    local/test-project
    Output Should Contain    ${exec1_result}    description: M1 E2E acceptance test action
    Output Should Contain    ${exec1_result}    strategy_actor: openai/gpt-4o-mini
    Output Should Contain    ${exec1_result}    execution_actor: openai/gpt-4o-mini
    Output Should Contain    ${exec1_result}    definition_of_done: Create a file called HELLO.md with a short greeting.

    # ── 8. Plan execute — advance to execute phase ───────────────
    ${exec2_result}=    Run CleverAgents Command
    ...    plan    execute    ${plan_id}    --format    plain
    ...    timeout=300s

    Log    Execute phase rc=${exec2_result.rc}
    ${exec2_combined}=    Set Variable    ${exec2_result.stdout}\n${exec2_result.stderr}
    Log    Execute output: ${exec2_combined}

    # Output matches step 7: both report the plan's current state after transition
    Should Be Equal As Integers    ${exec2_result.rc}    0
    Output Should Contain    ${exec2_result}    action_name: local/test-action
    Output Should Contain    ${exec2_result}    phase: execute
    Output Should Contain    ${exec2_result}    processing_state: complete
    Output Should Contain    ${exec2_result}    project_links:
    Output Should Contain    ${exec2_result}    local/test-project
    Output Should Contain    ${exec2_result}    description: M1 E2E acceptance test action
    Output Should Contain    ${exec2_result}    strategy_actor: openai/gpt-4o-mini
    Output Should Contain    ${exec2_result}    execution_actor: openai/gpt-4o-mini
    Output Should Contain    ${exec2_result}    definition_of_done: Create a file called HELLO.md with a short greeting.

    # ── 9. Plan diff — verify changeset contains HELLO.md ──────────
    ${diff_result}=    Run CleverAgents Command
    ...    plan    diff    ${plan_id}    --format    plain
    ...    timeout=120s

    Log    Diff rc=${diff_result.rc}
    ${diff_combined}=    Set Variable    ${diff_result.stdout}\n${diff_result.stderr}
    Log    Diff output: ${diff_combined}

    Should Be Equal As Integers    ${diff_result.rc}    0
    Should Not Be Empty    ${diff_result.stdout}
    ...    Plan diff produced no output
    # The LLM was asked to create HELLO.md — the diff must show it.
    # This assertion will fail until #1313 is resolved (sandbox_root
    # is not wired into the execute pipeline).
    Output Should Contain    ${diff_result}    HELLO

    # ── 10. Plan apply — apply changes to the repo ───────────────
    ${apply_result}=    Run CleverAgents Command
    ...    plan    apply    --yes    --format    plain    ${plan_id}
    ...    timeout=300s
    Log    Apply rc=${apply_result.rc}
    ${apply_combined}=    Set Variable    ${apply_result.stdout}\n${apply_result.stderr}
    Log    Apply output: ${apply_combined}

    Should Be Equal As Integers    ${apply_result.rc}    0
    Output Should Contain    ${apply_result}    phase: apply
    Output Should Contain    ${apply_result}    processing_state: applied
    Output Should Contain    ${apply_result}    state: applied
    Output Should Contain    ${apply_result}    project_links:
    Output Should Contain    ${apply_result}    - {"project_name": "local/test-project"}
    Output Should Contain    ${apply_result}    arguments:
    Output Should Contain    ${apply_result}    automation_profile: manual
    Output Should Contain    ${apply_result}    action_name: local/test-action
    Output Should Contain    ${apply_result}    description: M1 E2E acceptance test action
    Output Should Contain    ${apply_result}    definition_of_done: Create a file called HELLO.md with a short greeting.
    Output Should Contain    ${apply_result}    strategy_actor: openai/gpt-4o-mini
    Output Should Contain    ${apply_result}    execution_actor: openai/gpt-4o-mini
    Output Should Contain    ${apply_result}    estimation_actor: None
    Output Should Contain    ${apply_result}    invariant_actor: None

    # ── 11. Verify post-apply commit and HELLO.md in target repo ──
    ${git_log}=    Run Process    git    log    --oneline
    ...    cwd=${repo_path}    timeout=60s    on_timeout=kill
    Log    Git log after apply: ${git_log.stdout}
    Should Be Equal As Integers    ${git_log.rc}    0
    # After apply there must be at least 2 commits: the initial commit
    # from Create Temp Git Repo plus the CleverAgents apply commit.
    ${commit_lines}=    Get Line Count    ${git_log.stdout}
    Should Be True    ${commit_lines} >= 2
    ...    Expected at least 2 commits after apply, found ${commit_lines}

    # HELLO.md must exist in the repo after apply.
    # This assertion will fail until #1313 is resolved.
    File Should Exist    ${repo_path}${/}HELLO.md

    # ── 12. Structural validation summary ────────────────────────
    # Action create succeeded (rc=0 verified above)
    # Resource add succeeded (rc=0 verified above)
    # Project create succeeded (rc=0 verified above)
    # Plan use created a plan with a valid ID (verified above)
    # Execute phases completed (rc=0, output validated above)
    # Diff showed HELLO.md changes (validated above)
    # Apply succeeded and produced a commit (validated above)
    # HELLO.md exists in target repo (validated above)
    Log    M1 Full Plan Lifecycle E2E test completed successfully

*** Keywords ***
Extract Plan Id
    [Documentation]    Extract a ULID-style plan ID from command output.
    ...
    ...                Searches for a 26-character alphanumeric ULID pattern
    ...                in the output text. Returns the first match or EMPTY.
    [Arguments]    ${text}
    # ULID pattern: 26 uppercase alphanumeric characters (Crockford Base32)
    ${matches}=    Get Regexp Matches    ${text}    [0-9A-HJ-NP-Z]{26}    flags=IGNORECASE
    ${count}=    Get Length    ${matches}
    ${plan_id}=    Set Variable If    ${count} > 0    ${matches}[0]    ${EMPTY}
    RETURN    ${plan_id}