cleveragents-core/robot/e2e/wf07_cicd.robot

*** Settings ***
Documentation     E2E test for Workflow Example 7: CI/CD integration,
...               automated PR review and fix (ci profile).
...
...               Exercises the real CleverAgents CLI with zero mocking,
...               covering ci-profile configuration, idempotent resource and
...               project registration, validation registration with project
...               attachment, CI plan launch with action args and JSON output,
...               polling for plan completion, and JSON output verification.
...
...               Tests form an implicit sequential pipeline via shared
...               database state: each test creates CLI entities consumed
...               by later tests.  Execution order is file order.
Resource          common_e2e.resource
Suite Setup       WF07 Suite Setup
Suite Teardown    E2E Suite Teardown
Force Tags        E2E

*** Variables ***
${CI_PROJECT}           local/ci-workspace
${CI_RESOURCE}          local/ci-main
${CI_LINT_VAL}          local/ci-lint
${CI_TYPECHECK_VAL}     local/ci-typecheck
${CI_TESTS_VAL}         local/ci-tests

*** Test Cases ***
WF07 E2E CI Profile Configuration
    [Documentation]    Set and verify the ci automation profile, json format,
    ...                and log level per specification Step 1.
    [Tags]    E2E
    [Teardown]    Log    CI Profile Configuration teardown complete
    # Initialize the CleverAgents data directory and database
    # expected_rc=None: init may report the CWD is already initialized; this
    # is harmless because CLEVERAGENTS_HOME is set to a fresh temp dir by
    # Suite Setup.
    Run CleverAgents Command    init    --yes    expected_rc=None
    # Automation profile
    Run CleverAgents Command    config    set    core.automation-profile    ci
    ${get_profile}=    Run CleverAgents Command    config    get    core.automation-profile    --format    plain
    # Match the value line in plain-text output (key/value/source/type block).
    # Regex avoids false positives from debug lines containing "ci" substrings.
    Should Match Regexp    ${get_profile.stdout}    (?m)^value:\\s+ci\\s*$
    ...    Expected value line 'value: ci' in config output: ${get_profile.stdout}
    # Output format
    Run CleverAgents Command    config    set    core.format    json
    ${get_format}=    Run CleverAgents Command    config    get    core.format    --format    plain
    Should Match Regexp    ${get_format.stdout}    (?m)^value:\\s+json\\s*$
    ...    Expected value line 'value: json' in config output: ${get_format.stdout}
    # Log level (spec Step 1)
    Run CleverAgents Command    config    set    core.log.level    WARN
    ${get_log}=    Run CleverAgents Command    config    get    core.log.level    --format    plain
    Should Match Regexp    ${get_log.stdout}    (?m)^value:\\s+WARN\\s*$
    ...    Expected value line 'value: WARN' in config output: ${get_log.stdout}

WF07 E2E Idempotent Resource Registration
    [Documentation]    Register a git-checkout resource twice with --path and
    ...                --branch, and verify it appears in the resource list.
    [Tags]    E2E
    [Teardown]    Log    Idempotent Resource Registration teardown complete
    ${repo_dir}=    Create Temp Git Repo With Issues    ci-repo
    # First registration (--branch main per spec Step 3).
    Run CleverAgents Command    resource    add    git-checkout    ${CI_RESOURCE}
    ...    --path    ${repo_dir}    --branch    main    expected_rc=${0}
    # Second registration (idempotent — may succeed or fail gracefully)
    Run CleverAgents Command    resource    add    git-checkout    ${CI_RESOURCE}
    ...    --path    ${repo_dir}    --branch    main    expected_rc=None
    # Verify it appears in the list
    ${list_out}=    Run CleverAgents Command    resource    list    --format    plain
    Output Should Contain    ${list_out}    ci-main
    # Verify exactly one occurrence (idempotency: two adds must not duplicate)
    ${count}=    Get Count    ${list_out.stdout}    ci-main
    Should Be True    ${count} == 1    Resource ci-main should appear exactly once (idempotency check)

WF07 E2E Idempotent Project Registration
    [Documentation]    Create a project linked to the resource twice and verify
    ...                it appears in the project list (idempotent).
    [Tags]    E2E
    [Teardown]    Log    Idempotent Project Registration teardown complete
    # First creation with resource link and description (per spec Step 3)
    Run CleverAgents Command    project    create    ${CI_PROJECT}
    ...    --description    CI workspace for automated PR review
    ...    --resource    ${CI_RESOURCE}    expected_rc=${0}
    # Second creation (idempotent — may succeed or fail gracefully)
    Run CleverAgents Command    project    create    ${CI_PROJECT}
    ...    --description    CI workspace for automated PR review
    ...    --resource    ${CI_RESOURCE}    expected_rc=None
    ${list_out}=    Run CleverAgents Command    project    list    --format    plain
    Output Should Contain    ${list_out}    ci-workspace
    # Verify exactly one occurrence (idempotency: two creates must not duplicate).
    # Use the full namespaced name because plain-text output includes both
    # namespaced_name and name fields per entry — bare "ci-workspace" matches twice.
    ${count}=    Get Count    ${list_out.stdout}    ${CI_PROJECT}
    Should Be True    ${count} == 1    Project ${CI_PROJECT} should appear exactly once (idempotency check)

WF07 E2E Validation Registration
    [Documentation]    Register lint, typecheck, and test validations, attach
    ...                them to the project, and verify they appear in the tool
    ...                list and project show output.
    [Tags]    E2E
    [Teardown]    Log    Validation Registration teardown complete
    # --- Lint validation ---
    ${lint_yaml}=    Catenate    SEPARATOR=\n
    ...    name: ${CI_LINT_VAL}
    ...    description: Lint check validation for CI
    ...    source: custom
    ...    mode: required
    ...    code: |
    ...    ${SPACE}${SPACE}def run(inputs):
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "lint check passed"}
    ${lint_path}=    Set Variable    ${SUITE_HOME}${/}ci-lint.yaml
    Create File    ${lint_path}    ${lint_yaml}
    Run CleverAgents Command    validation    add    --config    ${lint_path}    expected_rc=None
    # --- Typecheck validation ---
    ${type_yaml}=    Catenate    SEPARATOR=\n
    ...    name: ${CI_TYPECHECK_VAL}
    ...    description: Type check validation for CI
    ...    source: custom
    ...    mode: required
    ...    code: |
    ...    ${SPACE}${SPACE}def run(inputs):
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "typecheck passed"}
    ${type_path}=    Set Variable    ${SUITE_HOME}${/}ci-typecheck.yaml
    Create File    ${type_path}    ${type_yaml}
    Run CleverAgents Command    validation    add    --config    ${type_path}    expected_rc=None
    # --- Test validation ---
    ${test_yaml}=    Catenate    SEPARATOR=\n
    ...    name: ${CI_TESTS_VAL}
    ...    description: Test runner validation for CI
    ...    source: custom
    ...    mode: required
    ...    code: |
    ...    ${SPACE}${SPACE}def run(inputs):
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "tests passed"}
    ${test_path}=    Set Variable    ${SUITE_HOME}${/}ci-tests.yaml
    Create File    ${test_path}    ${test_yaml}
    Run CleverAgents Command    validation    add    --config    ${test_path}    expected_rc=None
    # --- Attach validations to project (spec Step 3) ---
    Run CleverAgents Command    validation    attach
    ...    --project    ${CI_PROJECT}    ${CI_RESOURCE}    ${CI_LINT_VAL}    expected_rc=None
    Run CleverAgents Command    validation    attach
    ...    --project    ${CI_PROJECT}    ${CI_RESOURCE}    ${CI_TYPECHECK_VAL}    expected_rc=None
    Run CleverAgents Command    validation    attach
    ...    --project    ${CI_PROJECT}    ${CI_RESOURCE}    ${CI_TESTS_VAL}    expected_rc=None
    # Verify validations appear in tool list
    ${list_out}=    Run CleverAgents Command    tool    list    --type    validation    --format    plain
    Output Should Contain    ${list_out}    ci-lint
    Output Should Contain    ${list_out}    ci-typecheck
    Output Should Contain    ${list_out}    ci-tests
    # Verify the project itself is accessible (project show does not display
    # attached validations — validation registration is verified by tool list above)
    ${project_out}=    Run CleverAgents Command    project    show    ${CI_PROJECT}
    ...    --format    plain
    Log    Project show output: ${project_out.stdout}
    Output Should Contain    ${project_out}    ci-workspace

WF07 E2E CI Plan Launch
    [Documentation]    Create an action and launch a plan with --automation-profile ci,
    ...                --format json, and --arg flags per specification Steps 2-3.
    ...                Polls for plan completion.  Requires LLM API keys.
    [Tags]    E2E
    [Teardown]    Log    CI Plan Launch teardown complete
    Skip If No LLM Keys
    # Dynamically select actor based on available API keys.
    # Prefer OpenAI first to reduce Anthropic credit-quota flakiness.
    ${has_openai}=    Evaluate    bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
    ${has_anthropic}=    Evaluate    bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
    IF    ${has_openai}
        ${actor}=    Set Variable    openai/gpt-4o
    ELSE IF    ${has_anthropic}
        ${actor}=    Set Variable    anthropic/claude-sonnet-4-20250514
    ELSE
        ${actor}=    Set Variable    openai/gpt-4o
    END
    # Create action with spec-aligned fields and dynamic actor
    ${action_yaml}=    Catenate    SEPARATOR=\n
    ...    name: local/review-pr
    ...    description: Automatically review a PR and fix issues
    ...    strategy_actor: ${actor}
    ...    execution_actor: ${actor}
    ...    automation_profile: ci
    ...    reusable: true
    ...    state: available
    ...    definition_of_done: |
    ...    ${SPACE}${SPACE}- All lint issues are resolved
    ...    ${SPACE}${SPACE}- Type checking passes
    ...    ${SPACE}${SPACE}- Test coverage does not decrease
    ...    ${SPACE}${SPACE}- Security scan passes
    ...    ${SPACE}${SPACE}- All fixes are committed to the PR branch
    ...    arguments:
    ...    ${SPACE}${SPACE}- name: pr_branch
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}type: string
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}required: true
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}description: Branch name of the PR
    ...    ${SPACE}${SPACE}- name: base_branch
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}type: string
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}required: false
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}default: main
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}description: Base branch to compare against
    ...    invariants:
    ...    ${SPACE}${SPACE}- "Only modify files that are already changed in the PR"
    ...    ${SPACE}${SPACE}- "Do not change the intent of any code — only fix style, types, and test issues"
    ...    ${SPACE}${SPACE}- "All fixes must include a comment explaining what was changed and why"
    ${action_path}=    Set Variable    ${SUITE_HOME}${/}review-pr-action.yaml
    Create File    ${action_path}    ${action_yaml}
    # expected_rc=None: action may already exist if Suite Setup cleanup failed
    # and a prior E2E run left stale state; CI re-runnability requires tolerance.
    Run CleverAgents Command    action    create    --config    ${action_path}    expected_rc=None
    # Launch plan with --arg flags per spec Step 3
    ${plan_result}=    Run CleverAgents Command
    ...    plan    use    local/review-pr    ${CI_PROJECT}
    ...    --automation-profile    ci
    ...    --format    json
    ...    --arg    pr_branch=main
    ...    --arg    base_branch=main
    ...    expected_rc=None    timeout=180s
    # Parse and validate JSON output
    ${stdout}=    Set Variable    ${plan_result.stdout}
    ${plan_id}=    Safe Parse Json Field    ${stdout}    plan_id
    IF    '${plan_id}' != '${EMPTY}'
        Log    Plan launched with ID: ${plan_id}
        # Drive the plan through its lifecycle.  Per the specification
        # (§Automation Profiles), ``plan use`` returns the plan in
        # strategize/queued phase.  ``plan execute`` drives the plan
        # through Strategize → Execute, and auto_progress completes
        # Apply when the ci profile (auto_apply=0.0) permits it.
        ${exec_result}=    Run CleverAgents Command    plan    execute    ${plan_id}
        ...    --format    json    expected_rc=None    timeout=180s
        Should Be Equal As Integers    ${exec_result.rc}    0
        ...    plan execute failed (rc=${exec_result.rc}): ${exec_result.stderr}
        # Poll plan status until it reaches a terminal state (spec Step 3
        # polling loop).
        ${terminal_state}=    Poll Plan Until Terminal    ${plan_id}
        Log    Plan reached state: ${terminal_state}
        Should Be Equal As Strings    ${terminal_state}    applied
        ...    Plan did not reach 'applied' state. Final state: ${terminal_state}
        # If applied, verify plan diff is parseable JSON (spec Step 3 final command)
        ${diff_result}=    Run CleverAgents Command    plan    diff    ${plan_id}
        ...    --format    json    expected_rc=None
        Log    Plan diff output: ${diff_result.stdout}
        # Validate the diff output is parseable JSON.  The CLI may emit
        # debug/log lines before the JSON payload, so we use raw_decode
        # to locate the first valid JSON object.
        IF    len($diff_result.stdout) > 0
            TRY
                ${parsed_diff}=    Evaluate    json.loads($diff_result.stdout)    json
            EXCEPT
                TRY
                    ${idx}=    Evaluate    $diff_result.stdout.index('{')
                    ${parsed_diff}=    Evaluate    json.JSONDecoder().raw_decode($diff_result.stdout, $idx)[0]    json
                EXCEPT
                    Log    Plan diff stdout is not valid JSON; may contain only log lines.    WARN
                    ${parsed_diff}=    Set Variable    ${NONE}
                END
            END
            IF    $parsed_diff is not None
                Should Be True    isinstance($parsed_diff, (dict, list))
                ...    Expected JSON dict or list from plan diff
            END
        ELSE
            Log    Plan diff returned empty stdout; plan may have applied with no changes.    WARN
        END
    ELSE
        # Fail meaningfully when plan_id extraction fails
        Fail    Plan use command did not return a valid plan_id. rc=${plan_result.rc} stdout: ${plan_result.stdout} stderr: ${plan_result.stderr}
    END

WF07 E2E JSON Output Verification
    [Documentation]    Verify that --format json produces valid, parseable JSON
    ...                output from the version command.
    [Tags]    E2E
    [Teardown]    Log    JSON Output Verification teardown complete
    ${result}=    Run CleverAgents Command    version    --format    json
    Should Not Be Empty    ${result.stdout}
    # Parse as JSON and verify structure
    ${stdout}=    Set Variable    ${result.stdout}
    ${parsed}=    Evaluate    json.loads($stdout)    json
    Should Be True    isinstance($parsed, dict)
    ...    Expected JSON dict from version command, got: ${result.stdout}
    Output Should Contain    ${result}    version

*** Keywords ***
WF07 Suite Setup
    [Documentation]    E2E Suite Setup plus database initialisation.
    E2E Suite Setup
    # Initialise the database so config/resource/project commands work.
    ${init}=    Run CleverAgents Command    init    --force    --yes
    Should Be Equal As Integers    ${init.rc}    0

Create Temp Git Repo With Issues
    [Documentation]    Create a temporary git repo containing Python files with
    ...                lint and type issues for the CI review scenario.
    ...                Returns the path to the created repository.
    [Arguments]    ${name}=test-repo
    ${repo_dir}=    Create Temp Git Repo    ${name}
    # Ensure the branch is named 'main' for spec consistency
    ${rename_result}=    Run Process    git    branch    -M    main    cwd=${repo_dir}
    ...    timeout=30s    on_timeout=kill
    Should Be Equal As Integers    ${rename_result.rc}    0
    ...    Failed to rename branch to main: ${rename_result.stderr}
    # Add Python file with lint issues (unused imports, missing type hints,
    # unused variable) and type errors to simulate a PR needing review.
    ${py_content}=    Catenate    SEPARATOR=\n
    ...    import os
    ...    import sys
    ...    ${EMPTY}
    ...    ${EMPTY}
    ...    def greet(name: str) -> int:
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}msg = "Hello, " + name
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}return msg
    ...    ${EMPTY}
    ...    ${EMPTY}
    ...    def unused_function():
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}x = 42
    ...    ${SPACE}${SPACE}${SPACE}${SPACE}pass
    Create File    ${repo_dir}${/}app.py    ${py_content}
    # Verify git operations succeed
    ${add_result}=    Run Process    git    add    .    cwd=${repo_dir}
    ...    timeout=30s    on_timeout=kill
    Should Be Equal As Integers    ${add_result.rc}    0
    ...    git add failed: ${add_result.stderr}
    ${commit_result}=    Run Process    git    commit    -m    Add application with lint issues    cwd=${repo_dir}
    ...    timeout=30s    on_timeout=kill
    Should Be Equal As Integers    ${commit_result.rc}    0
    ...    git commit failed: ${commit_result.stderr}
    RETURN    ${repo_dir}

Poll Plan Until Terminal
    [Documentation]    Poll plan status until it reaches a terminal state.
    ...                Polling exit states (not all are terminal in the domain
    ...                model — ``errored`` is recoverable via ``resume_plan``
    ...                but will not auto-advance, so polling should exit):
    ...                applied, constrained, errored, cancelled.
    ...                Returns the state string, or 'timeout'.
    [Arguments]    ${plan_id}    ${max_polls}=18    ${interval}=10s
    FOR    ${i}    IN RANGE    ${max_polls}
        ${result}=    Run CleverAgents Command    plan    status    ${plan_id}
        ...    --format    json    expected_rc=None
        ${state}=    Safe Parse Json Field    ${result.stdout}    state
        IF    '${state}' in ['applied', 'constrained', 'cancelled', 'errored']
            RETURN    ${state}
        END
        Sleep    ${interval}
    END
    RETURN    timeout