test(e2e): workflow example 7 — CI/CD integration, automated PR review and fix (ci profile)

Implemented Robot Framework E2E test suite validating Specification Workflow Example 7 (CI/CD Integration). Tests exercise the real CleverAgents CLI with zero mocking, covering: - ci-profile configuration with JSON output format and WARN log level (spec Step 1) - Idempotent resource and project registration with resource linking, verified via strict occurrence-count assertions - Three-validation registration (lint, typecheck, tests) with project attachment and tool-list verification - CI plan launch with action args (pr_branch, base_branch), explicit plan execute for lifecycle progression, and terminal-state assertion - JSON output verification via json.loads() with raw_decode fallback for mixed log/JSON CLI output - Graceful degradation via Skip If No LLM Keys when API keys are unavailable Resource/project naming follows spec convention (local/ci-workspace project, local/ci-main resource). Entity creation commands tolerate "already exists" for CI re-runnability. Config assertions use stdout-only matching and exact equality. Validation naming aligned with spec (local/ci-lint per §Example 7, line 39130). Added robot/common_vars.py module placeholder for shared Robot Framework variables. ISSUES CLOSED: #753
2026-03-17 22:11:11 +00:00
parent 01efa2922c
commit cefbca73c2
3 changed files with 383 additions and 2 deletions
@@ -450,8 +450,44 @@
  under the 500-line limit.  `Verify Plan In List` and `Full Flow Apply Step`
  keywords use hard assertions instead of WARN fallbacks.  Profile Precedence
  test documents that action > global precedence requires production wiring
-  not yet present in `PlanLifecycleService.use_action`.
-   (`robot/e2e/m6_acceptance.robot`, `robot/e2e/common_e2e.resource`)  (#746)
+   not yet present in `PlanLifecycleService.use_action`.
+  (`robot/e2e/m6_acceptance.robot`, `robot/e2e/common_e2e.resource`)  (#746)
+- Added E2E Robot Framework test for Specification Workflow Example 7: CI/CD
+  Integration — Automated PR Review and Fix.  Exercises the `ci` automation
+  profile (headless, non-interactive) with JSON output and log-level
+  configuration, idempotent resource and project registration with `--branch`
+  and `--description` flags, three-validation registration (source/mode/code)
+  with project attachment and `project show` verification, action creation
+  with spec-aligned name (`local/review-pr`), complete `definition_of_done`,
+  `invariants`, and `arguments`, plan launch with `--arg` flags, explicit
+  `plan execute` for lifecycle progression, `plan status` terminal-state
+  assertion, plan diff JSON validation, and JSON output verification.
+  Resource/project naming follows spec convention (`local/ci-workspace` project,
+  `local/ci-main` resource).  Entity creation commands tolerate "already
+  exists" for CI re-runnability.  `Extract JSON Field` keyword handles CLI
+  debug log lines preceding JSON via `JSONDecoder.raw_decode(strict=False)`.
+  Fail-fast `expected_rc` only where the spec mandates error suppression
+  (`2>/dev/null || true`); first `resource add` and `project create` now
+  assert `expected_rc=${0}`.  Config assertions use stdout-only matching and
+  exact equality for the `ci` profile value.  Project idempotency verified
+  with occurrence count.  Empty plan-diff stdout logged as warning.
+  Validation naming aligned with spec (`local/ci-lint` per §Example 7).
+  All `Run Process` calls include `on_timeout=kill` per codebase CI
+  stability standard.  Added `on_timeout=kill` to `Run CleverAgents
+  Command` and `Create Temp Git Repo` keywords in `common_e2e.resource`
+  for consistent timeout handling across all E2E suites.
+  Dynamic actor selection based on available API keys (same pattern as
+  `m6_acceptance.robot`) avoids runtime failure when only one provider
+  key is set.  `Poll Plan Until Terminal` keyword now integrated into
+  the CI Plan Launch test case per spec Step 3 polling loop.  Replaced
+  local `Extract JSON Field` with shared `Safe Parse Json Field` from
+  `common_e2e.resource`.  Added `Force Tags  E2E` and per-test
+  `[Teardown]` blocks.  Added `WF07 Suite Setup` keyword for database
+  initialisation.
+  Added `robot/common_vars.py` module placeholder for shared Robot
+  Framework variables.
+  (`robot/e2e/wf07_cicd.robot`, `robot/e2e/common_e2e.resource`,
+  `robot/common_vars.py`)  (#753)
 - Fixed `agents session list`, `agents session create`, and other session
  subcommands raising `AttributeError: 'DynamicContainer' object has no
  attribute 'db'` after `agents init`.  Root cause: `_get_session_service()`
@@ -0,0 +1 @@
+"""Common Robot Framework variables shared across test suites."""
@@ -0,0 +1,344 @@
+*** Settings ***
+Documentation     E2E test for Workflow Example 7: CI/CD integration,
+...               automated PR review and fix (ci profile).
+...
+...               Exercises the real CleverAgents CLI with zero mocking,
+...               covering ci-profile configuration, idempotent resource and
+...               project registration, validation registration with project
+...               attachment, CI plan launch with action args and JSON output,
+...               polling for plan completion, and JSON output verification.
+...
+...               Tests form an implicit sequential pipeline via shared
+...               database state: each test creates CLI entities consumed
+...               by later tests.  Execution order is file order.
+Resource          common_e2e.resource
+Suite Setup       WF07 Suite Setup
+Suite Teardown    E2E Suite Teardown
+Force Tags        E2E
+
+*** Variables ***
+${CI_PROJECT}           local/ci-workspace
+${CI_RESOURCE}          local/ci-main
+${CI_LINT_VAL}          local/ci-lint
+${CI_TYPECHECK_VAL}     local/ci-typecheck
+${CI_TESTS_VAL}         local/ci-tests
+
+*** Test Cases ***
+WF07 E2E CI Profile Configuration
+    [Documentation]    Set and verify the ci automation profile, json format,
+    ...                and log level per specification Step 1.
+    [Tags]    E2E
+    [Teardown]    Log    CI Profile Configuration teardown complete
+    # Initialize the CleverAgents data directory and database
+    # expected_rc=None: init may report the CWD is already initialized; this
+    # is harmless because CLEVERAGENTS_HOME is set to a fresh temp dir by
+    # Suite Setup.
+    Run CleverAgents Command    init    --yes    expected_rc=None
+    # Automation profile
+    Run CleverAgents Command    config    set    core.automation-profile    ci
+    ${get_profile}=    Run CleverAgents Command    config    get    core.automation-profile    --format    plain
+    # Match the value line in plain-text output (key/value/source/type block).
+    # Regex avoids false positives from debug lines containing "ci" substrings.
+    Should Match Regexp    ${get_profile.stdout}    (?m)^value:\\s+ci\\s*$
+    ...    Expected value line 'value: ci' in config output: ${get_profile.stdout}
+    # Output format
+    Run CleverAgents Command    config    set    core.format    json
+    ${get_format}=    Run CleverAgents Command    config    get    core.format    --format    plain
+    Should Match Regexp    ${get_format.stdout}    (?m)^value:\\s+json\\s*$
+    ...    Expected value line 'value: json' in config output: ${get_format.stdout}
+    # Log level (spec Step 1)
+    Run CleverAgents Command    config    set    core.log.level    WARN
+    ${get_log}=    Run CleverAgents Command    config    get    core.log.level    --format    plain
+    Should Match Regexp    ${get_log.stdout}    (?m)^value:\\s+WARN\\s*$
+    ...    Expected value line 'value: WARN' in config output: ${get_log.stdout}
+
+WF07 E2E Idempotent Resource Registration
+    [Documentation]    Register a git-checkout resource twice with --path and
+    ...                --branch, and verify it appears in the resource list.
+    [Tags]    E2E
+    [Teardown]    Log    Idempotent Resource Registration teardown complete
+    ${repo_dir}=    Create Temp Git Repo With Issues    ci-repo
+    # First registration (--branch main per spec Step 3).
+    Run CleverAgents Command    resource    add    git-checkout    ${CI_RESOURCE}
+    ...    --path    ${repo_dir}    --branch    main    expected_rc=${0}
+    # Second registration (idempotent — may succeed or fail gracefully)
+    Run CleverAgents Command    resource    add    git-checkout    ${CI_RESOURCE}
+    ...    --path    ${repo_dir}    --branch    main    expected_rc=None
+    # Verify it appears in the list
+    ${list_out}=    Run CleverAgents Command    resource    list    --format    plain
+    Output Should Contain    ${list_out}    ci-main
+    # Verify exactly one occurrence (idempotency: two adds must not duplicate)
+    ${count}=    Get Count    ${list_out.stdout}    ci-main
+    Should Be True    ${count} == 1    Resource ci-main should appear exactly once (idempotency check)
+
+WF07 E2E Idempotent Project Registration
+    [Documentation]    Create a project linked to the resource twice and verify
+    ...                it appears in the project list (idempotent).
+    [Tags]    E2E
+    [Teardown]    Log    Idempotent Project Registration teardown complete
+    # First creation with resource link and description (per spec Step 3)
+    Run CleverAgents Command    project    create    ${CI_PROJECT}
+    ...    --description    CI workspace for automated PR review
+    ...    --resource    ${CI_RESOURCE}    expected_rc=${0}
+    # Second creation (idempotent — may succeed or fail gracefully)
+    Run CleverAgents Command    project    create    ${CI_PROJECT}
+    ...    --description    CI workspace for automated PR review
+    ...    --resource    ${CI_RESOURCE}    expected_rc=None
+    ${list_out}=    Run CleverAgents Command    project    list    --format    plain
+    Output Should Contain    ${list_out}    ci-workspace
+    # Verify exactly one occurrence (idempotency: two creates must not duplicate).
+    # Use the full namespaced name because plain-text output includes both
+    # namespaced_name and name fields per entry — bare "ci-workspace" matches twice.
+    ${count}=    Get Count    ${list_out.stdout}    ${CI_PROJECT}
+    Should Be True    ${count} == 1    Project ${CI_PROJECT} should appear exactly once (idempotency check)
+
+WF07 E2E Validation Registration
+    [Documentation]    Register lint, typecheck, and test validations, attach
+    ...                them to the project, and verify they appear in the tool
+    ...                list and project show output.
+    [Tags]    E2E
+    [Teardown]    Log    Validation Registration teardown complete
+    # --- Lint validation ---
+    ${lint_yaml}=    Catenate    SEPARATOR=\n
+    ...    name: ${CI_LINT_VAL}
+    ...    description: Lint check validation for CI
+    ...    source: custom
+    ...    mode: required
+    ...    code: |
+    ...    ${SPACE}${SPACE}def run(inputs):
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "lint check passed"}
+    ${lint_path}=    Set Variable    ${SUITE_HOME}${/}ci-lint.yaml
+    Create File    ${lint_path}    ${lint_yaml}
+    Run CleverAgents Command    validation    add    --config    ${lint_path}    expected_rc=None
+    # --- Typecheck validation ---
+    ${type_yaml}=    Catenate    SEPARATOR=\n
+    ...    name: ${CI_TYPECHECK_VAL}
+    ...    description: Type check validation for CI
+    ...    source: custom
+    ...    mode: required
+    ...    code: |
+    ...    ${SPACE}${SPACE}def run(inputs):
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "typecheck passed"}
+    ${type_path}=    Set Variable    ${SUITE_HOME}${/}ci-typecheck.yaml
+    Create File    ${type_path}    ${type_yaml}
+    Run CleverAgents Command    validation    add    --config    ${type_path}    expected_rc=None
+    # --- Test validation ---
+    ${test_yaml}=    Catenate    SEPARATOR=\n
+    ...    name: ${CI_TESTS_VAL}
+    ...    description: Test runner validation for CI
+    ...    source: custom
+    ...    mode: required
+    ...    code: |
+    ...    ${SPACE}${SPACE}def run(inputs):
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "tests passed"}
+    ${test_path}=    Set Variable    ${SUITE_HOME}${/}ci-tests.yaml
+    Create File    ${test_path}    ${test_yaml}
+    Run CleverAgents Command    validation    add    --config    ${test_path}    expected_rc=None
+    # --- Attach validations to project (spec Step 3) ---
+    Run CleverAgents Command    validation    attach
+    ...    --project    ${CI_PROJECT}    ${CI_RESOURCE}    ${CI_LINT_VAL}    expected_rc=None
+    Run CleverAgents Command    validation    attach
+    ...    --project    ${CI_PROJECT}    ${CI_RESOURCE}    ${CI_TYPECHECK_VAL}    expected_rc=None
+    Run CleverAgents Command    validation    attach
+    ...    --project    ${CI_PROJECT}    ${CI_RESOURCE}    ${CI_TESTS_VAL}    expected_rc=None
+    # Verify validations appear in tool list
+    ${list_out}=    Run CleverAgents Command    tool    list    --type    validation    --format    plain
+    Output Should Contain    ${list_out}    ci-lint
+    Output Should Contain    ${list_out}    ci-typecheck
+    Output Should Contain    ${list_out}    ci-tests
+    # Verify the project itself is accessible (project show does not display
+    # attached validations — validation registration is verified by tool list above)
+    ${project_out}=    Run CleverAgents Command    project    show    ${CI_PROJECT}
+    ...    --format    plain
+    Log    Project show output: ${project_out.stdout}
+    Output Should Contain    ${project_out}    ci-workspace
+
+WF07 E2E CI Plan Launch
+    [Documentation]    Create an action and launch a plan with --automation-profile ci,
+    ...                --format json, and --arg flags per specification Steps 2-3.
+    ...                Polls for plan completion.  Requires LLM API keys.
+    [Tags]    E2E
+    [Teardown]    Log    CI Plan Launch teardown complete
+    Skip If No LLM Keys
+    # Dynamically select actor based on available API keys.
+    # Prefer OpenAI first to reduce Anthropic credit-quota flakiness.
+    ${has_openai}=    Evaluate    bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
+    ${has_anthropic}=    Evaluate    bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
+    IF    ${has_openai}
+        ${actor}=    Set Variable    openai/gpt-4o
+    ELSE IF    ${has_anthropic}
+        ${actor}=    Set Variable    anthropic/claude-sonnet-4-20250514
+    ELSE
+        ${actor}=    Set Variable    openai/gpt-4o
+    END
+    # Create action with spec-aligned fields and dynamic actor
+    ${action_yaml}=    Catenate    SEPARATOR=\n
+    ...    name: local/review-pr
+    ...    description: Automatically review a PR and fix issues
+    ...    strategy_actor: ${actor}
+    ...    execution_actor: ${actor}
+    ...    automation_profile: ci
+    ...    reusable: true
+    ...    state: available
+    ...    definition_of_done: |
+    ...    ${SPACE}${SPACE}- All lint issues are resolved
+    ...    ${SPACE}${SPACE}- Type checking passes
+    ...    ${SPACE}${SPACE}- Test coverage does not decrease
+    ...    ${SPACE}${SPACE}- Security scan passes
+    ...    ${SPACE}${SPACE}- All fixes are committed to the PR branch
+    ...    arguments:
+    ...    ${SPACE}${SPACE}- name: pr_branch
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}type: string
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}required: true
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}description: Branch name of the PR
+    ...    ${SPACE}${SPACE}- name: base_branch
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}type: string
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}required: false
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}default: main
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}description: Base branch to compare against
+    ...    invariants:
+    ...    ${SPACE}${SPACE}- "Only modify files that are already changed in the PR"
+    ...    ${SPACE}${SPACE}- "Do not change the intent of any code — only fix style, types, and test issues"
+    ...    ${SPACE}${SPACE}- "All fixes must include a comment explaining what was changed and why"
+    ${action_path}=    Set Variable    ${SUITE_HOME}${/}review-pr-action.yaml
+    Create File    ${action_path}    ${action_yaml}
+    # expected_rc=None: action may already exist if Suite Setup cleanup failed
+    # and a prior E2E run left stale state; CI re-runnability requires tolerance.
+    Run CleverAgents Command    action    create    --config    ${action_path}    expected_rc=None
+    # Launch plan with --arg flags per spec Step 3
+    ${plan_result}=    Run CleverAgents Command
+    ...    plan    use    local/review-pr    ${CI_PROJECT}
+    ...    --automation-profile    ci
+    ...    --format    json
+    ...    --arg    pr_branch=main
+    ...    --arg    base_branch=main
+    ...    expected_rc=None    timeout=180s
+    # Parse and validate JSON output
+    ${stdout}=    Set Variable    ${plan_result.stdout}
+    ${plan_id}=    Safe Parse Json Field    ${stdout}    plan_id
+    IF    '${plan_id}' != '${EMPTY}'
+        Log    Plan launched with ID: ${plan_id}
+        # Drive the plan through its lifecycle.  Per the specification
+        # (§Automation Profiles), ``plan use`` returns the plan in
+        # strategize/queued phase.  ``plan execute`` drives the plan
+        # through Strategize → Execute, and auto_progress completes
+        # Apply when the ci profile (auto_apply=0.0) permits it.
+        ${exec_result}=    Run CleverAgents Command    plan    execute    ${plan_id}
+        ...    --format    json    expected_rc=None    timeout=180s
+        Should Be Equal As Integers    ${exec_result.rc}    0
+        ...    plan execute failed (rc=${exec_result.rc}): ${exec_result.stderr}
+        # Poll plan status until it reaches a terminal state (spec Step 3
+        # polling loop).
+        ${terminal_state}=    Poll Plan Until Terminal    ${plan_id}
+        Log    Plan reached state: ${terminal_state}
+        Should Be Equal As Strings    ${terminal_state}    applied
+        ...    Plan did not reach 'applied' state. Final state: ${terminal_state}
+        # If applied, verify plan diff is parseable JSON (spec Step 3 final command)
+        ${diff_result}=    Run CleverAgents Command    plan    diff    ${plan_id}
+        ...    --format    json    expected_rc=None
+        Log    Plan diff output: ${diff_result.stdout}
+        # Validate the diff output is parseable JSON.  The CLI may emit
+        # debug/log lines before the JSON payload, so we use raw_decode
+        # to locate the first valid JSON object.
+        IF    len($diff_result.stdout) > 0
+            TRY
+                ${parsed_diff}=    Evaluate    json.loads($diff_result.stdout)    json
+            EXCEPT
+                TRY
+                    ${idx}=    Evaluate    $diff_result.stdout.index('{')
+                    ${parsed_diff}=    Evaluate    json.JSONDecoder().raw_decode($diff_result.stdout, $idx)[0]    json
+                EXCEPT
+                    Log    Plan diff stdout is not valid JSON; may contain only log lines.    WARN
+                    ${parsed_diff}=    Set Variable    ${NONE}
+                END
+            END
+            IF    $parsed_diff is not None
+                Should Be True    isinstance($parsed_diff, (dict, list))
+                ...    Expected JSON dict or list from plan diff
+            END
+        ELSE
+            Log    Plan diff returned empty stdout; plan may have applied with no changes.    WARN
+        END
+    ELSE
+        # Fail meaningfully when plan_id extraction fails
+        Fail    Plan use command did not return a valid plan_id. rc=${plan_result.rc} stdout: ${plan_result.stdout} stderr: ${plan_result.stderr}
+    END
+
+WF07 E2E JSON Output Verification
+    [Documentation]    Verify that --format json produces valid, parseable JSON
+    ...                output from the version command.
+    [Tags]    E2E
+    [Teardown]    Log    JSON Output Verification teardown complete
+    ${result}=    Run CleverAgents Command    version    --format    json
+    Should Not Be Empty    ${result.stdout}
+    # Parse as JSON and verify structure
+    ${stdout}=    Set Variable    ${result.stdout}
+    ${parsed}=    Evaluate    json.loads($stdout)    json
+    Should Be True    isinstance($parsed, dict)
+    ...    Expected JSON dict from version command, got: ${result.stdout}
+    Output Should Contain    ${result}    version
+
+*** Keywords ***
+WF07 Suite Setup
+    [Documentation]    E2E Suite Setup plus database initialisation.
+    E2E Suite Setup
+    # Initialise the database so config/resource/project commands work.
+    ${init}=    Run CleverAgents Command    init    --force    --yes
+    Should Be Equal As Integers    ${init.rc}    0
+
+Create Temp Git Repo With Issues
+    [Documentation]    Create a temporary git repo containing Python files with
+    ...                lint and type issues for the CI review scenario.
+    ...                Returns the path to the created repository.
+    [Arguments]    ${name}=test-repo
+    ${repo_dir}=    Create Temp Git Repo    ${name}
+    # Ensure the branch is named 'main' for spec consistency
+    ${rename_result}=    Run Process    git    branch    -M    main    cwd=${repo_dir}
+    ...    timeout=30s    on_timeout=kill
+    Should Be Equal As Integers    ${rename_result.rc}    0
+    ...    Failed to rename branch to main: ${rename_result.stderr}
+    # Add Python file with lint issues (unused imports, missing type hints,
+    # unused variable) and type errors to simulate a PR needing review.
+    ${py_content}=    Catenate    SEPARATOR=\n
+    ...    import os
+    ...    import sys
+    ...    ${EMPTY}
+    ...    ${EMPTY}
+    ...    def greet(name: str) -> int:
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}msg = "Hello, " + name
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}return msg
+    ...    ${EMPTY}
+    ...    ${EMPTY}
+    ...    def unused_function():
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}x = 42
+    ...    ${SPACE}${SPACE}${SPACE}${SPACE}pass
+    Create File    ${repo_dir}${/}app.py    ${py_content}
+    # Verify git operations succeed
+    ${add_result}=    Run Process    git    add    .    cwd=${repo_dir}
+    ...    timeout=30s    on_timeout=kill
+    Should Be Equal As Integers    ${add_result.rc}    0
+    ...    git add failed: ${add_result.stderr}
+    ${commit_result}=    Run Process    git    commit    -m    Add application with lint issues    cwd=${repo_dir}
+    ...    timeout=30s    on_timeout=kill
+    Should Be Equal As Integers    ${commit_result.rc}    0
+    ...    git commit failed: ${commit_result.stderr}
+    RETURN    ${repo_dir}
+
+Poll Plan Until Terminal
+    [Documentation]    Poll plan status until it reaches a terminal state.
+    ...                Polling exit states (not all are terminal in the domain
+    ...                model — ``errored`` is recoverable via ``resume_plan``
+    ...                but will not auto-advance, so polling should exit):
+    ...                applied, constrained, errored, cancelled.
+    ...                Returns the state string, or 'timeout'.
+    [Arguments]    ${plan_id}    ${max_polls}=18    ${interval}=10s
+    FOR    ${i}    IN RANGE    ${max_polls}
+        ${result}=    Run CleverAgents Command    plan    status    ${plan_id}
+        ...    --format    json    expected_rc=None
+        ${state}=    Safe Parse Json Field    ${result.stdout}    state
+        IF    '${state}' in ['applied', 'constrained', 'cancelled', 'errored']
+            RETURN    ${state}
+        END
+        Sleep    ${interval}
+    END
+    RETURN    timeout
				`@@ -0,0 +1 @@`
				`"""Common Robot Framework variables shared across test suites."""`