forked from cleveragents/cleveragents-core
brent.edwards
998aaf25f8
Centralize E2E initialization in common suite setup so DB-dependent CLI commands no longer rely on per-suite or per-test init workarounds. This also sanitizes suite-home names to prevent invalid path-derived initialization failures in isolated Robot runs. ISSUES CLOSED: #1023
338 lines
18 KiB
Plaintext
338 lines
18 KiB
Plaintext
*** Settings ***
|
|
Documentation E2E test for Workflow Example 7: CI/CD integration,
|
|
... automated PR review and fix (ci profile).
|
|
...
|
|
... Exercises the real CleverAgents CLI with zero mocking,
|
|
... covering ci-profile configuration, idempotent resource and
|
|
... project registration, validation registration with project
|
|
... attachment, CI plan launch with action args and JSON output,
|
|
... polling for plan completion, and JSON output verification.
|
|
...
|
|
... Tests form an implicit sequential pipeline via shared
|
|
... database state: each test creates CLI entities consumed
|
|
... by later tests. Execution order is file order.
|
|
Resource common_e2e.resource
|
|
Suite Setup WF07 Suite Setup
|
|
Suite Teardown E2E Suite Teardown
|
|
Force Tags E2E
|
|
|
|
*** Variables ***
|
|
${CI_PROJECT} local/ci-workspace
|
|
${CI_RESOURCE} local/ci-main
|
|
${CI_LINT_VAL} local/ci-lint
|
|
${CI_TYPECHECK_VAL} local/ci-typecheck
|
|
${CI_TESTS_VAL} local/ci-tests
|
|
|
|
*** Test Cases ***
|
|
WF07 E2E CI Profile Configuration
|
|
[Documentation] Set and verify the ci automation profile, json format,
|
|
... and log level per specification Step 1.
|
|
[Tags] E2E
|
|
[Teardown] Log CI Profile Configuration teardown complete
|
|
# Automation profile
|
|
Run CleverAgents Command config set core.automation-profile ci
|
|
${get_profile}= Run CleverAgents Command config get core.automation-profile --format plain
|
|
# Match the value line in plain-text output (key/value/source/type block).
|
|
# Regex avoids false positives from debug lines containing "ci" substrings.
|
|
Should Match Regexp ${get_profile.stdout} (?m)^value:\\s+ci\\s*$
|
|
... Expected value line 'value: ci' in config output: ${get_profile.stdout}
|
|
# Output format
|
|
Run CleverAgents Command config set core.format json
|
|
${get_format}= Run CleverAgents Command config get core.format --format plain
|
|
Should Match Regexp ${get_format.stdout} (?m)^value:\\s+json\\s*$
|
|
... Expected value line 'value: json' in config output: ${get_format.stdout}
|
|
# Log level (spec Step 1)
|
|
Run CleverAgents Command config set core.log.level WARN
|
|
${get_log}= Run CleverAgents Command config get core.log.level --format plain
|
|
Should Match Regexp ${get_log.stdout} (?m)^value:\\s+WARN\\s*$
|
|
... Expected value line 'value: WARN' in config output: ${get_log.stdout}
|
|
|
|
WF07 E2E Idempotent Resource Registration
|
|
[Documentation] Register a git-checkout resource twice with --path and
|
|
... --branch, and verify it appears in the resource list.
|
|
[Tags] E2E
|
|
[Teardown] Log Idempotent Resource Registration teardown complete
|
|
${repo_dir}= Create Temp Git Repo With Issues ci-repo
|
|
# First registration (--branch main per spec Step 3).
|
|
Run CleverAgents Command resource add git-checkout ${CI_RESOURCE}
|
|
... --path ${repo_dir} --branch main expected_rc=${0}
|
|
# Second registration (idempotent — may succeed or fail gracefully)
|
|
Run CleverAgents Command resource add git-checkout ${CI_RESOURCE}
|
|
... --path ${repo_dir} --branch main expected_rc=None
|
|
# Verify it appears in the list
|
|
${list_out}= Run CleverAgents Command resource list --format plain
|
|
Output Should Contain ${list_out} ci-main
|
|
# Verify exactly one occurrence (idempotency: two adds must not duplicate)
|
|
${count}= Get Count ${list_out.stdout} ci-main
|
|
Should Be True ${count} == 1 Resource ci-main should appear exactly once (idempotency check)
|
|
|
|
WF07 E2E Idempotent Project Registration
|
|
[Documentation] Create a project linked to the resource twice and verify
|
|
... it appears in the project list (idempotent).
|
|
[Tags] E2E
|
|
[Teardown] Log Idempotent Project Registration teardown complete
|
|
# First creation with resource link and description (per spec Step 3)
|
|
Run CleverAgents Command project create ${CI_PROJECT}
|
|
... --description CI workspace for automated PR review
|
|
... --resource ${CI_RESOURCE} expected_rc=${0}
|
|
# Second creation (idempotent — may succeed or fail gracefully)
|
|
Run CleverAgents Command project create ${CI_PROJECT}
|
|
... --description CI workspace for automated PR review
|
|
... --resource ${CI_RESOURCE} expected_rc=None
|
|
${list_out}= Run CleverAgents Command project list --format plain
|
|
Output Should Contain ${list_out} ci-workspace
|
|
# Verify exactly one occurrence (idempotency: two creates must not duplicate).
|
|
# Use the full namespaced name because plain-text output includes both
|
|
# namespaced_name and name fields per entry — bare "ci-workspace" matches twice.
|
|
${count}= Get Count ${list_out.stdout} ${CI_PROJECT}
|
|
Should Be True ${count} == 1 Project ${CI_PROJECT} should appear exactly once (idempotency check)
|
|
|
|
WF07 E2E Validation Registration
|
|
[Documentation] Register lint, typecheck, and test validations, attach
|
|
... them to the project, and verify they appear in the tool
|
|
... list and project show output.
|
|
[Tags] E2E
|
|
[Teardown] Log Validation Registration teardown complete
|
|
# --- Lint validation ---
|
|
${lint_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${CI_LINT_VAL}
|
|
... description: Lint check validation for CI
|
|
... source: custom
|
|
... mode: required
|
|
... code: |
|
|
... ${SPACE}${SPACE}def run(inputs):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "lint check passed"}
|
|
${lint_path}= Set Variable ${SUITE_HOME}${/}ci-lint.yaml
|
|
Create File ${lint_path} ${lint_yaml}
|
|
Run CleverAgents Command validation add --config ${lint_path} expected_rc=None
|
|
# --- Typecheck validation ---
|
|
${type_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${CI_TYPECHECK_VAL}
|
|
... description: Type check validation for CI
|
|
... source: custom
|
|
... mode: required
|
|
... code: |
|
|
... ${SPACE}${SPACE}def run(inputs):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "typecheck passed"}
|
|
${type_path}= Set Variable ${SUITE_HOME}${/}ci-typecheck.yaml
|
|
Create File ${type_path} ${type_yaml}
|
|
Run CleverAgents Command validation add --config ${type_path} expected_rc=None
|
|
# --- Test validation ---
|
|
${test_yaml}= Catenate SEPARATOR=\n
|
|
... name: ${CI_TESTS_VAL}
|
|
... description: Test runner validation for CI
|
|
... source: custom
|
|
... mode: required
|
|
... code: |
|
|
... ${SPACE}${SPACE}def run(inputs):
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "tests passed"}
|
|
${test_path}= Set Variable ${SUITE_HOME}${/}ci-tests.yaml
|
|
Create File ${test_path} ${test_yaml}
|
|
Run CleverAgents Command validation add --config ${test_path} expected_rc=None
|
|
# --- Attach validations to project (spec Step 3) ---
|
|
Run CleverAgents Command validation attach
|
|
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_LINT_VAL} expected_rc=None
|
|
Run CleverAgents Command validation attach
|
|
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_TYPECHECK_VAL} expected_rc=None
|
|
Run CleverAgents Command validation attach
|
|
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_TESTS_VAL} expected_rc=None
|
|
# Verify validations appear in tool list
|
|
${list_out}= Run CleverAgents Command tool list --type validation --format plain
|
|
Output Should Contain ${list_out} ci-lint
|
|
Output Should Contain ${list_out} ci-typecheck
|
|
Output Should Contain ${list_out} ci-tests
|
|
# Verify the project itself is accessible (project show does not display
|
|
# attached validations — validation registration is verified by tool list above)
|
|
${project_out}= Run CleverAgents Command project show ${CI_PROJECT}
|
|
... --format plain
|
|
Log Project show output: ${project_out.stdout}
|
|
Output Should Contain ${project_out} ci-workspace
|
|
|
|
WF07 E2E CI Plan Launch
|
|
[Documentation] Create an action and launch a plan with --automation-profile ci,
|
|
... --format json, and --arg flags per specification Steps 2-3.
|
|
... Polls for plan completion. Requires LLM API keys.
|
|
[Tags] E2E
|
|
[Teardown] Log CI Plan Launch teardown complete
|
|
Skip If No LLM Keys
|
|
# Dynamically select actor based on available API keys.
|
|
# Prefer OpenAI first to reduce Anthropic credit-quota flakiness.
|
|
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
|
|
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
|
|
IF ${has_openai}
|
|
${actor}= Set Variable openai/gpt-4o
|
|
ELSE IF ${has_anthropic}
|
|
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
|
|
ELSE
|
|
${actor}= Set Variable openai/gpt-4o
|
|
END
|
|
# Create action with spec-aligned fields and dynamic actor
|
|
${action_yaml}= Catenate SEPARATOR=\n
|
|
... name: local/review-pr
|
|
... description: Automatically review a PR and fix issues
|
|
... strategy_actor: ${actor}
|
|
... execution_actor: ${actor}
|
|
... automation_profile: ci
|
|
... reusable: true
|
|
... state: available
|
|
... definition_of_done: |
|
|
... ${SPACE}${SPACE}- All lint issues are resolved
|
|
... ${SPACE}${SPACE}- Type checking passes
|
|
... ${SPACE}${SPACE}- Test coverage does not decrease
|
|
... ${SPACE}${SPACE}- Security scan passes
|
|
... ${SPACE}${SPACE}- All fixes are committed to the PR branch
|
|
... arguments:
|
|
... ${SPACE}${SPACE}- name: pr_branch
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Branch name of the PR
|
|
... ${SPACE}${SPACE}- name: base_branch
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}required: false
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}default: main
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Base branch to compare against
|
|
... invariants:
|
|
... ${SPACE}${SPACE}- "Only modify files that are already changed in the PR"
|
|
... ${SPACE}${SPACE}- "Do not change the intent of any code — only fix style, types, and test issues"
|
|
... ${SPACE}${SPACE}- "All fixes must include a comment explaining what was changed and why"
|
|
${action_path}= Set Variable ${SUITE_HOME}${/}review-pr-action.yaml
|
|
Create File ${action_path} ${action_yaml}
|
|
# expected_rc=None: action may already exist if Suite Setup cleanup failed
|
|
# and a prior E2E run left stale state; CI re-runnability requires tolerance.
|
|
Run CleverAgents Command action create --config ${action_path} expected_rc=None
|
|
# Launch plan with --arg flags per spec Step 3
|
|
${plan_result}= Run CleverAgents Command
|
|
... plan use local/review-pr ${CI_PROJECT}
|
|
... --automation-profile ci
|
|
... --format json
|
|
... --arg pr_branch=main
|
|
... --arg base_branch=main
|
|
... expected_rc=None timeout=180s
|
|
# Parse and validate JSON output
|
|
${stdout}= Set Variable ${plan_result.stdout}
|
|
${plan_id}= Safe Parse Json Field ${stdout} plan_id
|
|
IF '${plan_id}' != '${EMPTY}'
|
|
Log Plan launched with ID: ${plan_id}
|
|
# Drive the plan through its lifecycle. Per the specification
|
|
# (§Automation Profiles), ``plan use`` returns the plan in
|
|
# strategize/queued phase. ``plan execute`` drives the plan
|
|
# through Strategize → Execute, and auto_progress completes
|
|
# Apply when the ci profile (auto_apply=0.0) permits it.
|
|
${exec_result}= Run CleverAgents Command plan execute ${plan_id}
|
|
... --format json expected_rc=None timeout=180s
|
|
Should Be Equal As Integers ${exec_result.rc} 0
|
|
... plan execute failed (rc=${exec_result.rc}): ${exec_result.stderr}
|
|
# Poll plan status until it reaches a terminal state (spec Step 3
|
|
# polling loop).
|
|
${terminal_state}= Poll Plan Until Terminal ${plan_id}
|
|
Log Plan reached state: ${terminal_state}
|
|
Should Be Equal As Strings ${terminal_state} applied
|
|
... Plan did not reach 'applied' state. Final state: ${terminal_state}
|
|
# If applied, verify plan diff is parseable JSON (spec Step 3 final command)
|
|
${diff_result}= Run CleverAgents Command plan diff ${plan_id}
|
|
... --format json expected_rc=None
|
|
Log Plan diff output: ${diff_result.stdout}
|
|
# Validate the diff output is parseable JSON. The CLI may emit
|
|
# debug/log lines before the JSON payload, so we use raw_decode
|
|
# to locate the first valid JSON object.
|
|
IF len($diff_result.stdout) > 0
|
|
TRY
|
|
${parsed_diff}= Evaluate json.loads($diff_result.stdout) json
|
|
EXCEPT
|
|
TRY
|
|
${idx}= Evaluate $diff_result.stdout.index('{')
|
|
${parsed_diff}= Evaluate json.JSONDecoder().raw_decode($diff_result.stdout, $idx)[0] json
|
|
EXCEPT
|
|
Log Plan diff stdout is not valid JSON; may contain only log lines. WARN
|
|
${parsed_diff}= Set Variable ${NONE}
|
|
END
|
|
END
|
|
IF $parsed_diff is not None
|
|
Should Be True isinstance($parsed_diff, (dict, list))
|
|
... Expected JSON dict or list from plan diff
|
|
END
|
|
ELSE
|
|
Log Plan diff returned empty stdout; plan may have applied with no changes. WARN
|
|
END
|
|
ELSE
|
|
# Fail meaningfully when plan_id extraction fails
|
|
Fail Plan use command did not return a valid plan_id. rc=${plan_result.rc} stdout: ${plan_result.stdout} stderr: ${plan_result.stderr}
|
|
END
|
|
|
|
WF07 E2E JSON Output Verification
|
|
[Documentation] Verify that --format json produces valid, parseable JSON
|
|
... output from the version command.
|
|
[Tags] E2E
|
|
[Teardown] Log JSON Output Verification teardown complete
|
|
${result}= Run CleverAgents Command version --format json
|
|
Should Not Be Empty ${result.stdout}
|
|
# Parse as JSON and verify structure
|
|
${stdout}= Set Variable ${result.stdout}
|
|
${parsed}= Evaluate json.loads($stdout) json
|
|
Should Be True isinstance($parsed, dict)
|
|
... Expected JSON dict from version command, got: ${result.stdout}
|
|
Output Should Contain ${result} version
|
|
|
|
*** Keywords ***
|
|
WF07 Suite Setup
|
|
[Documentation] Delegate to shared E2E Suite Setup (includes centralized
|
|
... database initialization for DB-dependent CLI commands).
|
|
E2E Suite Setup
|
|
|
|
Create Temp Git Repo With Issues
|
|
[Documentation] Create a temporary git repo containing Python files with
|
|
... lint and type issues for the CI review scenario.
|
|
... Returns the path to the created repository.
|
|
[Arguments] ${name}=test-repo
|
|
${repo_dir}= Create Temp Git Repo ${name}
|
|
# Ensure the branch is named 'main' for spec consistency
|
|
${rename_result}= Run Process git branch -M main cwd=${repo_dir}
|
|
... timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${rename_result.rc} 0
|
|
... Failed to rename branch to main: ${rename_result.stderr}
|
|
# Add Python file with lint issues (unused imports, missing type hints,
|
|
# unused variable) and type errors to simulate a PR needing review.
|
|
${py_content}= Catenate SEPARATOR=\n
|
|
... import os
|
|
... import sys
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def greet(name: str) -> int:
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}msg = "Hello, " + name
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}return msg
|
|
... ${EMPTY}
|
|
... ${EMPTY}
|
|
... def unused_function():
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}x = 42
|
|
... ${SPACE}${SPACE}${SPACE}${SPACE}pass
|
|
Create File ${repo_dir}${/}app.py ${py_content}
|
|
# Verify git operations succeed
|
|
${add_result}= Run Process git add . cwd=${repo_dir}
|
|
... timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${add_result.rc} 0
|
|
... git add failed: ${add_result.stderr}
|
|
${commit_result}= Run Process git commit -m Add application with lint issues cwd=${repo_dir}
|
|
... timeout=30s on_timeout=kill
|
|
Should Be Equal As Integers ${commit_result.rc} 0
|
|
... git commit failed: ${commit_result.stderr}
|
|
RETURN ${repo_dir}
|
|
|
|
Poll Plan Until Terminal
|
|
[Documentation] Poll plan status until it reaches a terminal state.
|
|
... Polling exit states (not all are terminal in the domain
|
|
... model — ``errored`` is recoverable via ``resume_plan``
|
|
... but will not auto-advance, so polling should exit):
|
|
... applied, constrained, errored, cancelled.
|
|
... Returns the state string, or 'timeout'.
|
|
[Arguments] ${plan_id} ${max_polls}=18 ${interval}=10s
|
|
FOR ${i} IN RANGE ${max_polls}
|
|
${result}= Run CleverAgents Command plan status ${plan_id}
|
|
... --format json expected_rc=None
|
|
${state}= Safe Parse Json Field ${result.stdout} state
|
|
IF '${state}' in ['applied', 'constrained', 'cancelled', 'errored']
|
|
RETURN ${state}
|
|
END
|
|
Sleep ${interval}
|
|
END
|
|
RETURN timeout
|