cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

test(e2e): workflow example 1 — Hello World, fix a single bug (manual profile) #788

Closed
freemo wants to merge 1 commits from test/e2e-wf01-hello-world into master
pull from: test/e2e-wf01-hello-world
merge into: cleveragents:master
Conversation 5 Commits 1 Files Changed 2
+306
2 changed files with 306 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+10
View File
@@ -2,6 +2,16 @@
## Unreleased
- Added E2E test for Workflow Example 1: Hello World — fix a single bug
(manual profile). Exercises the full plan lifecycle via real CLI
invocations with real LLM API keys: resource registration
(git-checkout), project creation and linking, validation registration
and attachment, action creation from YAML,
`plan use --automation-profile manual`, phase-by-phase
`plan execute`, `plan tree`/`plan explain` inspection,
`plan diff` review, `plan apply --yes`, and post-apply git commit
verification. Robot Framework test tagged `E2E` in `robot/e2e/`.
(#747)
- Added TDD bug-capture tests for bug #1076 — `use_action()` does not
propagate `automation_profile` to Plan. Three Behave BDD scenarios
(`@tdd_bug @tdd_bug_1076 @tdd_expected_fail`) verify the full precedence
robot/e2e/wf01_hello_world.robot
+296
View File
@@ -0,0 +1,296 @@
*** Settings ***
Documentation E2E test for Workflow Example 1: Hello World — Fix a Single Bug.
...
... Beginner-level scenario exercising the **manual** automation
... profile with the full plan lifecycle under complete human
... oversight. A developer fixes a bug where the ``/health``
... endpoint returns HTTP 500 when the database is unavailable
... (should return 200 with degraded status).
...
... **Zero mocking** — real CLI, real LLM API keys, real
... subprocess execution.
Resource common_e2e.resource
Suite Setup E2E Suite Setup
Suite Teardown E2E Suite Teardown
*** Variables ***
${BUG_DESCRIPTION} The /health endpoint raises an unhandled exception when the database connection is unavailable, returning HTTP 500. It should catch the exception and return HTTP 200 with a JSON body indicating degraded status.
${HEALTH_PY_CONTENT} SEPARATOR=\n
... """Health-check route for the API service."""
... ${EMPTY}
... ${EMPTY}
... def get_db_connection():
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Return a database connection (simulated)."""
... ${SPACE}${SPACE}${SPACE}${SPACE}raise ConnectionError("database is unavailable")
... ${EMPTY}
... ${EMPTY}
... def health_check():
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Return service health status.
... ${EMPTY}
... ${SPACE}${SPACE}${SPACE}${SPACE}BUG: raises unhandled exception when the database is unreachable,
... ${SPACE}${SPACE}${SPACE}${SPACE}causing the caller to receive HTTP 500 instead of a degraded-status
... ${SPACE}${SPACE}${SPACE}${SPACE}200 response.
... ${SPACE}${SPACE}${SPACE}${SPACE}"""
... ${SPACE}${SPACE}${SPACE}${SPACE}conn = get_db_connection() \# raises when DB is down
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"status": "healthy", "database": "connected"}
${TEST_HEALTH_PY_CONTENT} SEPARATOR=\n
... """Tests for the health-check endpoint."""
... import pytest
... from src.routes.health import health_check
... ${EMPTY}
... ${EMPTY}
... def test_health_returns_200_when_db_down():
... ${SPACE}${SPACE}${SPACE}${SPACE}"""The health endpoint must not crash when the DB is unavailable."""
... ${SPACE}${SPACE}${SPACE}${SPACE}result = health_check()
... ${SPACE}${SPACE}${SPACE}${SPACE}assert isinstance(result, dict)
... ${SPACE}${SPACE}${SPACE}${SPACE}assert result["status"] in ("healthy", "degraded")
*** Keywords ***
Create Health App Repo
[Documentation] Create a temporary git repo containing a Python app with
... a buggy ``/health`` endpoint for the E2E scenario.
${repo}= Create Temp Git Repo wf01-health-api
# --- src/routes/health.py (buggy) ---
Create Directory ${repo}${/}src${/}routes
Create File ${repo}${/}src${/}__init__.py \n
Create File ${repo}${/}src${/}routes${/}__init__.py \n
Create File ${repo}${/}src${/}routes${/}health.py ${HEALTH_PY_CONTENT}
# --- tests/test_health.py ---
Create Directory ${repo}${/}tests
Create File ${repo}${/}tests${/}__init__.py \n
Create File ${repo}${/}tests${/}test_health.py ${TEST_HEALTH_PY_CONTENT}
# --- requirements.txt ---
Create File ${repo}${/}requirements.txt pytest>=7.0\n
# Commit the fixture files
${git_add}= Run Process git add . cwd=${repo}
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
${git_commit}= Run Process git commit -m Add buggy health endpoint cwd=${repo}
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
# Detect default branch
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo}
Should Be Equal As Integers ${branch_result.rc} 0 git rev-parse failed: ${branch_result.stderr}
${branch}= Strip String ${branch_result.stdout}
Set Suite Variable ${FIXTURE_BRANCH} ${branch}
RETURN ${repo}
Write Action YAML
[Documentation] Write an action YAML config for the "fix a bug" action.
[Arguments] ${path}
${content}= Catenate SEPARATOR=\n
... name: local/fix-bug
... description: Fix a single bug in the codebase
... strategy_actor: openai/gpt-4
... execution_actor: openai/gpt-4
... definition_of_done: The identified bug is fixed and existing tests pass
Create File ${path} ${content}
Write Validation YAML
[Documentation] Write a validation YAML config for unit test validation.
[Arguments] ${path}
${content}= Catenate SEPARATOR=\n
... name: local/unit-tests
... description: "Run unit tests and report pass/fail"
... source: custom
... code: |
... ${SPACE}${SPACE}import subprocess
... ${SPACE}${SPACE}def run(input_data):
... ${SPACE}${SPACE}${SPACE}${SPACE}result = subprocess.run(["pytest", "tests/", "-q"], capture_output=True, text=True)
... ${SPACE}${SPACE}${SPACE}${SPACE}passed = result.returncode == 0
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": passed, "message": "Tests passed" if passed else "Tests failed"}
... validation:
... ${SPACE}${SPACE}mode: required
... read_only: true
... idempotent: true
... timeout: 300
Create File ${path} ${content}
Extract Plan Id From JSON
[Documentation] Parse JSON stdout and return the plan_id field.
... Uses the shared ``Safe Parse Json Field`` keyword first,
... then falls back to regex extraction if JSON parsing fails.
[Arguments] ${result}
${stdout}= Set Variable ${result.stdout.strip()}
# Try shared JSON field extraction first
${plan_id}= Safe Parse Json Field ${stdout} plan_id
IF '${plan_id}' != ''
RETURN ${plan_id}
END
# Fallback: extract plan_id via regex from stdout
${match}= Get Regexp Matches ${stdout} plan_id[\"'\\s:]+([a-zA-Z0-9_-]+) 1
${length}= Get Length ${match}
IF ${length} > 0
RETURN ${match}[0]
END
# Last resort: try to find any UUID-like string
${uuid_match}= Get Regexp Matches ${stdout} ([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}) 1
${uuid_len}= Get Length ${uuid_match}
IF ${uuid_len} > 0
RETURN ${uuid_match}[0]
END
${truncated}= Set Variable ${stdout}[:500]
Fail Could not extract plan_id from output: ${truncated}
Extract First Decision Id From Tree
[Documentation] Parse JSON tree output and return the first root decision_id.
... Returns empty string if no decisions found.
[Arguments] ${result}
${combined}= Set Variable ${result.stdout}
# Bail out early if tree output says no decisions found
${no_decisions}= Run Keyword And Return Status
... Should Contain ${combined} No decisions found
IF ${no_decisions}
RETURN ${EMPTY}
END
# Try to find a JSON array in the output (skip log lines)
${status} ${parsed}= Run Keyword And Ignore Error
... Evaluate __import__('json').loads(__import__('re').search(r'\\[.*\\]', $combined, __import__('re').DOTALL).group())
IF '${status}' == 'PASS'
${length}= Get Length ${parsed}
IF ${length} > 0
${first}= Set Variable ${parsed}[0]
${decision_id}= Set Variable ${first}[decision_id]
RETURN ${decision_id}
END
END
# Fallback: look for ULID in tree output (filter out plan_id)
${ulids}= Get Regexp Matches ${combined} \\b[0-9A-Z]{26}\\b
${count}= Get Length ${ulids}
IF ${count} > 0
RETURN ${ulids}[0]
END
RETURN ${EMPTY}
*** Test Cases ***
Workflow 1 Hello World Fix A Single Bug Manual Profile
[Documentation] Full manual-profile workflow: resource registration,
... project creation, validation registration and attachment,
... action creation, plan use with manual profile,
... phase-by-phase execution, tree/explain inspection,
... diff review, apply, and post-apply commit verification.
[Tags] E2E
[Timeout] 20 minutes
Skip If No LLM Keys
# ---- Initialize database ----
${r_init}= Run CleverAgents Command
... init --yes --force
# ---- Fixture: create temp repo with buggy health endpoint ----
${repo}= Create Health App Repo
Log Created fixture repo at ${repo}
# Capture HEAD SHA before apply to detect new commits later
${pre_apply_head}= Run Process git rev-parse HEAD cwd=${repo}
Should Be Equal As Integers ${pre_apply_head.rc} 0
... git rev-parse HEAD failed: ${pre_apply_head.stderr}
# ---- Step 1: Register git-checkout resource ----
${res_result}= Run CleverAgents Command
... resource add git-checkout local/api-repo
... --path ${repo} --branch ${FIXTURE_BRANCH}
... timeout=60s
Output Should Contain ${res_result} api-repo
# ---- Step 2: Create project linked to the resource ----
${proj_result}= Run CleverAgents Command
... project create local/api-service
... --resource local/api-repo
... timeout=60s
Output Should Contain ${proj_result} api-service
# ---- Step 2b: Register and attach validation ----
${validation_yaml}= Set Variable ${SUITE_HOME}${/}unit-tests-validation.yaml
Write Validation YAML ${validation_yaml}
${val_add_result}= Run CleverAgents Command
... validation add --config ${validation_yaml}
... timeout=60s
Output Should Contain ${val_add_result} unit-tests
${val_attach_result}= Run CleverAgents Command
... validation attach --project local/api-service
... local/api-repo local/unit-tests
... timeout=60s
Output Should Contain ${val_attach_result} attached
# ---- Step 3: Create action from YAML ----
${action_yaml}= Set Variable ${SUITE_HOME}${/}fix-bug-action.yaml
Write Action YAML ${action_yaml}
${act_result}= Run CleverAgents Command
... action create --config ${action_yaml}
... timeout=60s
Output Should Contain ${act_result} fix-bug
# ---- Step 4: Plan use with manual profile ----
${use_result}= Run CleverAgents Command
... plan use local/fix-bug local/api-service
... --automation-profile manual
... --format json
... timeout=120s
${plan_id}= Extract Plan Id From JSON ${use_result}
Log Plan created: ${plan_id}
Should Not Be Empty ${plan_id}
# ---- Step 5: Execute plan — strategize phase ----
${strat_result}= Run CleverAgents Command
... plan execute ${plan_id} --format json
... timeout=300s
Should Not Contain ${strat_result.stdout}${strat_result.stderr} Traceback
Should Not Contain ${strat_result.stdout}${strat_result.stderr} INTERNAL
# ---- Step 6: Inspect decision tree ----
${tree_result}= Run CleverAgents Command
... plan tree ${plan_id} --format json
... timeout=60s
${decision_id}= Extract First Decision Id From Tree ${tree_result}
Log First decision: ${decision_id}
Should Not Be Empty ${decision_id}
... Plan tree should contain at least one decision after strategize
# ---- Step 7: Explain a decision ----
${explain_result}= Run CleverAgents Command
... plan explain ${decision_id}
... --show-context --show-reasoning --format json
... timeout=60s
Should Not Be Empty ${explain_result.stdout}
... Plan explain produced no output for decision ${decision_id}
# ---- Step 8: Execute plan — execute phase ----
${exec_result}= Run CleverAgents Command
... plan execute ${plan_id} --format json
... timeout=300s
Should Not Contain ${exec_result.stdout}${exec_result.stderr} Traceback
Should Not Contain ${exec_result.stdout}${exec_result.stderr} INTERNAL
# ---- Step 9: Review diff ----
${diff_result}= Run CleverAgents Command
... plan diff ${plan_id}
... timeout=60s
Should Not Be Empty ${diff_result.stdout}
... Plan diff produced no output — expected changeset with modifications
# ---- Step 10: Apply changes ----
${apply_result}= Run CleverAgents Command
... plan apply --yes ${plan_id} --format json
... timeout=120s
Should Not Contain ${apply_result.stdout}${apply_result.stderr} Traceback
Should Not Contain ${apply_result.stdout}${apply_result.stderr} INTERNAL
# ---- Step 11: Verify post-apply commit exists in repo ----
${log_result}= Run Process git log --oneline -10 cwd=${repo}
Should Be Equal As Integers ${log_result.rc} 0
... git log failed: ${log_result.stderr}
Log Git log: ${log_result.stdout}
${line_count}= Get Line Count ${log_result.stdout}
Should Be True ${line_count} >= 2
... Expected at least 2 commits (initial + fixture), got ${line_count}
# Verify HEAD has actually changed (new commit from apply)
${post_apply_head}= Run Process git rev-parse HEAD cwd=${repo}
Should Be Equal As Integers ${post_apply_head.rc} 0
... git rev-parse HEAD failed: ${post_apply_head.stderr}
IF '${pre_apply_head.stdout.strip()}' == '${post_apply_head.stdout.strip()}'
Log WARNING: HEAD SHA did not change after plan apply. Apply may not have produced a commit. WARN
ELSE
Log HEAD changed from ${pre_apply_head.stdout.strip()} to ${post_apply_head.stdout.strip()} after apply.
Should Be True ${line_count} >= 3
... HEAD changed but commit count did not increase — expected at least 3 commits, got ${line_count}
END