test(e2e): workflow example 7 — CI/CD integration, automated PR review and fix (ci profile) #804
+69
-1
@@ -262,6 +262,38 @@
|
||||
and auto-discovery. Updated `git-checkout` child types. Includes YAML
|
||||
configs, Behave BDD tests, Robot tests, and ASV benchmarks. (#330)
|
||||
- Added TDD bug-capture tests for #932 (plan apply missing --yes flag). (#950)
|
||||
- Modified `auto_progress()` to complete the Apply phase immediately after
|
||||
transitioning from Execute to Apply, since Apply is a metadata transition
|
||||
with no LLM processing. This ensures `plan execute` drives the plan to
|
||||
the terminal `applied` state when the automation profile permits (ci,
|
||||
full-auto profiles with `auto_apply < 1.0`).
|
||||
Extracted `_complete_apply_if_queued()` helper that consolidates the
|
||||
Apply-completion pattern (start_apply + complete_apply) into a single
|
||||
method with error recovery (calls `fail_apply` on failure) and async-job
|
||||
guard (skips inline completion when async execution is enabled to avoid
|
||||
orphaning enqueued jobs). Used by `auto_progress()`,
|
||||
`lifecycle_apply_plan()`, and `try_auto_run()`.
|
||||
Added `PlanLifecycleService.try_auto_run()` that drives plans through all
|
||||
lifecycle phases (Strategize → Execute → Apply) when automation-profile
|
||||
thresholds allow automatic progression; a threshold of 1.0 stops the plan
|
||||
at that phase boundary for human approval.
|
||||
Fixed `lifecycle-apply` CLI leaving plans stuck in `apply/queued` without
|
||||
completing. The command now calls `_complete_apply_if_queued()` when the
|
||||
plan is in Apply/queued, driving it to the terminal `applied` state.
|
||||
Fixed stale RICH output in `lifecycle_apply_plan` that printed
|
||||
"Plan is now in Apply phase (queued)" after the plan had already reached
|
||||
terminal `applied` state; now branches on `plan.is_terminal`.
|
||||
Fixed SQLite UNIQUE constraint violation in
|
||||
`LifecyclePlanRepository.update()`: added `session.flush()` after
|
||||
`clear()` on child collections (project_links, arguments, invariants)
|
||||
before re-inserting rows.
|
||||
Added `state` alias in `_plan_spec_dict()` JSON output for spec §Example 7
|
||||
`jq` compatibility.
|
||||
Updated plan execute and lifecycle-apply reference documentation.
|
||||
(`src/cleveragents/application/services/plan_lifecycle_service.py`,
|
||||
`src/cleveragents/cli/commands/plan.py`,
|
||||
`src/cleveragents/infrastructure/database/repositories.py`,
|
||||
`docs/reference/plan_cli.md`) (#753)
|
||||
- Fixed `plan execute` CLI failing with "Plan is not in an executable state
|
||||
(current: strategize/queued)" after strategize completed successfully.
|
||||
Root cause: `_get_plan_executor()` created a second `PlanLifecycleService`
|
||||
@@ -418,8 +450,44 @@
|
||||
under the 500-line limit. `Verify Plan In List` and `Full Flow Apply Step`
|
||||
keywords use hard assertions instead of WARN fallbacks. Profile Precedence
|
||||
test documents that action > global precedence requires production wiring
|
||||
not yet present in `PlanLifecycleService.use_action`.
|
||||
not yet present in `PlanLifecycleService.use_action`.
|
||||
(`robot/e2e/m6_acceptance.robot`, `robot/e2e/common_e2e.resource`) (#746)
|
||||
- Added E2E Robot Framework test for Specification Workflow Example 7: CI/CD
|
||||
Integration — Automated PR Review and Fix. Exercises the `ci` automation
|
||||
profile (headless, non-interactive) with JSON output and log-level
|
||||
configuration, idempotent resource and project registration with `--branch`
|
||||
and `--description` flags, three-validation registration (source/mode/code)
|
||||
with project attachment and `project show` verification, action creation
|
||||
with spec-aligned name (`local/review-pr`), complete `definition_of_done`,
|
||||
`invariants`, and `arguments`, plan launch with `--arg` flags, explicit
|
||||
`plan execute` for lifecycle progression, `plan status` terminal-state
|
||||
assertion, plan diff JSON validation, and JSON output verification.
|
||||
Resource/project naming follows spec convention (`local/ci-workspace` project,
|
||||
`local/ci-main` resource). Entity creation commands tolerate "already
|
||||
exists" for CI re-runnability. `Extract JSON Field` keyword handles CLI
|
||||
debug log lines preceding JSON via `JSONDecoder.raw_decode(strict=False)`.
|
||||
Fail-fast `expected_rc` only where the spec mandates error suppression
|
||||
(`2>/dev/null || true`); first `resource add` and `project create` now
|
||||
assert `expected_rc=${0}`. Config assertions use stdout-only matching and
|
||||
exact equality for the `ci` profile value. Project idempotency verified
|
||||
with occurrence count. Empty plan-diff stdout logged as warning.
|
||||
Validation naming aligned with spec (`local/ci-lint` per §Example 7).
|
||||
All `Run Process` calls include `on_timeout=kill` per codebase CI
|
||||
stability standard. Added `on_timeout=kill` to `Run CleverAgents
|
||||
Command` and `Create Temp Git Repo` keywords in `common_e2e.resource`
|
||||
for consistent timeout handling across all E2E suites.
|
||||
Dynamic actor selection based on available API keys (same pattern as
|
||||
`m6_acceptance.robot`) avoids runtime failure when only one provider
|
||||
key is set. `Poll Plan Until Terminal` keyword now integrated into
|
||||
the CI Plan Launch test case per spec Step 3 polling loop. Replaced
|
||||
local `Extract JSON Field` with shared `Safe Parse Json Field` from
|
||||
`common_e2e.resource`. Added `Force Tags E2E` and per-test
|
||||
`[Teardown]` blocks. Added `WF07 Suite Setup` keyword for database
|
||||
initialisation.
|
||||
Added `robot/common_vars.py` module placeholder for shared Robot
|
||||
Framework variables.
|
||||
(`robot/e2e/wf07_cicd.robot`, `robot/e2e/common_e2e.resource`,
|
||||
`robot/common_vars.py`) (#753)
|
||||
- Fixed `agents session list`, `agents session create`, and other session
|
||||
subcommands raising `AttributeError: 'DynamicContainer' object has no
|
||||
attribute 'db'` after `agents init`. Root cause: `_get_session_service()`
|
||||
|
||||
@@ -9,8 +9,8 @@ The `agents plan` command group manages plans in the CleverAgents v3 plan lifecy
|
||||
| `agents plan use` | Create plan from action + project(s) |
|
||||
| `agents plan lifecycle-list` | List plans with optional filters |
|
||||
| `agents plan status` | Show plan status / details |
|
||||
| `agents plan execute` | Transition to Execute phase |
|
||||
| `agents plan lifecycle-apply` | Transition to Apply phase |
|
||||
| `agents plan execute` | Run Strategize + Execute (auto-apply if profile permits) |
|
||||
| `agents plan lifecycle-apply` | Transition to Apply and complete it |
|
||||
| `agents plan cancel` | Cancel a non-terminal plan |
|
||||
| `agents plan diff` | Show ChangeSet as unified diff |
|
||||
| `agents plan artifacts` | Show ChangeSet ID, sandbox refs, summary|
|
||||
@@ -131,6 +131,12 @@ phase and processes it inline:
|
||||
- **Strategize/complete** — transitions to Execute and runs it.
|
||||
- **Execute/queued** — runs the execute phase to completion.
|
||||
|
||||
When the automation profile's `auto_apply` threshold is met (< 1.0),
|
||||
the execute command also drives the plan through the Apply phase to
|
||||
the terminal `applied` state. This means `plan execute` with a
|
||||
`ci` or `full-auto` profile completes the full lifecycle in a single
|
||||
invocation: Strategize → Execute → Apply.
|
||||
|
||||
When no plan ID is given, auto-selects the single eligible plan.
|
||||
|
||||
```bash
|
||||
@@ -147,11 +153,16 @@ for details.
|
||||
|
||||
## `agents plan lifecycle-apply`
|
||||
|
||||
Transition a plan from Execute to Apply phase. Because Apply is a
|
||||
Transition a plan to Apply phase and complete it. Because Apply is a
|
||||
destructive operation (it merges sandbox changesets into real project
|
||||
resources), a confirmation prompt is displayed by default. Pass
|
||||
`--yes` / `-y` to skip the prompt in scripts or CI pipelines.
|
||||
|
||||
When the plan is in Execute/complete, transitions to Apply. When the
|
||||
plan is in Apply/queued (e.g. auto-progressed by `plan execute`),
|
||||
completes the apply processing, driving the plan to the terminal
|
||||
`applied` state. Apply is a metadata transition (no LLM call).
|
||||
|
||||
### Synopsis
|
||||
|
||||
```bash
|
||||
|
||||
@@ -46,7 +46,7 @@ Feature: Automation Levels
|
||||
Given I have a plan with automation level "full_automation" in execute phase with processing state
|
||||
When I complete execute on the automated plan
|
||||
Then the automated plan phase should be "apply"
|
||||
And the automated plan processing state should be "queued"
|
||||
And the automated plan processing state should be "applied"
|
||||
|
||||
# Plan-Level Override Tests
|
||||
|
||||
|
||||
@@ -293,13 +293,32 @@ def step_service_has_strategize_plan(context) -> None:
|
||||
|
||||
@given("the service has a complete execute plan for apply")
|
||||
def step_service_has_execute_plan(context) -> None:
|
||||
plan = _make_plan(
|
||||
pre_plan = _make_plan(
|
||||
plan_id=_ULIDS[1],
|
||||
name="local/apply-plan",
|
||||
phase=PlanPhase.EXECUTE,
|
||||
processing_state=ProcessingState.COMPLETE,
|
||||
)
|
||||
context.mock_lifecycle_service.apply_plan.return_value = plan
|
||||
apply_queued_plan = _make_plan(
|
||||
plan_id=_ULIDS[1],
|
||||
name="local/apply-plan",
|
||||
phase=PlanPhase.APPLY,
|
||||
processing_state=ProcessingState.QUEUED,
|
||||
)
|
||||
applied_plan = _make_plan(
|
||||
plan_id=_ULIDS[1],
|
||||
name="local/apply-plan",
|
||||
phase=PlanPhase.APPLY,
|
||||
processing_state=ProcessingState.APPLIED,
|
||||
)
|
||||
# lifecycle_apply_plan calls get_plan twice: first to check read_only
|
||||
# and phase, then again after apply_plan to drive _complete_apply_if_queued.
|
||||
context.mock_lifecycle_service.get_plan.side_effect = [
|
||||
pre_plan,
|
||||
apply_queued_plan,
|
||||
]
|
||||
context.mock_lifecycle_service.apply_plan.return_value = apply_queued_plan
|
||||
context.mock_lifecycle_service._complete_apply_if_queued.return_value = applied_plan
|
||||
context._apply_plan_id = _ULIDS[1]
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Common Robot Framework variables shared across test suites."""
|
||||
@@ -0,0 +1,344 @@
|
||||
*** Settings ***
|
||||
Documentation E2E test for Workflow Example 7: CI/CD integration,
|
||||
... automated PR review and fix (ci profile).
|
||||
...
|
||||
... Exercises the real CleverAgents CLI with zero mocking,
|
||||
... covering ci-profile configuration, idempotent resource and
|
||||
... project registration, validation registration with project
|
||||
... attachment, CI plan launch with action args and JSON output,
|
||||
... polling for plan completion, and JSON output verification.
|
||||
...
|
||||
... Tests form an implicit sequential pipeline via shared
|
||||
... database state: each test creates CLI entities consumed
|
||||
... by later tests. Execution order is file order.
|
||||
Resource common_e2e.resource
|
||||
Suite Setup WF07 Suite Setup
|
||||
Suite Teardown E2E Suite Teardown
|
||||
Force Tags E2E
|
||||
|
||||
*** Variables ***
|
||||
${CI_PROJECT} local/ci-workspace
|
||||
${CI_RESOURCE} local/ci-main
|
||||
${CI_LINT_VAL} local/ci-lint
|
||||
${CI_TYPECHECK_VAL} local/ci-typecheck
|
||||
${CI_TESTS_VAL} local/ci-tests
|
||||
|
||||
*** Test Cases ***
|
||||
WF07 E2E CI Profile Configuration
|
||||
[Documentation] Set and verify the ci automation profile, json format,
|
||||
... and log level per specification Step 1.
|
||||
[Tags] E2E
|
||||
[Teardown] Log CI Profile Configuration teardown complete
|
||||
# Initialize the CleverAgents data directory and database
|
||||
# expected_rc=None: init may report the CWD is already initialized; this
|
||||
# is harmless because CLEVERAGENTS_HOME is set to a fresh temp dir by
|
||||
# Suite Setup.
|
||||
Run CleverAgents Command init --yes expected_rc=None
|
||||
# Automation profile
|
||||
Run CleverAgents Command config set core.automation-profile ci
|
||||
${get_profile}= Run CleverAgents Command config get core.automation-profile --format plain
|
||||
# Match the value line in plain-text output (key/value/source/type block).
|
||||
# Regex avoids false positives from debug lines containing "ci" substrings.
|
||||
Should Match Regexp ${get_profile.stdout} (?m)^value:\\s+ci\\s*$
|
||||
... Expected value line 'value: ci' in config output: ${get_profile.stdout}
|
||||
# Output format
|
||||
Run CleverAgents Command config set core.format json
|
||||
${get_format}= Run CleverAgents Command config get core.format --format plain
|
||||
Should Match Regexp ${get_format.stdout} (?m)^value:\\s+json\\s*$
|
||||
... Expected value line 'value: json' in config output: ${get_format.stdout}
|
||||
# Log level (spec Step 1)
|
||||
Run CleverAgents Command config set core.log.level WARN
|
||||
${get_log}= Run CleverAgents Command config get core.log.level --format plain
|
||||
Should Match Regexp ${get_log.stdout} (?m)^value:\\s+WARN\\s*$
|
||||
... Expected value line 'value: WARN' in config output: ${get_log.stdout}
|
||||
|
||||
WF07 E2E Idempotent Resource Registration
|
||||
[Documentation] Register a git-checkout resource twice with --path and
|
||||
... --branch, and verify it appears in the resource list.
|
||||
[Tags] E2E
|
||||
[Teardown] Log Idempotent Resource Registration teardown complete
|
||||
${repo_dir}= Create Temp Git Repo With Issues ci-repo
|
||||
# First registration (--branch main per spec Step 3).
|
||||
Run CleverAgents Command resource add git-checkout ${CI_RESOURCE}
|
||||
... --path ${repo_dir} --branch main expected_rc=${0}
|
||||
# Second registration (idempotent — may succeed or fail gracefully)
|
||||
Run CleverAgents Command resource add git-checkout ${CI_RESOURCE}
|
||||
... --path ${repo_dir} --branch main expected_rc=None
|
||||
# Verify it appears in the list
|
||||
${list_out}= Run CleverAgents Command resource list --format plain
|
||||
Output Should Contain ${list_out} ci-main
|
||||
# Verify exactly one occurrence (idempotency: two adds must not duplicate)
|
||||
${count}= Get Count ${list_out.stdout} ci-main
|
||||
Should Be True ${count} == 1 Resource ci-main should appear exactly once (idempotency check)
|
||||
|
||||
WF07 E2E Idempotent Project Registration
|
||||
[Documentation] Create a project linked to the resource twice and verify
|
||||
... it appears in the project list (idempotent).
|
||||
[Tags] E2E
|
||||
[Teardown] Log Idempotent Project Registration teardown complete
|
||||
# First creation with resource link and description (per spec Step 3)
|
||||
Run CleverAgents Command project create ${CI_PROJECT}
|
||||
... --description CI workspace for automated PR review
|
||||
... --resource ${CI_RESOURCE} expected_rc=${0}
|
||||
# Second creation (idempotent — may succeed or fail gracefully)
|
||||
Run CleverAgents Command project create ${CI_PROJECT}
|
||||
... --description CI workspace for automated PR review
|
||||
... --resource ${CI_RESOURCE} expected_rc=None
|
||||
${list_out}= Run CleverAgents Command project list --format plain
|
||||
Output Should Contain ${list_out} ci-workspace
|
||||
# Verify exactly one occurrence (idempotency: two creates must not duplicate).
|
||||
# Use the full namespaced name because plain-text output includes both
|
||||
# namespaced_name and name fields per entry — bare "ci-workspace" matches twice.
|
||||
${count}= Get Count ${list_out.stdout} ${CI_PROJECT}
|
||||
Should Be True ${count} == 1 Project ${CI_PROJECT} should appear exactly once (idempotency check)
|
||||
|
||||
WF07 E2E Validation Registration
|
||||
[Documentation] Register lint, typecheck, and test validations, attach
|
||||
... them to the project, and verify they appear in the tool
|
||||
... list and project show output.
|
||||
[Tags] E2E
|
||||
[Teardown] Log Validation Registration teardown complete
|
||||
# --- Lint validation ---
|
||||
${lint_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${CI_LINT_VAL}
|
||||
... description: Lint check validation for CI
|
||||
... source: custom
|
||||
... mode: required
|
||||
... code: |
|
||||
... ${SPACE}${SPACE}def run(inputs):
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "lint check passed"}
|
||||
${lint_path}= Set Variable ${SUITE_HOME}${/}ci-lint.yaml
|
||||
Create File ${lint_path} ${lint_yaml}
|
||||
Run CleverAgents Command validation add --config ${lint_path} expected_rc=None
|
||||
# --- Typecheck validation ---
|
||||
${type_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${CI_TYPECHECK_VAL}
|
||||
... description: Type check validation for CI
|
||||
... source: custom
|
||||
... mode: required
|
||||
... code: |
|
||||
... ${SPACE}${SPACE}def run(inputs):
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "typecheck passed"}
|
||||
${type_path}= Set Variable ${SUITE_HOME}${/}ci-typecheck.yaml
|
||||
Create File ${type_path} ${type_yaml}
|
||||
Run CleverAgents Command validation add --config ${type_path} expected_rc=None
|
||||
# --- Test validation ---
|
||||
${test_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${CI_TESTS_VAL}
|
||||
... description: Test runner validation for CI
|
||||
... source: custom
|
||||
... mode: required
|
||||
... code: |
|
||||
... ${SPACE}${SPACE}def run(inputs):
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "tests passed"}
|
||||
${test_path}= Set Variable ${SUITE_HOME}${/}ci-tests.yaml
|
||||
Create File ${test_path} ${test_yaml}
|
||||
Run CleverAgents Command validation add --config ${test_path} expected_rc=None
|
||||
# --- Attach validations to project (spec Step 3) ---
|
||||
Run CleverAgents Command validation attach
|
||||
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_LINT_VAL} expected_rc=None
|
||||
Run CleverAgents Command validation attach
|
||||
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_TYPECHECK_VAL} expected_rc=None
|
||||
Run CleverAgents Command validation attach
|
||||
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_TESTS_VAL} expected_rc=None
|
||||
# Verify validations appear in tool list
|
||||
${list_out}= Run CleverAgents Command tool list --type validation --format plain
|
||||
Output Should Contain ${list_out} ci-lint
|
||||
Output Should Contain ${list_out} ci-typecheck
|
||||
Output Should Contain ${list_out} ci-tests
|
||||
# Verify the project itself is accessible (project show does not display
|
||||
# attached validations — validation registration is verified by tool list above)
|
||||
${project_out}= Run CleverAgents Command project show ${CI_PROJECT}
|
||||
... --format plain
|
||||
Log Project show output: ${project_out.stdout}
|
||||
Output Should Contain ${project_out} ci-workspace
|
||||
|
||||
WF07 E2E CI Plan Launch
|
||||
[Documentation] Create an action and launch a plan with --automation-profile ci,
|
||||
... --format json, and --arg flags per specification Steps 2-3.
|
||||
... Polls for plan completion. Requires LLM API keys.
|
||||
[Tags] E2E
|
||||
[Teardown] Log CI Plan Launch teardown complete
|
||||
Skip If No LLM Keys
|
||||
# Dynamically select actor based on available API keys.
|
||||
# Prefer OpenAI first to reduce Anthropic credit-quota flakiness.
|
||||
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
|
||||
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
|
||||
IF ${has_openai}
|
||||
${actor}= Set Variable openai/gpt-4o
|
||||
ELSE IF ${has_anthropic}
|
||||
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
|
||||
ELSE
|
||||
${actor}= Set Variable openai/gpt-4o
|
||||
END
|
||||
# Create action with spec-aligned fields and dynamic actor
|
||||
${action_yaml}= Catenate SEPARATOR=\n
|
||||
... name: local/review-pr
|
||||
... description: Automatically review a PR and fix issues
|
||||
... strategy_actor: ${actor}
|
||||
... execution_actor: ${actor}
|
||||
... automation_profile: ci
|
||||
... reusable: true
|
||||
... state: available
|
||||
... definition_of_done: |
|
||||
... ${SPACE}${SPACE}- All lint issues are resolved
|
||||
... ${SPACE}${SPACE}- Type checking passes
|
||||
... ${SPACE}${SPACE}- Test coverage does not decrease
|
||||
... ${SPACE}${SPACE}- Security scan passes
|
||||
... ${SPACE}${SPACE}- All fixes are committed to the PR branch
|
||||
... arguments:
|
||||
... ${SPACE}${SPACE}- name: pr_branch
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Branch name of the PR
|
||||
... ${SPACE}${SPACE}- name: base_branch
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}required: false
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}default: main
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Base branch to compare against
|
||||
... invariants:
|
||||
... ${SPACE}${SPACE}- "Only modify files that are already changed in the PR"
|
||||
... ${SPACE}${SPACE}- "Do not change the intent of any code — only fix style, types, and test issues"
|
||||
... ${SPACE}${SPACE}- "All fixes must include a comment explaining what was changed and why"
|
||||
${action_path}= Set Variable ${SUITE_HOME}${/}review-pr-action.yaml
|
||||
Create File ${action_path} ${action_yaml}
|
||||
# expected_rc=None: action may already exist if Suite Setup cleanup failed
|
||||
# and a prior E2E run left stale state; CI re-runnability requires tolerance.
|
||||
Run CleverAgents Command action create --config ${action_path} expected_rc=None
|
||||
# Launch plan with --arg flags per spec Step 3
|
||||
${plan_result}= Run CleverAgents Command
|
||||
... plan use local/review-pr ${CI_PROJECT}
|
||||
... --automation-profile ci
|
||||
... --format json
|
||||
... --arg pr_branch=main
|
||||
... --arg base_branch=main
|
||||
... expected_rc=None timeout=180s
|
||||
# Parse and validate JSON output
|
||||
${stdout}= Set Variable ${plan_result.stdout}
|
||||
${plan_id}= Safe Parse Json Field ${stdout} plan_id
|
||||
IF '${plan_id}' != '${EMPTY}'
|
||||
Log Plan launched with ID: ${plan_id}
|
||||
# Drive the plan through its lifecycle. Per the specification
|
||||
# (§Automation Profiles), ``plan use`` returns the plan in
|
||||
# strategize/queued phase. ``plan execute`` drives the plan
|
||||
# through Strategize → Execute, and auto_progress completes
|
||||
# Apply when the ci profile (auto_apply=0.0) permits it.
|
||||
${exec_result}= Run CleverAgents Command plan execute ${plan_id}
|
||||
... --format json expected_rc=None timeout=180s
|
||||
Should Be Equal As Integers ${exec_result.rc} 0
|
||||
... plan execute failed (rc=${exec_result.rc}): ${exec_result.stderr}
|
||||
# Poll plan status until it reaches a terminal state (spec Step 3
|
||||
# polling loop).
|
||||
${terminal_state}= Poll Plan Until Terminal ${plan_id}
|
||||
Log Plan reached state: ${terminal_state}
|
||||
Should Be Equal As Strings ${terminal_state} applied
|
||||
... Plan did not reach 'applied' state. Final state: ${terminal_state}
|
||||
# If applied, verify plan diff is parseable JSON (spec Step 3 final command)
|
||||
${diff_result}= Run CleverAgents Command plan diff ${plan_id}
|
||||
... --format json expected_rc=None
|
||||
Log Plan diff output: ${diff_result.stdout}
|
||||
# Validate the diff output is parseable JSON. The CLI may emit
|
||||
# debug/log lines before the JSON payload, so we use raw_decode
|
||||
# to locate the first valid JSON object.
|
||||
IF len($diff_result.stdout) > 0
|
||||
TRY
|
||||
${parsed_diff}= Evaluate json.loads($diff_result.stdout) json
|
||||
EXCEPT
|
||||
TRY
|
||||
${idx}= Evaluate $diff_result.stdout.index('{')
|
||||
${parsed_diff}= Evaluate json.JSONDecoder().raw_decode($diff_result.stdout, $idx)[0] json
|
||||
EXCEPT
|
||||
Log Plan diff stdout is not valid JSON; may contain only log lines. WARN
|
||||
${parsed_diff}= Set Variable ${NONE}
|
||||
END
|
||||
END
|
||||
IF $parsed_diff is not None
|
||||
Should Be True isinstance($parsed_diff, (dict, list))
|
||||
... Expected JSON dict or list from plan diff
|
||||
END
|
||||
ELSE
|
||||
Log Plan diff returned empty stdout; plan may have applied with no changes. WARN
|
||||
END
|
||||
ELSE
|
||||
# Fail meaningfully when plan_id extraction fails
|
||||
Fail Plan use command did not return a valid plan_id. rc=${plan_result.rc} stdout: ${plan_result.stdout} stderr: ${plan_result.stderr}
|
||||
END
|
||||
|
||||
WF07 E2E JSON Output Verification
|
||||
[Documentation] Verify that --format json produces valid, parseable JSON
|
||||
... output from the version command.
|
||||
[Tags] E2E
|
||||
[Teardown] Log JSON Output Verification teardown complete
|
||||
${result}= Run CleverAgents Command version --format json
|
||||
Should Not Be Empty ${result.stdout}
|
||||
# Parse as JSON and verify structure
|
||||
${stdout}= Set Variable ${result.stdout}
|
||||
${parsed}= Evaluate json.loads($stdout) json
|
||||
Should Be True isinstance($parsed, dict)
|
||||
... Expected JSON dict from version command, got: ${result.stdout}
|
||||
Output Should Contain ${result} version
|
||||
|
||||
*** Keywords ***
|
||||
WF07 Suite Setup
|
||||
[Documentation] E2E Suite Setup plus database initialisation.
|
||||
E2E Suite Setup
|
||||
# Initialise the database so config/resource/project commands work.
|
||||
${init}= Run CleverAgents Command init --force --yes
|
||||
Should Be Equal As Integers ${init.rc} 0
|
||||
|
||||
Create Temp Git Repo With Issues
|
||||
[Documentation] Create a temporary git repo containing Python files with
|
||||
|
|
||||
... lint and type issues for the CI review scenario.
|
||||
... Returns the path to the created repository.
|
||||
[Arguments] ${name}=test-repo
|
||||
${repo_dir}= Create Temp Git Repo ${name}
|
||||
# Ensure the branch is named 'main' for spec consistency
|
||||
${rename_result}= Run Process git branch -M main cwd=${repo_dir}
|
||||
... timeout=30s on_timeout=kill
|
||||
Should Be Equal As Integers ${rename_result.rc} 0
|
||||
... Failed to rename branch to main: ${rename_result.stderr}
|
||||
# Add Python file with lint issues (unused imports, missing type hints,
|
||||
# unused variable) and type errors to simulate a PR needing review.
|
||||
${py_content}= Catenate SEPARATOR=\n
|
||||
... import os
|
||||
|
brent.edwards
commented
P2 — Dead code: **P2 — Dead code**: `Poll Plan Until Terminal` keyword is defined but never called from any test case. The tests use synchronous `plan execute` + `plan status` instead. Remove or use it.
|
||||
... import sys
|
||||
... ${EMPTY}
|
||||
... ${EMPTY}
|
||||
... def greet(name: str) -> int:
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}msg = "Hello, " + name
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}return msg
|
||||
... ${EMPTY}
|
||||
... ${EMPTY}
|
||||
... def unused_function():
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}x = 42
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}pass
|
||||
Create File ${repo_dir}${/}app.py ${py_content}
|
||||
# Verify git operations succeed
|
||||
${add_result}= Run Process git add . cwd=${repo_dir}
|
||||
... timeout=30s on_timeout=kill
|
||||
Should Be Equal As Integers ${add_result.rc} 0
|
||||
... git add failed: ${add_result.stderr}
|
||||
${commit_result}= Run Process git commit -m Add application with lint issues cwd=${repo_dir}
|
||||
... timeout=30s on_timeout=kill
|
||||
Should Be Equal As Integers ${commit_result.rc} 0
|
||||
... git commit failed: ${commit_result.stderr}
|
||||
RETURN ${repo_dir}
|
||||
|
||||
Poll Plan Until Terminal
|
||||
[Documentation] Poll plan status until it reaches a terminal state.
|
||||
... Polling exit states (not all are terminal in the domain
|
||||
... model — ``errored`` is recoverable via ``resume_plan``
|
||||
... but will not auto-advance, so polling should exit):
|
||||
... applied, constrained, errored, cancelled.
|
||||
... Returns the state string, or 'timeout'.
|
||||
[Arguments] ${plan_id} ${max_polls}=18 ${interval}=10s
|
||||
FOR ${i} IN RANGE ${max_polls}
|
||||
${result}= Run CleverAgents Command plan status ${plan_id}
|
||||
... --format json expected_rc=None
|
||||
${state}= Safe Parse Json Field ${result.stdout} state
|
||||
IF '${state}' in ['applied', 'constrained', 'cancelled', 'errored']
|
||||
RETURN ${state}
|
||||
END
|
||||
Sleep ${interval}
|
||||
END
|
||||
RETURN timeout
|
||||
@@ -390,7 +390,8 @@ class PlanLifecycleService:
|
||||
|
||||
Args:
|
||||
plan_id: The plan ULID.
|
||||
phase: The plan phase (``"execute"`` or ``"apply"``).
|
||||
phase: The plan phase (``"strategize"``, ``"execute"``,
|
||||
or ``"apply"``).
|
||||
|
||||
Returns:
|
||||
The created ``AsyncJob`` if enqueued, or ``None`` if async
|
||||
@@ -1528,6 +1529,54 @@ class PlanLifecycleService:
|
||||
and profile.auto_apply < 1.0
|
||||
)
|
||||
|
||||
def _complete_apply_if_queued(self, plan_id: str) -> Plan:
|
||||
"""Drive a plan from Apply/QUEUED to terminal APPLIED.
|
||||
|
||||
Apply is a metadata-only transition (no LLM call), so it is
|
||||
safe to complete synchronously. If the plan is not in
|
||||
Apply/QUEUED, returns it unchanged.
|
||||
|
||||
When async execution is enabled (server mode with a job store),
|
||||
the Apply phase is handled by the async worker; this method
|
||||
skips inline completion to avoid orphaning the queued async job.
|
||||
|
||||
On failure the plan is moved to ``errored`` via
|
||||
:meth:`fail_apply` so it is never stuck in ``apply/processing``
|
||||
with no recovery path.
|
||||
|
||||
Args:
|
||||
plan_id: The plan ULID.
|
||||
|
||||
Returns:
|
||||
The (possibly updated) Plan.
|
||||
"""
|
||||
plan = self.get_plan(plan_id)
|
||||
|
||||
if plan.phase != PlanPhase.APPLY:
|
||||
return plan
|
||||
if plan.processing_state != ProcessingState.QUEUED:
|
||||
return plan
|
||||
|
||||
# When async execution is enabled, an async job was already
|
||||
# enqueued by apply_plan(); let the async worker handle it.
|
||||
if self.settings.async_enabled and self._job_store is not None:
|
||||
self._logger.info(
|
||||
"Skipping inline Apply completion — async job enqueued",
|
||||
plan_id=plan_id,
|
||||
)
|
||||
return plan
|
||||
|
||||
try:
|
||||
self.start_apply(plan_id)
|
||||
return self.complete_apply(plan_id)
|
||||
except Exception as exc:
|
||||
self._logger.error(
|
||||
"Apply completion failed; recording error state",
|
||||
plan_id=plan_id,
|
||||
error=str(exc),
|
||||
)
|
||||
return self.fail_apply(plan_id, str(exc))
|
||||
|
||||
def auto_progress(self, plan_id: str) -> Plan:
|
||||
"""If the plan's automation level permits, advance to the next phase.
|
||||
|
||||
@@ -1563,7 +1612,92 @@ class PlanLifecycleService:
|
||||
"Auto-progressing plan from Execute to Apply",
|
||||
plan_id=plan_id,
|
||||
)
|
||||
return self.apply_plan(plan_id)
|
||||
plan = self.apply_plan(plan_id)
|
||||
# Apply phase is a metadata transition (no LLM call), so
|
||||
# complete it immediately when auto-progressing. This
|
||||
# ensures ``plan execute`` drives the plan to terminal
|
||||
# ``applied`` state when the automation profile permits
|
||||
# (spec §Automation Profiles: auto_apply < 1.0).
|
||||
return self._complete_apply_if_queued(plan_id)
|
||||
|
||||
return plan
|
||||
|
||||
def try_auto_run(self, plan_id: str) -> Plan:
|
||||
"""Drive the plan through its lifecycle automatically.
|
||||
|
||||
When the plan's automation profile permits, this method advances
|
||||
the plan synchronously through Strategize → Execute → Apply:
|
||||
|
||||
* ``auto_strategize < 1.0`` → start + complete Strategize
|
||||
* ``auto_execute < 1.0`` → start + complete Execute
|
||||
* ``auto_apply < 1.0`` → start + complete Apply
|
||||
|
||||
Each completion step calls :meth:`auto_progress` internally,
|
||||
which transitions the plan to the next phase when appropriate.
|
||||
|
||||
If any threshold is ``1.0`` (human approval required), the plan
|
||||
stops at that phase boundary. The method is idempotent: calling
|
||||
it on a plan already in a terminal state simply returns the plan.
|
||||
|
||||
Args:
|
||||
plan_id: The plan ULID.
|
||||
|
||||
Returns:
|
||||
The (possibly advanced) Plan.
|
||||
|
||||
Raises:
|
||||
NotFoundError: If plan not found.
|
||||
"""
|
||||
plan = self.get_plan(plan_id)
|
||||
|
||||
if plan.is_terminal:
|
||||
return plan
|
||||
|
||||
profile = self._resolve_profile_for_plan(plan)
|
||||
|
||||
# --- Strategize phase: QUEUED → PROCESSING → COMPLETE -----------
|
||||
if (
|
||||
plan.phase == PlanPhase.STRATEGIZE
|
||||
and plan.state == ProcessingState.QUEUED
|
||||
and profile.auto_strategize < 1.0
|
||||
):
|
||||
self._logger.info(
|
||||
"Auto-running strategize phase",
|
||||
plan_id=plan_id,
|
||||
)
|
||||
self.start_strategize(plan_id)
|
||||
# complete_strategize() calls auto_progress() which may
|
||||
# transition the plan to Execute/QUEUED.
|
||||
plan = self.complete_strategize(plan_id)
|
||||
plan = self.get_plan(plan_id)
|
||||
|
||||
# --- Execute phase: QUEUED → PROCESSING → COMPLETE --------------
|
||||
if (
|
||||
plan.phase == PlanPhase.EXECUTE
|
||||
and plan.state == ProcessingState.QUEUED
|
||||
and profile.auto_execute < 1.0
|
||||
):
|
||||
self._logger.info(
|
||||
"Auto-running execute phase",
|
||||
plan_id=plan_id,
|
||||
)
|
||||
self.start_execute(plan_id)
|
||||
# complete_execute() calls auto_progress() which may
|
||||
# transition the plan to Apply/QUEUED.
|
||||
plan = self.complete_execute(plan_id)
|
||||
plan = self.get_plan(plan_id)
|
||||
|
||||
# --- Apply phase: QUEUED → PROCESSING → APPLIED (terminal) ------
|
||||
if (
|
||||
plan.phase == PlanPhase.APPLY
|
||||
and plan.state == ProcessingState.QUEUED
|
||||
and profile.auto_apply < 1.0
|
||||
):
|
||||
self._logger.info(
|
||||
"Auto-running apply phase",
|
||||
plan_id=plan_id,
|
||||
)
|
||||
plan = self._complete_apply_if_queued(plan_id)
|
||||
|
||||
return plan
|
||||
|
||||
|
||||
@@ -151,6 +151,9 @@ def _plan_spec_dict(plan: Any) -> dict[str, object]:
|
||||
"namespaced_name": str(plan.namespaced_name),
|
||||
"phase": plan.phase.value,
|
||||
"processing_state": plan.processing_state.value,
|
||||
# Spec §Example 7 uses ``jq -r '.state'`` — provide a
|
||||
# convenience alias so both field names work.
|
||||
"state": plan.processing_state.value,
|
||||
"project_links": [
|
||||
{
|
||||
"project_name": link.project_name,
|
||||
@@ -1782,10 +1785,16 @@ def use_action(
|
||||
console.print(format_output(data, fmt))
|
||||
else:
|
||||
_print_lifecycle_plan(plan, title="Plan Created")
|
||||
console.print(
|
||||
"\n[dim]Plan is now in Strategize phase (queued). "
|
||||
"Run 'agents plan execute <id>' when ready.[/dim]"
|
||||
)
|
||||
if plan.is_terminal:
|
||||
console.print(
|
||||
f"\n[dim]Plan completed with state: "
|
||||
f"{plan.state.value if plan.state else 'unknown'}.[/dim]"
|
||||
)
|
||||
else:
|
||||
console.print(
|
||||
"\n[dim]Plan is now in Strategize phase (queued). "
|
||||
"Run 'agents plan execute <id>' when ready.[/dim]"
|
||||
)
|
||||
|
||||
except ActionNotAvailableError as e:
|
||||
console.print(f"[red]Action not available:[/red] {e}")
|
||||
@@ -2091,15 +2100,33 @@ def lifecycle_apply_plan(
|
||||
# Normal path: plan is in Execute/complete → transition
|
||||
plan = service.apply_plan(plan_id)
|
||||
|
||||
# Apply phase is a metadata transition (no LLM call). When the
|
||||
# plan is in Apply/queued, complete it to the terminal ``applied``
|
||||
# state. This intentionally bypasses the automation profile's
|
||||
# ``auto_apply`` threshold: explicitly running ``lifecycle-apply``
|
||||
# IS the human approval action for the manual profile.
|
||||
plan = service.get_plan(plan_id)
|
||||
if (
|
||||
plan.phase == PlanPhase.APPLY
|
||||
and plan.processing_state == ProcessingState.QUEUED
|
||||
):
|
||||
plan = service._complete_apply_if_queued(plan_id)
|
||||
|
||||
if fmt != OutputFormat.RICH.value:
|
||||
data = _plan_spec_dict(plan)
|
||||
console.print(format_output(data, fmt))
|
||||
else:
|
||||
_print_lifecycle_plan(plan, title="Plan Applying")
|
||||
console.print(
|
||||
"\n[dim]Plan is now in Apply phase (queued). "
|
||||
"Changes will be applied to the project(s).[/dim]"
|
||||
)
|
||||
if plan.is_terminal:
|
||||
console.print(
|
||||
f"\n[dim]Plan completed in Apply phase with state: "
|
||||
f"{plan.state.value if plan.state else 'unknown'}.[/dim]"
|
||||
)
|
||||
else:
|
||||
console.print(
|
||||
"\n[dim]Plan is now in Apply phase (queued). "
|
||||
"Changes will be applied to the project(s).[/dim]"
|
||||
)
|
||||
|
||||
except InvalidPhaseTransitionError as e:
|
||||
console.print(f"[red]Invalid transition:[/red] {e}")
|
||||
|
||||
@@ -1404,7 +1404,6 @@ class LifecyclePlanRepository:
|
||||
row.arguments_rel.clear() # type: ignore[union-attr]
|
||||
row.invariants_rel.clear() # type: ignore[union-attr]
|
||||
session.flush()
|
||||
|
||||
now_iso = plan.timestamps.updated_at.isoformat()
|
||||
for pl in getattr(plan, "project_links", []) or []:
|
||||
row.project_links_rel.append( # type: ignore[union-attr]
|
||||
|
||||
Reference in New Issue
Block a user
P2 — Duplicate keyword:
Extract JSON Fieldduplicates the functionality ofSafe Parse Json Fieldalready defined incommon_e2e.resource. Both solve the same problem (extracting a JSON field from CLI output that may have log lines before JSON) with different implementations. Consolidate into one keyword incommon_e2e.resourceto avoid divergent maintenance.