Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| d6046fb0c1 | |||
| 91730a7801 | |||
| 1b0cec265e | |||
| 1003605cba |
@@ -170,4 +170,4 @@ jobs:
|
||||
path: |
|
||||
build/nox-e2e-tests-output.log
|
||||
build/reports/robot-e2e/
|
||||
retention-days: 30
|
||||
retention-days: 30
|
||||
|
||||
+50
-55
@@ -1,18 +1,58 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to this project will be documented in this file.
|
||||
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Restored deleted E2E test files (`features/cli_main_cov3.feature`, `robot/plan_diff_artifacts.robot`) removed by commit 8ea00f51 which covered CLI main line coverage and plan diff/artifacts workflows. Step definitions and helper utilities were already present in the codebase. Resolves AUTO-WDOG announcement #8490 and blocks milestone acceptance criteria validation for v3.2.0.
|
||||
- Hardened the TDD bug-fix quality gate for issue #629: PR parsing now
|
||||
requires whole-word closing keywords (avoids false positives like
|
||||
"prefixes #12"), TDD bug tag discovery now uses exact token matching
|
||||
(avoids `@tdd_bug_42` matching `@tdd_bug_420`), and gate evaluation now
|
||||
requires expected-fail tag removal to be present in the PR diff. CI
|
||||
integration now fetches the PR base branch for diff analysis and includes
|
||||
`tdd_quality_gate` in `status-check` requirements for pull requests.
|
||||
Review-round fixes: `check_expected_fail_removed` now uses word-boundary
|
||||
matching via `_contains_tag_token` (avoids false positives on partial tag
|
||||
names); diff expected-fail removal detection now tracks flags at file level
|
||||
instead of per-hunk (fixes false negatives when tags span different hunks);
|
||||
`parse_bug_refs` filters out issue number zero; redundant double error
|
||||
reporting eliminated; regex compilation cached via `lru_cache`; nox session
|
||||
no longer installs the full project (script uses stdlib only); CI checkout
|
||||
uses `fetch-depth: 0` for reliable merge-base resolution.
|
||||
Review-round 2 fixes: diff expected-fail removal detection now requires the
|
||||
removed line to contain both the expected-fail tag and the specific bug tag
|
||||
(fixes false positives when two bugs' TDD tests reside in the same file);
|
||||
`check_expected_fail_removed` error messages now use the correct tag prefix
|
||||
per file type (`@tdd_bug_N` for `.feature`, `tdd_bug_N` for `.robot`);
|
||||
`bool` values are now rejected by bug-number validation guards; file-read
|
||||
error handling in `find_tdd_tests` and `check_expected_fail_removed` now
|
||||
catches `UnicodeDecodeError` (root-safe unreadable-file handling); temp
|
||||
directory cleanup added to `after_scenario` hook; 8 new Behave scenarios
|
||||
covering bool guards, co-located bug false-positive, `run_quality_gate`
|
||||
argument validation, and `main()` CLI entry point.
|
||||
Review-round 3 fixes: synthetic PR diff helper now auto-detects `.robot`
|
||||
vs `.feature` file type and generates the matching diff format (fixes
|
||||
under-tested robot-format diff code path); `check_expected_fail_removed`
|
||||
test step now filters files by bug tag via `find_tdd_tests` before
|
||||
checking (matches the production code path); `after_scenario` temp
|
||||
directory cleanup no longer sets `context.temp_dir = None` (fixes
|
||||
cleanup conflict with `cli_init_yes_flag_steps.py`); 2 new Behave
|
||||
scenarios covering multi-line PR description parsing and non-string
|
||||
`pr_diff` type guard.
|
||||
- Added Fix-then-Revalidate orchestration loop for required validations:
|
||||
bounded retry with configurable limits (0--100 per Safety Profile),
|
||||
strategy revision escalation via ``auto_strategy_revision`` float
|
||||
threshold, user escalation via ``needs_user_escalation`` result flag,
|
||||
and domain events (``VALIDATION_FIX_ATTEMPTED``, ``VALIDATION_FIX_SUCCEEDED``,
|
||||
``VALIDATION_FIX_EXHAUSTED``). Validation errors are treated as required
|
||||
failures regardless of mode. Includes ``auto_validation_fix`` threshold,
|
||||
per-resource retry tracking, early-exit signalling via ``None`` return from
|
||||
``FixCallback``, event bus circuit breaker with lock-protected failure
|
||||
counter, spec-required ``validation_summary`` and
|
||||
``final_validation_results`` fields on the result model, DI container
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
- **`task-implementor` posts work-started notification comments** (#11031): Both
|
||||
the `issue_impl` and `pr_fix` procedures now post an informational "work
|
||||
started" comment to the Forgejo issue/PR before beginning implementation.
|
||||
The comment includes the issue/PR title, procedure type, and expected
|
||||
duration. Posted asynchronously ("fire and move on") so it does not block
|
||||
the workflow. Step numbering in both procedures has been re-numbered to
|
||||
accommodate the new step.
|
||||
|
||||
- **`agents session tell` invokes real LLM orchestrator actor** (#5784): Replaced the
|
||||
M3 echo-stub with real actor invocation via `SessionWorkflow`, routing through
|
||||
`LangChainSessionCaller` → `ToolCallingRuntime.run_tool_loop()`. The user prompt
|
||||
@@ -24,10 +64,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
LLM streaming output. A `SessionActorNotConfiguredError` is raised with exit
|
||||
code 1 when no actor is configured.
|
||||
|
||||
### Added
|
||||
|
||||
- **A2A module rename BDD test suite** (#8615): Comprehensive Behave tests validating that the ACP→A2A module rename is complete — verifying all 22 A2A symbols are properly exported, no legacy ACP references remain in `.py` files under `cleveragents.a2a/`, and the module docstring uses current A2A naming. The step definitions include self-contained symbol lookups to avoid cross-scenario dependency failures.
|
||||
|
||||
- Fixed `ReactiveEventBus.emit()` exception handler to log the full exception
|
||||
message (`str(exc)`) and enable traceback forwarding (`exc_info=True`).
|
||||
Previously the handler logged only the exception type name (e.g.
|
||||
@@ -38,40 +74,9 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
### Added
|
||||
|
||||
- **`pr-review-worker` review-started notification** (#11028): The `first_review`
|
||||
and `re_review` modes now post a "review started" notification comment to the
|
||||
PR at the beginning of the review, giving PR authors immediate visibility
|
||||
that a review is in progress. Posted asynchronously so it does not block
|
||||
the workflow.
|
||||
|
||||
- **Plan Rollback Command** (#8557): Implemented `agents plan rollback <plan-id> [<checkpoint-id>]` for checkpoint-based plan state restoration in Epic #8493. The command restores a plan's sandbox to the state captured at a given checkpoint, discarding all decisions made after that checkpoint. The checkpoint can be specified as an optional positional second argument or via the `--to-checkpoint` named option. Supports `--yes/-y` flag to skip confirmation prompts and `--format/-f` for output format selection (rich/plain/json/yaml). Included with comprehensive BDD test coverage (>= 97%) and spec-aligned output formatting showing rollback summary, changes reverted, impact analysis, and post-rollback state panels.
|
||||
|
||||
### Fixed
|
||||
- **Guard cleanup_stale against execute/processing and execute/complete plans** (#11121):
|
||||
``_create_sandbox_for_plan()`` in ``src/cleveragents/cli/commands/plan.py`` now
|
||||
skips ``GitWorktreeSandbox.cleanup_stale()`` when the plan is in
|
||||
``execute/processing`` (execution in progress) or ``execute/complete`` (execution
|
||||
finished, awaiting apply) state. Previously, re-invoking ``agents plan execute``
|
||||
on a completed plan would silently destroy the ``cleveragents/plan-<id>`` git
|
||||
worktree branch, causing ``plan apply`` to merge zero artifacts. The guard
|
||||
preserves the branch per spec (§sandbox.cleanup defaults to ``on_apply``).
|
||||
|
||||
- **Global CLI options ``--data-dir``, ``--config-path``, and ``-v`` now work correctly**
|
||||
(#6785): These spec-required flags were absent from ``main_callback()`` in
|
||||
``src/cleveragents/cli/main.py``, causing any invocation with these flags to crash
|
||||
with ``NoSuchOption: No such option``. All three options are now implemented per
|
||||
ADR-021 §Global CLI Flags and ADR-024 §Resolution Chain. ``--data-dir`` sets
|
||||
``CLEVERAGENTS_DATA_DIR`` before any ``Settings``-reading code runs, ``--config-path``
|
||||
sets ``CLEVERAGENTS_CONFIG_PATH`` for ``ConfigService`` to pick up, and ``-v``
|
||||
(repeatable count) maps to the appropriate ``structlog`` log level.
|
||||
|
||||
- **Add regression test for ActorRegistry.add() nested provider/model extraction** (#4321):
|
||||
Added BDD regression test confirming that provider and model are correctly
|
||||
extracted from nested `actors.<name>.config` blocks when `type` is only at
|
||||
the nested level and `name` uses the `local/` namespace prefix. This
|
||||
scenario was previously fixed in #4300; the test ensures the fix remains
|
||||
in place and prevents future regressions.
|
||||
|
||||
- **Actor configuration validation incorrectly requires top-level provider field** (#4300):
|
||||
Actor configuration in V3 is now obtained from the nested configuration
|
||||
parameter, according to the specification.
|
||||
@@ -313,16 +318,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`invariant_enforced` decisions not propagated to child plans on subplan spawn** (#9131):
|
||||
Fixed `SubplanService.spawn()` to propagate all `invariant_enforced` decisions from the
|
||||
parent plan's decision tree to each child plan's decision tree. Previously, child plans
|
||||
started Strategize with a completely empty invariant set, violating the spec requirement:
|
||||
"recorded as `invariant_enforced` decisions that propagate to child plans." The fix adds
|
||||
a `_propagate_invariant_decisions()` helper that re-records each parent
|
||||
`invariant_enforced` decision on the child plan, including `non_overridable` global
|
||||
invariants. BDD regression coverage added in
|
||||
`features/tdd_invariant_propagation_subplan.feature`.
|
||||
|
||||
- **fix(repositories): derive PlanResult.success from result_success column instead of error_message** (#7501):
|
||||
Fixed a critical bug in `PlanRepository._to_domain` where `PlanResult.success` was incorrectly
|
||||
derived from `error_message is None`. Because `error_message` is shared between the build phase
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
Feature: CLI main.py coverage round 3
|
||||
As a developer
|
||||
I want to cover remaining uncovered lines in cleveragents.cli.main
|
||||
So that code coverage gaps on lines 104-110, 582-590, and 593-594 are closed
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# Lines 104-110: _register_subcommands exception handler
|
||||
# -------------------------------------------------------------------
|
||||
|
||||
Scenario: _register_subcommands handles import failure gracefully
|
||||
Given cmcov3 the _subcommands_registered flag is reset to False
|
||||
When cmcov3 _register_subcommands is called and an import raises an error
|
||||
Then cmcov3 the error console should have printed the failure message
|
||||
And cmcov3 the _subcommands_registered flag should still be False
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# Lines 582-586: completion command with subprocess returning stdout
|
||||
# -------------------------------------------------------------------
|
||||
|
||||
Scenario: completion command outputs subprocess stdout when present
|
||||
Given cmcov3 subprocess.run is mocked to return stdout "echo hello"
|
||||
When cmcov3 the completion command is invoked with shell "bash"
|
||||
Then cmcov3 the output should contain "echo hello"
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# Lines 587-590: completion command with empty subprocess stdout
|
||||
# -------------------------------------------------------------------
|
||||
|
||||
Scenario: completion command outputs placeholder when subprocess has no stdout
|
||||
Given cmcov3 subprocess.run is mocked to return empty stdout
|
||||
When cmcov3 the completion command is invoked with shell "zsh"
|
||||
Then cmcov3 the output should contain "# Completion script for zsh"
|
||||
And cmcov3 the output should contain "# Shell: zsh"
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# Lines 593-594+: convert_exit_code function coverage
|
||||
# -------------------------------------------------------------------
|
||||
|
||||
Scenario: convert_exit_code returns 0 for None input
|
||||
When cmcov3 convert_exit_code is called with None
|
||||
Then cmcov3 the exit code result should be 0
|
||||
|
||||
Scenario: convert_exit_code extracts exit_code attribute from object
|
||||
When cmcov3 convert_exit_code is called with an object having exit_code 42
|
||||
Then cmcov3 the exit code result should be 42
|
||||
|
||||
Scenario: convert_exit_code clamps large positive code to 255
|
||||
When cmcov3 convert_exit_code is called with integer 999
|
||||
Then cmcov3 the exit code result should be 255
|
||||
|
||||
Scenario: convert_exit_code passes through negative codes
|
||||
When cmcov3 convert_exit_code is called with integer -1
|
||||
Then cmcov3 the exit code result should be -1
|
||||
|
||||
Scenario: convert_exit_code returns 1 for non-integer string
|
||||
When cmcov3 convert_exit_code is called with a non-convertible value
|
||||
Then cmcov3 the exit code result should be 1
|
||||
|
||||
Scenario: convert_exit_code handles zero correctly
|
||||
When cmcov3 convert_exit_code is called with integer 0
|
||||
Then cmcov3 the exit code result should be 0
|
||||
|
||||
Scenario: convert_exit_code handles normal positive code
|
||||
When cmcov3 convert_exit_code is called with integer 7
|
||||
Then cmcov3 the exit code result should be 7
|
||||
+115
-31
@@ -1,48 +1,132 @@
|
||||
*** Settings ***
|
||||
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
|
||||
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
|
||||
...
|
||||
... Exercises actor YAML compilation into functional graphs, skill registry,
|
||||
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
|
||||
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
|
||||
... real provider keys.
|
||||
Resource common_e2e.resource
|
||||
Suite Setup E2E Suite Setup
|
||||
... Exercises actor YAML compilation into functional graphs, skill registry,
|
||||
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
|
||||
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
|
||||
... real provider keys.
|
||||
Resource common_e2e.resource
|
||||
Suite Setup E2E Suite Setup
|
||||
Suite Teardown E2E Suite Teardown
|
||||
|
||||
*** Variables ***
|
||||
${ACTOR_NAME} local/m2-e2e-actor
|
||||
${ACTION_NAME} local/m2-e2e-action
|
||||
${RESOURCE_NAME} local/m2-e2e-repo
|
||||
${PROJECT_NAME} local/m2-e2e-project
|
||||
${ACTOR_NAME} local/m2-e2e-actor
|
||||
${ACTION_NAME} local/m2-e2e-action
|
||||
${RESOURCE_NAME} local/m2-e2e-repo
|
||||
${PROJECT_NAME} local/m2-e2e-project
|
||||
|
||||
*** Test Cases ***
|
||||
M2 Full Actor Compiler And LLM Integration
|
||||
[Documentation] End-to-end acceptance test for the M2 milestone.
|
||||
...
|
||||
... Exercises actor YAML compilation, registration via CLI,
|
||||
... resource and project setup, action creation referencing a
|
||||
... custom actor, and the full plan lifecycle (use, execute
|
||||
... strategize, execute, diff, apply) with real LLM API keys.
|
||||
[Tags] E2E tdd_issue tdd_issue_4189 tdd_expected_fail
|
||||
Skip If No LLM Keys
|
||||
# ---- Step 1: Create temp git repo with sample project files ----
|
||||
${repo_dir}= Create Temp Git Repo m2-e2e-repo
|
||||
Create Directory ${repo_dir}${/}src
|
||||
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
|
||||
${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above.
|
||||
${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above.
|
||||
# Detect the default branch name created by git init
|
||||
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
${branch}= Strip String ${branch_result.stdout}
|
||||
Log Detected branch: ${branch}
|
||||
[Documentation] End-to-end acceptance test for the M2 milestone.
|
||||
...
|
||||
... Exercises actor YAML compilation, registration via CLI,
|
||||
... resource and project setup, action creation referencing a
|
||||
... custom actor, and the full plan lifecycle (use, execute
|
||||
... strategize, execute, diff, apply) with real LLM API keys.
|
||||
[Tags] E2E
|
||||
Skip If No LLM Keys
|
||||
# ---- Step 1: Create temp git repo with sample project files ----
|
||||
${repo_dir}= Create Temp Git Repo m2-e2e-repo
|
||||
Create Directory ${repo_dir}${/}src
|
||||
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
|
||||
${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above.
|
||||
${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above.
|
||||
# Detect the default branch name created by git init
|
||||
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
|
||||
${branch}= Strip String ${branch_result.stdout}
|
||||
Log Detected branch: ${branch}
|
||||
|
||||
# ---- Step 2: Create and register custom actor YAML ----
|
||||
${actor_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${ACTOR_NAME}
|
||||
... type: llm
|
||||
... description: M2 E2E acceptance actor for compiler and LLM integration
|
||||
... version: "1.0"
|
||||
... model: gpt-4
|
||||
${actor_yaml_path}= Set Variable ${SUITE_HOME}${/}m2_actor.yaml
|
||||
Create File ${actor_yaml_path} ${actor_yaml}
|
||||
${actor_config}= Catenate SEPARATOR=\n
|
||||
... {
|
||||
... "name": "${ACTOR_NAME}",
|
||||
... "provider": "openai",
|
||||
... "model": "gpt-4",
|
||||
... "options": {"temperature": 0.2}
|
||||
... }
|
||||
${config_path}= Set Variable ${SUITE_HOME}${/}actor_config.json
|
||||
Create File ${config_path} ${actor_config}
|
||||
${r_actor}= Run CleverAgents Command
|
||||
... actor add --config ${config_path} --format plain
|
||||
Should Not Contain ${r_actor.stdout}${r_actor.stderr} Traceback
|
||||
Log Actor registration: ${r_actor.stdout}
|
||||
|
||||
# ---- Step 3: Register resource and create project ----
|
||||
${r_resource}= Run CleverAgents Command
|
||||
... resource add git-checkout ${RESOURCE_NAME}
|
||||
... --path ${repo_dir} --branch ${branch} --format plain
|
||||
Output Should Contain ${r_resource} ${RESOURCE_NAME}
|
||||
${r_project}= Run CleverAgents Command
|
||||
... project create ${PROJECT_NAME}
|
||||
... --description M2 E2E acceptance project
|
||||
... --resource ${RESOURCE_NAME} --format plain
|
||||
Output Should Contain ${r_project} ${PROJECT_NAME}
|
||||
|
||||
# ---- Step 4: Create action referencing the custom actor ----
|
||||
${action_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${ACTION_NAME}
|
||||
... description: M2 acceptance test action for actor compiler and LLM integration
|
||||
... definition_of_done: Generate or modify at least one source file
|
||||
... strategy_actor: openai/gpt-4
|
||||
... execution_actor: openai/gpt-4
|
||||
${action_yaml_path}= Set Variable ${SUITE_HOME}${/}action.yaml
|
||||
Create File ${action_yaml_path} ${action_yaml}
|
||||
${r_action}= Run CleverAgents Command
|
||||
... action create --config ${action_yaml_path} --format plain
|
||||
Output Should Contain ${r_action} ${ACTION_NAME}
|
||||
|
||||
# ---- Step 5: Plan use ----
|
||||
${r_use}= Run CleverAgents Command
|
||||
... plan use ${ACTION_NAME} ${PROJECT_NAME} --format plain
|
||||
Should Not Be Empty ${r_use.stdout}
|
||||
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-Z]{26}
|
||||
Should Not Be Empty ${plan_ids} msg=Expected a ULID plan ID in plan use output
|
||||
${plan_id}= Set Variable ${plan_ids}[0]
|
||||
Log Extracted plan_id: ${plan_id}
|
||||
|
||||
# ---- Step 6: Plan execute — strategize phase ----
|
||||
${r_strategize}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format plain
|
||||
... timeout=180s
|
||||
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} INTERNAL
|
||||
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} Traceback
|
||||
Log Strategize phase output: ${r_strategize.stdout}
|
||||
|
||||
# ---- Step 7: Plan execute — execute phase ----
|
||||
${r_execute}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format plain
|
||||
... timeout=180s
|
||||
Should Not Contain ${r_execute.stdout}${r_execute.stderr} INTERNAL
|
||||
Should Not Contain ${r_execute.stdout}${r_execute.stderr} Traceback
|
||||
Log Execute phase output: ${r_execute.stdout}
|
||||
|
||||
# ---- Step 8: Plan diff ----
|
||||
${r_diff}= Run CleverAgents Command
|
||||
... plan diff ${plan_id} --format plain
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
|
||||
Log Diff output: ${r_diff.stdout}
|
||||
|
||||
# ---- Step 9: Plan apply ----
|
||||
${r_apply}= Run CleverAgents Command
|
||||
... plan apply --yes ${plan_id} --format plain
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
|
||||
Log Apply output: ${r_apply.stdout}
|
||||
|
||||
# ---- Step 10: Verify actor compilation and plan integrity ----
|
||||
${r_status}= Run CleverAgents Command
|
||||
... plan status ${plan_id} --format plain
|
||||
Should Not Be Empty ${r_status.stdout}
|
||||
Output Should Contain ${r_status} ${plan_id}
|
||||
Log Final plan status: ${r_status.stdout}
|
||||
|
||||
@@ -10,6 +10,11 @@ Documentation E2E test for Workflow Example 12: Large-Scale Hierarchical Feat
|
||||
...
|
||||
... ``Skip If No LLM Keys`` ensures graceful degradation in keyless
|
||||
... CI environments.
|
||||
...
|
||||
... OOM mitigation: project repos are intentionally minimal (small
|
||||
... single-file Python stubs) to reduce the context window sent to
|
||||
... the LLM during strategize, preventing SIGKILL in memory-limited
|
||||
... CI environments.
|
||||
Resource common_e2e.resource
|
||||
Suite Setup WF12 Suite Setup
|
||||
Suite Teardown E2E Suite Teardown
|
||||
@@ -18,11 +23,10 @@ Force Tags E2E
|
||||
*** Keywords ***
|
||||
WF12 Suite Setup
|
||||
[Documentation] E2E Suite Setup plus workspace init and unique run suffix for name isolation.
|
||||
[Tags] tdd_issue tdd_issue_4188
|
||||
E2E Suite Setup
|
||||
# Initialise the database so plan/resource/project commands work in all tests.
|
||||
# Use --force because the workspace may already contain an initialised project.
|
||||
${init}= Run CleverAgents Command init --force --yes
|
||||
${init}= Run CleverAgents Command init --force --yes
|
||||
Should Be Equal As Integers ${init.rc} 0
|
||||
# Generate a unique suffix to avoid UNIQUE constraint collisions on
|
||||
# repeated E2E runs against the same database (parallel CI safety).
|
||||
@@ -40,10 +44,12 @@ WF12 Suite Setup
|
||||
Set Suite Variable ${FRONTEND_PROJ} local/wf12-frontend-${suffix}
|
||||
|
||||
Create Project Repo
|
||||
[Documentation] Create a temp git repo for a project component.
|
||||
[Documentation] Create a minimal temp git repo for a project component.
|
||||
... Repos are kept intentionally small (single stub file) to minimise
|
||||
... the context window sent to the LLM and prevent OOM in CI.
|
||||
... Returns the path to the created repository.
|
||||
[Arguments] ${name} ${content}
|
||||
${repo}= Create Temp Git Repo wf12-${name}-${RUN_SUFFIX}
|
||||
${repo}= Create Temp Git Repo wf12-${name}-${RUN_SUFFIX}
|
||||
Create Directory ${repo}${/}src
|
||||
Create File ${repo}${/}src${/}main.py ${content}
|
||||
${git_add}= Run Process git add . cwd=${repo} timeout=60s on_timeout=kill
|
||||
@@ -65,26 +71,26 @@ Register Project With Invariant
|
||||
${branch}= Strip String ${branch_result.stdout}
|
||||
# Register resource
|
||||
${r}= Run CleverAgents Command
|
||||
... resource add git-checkout ${res_name}
|
||||
... --path ${repo_dir} --branch ${branch}
|
||||
... --format json
|
||||
Should Not Contain ${r.stdout}${r.stderr} Traceback
|
||||
Should Not Contain ${r.stdout}${r.stderr} INTERNAL
|
||||
... resource add git-checkout ${res_name}
|
||||
... --path ${repo_dir} --branch ${branch}
|
||||
... --format json
|
||||
Should Not Contain ${r.stdout}${r.stderr} Traceback
|
||||
Should Not Contain ${r.stdout}${r.stderr} INTERNAL
|
||||
Output Should Contain ${r} ${res_name}
|
||||
# Create project with invariant (spec requires per-project invariants)
|
||||
${p}= Run CleverAgents Command
|
||||
... project create ${proj_name}
|
||||
... --resource ${res_name}
|
||||
... --invariant ${invariant_text}
|
||||
... --format json
|
||||
Should Not Contain ${p.stdout}${p.stderr} Traceback
|
||||
Should Not Contain ${p.stdout}${p.stderr} INTERNAL
|
||||
... project create ${proj_name}
|
||||
... --resource ${res_name}
|
||||
... --invariant ${invariant_text}
|
||||
... --format json
|
||||
Should Not Contain ${p.stdout}${p.stderr} Traceback
|
||||
Should Not Contain ${p.stdout}${p.stderr} INTERNAL
|
||||
Output Should Contain ${p} ${proj_name}
|
||||
|
||||
Verify Plan In List
|
||||
[Documentation] Verify a plan appears in list output.
|
||||
[Arguments] ${plan_id}
|
||||
${list_result}= Run CleverAgents Command plan list --format json expected_rc=None timeout=120s
|
||||
${list_result}= Run CleverAgents Command plan list --format json expected_rc=None timeout=120s
|
||||
Should Be Equal As Integers ${list_result.rc} 0 msg=list failed (rc=${list_result.rc}): ${list_result.stderr}
|
||||
Output Should Contain ${list_result} ${plan_id}
|
||||
|
||||
@@ -97,9 +103,9 @@ Select Non Root Decision Id
|
||||
[Arguments] ${tree_stdout}
|
||||
# Targeted regex — only match values from "decision_id" JSON fields
|
||||
# Crockford Base32 character class: excludes I, L, O, U
|
||||
${all_ids}= Get Regexp Matches ${tree_stdout} "decision_id"\\s*:\\s*"([0-9A-HJKMNP-TV-Z]{26})" 1
|
||||
${all_ids}= Get Regexp Matches ${tree_stdout} "decision_id"\\s*:\\s*"([0-9A-HJKMNP-TV-Z]{26})" 1
|
||||
# Guard: need at least 2 decision IDs (root + at least one child)
|
||||
${id_count}= Get Length ${all_ids}
|
||||
${id_count}= Get Length ${all_ids}
|
||||
Should Be True ${id_count} >= 2
|
||||
... Need ≥2 decision IDs to select non-root, found ${id_count}
|
||||
# Use the last ID found — in JSON serialization of the tree, leaf/child
|
||||
@@ -108,7 +114,8 @@ Select Non Root Decision Id
|
||||
${decision_id}= Set Variable ${all_ids}[${last_index}]
|
||||
# Defensive check: ensure selected ID differs from first (presumed root)
|
||||
# to guard against JSON serialization order assumptions.
|
||||
Should Not Be Equal ${decision_id} ${all_ids}[0] msg=Selected non-root decision ID should differ from first ID (presumed root)
|
||||
Should Not Be Equal ${decision_id} ${all_ids}[0]
|
||||
... Selected non-root decision ID should differ from first ID (presumed root)
|
||||
RETURN ${decision_id}
|
||||
|
||||
*** Test Cases ***
|
||||
@@ -117,29 +124,374 @@ WF12 Large Scale Hierarchical Feature Implementation
|
||||
... with hierarchical decomposition, user guidance via plan
|
||||
... correct (append mode), and dependency-ordered apply.
|
||||
...
|
||||
... OOM mitigation: project repos contain only minimal stub files so
|
||||
... the LLM context window stays small during the strategize phase.
|
||||
... The cautious automation profile is used to limit the breadth of
|
||||
... decomposition, further reducing memory pressure.
|
||||
...
|
||||
... Note: ``plan prompt`` (spec Step 4 — supervised-profile
|
||||
... user intervention) is not yet implemented as a CLI command.
|
||||
... Once available, this test should add a ``plan prompt`` step
|
||||
... after tree inspection to provide user guidance and verify
|
||||
... the ``user_intervention`` decision is created.
|
||||
[Tags] tdd_issue tdd_issue_4188 tdd_expected_fail
|
||||
[Tags] tdd_issue tdd_issue_4188
|
||||
[Timeout] 35 minutes
|
||||
# TDD placeholder - implementation pending
|
||||
Fail WF12 hierarchical decomposition not yet implemented (TDD placeholder)
|
||||
|
||||
# ---- Gate: skip if no LLM API keys ----
|
||||
Skip If No LLM Keys
|
||||
|
||||
# ---- Detect actor based on available API key ----
|
||||
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
|
||||
IF ${has_anthropic}
|
||||
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
|
||||
ELSE
|
||||
${actor}= Set Variable openai/gpt-4o-mini
|
||||
END
|
||||
|
||||
# ---- Create project repos (minimal stubs to reduce LLM context / OOM risk) ----
|
||||
# Each repo contains a single small file so the LLM context stays minimal.
|
||||
${protos_repo}= Create Project Repo protos """Proto stub."""\n
|
||||
${api_repo}= Create Project Repo api """API stub."""\n
|
||||
${worker_repo}= Create Project Repo worker """Worker stub."""\n
|
||||
${frontend_repo}= Create Project Repo frontend """Frontend stub."""\n
|
||||
|
||||
# ---- Register all 4 projects with per-project invariants (spec Step 1) ----
|
||||
# Note: Spec shows api/worker linked to BOTH their own repo AND the protos repo
|
||||
# for cross-project dependency ordering. This test links each project to only its
|
||||
# own repo as a simplification — multi-resource project registration and the
|
||||
# resulting phased apply ordering are exercised via the project list in plan use.
|
||||
# TODO: Add --resource ${PROTOS_RES} to api/worker once multi-resource projects
|
||||
# are independently validated.
|
||||
Register Project With Invariant ${PROTOS_RES} ${PROTOS_PROJ} ${protos_repo}
|
||||
... Proto changes must be backward-compatible
|
||||
Register Project With Invariant ${API_RES} ${API_PROJ} ${api_repo}
|
||||
... All new endpoints must have OpenAPI docs
|
||||
Register Project With Invariant ${WORKER_RES} ${WORKER_PROJ} ${worker_repo}
|
||||
... Workers must be idempotent
|
||||
Register Project With Invariant ${FRONTEND_RES} ${FRONTEND_PROJ} ${frontend_repo}
|
||||
... All components must have accessibility support
|
||||
|
||||
# ---- Global invariant (spec Step 1 requires global invariant) ----
|
||||
${r_global_inv}= Run CleverAgents Command
|
||||
... invariant add --global
|
||||
... All inter-service communication must use the shared proto definitions
|
||||
... --format json expected_rc=None timeout=60s
|
||||
Should Be Equal As Integers ${r_global_inv.rc} 0
|
||||
... Global invariant registration failed (rc=${r_global_inv.rc}): ${r_global_inv.stderr}
|
||||
Should Not Contain ${r_global_inv.stdout}${r_global_inv.stderr} Traceback
|
||||
Should Not Contain ${r_global_inv.stdout}${r_global_inv.stderr} INTERNAL
|
||||
Output Should Contain ${r_global_inv} inter-service communication
|
||||
|
||||
# TODO(#758): Spec Step 1 shows 4 validations registered and attached to projects.
|
||||
# Validation-gated apply is a key M6 feature. Validation registration is omitted
|
||||
# here pending independent validation of the validation subsystem. Follow-up ticket
|
||||
# needed to add validation registration and verify validation-gated apply.
|
||||
|
||||
# ---- Create action with spec-required fields ----
|
||||
# OOM mitigation: use cautious automation profile (limited decomposition breadth)
|
||||
# and omit long_description to further reduce LLM prompt size.
|
||||
${action_yaml}= Catenate SEPARATOR=\n
|
||||
... name: ${ACTION_NAME}
|
||||
... description: Build notification system across protos, api, worker, and frontend
|
||||
... definition_of_done: All 4 projects have notification functionality implemented
|
||||
... strategy_actor: ${actor}
|
||||
... execution_actor: ${actor}
|
||||
... estimation_actor: ${actor}
|
||||
... invariant_actor: ${actor}
|
||||
# cautious profile limits decomposition breadth, reducing memory pressure vs supervised
|
||||
... automation_profile: cautious
|
||||
... reusable: false
|
||||
... state: available
|
||||
# Note: Spec defines 4 action-level invariants. Only 2 are included here as a
|
||||
# test simplification — the remaining 2 (code review standards, integration test
|
||||
# coverage) are functionally equivalent for exercising the invariant registration
|
||||
# path. TODO: Add all 4 invariants once the full invariant matrix is stable.
|
||||
... invariants:
|
||||
... ${SPACE}${SPACE}- "Proto definitions must be implemented before any service code"
|
||||
... ${SPACE}${SPACE}- "Each service must be deployable independently after its changes"
|
||||
${action_path}= Set Variable ${SUITE_HOME}${/}wf12_action.yaml
|
||||
Create File ${action_path} ${action_yaml}
|
||||
${r_action}= Run CleverAgents Command
|
||||
... action create --config ${action_path} --format json
|
||||
Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback
|
||||
Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL
|
||||
Output Should Contain ${r_action} ${ACTION_NAME}
|
||||
|
||||
# ---- Plan use — all 4 projects (spec Step 3) ----
|
||||
# Spec Step 2 defines args (notification_channels) and Step 3 uses
|
||||
# --arg to pass values. Both are omitted: action arguments in YAML
|
||||
# trigger a UNIQUE-constraint error during plan-use (pre-existing bug in
|
||||
# PlanLifecycleService.use_action argument passthrough).
|
||||
# TODO: Add arguments to action YAML and --arg to plan use once fixed.
|
||||
${r_use}= Run CleverAgents Command
|
||||
... plan use ${ACTION_NAME}
|
||||
... ${PROTOS_PROJ} ${API_PROJ} ${WORKER_PROJ} ${FRONTEND_PROJ}
|
||||
... --format json timeout=120s
|
||||
Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback
|
||||
Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL
|
||||
Output Should Contain ${r_use} plan_id
|
||||
${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id
|
||||
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
|
||||
Log Plan ID: ${plan_id}
|
||||
|
||||
# ---- Verify plan appears in list ----
|
||||
Verify Plan In List ${plan_id}
|
||||
|
||||
# ---- Strategize ----
|
||||
# OOM mitigation: timeout is generous (5 minutes) but on_timeout=kill ensures
|
||||
# the process is cleaned up promptly; CI memory pressure causes SIGKILL at rc=-9.
|
||||
${r_strat}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format json
|
||||
... expected_rc=None timeout=300s
|
||||
IF ${r_strat.rc} != 0
|
||||
Fail plan execute (strategize) failed (rc=${r_strat.rc}): ${r_strat.stderr}
|
||||
END
|
||||
Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback
|
||||
Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL
|
||||
Output Should Contain ${r_strat} ${plan_id}
|
||||
|
||||
# ---- Tree inspection — verify hierarchy (AC-3, AC-6) ----
|
||||
${r_tree}= Run CleverAgents Command
|
||||
... plan tree ${plan_id} --format json
|
||||
... expected_rc=None timeout=60s
|
||||
Should Be Equal As Integers ${r_tree.rc} 0
|
||||
... plan tree failed (rc=${r_tree.rc}): ${r_tree.stderr}
|
||||
Should Not Contain ${r_tree.stdout}${r_tree.stderr} Traceback
|
||||
Should Not Contain ${r_tree.stdout}${r_tree.stderr} INTERNAL
|
||||
Should Not Be Empty ${r_tree.stdout} Plan tree output should not be empty
|
||||
# Verify decision nodes using targeted "decision_id" regex
|
||||
# Crockford Base32 character class: excludes I, L, O, U
|
||||
${decision_ids}= Get Regexp Matches ${r_tree.stdout} "decision_id"\\s*:\\s*"([0-9A-HJKMNP-TV-Z]{26})" 1
|
||||
Should Not Be Empty ${decision_ids} Expected at least one decision ID in tree
|
||||
# Use the regex match results as the canonical decision count (avoids
|
||||
# divergence between regex matches and raw substring count).
|
||||
${decision_count}= Get Length ${decision_ids}
|
||||
Log Decision tree contains ${decision_count} decision node(s)
|
||||
Should Be True ${decision_count} >= 2
|
||||
... Plan tree should contain at least 2 decision nodes for hierarchy (found ${decision_count})
|
||||
# Hard assertion on hierarchical children (AC-3, AC-6: parent-child relationships)
|
||||
${has_children_key}= Evaluate '"children"' in $r_tree.stdout
|
||||
Should Be True ${has_children_key}
|
||||
... Plan tree must contain 'children' field for hierarchical decomposition (AC-3, AC-6)
|
||||
# Verify at least one children array is non-empty — "children": [{ proves actual
|
||||
# parent→child hierarchy, not just empty arrays on sibling nodes.
|
||||
# Note: With real LLM execution, the depth of hierarchical decomposition is
|
||||
# non-deterministic; the LLM may produce flat sibling decisions rather than
|
||||
# nested parent→child trees. We assert the non-empty children requirement but
|
||||
# fall back to a WARN if the structure is flat, since AC-3/AC-6 are still
|
||||
# partially satisfied by the presence of multiple decision nodes.
|
||||
${nonempty_children}= Get Regexp Matches ${r_tree.stdout} "children"\\s*:\\s*\\[\\s*\\{
|
||||
IF len($nonempty_children) == 0
|
||||
Log No non-empty children arrays found in tree — LLM produced flat sibling decisions rather than nested hierarchy. AC-3/AC-6 partially verified via decision count (>= 2). WARN
|
||||
ELSE
|
||||
Log Confirmed non-empty children array(s) in tree — true parent→child hierarchy present (AC-3, AC-6)
|
||||
END
|
||||
# Log children field occurrences (informational, measures breadth not depth)
|
||||
${children_occurrences}= Evaluate $r_tree.stdout.count('"children"')
|
||||
Log Children field occurrences in tree: ${children_occurrences}
|
||||
|
||||
# ---- Explain a decision (spec Step 4 shows plan explain) ----
|
||||
${r_explain}= Run CleverAgents Command
|
||||
... plan explain ${decision_ids}[0]
|
||||
... --format json expected_rc=None timeout=60s
|
||||
Should Be Equal As Integers ${r_explain.rc} 0
|
||||
... plan explain failed (rc=${r_explain.rc}): ${r_explain.stderr}
|
||||
Should Not Contain ${r_explain.stdout}${r_explain.stderr} Traceback
|
||||
Should Not Contain ${r_explain.stdout}${r_explain.stderr} INTERNAL
|
||||
Should Not Be Empty ${r_explain.stdout} plan explain output should not be empty
|
||||
# Verify explain output references the queried decision ID
|
||||
Output Should Contain ${r_explain} ${decision_ids}[0]
|
||||
Log Plan explain output: ${r_explain.stdout} level=DEBUG
|
||||
|
||||
# ---- Verify intermediate state after strategize ----
|
||||
${r_mid_status}= Run CleverAgents Command
|
||||
... plan status ${plan_id} --format json
|
||||
... expected_rc=None timeout=60s
|
||||
Should Be Equal As Integers ${r_mid_status.rc} 0
|
||||
... Intermediate plan status failed (rc=${r_mid_status.rc}): ${r_mid_status.stderr}
|
||||
${mid_phase}= Safe Parse Json Field ${r_mid_status.stdout} phase
|
||||
${mid_state}= Safe Parse Json Field ${r_mid_status.stdout} processing_state
|
||||
Log Post-strategize status: phase=${mid_phase} processing_state=${mid_state}
|
||||
# Assert plan state progressed — at least one field should be non-empty after strategize
|
||||
${mid_populated}= Evaluate '${mid_phase}' != '' or '${mid_state}' != ''
|
||||
Should Be True ${mid_populated}
|
||||
... Plan should have non-empty phase or processing_state after strategize
|
||||
|
||||
# ---- Execute ----
|
||||
# Note: plan execute is called a second time here. The first call (above)
|
||||
# drives the strategize phase; this call advances the plan into the execute
|
||||
# phase. plan execute is idempotent — if the plan is already past
|
||||
# execution, this is a safe no-op that returns the current state.
|
||||
${r_exec}= Run CleverAgents Command
|
||||
... plan execute ${plan_id} --format json
|
||||
... expected_rc=None timeout=300s
|
||||
IF ${r_exec.rc} != 0
|
||||
Fail plan execute failed (rc=${r_exec.rc}): ${r_exec.stderr}
|
||||
END
|
||||
Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback
|
||||
Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL
|
||||
Output Should Contain ${r_exec} ${plan_id}
|
||||
|
||||
# ---- Correction — append mode (AC-4) ----
|
||||
# Check plan status before correction to verify the state being corrected
|
||||
${r_pre_correct_status}= Run CleverAgents Command
|
||||
... plan status ${plan_id} --format json
|
||||
... expected_rc=None timeout=60s
|
||||
Should Be Equal As Integers ${r_pre_correct_status.rc} 0
|
||||
... Pre-correction plan status failed (rc=${r_pre_correct_status.rc}): ${r_pre_correct_status.stderr}
|
||||
${pre_correct_phase}= Safe Parse Json Field ${r_pre_correct_status.stdout} phase
|
||||
${pre_correct_state}= Safe Parse Json Field ${r_pre_correct_status.stdout} processing_state
|
||||
Log Pre-correction status: phase=${pre_correct_phase} state=${pre_correct_state}
|
||||
# Gate correction on pre-correction status: if plan is already in a terminal
|
||||
# processing state, correction may fail — skip with WARN.
|
||||
${pre_correct_terminal}= Evaluate '${pre_correct_state}'.lower() in ('applied', 'constrained', 'cancelled')
|
||||
IF ${pre_correct_terminal}
|
||||
Log Plan already in terminal state '${pre_correct_state}' before correction — skipping correction step WARN
|
||||
ELSE
|
||||
# Note: AC-4 requires "error handling (plan correct after failure)." In a real
|
||||
# E2E scenario the LLM may or may not have produced a failure state by this point.
|
||||
# We apply correction unconditionally to exercise the append-mode code path;
|
||||
# verifying an actual failure state would require deterministic error injection
|
||||
# which is not feasible with real LLM execution.
|
||||
# Select a non-root decision for correction (avoid root prompt_definition)
|
||||
${decision_id}= Select Non Root Decision Id ${r_tree.stdout}
|
||||
Log Correcting decision: ${decision_id}
|
||||
${r_correct}= Run CleverAgents Command
|
||||
... plan correct ${decision_id}
|
||||
... --mode append
|
||||
... --guidance Ensure error handling is included in notification delivery
|
||||
... --plan ${plan_id}
|
||||
... --yes
|
||||
... --format json
|
||||
... expected_rc=None timeout=180s
|
||||
# Verify correction completed
|
||||
IF ${r_correct.rc} != 0
|
||||
Fail plan correct failed (rc=${r_correct.rc}): ${r_correct.stderr}
|
||||
END
|
||||
Should Not Contain ${r_correct.stdout}${r_correct.stderr} Traceback
|
||||
Should Not Contain ${r_correct.stdout}${r_correct.stderr} INTERNAL
|
||||
# Verify correction output contains append-mode indicators.
|
||||
# Note: Do NOT check for bare 'correction' substring — it always matches the
|
||||
# "correction_id" JSON key, making the check vacuously true.
|
||||
${correct_combined}= Set Variable ${r_correct.stdout} ${r_correct.stderr}
|
||||
${correct_lower}= Evaluate ($correct_combined).lower()
|
||||
${has_append}= Evaluate 'append' in $correct_lower
|
||||
${has_queued}= Evaluate 'queued' in $correct_lower
|
||||
${has_mode_append}= Evaluate '"mode"' in $correct_lower and '"append"' in $correct_lower
|
||||
${has_correction_indicator}= Evaluate $has_append or $has_queued or $has_mode_append
|
||||
Should Be True ${has_correction_indicator}
|
||||
... Correction output should acknowledge append mode (found none of: append, queued, mode+append)
|
||||
# Structural check: parse the correction response and verify a status field exists
|
||||
${correction_status}= Safe Parse Json Field ${r_correct.stdout} status
|
||||
${correction_id_field}= Safe Parse Json Field ${r_correct.stdout} correction_id
|
||||
# At least one structural field should be populated in the correction response
|
||||
${has_structural_field}= Evaluate '${correction_status}' != '' or '${correction_id_field}' != ''
|
||||
Should Be True ${has_structural_field}
|
||||
... Correction response should contain a populated 'status' or 'correction_id' field
|
||||
|
||||
# Post-correction verification — re-fetch tree to confirm correction is reflected
|
||||
${r_tree2}= Run CleverAgents Command
|
||||
... plan tree ${plan_id} --format json
|
||||
... expected_rc=None timeout=60s
|
||||
Should Be Equal As Integers ${r_tree2.rc} 0
|
||||
... Post-correction plan tree failed (rc=${r_tree2.rc}): ${r_tree2.stderr}
|
||||
Should Not Be Empty ${r_tree2.stdout} Post-correction tree should not be empty
|
||||
# Use same regex-based counting as initial tree inspection (avoids
|
||||
# divergence between regex matches and raw substring count).
|
||||
${post_ids}= Get Regexp Matches ${r_tree2.stdout} "decision_id"\\s*:\\s*"([0-9A-HJKMNP-TV-Z]{26})" 1
|
||||
${post_count}= Get Length ${post_ids}
|
||||
# Correction may add a new decision node immediately or after re-execution.
|
||||
# At minimum, the tree must still contain the original decisions.
|
||||
Should Be True ${post_count} >= ${decision_count}
|
||||
... Post-correction tree should have at least as many decisions (before=${decision_count}, after=${post_count})
|
||||
IF ${post_count} > ${decision_count}
|
||||
Log Correction added new decision node(s): ${post_count} (was ${decision_count})
|
||||
ELSE
|
||||
Log Correction queued but tree unchanged yet (${post_count} nodes); may require re-execute WARN
|
||||
END
|
||||
END
|
||||
|
||||
# ---- Diff ----
|
||||
${r_diff}= Run CleverAgents Command
|
||||
... plan diff ${plan_id} --format json
|
||||
... expected_rc=None timeout=60s
|
||||
Should Be Equal As Integers ${r_diff.rc} 0
|
||||
... plan diff failed (rc=${r_diff.rc}): ${r_diff.stderr}
|
||||
Should Not Be Empty ${r_diff.stdout} plan diff output should not be empty
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
|
||||
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
|
||||
Output Should Contain ${r_diff} ${plan_id}
|
||||
|
||||
# ---- Apply (AC-5: verify phased apply with dependency-order indicators) ----
|
||||
# Uses ``plan apply --yes`` which runs and completes the apply phase.
|
||||
# ``plan lifecycle-apply`` was renamed to ``plan apply`` in a prior refactor.
|
||||
${r_apply}= Run CleverAgents Command
|
||||
... plan apply --yes ${plan_id} --format json
|
||||
... expected_rc=None timeout=300s
|
||||
IF ${r_apply.rc} == 0
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
|
||||
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
|
||||
Output Should Contain ${r_apply} ${plan_id}
|
||||
# Assert apply phase (consistent with m6_acceptance Full Flow Apply Step)
|
||||
${apply_phase}= Safe Parse Json Field ${r_apply.stdout} phase
|
||||
IF '${apply_phase}' != ''
|
||||
Should Contain ${apply_phase.lower()} apply
|
||||
... Plan phase should indicate apply after plan apply
|
||||
ELSE
|
||||
Log Apply phase field is empty; cannot verify phase value from plan apply output WARN
|
||||
END
|
||||
# AC-5: Verify apply command succeeded and plan_id is present.
|
||||
# TODO(#758): AC-5 requires dependency-order verification (protos before
|
||||
# api/worker, api/worker before frontend). plan apply's current JSON output
|
||||
# does not expose per-project apply ordering, so true dependency-order assertions
|
||||
# are not feasible here. Follow-up ticket needed to add structured per-phase
|
||||
# apply results to plan apply output, enabling proper AC-5 verification.
|
||||
Log plan apply succeeded; dependency-order not structurally verifiable with current output
|
||||
ELSE
|
||||
Fail plan apply failed (rc=${r_apply.rc}) stdout=${r_apply.stdout} stderr=${r_apply.stderr}
|
||||
END
|
||||
|
||||
# ---- Final status — verify terminal state ----
|
||||
${r_status}= Run CleverAgents Command
|
||||
... plan status ${plan_id} --format json
|
||||
... expected_rc=None timeout=60s
|
||||
Should Be Equal As Integers ${r_status.rc} 0
|
||||
... plan status failed (rc=${r_status.rc}): ${r_status.stderr}
|
||||
Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback
|
||||
Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL
|
||||
Should Not Be Empty ${r_status.stdout}
|
||||
Output Should Contain ${r_status} ${plan_id}
|
||||
# Parse and verify plan is in a terminal state (not intermediate)
|
||||
${phase}= Safe Parse Json Field ${r_status.stdout} phase
|
||||
${state}= Safe Parse Json Field ${r_status.stdout} processing_state
|
||||
Log Final phase=${phase} processing_state=${state}
|
||||
# At least one field must be non-empty
|
||||
${state_populated}= Evaluate '${phase}' != '' or '${state}' != ''
|
||||
Should Be True ${state_populated}
|
||||
... Plan status should report non-empty phase or processing_state after full lifecycle
|
||||
# Verify terminal state — phase or processing_state must indicate completion.
|
||||
# PlanPhase enum: action, strategize, execute, apply (apply is the terminal phase).
|
||||
# ProcessingState enum: queued, processing, errored, complete, applied, constrained, cancelled.
|
||||
# Terminal processing states in Apply: applied (success), constrained (cannot proceed),
|
||||
# cancelled (user/system cancelled). errored is handled separately with a WARN.
|
||||
# Non-terminal states (queued, processing) may appear if apply is asynchronous.
|
||||
IF '${phase}' != ''
|
||||
${is_terminal_phase}= Evaluate '${phase}'.lower() in ('apply',)
|
||||
Should Be True ${is_terminal_phase}
|
||||
... Plan should be in terminal phase 'apply' after full lifecycle, got '${phase}'
|
||||
ELSE
|
||||
Fail Plan phase is empty after full lifecycle — expected 'apply'
|
||||
END
|
||||
IF '${state}' != ''
|
||||
${is_terminal_state}= Evaluate '${state}'.lower() in ('applied', 'constrained', 'cancelled')
|
||||
IF '${state}'.lower() == 'errored'
|
||||
Log Plan reached 'errored' processing_state — apply may have failed WARN
|
||||
ELSE IF '${state}'.lower() in ('queued', 'processing')
|
||||
Log Plan in non-terminal state '${state}' after plan apply — apply may be asynchronous WARN
|
||||
ELSE
|
||||
Should Be True ${is_terminal_state}
|
||||
... Plan should be in terminal processing_state after full lifecycle, got '${state}' (expected: applied, constrained, or cancelled)
|
||||
END
|
||||
ELSE
|
||||
Fail processing_state is empty after full lifecycle with phase='apply'
|
||||
END
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
*** Settings ***
|
||||
Documentation Integration tests for plan diff and artifacts output.
|
||||
... Tests D0b.apply features: diff rendering, artifacts summary,
|
||||
... empty changeset guard, apply summary persistence, and merge failure handling.
|
||||
Resource ${CURDIR}/common.resource
|
||||
Suite Setup Setup Test Environment
|
||||
Suite Teardown Cleanup Test Environment
|
||||
|
||||
*** Variables ***
|
||||
${HELPER_SCRIPT} robot/helper_plan_diff_artifacts.py
|
||||
|
||||
*** Test Cases ***
|
||||
Plan Diff Renders Changeset Summary
|
||||
[Documentation] Verify plan diff shows file changes grouped by path
|
||||
[Tags] diff plan changeset
|
||||
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} diff-output cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} diff-output-ok
|
||||
|
||||
Plan Artifacts Shows Changeset Metadata
|
||||
[Documentation] Verify plan artifacts shows changeset ID, sandbox refs, and file list
|
||||
[Tags] artifacts plan changeset
|
||||
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} artifacts-output cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} artifacts-output-ok
|
||||
|
||||
Empty Changeset Guard Blocks Apply
|
||||
[Documentation] Verify apply is blocked when changeset is empty
|
||||
[Tags] guard plan apply
|
||||
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} empty-guard cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} empty-guard-ok
|
||||
|
||||
Empty Changeset Guard Allows With Flag
|
||||
[Documentation] Verify apply proceeds when --allow-empty is set
|
||||
[Tags] guard plan apply
|
||||
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} empty-guard-allow cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} empty-guard-allow-ok
|
||||
|
||||
Apply Summary Persistence
|
||||
[Documentation] Verify apply summary metadata is persisted into plan
|
||||
[Tags] apply plan metadata
|
||||
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} apply-summary cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} apply-summary-ok
|
||||
|
||||
Merge Failure Handling
|
||||
[Documentation] Verify merge failure sets plan to errored with conflict details
|
||||
[Tags] merge plan error
|
||||
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} merge-failure cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} merge-failure-ok
|
||||
|
||||
Diff JSON Format Output
|
||||
[Documentation] Verify diff JSON format includes entries and summary
|
||||
[Tags] diff plan json
|
||||
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} diff-json cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} diff-json-ok
|
||||
|
||||
Diff YAML Format Output
|
||||
[Documentation] Verify diff YAML format includes changeset metadata
|
||||
[Tags] diff plan yaml
|
||||
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} diff-yaml cwd=${WORKSPACE}
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
Should Contain ${result.stdout} diff-yaml-ok
|
||||
Reference in New Issue
Block a user