Compare commits

...

4 Commits

Author SHA1 Message Date
brent.edwards d4cbbaa41f fix(e2e): use Extract JSON From Stdout for WF05 decision tree parsing
WF05 was manually parsing the plan tree JSON output without unwrapping
the CLI envelope, causing the decision_id count to return 0. The envelope
wraps the decision array in a 'data' key.

Updated WF05 to use Extract JSON From Stdout which automatically unwraps
the envelope, ensuring the decision walker operates on the actual decision
array instead of the envelope structure.

Also added a documentation note to Extract JSON From Stdout explaining
which tests need refactoring to use this keyword instead of custom JSON
extraction logic.
2026-04-26 17:41:39 +00:00
brent.edwards 8bce7f45d0 fix(e2e): restore M5/WF14 tests broken by JSON envelope and stale config
Two independent root causes were causing 10 non-quota E2E failures.

**M5 acceptance (9 tests + 1 cascade):**
`format_output(..., "json")` wraps all CLI JSON output in the
spec-required envelope `{"command":…,"status":"ok","exit_code":0,
"data":{…},…}`.  The M5 tests were looking for payload fields
(`total_tokens`, `resolved_view`, `acms_config`, `tier_metrics`,
`plan_id`) at the top level of the extracted JSON object, but they
live inside `data`.

Fix: update `Extract JSON From Stdout` in `common_e2e.resource` to
auto-unwrap the envelope when both `exit_code` and `data` keys are
present, so callers receive the payload dict transparently.  Output
from helper scripts that do not use the CLI envelope (e.g. the WF04
snapshot helper) is returned unchanged.

Also includes the already-staged m5_acceptance.robot rework that fixed
the argument-passing bug (args were previously joined into single
space-containing strings instead of passed individually to Run CLI),
removed stale `tdd_expected_fail` tags, and standardised indentation.

**WF14 server mode (1 test):**
`~/.cleveragents/config.toml` is shared across all parallel pabot
workers and across nox sessions (the `CLEVERAGENTS_HOME` env var
isolates the SQLite DB but not the config file path, which is
hardcoded to `Path.home() / ".cleveragents"`).  A previous
`server_stubs.robot` run wrote `server.url = "https://stub.example.com"`
to that file.  The `config set` CLI command could not overwrite the
entry because tomllib parses TOML dotted keys as nested dicts
(`{"server": {"url": "…"}}`), causing `config set` to write a new
flat key alongside the original nested entry rather than replacing it.

Fix: add `WF14 Clean Global Config` — a keyword that uses
Python/tomlkit directly to remove `server.url`, `server.token`, and
`server.namespace` from the global config TOML, handling both nested
(`[server]` table) and flat quoted-key representations.  The keyword
is called from both `WF14 Suite Setup` (so any stale value is removed
before the test's `config set` runs) and `WF14 Suite Teardown` (so
subsequent runs start clean).  Failures are silently ignored — a
missing config file is normal on a fresh environment.

ISSUES CLOSED: #8459
2026-04-26 17:41:39 +00:00
brent.edwards 84f5d6eb6b fix(plan): wire sandbox_root into plan execute pipeline
Fixes three sub-bugs that caused LLM-generated files to be silently lost
during plan execute:

Bug 1 — Duplicate sandbox branch on second execute call:
_create_sandbox_for_plan() was called before phase detection, causing a
fatal duplicate-branch error when plan execute was invoked a second time on
a plan already in Execute/complete.  Moved sandbox creation inside the
Execute/QUEUED block so repeated invocations skip it safely.

Bug 2 — Changeset entries discarded after execute:
The lightweight ChangeSet from LLMExecuteActor was never persisted.  Now
serialised to plan.error_details["changeset_entries_json"] by
PlanExecutor._run_execute_with_stub(); PlanApplyService._resolve_changeset()
reconstructs a SpecChangeSet from this metadata when no changeset_store is
wired, so plan diff shows the generated file changes.

Bug 3 — Wrong output format for plan execute --format plain:
execute_plan() used the now-removed _execute_output_dict (envelope format)
instead of _plan_spec_dict, inconsistent with plan use / plan apply.
Switched to _plan_spec_dict; removed the dead _execute_output_dict function.

Also removes tdd_expected_fail from the M1 acceptance E2E test and adds 6
new BDD regression scenarios (@tdd_issue_1313) covering all three fixes.

ISSUES CLOSED: #1313
2026-04-26 17:41:13 +00:00
brent.edwards d81b5853a8 fix(e2e): restore M5/WF14 tests broken by JSON envelope and stale config
Two independent root causes were causing 10 non-quota E2E failures.

**M5 acceptance (9 tests + 1 cascade):**
`format_output(..., "json")` wraps all CLI JSON output in the
spec-required envelope `{"command":…,"status":"ok","exit_code":0,
"data":{…},…}`.  The M5 tests were looking for payload fields
(`total_tokens`, `resolved_view`, `acms_config`, `tier_metrics`,
`plan_id`) at the top level of the extracted JSON object, but they
live inside `data`.

Fix: update `Extract JSON From Stdout` in `common_e2e.resource` to
auto-unwrap the envelope when both `exit_code` and `data` keys are
present, so callers receive the payload dict transparently.  Output
from helper scripts that do not use the CLI envelope (e.g. the WF04
snapshot helper) is returned unchanged.

Also includes the already-staged m5_acceptance.robot rework that fixed
the argument-passing bug (args were previously joined into single
space-containing strings instead of passed individually to Run CLI),
removed stale `tdd_expected_fail` tags, and standardised indentation.

**WF14 server mode (1 test):**
`~/.cleveragents/config.toml` is shared across all parallel pabot
workers and across nox sessions (the `CLEVERAGENTS_HOME` env var
isolates the SQLite DB but not the config file path, which is
hardcoded to `Path.home() / ".cleveragents"`).  A previous
`server_stubs.robot` run wrote `server.url = "https://stub.example.com"`
to that file.  The `config set` CLI command could not overwrite the
entry because tomllib parses TOML dotted keys as nested dicts
(`{"server": {"url": "…"}}`), causing `config set` to write a new
flat key alongside the original nested entry rather than replacing it.

Fix: add `WF14 Clean Global Config` — a keyword that uses
Python/tomlkit directly to remove `server.url`, `server.token`, and
`server.namespace` from the global config TOML, handling both nested
(`[server]` table) and flat quoted-key representations.  The keyword
is called from both `WF14 Suite Setup` (so any stale value is removed
before the test's `config set` runs) and `WF14 Suite Teardown` (so
subsequent runs start clean).  Failures are silently ignored — a
missing config file is normal on a fresh environment.

ISSUES CLOSED: #8459
2026-04-26 17:40:43 +00:00
29 changed files with 7576 additions and 3297 deletions
+216 -180
View File
@@ -5,66 +5,42 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
## [Unreleased]
### Changed
- **Diagnostics spec examples expanded to all 9 providers** (#5320): Updated the
`agents diagnostics` command examples in the specification to show all 9 supported
providers (OpenAI, Anthropic, Google, Gemini, Azure, OpenRouter, Cohere, Groq,
Together), matching the implementation from PR #3469. Rich, plain, JSON, and YAML
example outputs now reflect comprehensive provider coverage with accurate warning
counts and per-provider recommendations.
### Added
- **Architecture Pool Supervisor Milestone Assignment** (#7521): Added a "PR Workflow
for Major Changes" section to the `architecture-pool-supervisor` agent definition
documenting the milestone assignment step for spec PRs. The agent now has
`forgejo_update_pull_request` permission to assign PRs to the current active
milestone after creation, improving traceability of specification changes within
project milestone planning. Includes BDD test coverage for the new workflow
documentation and permission configuration.
### Fixed
- **Trusted Automation Profile Description** (#9156): Corrected the `trusted` built-in automation profile description from `"Auto for most, human for apply and revert"` to `"Auto-exec, manual apply. Day-to-day development"` to match the specification at line 17197.
- **Actor v3 YAML Schema Validation in CLI** (#5869): The `agents actor add --config`
command now validates v3 YAML files using `ActorConfigSchema`, ensuring proper
schema compliance including cycle detection for GRAPH actors, required field
validation, and enum validation. v3 YAML is detected by the presence of ANY
`type` field (any value — invalid type values are then rejected by schema
validation) or a `version` field whose string value starts with `"3"` (e.g.
`"3"`, `"3.0"`, `"3.0.0"`). Configs with `type: null` are not treated as v3.
Invalid v3 actors are rejected with clear error messages before registration.
- **Atomic `load_from_metadata` for Autonomy Guardrails** (#7504): Fixed
`AutonomyGuardrailService.load_from_metadata()` to validate both
`AutonomyGuardrails` and `GuardrailAuditTrail` models before writing either
to state, ensuring atomic updates. Previously, a validation failure on the
audit trail after guardrails were already written would leave the system in
an inconsistent state with partial updates. The method now uses a two-phase
validate-then-write approach: all model validation occurs in Phase 1, and
state mutations only happen in Phase 2 after all validations succeed.
- **ActorRegistry.add() spec-compliant YAML support** (#4466): The registry now
accepts actor YAML using the spec's `actors:` map format with nested `config:`
blocks, in addition to the legacy top-level `provider`/`model` format. The
`unsafe` flag and graph descriptor from nested config are now correctly
preserved during registration. Multi-actor YAML (>1 entry in `actors:`/`agents:`
map) is now rejected by `add()` with a `ValidationError`. Nested
`config.options` are now correctly preserved. The `unsafe` coercion now uses
strict `is True or == 1` instead of `bool()` to prevent truthy non-boolean
YAML values (e.g. `unsafe: "no"`) from being treated as unsafe.
- **Checkpoint Trigger Names and Config Key Path** (#5163): Aligned
`docs/specification.md` and `docs/reference/checkpointing.md` with the
implementation. Trigger names `on_tool_write` and `on_tool_write_complete`
are renamed to `before_tool_execute` and `after_tool_execute` to match
`src/cleveragents/tool/runner.py` (`DEFAULT_AUTO_TRIGGERS`) and
`src/cleveragents/application/services/config_service.py`. The configuration
reference table entry is moved from `sandbox.checkpoint.auto-create-on` to
`core.checkpoints.auto-create-on` to match the implementation's
`_register("core", "checkpoints.auto_create_on", ...)` call.
- **UKO Runtime Layer 2 (Paradigm) Indexing** (#9351): Added missing `rdf:type
uko-oo:Class` triple emission in `PythonAnalyzer._extract_class()` so that
Python class definitions are now correctly classified at layer 2 (paradigm/OO)
in addition to layer 3 (technology). Added the corresponding Behave scenario
`Indexing a Python file populates layer 2 (paradigm)` to
`features/uko_runtime.feature`, completing four-layer guarantee verification
for the UKO runtime.
- **Alembic Files Missing from Wheel Distribution** (#4180): Alembic configuration
(`alembic.ini`) and migration files are now part of the Python package structure
at `src/cleveragents/infrastructure/database/migrations/`. Previously, when
`agents init` was run in Docker containers or any wheel-based installation,
`FileNotFoundError` was raised because alembic files were stored at the
repository root and excluded from the wheel distribution. Now alembic files
follow standard Python packaging conventions and are automatically included.
`MigrationRunner._find_alembic_ini()` has been updated to search the new package
location as the primary anchor point. This fix enables `agents init` to work
correctly in all deployment modes: Docker containers, local pip installs
(wheel or editable), and development environments.
- **E2E Test Restoration: JSON Envelope Unwrap + WF14 Config Cleanup** (#8459):
Fixed 10 non-quota E2E test failures across the M5 acceptance suite and WF14
server-mode suite. `Extract JSON From Stdout` in `common_e2e.resource` now
auto-unwraps the spec-required CLI output envelope (`exit_code` + `data`
co-presence), so all callers receive the payload dict directly without manual
unpacking. `m5_acceptance.robot` was also reworked to pass CLI arguments
individually (fixing a space-in-arg parsing bug) and `tdd_expected_fail` tags
were removed from tests that now pass. `WF14 Clean Global Config` was added
to `wf14_server_mode.robot` — a keyword that uses Python/tomlkit directly to
strip `server.url`, `server.token`, and `server.namespace` from
`~/.cleveragents/config.toml` in both suite setup and teardown, replacing the
unreliable `config set` teardown calls that could not overwrite existing TOML
dotted keys.
- **TDD Non-AssertionError Guard Visibility** (#8294): `apply_tdd_inversion` in
`features/environment.py` now emits its non-assertion exception guard warning to
both the structured logger and `stderr` via a new `_warning_with_stderr` helper.
@@ -84,16 +60,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
crash (unhandled exception) is detected via an all-zero summary and the
captured traceback is always surfaced.
- **Bug Hunt Pool Supervisor Non-Blocking Tracking**: Updated `bug-hunt-pool-supervisor` to make the automation tracking step non-blocking. The `automation-tracking-manager` call in step 5 is now best-effort — if it does not complete within a reasonable time or fails, the supervisor skips it and continues to the next cycle. Added explicit rule 9 clarifying that tracking must never block the main loop. Core functionality (module scanning and worker dispatch) takes priority over status reporting.
- **Name Validator Server-Qualified Format** (#9074): Updated actor, skill, and tool name
validators to accept the spec-required `[[server:]namespace/]name` format. Previously,
server-qualified names like `dev:freemo/custom-analysis` were incorrectly rejected.
Added BDD scenarios for server-qualified name acceptance and rejection. All three
validators (`ActorConfigSchema.validate_name`, `NAMESPACED_NAME_RE`,
`_TOOL_NAME_PATTERN`) now correctly support optional server prefixes while maintaining
backward compatibility with existing `namespace/name` names.
- **Automation Profile Silent Fallback** (#8232): `_resolve_profile_for_plan` in
`PlanLifecycleService` now raises a clear `ValidationError` when a plan's
automation profile name is not a known built-in profile, instead of silently
@@ -103,37 +69,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
logged at debug level for observability.
### Added
- Wired `StrategyActor` into the real plan execution path: `_get_plan_executor`
in `plan.py` now resolves the strategy actor via `resolve_strategy_actor()`
(reading the `actor.default.strategy` config key) instead of always
constructing `LLMStrategizeActor`. `run_strategize` in `PlanExecutor` now
passes `resources` (derived from `plan.project_links`) and `project_context`
to the actor so the LLM prompt receives full project context. Strategy
decisions are serialised as JSON in `plan.error_details["strategy_decisions_json"]`
so `_build_decisions` can reconstruct the full hierarchy (dependency ordering,
parent/child structure) during Execute instead of rebuilding from
`definition_of_done`. `StrategizeStubActor.execute` accepts `**kwargs` for
forward-compatibility. Added BDD coverage for the stored-JSON path,
corrupt-JSON fallback, resource-passing, and stub extra-kwargs scenarios.
(#828)
- **Implementation Worker PR Inspection Permissions** (#8175): Added
`forgejo_list_pull_request_files` and `forgejo_get_pull_request_diff` permissions
to `implementation-worker.md`. These read-only permissions enable the worker to
directly inspect PR changes in PR fix mode, reducing unnecessary full-repo clones
and aligning permissions with the `pr-reviewer` agent. Also added `curl *`,
`printenv *`, and `echo $*` bash permissions needed for Forgejo API calls,
environment variable checks, and basic shell operations.
- Added TDD bug-capture test for bug #991 — AuditService._ensure_session()
TOCTOU race. Behave BDD scenario (`@tdd_bug @tdd_bug_991
@tdd_expected_fail`) launches 10 concurrent threads through a
`threading.Barrier` to prove that `_ensure_session()` creates multiple
engines when called without a lock. The test asserts `create_engine` is
called exactly once — which currently fails, confirming the race. The
`@tdd_expected_fail` tag inverts this to a CI pass until the fix is
merged. (#1095)
- **TDD Issue-Capture Test Activation** (#7025): Replaced 234 bare `@skip` tags
across 82 Behave feature files with the correct `@tdd_expected_fail @tdd_issue
@tdd_issue_<N>` tag system. Scenarios whose referenced bugs were already fixed
@@ -173,10 +109,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
`new-issue-creator`, and `issue-state-updater` now delegate all label operations to this
subagent.
- **PR-Issue Label Synchronization**: PRs now inherit `Priority/`, `MoSCoW/`, `Points/`,
- **PRIssue Label Synchronization**: PRs now inherit `Priority/`, `MoSCoW/`, `Points/`,
and `State/` labels from their associated issues at creation time
(`pr-api-creator`). The `backlog-groomer` adds a continuous Pass 19 for ongoing
PR-issue label synchronization. The `issue-state-updater` syncs PR state labels whenever
PRissue label synchronization. The `issue-state-updater` syncs PR state labels whenever
issue states change.
- **Automation Tracking Announcements**: Extended `automation-tracking-manager` with
@@ -190,14 +126,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
the `*-pool-supervisor` naming pattern. New agents added: `pr-editor` (safe PR editing
with description preservation), `pr-manager` (unified PR interface), and
`pr-merge-pool-supervisor` (automated PR merging supervisor). Renamed:
`pr-api-creator` to `pr-creator`, `pr-checker` to `pr-ci-test-fixer`,
`pr-status-checker` to `pr-status-analyzer`, `pr-self-reviewer` to `pr-reviewer`,
`pr-fix-orchestrator` to `pr-fix-pool-supervisor`.
`pr-api-creator``pr-creator`, `pr-checker``pr-ci-test-fixer`,
`pr-status-checker``pr-status-analyzer`, `pr-self-reviewer``pr-reviewer`,
`pr-fix-orchestrator``pr-fix-pool-supervisor`.
- **Automated PR Merging** (`pr-merge-pool-supervisor`): New supervisor continuously
monitors for merge-ready PRs and merges them automatically when all criteria are met
(approvals, CI passing, no conflicts). Supports both formal reviews and comment-based
approvals (LGTM, ready to merge, etc.).
approvals (LGTM, ✅, "ready to merge", etc.).
- **Implementation Worker Workflow Completion**: `implementation-worker` now implements
work claiming protocols with conflict detection, comprehensive review feedback handling
@@ -227,22 +163,8 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
`docs/development/automation-tracking.md` and the new
`docs/development/docs-writer.md` reference.
- **ACMS / UKO API Documentation** (`docs/api/acms.md`): Added comprehensive API
reference for the `cleveragents.acms` package covering the four-layer UKO ontology
hierarchy, `VocabularyRegistry`, `ProvenanceInfo`, `UKOClass`, `UKOProperty`,
`UKOVocabulary`, `Layer2Dependency`, `ParadigmVocabulary`, `DetailLevelMapBuilder`,
and all Layer 3 language vocabulary types (Python, TypeScript, Rust, Java).
The new page is linked from the API Reference index and the MkDocs navigation.
### Changed
- **PR Merge Supervisor Criteria Documentation** (#8107): Updated `pr-merge-pool-supervisor.md` to
explicitly document all ten merge criteria that the supervisor applies at runtime. Replaces the
implicit six-criteria description with a dedicated "Ten Merge Criteria" section listing all
requirements: approval, CI passing, no conflicts, not stale, no `needs feedback` label, not
blocked, has milestone, has `Type/` label, has `Closes` reference, and changelog updated.
Eliminates the inconsistency between the agent definition and actual runtime behaviour.
- **Decision Tree Full ULID Display** (#5825): The `agents plan tree` command now
displays full 26-character ULIDs for all decisions instead of truncating them to
8 characters. This enables users to copy decision IDs directly from tree output
@@ -256,7 +178,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
- **PR Review Policy**: Reduced PR review requirement from 2 approvals to 1. Self-approval
is now permitted including for automated bot PRs. Approval can be a formal review OR an
approval comment (LGTM, Approved, ready to merge).
approval comment (LGTM, Approved, ✅, "ready to merge").
- **Label Delegation Enforcement**: `automation-tracking-manager` now enforces delegation
to `forgejo-label-manager` for all label operations, preventing "invalid label ID" errors
@@ -274,43 +196,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
### Fixed
- **Plan Concurrency Race Condition** (#7989): Fixed critical race condition in `execute_plan()` and
`apply_plan()` where concurrent CLI/worker sessions could simultaneously modify the same plan,
corrupting plan state. `LockService` is now wired into the plan lifecycle with plan-level advisory
locking. Each invocation generates a unique caller identity (UUID) to prevent re-entrant lock
acquisition by concurrent sessions on the same plan. Concurrent attempts now raise `LockConflictError`
instead of silently racing. Lock is acquired before phase transition and released in a `finally`
block to ensure cleanup even on error.
- **`--format color` ANSI Output** (#7910): Fixed `format_output` routing the `color` format
option to `_format_plain`, which produced plain uncoloured text instead of ANSI escape
sequences. The `color` format is now routed to `format_output_session` which uses the
`ColorMaterializer` to emit proper ANSI-coloured output. `--format plain` and all other
formats remain unaffected.
- **ContextTierService Thread Safety** (#7547): Added `threading.RLock` to
`ContextTierService` to prevent `RuntimeError: dictionary changed size during
iteration` and data corruption under concurrent plan execution. All public
methods (`store`, `get`, `promote`, `demote`, `evict_lru`, `enforce_staleness`,
`get_metrics`, `get_all_fragments`, `get_hot_fragments`, `get_for_actor`,
`get_scoped_view`, `get_scoped_by_resource`, `get_scoped_metrics`) now acquire
the reentrant lock before accessing the hot/warm/cold tier dicts. The service
was previously documented as single-threaded but registered as a DI Singleton,
causing potential data corruption when parallel subplans shared the same
instance. The `TierRuntimeMixin.enforce_staleness()` and
`ScopedTierMixin.get_scoped_by_resource()` / `get_scoped_metrics()` methods
are also protected. The DI container registration as `providers.Singleton`
is now correct and safe.
- **Migration Prompt Safe Default on Failure** (#7503): Fixed
`MigrationRunner._default_prompt_for_migration` silently auto-approving destructive
database migrations when the interactive prompt raised any exception. The handler now
catches only `(IOError, OSError, EOFError)` (broken stdin / non-interactive pipe),
logs a `WARNING` instead of a `DEBUG` message, and returns `False` (reject) so that
migrations are never applied without explicit user consent. `KeyboardInterrupt` is
re-raised so Ctrl-C always works. Non-interactive environments (stdin not a TTY) also
now return `False` by default; use `CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true` or the
`--yes` CLI flag to approve automatically.
- **Path Traversal Sandbox Escape via Prefix Collision** (#7558): Fixed
`validate_path()` in `file_tools.py` using `str.startswith()` for sandbox
containment, which allowed sibling directories with a matching name prefix
(e.g. `/tmp/sandbox-escape/` bypassing `/tmp/sandbox/`) to escape the
sandbox. Replaced with `Path.relative_to()` which performs a proper path
prefix check using OS path separators. Added regression test tagged
`@tdd_issue_7558`.
- **Validation Gate Empty-Run Guard** (#7508): Fixed `ApplyValidationSummary.all_required_passed`
returning `True` when zero validations were run, silently bypassing the apply gate. The property
@@ -332,16 +224,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
`sandbox_root=.cleveragents/sandbox/`, so LLM file output (`FILE:` blocks)
is written to disk during the execute phase. (#4222)
- **SubplanExecutionService fail_fast cancellation** (#7582): Fixed a race condition where
already-running parallel subplans were not cancelled when `fail_fast` fired. Previously,
`Future.cancel()` only prevented queued futures from starting but had no effect on
in-flight futures that completed after `stop_flag` was set -- their `COMPLETE` results
were incorrectly included in the merge output. The fix adds a post-completion guard that
overrides any non-`ERRORED`/non-`CANCELLED` result to `CANCELLED` when `stop_flag` is
active, and clears the associated output to prevent it from entering the merge. Also
replaces the O(n) linear `status` lookup in the `as_completed()` loop with an O(1)
`status_map` dict pre-computed before the executor block.
- **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the
`tdd_expected_fail_listener` `end_test()` function to prevent blindly inverting ALL test
failures to passes, which was masking infrastructure errors and causing flaky CI behavior.
@@ -350,11 +232,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
e2e test files and removed `tdd_expected_fail` from 4 context assembly e2e tests where
bugs were already fixed.
- **PluginLoader entry point prefix validation** (#7476): Parse entry point targets before
import, enforce the module allowlist ahead of loading, and add Behave plus Robot Framework
regression coverage to ensure disallowed prefixes never execute untrusted module-level code
in either unit or integration flows.
- **`issue-state-updater` Bash Script Errors**: Removed problematic bash script examples
that tried to invoke `task forgejo-label-manager` as a bash command (the Task tool cannot
be invoked from bash). Replaced with clear step-by-step operational instructions and
@@ -374,17 +251,18 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
a `sqlite3.IntegrityError: UNIQUE constraint failed` crash when `agents plan use` was
called on an action that already had arguments registered via `action create`. (#4197)
- Updated `resource_dag.robot` to use `StaticPool`-based SQLite connections
(`poolclass=StaticPool`, `connect_args={"check_same_thread": False}`) in
all three test cases, preventing connection sharing issues in the test
environment. Updated cycle detection test to use distinct resource types
(`robot/cycle-a` and `robot/cycle-b`) instead of a single shared type,
improving coverage of cross-type cycle detection. Split from PR #1204
per reviewer request for atomic commits. (#1226)
- Fixed CheckpointManager not wired into PlanExecutor — checkpoints were
never created during plan execution because `_get_plan_executor()` in
the CLI constructed PlanExecutor without a CheckpointManager (defaulted
to None, silently skipping all checkpoint hooks). `_get_plan_executor()`
now resolves the container singleton so CLI `plan execute` and `plan
rollback` share the same registry, and `_try_create_checkpoint()` raises
`PlanError` if checkpoint metadata cannot be persisted. Writable sandboxable
resources and write-capable tools now default to `checkpointable=True`, and
new Behave scenarios cover DI wiring, rollback, and capability defaults. (#1253)
---
## [3.8.0] -- 2026-04-05
## [3.8.0] 2026-04-05
### Added
@@ -395,13 +273,171 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
`INVARIANT_VIOLATED` events. Post-correction reconciliation runs
via `CORRECTION_APPLIED` event subscription (best-effort). Added
`InvariantService` Singleton provider in the DI container.
- **TUI -- Shell danger detection**: The TUI shell mode (`!` prefix) now detects
- **TUI Shell danger detection**: The TUI shell mode (`!` prefix) now detects
dangerous command patterns before execution. A configurable pattern registry
classifies commands by danger level (warning, critical) and surfaces a user
warning overlay before proceeding. Patterns cover destructive filesystem
operations, privilege escalation, network exfiltration, and more. (#1003)
- **TUI -- Permission Question Widget**: A new inline `PermissionQuestionWidget`
- **TUI Permission Question Widget**: A new inline `PermissionQuestionWidget`
renders permission requests directly in the conversation stream for single-file
operations. Users can allow/reject with single-key shortcuts (`a`/`A`/`r`/`R`),
navigate with arrow keys, confirm with `Enter`, or press `v` to open the full
permissions screen. (#1004)
- **TUI — Actor thought blocks**: Expandable reasoning trace widgets rendered
inline in the conversation stream with muted styling. Collapsed by default;
expand with `Space` or click. (#1005)
- **UKO provenance tracking**: Every typed triple now carries `sourceResource`,
`validFrom`, and `isCurrent` metadata. A revision chain enables temporal
queries and point-in-time ontology state reconstruction.
- **JSON-RPC 2.0 A2A wire format**: `A2aRequest`/`A2aResponse` fields renamed to
standard JSON-RPC 2.0 names (`method`, `id`, `result`, `error`). The
`A2aVersionNegotiator` handles backward compatibility.
- **Database resource handler**: Full CRUD and checkpoint/rollback support for
SQLite, PostgreSQL, MySQL, and DuckDB resources via the resource DAG.
- **Estimation lifecycle hook**: `actor.default.estimation` config key wires an
estimation actor into the Strategize-to-Estimate lifecycle hook.
- **Persona system**: YAML-backed personas bind actors, argument presets, and scope
references to named identities; persisted in `~/.config/cleveragents/personas/`.
- **Session management**: Create, list, export, and import conversation sessions;
full JSON export/import for portability; Markdown transcript export
(`--format md`) for human-readable sharing.
- **First-run experience**: `ActorSelectionOverlay` guides new users to pick an
actor on first TUI launch; creates a `"default"` persona automatically.
- **Server mode**: `agents server connect` configures a remote CleverAgents server;
Kubernetes Helm chart in `k8s/` for production deployment.
- **A2A integration**: Agent-to-Agent protocol facade wires CLI and TUI to live
application services (session, plan, registry, event).
- **Permissions screen**: TUI overlay for reviewing tool permission requests with
unified, side-by-side, and context diff views; session-scoped allow/reject decisions.
- **Inline permission questions**: `PermissionQuestionWidget` renders single-file
permission requests directly in the conversation stream with single-key shortcuts.
- **Invariant reconciliation**: `InvariantReconciliationActor` runs automatically at
every plan phase transition; failures block the transition and emit `INVARIANT_VIOLATED`.
- **UKO runtime**: Universal Knowledge Ontology query interface, inference engine, and
graph persistence for ACMS context strategies.
### Fixed
- `LangChainChatProvider.name` and `model_id` are now mutable properties with setters,
fixing an `AttributeError` when `PlanService` attempted to resolve provider names after
instantiation. (#1553)
---
## [3.7.0] — 2026-03-15
### Added
- **Interactive TUI** (`agents tui`) — full-screen Textual app with multi-session tabs,
persona switching, slash commands (67 commands across 14 groups), reference picker
(`@`), shell mode (`!`), context-sensitive F1 help, and `Ctrl+T` argument preset cycling.
- **Slash command system** — 67 commands across 14 groups accessible via `/` overlay.
- **Reference picker** — `@` key opens a file/resource reference picker that inserts
references into the input field.
- **TUI persona system** — YAML-backed personas bind actors, argument presets, and scope
references; persisted in `~/.config/cleveragents/personas/`.
- **TUI session export/import** — full JSON round-trip and Markdown transcript export
(`--format md`).
---
## [3.6.0] — 2026-02-28
### Added
- Advanced Context Management System (ACMS) with three-tier context strategy.
- UKO Runtime (Universal Knowledge Ontology) with graph persistence and inference engine.
- Implicit inference engine producing `uko:implicitSiblingOf`, `uko:implicitContains`,
and `uko:implicitDependsOn` triples with confidence 0.7.
---
## [3.5.0] — 2026-02-14
### Added
- Autonomy hardening: advisory locking, validation pipeline, definition-of-done gating.
- Resource DAG with dependency tracking and type hierarchy with multiple inheritance.
- Container resource types (`container.docker`, `container.podman`).
- LSP resource types (`lsp.*`).
---
## [3.4.0] — 2026-01-31
### Added
- ACMS v1 with context scaling strategies.
- Resource type inheritance system (ADR-042).
- Safety profile extraction (ADR-041).
---
## [3.3.0] — 2026-01-17
### Added
- Corrections and subplans support in plan lifecycle.
- Checkpoint and rollback for all resource writes.
- Decision tree versioning and history (ADR-034).
- Decision tree rollback and replay (ADR-035).
---
## [3.2.0] — 2026-01-03
### Added
- Decisions, validations, and invariants in plan lifecycle.
- Validation abstraction layer (ADR-013).
- Invariant system (ADR-016).
- Automation profiles (ADR-017).
- Semantic error prevention (ADR-018).
---
## [3.1.0] — 2025-12-20
### Added
- MCP (Model Context Protocol) adapter and client (ADR-029).
- LSP (Language Server Protocol) client integration (ADR-027).
- Agent Skills Standard (AgentSkills.io) support (ADR-028).
- Skill abstraction definition (ADR-030).
---
## [3.0.0] — 2025-12-06
### Added
- Initial public release of CleverAgents Core.
- Unified `agents` / `cleveragents` CLI entry points.
- Layered architecture: Entry Points → Application → Domain → Infrastructure → Integration → Core.
- Actor system with YAML-defined LangGraph node graphs.
- Tool system with four-stage lifecycle (activate → validate → execute → deactivate).
- Skill system with three-tier progressive disclosure.
- Resource system with DAG and type hierarchy.
- A2A (Agent-to-Agent) protocol facade.
- DI container (`cleveragents.application.container`).
- LangChain/LangGraph integration (ADR-022).
- Provider registry with fallback chain (OpenAI → Anthropic → Google → Azure → OpenRouter → Groq → Together → Cohere).
- Observability: structured logging, metrics, audit trail, token/cost tracking.
- BDD test suite (Behave + Robot Framework).
- Nox automation for lint, typecheck, tests, docs, benchmarks.
- MkDocs-powered documentation with CleverAgents branding.
+461
View File
@@ -0,0 +1,461 @@
# Changelog
All notable changes to this project will be documented in this file.
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
## [Unreleased]
### Fixed
<<<<<<< HEAD
- **E2E Test Restoration: JSON Envelope Unwrap + WF14 Config Cleanup** (#8459):
Fixed 10 non-quota E2E test failures across the M5 acceptance suite and WF14
server-mode suite. `Extract JSON From Stdout` in `common_e2e.resource` now
auto-unwraps the spec-required CLI output envelope (`exit_code` + `data`
co-presence), so all callers receive the payload dict directly without manual
unpacking. `m5_acceptance.robot` was also reworked to pass CLI arguments
individually (fixing a space-in-arg parsing bug) and `tdd_expected_fail` tags
were removed from tests that now pass. `WF14 Clean Global Config` was added
to `wf14_server_mode.robot` — a keyword that uses Python/tomlkit directly to
strip `server.url`, `server.token`, and `server.namespace` from
`~/.cleveragents/config.toml` in both suite setup and teardown, replacing the
unreliable `config set` teardown calls that could not overwrite existing TOML
dotted keys.
||||||| 703cf13c
=======
- **Actor v3 YAML Schema Validation in CLI** (#5869): The `agents actor add --config`
command now validates v3 YAML files using `ActorConfigSchema`, ensuring proper
schema compliance including cycle detection for GRAPH actors, required field
validation, and enum validation. v3 YAML is detected by the presence of ANY
`type` field (any value — invalid type values are then rejected by schema
validation) or a `version` field whose string value starts with `"3"` (e.g.
`"3"`, `"3.0"`, `"3.0.0"`). Configs with `type: null` are not treated as v3.
Invalid v3 actors are rejected with clear error messages before registration.
- **Alembic Files Missing from Wheel Distribution** (#4180): Alembic configuration
(`alembic.ini`) and migration files are now part of the Python package structure
at `src/cleveragents/infrastructure/database/migrations/`. Previously, when
`agents init` was run in Docker containers or any wheel-based installation,
`FileNotFoundError` was raised because alembic files were stored at the
repository root and excluded from the wheel distribution. Now alembic files
follow standard Python packaging conventions and are automatically included.
`MigrationRunner._find_alembic_ini()` has been updated to search the new package
location as the primary anchor point. This fix enables `agents init` to work
correctly in all deployment modes: Docker containers, local pip installs
(wheel or editable), and development environments.
- **E2E Test Restoration: JSON Envelope Unwrap + WF14 Config Cleanup** (#8459):
Fixed 10 non-quota E2E test failures across the M5 acceptance suite and WF14
server-mode suite. `Extract JSON From Stdout` in `common_e2e.resource` now
auto-unwraps the spec-required CLI output envelope (`exit_code` + `data`
co-presence), so all callers receive the payload dict directly without manual
unpacking. `m5_acceptance.robot` was also reworked to pass CLI arguments
individually (fixing a space-in-arg parsing bug) and `tdd_expected_fail` tags
were removed from tests that now pass. `WF14 Clean Global Config` was added
to `wf14_server_mode.robot` — a keyword that uses Python/tomlkit directly to
strip `server.url`, `server.token`, and `server.namespace` from
`~/.cleveragents/config.toml` in both suite setup and teardown, replacing the
unreliable `config set` teardown calls that could not overwrite existing TOML
dotted keys.
>>>>>>> test/restore-e2e-tests
- **TDD Non-AssertionError Guard Visibility** (#8294): `apply_tdd_inversion` in
`features/environment.py` now emits its non-assertion exception guard warning to
both the structured logger and `stderr` via a new `_warning_with_stderr` helper.
This makes the guard firing visible in standard Behave console output and CI log
snippets where the structured logging sink may not be displayed. BDD infrastructure
coverage added: a new scenario in `tdd_expected_fail_infrastructure.feature`
asserts that the warning is emitted to stderr when a non-AssertionError exception
is encountered in an `@tdd_expected_fail` scenario, and a second scenario asserts
the warning is NOT emitted when the exception is an `AssertionError`. The
`CONTRIBUTING.md` now documents that `@tdd_expected_fail` step definitions must
signal expected failures via `AssertionError`.
- **Parallel Behave Runner Log Noise Reduction** (#8351): The parallel behave
runner now suppresses captured stdout/stderr for passing worker chunks and
only replays diagnostics for failed, errored, or crashed chunks. This makes
failure output significantly easier to spot in CI and local runs. A worker
crash (unhandled exception) is detected via an all-zero summary and the
captured traceback is always surfaced.
- **Automation Profile Silent Fallback** (#8232): `_resolve_profile_for_plan` in
`PlanLifecycleService` now raises a clear `ValidationError` when a plan's
automation profile name is not a known built-in profile, instead of silently
falling back to `"manual"`. Users who configured custom automation profiles
(e.g. `"semi-auto"`, `"acme/strict"`) will now receive an actionable error
message listing available built-in profiles. The resolved profile name is also
logged at debug level for observability.
### Added
- **TDD Issue-Capture Test Activation** (#7025): Replaced 234 bare `@skip` tags
across 82 Behave feature files with the correct `@tdd_expected_fail @tdd_issue
@tdd_issue_<N>` tag system. Scenarios whose referenced bugs were already fixed
had `@tdd_expected_fail` removed and now run as permanent regression guards.
Net result: 629 features active in CI (up from ~545), zero `@skip` tags remain.
- **Git Worktree Sandbox Apply** (#4454): The `plan apply` command now merges
LLM-generated changes via `git merge` from an isolated worktree branch
instead of flat `shutil.copy2`. Displays spec-aligned Apply Summary
(plan ID, artifacts, insertions/deletions, project, timestamp), Sandbox
Cleanup panel, and `✓ OK Changes applied` footer. Non-git projects fall
back to the original flat file copy.
- **Context Hydration Fix** (#4454): Fixed `ContextFragment` metadata types
(`detail_depth` and `relevance_score` must be strings, not int/float) that
caused Pydantic validation errors during context assembly, resulting in the
LLM receiving zero file context.
- **Automation Tracking System**: Replaced shared session state issue tracking with
individual per-agent tracking issues. Each agent now creates its own `[AUTO-<PREFIX>]`
titled issues with standardized headers, reporting intervals, and health indicators.
Agents: `session-persister`, `implementation-orchestrator`, `system-watchdog`,
`backlog-groomer`, `human-liaison`. Documentation at
`docs/development/automation-tracking.md`.
- **Automated Health Monitoring and Recovery**: The `system-watchdog` now runs
`audit_automation_tracking_health()` every 5 minutes, detecting stalled agents when
tracking issues are >20% overdue from their declared reporting interval. On detection,
it terminates stalled sessions via the OpenCode Server API, performs root-cause analysis,
creates high-priority diagnostic issues, and closes stale tracking issues with recovery
notes.
- **Centralized Label Management** (`forgejo-label-manager`): A new specialized subagent
centralizes all Forgejo label operations across the agent system. Enforces the
organization-level label system, prohibits label creation, and validates label compliance.
Agents `backlog-groomer`, `human-liaison`, `project-owner`, `epic-planner`,
`new-issue-creator`, and `issue-state-updater` now delegate all label operations to this
subagent.
- **PRIssue Label Synchronization**: PRs now inherit `Priority/`, `MoSCoW/`, `Points/`,
and `State/` labels from their associated issues at creation time
(`pr-api-creator`). The `backlog-groomer` adds a continuous Pass 19 for ongoing
PRissue label synchronization. The `issue-state-updater` syncs PR state labels whenever
issue states change.
- **Automation Tracking Announcements**: Extended `automation-tracking-manager` with
announcement issue support (`CREATE_ANNOUNCEMENT_ISSUE`, `CLOSE_ANNOUNCEMENT_ISSUE`,
`LIST_TRACKING_ISSUES`, `READ_ANNOUNCEMENTS`, `REVIEW_OWN_ANNOUNCEMENTS`). Supervisors
and workers now read critical announcements before each cycle for cross-agent awareness.
Priority-based filtering (Critical/High/Medium/Low) reduces noise. Backlog-groomer
performs intelligent cleanup with age thresholds by priority.
- **PR Agent Reorganization**: All PR-related agents renamed and reorganized to follow
the `*-pool-supervisor` naming pattern. New agents added: `pr-editor` (safe PR editing
with description preservation), `pr-manager` (unified PR interface), and
`pr-merge-pool-supervisor` (automated PR merging supervisor). Renamed:
`pr-api-creator` → `pr-creator`, `pr-checker` → `pr-ci-test-fixer`,
`pr-status-checker` → `pr-status-analyzer`, `pr-self-reviewer` → `pr-reviewer`,
`pr-fix-orchestrator` → `pr-fix-pool-supervisor`.
- **Automated PR Merging** (`pr-merge-pool-supervisor`): New supervisor continuously
monitors for merge-ready PRs and merges them automatically when all criteria are met
(approvals, CI passing, no conflicts). Supports both formal reviews and comment-based
approvals (LGTM, ✅, "ready to merge", etc.).
- **Implementation Worker Workflow Completion**: `implementation-worker` now implements
work claiming protocols with conflict detection, comprehensive review feedback handling
with intelligent parsing, sophisticated merge conflict resolution with multiple
strategies, and parallel subtask execution with wave-based dependency analysis.
Pass rate improved from 48.15% to 84.8%.
- **Container Resource Stop Support**: `agents resource stop` now correctly stops
`container-instance` and `devcontainer-instance` resource types.
- **Centralized Automation Tracking Manager** (`automation-tracking-manager`): The manager
is now the single interface for all tracking issue operations (`CREATE_TRACKING_ISSUE`,
`UPDATE_TRACKING_ISSUE`, `CLOSE_TRACKING_ISSUE`, `READ_TRACKING_STATE`,
`GET_NEXT_CYCLE_NUMBER`). Agents delegate to the manager rather than calling the Forgejo
API directly, ensuring sequential cycle numbers across restarts and preventing
duplicate issues. Migrated agents include `system-watchdog`,
`implementation-pool-supervisor`, `timeline-update-pool-supervisor`,
`project-owner-pool-supervisor`, `product-builder`, and
`backlog-grooming-pool-supervisor`. The legacy `shared/automation_tracking.md` module
was removed.
- **Documentation Writer Tracking** (`docs-writer`): The documentation writer now
participates in the automation tracking system by creating individual
`[AUTO-DOCS] Documentation Report (Cycle N)` issues every 10 cycles (~3.3 hours).
The manager applies the mandatory `Automation Tracking` label automatically, while
teams may add additional workflow labels as needed. See
`docs/development/automation-tracking.md` and the new
`docs/development/docs-writer.md` reference.
### Changed
- **Decision Tree Full ULID Display** (#5825): The `agents plan tree` command now
displays full 26-character ULIDs for all decisions instead of truncating them to
8 characters. This enables users to copy decision IDs directly from tree output
and use them in follow-up CLI commands like `agents plan correct` without manual
ID reconstruction. Added "Decision IDs (for correction)" section with human-readable
labels for easy reference. Applies to both table and rich/plain text output formats.
- **Automation Tracking Format**: All automation tracking issues now use a standardized
header format with mandatory `Reporting Interval: <interval> (Next report expected: <ts>)`
declarations, enabling precise staleness detection.
- **PR Review Policy**: Reduced PR review requirement from 2 approvals to 1. Self-approval
is now permitted including for automated bot PRs. Approval can be a formal review OR an
approval comment (LGTM, Approved, ✅, "ready to merge").
- **Label Delegation Enforcement**: `automation-tracking-manager` now enforces delegation
to `forgejo-label-manager` for all label operations, preventing "invalid label ID" errors
and ensuring label application uses correct name-to-ID mapping.
- **Automation Tracking Label Guidance**: Documentation now clarifies that the manager
automatically applies the `Automation Tracking` label and that additional labels such as
`Type/Automation`, `State/In Progress`, or `Priority/Medium` remain optional workflow
choices rather than mandatory.
- **Automation Tracking Agent Prefix Registry**: Expanded from 5 agents to 18 agents.
New prefixes include `AUTO-DOCS`, `AUTO-REV-POOL`, `AUTO-UAT-POOL`, `AUTO-BUG-POOL`,
`AUTO-INF-POOL`, `AUTO-ARCH`, `AUTO-EPIC`, `AUTO-EVLV`, `AUTO-GUARD`, `AUTO-SPEC`,
`AUTO-TIME`, `AUTO-PROJ-OWN`, and `AUTO-PROD-BLDR`.
### Fixed
- **Path Traversal Sandbox Escape via Prefix Collision** (#7558): Fixed
`validate_path()` in `file_tools.py` using `str.startswith()` for sandbox
containment, which allowed sibling directories with a matching name prefix
(e.g. `/tmp/sandbox-escape/` bypassing `/tmp/sandbox/`) to escape the
sandbox. Replaced with `Path.relative_to()` which performs a proper path
prefix check using OS path separators. Added regression test tagged
`@tdd_issue_7558`.
- **Validation Gate Empty-Run Guard** (#7508): Fixed `ApplyValidationSummary.all_required_passed`
returning `True` when zero validations were run, silently bypassing the apply gate. The property
now returns `False` when the validation result set is empty (`is_empty` is `True`), ensuring
that apply is blocked unless at least one validation was actually executed. Also added
`required_total` property for completeness. Updated `consolidated_validation.feature` scenarios
to reflect the corrected blocking behavior for empty summaries and no-attachment runs.
- **ACMS context tier hydration**: `ContextTierService` no longer starts empty
on every CLI invocation. A new `context_tier_hydrator.py` reads files from
linked project resources (via `git ls-files` or `os.walk`), creates
`TieredFragment` objects, and stores them in the tier service before context
assembly in `LLMExecuteActor.execute()`. The LLM now receives real file
context during plan execution. Respects max file size (256 KB), total budget
(10 MB), binary file exclusion, and `.git`/`node_modules`/`__pycache__`
directory skipping. (#1028)
- **Sandbox root wiring**: `_get_plan_executor()` now passes
`sandbox_root=.cleveragents/sandbox/`, so LLM file output (`FILE:` blocks)
is written to disk during the execute phase. (#4222)
- **Robot Framework TDD Listener Guards** (#5436): Added three guard conditions to the
`tdd_expected_fail_listener` `end_test()` function to prevent blindly inverting ALL test
failures to passes, which was masking infrastructure errors and causing flaky CI behavior.
Guards: setup/teardown error detection, non-assertion failure detection (infrastructure
errors), and dry-run mode detection. Also fixed `Variable Should Exist` syntax errors in
e2e test files and removed `tdd_expected_fail` from 4 context assembly e2e tests where
bugs were already fixed.
- **`issue-state-updater` Bash Script Errors**: Removed problematic bash script examples
that tried to invoke `task forgejo-label-manager` as a bash command (the Task tool cannot
be invoked from bash). Replaced with clear step-by-step operational instructions and
direct label management via API.
- **`automation-tracking-manager` Label Delegation Syntax**: Fixed incorrect delegation
syntax when calling `forgejo-label-manager`. The manager now uses correct natural language
requests (e.g., "Apply labels to issue #123: Automation Tracking") instead of structured
parameters, ensuring tracking issues receive proper labels.
- **`product-builder` Missing Supervisors**: Added missing `pr-fix-pool-supervisor` and
`pr-merge-pool-supervisor` to the product-builder's supervisor launch list (18 total
supervisors). Updated all numeric references, pre-flight checklists, and validation logic.
- `ActionRepository.update()` now uses explicit bulk `sa_delete()` + `session.flush()`
before re-inserting child rows for `action_arguments` and `action_invariants`, fixing
a `sqlite3.IntegrityError: UNIQUE constraint failed` crash when `agents plan use` was
called on an action that already had arguments registered via `action create`. (#4197)
- Fixed CheckpointManager not wired into PlanExecutor — checkpoints were
never created during plan execution because `_get_plan_executor()` in
the CLI constructed PlanExecutor without a CheckpointManager (defaulted
to None, silently skipping all checkpoint hooks). `_get_plan_executor()`
now resolves the container singleton so CLI `plan execute` and `plan
rollback` share the same registry, and `_try_create_checkpoint()` raises
`PlanError` if checkpoint metadata cannot be persisted. Writable sandboxable
resources and write-capable tools now default to `checkpointable=True`, and
new Behave scenarios cover DI wiring, rollback, and capability defaults. (#1253)
---
## [3.8.0] — 2026-04-05
### Added
- Wired Invariant Reconciliation Actor auto-invocation into
`PlanLifecycleService` phase transitions (`start_strategize`,
`execute_plan`, `apply_plan`). Reconciliation failures now block
the transition with `ReconciliationBlockedError` and emit
`INVARIANT_VIOLATED` events. Post-correction reconciliation runs
via `CORRECTION_APPLIED` event subscription (best-effort). Added
`InvariantService` Singleton provider in the DI container.
- **TUI — Shell danger detection**: The TUI shell mode (`!` prefix) now detects
dangerous command patterns before execution. A configurable pattern registry
classifies commands by danger level (warning, critical) and surfaces a user
warning overlay before proceeding. Patterns cover destructive filesystem
operations, privilege escalation, network exfiltration, and more. (#1003)
- **TUI — Permission Question Widget**: A new inline `PermissionQuestionWidget`
renders permission requests directly in the conversation stream for single-file
operations. Users can allow/reject with single-key shortcuts (`a`/`A`/`r`/`R`),
navigate with arrow keys, confirm with `Enter`, or press `v` to open the full
permissions screen. (#1004)
- **TUI — Actor thought blocks**: Expandable reasoning trace widgets rendered
inline in the conversation stream with muted styling. Collapsed by default;
expand with `Space` or click. (#1005)
- **UKO provenance tracking**: Every typed triple now carries `sourceResource`,
`validFrom`, and `isCurrent` metadata. A revision chain enables temporal
queries and point-in-time ontology state reconstruction.
- **JSON-RPC 2.0 A2A wire format**: `A2aRequest`/`A2aResponse` fields renamed to
standard JSON-RPC 2.0 names (`method`, `id`, `result`, `error`). The
`A2aVersionNegotiator` handles backward compatibility.
- **Database resource handler**: Full CRUD and checkpoint/rollback support for
SQLite, PostgreSQL, MySQL, and DuckDB resources via the resource DAG.
- **Estimation lifecycle hook**: `actor.default.estimation` config key wires an
estimation actor into the Strategize-to-Estimate lifecycle hook.
- **Persona system**: YAML-backed personas bind actors, argument presets, and scope
references to named identities; persisted in `~/.config/cleveragents/personas/`.
- **Session management**: Create, list, export, and import conversation sessions;
full JSON export/import for portability; Markdown transcript export
(`--format md`) for human-readable sharing.
- **First-run experience**: `ActorSelectionOverlay` guides new users to pick an
actor on first TUI launch; creates a `"default"` persona automatically.
- **Server mode**: `agents server connect` configures a remote CleverAgents server;
Kubernetes Helm chart in `k8s/` for production deployment.
- **A2A integration**: Agent-to-Agent protocol facade wires CLI and TUI to live
application services (session, plan, registry, event).
- **Permissions screen**: TUI overlay for reviewing tool permission requests with
unified, side-by-side, and context diff views; session-scoped allow/reject decisions.
- **Inline permission questions**: `PermissionQuestionWidget` renders single-file
permission requests directly in the conversation stream with single-key shortcuts.
- **Invariant reconciliation**: `InvariantReconciliationActor` runs automatically at
every plan phase transition; failures block the transition and emit `INVARIANT_VIOLATED`.
- **UKO runtime**: Universal Knowledge Ontology query interface, inference engine, and
graph persistence for ACMS context strategies.
### Fixed
- `LangChainChatProvider.name` and `model_id` are now mutable properties with setters,
fixing an `AttributeError` when `PlanService` attempted to resolve provider names after
instantiation. (#1553)
---
## [3.7.0] — 2026-03-15
### Added
- **Interactive TUI** (`agents tui`) — full-screen Textual app with multi-session tabs,
persona switching, slash commands (67 commands across 14 groups), reference picker
(`@`), shell mode (`!`), context-sensitive F1 help, and `Ctrl+T` argument preset cycling.
- **Slash command system** — 67 commands across 14 groups accessible via `/` overlay.
- **Reference picker** — `@` key opens a file/resource reference picker that inserts
references into the input field.
- **TUI persona system** — YAML-backed personas bind actors, argument presets, and scope
references; persisted in `~/.config/cleveragents/personas/`.
- **TUI session export/import** — full JSON round-trip and Markdown transcript export
(`--format md`).
---
## [3.6.0] — 2026-02-28
### Added
- Advanced Context Management System (ACMS) with three-tier context strategy.
- UKO Runtime (Universal Knowledge Ontology) with graph persistence and inference engine.
- Implicit inference engine producing `uko:implicitSiblingOf`, `uko:implicitContains`,
and `uko:implicitDependsOn` triples with confidence 0.7.
---
## [3.5.0] — 2026-02-14
### Added
- Autonomy hardening: advisory locking, validation pipeline, definition-of-done gating.
- Resource DAG with dependency tracking and type hierarchy with multiple inheritance.
- Container resource types (`container.docker`, `container.podman`).
- LSP resource types (`lsp.*`).
---
## [3.4.0] — 2026-01-31
### Added
- ACMS v1 with context scaling strategies.
- Resource type inheritance system (ADR-042).
- Safety profile extraction (ADR-041).
---
## [3.3.0] — 2026-01-17
### Added
- Corrections and subplans support in plan lifecycle.
- Checkpoint and rollback for all resource writes.
- Decision tree versioning and history (ADR-034).
- Decision tree rollback and replay (ADR-035).
---
## [3.2.0] — 2026-01-03
### Added
- Decisions, validations, and invariants in plan lifecycle.
- Validation abstraction layer (ADR-013).
- Invariant system (ADR-016).
- Automation profiles (ADR-017).
- Semantic error prevention (ADR-018).
---
## [3.1.0] — 2025-12-20
### Added
- MCP (Model Context Protocol) adapter and client (ADR-029).
- LSP (Language Server Protocol) client integration (ADR-027).
- Agent Skills Standard (AgentSkills.io) support (ADR-028).
- Skill abstraction definition (ADR-030).
---
## [3.0.0] — 2025-12-06
### Added
- Initial public release of CleverAgents Core.
- Unified `agents` / `cleveragents` CLI entry points.
- Layered architecture: Entry Points → Application → Domain → Infrastructure → Integration → Core.
- Actor system with YAML-defined LangGraph node graphs.
- Tool system with four-stage lifecycle (activate → validate → execute → deactivate).
- Skill system with three-tier progressive disclosure.
- Resource system with DAG and type hierarchy.
- A2A (Agent-to-Agent) protocol facade.
- DI container (`cleveragents.application.container`).
- LangChain/LangGraph integration (ADR-022).
- Provider registry with fallback chain (OpenAI → Anthropic → Google → Azure → OpenRouter → Groq → Together → Cohere).
- Observability: structured logging, metrics, audit trail, token/cost tracking.
- BDD test suite (Behave + Robot Framework).
- Nox automation for lint, typecheck, tests, docs, benchmarks.
- MkDocs-powered documentation with CleverAgents branding.
+1808
View File
File diff suppressed because it is too large Load Diff
+1
View File
@@ -0,0 +1 @@
10081,9437,8297,8285,8280,8269,8176,5968,5276,4652,4224,3466,3455,3152,1410,1258,1224,1221,1211,1202,1160,1117
+65
View File
@@ -0,0 +1,65 @@
Feature: CLI main.py coverage round 3
As a developer
I want to cover remaining uncovered lines in cleveragents.cli.main
So that code coverage gaps on lines 104-110, 582-590, and 593-594 are closed
# -------------------------------------------------------------------
# Lines 104-110: _register_subcommands exception handler
# -------------------------------------------------------------------
Scenario: _register_subcommands handles import failure gracefully
Given cmcov3 the _subcommands_registered flag is reset to False
When cmcov3 _register_subcommands is called and an import raises an error
Then cmcov3 the error console should have printed the failure message
And cmcov3 the _subcommands_registered flag should still be False
# -------------------------------------------------------------------
# Lines 582-586: completion command with subprocess returning stdout
# -------------------------------------------------------------------
Scenario: completion command outputs subprocess stdout when present
Given cmcov3 subprocess.run is mocked to return stdout "echo hello"
When cmcov3 the completion command is invoked with shell "bash"
Then cmcov3 the output should contain "echo hello"
# -------------------------------------------------------------------
# Lines 587-590: completion command with empty subprocess stdout
# -------------------------------------------------------------------
Scenario: completion command outputs placeholder when subprocess has no stdout
Given cmcov3 subprocess.run is mocked to return empty stdout
When cmcov3 the completion command is invoked with shell "zsh"
Then cmcov3 the output should contain "# Completion script for zsh"
And cmcov3 the output should contain "# Shell: zsh"
# -------------------------------------------------------------------
# Lines 593-594+: convert_exit_code function coverage
# -------------------------------------------------------------------
Scenario: convert_exit_code returns 0 for None input
When cmcov3 convert_exit_code is called with None
Then cmcov3 the exit code result should be 0
Scenario: convert_exit_code extracts exit_code attribute from object
When cmcov3 convert_exit_code is called with an object having exit_code 42
Then cmcov3 the exit code result should be 42
Scenario: convert_exit_code clamps large positive code to 255
When cmcov3 convert_exit_code is called with integer 999
Then cmcov3 the exit code result should be 255
Scenario: convert_exit_code passes through negative codes
When cmcov3 convert_exit_code is called with integer -1
Then cmcov3 the exit code result should be -1
Scenario: convert_exit_code returns 1 for non-integer string
When cmcov3 convert_exit_code is called with a non-convertible value
Then cmcov3 the exit code result should be 1
Scenario: convert_exit_code handles zero correctly
When cmcov3 convert_exit_code is called with integer 0
Then cmcov3 the exit code result should be 0
Scenario: convert_exit_code handles normal positive code
When cmcov3 convert_exit_code is called with integer 7
Then cmcov3 the exit code result should be 7
+3 -5
View File
@@ -57,11 +57,9 @@ Feature: Plan CLI coverage boost
When I invoke execute with "--format" "json" and plan id
Then the plan coverage command should succeed
And the plan coverage output should contain "plan_id"
And the plan coverage output should contain "sandbox"
And the plan coverage output should contain "worker"
And the plan coverage output should contain "progress"
And the plan coverage output should contain "strategy_summary"
And the plan coverage output should contain "command"
And the plan coverage output should contain "phase"
And the plan coverage output should contain "processing_state"
And the plan coverage output should contain "action_name"
And the plan coverage output should contain "exit_code"
@tdd_issue @tdd_issue_4251 @tdd_expected_fail
+3 -1
View File
@@ -54,7 +54,9 @@ def step_register_subcommands_import_error(context: Any) -> None:
try:
mod._register_subcommands()
except Exception as exc:
except (
BaseException
) as exc: # catch SystemExit(1) raised by _register_subcommands on failure
context.cmcov3_error = exc
finally:
patcher_builtins.stop()
@@ -0,0 +1,374 @@
"""Step definitions for tdd_plan_execute_sandbox_root.feature.
Regression guards for issue #1313 — sandbox_root wiring in the plan execute
pipeline. Tests cover three aspects of the fix:
1. ``PlanApplyService._resolve_changeset`` reconstructs a ``SpecChangeSet``
from entries persisted in ``plan.error_details["changeset_entries_json"]``
when no ``changeset_store`` is wired.
2. ``PlanExecutor._run_execute_with_stub`` serialises lightweight
``ChangeSetEntry`` objects into ``plan.error_details`` after execute so
that ``plan diff`` can show them.
3. ``execute_plan`` (CLI handler) creates the git-worktree sandbox only when
the execute phase actually runs (plan in Execute/QUEUED), not on repeated
invocations where the plan is already in Execute/COMPLETE.
"""
from __future__ import annotations
import json
from datetime import datetime
from unittest.mock import MagicMock, patch
from behave import given, then, when
from behave.runner import Context # type: ignore[import-untyped]
from typer.testing import CliRunner
from ulid import ULID
from cleveragents.application.services.plan_apply_service import PlanApplyService
from cleveragents.application.services.plan_executor import ExecuteResult, PlanExecutor
from cleveragents.cli.commands.plan import app as plan_app
from cleveragents.domain.models.core.change import ChangeOperation
from cleveragents.domain.models.core.plan import (
NamespacedName,
Plan,
PlanIdentity,
PlanPhase,
PlanTimestamps,
ProcessingState,
ProjectLink,
)
from cleveragents.tool.builtins.changeset import ChangeSet, ChangeSetEntry
__all__: list[str] = []
# ---------------------------------------------------------------------------
# Constants — generated once at import time so they are always valid ULIDs
# ---------------------------------------------------------------------------
_PLAN_ID = str(ULID())
_CHANGESET_ID = str(ULID())
_CLI_PLAN_ID = str(ULID())
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_test_plan(
*,
plan_id: str = _PLAN_ID,
phase: PlanPhase = PlanPhase.EXECUTE,
state: ProcessingState = ProcessingState.QUEUED,
decision_root_id: str | None = "01DECISION0000000000001313",
definition_of_done: str = "Create HELLO.md",
changeset_id: str | None = None,
error_details: dict | None = None,
) -> Plan:
"""Build a real Plan domain object for issue #1313 tests."""
return Plan(
identity=PlanIdentity(plan_id=plan_id),
namespaced_name=NamespacedName(namespace="local", name="tdd-1313-plan"),
action_name="local/tdd-1313-action",
description="TDD plan for issue #1313",
definition_of_done=definition_of_done,
phase=phase,
processing_state=state,
decision_root_id=decision_root_id,
project_links=[ProjectLink(project_name="local/test-project")],
strategy_actor="openai/gpt-4o-mini",
execution_actor="openai/gpt-4o-mini",
timestamps=PlanTimestamps(created_at=datetime.now(), updated_at=datetime.now()),
changeset_id=changeset_id,
error_details=error_details,
)
def _make_lifecycle_mock(plan: Plan) -> MagicMock:
"""Return a mock lifecycle service whose get_plan always returns *plan*."""
svc = MagicMock(name="lifecycle_service")
svc.get_plan.return_value = plan
return svc
# ---------------------------------------------------------------------------
# Background
# ---------------------------------------------------------------------------
@given("the plan_apply_service module is available for issue 1313")
def step_module_available(context: Context) -> None:
"""Confirm the PlanApplyService is importable."""
assert PlanApplyService is not None
# ---------------------------------------------------------------------------
# _resolve_changeset: reconstruct from plan metadata (Scenarios 1-4)
# ---------------------------------------------------------------------------
def _build_entries_json(path: str = "HELLO.md") -> str:
"""Return a JSON string with one changeset entry."""
return json.dumps(
[
{
"operation": "create",
"path": path,
"resource_id": _PLAN_ID,
"tool_name": "llm_execute",
"before_hash": None,
"after_hash": None,
}
]
)
@given(
"a plan with changeset_id and changeset_entries_json containing HELLO.md"
" for issue 1313"
)
def step_plan_with_entries(context: Context) -> None:
"""Create a plan with changeset metadata serialised in error_details."""
context.test_plan_1313 = _make_test_plan(
changeset_id=_CHANGESET_ID,
error_details={"changeset_entries_json": _build_entries_json("HELLO.md")},
)
@given("a PlanApplyService with no changeset_store for issue 1313")
def step_apply_service_no_store(context: Context) -> None:
"""Create a PlanApplyService without a changeset_store."""
plan = context.test_plan_1313
lifecycle_mock = _make_lifecycle_mock(plan)
context.apply_service_1313 = PlanApplyService(lifecycle_service=lifecycle_mock)
@when("_resolve_changeset is called for issue 1313")
def step_call_resolve_changeset(context: Context) -> None:
"""Invoke the internal _resolve_changeset method."""
context.resolved_changeset_1313 = context.apply_service_1313._resolve_changeset(
context.test_plan_1313
)
@then("the changeset result has one entry for issue 1313")
def step_result_one_entry(context: Context) -> None:
cs = context.resolved_changeset_1313
assert cs is not None, "Expected a SpecChangeSet but got None"
assert len(cs.entries) == 1, f"Expected 1 entry, got {len(cs.entries)}"
@then('the entry path is "HELLO.md" for issue 1313')
def step_entry_path_hello(context: Context) -> None:
entry = context.resolved_changeset_1313.entries[0]
assert entry.path == "HELLO.md", f"Expected path 'HELLO.md', got '{entry.path}'"
@then('the entry operation is "create" for issue 1313')
def step_entry_operation_create(context: Context) -> None:
entry = context.resolved_changeset_1313.entries[0]
assert entry.operation == ChangeOperation.CREATE, (
f"Expected operation CREATE, got '{entry.operation}'"
)
@when("diff is called with plain format for issue 1313")
def step_call_diff_plain(context: Context) -> None:
"""Call PlanApplyService.diff with plain format."""
context.diff_output_1313 = context.apply_service_1313.diff(_PLAN_ID, fmt="plain")
@then('the diff output contains "HELLO.md" for issue 1313')
def step_diff_contains_hello(context: Context) -> None:
output = context.diff_output_1313
assert "HELLO.md" in output, (
f"Expected 'HELLO.md' in diff output but got:\n{output}"
)
@given("a plan with only changeset_id set and no metadata entries for issue 1313")
def step_plan_no_metadata(context: Context) -> None:
"""Plan has a changeset_id but no changeset_entries_json in error_details."""
context.test_plan_1313 = _make_test_plan(
changeset_id=_CHANGESET_ID,
error_details={"mode": "stub"}, # no changeset_entries_json key
)
@then("the changeset result has no entries for issue 1313")
def step_result_no_entries(context: Context) -> None:
cs = context.resolved_changeset_1313
assert cs is not None, "Expected a SpecChangeSet stub but got None"
assert len(cs.entries) == 0, f"Expected 0 entries, got {len(cs.entries)}"
@given("a plan with changeset_id and malformed changeset_entries_json for issue 1313")
def step_plan_malformed_json(context: Context) -> None:
"""Plan has changeset_entries_json that cannot be parsed as JSON."""
context.test_plan_1313 = _make_test_plan(
changeset_id=_CHANGESET_ID,
error_details={"changeset_entries_json": "NOT VALID JSON {{"},
)
# ---------------------------------------------------------------------------
# Changeset persistence during execute (Scenario 5)
# ---------------------------------------------------------------------------
@given(
"a PlanExecutor wired with a mock execute actor returning HELLO.md for issue 1313"
)
def step_executor_with_mock_actor(context: Context) -> None:
"""Create a PlanExecutor whose execute_actor returns a HELLO.md entry."""
# Build the plan in Execute/QUEUED state with a decision root
context.executor_plan_1313 = _make_test_plan(
phase=PlanPhase.EXECUTE,
state=ProcessingState.QUEUED,
decision_root_id=str(ULID()),
definition_of_done="Create HELLO.md",
)
# Lifecycle mock: get_plan always returns the same plan; capture _commit_plan
context.lifecycle_mock_1313 = _make_lifecycle_mock(context.executor_plan_1313)
# Build the ExecuteResult the mock actor will return
changeset_id = str(ULID())
changeset = ChangeSet(
plan_id=_PLAN_ID,
entries=[
ChangeSetEntry(
operation="create",
path="HELLO.md",
resource_id=_PLAN_ID,
tool_name="llm_execute",
)
],
)
execute_result = ExecuteResult(
changeset_id=changeset_id,
changeset=changeset,
tool_calls_count=1,
sandbox_refs=[],
)
context.mock_execute_actor_1313 = MagicMock(name="execute_actor")
context.mock_execute_actor_1313.execute.return_value = execute_result
context.executor_1313 = PlanExecutor(
lifecycle_service=context.lifecycle_mock_1313,
execute_actor=context.mock_execute_actor_1313,
)
@when("the stub execute phase runs for issue 1313")
def step_run_stub_execute(context: Context) -> None:
"""Call PlanExecutor.run_execute to trigger _run_execute_with_stub."""
context.run_execute_error_1313 = None
try:
context.executor_1313.run_execute(_PLAN_ID)
except Exception as exc:
# Capture but do not re-raise; we check the plan state below.
context.run_execute_error_1313 = exc
@then("plan error_details contains changeset_entries_json for issue 1313")
def step_error_details_has_entries(context: Context) -> None:
"""Assert that _commit_plan was called with a plan having the new key."""
# The plan object is modified in-place; check it directly.
plan = context.executor_plan_1313
assert isinstance(plan.error_details, dict), (
f"error_details should be dict, got {type(plan.error_details)}"
)
assert "changeset_entries_json" in plan.error_details, (
f"'changeset_entries_json' missing from error_details: {plan.error_details}"
)
@then('the persisted entry path is "HELLO.md" for issue 1313')
def step_persisted_entry_path(context: Context) -> None:
"""Assert the serialised entry contains path='HELLO.md'."""
entries_json = context.executor_plan_1313.error_details["changeset_entries_json"]
entries = json.loads(entries_json)
assert len(entries) == 1, f"Expected 1 serialised entry, got {len(entries)}"
assert entries[0]["path"] == "HELLO.md", (
f"Expected path 'HELLO.md', got '{entries[0]['path']}'"
)
# ---------------------------------------------------------------------------
# CLI: sandbox not created for Execute/COMPLETE plan (Scenario 6)
# ---------------------------------------------------------------------------
@given("a CLI runner and mocked services for issue 1313")
def step_cli_runner_and_mocks(context: Context) -> None:
"""Set up a CliRunner and mock service/executor."""
context.cli_runner_1313 = CliRunner()
context.mock_cli_service_1313 = MagicMock(name="lifecycle_service")
context.mock_cli_executor_1313 = MagicMock(name="executor")
context.mock_sandbox_fn_1313 = MagicMock(
name="_create_sandbox_for_plan",
return_value=(None, None),
)
@given("a plan already in Execute/complete state for issue 1313")
def step_plan_execute_complete(context: Context) -> None:
"""Configure the mock service to return an Execute/COMPLETE plan."""
complete_plan = _make_test_plan(
plan_id=_CLI_PLAN_ID,
phase=PlanPhase.EXECUTE,
state=ProcessingState.COMPLETE,
decision_root_id=str(ULID()),
)
# Both get_plan calls in execute_plan return the same complete plan
context.mock_cli_service_1313.get_plan.return_value = complete_plan
@when("plan execute CLI is invoked for issue 1313")
def step_invoke_cli_execute_complete(context: Context) -> None:
"""Invoke plan execute on a plan already in Execute/complete."""
with (
patch(
"cleveragents.cli.commands.plan._get_lifecycle_service",
return_value=context.mock_cli_service_1313,
),
patch(
"cleveragents.cli.commands.plan._create_sandbox_for_plan",
context.mock_sandbox_fn_1313,
),
patch(
"cleveragents.cli.commands.plan._get_plan_executor",
return_value=context.mock_cli_executor_1313,
),
patch(
"cleveragents.cli.commands.plan._notify_facade",
),
):
context.cli_result_1313 = context.cli_runner_1313.invoke(
plan_app,
["execute", _CLI_PLAN_ID, "--format", "plain"],
)
@then("the CLI exits with code 0 for issue 1313")
def step_cli_exits_zero(context: Context) -> None:
result = context.cli_result_1313
assert result.exit_code == 0, (
f"Expected exit code 0 but got {result.exit_code}.\n"
f"Output:\n{result.output}\n"
f"Exception: {result.exception}"
)
@then("_create_sandbox_for_plan was not called for issue 1313")
def step_sandbox_not_called(context: Context) -> None:
"""Assert _create_sandbox_for_plan was never invoked."""
assert context.mock_sandbox_fn_1313.call_count == 0, (
f"_create_sandbox_for_plan was unexpectedly called "
f"{context.mock_sandbox_fn_1313.call_count} time(s)"
)
@@ -0,0 +1,61 @@
@tdd_issue @tdd_issue_1313 @mock_only
Feature: Fix sandbox_root wiring in plan execute pipeline (#1313)
As a developer
I want the git-worktree sandbox to be created only when the execute phase
actually runs, changeset entries to be persisted for plan diff, and the
plan diff to reconstruct those entries even without a changeset_store
So that the M1 acceptance test passes end-to-end
Background:
Given the plan_apply_service module is available for issue 1313
# ------------------------------------------------------------------
# _resolve_changeset: reconstruct from plan metadata
# ------------------------------------------------------------------
Scenario: _resolve_changeset reconstructs SpecChangeSet from plan metadata
Given a plan with changeset_id and changeset_entries_json containing HELLO.md for issue 1313
And a PlanApplyService with no changeset_store for issue 1313
When _resolve_changeset is called for issue 1313
Then the changeset result has one entry for issue 1313
And the entry path is "HELLO.md" for issue 1313
And the entry operation is "create" for issue 1313
Scenario: Plan diff plain output contains file path from metadata entries
Given a plan with changeset_id and changeset_entries_json containing HELLO.md for issue 1313
And a PlanApplyService with no changeset_store for issue 1313
When diff is called with plain format for issue 1313
Then the diff output contains "HELLO.md" for issue 1313
Scenario: _resolve_changeset returns empty stub when no metadata entries present
Given a plan with only changeset_id set and no metadata entries for issue 1313
And a PlanApplyService with no changeset_store for issue 1313
When _resolve_changeset is called for issue 1313
Then the changeset result has no entries for issue 1313
Scenario: _resolve_changeset handles malformed changeset_entries_json gracefully
Given a plan with changeset_id and malformed changeset_entries_json for issue 1313
And a PlanApplyService with no changeset_store for issue 1313
When _resolve_changeset is called for issue 1313
Then the changeset result has no entries for issue 1313
# ------------------------------------------------------------------
# plan_executor: persist changeset entries after execute
# ------------------------------------------------------------------
Scenario: Changeset entries are persisted in plan metadata during execute
Given a PlanExecutor wired with a mock execute actor returning HELLO.md for issue 1313
When the stub execute phase runs for issue 1313
Then plan error_details contains changeset_entries_json for issue 1313
And the persisted entry path is "HELLO.md" for issue 1313
# ------------------------------------------------------------------
# CLI: sandbox not created for already-completed plan
# ------------------------------------------------------------------
Scenario: Sandbox is not created when plan is already in Execute/complete
Given a CLI runner and mocked services for issue 1313
And a plan already in Execute/complete state for issue 1313
When plan execute CLI is invoked for issue 1313
Then the CLI exits with code 0 for issue 1313
And _create_sandbox_for_plan was not called for issue 1313
+41 -14
View File
@@ -226,20 +226,42 @@ Run CLI
RETURN ${result}
Extract JSON From Stdout
[Documentation] Extract the first JSON object from stdout that may contain
... leading non-JSON text (e.g. structlog debug messages).
... Uses ``raw_decode`` with ``strict=False`` to tolerate
... literal control characters that Rich's ``console.print``
... may inject when it wraps long lines inside JSON string
... values. Wrapped in ``TRY/EXCEPT`` for clear failure messages.
...
... **Note:** This picks the *first* JSON object (by scanning for
... the first ``{``). Non-JSON preamble (e.g. structlog lines)
... is mitigated by ``NO_COLOR=1`` (disables Rich formatting) and
... structlog routing to stderr, so stdout typically starts with
... the JSON payload. See ``Safe Parse Json Field`` in
... ``common_e2e.resource`` for a last-line fallback strategy
... that scans in reverse.
[Documentation] Extract the first JSON object from stdout that may contain
... leading non-JSON text (e.g. structlog debug messages).
... Uses ``raw_decode`` with ``strict=False`` to tolerate
... literal control characters that Rich's ``console.print``
... may inject when it wraps long lines inside JSON string
... values. Wrapped in ``TRY/EXCEPT`` for clear failure messages.
...
... **Note:** This picks the *first* JSON object (by scanning for
... the first ``{``). Non-JSON preamble (e.g. structlog lines)
... is mitigated by ``NO_COLOR=1`` (disables Rich formatting) and
... structlog routing to stderr, so stdout typically starts with
... the JSON payload. See ``Safe Parse Json Field`` in
... ``common_e2e.resource`` for a last-line fallback strategy
... that scans in reverse.
...
... **Envelope unwrapping:** When the CLI is invoked with
... ``--format json``, ``format_output`` wraps the payload in a
... spec-required envelope containing ``command``, ``status``,
... ``exit_code``, ``data``, ``timing``, and ``messages`` keys.
... This keyword detects the envelope by the co-presence of
... ``exit_code`` and ``data`` and returns ``envelope["data"]``
... transparently, so callers access payload fields directly
... without needing to unwrap the envelope themselves. Output
... from helper scripts that do not use the CLI envelope (e.g.
... the WF04 snapshot helper) is returned as-is because those
... JSON objects do not contain ``exit_code``.
...
... **Usage recommendation:** All E2E tests that parse JSON from
... ``--format json`` CLI commands should use this keyword to
... ensure automatic envelope unwrapping. Tests that manually
... extract and parse JSON (using custom lambdas, regexes, or
... JSON parsing) should be refactored to use this keyword for
... consistency and correctness. Current candidates for refactoring
... include: wf05_db_migration.robot (decision tree parsing),
... wf12_hierarchical.robot (decision tree parsing), and any
... test that implements custom JSON extraction logic.
[Arguments] ${text}
TRY
${start}= Evaluate $text.index('{')
@@ -247,6 +269,11 @@ Extract JSON From Stdout
EXCEPT AS ${err}
Fail Failed to extract JSON object from stdout: ${err}
END
# Auto-unwrap the spec-required CLI output envelope when present.
# The envelope is identified by the co-presence of 'exit_code' and 'data'.
IF 'exit_code' in $json_obj and 'data' in $json_obj
RETURN ${json_obj}[data]
END
RETURN ${json_obj}
Link Resource To Project
+183 -33
View File
@@ -1,65 +1,215 @@
*** Settings ***
Documentation E2E acceptance test for M1 - minimal plan execution flow.
Documentation E2E acceptance test for M1 minimal plan execution flow.
...
... Exercises the complete M1 (v3.0.0) milestone success criteria
... with **zero mocking**: real CLI invocations, real LLM API keys,
... and real subprocess execution against a temporary git repository.
... Exercises the complete M1 (v3.0.0) milestone success criteria
... with **zero mocking**: real CLI invocations, real LLM API keys,
... and real subprocess execution against a temporary git repository.
...
... Flow: action create → resource add → project create → plan use
... → plan execute (strategize) → plan execute (execute) → plan diff
... → plan apply.
... Flow: action create → resource add → project create → plan use
... → plan execute (strategize) → plan execute (execute) → plan diff
... → plan apply.
...
... Tagged ``tdd_expected_fail`` because ``_get_plan_executor()``
... does not pass ``sandbox_root`` to ``PlanExecutor``, so
... ``LLMExecuteActor._write_to_sandbox()`` is never called and
... LLM-generated files (HELLO.md) are silently discarded.
... See `#1313 <https://git.cleverthis.com/cleveragents/cleveragents-core/issues/1313>`_.
Resource common_e2e.resource
Suite Setup E2E Suite Setup
... Tagged ``tdd_expected_fail`` because ``_get_plan_executor()``
... does not pass ``sandbox_root`` to ``PlanExecutor``, so
... ``LLMExecuteActor._write_to_sandbox()`` is never called and
... LLM-generated files (HELLO.md) are silently discarded.
... See `#1313 <https://git.cleverthis.com/cleveragents/cleveragents-core/issues/1313>`_.
Resource common_e2e.resource
Suite Setup E2E Suite Setup
Suite Teardown E2E Suite Teardown
*** Test Cases ***
M1 Full Plan Lifecycle
[Documentation] Exercise the complete M1 plan lifecycle with real LLM.
...
... Creates an action from YAML, registers a git-checkout
... resource, creates a project, and runs the full plan
... lifecycle through apply with post-apply commit verification.
[Tags] E2E tdd_expected_fail tdd_issue tdd_issue_1313
Skip If No LLM Keys
# ── 1. Create a temporary git repo for isolation ──────────────
${repo_path}= Create Temp Git Repo m1-acceptance-repo
[Documentation] Exercise the complete M1 plan lifecycle with real LLM.
...
... Creates an action from YAML, registers a git-checkout
... resource, creates a project, and runs the full plan
... lifecycle through apply with post-apply commit verification.
[Tags] E2E tdd_expected_fail tdd_issue tdd_issue_1313
Skip If No LLM Keys
# ── 1. Create a temporary git repo for isolation ──────────────
${repo_path}= Create Temp Git Repo m1-acceptance-repo
# ── 2. Write action YAML config ──────────────────────────────
${action_yaml}= Set Variable ${SUITE_HOME}${/}m1_test_action.yaml
${yaml_content}= Catenate SEPARATOR=\n
... name: local/test-action
... description: "M1 E2E acceptance test action"
... strategy_actor: openai/gpt-4o-mini
... execution_actor: openai/gpt-4o-mini
... definition_of_done: |
... ${SPACE}${SPACE}Create a file called HELLO.md with a short greeting.
... reusable: true
... read_only: false
Create File ${action_yaml} ${yaml_content}
# ── 3. Create the action ─────────────────────────────────────
${action_result}= Run CleverAgents Command
... action create --config ${action_yaml}
... --format plain
Should Be Equal As Integers ${action_result.rc} 0
Output Should Contain ${action_result} local/test-action
Output Should Contain ${action_result} M1 E2E acceptance test action
Output Should Contain ${action_result} State: available
# ── 4. Register git-checkout resource ────────────────────────
${resource_result}= Run CleverAgents Command
... resource add git-checkout local/test-repo
... --path ${repo_path} --branch master
... --format plain
Should Be Equal As Integers ${resource_result.rc} 0
... Resource add failed: ${resource_result.stderr}
Output Should Contain ${resource_result} name: local/test-repo
Output Should Contain ${resource_result} type: git-checkout
# ── 5. Create project linked to resource ─────────────────────
${project_result}= Run CleverAgents Command
... project create --resource local/test-repo local/test-project
... --format plain
Should Be Equal As Integers ${project_result.rc} 0
... Project create failed: ${project_result.stderr}
Output Should Contain ${project_result} namespaced_name: local/test-project
Output Should Contain ${project_result} linked_resources:
# ── 6. Plan use — create plan from action + project ──────────
${plan_use_result}= Run CleverAgents Command
... plan use local/test-action local/test-project
... --format plain expected_rc=${0}
# Extract plan ID from output (ULID pattern: 26 alphanumeric chars)
${plan_id}= Extract Plan Id ${plan_use_result.stdout}
Should Not Be Empty ${plan_id} Could not extract plan ID from plan use output
Should Be Equal As Integers ${plan_use_result.rc} 0
... Plan use failed: ${plan_use_result.stderr}
Output Should Contain ${plan_use_result} phase: strategize
Output Should Contain ${plan_use_result} action_name: local/test-action
Output Should Contain ${plan_use_result} automation_profile: manual
Output Should Contain ${plan_use_result} description: M1 E2E acceptance test action
Output Should Contain ${plan_use_result} definition_of_done: Create a file called HELLO.md with a short greeting.
Output Should Contain ${plan_use_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${plan_use_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${plan_use_result} estimation_actor: None
Output Should Contain ${plan_use_result} invariant_actor: None
# ── 7. Plan execute — strategize phase ───────────────────────
${exec1_result}= Run CleverAgents Command
... plan execute ${plan_id} --format plain
... timeout=300s
Log Strategize execute rc=${exec1_result.rc}
# Strategize must succeed (rc=0 enforced)
${exec1_combined}= Set Variable ${exec1_result.stdout}\n${exec1_result.stderr}
Log Strategize output: ${exec1_combined}
Should Be Equal As Integers ${exec1_result.rc} 0
Output Should Contain ${exec1_result} action_name: local/test-action
Output Should Contain ${exec1_result} phase: execute
Output Should Contain ${exec1_result} processing_state: complete
Output Should Contain ${exec1_result} project_links:
Output Should Contain ${exec1_result} local/test-project
Output Should Contain ${exec1_result} description: M1 E2E acceptance test action
Output Should Contain ${exec1_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${exec1_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${exec1_result} definition_of_done: Create a file called HELLO.md with a short greeting.
# ── 8. Plan execute — advance to execute phase ───────────────
${exec2_result}= Run CleverAgents Command
... plan execute ${plan_id} --format plain
... timeout=300s
Log Execute phase rc=${exec2_result.rc}
${exec2_combined}= Set Variable ${exec2_result.stdout}\n${exec2_result.stderr}
Log Execute output: ${exec2_combined}
# Output matches step 7: both report the plan's current state after transition
Should Be Equal As Integers ${exec2_result.rc} 0
Output Should Contain ${exec2_result} action_name: local/test-action
Output Should Contain ${exec2_result} phase: execute
Output Should Contain ${exec2_result} processing_state: complete
Output Should Contain ${exec2_result} project_links:
Output Should Contain ${exec2_result} local/test-project
Output Should Contain ${exec2_result} description: M1 E2E acceptance test action
Output Should Contain ${exec2_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${exec2_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${exec2_result} definition_of_done: Create a file called HELLO.md with a short greeting.
# ── 9. Plan diff — verify changeset contains HELLO.md ──────────
${diff_result}= Run CleverAgents Command
... plan diff ${plan_id} --format plain
... timeout=120s
Log Diff rc=${diff_result.rc}
${diff_combined}= Set Variable ${diff_result.stdout}\n${diff_result.stderr}
Log Diff output: ${diff_combined}
Should Be Equal As Integers ${diff_result.rc} 0
Should Not Be Empty ${diff_result.stdout}
... Plan diff produced no output
# The LLM was asked to create HELLO.md — the diff must show it.
# This assertion will fail until #1313 is resolved (sandbox_root
# is not wired into the execute pipeline).
Output Should Contain ${diff_result} HELLO
# ── 10. Plan apply — apply changes to the repo ───────────────
${apply_result}= Run CleverAgents Command
... plan apply --yes --format plain ${plan_id}
... timeout=300s
Log Apply rc=${apply_result.rc}
${apply_combined}= Set Variable ${apply_result.stdout}\n${apply_result.stderr}
Log Apply output: ${apply_combined}
Should Be Equal As Integers ${apply_result.rc} 0
Output Should Contain ${apply_result} phase: apply
Output Should Contain ${apply_result} processing_state: applied
Output Should Contain ${apply_result} state: applied
Output Should Contain ${apply_result} project_links:
Output Should Contain ${apply_result} - {"project_name": "local/test-project"}
Output Should Contain ${apply_result} arguments:
Output Should Contain ${apply_result} automation_profile: manual
Output Should Contain ${apply_result} action_name: local/test-action
Output Should Contain ${apply_result} description: M1 E2E acceptance test action
Output Should Contain ${apply_result} definition_of_done: Create a file called HELLO.md with a short greeting.
Output Should Contain ${apply_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${apply_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${apply_result} estimation_actor: None
Output Should Contain ${apply_result} invariant_actor: None
# ── 11. Verify post-apply commit and HELLO.md in target repo ──
${git_log}= Run Process git log --oneline
... cwd=${repo_path} timeout=60s on_timeout=kill
Log Git log after apply: ${git_log.stdout}
Should Be Equal As Integers ${git_log.rc} 0
# After apply there must be at least 2 commits: the initial commit
# from Create Temp Git Repo plus the CleverAgents apply commit.
${commit_lines}= Get Line Count ${git_log.stdout}
Should Be True ${commit_lines} >= 2
... Expected at least 2 commits after apply, found ${commit_lines}
# HELLO.md must exist in the repo after apply.
# This assertion will fail until #1313 is resolved.
File Should Exist ${repo_path}${/}HELLO.md
# ── 12. Structural validation summary ────────────────────────
# Action create succeeded (rc=0 verified above)
# Resource add succeeded (rc=0 verified above)
# Project create succeeded (rc=0 verified above)
# Plan use created a plan with a valid ID (verified above)
# Execute phases completed (rc=0, output validated above)
# Diff showed HELLO.md changes (validated above)
# Apply succeeded and produced a commit (validated above)
# HELLO.md exists in target repo (validated above)
Log M1 Full Plan Lifecycle E2E test completed successfully
*** Keywords ***
Extract Plan Id
[Documentation] Extract a ULID-style plan ID from command output.
...
... Searches for a 26-character alphanumeric ULID pattern
... in the output text. Returns the first match or EMPTY.
[Arguments] ${text}
# ULID pattern: 26 uppercase alphanumeric characters (Crockford Base32)
${matches}= Get Regexp Matches ${text} [0-9A-HJ-NP-Z]{26} flags=IGNORECASE
${count}= Get Length ${matches}
${plan_id}= Set Variable If ${count} > 0 ${matches}[0] ${EMPTY}
RETURN ${plan_id}
[Documentation] Extract a ULID-style plan ID from command output.
...
... Searches for a 26-character alphanumeric ULID pattern
... in the output text. Returns the first match or EMPTY.
[Arguments] ${text}
# ULID pattern: 26 uppercase alphanumeric characters (Crockford Base32)
${matches}= Get Regexp Matches ${text} [0-9A-HJ-NP-Z]{26} flags=IGNORECASE
${count}= Get Length ${matches}
${plan_id}= Set Variable If ${count} > 0 ${matches}[0] ${EMPTY}
RETURN ${plan_id}
+244
View File
@@ -0,0 +1,244 @@
*** Settings ***
Documentation E2E acceptance test for M1 — minimal plan execution flow.
...
... Exercises the complete M1 (v3.0.0) milestone success criteria
... with **zero mocking**: real CLI invocations, real LLM API keys,
... and real subprocess execution against a temporary git repository.
...
... Flow: action create → resource add → project create → plan use
... → plan execute (strategize) → plan execute (execute) → plan diff
... → plan apply.
<<<<<<< HEAD
||||||| 703cf13c
... Flow: action create → resource add → project create → plan use
... → plan execute (strategize) → plan execute (execute) → plan diff
... → plan apply.
...
... Tagged ``tdd_expected_fail`` because ``_get_plan_executor()``
... does not pass ``sandbox_root`` to ``PlanExecutor``, so
... ``LLMExecuteActor._write_to_sandbox()`` is never called and
... LLM-generated files (HELLO.md) are silently discarded.
... See `#1313 <https://git.cleverthis.com/cleveragents/cleveragents-core/issues/1313>`_.
Resource common_e2e.resource
Suite Setup E2E Suite Setup
=======
...
... Tagged ``tdd_expected_fail`` because ``_get_plan_executor()``
... does not pass ``sandbox_root`` to ``PlanExecutor``, so
... ``LLMExecuteActor._write_to_sandbox()`` is never called and
... LLM-generated files (HELLO.md) are silently discarded.
... See `#1313 <https://git.cleverthis.com/cleveragents/cleveragents-core/issues/1313>`_.
>>>>>>> test/restore-e2e-tests
Resource common_e2e.resource
Suite Setup E2E Suite Setup
Suite Teardown E2E Suite Teardown
*** Test Cases ***
M1 Full Plan Lifecycle
[Documentation] Exercise the complete M1 plan lifecycle with real LLM.
...
... Creates an action from YAML, registers a git-checkout
... resource, creates a project, and runs the full plan
... lifecycle through apply with post-apply commit verification.
<<<<<<< HEAD
[Tags] E2E tdd_issue tdd_issue_1313
||||||| 703cf13c
[Documentation] Exercise the complete M1 plan lifecycle with real LLM.
...
... Creates an action from YAML, registers a git-checkout
... resource, creates a project, and runs the full plan
... lifecycle through apply with post-apply commit verification.
[Tags] E2E tdd_expected_fail tdd_issue tdd_issue_1313
Skip If No LLM Keys
# ── 1. Create a temporary git repo for isolation ──────────────
${repo_path}= Create Temp Git Repo m1-acceptance-repo
=======
[Tags] E2E tdd_expected_fail tdd_issue tdd_issue_1313
>>>>>>> test/restore-e2e-tests
Skip If No LLM Keys
# ── 1. Create a temporary git repo for isolation ──────────────
${repo_path}= Create Temp Git Repo m1-acceptance-repo
# ── 2. Write action YAML config ──────────────────────────────
${action_yaml}= Set Variable ${SUITE_HOME}${/}m1_test_action.yaml
${yaml_content}= Catenate SEPARATOR=\n
... name: local/test-action
... description: "M1 E2E acceptance test action"
... strategy_actor: openai/gpt-4o-mini
... execution_actor: openai/gpt-4o-mini
... definition_of_done: |
... ${SPACE}${SPACE}Create a file called HELLO.md with a short greeting.
... reusable: true
... read_only: false
Create File ${action_yaml} ${yaml_content}
# ── 3. Create the action ─────────────────────────────────────
${action_result}= Run CleverAgents Command
... action create --config ${action_yaml}
... --format plain
Should Be Equal As Integers ${action_result.rc} 0
Output Should Contain ${action_result} local/test-action
Output Should Contain ${action_result} M1 E2E acceptance test action
Output Should Contain ${action_result} State: available
# ── 4. Register git-checkout resource ────────────────────────
${resource_result}= Run CleverAgents Command
... resource add git-checkout local/test-repo
... --path ${repo_path} --branch master
... --format plain
Should Be Equal As Integers ${resource_result.rc} 0
... Resource add failed: ${resource_result.stderr}
Output Should Contain ${resource_result} name: local/test-repo
Output Should Contain ${resource_result} type: git-checkout
# ── 5. Create project linked to resource ─────────────────────
${project_result}= Run CleverAgents Command
... project create --resource local/test-repo local/test-project
... --format plain
Should Be Equal As Integers ${project_result.rc} 0
... Project create failed: ${project_result.stderr}
Output Should Contain ${project_result} namespaced_name: local/test-project
Output Should Contain ${project_result} linked_resources:
# ── 6. Plan use — create plan from action + project ──────────
${plan_use_result}= Run CleverAgents Command
... plan use local/test-action local/test-project
... --format plain expected_rc=${0}
# Extract plan ID from output (ULID pattern: 26 alphanumeric chars)
${plan_id}= Extract Plan Id ${plan_use_result.stdout}
Should Not Be Empty ${plan_id} Could not extract plan ID from plan use output
Should Be Equal As Integers ${plan_use_result.rc} 0
... Plan use failed: ${plan_use_result.stderr}
Output Should Contain ${plan_use_result} phase: strategize
Output Should Contain ${plan_use_result} action_name: local/test-action
Output Should Contain ${plan_use_result} automation_profile: manual
Output Should Contain ${plan_use_result} description: M1 E2E acceptance test action
Output Should Contain ${plan_use_result} definition_of_done: Create a file called HELLO.md with a short greeting.
Output Should Contain ${plan_use_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${plan_use_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${plan_use_result} estimation_actor: None
Output Should Contain ${plan_use_result} invariant_actor: None
# ── 7. Plan execute — strategize phase ───────────────────────
${exec1_result}= Run CleverAgents Command
... plan execute ${plan_id} --format plain
... timeout=300s
Log Strategize execute rc=${exec1_result.rc}
# Strategize must succeed (rc=0 enforced)
${exec1_combined}= Set Variable ${exec1_result.stdout}\n${exec1_result.stderr}
Log Strategize output: ${exec1_combined}
Should Be Equal As Integers ${exec1_result.rc} 0
Output Should Contain ${exec1_result} action_name: local/test-action
Output Should Contain ${exec1_result} phase: execute
Output Should Contain ${exec1_result} processing_state: complete
Output Should Contain ${exec1_result} project_links:
Output Should Contain ${exec1_result} local/test-project
Output Should Contain ${exec1_result} description: M1 E2E acceptance test action
Output Should Contain ${exec1_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${exec1_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${exec1_result} definition_of_done: Create a file called HELLO.md with a short greeting.
# ── 8. Plan execute — advance to execute phase ───────────────
${exec2_result}= Run CleverAgents Command
... plan execute ${plan_id} --format plain
... timeout=300s
Log Execute phase rc=${exec2_result.rc}
${exec2_combined}= Set Variable ${exec2_result.stdout}\n${exec2_result.stderr}
Log Execute output: ${exec2_combined}
# Output matches step 7: both report the plan's current state after transition
Should Be Equal As Integers ${exec2_result.rc} 0
Output Should Contain ${exec2_result} action_name: local/test-action
Output Should Contain ${exec2_result} phase: execute
Output Should Contain ${exec2_result} processing_state: complete
Output Should Contain ${exec2_result} project_links:
Output Should Contain ${exec2_result} local/test-project
Output Should Contain ${exec2_result} description: M1 E2E acceptance test action
Output Should Contain ${exec2_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${exec2_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${exec2_result} definition_of_done: Create a file called HELLO.md with a short greeting.
# ── 9. Plan diff — verify changeset contains HELLO.md ──────────
${diff_result}= Run CleverAgents Command
... plan diff ${plan_id} --format plain
... timeout=120s
Log Diff rc=${diff_result.rc}
${diff_combined}= Set Variable ${diff_result.stdout}\n${diff_result.stderr}
Log Diff output: ${diff_combined}
Should Be Equal As Integers ${diff_result.rc} 0
Should Not Be Empty ${diff_result.stdout}
... Plan diff produced no output
# The LLM was asked to create HELLO.md — the diff must show it.
# This assertion will fail until #1313 is resolved (sandbox_root
# is not wired into the execute pipeline).
Output Should Contain ${diff_result} HELLO
# ── 10. Plan apply — apply changes to the repo ───────────────
${apply_result}= Run CleverAgents Command
... plan apply --yes --format plain ${plan_id}
... timeout=300s
Log Apply rc=${apply_result.rc}
${apply_combined}= Set Variable ${apply_result.stdout}\n${apply_result.stderr}
Log Apply output: ${apply_combined}
Should Be Equal As Integers ${apply_result.rc} 0
Output Should Contain ${apply_result} phase: apply
Output Should Contain ${apply_result} processing_state: applied
Output Should Contain ${apply_result} state: applied
Output Should Contain ${apply_result} project_links:
Output Should Contain ${apply_result} - {"project_name": "local/test-project"}
Output Should Contain ${apply_result} arguments:
Output Should Contain ${apply_result} automation_profile: manual
Output Should Contain ${apply_result} action_name: local/test-action
Output Should Contain ${apply_result} description: M1 E2E acceptance test action
Output Should Contain ${apply_result} definition_of_done: Create a file called HELLO.md with a short greeting.
Output Should Contain ${apply_result} strategy_actor: openai/gpt-4o-mini
Output Should Contain ${apply_result} execution_actor: openai/gpt-4o-mini
Output Should Contain ${apply_result} estimation_actor: None
Output Should Contain ${apply_result} invariant_actor: None
# ── 11. Verify post-apply commit and HELLO.md in target repo ──
${git_log}= Run Process git log --oneline
... cwd=${repo_path} timeout=60s on_timeout=kill
Log Git log after apply: ${git_log.stdout}
Should Be Equal As Integers ${git_log.rc} 0
# After apply there must be at least 2 commits: the initial commit
# from Create Temp Git Repo plus the CleverAgents apply commit.
${commit_lines}= Get Line Count ${git_log.stdout}
Should Be True ${commit_lines} >= 2
... Expected at least 2 commits after apply, found ${commit_lines}
# HELLO.md must exist in the repo after apply.
# This assertion will fail until #1313 is resolved.
File Should Exist ${repo_path}${/}HELLO.md
# ── 12. Structural validation summary ────────────────────────
# Action create succeeded (rc=0 verified above)
# Resource add succeeded (rc=0 verified above)
# Project create succeeded (rc=0 verified above)
# Plan use created a plan with a valid ID (verified above)
# Execute phases completed (rc=0, output validated above)
# Diff showed HELLO.md changes (validated above)
# Apply succeeded and produced a commit (validated above)
# HELLO.md exists in target repo (validated above)
Log M1 Full Plan Lifecycle E2E test completed successfully
*** Keywords ***
Extract Plan Id
[Documentation] Extract a ULID-style plan ID from command output.
...
... Searches for a 26-character alphanumeric ULID pattern
... in the output text. Returns the first match or EMPTY.
[Arguments] ${text}
# ULID pattern: 26 uppercase alphanumeric characters (Crockford Base32)
${matches}= Get Regexp Matches ${text} [0-9A-HJ-NP-Z]{26} flags=IGNORECASE
${count}= Get Length ${matches}
${plan_id}= Set Variable If ${count} > 0 ${matches}[0] ${EMPTY}
RETURN ${plan_id}
+115 -31
View File
@@ -1,48 +1,132 @@
*** Settings ***
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
Documentation E2E acceptance test for M2 (v3.1.0): Actor Compiler + Full LLM Integration.
...
... Exercises actor YAML compilation into functional graphs, skill registry,
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
... real provider keys.
Resource common_e2e.resource
Suite Setup E2E Suite Setup
... Exercises actor YAML compilation into functional graphs, skill registry,
... tool lifecycle, and plan execution with a custom actor using real LLM keys.
... Zero mocking — all CLI invocations hit the real CleverAgents binary with
... real provider keys.
Resource common_e2e.resource
Suite Setup E2E Suite Setup
Suite Teardown E2E Suite Teardown
*** Variables ***
${ACTOR_NAME} local/m2-e2e-actor
${ACTION_NAME} local/m2-e2e-action
${RESOURCE_NAME} local/m2-e2e-repo
${PROJECT_NAME} local/m2-e2e-project
${ACTOR_NAME} local/m2-e2e-actor
${ACTION_NAME} local/m2-e2e-action
${RESOURCE_NAME} local/m2-e2e-repo
${PROJECT_NAME} local/m2-e2e-project
*** Test Cases ***
M2 Full Actor Compiler And LLM Integration
[Documentation] End-to-end acceptance test for the M2 milestone.
...
... Exercises actor YAML compilation, registration via CLI,
... resource and project setup, action creation referencing a
... custom actor, and the full plan lifecycle (use, execute
... strategize, execute, diff, apply) with real LLM API keys.
[Tags] E2E tdd_issue tdd_issue_4189 tdd_expected_fail
Skip If No LLM Keys
# ---- Step 1: Create temp git repo with sample project files ----
${repo_dir}= Create Temp Git Repo m2-e2e-repo
Create Directory ${repo_dir}${/}src
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above.
${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above.
# Detect the default branch name created by git init
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
${branch}= Strip String ${branch_result.stdout}
Log Detected branch: ${branch}
[Documentation] End-to-end acceptance test for the M2 milestone.
...
... Exercises actor YAML compilation, registration via CLI,
... resource and project setup, action creation referencing a
... custom actor, and the full plan lifecycle (use, execute
... strategize, execute, diff, apply) with real LLM API keys.
[Tags] E2E
Skip If No LLM Keys
# ---- Step 1: Create temp git repo with sample project files ----
${repo_dir}= Create Temp Git Repo m2-e2e-repo
Create Directory ${repo_dir}${/}src
Create File ${repo_dir}${/}src${/}main.py print("hello world")\n
${r_add}= Run Process git add . cwd=${repo_dir} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc}). Check DEBUG logs above.
${r_commit}= Run Process git commit -m Add source files cwd=${repo_dir} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc}). Check DEBUG logs above.
# Detect the default branch name created by git init
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
${branch}= Strip String ${branch_result.stdout}
Log Detected branch: ${branch}
# ---- Step 2: Create and register custom actor YAML ----
${actor_yaml}= Catenate SEPARATOR=\n
... name: ${ACTOR_NAME}
... type: llm
... description: M2 E2E acceptance actor for compiler and LLM integration
... version: "1.0"
... model: gpt-4
${actor_yaml_path}= Set Variable ${SUITE_HOME}${/}m2_actor.yaml
Create File ${actor_yaml_path} ${actor_yaml}
${actor_config}= Catenate SEPARATOR=\n
... {
... "name": "${ACTOR_NAME}",
... "provider": "openai",
... "model": "gpt-4",
... "options": {"temperature": 0.2}
... }
${config_path}= Set Variable ${SUITE_HOME}${/}actor_config.json
Create File ${config_path} ${actor_config}
${r_actor}= Run CleverAgents Command
... actor add ${ACTOR_NAME} --config ${config_path} --format plain
Should Not Contain ${r_actor.stdout}${r_actor.stderr} Traceback
Log Actor registration: ${r_actor.stdout}
# ---- Step 3: Register resource and create project ----
${r_resource}= Run CleverAgents Command
... resource add git-checkout ${RESOURCE_NAME}
... --path ${repo_dir} --branch ${branch} --format plain
Output Should Contain ${r_resource} ${RESOURCE_NAME}
${r_project}= Run CleverAgents Command
... project create ${PROJECT_NAME}
... --description 'M2 E2E acceptance project'
... --resource ${RESOURCE_NAME} --format plain
Output Should Contain ${r_project} ${PROJECT_NAME}
# ---- Step 4: Create action referencing the custom actor ----
${action_yaml}= Catenate SEPARATOR=\n
... name: ${ACTION_NAME}
... description: M2 acceptance test action for actor compiler and LLM integration
... definition_of_done: Generate or modify at least one source file
... strategy_actor: openai/gpt-4
... execution_actor: openai/gpt-4
${action_yaml_path}= Set Variable ${SUITE_HOME}${/}action.yaml
Create File ${action_yaml_path} ${action_yaml}
${r_action}= Run CleverAgents Command
... action create --config ${action_yaml_path} --format plain
Output Should Contain ${r_action} ${ACTION_NAME}
# ---- Step 5: Plan use ----
${r_use}= Run CleverAgents Command
... plan use ${ACTION_NAME} ${PROJECT_NAME} --format plain
Should Not Be Empty ${r_use.stdout}
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-Z]{26}
Should Not Be Empty ${plan_ids} msg=Expected a ULID plan ID in plan use output
${plan_id}= Set Variable ${plan_ids}[0]
Log Extracted plan_id: ${plan_id}
# ---- Step 6: Plan execute — strategize phase ----
${r_strategize}= Run CleverAgents Command
... plan execute ${plan_id} --format plain
... timeout=180s
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} INTERNAL
Should Not Contain ${r_strategize.stdout}${r_strategize.stderr} Traceback
Log Strategize phase output: ${r_strategize.stdout}
# ---- Step 7: Plan execute — execute phase ----
${r_execute}= Run CleverAgents Command
... plan execute ${plan_id} --format plain
... timeout=180s
Should Not Contain ${r_execute.stdout}${r_execute.stderr} INTERNAL
Should Not Contain ${r_execute.stdout}${r_execute.stderr} Traceback
Log Execute phase output: ${r_execute.stdout}
# ---- Step 8: Plan diff ----
${r_diff}= Run CleverAgents Command
... plan diff ${plan_id} --format plain
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
Log Diff output: ${r_diff.stdout}
# ---- Step 9: Plan apply ----
${r_apply}= Run CleverAgents Command
... plan apply --yes ${plan_id} --format plain
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
Log Apply output: ${r_apply.stdout}
# ---- Step 10: Verify actor compilation and plan integrity ----
${r_status}= Run CleverAgents Command
... plan status ${plan_id} --format plain
Should Not Be Empty ${r_status.stdout}
Output Should Contain ${r_status} ${plan_id}
Log Final plan status: ${r_status.stdout}
File diff suppressed because it is too large Load Diff
+400 -389
View File
@@ -1,430 +1,441 @@
*** Settings ***
Documentation E2E acceptance criteria for M6 (v3.5.0) — autonomy hardening.
Documentation E2E acceptance criteria for M6 (v3.5.0) — autonomy hardening.
...
... Validates the full CleverAgents CLI with **zero mocking**:
... session lifecycle, automation profiles, project setup,
... plan lifecycle via A2A facade, guard enforcement, and
... a full autonomy acceptance flow. LLM-dependent tests
... use ``Skip If No LLM Keys`` for graceful degradation.
Resource common_e2e.resource
Suite Setup M6 Suite Setup
... Validates the full CleverAgents CLI with **zero mocking**:
... session lifecycle, automation profiles, project setup,
... plan lifecycle via A2A facade, guard enforcement, and
... a full autonomy acceptance flow. LLM-dependent tests
... use ``Skip If No LLM Keys`` for graceful degradation.
Resource common_e2e.resource
Suite Setup M6 Suite Setup
Suite Teardown E2E Suite Teardown
Force Tags E2E
Force Tags E2E
*** Keywords ***
M6 Suite Setup
[Documentation] E2E Suite Setup plus unique run suffix and action registration.
E2E Suite Setup
# Generate a unique suffix for resource/project names to avoid UNIQUE
# constraint collisions on repeated E2E runs against the same database.
# uuid4 provides ~4 billion possibilities vs randint's 9000 for parallel CI safety.
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
# Register the local/code-review action needed by plan lifecycle tests.
# Probe the OpenAI API to confirm the key is usable before selecting actor.
# Falls back to Anthropic when the key is absent or quota-exhausted.
${actor}= Resolve LLM Actor
${action_yaml}= Catenate SEPARATOR=\n
... name: local/code-review
... description: "Perform a code review on project sources"
... strategy_actor: ${actor}
... execution_actor: ${actor}
... definition_of_done: "Code review completed: issues identified and documented."
... reusable: true
... read_only: false
... state: available
${yaml_path}= Set Variable ${SUITE_HOME}${/}code-review-action.yaml
Create File ${yaml_path} ${action_yaml}
${action_create}= Run CleverAgents Command action create --config ${yaml_path} expected_rc=None
IF ${action_create.rc} != 0
Log Could not register local/code-review action (rc=${action_create.rc}): ${action_create.stderr} WARN
END
[Documentation] E2E Suite Setup plus unique run suffix and action registration.
E2E Suite Setup
# Generate a unique suffix for resource/project names to avoid UNIQUE
# constraint collisions on repeated E2E runs against the same database.
# uuid4 provides ~4 billion possibilities vs randint's 9000 for parallel CI safety.
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
# Register the local/code-review action needed by plan lifecycle tests.
# Pick an actor that matches the available API key.
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
IF ${has_openai}
${actor}= Set Variable openai/gpt-4o
ELSE
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
END
${action_yaml}= Catenate SEPARATOR=\n
... name: local/code-review
... description: "Perform a code review on project sources"
... strategy_actor: ${actor}
... execution_actor: ${actor}
... definition_of_done: "Code review completed: issues identified and documented."
... reusable: true
... read_only: false
... state: available
${yaml_path}= Set Variable ${SUITE_HOME}${/}code-review-action.yaml
Create File ${yaml_path} ${action_yaml}
${action_create}= Run CleverAgents Command action create --config ${yaml_path} expected_rc=None
IF ${action_create.rc} != 0
Log Could not register local/code-review action (rc=${action_create.rc}): ${action_create.stderr} WARN
END
Setup Plan Test Resources
[Documentation] Create temp git repo, register as resource, and create a project.
... Returns the project name. Shared across LLM-dependent plan tests
... to eliminate boilerplate (P2-22: keep file under 500 lines).
[Arguments] ${prefix}
${repo_path}= Create Temp Git Repo ${prefix}-repo-${RUN_SUFFIX}
${res_name}= Set Variable ${prefix}-res-${RUN_SUFFIX}
${add_res}= Run CleverAgents Command resource add git-checkout ${res_name} --path ${repo_path}
Should Be Equal As Integers ${add_res.rc} 0
${proj_name}= Set Variable ${prefix}-proj-${RUN_SUFFIX}
${create_proj}= Run CleverAgents Command project create --resource ${res_name} ${proj_name}
Should Be Equal As Integers ${create_proj.rc} 0
RETURN ${proj_name}
[Documentation] Create temp git repo, register as resource, and create a project.
... Returns the project name. Shared across LLM-dependent plan tests
... to eliminate boilerplate (P2-22: keep file under 500 lines).
[Arguments] ${prefix}
${repo_path}= Create Temp Git Repo ${prefix}-repo-${RUN_SUFFIX}
${res_name}= Set Variable ${prefix}-res-${RUN_SUFFIX}
${add_res}= Run CleverAgents Command resource add git-checkout ${res_name} --path ${repo_path}
Should Be Equal As Integers ${add_res.rc} 0
${proj_name}= Set Variable ${prefix}-proj-${RUN_SUFFIX}
${create_proj}= Run CleverAgents Command project create --resource ${res_name} ${proj_name}
Should Be Equal As Integers ${create_proj.rc} 0
RETURN ${proj_name}
Plan Lifecycle Assertions
[Documentation] Assert plan lifecycle output after successful plan use.
[Arguments] ${plan_use}
Output Should Contain ${plan_use} plan_id
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
Verify Plan In List ${plan_id}
[Documentation] Assert plan lifecycle output after successful plan use.
[Arguments] ${plan_use}
Output Should Contain ${plan_use} plan_id
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
Verify Plan In List ${plan_id}
Verify Plan In List
[Documentation] Verify a plan appears in list output.
[Arguments] ${plan_id}
${list_result}= Run CleverAgents Command plan list --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${list_result.rc} 0 msg=list failed (rc=${list_result.rc}): ${list_result.stderr}
Output Should Contain ${list_result} ${plan_id}
[Documentation] Verify a plan appears in list output.
[Arguments] ${plan_id}
${list_result}= Run CleverAgents Command plan list --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${list_result.rc} 0
... list failed (rc=${list_result.rc}): ${list_result.stderr}
Output Should Contain ${list_result} ${plan_id}
Guard Enforcement Assertions
[Documentation] Assert guard/profile information is present in plan output.
...
... Verifies the resolved automation profile matches the expected
... profile name and checks for guard-specific fields in output.
[Arguments] ${plan_use} ${expected_profile}=manual
Output Should Contain ${plan_use} plan_id
# Verify the resolved automation profile matches expectation (P2-24)
${resolved_profile}= Safe Parse Json Field ${plan_use.stdout} automation_profile
IF '${resolved_profile}' != ''
Should Be Equal As Strings ${resolved_profile} ${expected_profile}
END
# Verify automation-profile-specific fields appear in combined output
${combined}= Set Variable ${plan_use.stdout} ${plan_use.stderr}
${combined_lower}= Evaluate ($combined).lower()
${has_profile}= Evaluate 'automation_profile' in $combined_lower or 'automation-profile' in $combined_lower or 'require_sandbox' in $combined_lower or 'decompose_task' in $combined_lower or 'create_tool' in $combined_lower or 'safety_profile' in $combined_lower or '"${expected_profile}"' in $combined_lower
Should Be True ${has_profile} Plan output should reference automation profile fields (automation_profile, require_sandbox, decompose_task, etc.)
[Documentation] Assert guard/profile information is present in plan output.
...
... Verifies the resolved automation profile matches the expected
... profile name and checks for guard-specific fields in output.
[Arguments] ${plan_use} ${expected_profile}=manual
Output Should Contain ${plan_use} plan_id
# Verify the resolved automation profile matches expectation (P2-24)
${resolved_profile}= Safe Parse Json Field ${plan_use.stdout} automation_profile
IF '${resolved_profile}' != ''
Should Be Equal As Strings ${resolved_profile} ${expected_profile}
END
# Verify automation-profile-specific fields appear in combined output
${combined}= Set Variable ${plan_use.stdout} ${plan_use.stderr}
${combined_lower}= Evaluate ($combined).lower()
${has_profile}= Evaluate 'automation_profile' in $combined_lower or 'automation-profile' in $combined_lower or 'require_sandbox' in $combined_lower or 'decompose_task' in $combined_lower or 'create_tool' in $combined_lower or 'safety_profile' in $combined_lower or '"${expected_profile}"' in $combined_lower
Should Be True ${has_profile} Plan output should reference automation profile fields (automation_profile, require_sandbox, decompose_task, etc.)
Full Flow Apply Step
[Documentation] Attempt apply after execute succeeds and verify state transition.
... Apply may fail if the plan is not in the correct state after
... LLM execution; hard assertions on the success path only.
[Arguments] ${plan_id}
${apply}= Run CleverAgents Command plan apply --yes ${plan_id} --format json expected_rc=None timeout=180s
Log Apply result rc=${apply.rc} stdout=${apply.stdout} stderr=${apply.stderr}
IF ${apply.rc} == 0
Output Should Contain ${apply} ${plan_id}
# Verify the plan actually transitioned — check for apply-phase indicators
${apply_phase}= Safe Parse Json Field ${apply.stdout} phase
IF '${apply_phase}' != ''
Should Contain ${apply_phase.lower()} apply Plan phase should indicate apply after apply
END
ELSE
Fail apply failed (rc=${apply.rc}) stdout=${apply.stdout} stderr=${apply.stderr}
END
[Documentation] Attempt apply after execute succeeds and verify state transition.
... Apply may fail if the plan is not in the correct state after
... LLM execution; hard assertions on the success path only.
[Arguments] ${plan_id}
${apply}= Run CleverAgents Command plan apply --yes ${plan_id} --format json expected_rc=None timeout=180s
Log Apply result rc=${apply.rc} stdout=${apply.stdout} stderr=${apply.stderr}
IF ${apply.rc} == 0
Output Should Contain ${apply} ${plan_id}
# Verify the plan actually transitioned — check for apply-phase indicators
${apply_phase}= Safe Parse Json Field ${apply.stdout} phase
IF '${apply_phase}' != ''
Should Contain ${apply_phase.lower()} apply Plan phase should indicate apply after apply
END
ELSE
Fail apply failed (rc=${apply.rc}) stdout=${apply.stdout} stderr=${apply.stderr}
END
*** Test Cases ***
M6 E2E Session Lifecycle
[Documentation] Create, list, show, and delete a session via the CLI.
[Teardown] Run Keyword And Ignore Error Run CleverAgents Command session delete ${session_id} --yes expected_rc=None
# Initialise variable so teardown never references an undefined name.
Set Test Variable ${session_id} ${EMPTY}
# Create a session and capture its ID via JSON output
${create}= Run CleverAgents Command session create --format json
Should Not Be Empty ${create.stdout}
Output Should Contain ${create} session_id
# Parse session_id from JSON output (handle potential leading non-JSON output)
${session_id}= Safe Parse Json Field ${create.stdout} session_id
Set Test Variable ${session_id}
Should Not Be Empty ${session_id}
# List sessions — should include the new one
${list_result}= Run CleverAgents Command session list --format json
Output Should Contain ${list_result} ${session_id}
# Show session details
${show_result}= Run CleverAgents Command session show ${session_id} --format json
Output Should Contain ${show_result} ${session_id}
# Delete session
${delete_result}= Run CleverAgents Command session delete ${session_id} --yes
Should Be Equal As Integers ${delete_result.rc} 0
Output Should Contain ${delete_result} deleted
# Confirm deletion by re-listing — the session should no longer appear.
${post_delete_list}= Run CleverAgents Command session list --format json
${post_combined}= Set Variable ${post_delete_list.stdout}\n${post_delete_list.stderr}
Should Not Contain ${post_combined} ${session_id} Session ${session_id} still appears after deletion
[Documentation] Create, list, show, and delete a session via the CLI.
[Teardown] Run Keyword And Ignore Error Run CleverAgents Command session delete ${session_id} --yes expected_rc=None
# Initialise variable so teardown never references an undefined name.
Set Test Variable ${session_id} ${EMPTY}
# Create a session and capture its ID via JSON output
${create}= Run CleverAgents Command session create --format json
Should Not Be Empty ${create.stdout}
Output Should Contain ${create} session_id
# Parse session_id from JSON output (handle potential leading non-JSON output)
${session_id}= Safe Parse Json Field ${create.stdout} session_id
Set Test Variable ${session_id}
Should Not Be Empty ${session_id}
# List sessions — should include the new one
${list_result}= Run CleverAgents Command session list --format json
Output Should Contain ${list_result} ${session_id}
# Show session details
${show_result}= Run CleverAgents Command session show ${session_id} --format json
Output Should Contain ${show_result} ${session_id}
# Delete session
${delete_result}= Run CleverAgents Command session delete ${session_id} --yes
Should Be Equal As Integers ${delete_result.rc} 0
Output Should Contain ${delete_result} deleted
# Confirm deletion by re-listing — the session should no longer appear.
${post_delete_list}= Run CleverAgents Command session list --format json
${post_combined}= Set Variable ${post_delete_list.stdout}\n${post_delete_list.stderr}
Should Not Contain ${post_combined} ${session_id} Session ${session_id} still appears after deletion
M6 E2E Automation Profile List
[Documentation] Verify all eight built-in automation profiles are listed.
${result}= Run CleverAgents Command automation-profile list --format json
Should Be Equal As Integers ${result.rc} 0
Should Not Be Empty ${result.stdout}
# Verify all 8 built-in profiles are present (spec: manual through full-auto).
Output Should Contain ${result} manual
Output Should Contain ${result} review
Output Should Contain ${result} supervised
Output Should Contain ${result} cautious
Output Should Contain ${result} trusted
# Use case-sensitive match with JSON-quoted forms for short profile names
# to avoid false-positive substring matches (e.g. "ci" in "deficit").
Output Should Contain ${result} "auto" case_insensitive=${FALSE}
Output Should Contain ${result} "ci" case_insensitive=${FALSE}
Output Should Contain ${result} full-auto
[Documentation] Verify all eight built-in automation profiles are listed.
${result}= Run CleverAgents Command automation-profile list --format json
Should Be Equal As Integers ${result.rc} 0
Should Not Be Empty ${result.stdout}
# Verify all 8 built-in profiles are present (spec: manual through full-auto).
Output Should Contain ${result} manual
Output Should Contain ${result} review
Output Should Contain ${result} supervised
Output Should Contain ${result} cautious
Output Should Contain ${result} trusted
# Use case-sensitive match with JSON-quoted forms for short profile names
# to avoid false-positive substring matches (e.g. "ci" in "deficit").
Output Should Contain ${result} "auto" case_insensitive=${FALSE}
Output Should Contain ${result} "ci" case_insensitive=${FALSE}
Output Should Contain ${result} full-auto
M6 E2E Automation Profile Show
[Documentation] Verify showing a single automation profile returns threshold info.
${result}= Run CleverAgents Command automation-profile show manual
Should Be Equal As Integers ${result.rc} 0
Should Not Be Empty ${result.stdout}
Output Should Contain ${result} manual
# Threshold fields should be present
Output Should Contain ${result} decompose_task
Output Should Contain ${result} create_tool
Output Should Contain ${result} select_tool
[Documentation] Verify showing a single automation profile returns threshold info.
${result}= Run CleverAgents Command automation-profile show manual
Should Be Equal As Integers ${result.rc} 0
Should Not Be Empty ${result.stdout}
Output Should Contain ${result} manual
# Threshold fields should be present
Output Should Contain ${result} decompose_task
Output Should Contain ${result} create_tool
Output Should Contain ${result} select_tool
M6 E2E Config Automation Profile
[Documentation] Set and get the automation profile configuration key.
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
# Set the profile to ci
${set_result}= Run CleverAgents Command config set core.automation-profile ci --format json
Should Be Equal As Integers ${set_result.rc} 0
# Use case-sensitive JSON-quoted match for "ci" to avoid false positives on short strings
Output Should Contain ${set_result} "ci" case_insensitive=${FALSE}
# Get and verify via JSON — parse the value field for precise assertion
${get_result}= Run CleverAgents Command config get core.automation-profile --format json
Should Be Equal As Integers ${get_result.rc} 0
${config_value}= Safe Parse Json Field ${get_result.stdout} value
Should Be Equal As Strings ${config_value} ci
# Reset to default (manual) is handled by [Teardown].
[Documentation] Set and get the automation profile configuration key.
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
# Set the profile to ci
${set_result}= Run CleverAgents Command config set core.automation-profile ci --format json
Should Be Equal As Integers ${set_result.rc} 0
# Use case-sensitive JSON-quoted match for "ci" to avoid false positives on short strings
Output Should Contain ${set_result} "ci" case_insensitive=${FALSE}
# Get and verify via JSON — parse the value field for precise assertion
${get_result}= Run CleverAgents Command config get core.automation-profile --format json
Should Be Equal As Integers ${get_result.rc} 0
${config_value}= Safe Parse Json Field ${get_result.stdout} value
Should Be Equal As Strings ${config_value} ci
# Reset to default (manual) is handled by [Teardown].
M6 E2E Init And Project Setup
[Documentation] Initialize, add a git resource, create a project, and verify.
[Teardown] Log Init And Project Setup test completed (cleanup via suite teardown)
# Create a temp git repo for the resource
${repo_path}= Create Temp Git Repo m6-setup-repo-${RUN_SUFFIX}
# Add a git-checkout resource with a run-unique name
${res_name}= Set Variable m6-setup-res-${RUN_SUFFIX}
${add_res}= Run CleverAgents Command resource add git-checkout ${res_name} --path ${repo_path}
Should Be Equal As Integers ${add_res.rc} 0
Output Should Contain ${add_res} ${res_name}
# Create a project linked to the resource
${proj_name}= Set Variable m6-setup-proj-${RUN_SUFFIX}
${create_proj}= Run CleverAgents Command project create --resource ${res_name} ${proj_name}
Should Be Equal As Integers ${create_proj.rc} 0
Output Should Contain ${create_proj} ${proj_name}
# Verify project appears in the list (use --format json to avoid Rich table truncation)
${list_proj}= Run CleverAgents Command project list --format json
Should Be Equal As Integers ${list_proj.rc} 0
Output Should Contain ${list_proj} ${proj_name}
[Documentation] Initialize, add a git resource, create a project, and verify.
[Teardown] Log Init And Project Setup test completed (cleanup via suite teardown)
# Create a temp git repo for the resource
${repo_path}= Create Temp Git Repo m6-setup-repo-${RUN_SUFFIX}
# Add a git-checkout resource with a run-unique name
${res_name}= Set Variable m6-setup-res-${RUN_SUFFIX}
${add_res}= Run CleverAgents Command resource add git-checkout ${res_name} --path ${repo_path}
Should Be Equal As Integers ${add_res.rc} 0
Output Should Contain ${add_res} ${res_name}
# Create a project linked to the resource
${proj_name}= Set Variable m6-setup-proj-${RUN_SUFFIX}
${create_proj}= Run CleverAgents Command project create --resource ${res_name} ${proj_name}
Should Be Equal As Integers ${create_proj.rc} 0
Output Should Contain ${create_proj} ${proj_name}
# Verify project appears in the list (use --format json to avoid Rich table truncation)
${list_proj}= Run CleverAgents Command project list --format json
Should Be Equal As Integers ${list_proj.rc} 0
Output Should Contain ${list_proj} ${proj_name}
M6 E2E Guard Enforcement Denylist Budget Limits
[Documentation] Register a custom profile with explicit guards (denylist, budget,
... tool-call limits) and verify the guard fields are stored correctly.
... Covers AC-4: "Test verifies guard enforcement (denylist, budget
... caps, tool call limits)".
[Teardown] Run Keyword And Ignore Error Run CleverAgents Command automation-profile remove ${guard_profile_name} --yes expected_rc=None
Set Test Variable ${guard_profile_name} ${EMPTY}
${guard_profile_name}= Set Variable e2e-guard-${RUN_SUFFIX}
Set Test Variable ${guard_profile_name}
# Build YAML config for a custom profile with all three guard categories
${yaml}= Catenate SEPARATOR=\n
... name: ${guard_profile_name}
... description: E2E test profile with denylist budget and tool-call limits
... schema_version: "1.0"
... decompose_task: 0.5
... create_tool: 0.5
... select_tool: 1.0
... safety:
... ${SPACE}${SPACE}require_sandbox: true
... ${SPACE}${SPACE}require_checkpoints: true
... ${SPACE}${SPACE}max_cost_per_plan: 50.0
... ${SPACE}${SPACE}max_retries_per_step: 3
... guards:
... ${SPACE}${SPACE}max_tool_calls_per_step: 10
... ${SPACE}${SPACE}max_total_cost: 25.0
... ${SPACE}${SPACE}tool_denylist:
... ${SPACE}${SPACE}${SPACE}${SPACE}- shell_exec
... ${SPACE}${SPACE}${SPACE}${SPACE}- rm_rf
... ${SPACE}${SPACE}${SPACE}${SPACE}- drop_database
... ${SPACE}${SPACE}require_approval_for_writes: true
... ${SPACE}${SPACE}require_approval_for_apply: true
${yaml_path}= Set Variable ${SUITE_HOME}${/}guard-profile.yaml
Create File ${yaml_path} ${yaml}
# Register the custom profile
${add_result}= Run CleverAgents Command automation-profile add --config ${yaml_path} --format json
Should Be Equal As Integers ${add_result.rc} 0
Output Should Contain ${add_result} ${guard_profile_name}
# Show the profile and verify guard fields
${show_result}= Run CleverAgents Command automation-profile show ${guard_profile_name} --format json
Should Be Equal As Integers ${show_result.rc} 0
# Verify denylist entries
Output Should Contain ${show_result} tool_denylist
Output Should Contain ${show_result} shell_exec
Output Should Contain ${show_result} rm_rf
Output Should Contain ${show_result} drop_database
# Verify budget cap
Output Should Contain ${show_result} max_total_cost
# Verify tool call limit
Output Should Contain ${show_result} max_tool_calls_per_step
# Verify approval requirements
Output Should Contain ${show_result} require_approval_for_writes
Output Should Contain ${show_result} require_approval_for_apply
[Documentation] Register a custom profile with explicit guards (denylist, budget,
... tool-call limits) and verify the guard fields are stored correctly.
... Covers AC-4: "Test verifies guard enforcement (denylist, budget
... caps, tool call limits)".
[Teardown] Run Keyword And Ignore Error Run CleverAgents Command automation-profile remove ${guard_profile_name} --yes expected_rc=None
Set Test Variable ${guard_profile_name} ${EMPTY}
${guard_profile_name}= Set Variable e2e-guard-${RUN_SUFFIX}
Set Test Variable ${guard_profile_name}
# Build YAML config for a custom profile with all three guard categories
${yaml}= Catenate SEPARATOR=\n
... name: ${guard_profile_name}
... description: E2E test profile with denylist budget and tool-call limits
... schema_version: "1.0"
... decompose_task: 0.5
... create_tool: 0.5
... select_tool: 1.0
... safety:
... ${SPACE}${SPACE}require_sandbox: true
... ${SPACE}${SPACE}require_checkpoints: true
... ${SPACE}${SPACE}max_cost_per_plan: 50.0
... ${SPACE}${SPACE}max_retries_per_step: 3
... guards:
... ${SPACE}${SPACE}max_tool_calls_per_step: 10
... ${SPACE}${SPACE}max_total_cost: 25.0
... ${SPACE}${SPACE}tool_denylist:
... ${SPACE}${SPACE}${SPACE}${SPACE}- shell_exec
... ${SPACE}${SPACE}${SPACE}${SPACE}- rm_rf
... ${SPACE}${SPACE}${SPACE}${SPACE}- drop_database
... ${SPACE}${SPACE}require_approval_for_writes: true
... ${SPACE}${SPACE}require_approval_for_apply: true
${yaml_path}= Set Variable ${SUITE_HOME}${/}guard-profile.yaml
Create File ${yaml_path} ${yaml}
# Register the custom profile
${add_result}= Run CleverAgents Command automation-profile add --config ${yaml_path} --format json
Should Be Equal As Integers ${add_result.rc} 0
Output Should Contain ${add_result} ${guard_profile_name}
# Show the profile and verify guard fields
${show_result}= Run CleverAgents Command automation-profile show ${guard_profile_name} --format json
Should Be Equal As Integers ${show_result.rc} 0
# Verify denylist entries
Output Should Contain ${show_result} tool_denylist
Output Should Contain ${show_result} shell_exec
Output Should Contain ${show_result} rm_rf
Output Should Contain ${show_result} drop_database
# Verify budget cap
Output Should Contain ${show_result} max_total_cost
# Verify tool call limit
Output Should Contain ${show_result} max_tool_calls_per_step
# Verify approval requirements
Output Should Contain ${show_result} require_approval_for_writes
Output Should Contain ${show_result} require_approval_for_apply
M6 E2E Plan Lifecycle Via CLI
[Documentation] Test A2A facade plan lifecycle: create plan via action, list, and status.
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
${proj_name}= Setup Plan Test Resources plan-lc
# Use an action to create a plan with automation profile
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile ci --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
Plan Lifecycle Assertions ${plan_use}
[Documentation] Test A2A facade plan lifecycle: create plan via action, list, and status.
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
${proj_name}= Setup Plan Test Resources plan-lc
# Use an action to create a plan with automation profile
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile ci --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
Plan Lifecycle Assertions ${plan_use}
M6 E2E Guard Enforcement Via Profile
[Documentation] Verify guard enforcement with a strict automation profile.
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
${proj_name}= Setup Plan Test Resources guard
# Attempt to use an action; with manual profile guards should constrain autonomy
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile manual --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
Guard Enforcement Assertions ${plan_use} expected_profile=manual
[Documentation] Verify guard enforcement with a strict automation profile.
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
${proj_name}= Setup Plan Test Resources guard
# Attempt to use an action; with manual profile guards should constrain autonomy
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile manual --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
Guard Enforcement Assertions ${plan_use} expected_profile=manual
M6 E2E Profile Precedence Plan Overrides Global
[Documentation] Verify plan-level automation profile overrides global config.
[Tags] tdd_issue tdd_issue_4196
... Sets global profile to "review", then creates a plan with
... explicit --automation-profile trusted. The plan output must
... show "trusted" (not "review"), proving plan > global precedence.
... Covers AC-5: "Test verifies automation profile resolution
... precedence (plan > action > global)".
... Note: action > global precedence requires the action's
... automation_profile to propagate to the Plan during plan-use,
... which is not yet wired in PlanLifecycleService.use_action;
... this test covers plan > global.
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
# Set global profile to "review"
${set_global}= Run CleverAgents Command config set core.automation-profile review --format json
Should Be Equal As Integers ${set_global.rc} 0
${get_global}= Run CleverAgents Command config get core.automation-profile --format json
${global_value}= Safe Parse Json Field ${get_global.stdout} value
Should Be Equal As Strings ${global_value} review
# Setup resources and project
${proj_name}= Setup Plan Test Resources prec
# Create plan with EXPLICIT "trusted" profile (should override "review" global)
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile trusted --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
# Parse automation_profile from plan output: must be "trusted", NOT "review"
${resolved_profile}= Safe Parse Json Field ${plan_use.stdout} automation_profile
IF '${resolved_profile}' != ''
Should Be Equal As Strings ${resolved_profile} trusted
Should Not Be Equal As Strings ${resolved_profile} review
Log Precedence confirmed: plan-level "trusted" overrides global "review"
ELSE
# Fallback: check combined output for "trusted" profile name
Output Should Contain ${plan_use} trusted
END
[Documentation] Verify plan-level automation profile overrides global config.
... Sets global profile to "review", then creates a plan with
... explicit --automation-profile trusted. The plan output must
... show "trusted" (not "review"), proving plan > global precedence.
... Covers AC-5: "Test verifies automation profile resolution
... precedence (plan > action > global)".
... Note: action > global precedence requires the action's
... automation_profile to propagate to the Plan during plan-use,
... which is not yet wired in PlanLifecycleService.use_action;
... this test covers plan > global.
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
# Set global profile to "review"
${set_global}= Run CleverAgents Command config set core.automation-profile review --format json
Should Be Equal As Integers ${set_global.rc} 0
${get_global}= Run CleverAgents Command config get core.automation-profile --format json
${global_value}= Safe Parse Json Field ${get_global.stdout} value
Should Be Equal As Strings ${global_value} review
# Setup resources and project
${proj_name}= Setup Plan Test Resources prec
# Create plan with EXPLICIT "trusted" profile (should override "review" global)
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile trusted --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
# Parse automation_profile from plan output: must be "trusted", NOT "review"
${resolved_profile}= Safe Parse Json Field ${plan_use.stdout} automation_profile
IF '${resolved_profile}' != ''
Should Be Equal As Strings ${resolved_profile} trusted
Should Not Be Equal As Strings ${resolved_profile} review
Log Precedence confirmed: plan-level "trusted" overrides global "review"
ELSE
# Fallback: check combined output for "trusted" profile name
Output Should Contain ${plan_use} trusted
END
M6 E2E Event Queue Via Plan Lifecycle Transitions
[Documentation] Exercise event queue publish/subscribe through plan lifecycle.
[Tags] tdd_issue tdd_issue_4189
... Each plan operation (create, execute) publishes domain events
... (PLAN_CREATED, PHASE_TRANSITION) consumed by event handlers that
... update plan state. Verifying state transitions via CLI proves
... the event bus delivered and processed the published events.
... Covers AC-3: "Test exercises event queue publish/subscribe via
... real CLI".
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
${proj_name}= Setup Plan Test Resources evq
# 1. Create plan — publishes PLAN_CREATED on the domain event bus
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile ci --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
# 2. Verify plan appears in list (proves creation event was processed)
Verify Plan In List ${plan_id}
# 3. Capture initial plan state via status
${status1}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${status1.rc} 0 msg=plan status failed (rc=${status1.rc}): ${status1.stderr}
${initial_phase}= Safe Parse Json Field ${status1.stdout} phase
${initial_state}= Safe Parse Json Field ${status1.stdout} processing_state
Log Initial phase=${initial_phase} processing_state=${initial_state}
# 4. Execute plan — triggers PHASE_TRANSITION and execution events
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
# P0-3: Hard assertions on SUCCESS path; LLM execution may legitimately fail
# (e.g. strategize phase not yet complete — no background worker in E2E).
IF ${execute.rc} == 0
Output Should Contain ${execute} ${plan_id}
# 5. Re-check status — should reflect post-execute state
${status2}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${status2.rc} 0 msg=plan status failed after execute (rc=${status2.rc}): ${status2.stderr}
${post_phase}= Safe Parse Json Field ${status2.stdout} phase
${post_state}= Safe Parse Json Field ${status2.stdout} processing_state
Log Post-execute phase=${post_phase} processing_state=${post_state}
# P0-3: Hard assertion — at least one state field must be populated,
# proving the event bus delivered and processed lifecycle events.
${state_populated}= Evaluate '${post_phase}' != '' or '${post_state}' != ''
Should Be True ${state_populated}
... Event queue should process lifecycle events — expected non-empty phase or processing_state after execute
ELSE
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
END
# 6. Final: plan should still appear in list
Verify Plan In List ${plan_id}
[Documentation] Exercise event queue publish/subscribe through plan lifecycle.
... Each plan operation (create, execute) publishes domain events
... (PLAN_CREATED, PHASE_TRANSITION) consumed by event handlers that
... update plan state. Verifying state transitions via CLI proves
... the event bus delivered and processed the published events.
... Covers AC-3: "Test exercises event queue publish/subscribe via
... real CLI".
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
${proj_name}= Setup Plan Test Resources evq
# 1. Create plan — publishes PLAN_CREATED on the domain event bus
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile ci --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
# 2. Verify plan appears in list (proves creation event was processed)
Verify Plan In List ${plan_id}
# 3. Capture initial plan state via status
${status1}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${status1.rc} 0
... plan status failed (rc=${status1.rc}): ${status1.stderr}
${initial_phase}= Safe Parse Json Field ${status1.stdout} phase
${initial_state}= Safe Parse Json Field ${status1.stdout} processing_state
Log Initial phase=${initial_phase} processing_state=${initial_state}
# 4. Execute plan — triggers PHASE_TRANSITION and execution events
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
# P0-3: Hard assertions on SUCCESS path; LLM execution may legitimately fail
# (e.g. strategize phase not yet complete — no background worker in E2E).
IF ${execute.rc} == 0
Output Should Contain ${execute} ${plan_id}
# 5. Re-check status — should reflect post-execute state
${status2}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${status2.rc} 0
... plan status failed after execute (rc=${status2.rc}): ${status2.stderr}
${post_phase}= Safe Parse Json Field ${status2.stdout} phase
${post_state}= Safe Parse Json Field ${status2.stdout} processing_state
Log Post-execute phase=${post_phase} processing_state=${post_state}
# P0-3: Hard assertion — at least one state field must be populated,
# proving the event bus delivered and processed lifecycle events.
${state_populated}= Evaluate '${post_phase}' != '' or '${post_state}' != ''
Should Be True ${state_populated}
... Event queue should process lifecycle events — expected non-empty phase or processing_state after execute
ELSE
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
END
# 6. Final: plan should still appear in list
Verify Plan In List ${plan_id}
M6 E2E Hierarchical Decomposition Via Plan Tree
[Documentation] Verify hierarchical plan decomposition via the plan tree command.
[Tags] tdd_issue tdd_issue_4189
... After plan creation and execution, ``plan tree`` should return
... a decision tree with at least root-level decisions. If the LLM
... decomposes the task into subplans the tree will show nested
... children with hierarchy indicators.
... Covers AC-6: "Test exercises a full autonomy acceptance flow
... with hierarchical decomposition".
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
${proj_name}= Setup Plan Test Resources tree
# Create plan with full-auto profile (enables install_dependency for decomposition)
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile full-auto --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
# Execute plan to populate the decision tree with strategy/execution decisions
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
IF ${execute.rc} != 0
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
END
# View decision tree — must contain at least root-level decisions
${tree}= Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=120s
# P0-4: Hard assertion — tree command must succeed.
Should Be Equal As Integers ${tree.rc} 0 msg=plan tree failed (rc=${tree.rc}): ${tree.stderr}
Should Not Be Empty ${tree.stdout} Plan tree output should not be empty
# P0-4: Hard assertion — at least one decision node must exist after execution.
${decision_count}= Evaluate $tree.stdout.count('"decision_id"')
Log Decision tree contains ${decision_count} decision node(s)
Should Be True ${decision_count} >= 1
... Plan tree should contain at least one decision node after execution (found ${decision_count})
# Check for hierarchical children (decomposition infrastructure indicator)
${has_children_key}= Evaluate '"children"' in $tree.stdout
IF ${has_children_key}
Log Tree structure has children field (decomposition infrastructure functional)
END
[Documentation] Verify hierarchical plan decomposition via the plan tree command.
... After plan creation and execution, ``plan tree`` should return
... a decision tree with at least root-level decisions. If the LLM
... decomposes the task into subplans the tree will show nested
... children with hierarchy indicators.
... Covers AC-6: "Test exercises a full autonomy acceptance flow
... with hierarchical decomposition".
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
${proj_name}= Setup Plan Test Resources tree
# Create plan with full-auto profile (enables install_dependency for decomposition)
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile full-auto --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
# Execute plan to populate the decision tree with strategy/execution decisions
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
IF ${execute.rc} != 0
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
END
# View decision tree — must contain at least root-level decisions
${tree}= Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=120s
# P0-4: Hard assertion — tree command must succeed.
Should Be Equal As Integers ${tree.rc} 0
... plan tree failed (rc=${tree.rc}): ${tree.stderr}
Should Not Be Empty ${tree.stdout} Plan tree output should not be empty
# P0-4: Hard assertion — at least one decision node must exist after execution.
${decision_count}= Evaluate $tree.stdout.count('"decision_id"')
Log Decision tree contains ${decision_count} decision node(s)
Should Be True ${decision_count} >= 1
... Plan tree should contain at least one decision node after execution (found ${decision_count})
# Check for hierarchical children (decomposition infrastructure indicator)
${has_children_key}= Evaluate '"children"' in $tree.stdout
IF ${has_children_key}
Log Tree structure has children field (decomposition infrastructure functional)
END
M6 E2E Full Autonomy Acceptance Flow
[Documentation] End-to-end: init, resource, project, action, plan use, execute, status, apply.
[Tags] tdd_issue tdd_issue_4189
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
${proj_name}= Setup Plan Test Resources flow
# Use an action to create a plan
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile full-auto --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0 msg=plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
# Parse plan_id from JSON output
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
# Check plan status
${status}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${status.rc} 0 msg=plan status failed (rc=${status.rc}): ${status.stderr}
Output Should Contain ${status} ${plan_id}
# Execute
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
IF ${execute.rc} == 0
Output Should Contain ${execute} ${plan_id}
# Verify execute output contains phase-transition indicators
${exec_phase}= Safe Parse Json Field ${execute.stdout} phase
Log Execute phase: ${exec_phase}
# Apply
Full Flow Apply Step ${plan_id}
ELSE
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
END
# Final verification: list should show the plan
Verify Plan In List ${plan_id}
[Documentation] End-to-end: init, resource, project, action, plan use, execute, status, apply.
[Teardown] Run CleverAgents Command config set core.automation-profile manual expected_rc=None
Skip If No LLM Keys
${proj_name}= Setup Plan Test Resources flow
# Use an action to create a plan
${plan_use}= Run CleverAgents Command plan use local/code-review ${proj_name} --automation-profile full-auto --format json expected_rc=None timeout=180s
# P0-2: Fail (not Skip) when API keys are present but plan use fails.
Should Be Equal As Integers ${plan_use.rc} 0
... plan use failed (rc=${plan_use.rc}): ${plan_use.stderr}
# Parse plan_id from JSON output
${plan_id}= Safe Parse Json Field ${plan_use.stdout} plan_id
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
# Check plan status
${status}= Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${status.rc} 0
... plan status failed (rc=${status.rc}): ${status.stderr}
Output Should Contain ${status} ${plan_id}
# Execute
${execute}= Run CleverAgents Command plan execute ${plan_id} --format json expected_rc=None timeout=180s
IF ${execute.rc} == 0
Output Should Contain ${execute} ${plan_id}
# Verify execute output contains phase-transition indicators
${exec_phase}= Safe Parse Json Field ${execute.stdout} phase
Log Execute phase: ${exec_phase}
# Apply
Full Flow Apply Step ${plan_id}
ELSE
Fail plan execute failed (rc=${execute.rc}) stdout=${execute.stdout} stderr=${execute.stderr}
END
# Final verification: list should show the plan
Verify Plan In List ${plan_id}
+215 -212
View File
@@ -1,238 +1,241 @@
*** Settings ***
Documentation TDD Issue #1028 — ACMS indexing pipeline not wired into CLI.
Documentation TDD Issue #1028 — ACMS indexing pipeline not wired into CLI.
...
... Behavioral E2E tests proving that ``ContextTierService`` starts
... empty on every CLI invocation because the ACMS indexing pipeline
... is not wired into the CLI entry points. The ``project context
... simulate`` and ``project context inspect`` commands operate on
... zero data even when run against a project directory containing
... files.
... Behavioral E2E tests proving that ``ContextTierService`` starts
... empty on every CLI invocation because the ACMS indexing pipeline
... is not wired into the CLI entry points. The ``project context
... simulate`` and ``project context inspect`` commands operate on
... zero data even when run against a project directory containing
... files.
...
... See CONTRIBUTING.md > Bug Fix Workflow for the full TDD
... issue-capture lifecycle.
Resource common_e2e.resource
Library OperatingSystem
Library String
Library Collections
Library Process
Suite Setup ACMS Behavioral Suite Setup
... See CONTRIBUTING.md > Bug Fix Workflow for the full TDD
... issue-capture lifecycle.
Resource common_e2e.resource
Library OperatingSystem
Library String
Library Collections
Library Process
Suite Setup ACMS Behavioral Suite Setup
Suite Teardown ACMS Behavioral Suite Teardown
*** Variables ***
${PROJECT_SIMULATE} local/tdd-1028-simulate
${PROJECT_INSPECT} local/tdd-1028-inspect
${PROJECT_BUDGET} local/tdd-1028-budget
${PROJECT_SCALE} local/tdd-1028-scale
${PROJECT_INSPECT} local/tdd-1028-inspect
${PROJECT_BUDGET} local/tdd-1028-budget
${PROJECT_SCALE} local/tdd-1028-scale
*** Keywords ***
ACMS Behavioral Suite Setup
[Documentation] Create an isolated workspace with ``agents init``, build a
... synthetic codebase, initialise a git repo, register it as a
... resource, and prepare for behavioral ACMS tests.
E2E Suite Setup
# Create workspace directory inside the suite home
${ws}= Set Variable ${SUITE_HOME}${/}workspace
Create Directory ${ws}
Set Suite Variable ${WS} ${ws}
# Initialize the CleverAgents workspace
${result}= Run CLI init m5-tdd-workspace
Log Init stdout: ${result.stdout} level=DEBUG
Log Init stderr: ${result.stderr} level=DEBUG
Should Be Equal As Integers ${result.rc} 0 msg=Workspace init failed (rc=${result.rc}). Check DEBUG-level log entries above.
# Create synthetic source files for context testing
Create Synthetic Codebase ${ws} TDD 1028 test
# Initialize a git repository — check every return code
${git_init}= Run Process git init cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_init.rc} 0 msg=git init failed (rc=${git_init.rc}). Check DEBUG logs above.
${git_cfg_name}= Run Process git config user.name E2E Test cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_cfg_name.rc} 0 msg=git config user.name failed (rc=${git_cfg_name.rc}). Check DEBUG logs above.
${git_cfg_email}= Run Process git config user.email e2e@test.local cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_cfg_email.rc} 0 msg=git config user.email failed (rc=${git_cfg_email.rc}). Check DEBUG logs above.
${git_add}= Run Process git add . cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_add.rc} 0 msg=git add failed (rc=${git_add.rc}). Check DEBUG logs above.
${git_commit}= Run Process git commit -m Initial commit cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_commit.rc} 0 msg=git commit failed (rc=${git_commit.rc}). Check DEBUG logs above.
# Detect the default branch created by git init
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${branch_result.rc} 0 msg=git rev-parse failed (rc=${branch_result.rc}). Check DEBUG logs above.
${branch}= Strip String ${branch_result.stdout}
Set Suite Variable ${WS_BRANCH} ${branch}
# Register the workspace as a git-checkout resource
${res_name}= Set Variable local/tdd-1028-ws-resource
Set Suite Variable ${WS_RESOURCE} ${res_name}
${r_add}= Run CLI resource add git-checkout ${res_name} --path ${ws} --branch ${branch}
Should Be Equal As Integers ${r_add.rc} 0 msg=resource add failed (rc=${r_add.rc}). Check DEBUG logs above.
Set Suite Variable ${SUITE_SETUP_COMPLETE} ${TRUE}
[Documentation] Create an isolated workspace with ``agents init``, build a
... synthetic codebase, initialise a git repo, register it as a
... resource, and prepare for behavioral ACMS tests.
E2E Suite Setup
# Create workspace directory inside the suite home
${ws}= Set Variable ${SUITE_HOME}${/}workspace
Create Directory ${ws}
Set Suite Variable ${WS} ${ws}
# Initialize the CleverAgents workspace
${result}= Run CLI init m5-tdd-workspace
Log Init stdout: ${result.stdout} level=DEBUG
Log Init stderr: ${result.stderr} level=DEBUG
Should Be Equal As Integers ${result.rc} 0
... Workspace init failed (rc=${result.rc}). Check DEBUG-level log entries above.
# Create synthetic source files for context testing
Create Synthetic Codebase ${ws} TDD 1028 test
# Initialize a git repository — check every return code
${git_init}= Run Process git init cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_init.rc} 0 msg=git init failed (rc=${git_init.rc}). Check DEBUG logs above.
${git_cfg_name}= Run Process git config user.name E2E Test cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_cfg_name.rc} 0 msg=git config user.name failed (rc=${git_cfg_name.rc}). Check DEBUG logs above.
${git_cfg_email}= Run Process git config user.email e2e@test.local cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_cfg_email.rc} 0 msg=git config user.email failed (rc=${git_cfg_email.rc}). Check DEBUG logs above.
${git_add}= Run Process git add . cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_add.rc} 0 msg=git add failed (rc=${git_add.rc}). Check DEBUG logs above.
${git_commit}= Run Process git commit -m Initial commit cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_commit.rc} 0 msg=git commit failed (rc=${git_commit.rc}). Check DEBUG logs above.
# Detect the default branch created by git init
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${ws} timeout=60s on_timeout=kill
Should Be Equal As Integers ${branch_result.rc} 0 msg=git rev-parse failed (rc=${branch_result.rc}). Check DEBUG logs above.
${branch}= Strip String ${branch_result.stdout}
Set Suite Variable ${WS_BRANCH} ${branch}
# Register the workspace as a git-checkout resource
${res_name}= Set Variable local/tdd-1028-ws-resource
Set Suite Variable ${WS_RESOURCE} ${res_name}
${r_add}= Run CLI resource add git-checkout ${res_name} --path ${ws} --branch ${branch}
Should Be Equal As Integers ${r_add.rc} 0 msg=resource add failed (rc=${r_add.rc}). Check DEBUG logs above.
Set Suite Variable ${SUITE_SETUP_COMPLETE} ${TRUE}
ACMS Behavioral Suite Teardown
[Documentation] Delegate to the common E2E teardown.
E2E Suite Teardown
[Documentation] Delegate to the common E2E teardown.
E2E Suite Teardown
*** Test Cases ***
# ----------------------------------------------------------------------- # TDD Issue #1028 — Behavioral ACMS validation
# # NOTE: Each test guards against incomplete suite setup with
# [Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
# -----------------------------------------------------------------------
# TDD Issue #1028 — Behavioral ACMS validation
#
# NOTE: Each test guards against incomplete suite setup with
# [Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
# If suite setup fails, this guard raises an error and the test
# will fail without executing any assertions. The
# will fail without executing any assertions. The
# m5_acceptance.robot tests cover the same CLI plumbing and will
# surface setup failures independently.
# -----------------------------------------------------------------------
Context Simulate Returns Non-Empty Tier Data
[Documentation] Run ``project context simulate`` against a project with
... files and assert that the output contains actual indexed
... fragments (not empty tiers).
...
... **Expected bug behavior:** ``total_tokens`` is 0 and
... ``fragment_count`` is 0 because the ACMS indexing pipeline
... is not wired into the CLI — ``ContextTierService`` starts
... empty on every invocation.
...
[Tags] tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E tdd_issue tdd_issue_4306 tdd_expected_fail
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
# Create project and configure context policy
Run CLI project create ${PROJECT_SIMULATE}
Link Resource To Project ${PROJECT_SIMULATE}
Run CLI ... project context set ${PROJECT_SIMULATE}
... --view default
... --include-path **/*.py
# Run simulate and check for actual indexed data
${result}= Run CLI
... project context simulate ${PROJECT_SIMULATE}
... --format json
${sim_json}= Extract JSON From Stdout ${result.stdout}
# Behavioral assertion: the project has Python files, so simulate
# must produce non-zero fragment data after indexing.
${fragment_count}= Evaluate int($sim_json.get('fragment_count', 0))
Should Be True ${fragment_count} > 0
... msg=Bug #1028: fragment_count is ${fragment_count} (expected > 0). ACMS indexing pipeline is not wired into CLI — ContextTierService starts empty.
${total_tokens}= Evaluate int($sim_json.get('total_tokens', 0))
Should Be True ${total_tokens} > 0
... msg=Bug #1028: total_tokens is ${total_tokens} (expected > 0). No fragments were indexed from project files.
[Documentation] Run ``project context simulate`` against a project with
... files and assert that the output contains actual indexed
... fragments (not empty tiers).
...
... **Expected bug behavior:** ``total_tokens`` is 0 and
... ``fragment_count`` is 0 because the ACMS indexing pipeline
... is not wired into the CLI — ``ContextTierService`` starts
... empty on every invocation.
...
[Tags] tdd_expected_fail tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
# Create project and configure context policy
Run CLI project create ${PROJECT_SIMULATE}
Link Resource To Project ${PROJECT_SIMULATE}
Run CLI
... project context set ${PROJECT_SIMULATE}
... --view default
... --include-path **/*.py
# Run simulate and check for actual indexed data
${result}= Run CLI
... project context simulate ${PROJECT_SIMULATE}
... --format json
${sim_json}= Extract JSON From Stdout ${result.stdout}
# Behavioral assertion: the project has Python files, so simulate
# must produce non-zero fragment data after indexing.
${fragment_count}= Evaluate int($sim_json.get('fragment_count', 0))
Should Be True ${fragment_count} > 0
... msg=Bug #1028: fragment_count is ${fragment_count} (expected > 0). ACMS indexing pipeline is not wired into CLI — ContextTierService starts empty.
${total_tokens}= Evaluate int($sim_json.get('total_tokens', 0))
Should Be True ${total_tokens} > 0
... msg=Bug #1028: total_tokens is ${total_tokens} (expected > 0). No fragments were indexed from project files.
Context Inspect Shows Indexed Resources
[Documentation] Run ``project context inspect`` against a project with
... files and assert that the indexed resource count is > 0.
...
... **Expected bug behavior:** ``tier_metrics`` counters are
... all zero because no indexing occurs — the ACMS pipeline
... is disconnected from the CLI.
...
[Tags] tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E tdd_issue tdd_issue_4306 tdd_expected_fail
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
# Create project and configure context policy
Run CLI project create ${PROJECT_INSPECT}
Link Resource To Project ${PROJECT_INSPECT}
Run CLI ... project context set ${PROJECT_INSPECT}
... --view default
... --include-path **/*.py
# Run inspect and check for indexed resources
${result}= Run CLI
... project context inspect ${PROJECT_INSPECT}
... --format json
${inspect_json}= Extract JSON From Stdout ${result.stdout}
# Behavioral assertion: at least one tier should have fragments
${metrics}= Evaluate $inspect_json.get('tier_metrics', {})
${hot_count}= Evaluate int($metrics.get('hot_count', 0))
${warm_count}= Evaluate int($metrics.get('warm_count', 0))
${cold_count}= Evaluate int($metrics.get('cold_count', 0))
${total_indexed}= Evaluate ${hot_count} + ${warm_count} + ${cold_count}
Should Be True ${total_indexed} > 0
... msg=Bug #1028: total indexed fragments is ${total_indexed} (expected > 0). tier_metrics: hot=${hot_count}, warm=${warm_count}, cold=${cold_count}. ACMS indexing pipeline is not wired into CLI.
[Documentation] Run ``project context inspect`` against a project with
... files and assert that the indexed resource count is > 0.
...
... **Expected bug behavior:** ``tier_metrics`` counters are
... all zero because no indexing occurs — the ACMS pipeline
... is disconnected from the CLI.
...
[Tags] tdd_expected_fail tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
# Create project and configure context policy
Run CLI project create ${PROJECT_INSPECT}
Link Resource To Project ${PROJECT_INSPECT}
Run CLI
... project context set ${PROJECT_INSPECT}
... --view default
... --include-path **/*.py
# Run inspect and check for indexed resources
${result}= Run CLI
... project context inspect ${PROJECT_INSPECT}
... --format json
${inspect_json}= Extract JSON From Stdout ${result.stdout}
# Behavioral assertion: at least one tier should have fragments
${metrics}= Evaluate $inspect_json.get('tier_metrics', {})
${hot_count}= Evaluate int($metrics.get('hot_count', 0))
${warm_count}= Evaluate int($metrics.get('warm_count', 0))
${cold_count}= Evaluate int($metrics.get('cold_count', 0))
${total_indexed}= Evaluate ${hot_count} + ${warm_count} + ${cold_count}
Should Be True ${total_indexed} > 0
... msg=Bug #1028: total indexed fragments is ${total_indexed} (expected > 0). tier_metrics: hot=${hot_count}, warm=${warm_count}, cold=${cold_count}. ACMS indexing pipeline is not wired into CLI.
Budget Enforcement Excludes Oversized Files
[Documentation] Configure ``max_file_size`` policy, add a file exceeding
... that limit, run simulate, and assert that the oversized
... file is excluded while smaller files are indexed.
...
... **Expected bug behavior:** ``fragment_count`` is 0 because
... the indexing pipeline does not run at all — regardless of
... ``max_file_size`` configuration, no files are scanned.
...
[Tags] tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E tdd_issue tdd_issue_4306 tdd_expected_fail
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
# Create project with tight max_file_size (1024 bytes)
Run CLI project create ${PROJECT_BUDGET}
Link Resource To Project ${PROJECT_BUDGET}
Run CLI ... project context set ${PROJECT_BUDGET}
... --view default
... --include-path **/*.py
... --max-file-size 1024
... --max-total-size 8192
# Simulate should index small files but exclude large_file.py (>1KiB)
${result}= Run CLI
... project context simulate ${PROJECT_BUDGET}
... --format json
${sim_json}= Extract JSON From Stdout ${result.stdout}
# Behavioral assertion: at least the small files (main.py, utils.py,
# config.py) should be indexed — large_file.py should be excluded by
# max_file_size policy. If fragment_count > 0, some files were indexed
# (and budget enforcement partially works).
# TODO(bugfix/m5-acms-cli-indexing-pipeline-wiring): After the indexing
# pipeline is wired, add a second assertion verifying that large_file.py
# is absent from the fragment list (i.e. budget enforcement actually
# excludes oversized files, not just that *some* files are indexed).
${fragment_count}= Evaluate int($sim_json.get('fragment_count', 0))
Should Be True ${fragment_count} > 0
... msg=Bug #1028: fragment_count is ${fragment_count} (expected > 0). Budget enforcement cannot exclude oversized files because the indexing pipeline does not run at all.
[Documentation] Configure ``max_file_size`` policy, add a file exceeding
... that limit, run simulate, and assert that the oversized
... file is excluded while smaller files are indexed.
...
... **Expected bug behavior:** ``fragment_count`` is 0 because
... the indexing pipeline does not run at all — regardless of
... ``max_file_size`` configuration, no files are scanned.
...
[Tags] tdd_expected_fail tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
# Create project with tight max_file_size (1024 bytes)
Run CLI project create ${PROJECT_BUDGET}
Link Resource To Project ${PROJECT_BUDGET}
Run CLI
... project context set ${PROJECT_BUDGET}
... --view default
... --include-path **/*.py
... --max-file-size 1024
... --max-total-size 8192
# Simulate should index small files but exclude large_file.py (>1KiB)
${result}= Run CLI
... project context simulate ${PROJECT_BUDGET}
... --format json
${sim_json}= Extract JSON From Stdout ${result.stdout}
# Behavioral assertion: at least the small files (main.py, utils.py,
# config.py) should be indexed — large_file.py should be excluded by
# max_file_size policy. If fragment_count > 0, some files were indexed
# (and budget enforcement partially works).
# TODO(bugfix/m5-acms-cli-indexing-pipeline-wiring): After the indexing
# pipeline is wired, add a second assertion verifying that large_file.py
# is absent from the fragment list (i.e. budget enforcement actually
# excludes oversized files, not just that *some* files are indexed).
${fragment_count}= Evaluate int($sim_json.get('fragment_count', 0))
Should Be True ${fragment_count} > 0
... msg=Bug #1028: fragment_count is ${fragment_count} (expected > 0). Budget enforcement cannot exclude oversized files because the indexing pipeline does not run at all.
Large Project Indexes Without Timeout
[Documentation] Create a synthetic 10,000+ file project, run
... ``project context simulate``, and assert completion
... within a reasonable timeout with non-empty results.
...
... **Expected bug behavior:** The simulate command completes
... but returns zero fragments because the indexing pipeline
... is not wired — the 10K files are never scanned.
...
[Tags] tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E tdd_issue tdd_issue_4306 tdd_expected_fail
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
# Generate 10,000 tiny .py files in a subdirectory
${scale_dir}= Set Variable ${WS}${/}scale_src
Create Directory ${scale_dir}
${script}= Catenate SEPARATOR=\n
... import os, sys
... d = sys.argv[1]
... for i in range(10000):
... ${SPACE}${SPACE}${SPACE}${SPACE}with open(os.path.join(d, f"mod_{i:05d}.py"), "w") as f:
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}f.write(f"# module {i}\\ndef fn_{i}(): return {i}\\n")
${gen_script}= Set Variable ${WS}${/}_gen_10k.py
Create File ${gen_script} ${script}
${gen}= Run Process ${PYTHON} ${gen_script} ${scale_dir}
... timeout=120s on_timeout=kill
Should Be Equal As Integers ${gen.rc} 0 msg=10K file generation failed
# NOTE: The 10K generated files are written to the filesystem but NOT
# committed to the workspace git repo. The resource is registered as
# git-checkout type. The bug-fix developer MUST evaluate whether the
# fix routes indexing through the git sandbox (which only exposes
# git-tracked content) or the filesystem. If the fix uses git-tracked
# content, uncomment the following lines to commit the generated files:
# Run Process git add scale_src/ cwd=${WS}
# Run Process git commit -m Add 10K test files cwd=${WS}
# The m5_acceptance.robot structural test has the same pattern —
# neither suite commits generated files.
Remove File ${gen_script}
# Create project with include path covering the 10K files
Run CLI project create ${PROJECT_SCALE}
Link Resource To Project ${PROJECT_SCALE}
Run CLI ... project context set ${PROJECT_SCALE}
... --view default
... --include-path scale_src/**/*.py
# Simulate must complete within 600s (10 minutes) and return data
${result}= Run CLI
... project context simulate ${PROJECT_SCALE}
... --format json timeout=600s
${sim_json}= Extract JSON From Stdout ${result.stdout}
# Behavioral assertion: with 10,000+ files, fragment_count must be
# non-zero if the indexing pipeline is operational.
${fragment_count}= Evaluate int($sim_json.get('fragment_count', 0))
Should Be True ${fragment_count} > 0
... msg=Bug #1028: fragment_count is ${fragment_count} (expected > 0). 10,000+ files were generated but none were indexed. ACMS indexing pipeline is not wired into CLI.
[Documentation] Create a synthetic 10,000+ file project, run
... ``project context simulate``, and assert completion
... within a reasonable timeout with non-empty results.
...
... **Expected bug behavior:** The simulate command completes
... but returns zero fragments because the indexing pipeline
... is not wired — the 10K files are never scanned.
...
[Tags] tdd_expected_fail tdd_bug tdd_bug_1028 tdd_issue tdd_issue_1028 E2E
[Setup] Variable Should Exist ${SUITE_SETUP_COMPLETE}
... msg=Prerequisite not met: suite setup did not complete
# Generate 10,000 tiny .py files in a subdirectory
${scale_dir}= Set Variable ${WS}${/}scale_src
Create Directory ${scale_dir}
${script}= Catenate SEPARATOR=\n
... import os, sys
... d = sys.argv[1]
... for i in range(10000):
... ${SPACE}${SPACE}${SPACE}${SPACE}with open(os.path.join(d, f"mod_{i:05d}.py"), "w") as f:
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}f.write(f"# module {i}\\ndef fn_{i}(): return {i}\\n")
${gen_script}= Set Variable ${WS}${/}_gen_10k.py
Create File ${gen_script} ${script}
${gen}= Run Process ${PYTHON} ${gen_script} ${scale_dir}
... timeout=120s on_timeout=kill
Should Be Equal As Integers ${gen.rc} 0 msg=10K file generation failed
# NOTE: The 10K generated files are written to the filesystem but NOT
# committed to the workspace git repo. The resource is registered as
# git-checkout type. The bug-fix developer MUST evaluate whether the
# fix routes indexing through the git sandbox (which only exposes
# git-tracked content) or the filesystem. If the fix uses git-tracked
# content, uncomment the following lines to commit the generated files:
# Run Process git add scale_src/ cwd=${WS}
# Run Process git commit -m Add 10K test files cwd=${WS}
# The m5_acceptance.robot structural test has the same pattern —
# neither suite commits generated files.
Remove File ${gen_script}
# Create project with include path covering the 10K files
Run CLI project create ${PROJECT_SCALE}
Link Resource To Project ${PROJECT_SCALE}
Run CLI
... project context set ${PROJECT_SCALE}
... --view default
... --include-path scale_src/**/*.py
# Simulate must complete within 600s (10 minutes) and return data
${result}= Run CLI
... project context simulate ${PROJECT_SCALE}
... --format json timeout=600s
${sim_json}= Extract JSON From Stdout ${result.stdout}
# Behavioral assertion: with 10,000+ files, fragment_count must be
# non-zero if the indexing pipeline is operational.
${fragment_count}= Evaluate int($sim_json.get('fragment_count', 0))
Should Be True ${fragment_count} > 0
... msg=Bug #1028: fragment_count is ${fragment_count} (expected > 0). 10,000+ files were generated but none were indexed. ACMS indexing pipeline is not wired into CLI.
+517 -279
View File
@@ -1,339 +1,577 @@
*** Settings ***
Documentation E2E test for Workflow Example 4: Multi-Project Dependency Update.
Documentation E2E test for Workflow Example 4: Multi-Project Dependency Update.
...
... Advanced scenario using the supervised automation profile.
... Three microservices share a common library with a breaking
... change (v1 to v2). CleverAgents creates a parent plan
... targeting all 4 projects, spawns child subplans per project,
... executes in dependency order (common-lib first, then services),
... validates per project, and applies in dependency order.
... Advanced scenario using the supervised automation profile.
... Three microservices share a common library with a breaking
... change (v1 to v2). CleverAgents creates a parent plan
... targeting all 4 projects, spawns child subplans per project,
... executes in dependency order (common-lib first, then services),
... validates per project, and applies in dependency order.
...
... Zero mocking — real CLI, real LLM API keys.
Resource common_e2e.resource
Suite Setup WF04 Suite Setup
... Zero mocking — real CLI, real LLM API keys.
Resource common_e2e.resource
Suite Setup WF04 Suite Setup
Suite Teardown E2E Suite Teardown
Force Tags E2E
Force Tags E2E
*** Variables ***
${ACTION_BASE} local/wf04-dep-update
${LIB_BASE} wf04-common-lib
${SVC1_BASE} wf04-svc-auth
${SVC2_BASE} wf04-svc-billing
${SVC3_BASE} wf04-svc-gateway
${ACTION_BASE} local/wf04-dep-update
${LIB_BASE} wf04-common-lib
${SVC1_BASE} wf04-svc-auth
${SVC2_BASE} wf04-svc-billing
${SVC3_BASE} wf04-svc-gateway
${WF04_SNAPSHOT_HELPER} ${CURDIR}${/}wf04_snapshot_helper.py
*** Keywords ***
WF04 Suite Setup
[Documentation] E2E Suite Setup plus unique suffix generation and actor selection.
E2E Suite Setup
# Initialise the database so commands work in all tests.
${init}= Run CleverAgents Command init --force --yes
Should Be Equal As Integers ${init.rc} 0
Should Not Contain ${init.stdout}${init.stderr} Traceback
Should Not Contain ${init.stdout}${init.stderr} INTERNAL
# Generate a unique suffix for resource/project names to avoid
# UNIQUE constraint collisions on repeated or parallel CI runs.
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
# Derive unique names from base + suffix
Set Suite Variable ${ACTION_NAME} ${ACTION_BASE}-${suffix}
Set Suite Variable ${LIB_RESOURCE} ${LIB_BASE}-res-${suffix}
Set Suite Variable ${SVC1_RESOURCE} ${SVC1_BASE}-res-${suffix}
Set Suite Variable ${SVC2_RESOURCE} ${SVC2_BASE}-res-${suffix}
Set Suite Variable ${SVC3_RESOURCE} ${SVC3_BASE}-res-${suffix}
Set Suite Variable ${LIB_PROJECT} ${LIB_BASE}-proj-${suffix}
Set Suite Variable ${SVC1_PROJECT} ${SVC1_BASE}-proj-${suffix}
Set Suite Variable ${SVC2_PROJECT} ${SVC2_BASE}-proj-${suffix}
Set Suite Variable ${SVC3_PROJECT} ${SVC3_BASE}-proj-${suffix}
# Probe the OpenAI API to confirm the key is usable before selecting actor.
# Falls back to Anthropic when the key is absent or quota-exhausted.
${actor}= Resolve LLM Actor
Set Suite Variable ${LLM_ACTOR} ${actor}
[Documentation] E2E Suite Setup plus unique suffix generation and actor selection.
E2E Suite Setup
# Initialise the database so commands work in all tests.
${init}= Run CleverAgents Command init --force --yes
Should Be Equal As Integers ${init.rc} 0
Should Not Contain ${init.stdout}${init.stderr} Traceback
Should Not Contain ${init.stdout}${init.stderr} INTERNAL
# Generate a unique suffix for resource/project names to avoid
# UNIQUE constraint collisions on repeated or parallel CI runs.
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
# Derive unique names from base + suffix
Set Suite Variable ${ACTION_NAME} ${ACTION_BASE}-${suffix}
Set Suite Variable ${LIB_RESOURCE} ${LIB_BASE}-res-${suffix}
Set Suite Variable ${SVC1_RESOURCE} ${SVC1_BASE}-res-${suffix}
Set Suite Variable ${SVC2_RESOURCE} ${SVC2_BASE}-res-${suffix}
Set Suite Variable ${SVC3_RESOURCE} ${SVC3_BASE}-res-${suffix}
Set Suite Variable ${LIB_PROJECT} ${LIB_BASE}-proj-${suffix}
Set Suite Variable ${SVC1_PROJECT} ${SVC1_BASE}-proj-${suffix}
Set Suite Variable ${SVC2_PROJECT} ${SVC2_BASE}-proj-${suffix}
Set Suite Variable ${SVC3_PROJECT} ${SVC3_BASE}-proj-${suffix}
# Pick an actor that matches the available API key.
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
IF ${has_openai}
${actor}= Set Variable openai/gpt-4o
ELSE IF ${has_anthropic}
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
ELSE
${actor}= Set Variable openai/gpt-4o
END
Set Suite Variable ${LLM_ACTOR} ${actor}
Create Library Repo
[Documentation] Create temp git repo for the common library.
${repo}= Create Temp Git Repo ${LIB_BASE}-${RUN_SUFFIX}
Create Directory ${repo}${/}src
${lib_content}= Catenate SEPARATOR=\n
... """Common library v1 — shared utilities."""
... ${EMPTY}
... ${EMPTY}
... __version__ = "1.0.0"
... ${EMPTY}
... ${EMPTY}
... def connect(host, port):
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Connect to a service (v1 API)."""
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"host": host, "port": port, "status": "connected"}
Create File ${repo}${/}src${/}client.py ${lib_content}
${git_add}= Run Process git add . cwd=${repo} timeout=30s on_timeout=kill
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
${git_commit}= Run Process git commit -m Initial common library v1 cwd=${repo} timeout=30s on_timeout=kill
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
RETURN ${repo}
[Documentation] Create temp git repo for the common library.
${repo}= Create Temp Git Repo ${LIB_BASE}-${RUN_SUFFIX}
Create Directory ${repo}${/}src
${lib_content}= Catenate SEPARATOR=\n
... """Common library v1 — shared utilities."""
... ${EMPTY}
... ${EMPTY}
... __version__ = "1.0.0"
... ${EMPTY}
... ${EMPTY}
... def connect(host, port):
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Connect to a service (v1 API)."""
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"host": host, "port": port, "status": "connected"}
Create File ${repo}${/}src${/}client.py ${lib_content}
${git_add}= Run Process git add . cwd=${repo} timeout=30s on_timeout=kill
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
${git_commit}= Run Process git commit -m Initial common library v1 cwd=${repo} timeout=30s on_timeout=kill
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
RETURN ${repo}
Create Service Repo
[Documentation] Create temp git repo for a microservice.
[Arguments] ${name} ${import_line}
${repo}= Create Temp Git Repo ${name}-${RUN_SUFFIX}
Create Directory ${repo}${/}src
${svc_content}= Catenate SEPARATOR=\n
... """${name} service — depends on common-lib v1."""
... ${import_line}
... ${EMPTY}
... ${EMPTY}
... def start():
... ${SPACE}${SPACE}${SPACE}${SPACE}conn = connect("localhost", 8080)
... ${SPACE}${SPACE}${SPACE}${SPACE}return conn
Create File ${repo}${/}src${/}app.py ${svc_content}
Create File ${repo}${/}requirements.txt common-lib==1.0.0\n
${git_add}= Run Process git add . cwd=${repo} timeout=30s on_timeout=kill
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
${git_commit}= Run Process git commit -m Initial ${name} service cwd=${repo} timeout=30s on_timeout=kill
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
RETURN ${repo}
[Documentation] Create temp git repo for a microservice.
[Arguments] ${name} ${import_line}
${repo}= Create Temp Git Repo ${name}-${RUN_SUFFIX}
Create Directory ${repo}${/}src
${svc_content}= Catenate SEPARATOR=\n
... """${name} service — depends on common-lib v1."""
... ${import_line}
... ${EMPTY}
... ${EMPTY}
... def start():
... ${SPACE}${SPACE}${SPACE}${SPACE}conn = connect("localhost", 8080)
... ${SPACE}${SPACE}${SPACE}${SPACE}return conn
Create File ${repo}${/}src${/}app.py ${svc_content}
Create File ${repo}${/}requirements.txt common-lib==1.0.0\n
${git_add}= Run Process git add . cwd=${repo} timeout=30s on_timeout=kill
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
${git_commit}= Run Process git commit -m Initial ${name} service cwd=${repo} timeout=30s on_timeout=kill
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
RETURN ${repo}
Register Resource And Project
[Documentation] Register a git-checkout resource and create a project.
[Arguments] ${resource_name} ${project_name} ${repo_dir}
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=30s on_timeout=kill
Should Be Equal As Integers ${branch_result.rc} 0 msg=git rev-parse failed: ${branch_result.stderr}
${branch}= Strip String ${branch_result.stdout}
${r_res}= Run CleverAgents Command
... resource add git-checkout ${resource_name}
... --path ${repo_dir} --branch ${branch}
Should Be Equal As Integers ${r_res.rc} 0 resource add failed: ${r_res.stderr}
Should Not Contain ${r_res.stdout}${r_res.stderr} Traceback
Should Not Contain ${r_res.stdout}${r_res.stderr} INTERNAL
${r_proj}= Run CleverAgents Command
... project create ${project_name}
... --resource ${resource_name}
Should Be Equal As Integers ${r_proj.rc} 0 project create failed: ${r_proj.stderr}
Should Not Contain ${r_proj.stdout}${r_proj.stderr} Traceback
Should Not Contain ${r_proj.stdout}${r_proj.stderr} INTERNAL
[Documentation] Register a git-checkout resource and create a project.
[Arguments] ${resource_name} ${project_name} ${repo_dir}
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=30s on_timeout=kill
Should Be Equal As Integers ${branch_result.rc} 0 git rev-parse failed: ${branch_result.stderr}
${branch}= Strip String ${branch_result.stdout}
${r_res}= Run CleverAgents Command
... resource add git-checkout ${resource_name}
... --path ${repo_dir} --branch ${branch}
Should Be Equal As Integers ${r_res.rc} 0 resource add failed: ${r_res.stderr}
Should Not Contain ${r_res.stdout}${r_res.stderr} Traceback
Should Not Contain ${r_res.stdout}${r_res.stderr} INTERNAL
${r_proj}= Run CleverAgents Command
... project create ${project_name}
... --resource ${resource_name}
Should Be Equal As Integers ${r_proj.rc} 0 project create failed: ${r_proj.stderr}
Should Not Contain ${r_proj.stdout}${r_proj.stderr} Traceback
Should Not Contain ${r_proj.stdout}${r_proj.stderr} INTERNAL
WF04 Test Teardown
[Documentation] Log diagnostic context on failure for debugging.
... Captures plan status and decision tree so CI failures
... in this 25-minute LLM-dependent test have actionable data.
${plan_id}= Get Variable Value ${WF04_PLAN_ID} ${EMPTY}
IF '${plan_id}' != ''
${status} ${result}= Run Keyword And Ignore Error
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
IF '${status}' == 'PASS'
Log Teardown plan status: ${result.stdout} WARN
END
${tree_status} ${tree_result}= Run Keyword And Ignore Error
... Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=30s
IF '${tree_status}' == 'PASS'
Log Teardown plan tree: ${tree_result.stdout} WARN
END
END
[Documentation] Log diagnostic context on failure for debugging.
... Captures plan status and decision tree so CI failures
... in this 25-minute LLM-dependent test have actionable data.
${plan_id}= Get Variable Value ${WF04_PLAN_ID} ${EMPTY}
IF '${plan_id}' != ''
${status} ${result}= Run Keyword And Ignore Error
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
IF '${status}' == 'PASS'
Log Teardown plan status: ${result.stdout} WARN
END
${tree_status} ${tree_result}= Run Keyword And Ignore Error
... Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=30s
IF '${tree_status}' == 'PASS'
Log Teardown plan tree: ${tree_result.stdout} WARN
END
END
Register Validation For Project
[Documentation] Create a validation YAML, register it, and attach it to a project's resource.
[Arguments] ${validation_name} ${resource_name} ${project_name}
${val_yaml}= Catenate SEPARATOR=\n
... name: ${validation_name}
... description: "Simple pass-through validation for E2E testing"
... source: custom
... mode: required
... code: |
... ${SPACE}${SPACE}return {"passed": True, "data": {}, "message": "Validation passed"}
... input_schema:
... ${SPACE}${SPACE}type: object
... ${SPACE}${SPACE}properties: {}
... timeout: 30
${val_path}= Set Variable ${SUITE_HOME}${/}${validation_name}.yaml
Create File ${val_path} ${val_yaml}
${r_add}= Run CleverAgents Command
... validation add --config ${val_path} expected_rc=None
Log Validation add rc=${r_add.rc} stdout=${r_add.stdout} stderr=${r_add.stderr}
Should Be Equal As Integers ${r_add.rc} 0 msg=validation add failed (rc=${r_add.rc}): ${r_add.stderr}
Should Not Contain ${r_add.stdout}${r_add.stderr} Traceback
Should Not Contain ${r_add.stdout}${r_add.stderr} INTERNAL
# Positional args: <resource_name> <validation_name> (resource first, validation second)
${r_attach}= Run CleverAgents Command
... validation attach --project ${project_name}
... ${resource_name} ${validation_name} expected_rc=None
Log Validation attach rc=${r_attach.rc} stdout=${r_attach.stdout} stderr=${r_attach.stderr}
Should Be Equal As Integers ${r_attach.rc} 0 msg=validation attach failed for ${project_name} (rc=${r_attach.rc}): ${r_attach.stderr}
Should Not Contain ${r_attach.stdout}${r_attach.stderr} Traceback
Should Not Contain ${r_attach.stdout}${r_attach.stderr} INTERNAL
[Documentation] Create a validation YAML, register it, and attach it to a project's resource.
[Arguments] ${validation_name} ${resource_name} ${project_name}
${val_yaml}= Catenate SEPARATOR=\n
... name: ${validation_name}
... description: "Simple pass-through validation for E2E testing"
... source: custom
... mode: required
... code: |
... ${SPACE}${SPACE}return {"passed": True, "data": {}, "message": "Validation passed"}
... input_schema:
... ${SPACE}${SPACE}type: object
... ${SPACE}${SPACE}properties: {}
... timeout: 30
${val_path}= Set Variable ${SUITE_HOME}${/}${validation_name}.yaml
Create File ${val_path} ${val_yaml}
${r_add}= Run CleverAgents Command
... validation add --config ${val_path} expected_rc=None
Log Validation add rc=${r_add.rc} stdout=${r_add.stdout} stderr=${r_add.stderr}
Should Be Equal As Integers ${r_add.rc} 0
... validation add failed (rc=${r_add.rc}): ${r_add.stderr}
Should Not Contain ${r_add.stdout}${r_add.stderr} Traceback
Should Not Contain ${r_add.stdout}${r_add.stderr} INTERNAL
# Positional args: <resource_name> <validation_name> (resource first, validation second)
${r_attach}= Run CleverAgents Command
... validation attach --project ${project_name}
... ${resource_name} ${validation_name} expected_rc=None
Log Validation attach rc=${r_attach.rc} stdout=${r_attach.stdout} stderr=${r_attach.stderr}
Should Be Equal As Integers ${r_attach.rc} 0
... validation attach failed for ${project_name} (rc=${r_attach.rc}): ${r_attach.stderr}
Should Not Contain ${r_attach.stdout}${r_attach.stderr} Traceback
Should Not Contain ${r_attach.stdout}${r_attach.stderr} INTERNAL
Attach Validation To Project
[Documentation] Attach an already-registered validation to a project's resource.
[Arguments] ${validation_name} ${resource_name} ${project_name}
# Positional args: <resource_name> <validation_name> (resource first, validation second)
${r_attach}= Run CleverAgents Command
... validation attach --project ${project_name}
... ${resource_name} ${validation_name} expected_rc=None
Log Validation attach rc=${r_attach.rc} stdout=${r_attach.stdout} stderr=${r_attach.stderr}
Should Be Equal As Integers ${r_attach.rc} 0 msg=validation attach failed for ${project_name} (rc=${r_attach.rc}): ${r_attach.stderr}
Should Not Contain ${r_attach.stdout}${r_attach.stderr} Traceback
Should Not Contain ${r_attach.stdout}${r_attach.stderr} INTERNAL
[Documentation] Attach an already-registered validation to a project's resource.
[Arguments] ${validation_name} ${resource_name} ${project_name}
# Positional args: <resource_name> <validation_name> (resource first, validation second)
${r_attach}= Run CleverAgents Command
... validation attach --project ${project_name}
... ${resource_name} ${validation_name} expected_rc=None
Log Validation attach rc=${r_attach.rc} stdout=${r_attach.stdout} stderr=${r_attach.stderr}
Should Be Equal As Integers ${r_attach.rc} 0
... validation attach failed for ${project_name} (rc=${r_attach.rc}): ${r_attach.stderr}
Should Not Contain ${r_attach.stdout}${r_attach.stderr} Traceback
Should Not Contain ${r_attach.stdout}${r_attach.stderr} INTERNAL
Parse Json Payload
[Documentation] Parse JSON object/array from stdout with optional log preamble.
... Delegates to ``Extract JSON From Stdout`` which uses
... ``json.JSONDecoder().raw_decode()`` for robustness against
... trailing non-JSON output.
[Arguments] ${stdout}
${parsed}= Extract JSON From Stdout ${stdout}
RETURN ${parsed}
[Documentation] Parse JSON object/array from stdout with optional log preamble.
... Delegates to ``Extract JSON From Stdout`` which uses
... ``json.JSONDecoder().raw_decode()`` for robustness against
... trailing non-JSON output.
[Arguments] ${stdout}
${parsed}= Extract JSON From Stdout ${stdout}
RETURN ${parsed}
Get WF04 Plan Snapshot
[Documentation] Read parent/subplan metadata for deterministic WF04 assertions.
[Arguments] ${plan_id}
${snapshot_result}= Run Process
... ${PYTHON} ${WF04_SNAPSHOT_HELPER} ${plan_id}
... cwd=${SUITE_HOME} timeout=120s on_timeout=kill
... env:CLEVERAGENTS_HOME=${SUITE_HOME}
... env:CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true
... env:NO_COLOR=1
... env:PYTHONPATH=${WORKSPACE}${/}src
Should Be Equal As Integers ${snapshot_result.rc} 0 msg=Snapshot helper failed (rc=${snapshot_result.rc}): ${snapshot_result.stderr}
Should Not Be Empty ${snapshot_result.stdout}
${snapshot}= Parse Json Payload ${snapshot_result.stdout}
RETURN ${snapshot}
[Documentation] Read parent/subplan metadata for deterministic WF04 assertions.
[Arguments] ${plan_id}
${snapshot_result}= Run Process
... ${PYTHON} ${WF04_SNAPSHOT_HELPER} ${plan_id}
... cwd=${SUITE_HOME} timeout=120s on_timeout=kill
... env:CLEVERAGENTS_HOME=${SUITE_HOME}
... env:CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true
... env:NO_COLOR=1
... env:PYTHONPATH=${WORKSPACE}${/}src
Should Be Equal As Integers ${snapshot_result.rc} 0
... Snapshot helper failed (rc=${snapshot_result.rc}): ${snapshot_result.stderr}
Should Not Be Empty ${snapshot_result.stdout}
${snapshot}= Parse Json Payload ${snapshot_result.stdout}
RETURN ${snapshot}
Verify WF04 Child Plan Spawning
[Documentation] AC-4: verify exactly 4 child plans mapped one-per-project.
[Arguments] ${snapshot}
${subplans}= Evaluate $snapshot.get('subplans', [])
${subplan_count}= Evaluate len($subplans)
IF ${subplan_count} == 0
Skip LLM produced no child plans - AC-4 child-plan mapping assertions were not exercised
END
Should Be Equal As Integers ${subplan_count} 4 msg=Expected exactly 4 child plans (common-lib + 3 services), found ${subplan_count}
${mapped_projects}= Evaluate sorted({p for sp in $subplans for p in sp.get('mapped_projects', []) if p})
${expected_projects}= Create List ${LIB_PROJECT} ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
${expected_sorted}= Evaluate sorted($expected_projects)
${all_projects_covered}= Evaluate $mapped_projects == $expected_sorted
Should Be True ${all_projects_covered}
... Child plans should cover all 4 projects. expected=${expected_sorted} actual=${mapped_projects}
${single_mapping}= Evaluate all(len(sp.get('mapped_projects', [])) == 1 for sp in $subplans)
Should Be True ${single_mapping}
... Each child plan should map to exactly one project scope
[Documentation] AC-4: verify exactly 4 child plans mapped one-per-project.
[Arguments] ${snapshot}
${subplans}= Evaluate $snapshot.get('subplans', [])
${subplan_count}= Evaluate len($subplans)
IF ${subplan_count} == 0
Skip LLM produced no child plans AC-4 child-plan mapping assertions were not exercised
END
Should Be Equal As Integers ${subplan_count} 4
... Expected exactly 4 child plans (common-lib + 3 services), found ${subplan_count}
${mapped_projects}= Evaluate sorted({p for sp in $subplans for p in sp.get('mapped_projects', []) if p})
${expected_projects}= Create List ${LIB_PROJECT} ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
${expected_sorted}= Evaluate sorted($expected_projects)
${all_projects_covered}= Evaluate $mapped_projects == $expected_sorted
Should Be True ${all_projects_covered}
... Child plans should cover all 4 projects. expected=${expected_sorted} actual=${mapped_projects}
${single_mapping}= Evaluate all(len(sp.get('mapped_projects', [])) == 1 for sp in $subplans)
Should Be True ${single_mapping}
... Each child plan should map to exactly one project scope
Verify WF04 Execution Order
[Documentation] AC-5: common-lib executes before service subplans.
[Arguments] ${snapshot}
${subplans}= Evaluate $snapshot.get('subplans', [])
${subplan_count}= Evaluate len($subplans)
IF ${subplan_count} == 0
Skip LLM produced no child plans - AC-5 execution-order assertions were not exercised
END
${lib_subplans}= Evaluate [sp for sp in $subplans if '${LIB_PROJECT}' in sp.get('mapped_projects', [])]
${lib_count}= Evaluate len($lib_subplans)
Should Be Equal As Integers ${lib_count} 1 msg=Expected exactly one common-lib child plan, found ${lib_count}
${lib_completed}= Evaluate $lib_subplans[0].get('completed_at') or $lib_subplans[0].get('execute_completed_at') or ''
Should Not Be Empty ${lib_completed}
... common-lib child plan completion timestamp is required for ordering checks
${service_projects}= Create List ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
${svc_subplans}= Evaluate [sp for sp in $subplans if any(p in $service_projects for p in sp.get('mapped_projects', []))]
${svc_count}= Evaluate len($svc_subplans)
Should Be Equal As Integers ${svc_count} 3 msg=Expected exactly three service child plans, found ${svc_count}
${svc_starts_present}= Evaluate all((sp.get('started_at') or sp.get('execute_started_at') or '') != '' for sp in $svc_subplans)
Should Be True ${svc_starts_present}
... Service child plans must expose start timestamps for execution-order checks
${services_after_lib}= Evaluate all((sp.get('started_at') or sp.get('execute_started_at')) >= $lib_completed for sp in $svc_subplans)
Should Be True ${services_after_lib}
... Service execution must start only after common-lib execution completes
[Documentation] AC-5: common-lib executes before service subplans.
[Arguments] ${snapshot}
${subplans}= Evaluate $snapshot.get('subplans', [])
${subplan_count}= Evaluate len($subplans)
IF ${subplan_count} == 0
Skip LLM produced no child plans AC-5 execution-order assertions were not exercised
END
${lib_subplans}= Evaluate [sp for sp in $subplans if '${LIB_PROJECT}' in sp.get('mapped_projects', [])]
${lib_count}= Evaluate len($lib_subplans)
Should Be Equal As Integers ${lib_count} 1
... Expected exactly one common-lib child plan, found ${lib_count}
${lib_completed}= Evaluate $lib_subplans[0].get('completed_at') or $lib_subplans[0].get('execute_completed_at') or ''
Should Not Be Empty ${lib_completed}
... common-lib child plan completion timestamp is required for ordering checks
${service_projects}= Create List ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
${svc_subplans}= Evaluate [sp for sp in $subplans if any(p in $service_projects for p in sp.get('mapped_projects', []))]
${svc_count}= Evaluate len($svc_subplans)
Should Be Equal As Integers ${svc_count} 3
... Expected exactly three service child plans, found ${svc_count}
${svc_starts_present}= Evaluate all((sp.get('started_at') or sp.get('execute_started_at') or '') != '' for sp in $svc_subplans)
Should Be True ${svc_starts_present}
... Service child plans must expose start timestamps for execution-order checks
${services_after_lib}= Evaluate all((sp.get('started_at') or sp.get('execute_started_at')) >= $lib_completed for sp in $svc_subplans)
Should Be True ${services_after_lib}
... Service execution must start only after common-lib execution completes
Verify WF04 Validation Outcomes
[Documentation] AC-6: each child plan must report validation pass results.
[Arguments] ${snapshot}
${subplans}= Evaluate $snapshot.get('subplans', [])
${subplan_count}= Evaluate len($subplans)
IF ${subplan_count} == 0
Skip LLM produced no child plans - AC-6 per-project validation assertions were not exercised
END
${validation_present}= Evaluate all(isinstance(sp.get('child_validation_summary'), dict) and len(sp.get('child_validation_summary')) > 0 for sp in $subplans)
Should Be True ${validation_present}
... Each child plan must expose a non-empty validation summary
${validation_passed}= Evaluate all(int((sp.get('child_validation_summary') or {}).get('required_passed', 0) or 0) >= 1 and int((sp.get('child_validation_summary') or {}).get('required_failed', 0) or 0) == 0 for sp in $subplans)
Should Be True ${validation_passed}
... Each child plan must pass required validations (required_failed == 0)
[Documentation] AC-6: each child plan must report validation pass results.
[Arguments] ${snapshot}
${subplans}= Evaluate $snapshot.get('subplans', [])
${subplan_count}= Evaluate len($subplans)
IF ${subplan_count} == 0
Skip LLM produced no child plans AC-6 per-project validation assertions were not exercised
END
${validation_present}= Evaluate all(isinstance(sp.get('child_validation_summary'), dict) and len(sp.get('child_validation_summary')) > 0 for sp in $subplans)
Should Be True ${validation_present}
... Each child plan must expose a non-empty validation summary
${validation_passed}= Evaluate all(int((sp.get('child_validation_summary') or {}).get('required_passed', 0) or 0) >= 1 and int((sp.get('child_validation_summary') or {}).get('required_failed', 0) or 0) == 0 for sp in $subplans)
Should Be True ${validation_passed}
... Each child plan must pass required validations (required_failed == 0)
Verify WF04 Apply Order
[Documentation] AC-7: common-lib apply must complete before service applies.
[Arguments] ${snapshot}
${subplans}= Evaluate $snapshot.get('subplans', [])
${subplan_count}= Evaluate len($subplans)
IF ${subplan_count} == 0
Skip LLM produced no child plans - AC-7 apply-order assertions were not exercised
END
${lib_subplans}= Evaluate [sp for sp in $subplans if '${LIB_PROJECT}' in sp.get('mapped_projects', [])]
${lib_count}= Evaluate len($lib_subplans)
Should Be Equal As Integers ${lib_count} 1 msg=Expected exactly one common-lib child plan, found ${lib_count}
${lib_applied}= Evaluate $lib_subplans[0].get('applied_at') or $lib_subplans[0].get('child_updated_at') or ''
Should Not Be Empty ${lib_applied}
... common-lib child plan apply timestamp is required for apply-order checks
${service_projects}= Create List ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
${svc_subplans}= Evaluate [sp for sp in $subplans if any(p in $service_projects for p in sp.get('mapped_projects', []))]
${svc_count}= Evaluate len($svc_subplans)
Should Be Equal As Integers ${svc_count} 3 msg=Expected exactly three service child plans, found ${svc_count}
${svc_apply_present}= Evaluate all((sp.get('applied_at') or sp.get('child_updated_at') or '') != '' for sp in $svc_subplans)
Should Be True ${svc_apply_present}
... Service child plans must expose apply/updated timestamps for apply-order checks
${services_after_lib_apply}= Evaluate all((sp.get('applied_at') or sp.get('child_updated_at')) >= $lib_applied for sp in $svc_subplans)
Should Be True ${services_after_lib_apply}
... Service applies must occur after common-lib apply
[Documentation] AC-7: common-lib apply must complete before service applies.
[Arguments] ${snapshot}
${subplans}= Evaluate $snapshot.get('subplans', [])
${subplan_count}= Evaluate len($subplans)
IF ${subplan_count} == 0
Skip LLM produced no child plans AC-7 apply-order assertions were not exercised
END
${lib_subplans}= Evaluate [sp for sp in $subplans if '${LIB_PROJECT}' in sp.get('mapped_projects', [])]
${lib_count}= Evaluate len($lib_subplans)
Should Be Equal As Integers ${lib_count} 1
... Expected exactly one common-lib child plan, found ${lib_count}
${lib_applied}= Evaluate $lib_subplans[0].get('applied_at') or $lib_subplans[0].get('child_updated_at') or ''
Should Not Be Empty ${lib_applied}
... common-lib child plan apply timestamp is required for apply-order checks
${service_projects}= Create List ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
${svc_subplans}= Evaluate [sp for sp in $subplans if any(p in $service_projects for p in sp.get('mapped_projects', []))]
${svc_count}= Evaluate len($svc_subplans)
Should Be Equal As Integers ${svc_count} 3
... Expected exactly three service child plans, found ${svc_count}
${svc_apply_present}= Evaluate all((sp.get('applied_at') or sp.get('child_updated_at') or '') != '' for sp in $svc_subplans)
Should Be True ${svc_apply_present}
... Service child plans must expose apply/updated timestamps for apply-order checks
${services_after_lib_apply}= Evaluate all((sp.get('applied_at') or sp.get('child_updated_at')) >= $lib_applied for sp in $svc_subplans)
Should Be True ${services_after_lib_apply}
... Service applies must occur after common-lib apply
Count Decision Nodes
[Documentation] Recursively count decision nodes in a plan tree JSON structure.
... Invokes ``wf04_snapshot_helper.py --count-nodes`` as a subprocess
... to avoid importing the application DI container into the Robot
... test runner process.
[Arguments] ${tree_payload}
${tree_json}= Evaluate __import__('json').dumps($tree_payload)
${tmp_path}= Evaluate __import__('tempfile').NamedTemporaryFile(mode='w', suffix='.json', delete=False).name
Evaluate __import__('pathlib').Path(r'${tmp_path}').write_text($tree_json, encoding='utf-8')
${result}= Run Process
... ${PYTHON} ${WF04_SNAPSHOT_HELPER} --count-nodes ${tmp_path}
... timeout=30s on_timeout=kill
Evaluate __import__('os').unlink(r'${tmp_path}')
Should Be Equal As Integers ${result.rc} 0 msg=count-nodes failed (rc=${result.rc}): ${result.stderr}
${count}= Convert To Integer ${result.stdout.strip()}
RETURN ${count}
[Documentation] Recursively count decision nodes in a plan tree JSON structure.
... Invokes ``wf04_snapshot_helper.py --count-nodes`` as a subprocess
... to avoid importing the application DI container into the Robot
... test runner process.
[Arguments] ${tree_payload}
${tree_json}= Evaluate __import__('json').dumps($tree_payload)
${tmp_path}= Evaluate __import__('tempfile').NamedTemporaryFile(mode='w', suffix='.json', delete=False).name
Evaluate __import__('pathlib').Path(r'${tmp_path}').write_text($tree_json, encoding='utf-8')
${result}= Run Process
... ${PYTHON} ${WF04_SNAPSHOT_HELPER} --count-nodes ${tmp_path}
... timeout=30s on_timeout=kill
Evaluate __import__('os').unlink(r'${tmp_path}')
Should Be Equal As Integers ${result.rc} 0
... count-nodes failed (rc=${result.rc}): ${result.stderr}
${count}= Convert To Integer ${result.stdout.strip()}
RETURN ${count}
Verify Plan In List
[Documentation] Verify a plan appears in list output.
[Arguments] ${plan_id}
${list_result}= Run CleverAgents Command
... plan list --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${list_result.rc} 0 msg=list failed (rc=${list_result.rc}): ${list_result.stderr}
Output Should Contain ${list_result} ${plan_id}
[Documentation] Verify a plan appears in list output.
[Arguments] ${plan_id}
${list_result}= Run CleverAgents Command
... plan list --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${list_result.rc} 0
... list failed (rc=${list_result.rc}): ${list_result.stderr}
Output Should Contain ${list_result} ${plan_id}
*** Test Cases ***
WF04 Multi Project Dependency Update Supervised Profile
[Documentation] Full supervised-profile workflow: register 4 repos,
... create multi-project action with invariants, plan use
... targeting all 4 projects with --automation-profile supervised,
... execute with child plan spawning in dependency order,
... verify per-project validation, and apply in dependency order.
[Tags] tdd_issue tdd_issue_4189
[Timeout] 25 minutes
[Teardown] WF04 Test Teardown
Skip If No LLM Keys
# Initialise test variable for teardown access.
Set Test Variable ${WF04_PLAN_ID} ${EMPTY}
[Documentation] Full supervised-profile workflow: register 4 repos,
... create multi-project action with invariants, plan use
... targeting all 4 projects with --automation-profile supervised,
... execute with child plan spawning in dependency order,
... verify per-project validation, and apply in dependency order.
[Timeout] 25 minutes
[Teardown] WF04 Test Teardown
Skip If No LLM Keys
# Initialise test variable for teardown access.
Set Test Variable ${WF04_PLAN_ID} ${EMPTY}
# ---- Create fixture repos ----
${lib_repo}= Create Library Repo
${svc1_repo}= Create Service Repo svc-auth from common_lib.client import connect
${svc2_repo}= Create Service Repo svc-billing from common_lib.client import connect
${svc3_repo}= Create Service Repo svc-gateway from common_lib.client import connect
# ---- Register resources and projects ----
Register Resource And Project ${LIB_RESOURCE} ${LIB_PROJECT} ${lib_repo}
Register Resource And Project ${SVC1_RESOURCE} ${SVC1_PROJECT} ${svc1_repo}
Register Resource And Project ${SVC2_RESOURCE} ${SVC2_PROJECT} ${svc2_repo}
Register Resource And Project ${SVC3_RESOURCE} ${SVC3_PROJECT} ${svc3_repo}
# ---- Register and attach validations for all 4 projects (AC-6) ----
${val_name}= Set Variable local/wf04-val-${RUN_SUFFIX}
Register Validation For Project ${val_name} ${LIB_RESOURCE} ${LIB_PROJECT}
# Validation already registered; attach to remaining 3 projects
Attach Validation To Project ${val_name} ${SVC1_RESOURCE} ${SVC1_PROJECT}
Attach Validation To Project ${val_name} ${SVC2_RESOURCE} ${SVC2_PROJECT}
Attach Validation To Project ${val_name} ${SVC3_RESOURCE} ${SVC3_PROJECT}
# ---- Create action with supervised profile and invariants ----
${action_yaml}= Catenate SEPARATOR=\n
... name: ${ACTION_NAME}
... description: Update common-lib from v1 to v2 across all dependent services
... definition_of_done: All services updated to use common-lib v2 API
... strategy_actor: ${LLM_ACTOR}
... execution_actor: ${LLM_ACTOR}
... automation_profile: supervised
... reusable: true
... state: available
... invariants:
... ${SPACE}${SPACE}- "Each dependent project must be updated in its own child plan"
... ${SPACE}${SPACE}- "All child plans must pass validation before any can be applied"
... ${SPACE}${SPACE}- "The library update in common-lib must be applied first"
${action_path}= Set Variable ${SUITE_HOME}${/}wf04_action.yaml
Create File ${action_path} ${action_yaml}
${r_action}= Run CleverAgents Command
... action create --config ${action_path}
Should Be Equal As Integers ${r_action.rc} 0
... action create failed (rc=${r_action.rc}): ${r_action.stderr}
Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback
Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL
Output Should Contain ${r_action} ${ACTION_NAME}
# ---- Plan use targeting ALL 4 projects with supervised profile (AC-3) ----
${r_use}= Run CleverAgents Command
... plan use ${ACTION_NAME}
... ${LIB_PROJECT} ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
... --automation-profile supervised
... --format json
... timeout=120s
Should Be Equal As Integers ${r_use.rc} 0
... plan use failed (rc=${r_use.rc}): ${r_use.stderr}
Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback
Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL
${use_payload}= Parse Json Payload ${r_use.stdout}
${plan_id}= Evaluate str($use_payload.get('plan_id', ''))
Should Not Be Empty ${plan_id} msg=Expected plan_id in plan use JSON output
Log Plan ID: ${plan_id}
Set Test Variable ${WF04_PLAN_ID} ${plan_id}
# Verify plan targets all 4 projects exactly (AC-3)
${use_projects}= Evaluate sorted([link.get('project_name', '') for link in $use_payload.get('project_links', []) if isinstance(link, dict)])
${expected_projects}= Create List ${LIB_PROJECT} ${SVC1_PROJECT} ${SVC2_PROJECT} ${SVC3_PROJECT}
${expected_sorted}= Evaluate sorted($expected_projects)
${projects_match}= Evaluate $use_projects == $expected_sorted
Should Be True ${projects_match}
... plan use should target all 4 projects. expected=${expected_sorted} actual=${use_projects}
# ---- Strategize ----
# Supervised profile requires two explicit ``plan execute`` calls:
# the first advances the plan through strategize, the second runs
# actual execution. Both use the same CLI command.
${r_strat}= Run CleverAgents Command
... plan execute ${plan_id}
... --format json expected_rc=None timeout=180s
Log Strategize rc=${r_strat.rc} stdout=${r_strat.stdout} stderr=${r_strat.stderr}
Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback
Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL
IF ${r_strat.rc} != 0
Fail plan execute (strategize) failed (rc=${r_strat.rc}): ${r_strat.stderr}
END
# ---- Decision tree — verify child plan spawning (AC-4) ----
${r_tree}= Run CleverAgents Command
... plan tree ${plan_id} --format json
... expected_rc=None timeout=60s
Should Be Equal As Integers ${r_tree.rc} 0
... plan tree failed (rc=${r_tree.rc}): ${r_tree.stderr}
Should Not Contain ${r_tree.stdout}${r_tree.stderr} Traceback
Should Not Contain ${r_tree.stdout}${r_tree.stderr} INTERNAL
Should Not Be Empty ${r_tree.stdout} Plan tree output should not be empty
Log Decision tree: ${r_tree.stdout}
# Parse tree for child plan / decision structure
${tree_payload}= Parse Json Payload ${r_tree.stdout}
${decision_count}= Count Decision Nodes ${tree_payload}
Log Decision tree contains ${decision_count} decision node(s)
Should Be True ${decision_count} >= 1
... Plan tree should contain at least one decision node after strategize (found ${decision_count})
# Some providers/runs expose a minimal strategize tree before execute.
# Child spawning is asserted deterministically after execute via snapshot.
# ---- Execute — dependency-ordered execution (AC-5) ----
${r_exec}= Run CleverAgents Command
... plan execute ${plan_id}
... --format json expected_rc=None timeout=300s
Log Execute rc=${r_exec.rc} stdout=${r_exec.stdout} stderr=${r_exec.stderr}
Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback
Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL
IF ${r_exec.rc} != 0
Fail plan execute failed (rc=${r_exec.rc}): ${r_exec.stderr}
END
# Deterministic WF04 assertions after execute (AC-4/AC-5/AC-6)
${exec_snapshot}= Get WF04 Plan Snapshot ${plan_id}
# Guard: if the snapshot contains zero subplans, skip the entire test rather
# than letting individual verification keywords silently skip all ACs.
# This ensures CI reports show SKIPPED (visible) rather than PASSED (misleading).
${exec_subplan_count}= Evaluate int($exec_snapshot.get('subplan_count', 0))
IF ${exec_subplan_count} == 0
Skip LLM produced 0 subplans — AC-4/5/6/7 verification cannot be exercised (entire test skipped)
END
Verify WF04 Child Plan Spawning ${exec_snapshot}
Verify WF04 Execution Order ${exec_snapshot}
Verify WF04 Validation Outcomes ${exec_snapshot}
# ---- Post-execute decision tree — verify child plan count (AC-4) ----
${r_tree_post}= Run CleverAgents Command
... plan tree ${plan_id} --format json
... expected_rc=None timeout=60s
Should Be Equal As Integers ${r_tree_post.rc} 0
... plan tree (post-execute) failed (rc=${r_tree_post.rc}): ${r_tree_post.stderr}
Should Not Contain ${r_tree_post.stdout}${r_tree_post.stderr} Traceback
Should Not Contain ${r_tree_post.stdout}${r_tree_post.stderr} INTERNAL
${tree_post_payload}= Parse Json Payload ${r_tree_post.stdout}
${post_exec_decision_count}= Count Decision Nodes ${tree_post_payload}
Log Post-execute decision tree contains ${post_exec_decision_count} decision node(s)
# Require non-trivial tree depth after execute. Subplan spawning is
# asserted deterministically via internal snapshot checks above.
Should Be True ${post_exec_decision_count} >= 2
... Plan tree should contain at least 2 decision nodes after execute (found ${post_exec_decision_count})
# Execute should preserve or grow the tree — never shrink it.
Should Be True ${post_exec_decision_count} >= ${decision_count}
... Post-execute decision count (${post_exec_decision_count}) should not be less than post-strategize count (${decision_count})
# ---- AC-6 verified above via per-child validation summaries ----
# ---- Plan list — verify plan exists ----
Verify Plan In List ${plan_id}
# ---- Verify plan status shows multi-project state ----
${r_status_mid}= Run CleverAgents Command
... plan status ${plan_id} --format json
... expected_rc=None timeout=60s
Should Be Equal As Integers ${r_status_mid.rc} 0
... plan status failed (rc=${r_status_mid.rc}): ${r_status_mid.stderr}
Should Not Contain ${r_status_mid.stdout}${r_status_mid.stderr} Traceback
Should Not Contain ${r_status_mid.stdout}${r_status_mid.stderr} INTERNAL
Output Should Contain ${r_status_mid} ${plan_id}
# Check for automation profile reference in status
${status_combined}= Set Variable ${r_status_mid.stdout}${r_status_mid.stderr}
${status_lower}= Evaluate ($status_combined).lower()
${has_profile_ref}= Evaluate 'supervised' in $status_lower or 'automation' in $status_lower or 'profile' in $status_lower
Should Be True ${has_profile_ref}
... Plan status should reference automation profile (supervised)
# Parse automation_profile field for precise assertion
${mid_profile}= Safe Parse Json Field ${r_status_mid.stdout} automation_profile
Should Not Be Empty ${mid_profile}
... automation_profile field should be present in plan status JSON
Should Be Equal As Strings ${mid_profile} supervised
... Plan automation profile should be supervised (found ${mid_profile})
# ---- Diff ----
${r_diff}= Run CleverAgents Command
... plan diff ${plan_id} --format plain
... expected_rc=None timeout=60s
Log Diff rc=${r_diff.rc} stdout=${r_diff.stdout} stderr=${r_diff.stderr}
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
Should Be Equal As Integers ${r_diff.rc} 0
... plan diff failed (rc=${r_diff.rc}): ${r_diff.stderr}
# ---- Apply — dependency-ordered apply (AC-7) ----
${r_apply}= Run CleverAgents Command
... plan apply ${plan_id} --yes --format json
... expected_rc=None timeout=180s
Log Apply rc=${r_apply.rc} stdout=${r_apply.stdout} stderr=${r_apply.stderr}
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
IF ${r_apply.rc} == 0
Output Should Contain ${r_apply} ${plan_id}
# Verify the plan transitioned — check for apply-phase indicators (AC-7)
${apply_phase}= Safe Parse Json Field ${r_apply.stdout} phase
Should Not Be Empty ${apply_phase}
... phase field should be present in apply JSON output
${apply_phase_lower}= Evaluate ($apply_phase).lower()
Should Contain ${apply_phase_lower} apply
... Plan phase should indicate apply after apply (found ${apply_phase})
${apply_snapshot}= Get WF04 Plan Snapshot ${plan_id}
# Guard: apply snapshot must also contain subplans for AC-7 verification.
# The exec guard above already skips the whole test if 0 subplans, so
# reaching this point implies subplans existed post-execute. If they
# disappeared post-apply, that would be a real regression worth flagging.
${apply_subplan_count}= Evaluate int($apply_snapshot.get('subplan_count', 0))
Should Be True ${apply_subplan_count} >= 1
... Snapshot reported 0 subplans after apply — subplans existed post-execute but vanished post-apply
Verify WF04 Apply Order ${apply_snapshot}
Verify WF04 Validation Outcomes ${apply_snapshot}
ELSE
Fail apply failed (rc=${r_apply.rc}) stdout=${r_apply.stdout} stderr=${r_apply.stderr}
END
# ---- Verify final status ----
${r_status}= Run CleverAgents Command
... plan status ${plan_id} --format json
... expected_rc=None timeout=60s
Should Be Equal As Integers ${r_status.rc} 0
... plan status (final) failed (rc=${r_status.rc}): ${r_status.stderr}
Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback
Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL
Should Not Be Empty ${r_status.stdout} Final plan status output should not be empty
Output Should Contain ${r_status} ${plan_id}
# Parse phase from final status and assert non-empty (proves lifecycle events processed)
${final_phase}= Safe Parse Json Field ${r_status.stdout} phase
${final_state}= Safe Parse Json Field ${r_status.stdout} processing_state
Log Final phase=${final_phase} processing_state=${final_state}
${final_state_populated}= Evaluate $final_phase != '' or $final_state != ''
Should Be True ${final_state_populated}
... Final plan status should have non-empty phase or processing_state
# After a full lifecycle (execute + apply), phase or state should reflect completion
${final_lower}= Evaluate ($final_phase).lower() if $final_phase else ''
${state_lower}= Evaluate ($final_state).lower() if $final_state else ''
${is_terminal}= Evaluate 'apply' in $final_lower or 'complete' in $final_lower or 'done' in $final_lower or 'complete' in $state_lower or 'done' in $state_lower or 'applied' in $state_lower
Should Be True ${is_terminal}
... Final status should indicate a terminal/applied state (phase=${final_phase}, state=${final_state})
# ---- Final: plan should still appear in list ----
Verify Plan In List ${plan_id}
+525 -106
View File
@@ -1,145 +1,564 @@
*** Settings ***
Documentation E2E test for Workflow Example 5: Database Schema Migration with Safety Nets.
Documentation E2E test for Workflow Example 5: Database Schema Migration with Safety Nets.
...
... Advanced scenario using the **review** automation profile.
... Registers a custom resource type (postgres-db), creates custom
... skills with spec-aligned database tools (query_db, execute_migration,
... backfill_column), exercises phased child plan execution via
... ``plan tree``, attempts checkpoint-based rollback via
... ``plan rollback``, and verifies migration changes after apply.
... Advanced scenario using the **review** automation profile.
... Registers a custom resource type (postgres-db), creates custom
... skills with spec-aligned database tools (query_db, execute_migration,
... backfill_column), exercises phased child plan execution via
... ``plan tree``, attempts checkpoint-based rollback via
... ``plan rollback``, and verifies migration changes after apply.
...
... Zero mocking — real CLI, real LLM API keys.
Resource common_e2e.resource
Suite Setup WF05 Suite Setup
... Zero mocking — real CLI, real LLM API keys.
Resource common_e2e.resource
Suite Setup WF05 Suite Setup
Suite Teardown E2E Suite Teardown
Force Tags E2E
Force Tags E2E
*** Variables ***
${ACTION_NAME} local/wf05-db-migration
${SKILL_NAME} local/wf05-db-tools
${ACTION_NAME} local/wf05-db-migration
${SKILL_NAME} local/wf05-db-tools
${RESOURCE_TYPE_NAME} local/wf05-postgres-db
*** Keywords ***
WF05 Suite Setup
[Documentation] E2E Suite Setup plus dynamic actor selection.
E2E Suite Setup
# Generate unique suffix for resource/project names to avoid UNIQUE
# constraint collisions on repeated E2E runs against the same database.
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
# Probe the OpenAI API to confirm the key is usable before selecting actor.
# Falls back to Anthropic when the key is absent or quota-exhausted.
${actor}= Resolve LLM Actor
Set Suite Variable ${WF05_ACTOR} ${actor}
[Documentation] E2E Suite Setup plus dynamic actor selection.
E2E Suite Setup
# Generate unique suffix for resource/project names to avoid UNIQUE
# constraint collisions on repeated E2E runs against the same database.
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
# Pick an actor that matches available API keys.
# Prefer OpenAI first to reduce Anthropic credit-quota flakiness.
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
IF ${has_openai}
${actor}= Set Variable openai/gpt-4o
ELSE IF ${has_anthropic}
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
ELSE
${actor}= Set Variable openai/gpt-4o
END
Set Suite Variable ${WF05_ACTOR} ${actor}
Create DB App Repo
[Documentation] Create temp git repo with a Python app simulating
... a database schema and application code.
${repo}= Create Temp Git Repo wf05-db-app-${RUN_SUFFIX}
Create Directory ${repo}${/}src
Create Directory ${repo}${/}migrations
Create Directory ${repo}${/}tests
${schema_content}= Catenate SEPARATOR=\n
... """Database schema — users table (missing last_login_at)."""
... ${EMPTY}
... USERS_SCHEMA = {
... ${SPACE}${SPACE}${SPACE}${SPACE}"table": "users",
... ${SPACE}${SPACE}${SPACE}${SPACE}"columns": [
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "id", "type": "INTEGER", "primary_key": True},
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "email", "type": "VARCHAR(255)"},
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "created_at", "type": "TIMESTAMP"},
... ${SPACE}${SPACE}${SPACE}${SPACE}],
... }
Create File ${repo}${/}src${/}schema.py ${schema_content}
${app_content}= Catenate SEPARATOR=\n
... """Application code — user service."""
... from src.schema import USERS_SCHEMA
... ${EMPTY}
... ${EMPTY}
... def get_user(user_id):
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Fetch a user by ID."""
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"id": user_id, "email": "user@example.com"}
... ${EMPTY}
... ${EMPTY}
... def get_user_activity(user_id):
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Get user activity — needs last_login_at but column is missing."""
... ${SPACE}${SPACE}${SPACE}${SPACE}user = get_user(user_id)
... ${SPACE}${SPACE}${SPACE}${SPACE}# BUG: no last_login_at field available
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"user": user, "last_login": None}
Create File ${repo}${/}src${/}app.py ${app_content}
${test_content}= Catenate SEPARATOR=\n
... """Tests for user service."""
... from src.app import get_user, get_user_activity
... ${EMPTY}
... ${EMPTY}
... def test_get_user():
... ${SPACE}${SPACE}${SPACE}${SPACE}user = get_user(1)
... ${SPACE}${SPACE}${SPACE}${SPACE}assert user["id"] == 1
... ${EMPTY}
... ${EMPTY}
... def test_get_user_activity():
... ${SPACE}${SPACE}${SPACE}${SPACE}result = get_user_activity(1)
... ${SPACE}${SPACE}${SPACE}${SPACE}assert result["last_login"] is None
Create File ${repo}${/}tests${/}test_app.py ${test_content}
Create File ${repo}${/}src${/}__init__.py \n
Create File ${repo}${/}tests${/}__init__.py \n
Create File ${repo}${/}migrations${/}__init__.py \n
Create File ${repo}${/}requirements.txt pytest>=7.0\n
${git_add}= Run Process git add . cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
${git_commit}= Run Process git commit -m Initial DB app with missing last_login_at cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
RETURN ${repo}
[Documentation] Create temp git repo with a Python app simulating
... a database schema and application code.
${repo}= Create Temp Git Repo wf05-db-app-${RUN_SUFFIX}
Create Directory ${repo}${/}src
Create Directory ${repo}${/}migrations
Create Directory ${repo}${/}tests
${schema_content}= Catenate SEPARATOR=\n
... """Database schema — users table (missing last_login_at)."""
... ${EMPTY}
... USERS_SCHEMA = {
... ${SPACE}${SPACE}${SPACE}${SPACE}"table": "users",
... ${SPACE}${SPACE}${SPACE}${SPACE}"columns": [
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "id", "type": "INTEGER", "primary_key": True},
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "email", "type": "VARCHAR(255)"},
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}{"name": "created_at", "type": "TIMESTAMP"},
... ${SPACE}${SPACE}${SPACE}${SPACE}],
... }
Create File ${repo}${/}src${/}schema.py ${schema_content}
${app_content}= Catenate SEPARATOR=\n
... """Application code — user service."""
... from src.schema import USERS_SCHEMA
... ${EMPTY}
... ${EMPTY}
... def get_user(user_id):
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Fetch a user by ID."""
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"id": user_id, "email": "user@example.com"}
... ${EMPTY}
... ${EMPTY}
... def get_user_activity(user_id):
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Get user activity — needs last_login_at but column is missing."""
... ${SPACE}${SPACE}${SPACE}${SPACE}user = get_user(user_id)
... ${SPACE}${SPACE}${SPACE}${SPACE}# BUG: no last_login_at field available
... ${SPACE}${SPACE}${SPACE}${SPACE}return {"user": user, "last_login": None}
Create File ${repo}${/}src${/}app.py ${app_content}
${test_content}= Catenate SEPARATOR=\n
... """Tests for user service."""
... from src.app import get_user, get_user_activity
... ${EMPTY}
... ${EMPTY}
... def test_get_user():
... ${SPACE}${SPACE}${SPACE}${SPACE}user = get_user(1)
... ${SPACE}${SPACE}${SPACE}${SPACE}assert user["id"] == 1
... ${EMPTY}
... ${EMPTY}
... def test_get_user_activity():
... ${SPACE}${SPACE}${SPACE}${SPACE}result = get_user_activity(1)
... ${SPACE}${SPACE}${SPACE}${SPACE}assert result["last_login"] is None
Create File ${repo}${/}tests${/}test_app.py ${test_content}
Create File ${repo}${/}src${/}__init__.py \n
Create File ${repo}${/}tests${/}__init__.py \n
Create File ${repo}${/}migrations${/}__init__.py \n
Create File ${repo}${/}requirements.txt pytest>=7.0\n
${git_add}= Run Process git add . cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_add.rc} 0 git add failed: ${git_add.stderr}
${git_commit}= Run Process git commit -m Initial DB app with missing last_login_at cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_commit.rc} 0 git commit failed: ${git_commit.stderr}
RETURN ${repo}
WF05 Test Teardown
[Documentation] Log diagnostic context on failure for debugging.
${plan_id}= Get Variable Value ${WF05_PLAN_ID} ${EMPTY}
IF '${plan_id}' != ''
${status} ${result}= Run Keyword And Ignore Error
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
IF '${status}' == 'PASS'
Log Teardown plan status: ${result.stdout} WARN
END
${tree_status} ${tree_result}= Run Keyword And Ignore Error
... Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=30s
IF '${tree_status}' == 'PASS'
Log Teardown plan tree: ${tree_result.stdout} WARN
END
END
[Documentation] Log diagnostic context on failure for debugging.
${plan_id}= Get Variable Value ${WF05_PLAN_ID} ${EMPTY}
IF '${plan_id}' != ''
${status} ${result}= Run Keyword And Ignore Error
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
IF '${status}' == 'PASS'
Log Teardown plan status: ${result.stdout} WARN
END
${tree_status} ${tree_result}= Run Keyword And Ignore Error
... Run CleverAgents Command plan tree ${plan_id} --format json expected_rc=None timeout=30s
IF '${tree_status}' == 'PASS'
Log Teardown plan tree: ${tree_result.stdout} WARN
END
END
*** Test Cases ***
WF05 Database Schema Migration With Safety Nets Review Profile
[Documentation] Full review-profile workflow: register custom resource type,
... register git-checkout resource, create custom skill with
... spec-aligned DB tools (query_db, execute_migration,
... backfill_column), create migration action with review
[Tags] tdd_issue tdd_issue_4189
... automation profile, exercise phased child plan execution,
... attempt checkpoint-based rollback, verify migration changes.
[Timeout] 30 minutes
[Teardown] WF05 Test Teardown
Skip If No LLM Keys
# Initialise test variable for teardown access.
Set Test Variable ${WF05_PLAN_ID} ${EMPTY}
[Documentation] Full review-profile workflow: register custom resource type,
... register git-checkout resource, create custom skill with
... spec-aligned DB tools (query_db, execute_migration,
... backfill_column), create migration action with review
... automation profile, exercise phased child plan execution,
... attempt checkpoint-based rollback, verify migration changes.
[Timeout] 30 minutes
[Teardown] WF05 Test Teardown
Skip If No LLM Keys
# Initialise test variable for teardown access.
Set Test Variable ${WF05_PLAN_ID} ${EMPTY}
# ── 1. Fixture: create temp repo ──
${repo}= Create DB App Repo
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${branch_result.rc} 0 git rev-parse failed: ${branch_result.stderr}
${branch}= Strip String ${branch_result.stdout}
# ── 2. Register custom resource type (AC #2: custom resource type via CLI) ──
${res_type_yaml}= Catenate SEPARATOR=\n
... name: ${RESOURCE_TYPE_NAME}
... description: PostgreSQL database resource for WF05 E2E test
... resource_kind: physical
... sandbox_strategy: transaction_rollback
... user_addable: true
... cli_args:
... ${SPACE}${SPACE}- name: host
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Database hostname
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
... ${SPACE}${SPACE}- name: port
... ${SPACE}${SPACE}${SPACE}${SPACE}required: false
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Port (default 5432)
... ${SPACE}${SPACE}${SPACE}${SPACE}type: integer
... ${SPACE}${SPACE}${SPACE}${SPACE}default: 5432
... ${SPACE}${SPACE}- name: database
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Database name
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
... ${SPACE}${SPACE}- name: schema
... ${SPACE}${SPACE}${SPACE}${SPACE}required: false
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Schema (default public)
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
... ${SPACE}${SPACE}${SPACE}${SPACE}default: public
... capabilities:
... ${SPACE}${SPACE}read: true
... ${SPACE}${SPACE}write: true
... ${SPACE}${SPACE}sandbox: true
... ${SPACE}${SPACE}checkpoint: true
${res_type_path}= Set Variable ${SUITE_HOME}${/}wf05_resource_type.yaml
Create File ${res_type_path} ${res_type_yaml}
${r_type}= Run CleverAgents Command
... resource type add --config ${res_type_path} --format json
... expected_rc=None
Should Be Equal As Integers ${r_type.rc} 0
... resource type add failed (rc=${r_type.rc}): ${r_type.stderr}
Should Not Contain ${r_type.stdout}${r_type.stderr} Traceback
Should Not Contain ${r_type.stdout}${r_type.stderr} INTERNAL
Output Should Contain ${r_type} ${RESOURCE_TYPE_NAME}
# ── 3. Register git-checkout resource and create project ──
${res_name}= Set Variable wf05-res-${RUN_SUFFIX}
${r_res}= Run CleverAgents Command
... resource add git-checkout ${res_name}
... --path ${repo} --branch ${branch}
Should Be Equal As Integers ${r_res.rc} 0
... resource add failed (rc=${r_res.rc}): ${r_res.stderr}
Should Not Contain ${r_res.stdout}${r_res.stderr} Traceback
Should Not Contain ${r_res.stdout}${r_res.stderr} INTERNAL
Output Should Contain ${r_res} ${res_name}
${proj_name}= Set Variable wf05-proj-${RUN_SUFFIX}
${r_proj}= Run CleverAgents Command
... project create --resource ${res_name} ${proj_name}
Should Be Equal As Integers ${r_proj.rc} 0
... project create failed (rc=${r_proj.rc}): ${r_proj.stderr}
Should Not Contain ${r_proj.stdout}${r_proj.stderr} Traceback
Should Not Contain ${r_proj.stdout}${r_proj.stderr} INTERNAL
Output Should Contain ${r_proj} ${proj_name}
# ── 4. Instantiate custom postgres-db resource and link to project ──
${db_res_name}= Set Variable wf05-db-${RUN_SUFFIX}
${r_db_res}= Run CleverAgents Command
... resource add ${RESOURCE_TYPE_NAME} ${db_res_name}
... --host localhost --database wf05_test_db
... expected_rc=None
IF ${r_db_res.rc} == 0
Output Should Contain ${r_db_res} ${db_res_name}
Log Custom postgres-db resource instance created: ${db_res_name}
# Link custom DB resource to project (spec Step 1: project link-resource)
${r_link_db}= Run CleverAgents Command
... project link-resource ${proj_name} ${db_res_name}
... expected_rc=None
IF ${r_link_db.rc} == 0
Log Custom DB resource linked to project: ${db_res_name}
ELSE
${link_err}= Set Variable ${r_link_db.stdout}${r_link_db.stderr}
Should Not Contain ${link_err} Traceback
${link_has_no_such_option}= Evaluate 'No such option' in $link_err or 'NoSuchOption' in $link_err
IF not ${link_has_no_such_option}
Should Not Contain ${link_err} INTERNAL
END
Log project link-resource returned rc=${r_link_db.rc} (may require additional CLI support) WARN
END
ELSE
${db_err}= Set Variable ${r_db_res.stdout}${r_db_res.stderr}
Should Not Contain ${db_err} Traceback
${db_has_no_such_option}= Evaluate 'No such option' in $db_err or 'NoSuchOption' in $db_err
IF not ${db_has_no_such_option}
Should Not Contain ${db_err} INTERNAL
END
Log Custom resource instantiation returned rc=${r_db_res.rc} (may require additional CLI support) WARN
END
# ── 5. Create custom skill with spec-aligned DB tools (AC #3) ──
# Note: writes/checkpointable are schema-validated at registration time.
# The persisted Skill model currently stores tool refs by namespaced name.
${skill_yaml}= Catenate SEPARATOR=\n
... name: ${SKILL_NAME}
... description: Safe database operations with transaction support
... tools:
... ${SPACE}${SPACE}- name: local/query_db
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Execute a read-only SQL query and return results
... ${SPACE}${SPACE}${SPACE}${SPACE}writes: false
... ${SPACE}${SPACE}${SPACE}${SPACE}checkpointable: false
... ${SPACE}${SPACE}- name: local/execute_migration
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Execute a DDL migration within a transaction
... ${SPACE}${SPACE}${SPACE}${SPACE}writes: true
... ${SPACE}${SPACE}${SPACE}${SPACE}checkpointable: true
... ${SPACE}${SPACE}- name: local/backfill_column
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Batch-update a column using a source query
... ${SPACE}${SPACE}${SPACE}${SPACE}writes: true
... ${SPACE}${SPACE}${SPACE}${SPACE}checkpointable: true
${skill_path}= Set Variable ${SUITE_HOME}${/}wf05_skill.yaml
Create File ${skill_path} ${skill_yaml}
${r_skill}= Run CleverAgents Command
... skill add --config ${skill_path} --format json
... expected_rc=None
Should Be Equal As Integers ${r_skill.rc} 0
... skill add failed (rc=${r_skill.rc}): ${r_skill.stderr}
Should Not Contain ${r_skill.stdout}${r_skill.stderr} Traceback
Should Not Contain ${r_skill.stdout}${r_skill.stderr} INTERNAL
Output Should Contain ${r_skill} ${SKILL_NAME}
# Verify individual tool names are registered
Output Should Contain ${r_skill} local/query_db
Output Should Contain ${r_skill} local/execute_migration
Output Should Contain ${r_skill} local/backfill_column
# ── 6. Create action with review automation profile ──
${action_yaml}= Catenate SEPARATOR=\n
... name: ${ACTION_NAME}
... description: Add last_login_at column to users table with backfill and app update
... definition_of_done: |
... ${SPACE}${SPACE}Database migration adds the column with a sensible default.
... ${SPACE}${SPACE}Backfill completes for all rows using the specified source.
... ${SPACE}${SPACE}Application code reads/writes the new column.
... ${SPACE}${SPACE}All tests pass with the new schema.
... ${SPACE}${SPACE}A rollback migration is available and tested.
... strategy_actor: ${WF05_ACTOR}
... execution_actor: ${WF05_ACTOR}
... automation_profile: review
... reusable: true
... state: available
... invariants:
... ${SPACE}${SPACE}- Migration must be backward-compatible (add column, don't rename or drop)
... ${SPACE}${SPACE}- Backfill must be batched to avoid locking the table
... ${SPACE}${SPACE}- Rollback migration must be provided and tested
... ${SPACE}${SPACE}- Application code must handle both old (null) and new values gracefully
... arguments:
... ${SPACE}${SPACE}- name: table_name
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Target table for schema migration
... ${SPACE}${SPACE}- name: column_name
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
... ${SPACE}${SPACE}${SPACE}${SPACE}description: New column to add
... ${SPACE}${SPACE}- name: column_type
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
... ${SPACE}${SPACE}${SPACE}${SPACE}description: SQL type for the new column
... ${SPACE}${SPACE}- name: backfill_source
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Where to source backfill data
${action_path}= Set Variable ${SUITE_HOME}${/}wf05_action.yaml
Create File ${action_path} ${action_yaml}
${r_action}= Run CleverAgents Command
... action create --config ${action_path} --format json
... expected_rc=None
Should Be Equal As Integers ${r_action.rc} 0
... action create failed (rc=${r_action.rc}): ${r_action.stderr}
Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback
Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL
Output Should Contain ${r_action} ${ACTION_NAME}
# ── 7. Plan use — create plan from action + project with args ──
${r_use}= Run CleverAgents Command
... plan use ${ACTION_NAME} ${proj_name}
... --arg table_name=users --arg column_name=last_login_at
... --arg column_type=TIMESTAMP --arg backfill_source=audit_log
... --automation-profile review
... --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${r_use.rc} 0
... plan use failed (rc=${r_use.rc}): ${r_use.stderr}
Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback
Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL
${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id
Should Not Be Empty ${plan_id} Could not extract plan_id from plan use output
Set Test Variable ${WF05_PLAN_ID} ${plan_id}
Log Plan ID: ${plan_id}
# Verify review automation profile was applied.
# If plan use output omits automation_profile, fall back to plan status.
${resolved_profile}= Safe Parse Json Field ${r_use.stdout} automation_profile
${resolved_profile}= Set Variable If $resolved_profile is None ${EMPTY} ${resolved_profile}
${resolved_profile}= Set Variable If '${resolved_profile}' == 'None' ${EMPTY} ${resolved_profile}
IF '${resolved_profile}' == ''
${r_profile_status}= Run CleverAgents Command
... plan status ${plan_id} --format json
... expected_rc=None timeout=120s
Should Be Equal As Integers ${r_profile_status.rc} 0
... plan status for profile verification failed (rc=${r_profile_status.rc}): ${r_profile_status.stderr}
Should Not Contain ${r_profile_status.stdout}${r_profile_status.stderr} Traceback
Should Not Contain ${r_profile_status.stdout}${r_profile_status.stderr} INTERNAL
${resolved_profile}= Safe Parse Json Field ${r_profile_status.stdout} automation_profile
${resolved_profile}= Set Variable If $resolved_profile is None ${EMPTY} ${resolved_profile}
${resolved_profile}= Set Variable If '${resolved_profile}' == 'None' ${EMPTY} ${resolved_profile}
END
Should Be Equal As Strings ${resolved_profile} review
... Expected automation_profile 'review' but got '${resolved_profile}'
# ── 8. Strategize ──
${r_strat}= Run CleverAgents Command
... plan execute ${plan_id} --format json
... expected_rc=None timeout=300s
Should Be Equal As Integers ${r_strat.rc} 0
... plan execute (strategize) failed (rc=${r_strat.rc}): ${r_strat.stderr}
Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback
Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL
Output Should Contain ${r_strat} ${plan_id}
Log Strategize stdout: ${r_strat.stdout}
# ── 9. Decision tree — verify phased child plans (AC #4) ──
${r_tree}= Run CleverAgents Command
... plan tree ${plan_id} --format json
... expected_rc=None timeout=120s
Should Be Equal As Integers ${r_tree.rc} 0
... plan tree failed (rc=${r_tree.rc}): ${r_tree.stderr}
Should Not Contain ${r_tree.stdout}${r_tree.stderr} Traceback
Should Not Contain ${r_tree.stdout}${r_tree.stderr} INTERNAL
Should Not Be Empty ${r_tree.stdout} Plan tree output should not be empty
# Parse JSON and assert structural evidence of phased decomposition.
# Extract JSON From Stdout unwraps the CLI envelope automatically, returning the data payload.
${tree_json}= Extract JSON From Stdout ${r_tree.stdout}
${decision_count}= Evaluate
... (lambda root: (lambda walk: walk(walk, root))(lambda self, node: (1 if isinstance(node, dict) and str(node.get('decision_id', '')).strip() != '' else 0) + sum(self(self, child) for child in ((node.get('children') if isinstance(node, dict) else []) or [])) + (sum(self(self, item) for item in node) if isinstance(node, list) else 0)))($tree_json)
Log Decision tree contains ${decision_count} decision node(s)
Should Be True ${decision_count} >= 2
... Plan tree should contain at least 2 decision nodes (found ${decision_count})
${children_key_count}= Evaluate __import__('json').dumps($tree_json).count('"children"')
Should Be True ${children_key_count} >= 1
... Plan tree should include structured children fields
${child_link_count}= Evaluate
... (lambda root: (lambda walk: walk(walk, root))(lambda self, node: ((len((node.get('children') if isinstance(node, dict) else []) or [])) if isinstance(node, dict) else 0) + sum(self(self, child) for child in ((node.get('children') if isinstance(node, dict) else []) or [])) + (sum(self(self, item) for item in node) if isinstance(node, list) else 0)))($tree_json)
IF ${decision_count} < 3
Log Decision count (${decision_count}) is below 3 — decomposition may be minimal in this run WARN
END
IF ${child_link_count} < 1
Log Decision tree has no child links in this run — phased decomposition may be minimal WARN
END
# ── 10. Execute ──
${r_exec}= Run CleverAgents Command
... plan execute ${plan_id} --format json
... expected_rc=None timeout=300s
IF ${r_exec.rc} != 0
Fail plan execute failed (rc=${r_exec.rc}) stdout=${r_exec.stdout} stderr=${r_exec.stderr}
END
Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback
Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL
Output Should Contain ${r_exec} ${plan_id}
Log Execute stdout: ${r_exec.stdout}
# ── 11. Plan status — verify phase and check for checkpoints ──
${r_status}= Run CleverAgents Command
... plan status ${plan_id} --format json
... expected_rc=None timeout=120s
Should Be Equal As Integers ${r_status.rc} 0
... plan status failed (rc=${r_status.rc}): ${r_status.stderr}
Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback
Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL
Should Not Be Empty ${r_status.stdout}
Output Should Contain ${r_status} ${plan_id}
# Assert phase field is populated after execute
${status_phase}= Safe Parse Json Field ${r_status.stdout} phase
Should Not Be Empty ${status_phase} Plan phase should be populated after execute
Log Plan phase after execute: ${status_phase}
# Extract checkpoint ID if available (for rollback attempt)
${checkpoint_id}= Safe Parse Json Field ${r_status.stdout} last_checkpoint_id
# Guard against Python None being returned as string "None" from JSON null
${checkpoint_id}= Set Variable If $checkpoint_id is None ${EMPTY} ${checkpoint_id}
${checkpoint_id}= Set Variable If '${checkpoint_id}' == 'None' ${EMPTY} ${checkpoint_id}
Log Checkpoint ID from status: ${checkpoint_id}
# ── 12. Checkpoint rollback attempt (AC #5) ──
IF '${checkpoint_id}' != ''
Log Attempting plan rollback to checkpoint ${checkpoint_id}
${r_rollback}= Run CleverAgents Command
... plan rollback --yes ${plan_id} ${checkpoint_id}
... --format json expected_rc=None timeout=120s
Log Rollback rc=${r_rollback.rc} stdout=${r_rollback.stdout} stderr=${r_rollback.stderr}
Should Be Equal As Integers ${r_rollback.rc} 0
... plan rollback to checkpoint ${checkpoint_id} failed (rc=${r_rollback.rc}): ${r_rollback.stderr}
Should Not Contain ${r_rollback.stdout}${r_rollback.stderr} Traceback
Should Not Contain ${r_rollback.stdout}${r_rollback.stderr} INTERNAL
Output Should Contain ${r_rollback} ${plan_id}
Log Rollback succeeded — re-executing plan to continue lifecycle
# Re-execute after rollback so we can proceed to diff/apply
${r_reexec}= Run CleverAgents Command
... plan execute ${plan_id} --format json
... expected_rc=None timeout=300s
Log Re-execute after rollback rc=${r_reexec.rc}
IF ${r_reexec.rc} != 0
# Re-execute failure is non-fatal: the plan may still be in an applicable
# state from the original execute, so we continue to diff/apply rather
# than aborting. If those steps also fail the test will still surface a
# clear failure at the relevant assertion.
Log Re-execute after rollback failed (rc=${r_reexec.rc}); continuing to diff/apply WARN
ELSE
Should Not Contain ${r_reexec.stdout}${r_reexec.stderr} Traceback
Should Not Contain ${r_reexec.stdout}${r_reexec.stderr} INTERNAL
END
ELSE
Log AC #5 visibility: no checkpoint_id present, so real rollback path was not executed in this run WARN
END
# Also validate graceful failure path on a fake checkpoint ID.
${r_rollback_noop}= Run CleverAgents Command
... plan rollback --yes ${plan_id} no-such-checkpoint
... --format json expected_rc=None timeout=60s
Log Rollback (no checkpoint) rc=${r_rollback_noop.rc} stderr=${r_rollback_noop.stderr}
Should Not Be Equal As Integers ${r_rollback_noop.rc} 0
... plan rollback with fake checkpoint should fail gracefully (got rc=0)
Should Not Contain ${r_rollback_noop.stdout}${r_rollback_noop.stderr} Traceback
Should Not Contain ${r_rollback_noop.stdout}${r_rollback_noop.stderr} INTERNAL
# ── 13. Diff — verify changeset is non-empty (AC #6: migration verification) ──
${r_diff}= Run CleverAgents Command
... plan diff ${plan_id} --format json
... expected_rc=None timeout=120s
Should Be Equal As Integers ${r_diff.rc} 0
... plan diff failed (rc=${r_diff.rc}): ${r_diff.stderr}
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
Should Not Be Empty ${r_diff.stdout} Plan diff produced no output
${diff_preview_lower}= Evaluate ($r_diff.stdout).lower()
${has_diff_signal}= Evaluate 'migration' in $diff_preview_lower or 'backfill' in $diff_preview_lower or 'last_login' in $diff_preview_lower or 'column' in $diff_preview_lower or 'audit_log' in $diff_preview_lower or 'diff' in $diff_preview_lower or 'change' in $diff_preview_lower or 'file' in $diff_preview_lower
Should Be True ${has_diff_signal}
... plan diff output should include meaningful diff/change indicators before apply
# ── 14. Apply ──
# Save baseline SHA before apply for accurate diff comparison
${baseline_sha_result}= Run Process git rev-parse HEAD cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${baseline_sha_result.rc} 0
... git rev-parse HEAD failed: ${baseline_sha_result.stderr}
${baseline_sha}= Strip String ${baseline_sha_result.stdout}
Log Baseline SHA before apply: ${baseline_sha}
${r_apply}= Run CleverAgents Command
... plan apply --yes ${plan_id} --format json
... expected_rc=None timeout=180s
Log Apply rc=${r_apply.rc} stdout=${r_apply.stdout}
IF ${r_apply.rc} == 0
Output Should Contain ${r_apply} ${plan_id}
ELSE
Fail apply failed (rc=${r_apply.rc}) stdout=${r_apply.stdout} stderr=${r_apply.stderr}
END
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
# Verify terminal state after apply.
${r_status_after_apply}= Run CleverAgents Command
... plan status ${plan_id} --format json
... expected_rc=None timeout=120s
Should Be Equal As Integers ${r_status_after_apply.rc} 0
... plan status after apply failed (rc=${r_status_after_apply.rc}): ${r_status_after_apply.stderr}
Should Not Contain ${r_status_after_apply.stdout}${r_status_after_apply.stderr} Traceback
Should Not Contain ${r_status_after_apply.stdout}${r_status_after_apply.stderr} INTERNAL
${apply_phase}= Safe Parse Json Field ${r_status_after_apply.stdout} phase
${apply_state}= Safe Parse Json Field ${r_status_after_apply.stdout} processing_state
Should Not Be Empty ${apply_phase} plan status after apply should include phase
Should Not Be Empty ${apply_state} plan status after apply should include processing_state
${apply_phase_lower}= Evaluate ($apply_phase).lower()
IF 'apply' not in $apply_phase_lower
Log Post-apply phase is '${apply_phase}' instead of apply; treating terminal state as authoritative WARN
END
${is_terminal_state}= Evaluate ($apply_state.lower() in ['applied', 'constrained', 'errored', 'cancelled', 'complete'])
${is_apply_progress_state}= Evaluate ('apply' in $apply_phase_lower)
Should Be True ${is_terminal_state} or ${is_apply_progress_state}
... expected apply to produce terminal state or apply-phase progress (phase=${apply_phase}, state=${apply_state})
IF not ${is_terminal_state}
Log Post-apply state '${apply_state}' is non-terminal; apply may complete asynchronously WARN
END
# ── 15. Verify repo state and migration content (AC #6) ──
${log_result}= Run Process git log --oneline -10 cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${log_result.rc} 0
... git log failed (rc=${log_result.rc}): ${log_result.stderr}
Log Git log: ${log_result.stdout}
${line_count}= Get Line Count ${log_result.stdout}
# Hard assert minimum fixture commits exist; LLM-generated commits are
# non-deterministic, so we only hard-assert the fixture baseline (2 commits:
# Create Temp Git Repo initial + DB app fixture).
Should Be True ${line_count} >= 2
... Expected at least 2 commits (fixture baseline), got ${line_count}
IF ${line_count} < 3
Log No new commits from apply (${line_count} total) — LLM may not have produced file changes WARN
END
# Diff against baseline SHA to capture only changes from apply
${diff_all}= Run Process git diff ${baseline_sha} HEAD --name-only cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${diff_all.rc} 0
... git diff --name-only failed (rc=${diff_all.rc}): ${diff_all.stderr}
Log Files changed since apply: ${diff_all.stdout}
${full_diff}= Run Process git diff ${baseline_sha} HEAD cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${full_diff.rc} 0
... git diff failed (rc=${full_diff.rc}): ${full_diff.stderr}
${migration_evidence_text}= Evaluate ($full_diff.stdout + ' ' + $r_diff.stdout).lower()
${has_migration_content}= Evaluate 'last_login' in $migration_evidence_text or 'schema' in $migration_evidence_text or 'migration' in $migration_evidence_text or 'column' in $migration_evidence_text or 'alter' in $migration_evidence_text
IF not ${has_migration_content}
Log AC #6 visibility: no migration keywords found in diff/apply outputs for this run WARN
END
# Check for backfill-related evidence in plan/decomposition outputs (AC #6).
${combined_output_lower}= Evaluate ($r_tree.stdout + ' ' + $r_exec.stdout + ' ' + $r_diff.stdout).lower()
${has_backfill_evidence}= Evaluate 'backfill' in $combined_output_lower or 'batch' in $combined_output_lower or 'populate' in $combined_output_lower or 'last_login' in $combined_output_lower
IF not ${has_backfill_evidence}
Log AC #6 visibility: no backfill keywords found in plan/decomposition outputs for this run WARN
END
${has_ac6_evidence}= Evaluate ${has_migration_content} or ${has_backfill_evidence}
IF not ${has_ac6_evidence}
Log AC #6 visibility: neither migration nor backfill evidence was observed in this run (output is flexible) WARN
END
Log WF05 Database Schema Migration E2E test completed successfully
+311 -299
View File
@@ -1,325 +1,337 @@
*** Settings ***
Documentation E2E test for Workflow Example 7: CI/CD integration,
... automated PR review and fix (ci profile).
Documentation E2E test for Workflow Example 7: CI/CD integration,
... automated PR review and fix (ci profile).
...
... Exercises the real CleverAgents CLI with zero mocking,
... covering ci-profile configuration, idempotent resource and
... project registration, validation registration with project
... attachment, CI plan launch with action args and JSON output,
... polling for plan completion, and JSON output verification.
... Exercises the real CleverAgents CLI with zero mocking,
... covering ci-profile configuration, idempotent resource and
... project registration, validation registration with project
... attachment, CI plan launch with action args and JSON output,
... polling for plan completion, and JSON output verification.
...
... Tests form an implicit sequential pipeline via shared
... database state: each test creates CLI entities consumed
... by later tests. Execution order is file order.
Resource common_e2e.resource
Suite Setup WF07 Suite Setup
... Tests form an implicit sequential pipeline via shared
... database state: each test creates CLI entities consumed
... by later tests. Execution order is file order.
Resource common_e2e.resource
Suite Setup WF07 Suite Setup
Suite Teardown E2E Suite Teardown
Force Tags E2E
Force Tags E2E
*** Variables ***
${CI_PROJECT} local/ci-workspace
${CI_RESOURCE} local/ci-main
${CI_LINT_VAL} local/ci-lint
${CI_TYPECHECK_VAL} local/ci-typecheck
${CI_TESTS_VAL} local/ci-tests
${CI_PROJECT} local/ci-workspace
${CI_RESOURCE} local/ci-main
${CI_LINT_VAL} local/ci-lint
${CI_TYPECHECK_VAL} local/ci-typecheck
${CI_TESTS_VAL} local/ci-tests
*** Test Cases ***
WF07 E2E CI Profile Configuration
[Documentation] Set and verify the ci automation profile, json format,
... and log level per specification Step 1.
[Tags] E2E tdd_issue tdd_issue_4196
[Teardown] Log CI Profile Configuration teardown complete
# Automation profile
Run CleverAgents Command config set core.automation-profile ci
${get_profile}= Run CleverAgents Command config get core.automation-profile --format plain
# Match the value line in plain-text output (key/value/source/type block).
# Regex avoids false positives from debug lines containing "ci" substrings.
Should Match Regexp ${get_profile.stdout} (?m)^value:\\s+ci\\s*$
... Expected value line 'value: ci' in config output: ${get_profile.stdout}
# Output format
Run CleverAgents Command config set core.format json
${get_format}= Run CleverAgents Command config get core.format --format plain
Should Match Regexp ${get_format.stdout} (?m)^value:\\s+json\\s*$
... Expected value line 'value: json' in config output: ${get_format.stdout}
# Log level (spec Step 1)
Run CleverAgents Command config set core.log.level WARN
${get_log}= Run CleverAgents Command config get core.log.level --format plain
Should Match Regexp ${get_log.stdout} (?m)^value:\\s+WARN\\s*$
... Expected value line 'value: WARN' in config output: ${get_log.stdout}
[Documentation] Set and verify the ci automation profile, json format,
... and log level per specification Step 1.
[Tags] E2E
[Teardown] Log CI Profile Configuration teardown complete
# Automation profile
Run CleverAgents Command config set core.automation-profile ci
${get_profile}= Run CleverAgents Command config get core.automation-profile --format plain
# Match the value line in plain-text output (key/value/source/type block).
# Regex avoids false positives from debug lines containing "ci" substrings.
Should Match Regexp ${get_profile.stdout} (?m)^value:\\s+ci\\s*$
... Expected value line 'value: ci' in config output: ${get_profile.stdout}
# Output format
Run CleverAgents Command config set core.format json
${get_format}= Run CleverAgents Command config get core.format --format plain
Should Match Regexp ${get_format.stdout} (?m)^value:\\s+json\\s*$
... Expected value line 'value: json' in config output: ${get_format.stdout}
# Log level (spec Step 1)
Run CleverAgents Command config set core.log.level WARN
${get_log}= Run CleverAgents Command config get core.log.level --format plain
Should Match Regexp ${get_log.stdout} (?m)^value:\\s+WARN\\s*$
... Expected value line 'value: WARN' in config output: ${get_log.stdout}
WF07 E2E Idempotent Resource Registration
[Documentation] Register a git-checkout resource twice with --path and
... --branch, and verify it appears in the resource list.
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
[Teardown] Log Idempotent Resource Registration teardown complete
${repo_dir}= Create Temp Git Repo With Issues ci-repo
# First registration (--branch main per spec Step 3).
Run CleverAgents Command resource add git-checkout ${CI_RESOURCE}
... --path ${repo_dir} --branch main expected_rc=${0}
# Second registration (idempotent — may succeed or fail gracefully)
Run CleverAgents Command resource add git-checkout ${CI_RESOURCE}
... --path ${repo_dir} --branch main expected_rc=None
# Verify it appears in the list
${list_out}= Run CleverAgents Command resource list --format plain
Output Should Contain ${list_out} ci-main
# Verify exactly one occurrence (idempotency: two adds must not duplicate)
${count}= Get Count ${list_out.stdout} ci-main
Should Be True ${count} == 1 Resource ci-main should appear exactly once (idempotency check)
[Documentation] Register a git-checkout resource twice with --path and
... --branch, and verify it appears in the resource list.
[Tags] E2E
[Teardown] Log Idempotent Resource Registration teardown complete
${repo_dir}= Create Temp Git Repo With Issues ci-repo
# First registration (--branch main per spec Step 3).
Run CleverAgents Command resource add git-checkout ${CI_RESOURCE}
... --path ${repo_dir} --branch main expected_rc=${0}
# Second registration (idempotent — may succeed or fail gracefully)
Run CleverAgents Command resource add git-checkout ${CI_RESOURCE}
... --path ${repo_dir} --branch main expected_rc=None
# Verify it appears in the list
${list_out}= Run CleverAgents Command resource list --format plain
Output Should Contain ${list_out} ci-main
# Verify exactly one occurrence (idempotency: two adds must not duplicate)
${count}= Get Count ${list_out.stdout} ci-main
Should Be True ${count} == 1 Resource ci-main should appear exactly once (idempotency check)
WF07 E2E Idempotent Project Registration
[Documentation] Create a project linked to the resource twice and verify
... it appears in the project list (idempotent).
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
[Teardown] Log Idempotent Project Registration teardown complete
# First creation with resource link and description (per spec Step 3)
Run CleverAgents Command project create ${CI_PROJECT}
... --description CI workspace for automated PR review
... --resource ${CI_RESOURCE} expected_rc=${0}
# Second creation (idempotent — may succeed or fail gracefully)
Run CleverAgents Command project create ${CI_PROJECT}
... --description CI workspace for automated PR review
... --resource ${CI_RESOURCE} expected_rc=None
${list_out}= Run CleverAgents Command project list --format plain
Output Should Contain ${list_out} ci-workspace
# Verify exactly one occurrence (idempotency: two creates must not duplicate).
# Use the full namespaced name because plain-text output includes both
# namespaced_name and name fields per entry — bare "ci-workspace" matches twice.
${count}= Get Count ${list_out.stdout} ${CI_PROJECT}
Should Be True ${count} == 1 Project ${CI_PROJECT} should appear exactly once (idempotency check)
[Documentation] Create a project linked to the resource twice and verify
... it appears in the project list (idempotent).
[Tags] E2E
[Teardown] Log Idempotent Project Registration teardown complete
# First creation with resource link and description (per spec Step 3)
Run CleverAgents Command project create ${CI_PROJECT}
... --description 'CI workspace for automated PR review'
... --resource ${CI_RESOURCE} expected_rc=${0}
# Second creation (idempotent — may succeed or fail gracefully)
Run CleverAgents Command project create ${CI_PROJECT}
... --description 'CI workspace for automated PR review'
... --resource ${CI_RESOURCE} expected_rc=None
${list_out}= Run CleverAgents Command project list --format plain
Output Should Contain ${list_out} ci-workspace
# Verify exactly one occurrence (idempotency: two creates must not duplicate).
# Use the full namespaced name because plain-text output includes both
# namespaced_name and name fields per entry — bare "ci-workspace" matches twice.
${count}= Get Count ${list_out.stdout} ${CI_PROJECT}
Should Be True ${count} == 1 Project ${CI_PROJECT} should appear exactly once (idempotency check)
WF07 E2E Validation Registration
[Documentation] Register lint, typecheck, and test validations, attach
... them to the project, and verify they appear in the tool
... list and project show output.
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
[Teardown] Log Validation Registration teardown complete
# --- Lint validation ---
${lint_yaml}= Catenate SEPARATOR=\n
... name: ${CI_LINT_VAL}
... description: Lint check validation for CI
... source: custom
... mode: required
... code: |
... ${SPACE}${SPACE}def run(inputs):
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "lint check passed"}
${lint_path}= Set Variable ${SUITE_HOME}${/}ci-lint.yaml
Create File ${lint_path} ${lint_yaml}
Run CleverAgents Command validation add --config ${lint_path} expected_rc=None
# --- Typecheck validation ---
${type_yaml}= Catenate SEPARATOR=\n
... name: ${CI_TYPECHECK_VAL}
... description: Type check validation for CI
... source: custom
... mode: required
... code: |
... ${SPACE}${SPACE}def run(inputs):
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "typecheck passed"}
${type_path}= Set Variable ${SUITE_HOME}${/}ci-typecheck.yaml
Create File ${type_path} ${type_yaml}
Run CleverAgents Command validation add --config ${type_path} expected_rc=None
# --- Test validation ---
${test_yaml}= Catenate SEPARATOR=\n
... name: ${CI_TESTS_VAL}
... description: Test runner validation for CI
... source: custom
... mode: required
... code: |
... ${SPACE}${SPACE}def run(inputs):
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "tests passed"}
${test_path}= Set Variable ${SUITE_HOME}${/}ci-tests.yaml
Create File ${test_path} ${test_yaml}
Run CleverAgents Command validation add --config ${test_path} expected_rc=None
# --- Attach validations to project (spec Step 3) ---
Run CleverAgents Command validation attach
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_LINT_VAL} expected_rc=None
Run CleverAgents Command validation attach
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_TYPECHECK_VAL} expected_rc=None
Run CleverAgents Command validation attach
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_TESTS_VAL} expected_rc=None
# Verify validations appear in tool list
${list_out}= Run CleverAgents Command tool list --type validation --format plain
Output Should Contain ${list_out} ci-lint
Output Should Contain ${list_out} ci-typecheck
Output Should Contain ${list_out} ci-tests
# Verify the project itself is accessible (project show does not display
# attached validations — validation registration is verified by tool list above)
${project_out}= Run CleverAgents Command project show ${CI_PROJECT}
... --format plain
Log Project show output: ${project_out.stdout}
Output Should Contain ${project_out} ci-workspace
[Documentation] Register lint, typecheck, and test validations, attach
... them to the project, and verify they appear in the tool
... list and project show output.
[Tags] E2E
[Teardown] Log Validation Registration teardown complete
# --- Lint validation ---
${lint_yaml}= Catenate SEPARATOR=\n
... name: ${CI_LINT_VAL}
... description: Lint check validation for CI
... source: custom
... mode: required
... code: |
... ${SPACE}${SPACE}def run(inputs):
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "lint check passed"}
${lint_path}= Set Variable ${SUITE_HOME}${/}ci-lint.yaml
Create File ${lint_path} ${lint_yaml}
Run CleverAgents Command validation add --config ${lint_path} expected_rc=None
# --- Typecheck validation ---
${type_yaml}= Catenate SEPARATOR=\n
... name: ${CI_TYPECHECK_VAL}
... description: Type check validation for CI
... source: custom
... mode: required
... code: |
... ${SPACE}${SPACE}def run(inputs):
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "typecheck passed"}
${type_path}= Set Variable ${SUITE_HOME}${/}ci-typecheck.yaml
Create File ${type_path} ${type_yaml}
Run CleverAgents Command validation add --config ${type_path} expected_rc=None
# --- Test validation ---
${test_yaml}= Catenate SEPARATOR=\n
... name: ${CI_TESTS_VAL}
... description: Test runner validation for CI
... source: custom
... mode: required
... code: |
... ${SPACE}${SPACE}def run(inputs):
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"passed": True, "message": "tests passed"}
${test_path}= Set Variable ${SUITE_HOME}${/}ci-tests.yaml
Create File ${test_path} ${test_yaml}
Run CleverAgents Command validation add --config ${test_path} expected_rc=None
# --- Attach validations to project (spec Step 3) ---
Run CleverAgents Command validation attach
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_LINT_VAL} expected_rc=None
Run CleverAgents Command validation attach
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_TYPECHECK_VAL} expected_rc=None
Run CleverAgents Command validation attach
... --project ${CI_PROJECT} ${CI_RESOURCE} ${CI_TESTS_VAL} expected_rc=None
# Verify validations appear in tool list
${list_out}= Run CleverAgents Command tool list --type validation --format plain
Output Should Contain ${list_out} ci-lint
Output Should Contain ${list_out} ci-typecheck
Output Should Contain ${list_out} ci-tests
# Verify the project itself is accessible (project show does not display
# attached validations — validation registration is verified by tool list above)
${project_out}= Run CleverAgents Command project show ${CI_PROJECT}
... --format plain
Log Project show output: ${project_out.stdout}
Output Should Contain ${project_out} ci-workspace
WF07 E2E CI Plan Launch
[Documentation] Create an action and launch a plan with --automation-profile ci,
... --format json, and --arg flags per specification Steps 2-3.
... Polls for plan completion. Requires LLM API keys.
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
[Teardown] Log CI Plan Launch teardown complete
Skip If No LLM Keys
# Probe the OpenAI API to confirm the key is usable before selecting actor.
# Falls back to Anthropic when the key is absent or quota-exhausted.
${actor}= Resolve LLM Actor
# Create action with spec-aligned fields and dynamic actor
${action_yaml}= Catenate SEPARATOR=\n
... name: local/review-pr
... description: Automatically review a PR and fix issues
... strategy_actor: ${actor}
... execution_actor: ${actor}
... automation_profile: ci
... reusable: true
... state: available
... definition_of_done: |
... ${SPACE}${SPACE}- All lint issues are resolved
... ${SPACE}${SPACE}- Type checking passes
... ${SPACE}${SPACE}- Test coverage does not decrease
... ${SPACE}${SPACE}- Security scan passes
... ${SPACE}${SPACE}- All fixes are committed to the PR branch
... arguments:
... ${SPACE}${SPACE}- name: pr_branch
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Branch name of the PR
... ${SPACE}${SPACE}- name: base_branch
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
... ${SPACE}${SPACE}${SPACE}${SPACE}required: false
... ${SPACE}${SPACE}${SPACE}${SPACE}default: main
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Base branch to compare against
... invariants:
... ${SPACE}${SPACE}- "Only modify files that are already changed in the PR"
... ${SPACE}${SPACE}- "Do not change the intent of any code — only fix style, types, and test issues"
... ${SPACE}${SPACE}- "All fixes must include a comment explaining what was changed and why"
${action_path}= Set Variable ${SUITE_HOME}${/}review-pr-action.yaml
Create File ${action_path} ${action_yaml}
# expected_rc=None: action may already exist if Suite Setup cleanup failed
# and a prior E2E run left stale state; CI re-runnability requires tolerance.
Run CleverAgents Command action create --config ${action_path} expected_rc=None
# Launch plan with --arg flags per spec Step 3
${plan_result}= Run CleverAgents Command
... plan use local/review-pr ${CI_PROJECT}
... --automation-profile ci
... --format json
... --arg pr_branch=main
... --arg base_branch=main
... expected_rc=None timeout=180s
# Parse and validate JSON output
${stdout}= Set Variable ${plan_result.stdout}
${plan_id}= Safe Parse Json Field ${stdout} plan_id
IF '${plan_id}' != '${EMPTY}'
Log Plan launched with ID: ${plan_id}
# Drive the plan through its lifecycle. Per the specification
# (§Automation Profiles), ``plan use`` returns the plan in
# strategize/queued phase. ``plan execute`` drives the plan
# through Strategize → Execute, and auto_progress completes
# Apply when the ci profile (auto_apply=0.0) permits it.
${exec_result}= Run CleverAgents Command plan execute ${plan_id}
... --format json expected_rc=None timeout=180s
Should Be Equal As Integers ${exec_result.rc} 0 msg=plan execute failed (rc=${exec_result.rc}): ${exec_result.stderr}
# Poll plan status until it reaches a terminal state (spec Step 3
# polling loop).
${terminal_state}= Poll Plan Until Terminal ${plan_id}
Log Plan reached state: ${terminal_state}
Should Be Equal As Strings ${terminal_state} applied
... Plan did not reach 'applied' state. Final state: ${terminal_state}
# If applied, verify plan diff is parseable JSON (spec Step 3 final command)
${diff_result}= Run CleverAgents Command plan diff ${plan_id}
... --format json expected_rc=None
Log Plan diff output: ${diff_result.stdout}
# Validate the diff output is parseable JSON. The CLI may emit
# debug/log lines before the JSON payload, so we use raw_decode
# to locate the first valid JSON object.
IF len($diff_result.stdout) > 0
TRY
${parsed_diff}= Evaluate json.loads($diff_result.stdout) json
EXCEPT
TRY
${idx}= Evaluate $diff_result.stdout.index('{')
${parsed_diff}= Evaluate json.JSONDecoder().raw_decode($diff_result.stdout, $idx)[0] json
EXCEPT
Log Plan diff stdout is not valid JSON; may contain only log lines. WARN
${parsed_diff}= Set Variable ${NONE}
END
END
IF $parsed_diff is not None
Should Be True isinstance($parsed_diff, (dict, list))
... Expected JSON dict or list from plan diff
END
ELSE
Log Plan diff returned empty stdout; plan may have applied with no changes. WARN
END
ELSE
# Fail meaningfully when plan_id extraction fails
Fail Plan use command did not return a valid plan_id. rc=${plan_result.rc} stdout: ${plan_result.stdout} stderr: ${plan_result.stderr}
END
[Documentation] Create an action and launch a plan with --automation-profile ci,
... --format json, and --arg flags per specification Steps 2-3.
... Polls for plan completion. Requires LLM API keys.
[Tags] E2E
[Teardown] Log CI Plan Launch teardown complete
Skip If No LLM Keys
# Dynamically select actor based on available API keys.
# Prefer OpenAI first to reduce Anthropic credit-quota flakiness.
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
IF ${has_openai}
${actor}= Set Variable openai/gpt-4o
ELSE IF ${has_anthropic}
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
ELSE
${actor}= Set Variable openai/gpt-4o
END
# Create action with spec-aligned fields and dynamic actor
${action_yaml}= Catenate SEPARATOR=\n
... name: local/review-pr
... description: Automatically review a PR and fix issues
... strategy_actor: ${actor}
... execution_actor: ${actor}
... automation_profile: ci
... reusable: true
... state: available
... definition_of_done: |
... ${SPACE}${SPACE}- All lint issues are resolved
... ${SPACE}${SPACE}- Type checking passes
... ${SPACE}${SPACE}- Test coverage does not decrease
... ${SPACE}${SPACE}- Security scan passes
... ${SPACE}${SPACE}- All fixes are committed to the PR branch
... arguments:
... ${SPACE}${SPACE}- name: pr_branch
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
... ${SPACE}${SPACE}${SPACE}${SPACE}required: true
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Branch name of the PR
... ${SPACE}${SPACE}- name: base_branch
... ${SPACE}${SPACE}${SPACE}${SPACE}type: string
... ${SPACE}${SPACE}${SPACE}${SPACE}required: false
... ${SPACE}${SPACE}${SPACE}${SPACE}default: main
... ${SPACE}${SPACE}${SPACE}${SPACE}description: Base branch to compare against
... invariants:
... ${SPACE}${SPACE}- "Only modify files that are already changed in the PR"
... ${SPACE}${SPACE}- "Do not change the intent of any code — only fix style, types, and test issues"
... ${SPACE}${SPACE}- "All fixes must include a comment explaining what was changed and why"
${action_path}= Set Variable ${SUITE_HOME}${/}review-pr-action.yaml
Create File ${action_path} ${action_yaml}
# expected_rc=None: action may already exist if Suite Setup cleanup failed
# and a prior E2E run left stale state; CI re-runnability requires tolerance.
Run CleverAgents Command action create --config ${action_path} expected_rc=None
# Launch plan with --arg flags per spec Step 3
${plan_result}= Run CleverAgents Command
... plan use local/review-pr ${CI_PROJECT}
... --automation-profile ci
... --format json
... --arg pr_branch=main
... --arg base_branch=main
... expected_rc=None timeout=180s
# Parse and validate JSON output
${stdout}= Set Variable ${plan_result.stdout}
${plan_id}= Safe Parse Json Field ${stdout} plan_id
IF '${plan_id}' != '${EMPTY}'
Log Plan launched with ID: ${plan_id}
# Drive the plan through its lifecycle. Per the specification
# (§Automation Profiles), ``plan use`` returns the plan in
# strategize/queued phase. ``plan execute`` drives the plan
# through Strategize → Execute, and auto_progress completes
# Apply when the ci profile (auto_apply=0.0) permits it.
${exec_result}= Run CleverAgents Command plan execute ${plan_id}
... --format json expected_rc=None timeout=180s
Should Be Equal As Integers ${exec_result.rc} 0
... plan execute failed (rc=${exec_result.rc}): ${exec_result.stderr}
# Poll plan status until it reaches a terminal state (spec Step 3
# polling loop).
${terminal_state}= Poll Plan Until Terminal ${plan_id}
Log Plan reached state: ${terminal_state}
Should Be Equal As Strings ${terminal_state} applied
... Plan did not reach 'applied' state. Final state: ${terminal_state}
# If applied, verify plan diff is parseable JSON (spec Step 3 final command)
${diff_result}= Run CleverAgents Command plan diff ${plan_id}
... --format json expected_rc=None
Log Plan diff output: ${diff_result.stdout}
# Validate the diff output is parseable JSON. The CLI may emit
# debug/log lines before the JSON payload, so we use raw_decode
# to locate the first valid JSON object.
IF len($diff_result.stdout) > 0
TRY
${parsed_diff}= Evaluate json.loads($diff_result.stdout) json
EXCEPT
TRY
${idx}= Evaluate $diff_result.stdout.index('{')
${parsed_diff}= Evaluate json.JSONDecoder().raw_decode($diff_result.stdout, $idx)[0] json
EXCEPT
Log Plan diff stdout is not valid JSON; may contain only log lines. WARN
${parsed_diff}= Set Variable ${NONE}
END
END
IF $parsed_diff is not None
Should Be True isinstance($parsed_diff, (dict, list))
... Expected JSON dict or list from plan diff
END
ELSE
Log Plan diff returned empty stdout; plan may have applied with no changes. WARN
END
ELSE
# Fail meaningfully when plan_id extraction fails
Fail Plan use command did not return a valid plan_id. rc=${plan_result.rc} stdout: ${plan_result.stdout} stderr: ${plan_result.stderr}
END
WF07 E2E JSON Output Verification
[Documentation] Verify that --format json produces valid, parseable JSON
... output from the version command.
[Tags] E2E tdd_issue tdd_issue_4188 tdd_expected_fail
[Teardown] Log JSON Output Verification teardown complete
${result}= Run CleverAgents Command --format json version
Should Not Be Empty ${result.stdout}
# Parse as JSON and verify structure
${stdout}= Set Variable ${result.stdout}
${parsed}= Evaluate json.loads($stdout) json
Should Be True isinstance($parsed, dict)
... Expected JSON dict from version command, got: ${result.stdout}
Output Should Contain ${result} version
[Documentation] Verify that --format json produces valid, parseable JSON
... output from the version command.
[Tags] E2E
[Teardown] Log JSON Output Verification teardown complete
${result}= Run CleverAgents Command --format json version
Should Not Be Empty ${result.stdout}
# Parse as JSON and verify structure
${stdout}= Set Variable ${result.stdout}
${parsed}= Evaluate json.loads($stdout) json
Should Be True isinstance($parsed, dict)
... Expected JSON dict from version command, got: ${result.stdout}
Output Should Contain ${result} version
*** Keywords ***
WF07 Suite Setup
[Documentation] Delegate to shared E2E Suite Setup (includes centralized
... database initialization for DB-dependent CLI commands).
E2E Suite Setup
[Documentation] Delegate to shared E2E Suite Setup (includes centralized
... database initialization for DB-dependent CLI commands).
E2E Suite Setup
Create Temp Git Repo With Issues
[Documentation] Create a temporary git repo containing Python files with
... lint and type issues for the CI review scenario.
... Returns the path to the created repository.
[Arguments] ${name}=test-repo
${repo_dir}= Create Temp Git Repo ${name}
# Ensure the branch is named 'main' for spec consistency
${rename_result}= Run Process git branch -M main cwd=${repo_dir}
... timeout=30s on_timeout=kill
Should Be Equal As Integers ${rename_result.rc} 0 msg=Failed to rename branch to main: ${rename_result.stderr}
# Add Python file with lint issues (unused imports, missing type hints,
# unused variable) and type errors to simulate a PR needing review.
${py_content}= Catenate SEPARATOR=\n
... import os
... import sys
... ${EMPTY}
... ${EMPTY}
... def greet(name: str) -> int:
... ${SPACE}${SPACE}${SPACE}${SPACE}msg = "Hello, " + name
... ${SPACE}${SPACE}${SPACE}${SPACE}return msg
... ${EMPTY}
... ${EMPTY}
... def unused_function():
... ${SPACE}${SPACE}${SPACE}${SPACE}x = 42
... ${SPACE}${SPACE}${SPACE}${SPACE}pass
Create File ${repo_dir}${/}app.py ${py_content}
# Verify git operations succeed
${add_result}= Run Process git add . cwd=${repo_dir}
... timeout=30s on_timeout=kill
Should Be Equal As Integers ${add_result.rc} 0 msg=git add failed: ${add_result.stderr}
${commit_result}= Run Process git commit -m Add application with lint issues cwd=${repo_dir}
... timeout=30s on_timeout=kill
Should Be Equal As Integers ${commit_result.rc} 0 msg=git commit failed: ${commit_result.stderr}
RETURN ${repo_dir}
[Documentation] Create a temporary git repo containing Python files with
... lint and type issues for the CI review scenario.
... Returns the path to the created repository.
[Arguments] ${name}=test-repo
${repo_dir}= Create Temp Git Repo ${name}
# Ensure the branch is named 'main' for spec consistency
${rename_result}= Run Process git branch -M main cwd=${repo_dir}
... timeout=30s on_timeout=kill
Should Be Equal As Integers ${rename_result.rc} 0
... Failed to rename branch to main: ${rename_result.stderr}
# Add Python file with lint issues (unused imports, missing type hints,
# unused variable) and type errors to simulate a PR needing review.
${py_content}= Catenate SEPARATOR=\n
... import os
... import sys
... ${EMPTY}
... ${EMPTY}
... def greet(name: str) -> int:
... ${SPACE}${SPACE}${SPACE}${SPACE}msg = "Hello, " + name
... ${SPACE}${SPACE}${SPACE}${SPACE}return msg
... ${EMPTY}
... ${EMPTY}
... def unused_function():
... ${SPACE}${SPACE}${SPACE}${SPACE}x = 42
... ${SPACE}${SPACE}${SPACE}${SPACE}pass
Create File ${repo_dir}${/}app.py ${py_content}
# Verify git operations succeed
${add_result}= Run Process git add . cwd=${repo_dir}
... timeout=30s on_timeout=kill
Should Be Equal As Integers ${add_result.rc} 0
... git add failed: ${add_result.stderr}
${commit_result}= Run Process git commit -m Add application with lint issues cwd=${repo_dir}
... timeout=30s on_timeout=kill
Should Be Equal As Integers ${commit_result.rc} 0
... git commit failed: ${commit_result.stderr}
RETURN ${repo_dir}
Poll Plan Until Terminal
[Documentation] Poll plan status until it reaches a terminal state.
... Polling exit states (not all are terminal in the domain
... model — ``errored`` is recoverable via ``resume_plan``
... but will not auto-advance, so polling should exit):
... applied, constrained, errored, cancelled.
... Returns the state string, or 'timeout'.
[Arguments] ${plan_id} ${max_polls}=18 ${interval}=10s
FOR ${i} IN RANGE ${max_polls}
${result}= Run CleverAgents Command plan status ${plan_id}
... --format json expected_rc=None
${state}= Safe Parse Json Field ${result.stdout} state
IF '${state}' in ['applied', 'constrained', 'cancelled', 'errored']
RETURN ${state}
END
Sleep ${interval}
END
RETURN timeout
[Documentation] Poll plan status until it reaches a terminal state.
... Polling exit states (not all are terminal in the domain
... model — ``errored`` is recoverable via ``resume_plan``
... but will not auto-advance, so polling should exit):
... applied, constrained, errored, cancelled.
... Returns the state string, or 'timeout'.
[Arguments] ${plan_id} ${max_polls}=18 ${interval}=10s
FOR ${i} IN RANGE ${max_polls}
${result}= Run CleverAgents Command plan status ${plan_id}
... --format json expected_rc=None
${state}= Safe Parse Json Field ${result.stdout} state
IF '${state}' in ['applied', 'constrained', 'cancelled', 'errored']
RETURN ${state}
END
Sleep ${interval}
END
RETURN timeout
+450 -109
View File
@@ -1,145 +1,486 @@
*** Settings ***
Documentation E2E test for Workflow Example 12: Large-Scale Hierarchical Feature
... Implementation (supervised profile).
Documentation E2E test for Workflow Example 12: Large-Scale Hierarchical Feature
... Implementation (supervised profile).
...
... Expert-level scenario building a notification system across 4
... projects with hierarchical plan decomposition, error recovery
... via plan correct, and phased apply.
... Expert-level scenario building a notification system across 4
... projects with hierarchical plan decomposition, error recovery
... via plan correct, and phased apply.
...
... Zero mocking — real CLI, real LLM API keys.
... Zero mocking — real CLI, real LLM API keys.
...
... ``Skip If No LLM Keys`` ensures graceful degradation in keyless
... CI environments.
Resource common_e2e.resource
Suite Setup WF12 Suite Setup
... ``Skip If No LLM Keys`` ensures graceful degradation in keyless
... CI environments.
Resource common_e2e.resource
Suite Setup WF12 Suite Setup
Suite Teardown E2E Suite Teardown
Force Tags E2E
Force Tags E2E
*** Keywords ***
WF12 Suite Setup
[Documentation] E2E Suite Setup plus workspace init and unique run suffix for name isolation.
[Tags] tdd_issue tdd_issue_4188
E2E Suite Setup
# Initialise the database so plan/resource/project commands work in all tests.
# Use --force because the workspace may already contain an initialised project.
${init}= Run CleverAgents Command init --force --yes
Should Be Equal As Integers ${init.rc} 0
# Generate a unique suffix to avoid UNIQUE constraint collisions on
# repeated E2E runs against the same database (parallel CI safety).
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
# Derive run-unique resource/project/action names.
Set Suite Variable ${ACTION_NAME} local/wf12-notifications-${suffix}
Set Suite Variable ${PROTOS_RES} local/wf12-protos-res-${suffix}
Set Suite Variable ${API_RES} local/wf12-api-res-${suffix}
Set Suite Variable ${WORKER_RES} local/wf12-worker-res-${suffix}
Set Suite Variable ${FRONTEND_RES} local/wf12-frontend-res-${suffix}
Set Suite Variable ${PROTOS_PROJ} local/wf12-protos-${suffix}
Set Suite Variable ${API_PROJ} local/wf12-api-${suffix}
Set Suite Variable ${WORKER_PROJ} local/wf12-worker-${suffix}
Set Suite Variable ${FRONTEND_PROJ} local/wf12-frontend-${suffix}
[Documentation] E2E Suite Setup plus workspace init and unique run suffix for name isolation.
E2E Suite Setup
# Initialise the database so plan/resource/project commands work in all tests.
# Use --force because the workspace may already contain an initialised project.
${init}= Run CleverAgents Command init --force --yes
Should Be Equal As Integers ${init.rc} 0
# Generate a unique suffix to avoid UNIQUE constraint collisions on
# repeated E2E runs against the same database (parallel CI safety).
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
# Derive run-unique resource/project/action names.
Set Suite Variable ${ACTION_NAME} local/wf12-notifications-${suffix}
Set Suite Variable ${PROTOS_RES} local/wf12-protos-res-${suffix}
Set Suite Variable ${API_RES} local/wf12-api-res-${suffix}
Set Suite Variable ${WORKER_RES} local/wf12-worker-res-${suffix}
Set Suite Variable ${FRONTEND_RES} local/wf12-frontend-res-${suffix}
Set Suite Variable ${PROTOS_PROJ} local/wf12-protos-${suffix}
Set Suite Variable ${API_PROJ} local/wf12-api-${suffix}
Set Suite Variable ${WORKER_PROJ} local/wf12-worker-${suffix}
Set Suite Variable ${FRONTEND_PROJ} local/wf12-frontend-${suffix}
Create Project Repo
[Documentation] Create a temp git repo for a project component.
... Returns the path to the created repository.
[Arguments] ${name} ${content}
${repo}= Create Temp Git Repo wf12-${name}-${RUN_SUFFIX}
Create Directory ${repo}${/}src
Create File ${repo}${/}src${/}main.py ${content}
${git_add}= Run Process git add . cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_add.rc} 0 msg=git add failed for ${name}: ${git_add.stderr}
${git_commit}= Run Process git commit -m Initial ${name} cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_commit.rc} 0 msg=git commit failed for ${name}: ${git_commit.stderr}
RETURN ${repo}
[Documentation] Create a temp git repo for a project component.
... Returns the path to the created repository.
[Arguments] ${name} ${content}
${repo}= Create Temp Git Repo wf12-${name}-${RUN_SUFFIX}
Create Directory ${repo}${/}src
Create File ${repo}${/}src${/}main.py ${content}
${git_add}= Run Process git add . cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_add.rc} 0
... git add failed for ${name}: ${git_add.stderr}
${git_commit}= Run Process git commit -m Initial ${name} cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${git_commit.rc} 0
... git commit failed for ${name}: ${git_commit.stderr}
RETURN ${repo}
Register Project With Invariant
[Documentation] Register resource, create project with an invariant.
... Note: Spec shows most projects with 2 invariants. This keyword
... accepts 1 invariant per project as a simplification. The invariant
... registration path is exercised identically regardless of count.
... TODO: Add a second invariant per project once the full invariant
... matrix is stable.
[Arguments] ${res_name} ${proj_name} ${repo_dir} ${invariant_text}
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
Should Be Equal As Integers ${branch_result.rc} 0 msg=git rev-parse failed: ${branch_result.stderr}
${branch}= Strip String ${branch_result.stdout}
# Register resource
${r}= Run CleverAgents Command
... resource add git-checkout ${res_name}
... --path ${repo_dir} --branch ${branch}
... --format json
Should Not Contain ${r.stdout}${r.stderr} Traceback
Should Not Contain ${r.stdout}${r.stderr} INTERNAL
Output Should Contain ${r} ${res_name}
# Create project with invariant (spec requires per-project invariants)
${p}= Run CleverAgents Command
... project create ${proj_name}
... --resource ${res_name}
... --invariant ${invariant_text}
... --format json
Should Not Contain ${p.stdout}${p.stderr} Traceback
Should Not Contain ${p.stdout}${p.stderr} INTERNAL
Output Should Contain ${p} ${proj_name}
[Documentation] Register resource, create project with an invariant.
... Note: Spec shows most projects with 2 invariants. This keyword
... accepts 1 invariant per project as a simplification. The invariant
... registration path is exercised identically regardless of count.
... TODO: Add a second invariant per project once the full invariant
... matrix is stable.
[Arguments] ${res_name} ${proj_name} ${repo_dir} ${invariant_text}
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo_dir} timeout=60s on_timeout=kill
Should Be Equal As Integers ${branch_result.rc} 0
... git rev-parse failed: ${branch_result.stderr}
${branch}= Strip String ${branch_result.stdout}
# Register resource
${r}= Run CleverAgents Command
... resource add git-checkout ${res_name}
... --path ${repo_dir} --branch ${branch}
... --format json
Should Not Contain ${r.stdout}${r.stderr} Traceback
Should Not Contain ${r.stdout}${r.stderr} INTERNAL
Output Should Contain ${r} ${res_name}
# Create project with invariant (spec requires per-project invariants)
${p}= Run CleverAgents Command
... project create ${proj_name}
... --resource ${res_name}
... --invariant ${invariant_text}
... --format json
Should Not Contain ${p.stdout}${p.stderr} Traceback
Should Not Contain ${p.stdout}${p.stderr} INTERNAL
Output Should Contain ${p} ${proj_name}
Verify Plan In List
[Documentation] Verify a plan appears in list output.
[Arguments] ${plan_id}
${list_result}= Run CleverAgents Command plan list --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${list_result.rc} 0 msg=list failed (rc=${list_result.rc}): ${list_result.stderr}
Output Should Contain ${list_result} ${plan_id}
[Documentation] Verify a plan appears in list output.
[Arguments] ${plan_id}
${list_result}= Run CleverAgents Command plan list --format json expected_rc=None timeout=120s
Should Be Equal As Integers ${list_result.rc} 0
... list failed (rc=${list_result.rc}): ${list_result.stderr}
Output Should Contain ${list_result} ${plan_id}
Select Non Root Decision Id
[Documentation] Parse JSON tree output and select a non-root decision ID
... suitable for correction. Uses a targeted regex to extract
... only values from ``"decision_id"`` fields (avoid matching
... plan_id / resource_id). Requires at least 2 decision IDs to
... guarantee the returned ID is not the root.
[Arguments] ${tree_stdout}
# Targeted regex — only match values from "decision_id" JSON fields
# Crockford Base32 character class: excludes I, L, O, U
${all_ids}= Get Regexp Matches ${tree_stdout} "decision_id"\\s*:\\s*"([0-9A-HJKMNP-TV-Z]{26})" 1
# Guard: need at least 2 decision IDs (root + at least one child)
${id_count}= Get Length ${all_ids}
Should Be True ${id_count} >= 2
... Need ≥2 decision IDs to select non-root, found ${id_count}
# Use the last ID found — in JSON serialization of the tree, leaf/child
# decisions appear after the root prompt_definition decision.
${last_index}= Evaluate len($all_ids) - 1
${decision_id}= Set Variable ${all_ids}[${last_index}]
# Defensive check: ensure selected ID differs from first (presumed root)
# to guard against JSON serialization order assumptions.
Should Not Be Equal ${decision_id} ${all_ids}[0] msg=Selected non-root decision ID should differ from first ID (presumed root)
RETURN ${decision_id}
[Documentation] Parse JSON tree output and select a non-root decision ID
... suitable for correction. Uses a targeted regex to extract
... only values from ``"decision_id"`` fields (avoid matching
... plan_id / resource_id). Requires at least 2 decision IDs to
... guarantee the returned ID is not the root.
[Arguments] ${tree_stdout}
# Targeted regex — only match values from "decision_id" JSON fields
# Crockford Base32 character class: excludes I, L, O, U
${all_ids}= Get Regexp Matches ${tree_stdout} "decision_id"\\s*:\\s*"([0-9A-HJKMNP-TV-Z]{26})" 1
# Guard: need at least 2 decision IDs (root + at least one child)
${id_count}= Get Length ${all_ids}
Should Be True ${id_count} >= 2
... Need ≥2 decision IDs to select non-root, found ${id_count}
# Use the last ID found — in JSON serialization of the tree, leaf/child
# decisions appear after the root prompt_definition decision.
${last_index}= Evaluate len($all_ids) - 1
${decision_id}= Set Variable ${all_ids}[${last_index}]
# Defensive check: ensure selected ID differs from first (presumed root)
# to guard against JSON serialization order assumptions.
Should Not Be Equal ${decision_id} ${all_ids}[0]
... Selected non-root decision ID should differ from first ID (presumed root)
RETURN ${decision_id}
*** Test Cases ***
WF12 Large Scale Hierarchical Feature Implementation
[Documentation] Supervised-profile workflow: 4-project notification system
... with hierarchical decomposition, user guidance via plan
... correct (append mode), and dependency-ordered apply.
...
... Note: ``plan prompt`` (spec Step 4 — supervised-profile
... user intervention) is not yet implemented as a CLI command.
... Once available, this test should add a ``plan prompt`` step
... after tree inspection to provide user guidance and verify
... the ``user_intervention`` decision is created.
[Tags] tdd_issue tdd_issue_4188 tdd_expected_fail
[Timeout] 35 minutes
# TDD placeholder - implementation pending
Fail WF12 hierarchical decomposition not yet implemented (TDD placeholder)
[Documentation] Supervised-profile workflow: 4-project notification system
... with hierarchical decomposition, user guidance via plan
... correct (append mode), and dependency-ordered apply.
...
... Note: ``plan prompt`` (spec Step 4 — supervised-profile
... user intervention) is not yet implemented as a CLI command.
... Once available, this test should add a ``plan prompt`` step
... after tree inspection to provide user guidance and verify
... the ``user_intervention`` decision is created.
[Timeout] 35 minutes
# ---- Gate: skip if no LLM API keys ----
Skip If No LLM Keys
# ---- Detect actor based on available API key ----
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
IF ${has_anthropic}
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
ELSE
${actor}= Set Variable openai/gpt-4o-mini
END
# ---- Create project repos ----
${protos_repo}= Create Project Repo protos """Proto definitions for notification service."""\n
${api_repo}= Create Project Repo api """API server for notifications."""\n
${worker_repo}= Create Project Repo worker """Background worker for sending notifications."""\n
${frontend_repo}= Create Project Repo frontend """Frontend notification UI components."""\n
# ---- Register all 4 projects with per-project invariants (spec Step 1) ----
# Note: Spec shows api/worker linked to BOTH their own repo AND the protos repo
# for cross-project dependency ordering. This test links each project to only its
# own repo as a simplification — multi-resource project registration and the
# resulting phased apply ordering are exercised via the project list in plan use.
# TODO: Add --resource ${PROTOS_RES} to api/worker once multi-resource projects
# are independently validated.
Register Project With Invariant ${PROTOS_RES} ${PROTOS_PROJ} ${protos_repo}
... 'Proto changes must be backward-compatible'
Register Project With Invariant ${API_RES} ${API_PROJ} ${api_repo}
... 'All new endpoints must have OpenAPI docs'
Register Project With Invariant ${WORKER_RES} ${WORKER_PROJ} ${worker_repo}
... 'Workers must be idempotent'
Register Project With Invariant ${FRONTEND_RES} ${FRONTEND_PROJ} ${frontend_repo}
... 'All components must have accessibility support'
# ---- Global invariant (spec Step 1 requires global invariant) ----
${r_global_inv}= Run CleverAgents Command
... invariant add --global
... 'All inter-service communication must use the shared proto definitions'
... --format json expected_rc=None timeout=60s
Should Be Equal As Integers ${r_global_inv.rc} 0
... Global invariant registration failed (rc=${r_global_inv.rc}): ${r_global_inv.stderr}
Should Not Contain ${r_global_inv.stdout}${r_global_inv.stderr} Traceback
Should Not Contain ${r_global_inv.stdout}${r_global_inv.stderr} INTERNAL
Output Should Contain ${r_global_inv} inter-service communication
# TODO(#758): Spec Step 1 shows 4 validations registered and attached to projects.
# Validation-gated apply is a key M6 feature. Validation registration is omitted
# here pending independent validation of the validation subsystem. Follow-up ticket
# needed to add validation registration and verify validation-gated apply.
# ---- Create action with spec-required fields ----
${action_yaml}= Catenate SEPARATOR=\n
... name: ${ACTION_NAME}
... description: Build notification system across protos, api, worker, and frontend
... long_description: |
... ${SPACE}${SPACE}Implement a full notification system with backend API, message queue,
... ${SPACE}${SPACE}worker service, and frontend dashboard. The system must be designed for
... ${SPACE}${SPACE}reliability (at-least-once delivery), scalability (async processing),
... ${SPACE}${SPACE}and user control (per-channel preferences with quiet hours).
... definition_of_done: All 4 projects have notification functionality implemented
... strategy_actor: ${actor}
... execution_actor: ${actor}
... estimation_actor: ${actor}
... invariant_actor: ${actor}
# Ticket says 'supervised' but spec uses 'cautious' — following spec.
... automation_profile: cautious
... reusable: false
... state: available
# Note: Spec defines 4 action-level invariants. Only 2 are included here as a
# test simplification — the remaining 2 (code review standards, integration test
# coverage) are functionally equivalent for exercising the invariant registration
# path. TODO: Add all 4 invariants once the full invariant matrix is stable.
... invariants:
... ${SPACE}${SPACE}- "Proto definitions must be implemented before any service code"
... ${SPACE}${SPACE}- "Each service must be deployable independently after its changes"
${action_path}= Set Variable ${SUITE_HOME}${/}wf12_action.yaml
Create File ${action_path} ${action_yaml}
${r_action}= Run CleverAgents Command
... action create --config ${action_path} --format json
Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback
Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL
Output Should Contain ${r_action} ${ACTION_NAME}
# ---- Plan use — all 4 projects (spec Step 3) ----
# Spec Step 2 defines args (notification_channels) and Step 3 uses
# --arg to pass values. Both are omitted: action arguments in YAML
# trigger a UNIQUE-constraint error during plan-use (pre-existing bug in
# PlanLifecycleService.use_action argument passthrough).
# TODO: Add arguments to action YAML and --arg to plan use once fixed.
${r_use}= Run CleverAgents Command
... plan use ${ACTION_NAME}
... ${PROTOS_PROJ} ${API_PROJ} ${WORKER_PROJ} ${FRONTEND_PROJ}
... --format json timeout=120s
Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback
Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL
Output Should Contain ${r_use} plan_id
${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id
Should Not Be Empty ${plan_id} Could not parse plan_id from plan use output
Log Plan ID: ${plan_id}
# ---- Verify plan appears in list (consistent with m6_acceptance pattern) ----
Verify Plan In List ${plan_id}
# ---- Strategize ----
${r_strat}= Run CleverAgents Command
... plan execute ${plan_id} --format json
... expected_rc=None timeout=300s
IF ${r_strat.rc} != 0
Fail plan execute (strategize) failed (rc=${r_strat.rc}): ${r_strat.stderr}
END
Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback
Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL
Output Should Contain ${r_strat} ${plan_id}
# ---- Tree inspection — verify hierarchy (AC-3, AC-6) ----
${r_tree}= Run CleverAgents Command
... plan tree ${plan_id} --format json
... expected_rc=None timeout=60s
Should Be Equal As Integers ${r_tree.rc} 0
... plan tree failed (rc=${r_tree.rc}): ${r_tree.stderr}
Should Not Contain ${r_tree.stdout}${r_tree.stderr} Traceback
Should Not Contain ${r_tree.stdout}${r_tree.stderr} INTERNAL
Should Not Be Empty ${r_tree.stdout} Plan tree output should not be empty
# Verify decision nodes using targeted "decision_id" regex
# Crockford Base32 character class: excludes I, L, O, U
${decision_ids}= Get Regexp Matches ${r_tree.stdout} "decision_id"\\s*:\\s*"([0-9A-HJKMNP-TV-Z]{26})" 1
Should Not Be Empty ${decision_ids} Expected at least one decision ID in tree
# Use the regex match results as the canonical decision count (n3 fix: avoids
# divergence between regex matches and substring count).
${decision_count}= Get Length ${decision_ids}
Log Decision tree contains ${decision_count} decision node(s)
Should Be True ${decision_count} >= 2
... Plan tree should contain at least 2 decision nodes for hierarchy (found ${decision_count})
# Hard assertion on hierarchical children (AC-3, AC-6: parent-child relationships)
${has_children_key}= Evaluate '"children"' in $r_tree.stdout
Should Be True ${has_children_key}
... Plan tree must contain 'children' field for hierarchical decomposition (AC-3, AC-6)
# Verify at least one children array is non-empty — "children": [{ proves actual
# parent→child hierarchy, not just empty arrays on sibling nodes (M4 fix).
# Note: With real LLM execution, the depth of hierarchical decomposition is
# non-deterministic; the LLM may produce flat sibling decisions rather than
# nested parent→child trees. We assert the non-empty children requirement but
# fall back to a WARN if the structure is flat, since AC-3/AC-6 are still
# partially satisfied by the presence of multiple decision nodes.
${nonempty_children}= Get Regexp Matches ${r_tree.stdout} "children"\\s*:\\s*\\[\\s*\\{
IF len($nonempty_children) == 0
Log No non-empty children arrays found in tree — LLM produced flat sibling decisions rather than nested hierarchy. AC-3/AC-6 partially verified via decision count (>= 2). WARN
ELSE
Log Confirmed non-empty children array(s) in tree — true parent→child hierarchy present (AC-3, AC-6)
END
# Log children field occurrences (informational, measures breadth not depth)
${children_occurrences}= Evaluate $r_tree.stdout.count('"children"')
Log Children field occurrences in tree: ${children_occurrences}
# ---- Explain a decision (spec Step 4 shows plan explain) ----
${r_explain}= Run CleverAgents Command
... plan explain ${decision_ids}[0]
... --format json expected_rc=None timeout=60s
Should Be Equal As Integers ${r_explain.rc} 0
... plan explain failed (rc=${r_explain.rc}): ${r_explain.stderr}
Should Not Contain ${r_explain.stdout}${r_explain.stderr} Traceback
Should Not Contain ${r_explain.stdout}${r_explain.stderr} INTERNAL
Should Not Be Empty ${r_explain.stdout} plan explain output should not be empty
# Verify explain output references the queried decision ID (m3 fix)
Output Should Contain ${r_explain} ${decision_ids}[0]
Log Plan explain output: ${r_explain.stdout} level=DEBUG
# ---- Verify intermediate state after strategize ----
${r_mid_status}= Run CleverAgents Command
... plan status ${plan_id} --format json
... expected_rc=None timeout=60s
Should Be Equal As Integers ${r_mid_status.rc} 0
... Intermediate plan status failed (rc=${r_mid_status.rc}): ${r_mid_status.stderr}
${mid_phase}= Safe Parse Json Field ${r_mid_status.stdout} phase
${mid_state}= Safe Parse Json Field ${r_mid_status.stdout} processing_state
Log Post-strategize status: phase=${mid_phase} processing_state=${mid_state}
# Assert plan state progressed — at least one field should be non-empty after strategize (M3 fix)
${mid_populated}= Evaluate '${mid_phase}' != '' or '${mid_state}' != ''
Should Be True ${mid_populated}
... Plan should have non-empty phase or processing_state after strategize
# ---- Execute ----
# Note: plan execute is called a second time here. The first call (above)
# drives the strategize phase; this call advances the plan into the execute
# phase. plan execute is idempotent — if the plan is already past
# execution, this is a safe no-op that returns the current state.
${r_exec}= Run CleverAgents Command
... plan execute ${plan_id} --format json
... expected_rc=None timeout=300s
IF ${r_exec.rc} != 0
Fail plan execute failed (rc=${r_exec.rc}): ${r_exec.stderr}
END
Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback
Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL
Output Should Contain ${r_exec} ${plan_id}
# ---- Correction — append mode (AC-4) ----
# Check plan status before correction to verify the state being corrected
${r_pre_correct_status}= Run CleverAgents Command
... plan status ${plan_id} --format json
... expected_rc=None timeout=60s
Should Be Equal As Integers ${r_pre_correct_status.rc} 0
... Pre-correction plan status failed (rc=${r_pre_correct_status.rc}): ${r_pre_correct_status.stderr}
${pre_correct_phase}= Safe Parse Json Field ${r_pre_correct_status.stdout} phase
${pre_correct_state}= Safe Parse Json Field ${r_pre_correct_status.stdout} processing_state
Log Pre-correction status: phase=${pre_correct_phase} state=${pre_correct_state}
# Gate correction on pre-correction status: if plan is already in a terminal
# processing state, correction may fail — skip with WARN (m4 fix).
${pre_correct_terminal}= Evaluate '${pre_correct_state}'.lower() in ('applied', 'constrained', 'cancelled')
IF ${pre_correct_terminal}
Log Plan already in terminal state '${pre_correct_state}' before correction — skipping correction step WARN
ELSE
# Note: AC-4 requires "error handling (plan correct after failure)." In a real
# E2E scenario the LLM may or may not have produced a failure state by this point.
# We apply correction unconditionally to exercise the append-mode code path;
# verifying an actual failure state would require deterministic error injection
# which is not feasible with real LLM execution.
# Select a non-root decision for correction (avoid root prompt_definition)
${decision_id}= Select Non Root Decision Id ${r_tree.stdout}
Log Correcting decision: ${decision_id}
${r_correct}= Run CleverAgents Command
... plan correct ${decision_id}
... --mode append
... --guidance Ensure error handling is included in notification delivery
... --plan ${plan_id}
... --yes
... --format json
... expected_rc=None timeout=180s
# Verify correction completed
IF ${r_correct.rc} != 0
Fail plan correct failed (rc=${r_correct.rc}): ${r_correct.stderr}
END
Should Not Contain ${r_correct.stdout}${r_correct.stderr} Traceback
Should Not Contain ${r_correct.stdout}${r_correct.stderr} INTERNAL
# Verify correction output contains append-mode indicators.
# Note: Do NOT check for bare 'correction' substring — it always matches the
# "correction_id" JSON key, making the check vacuously true (M1 fix).
${correct_combined}= Set Variable ${r_correct.stdout} ${r_correct.stderr}
${correct_lower}= Evaluate ($correct_combined).lower()
${has_append}= Evaluate 'append' in $correct_lower
${has_queued}= Evaluate 'queued' in $correct_lower
${has_mode_append}= Evaluate '"mode"' in $correct_lower and '"append"' in $correct_lower
${has_correction_indicator}= Evaluate $has_append or $has_queued or $has_mode_append
Should Be True ${has_correction_indicator}
... Correction output should acknowledge append mode (found none of: append, queued, mode+append)
# Structural check: parse the correction response and verify a status field exists
${correction_status}= Safe Parse Json Field ${r_correct.stdout} status
${correction_id_field}= Safe Parse Json Field ${r_correct.stdout} correction_id
# At least one structural field should be populated in the correction response
${has_structural_field}= Evaluate '${correction_status}' != '' or '${correction_id_field}' != ''
Should Be True ${has_structural_field}
... Correction response should contain a populated 'status' or 'correction_id' field
# Post-correction verification — re-fetch tree to confirm correction is reflected
${r_tree2}= Run CleverAgents Command
... plan tree ${plan_id} --format json
... expected_rc=None timeout=60s
Should Be Equal As Integers ${r_tree2.rc} 0
... Post-correction plan tree failed (rc=${r_tree2.rc}): ${r_tree2.stderr}
Should Not Be Empty ${r_tree2.stdout} Post-correction tree should not be empty
# Use same regex-based counting as initial tree inspection (m1 fix: avoids
# divergence between regex matches and raw substring count).
${post_ids}= Get Regexp Matches ${r_tree2.stdout} "decision_id"\\s*:\\s*"([0-9A-HJKMNP-TV-Z]{26})" 1
${post_count}= Get Length ${post_ids}
# Correction may add a new decision node immediately or after re-execution.
# At minimum, the tree must still contain the original decisions.
Should Be True ${post_count} >= ${decision_count}
... Post-correction tree should have at least as many decisions (before=${decision_count}, after=${post_count})
IF ${post_count} > ${decision_count}
Log Correction added new decision node(s): ${post_count} (was ${decision_count})
ELSE
Log Correction queued but tree unchanged yet (${post_count} nodes); may require re-execute WARN
END
END
# ---- Diff ----
${r_diff}= Run CleverAgents Command
... plan diff ${plan_id} --format json
... expected_rc=None timeout=60s
Should Be Equal As Integers ${r_diff.rc} 0
... plan diff failed (rc=${r_diff.rc}): ${r_diff.stderr}
Should Not Be Empty ${r_diff.stdout} plan diff output should not be empty
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
Output Should Contain ${r_diff} ${plan_id}
# ---- Apply (AC-5: verify phased apply with dependency-order indicators) ----
${r_apply}= Run CleverAgents Command
... plan apply --yes ${plan_id} --format json
... expected_rc=None timeout=300s
IF ${r_apply.rc} == 0
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
Output Should Contain ${r_apply} ${plan_id}
# Assert apply phase (consistent with m6_acceptance Full Flow Apply Step)
${apply_phase}= Safe Parse Json Field ${r_apply.stdout} phase
IF '${apply_phase}' != ''
Should Contain ${apply_phase.lower()} apply
... Plan phase should indicate apply after apply
ELSE
Log Apply phase field is empty; cannot verify phase value from apply output WARN
END
# AC-5: Verify apply command succeeded and plan_id is present.
# TODO(#758): AC-5 requires dependency-order verification (protos before
# api/worker, api/worker before frontend). apply's current JSON output
# does not expose per-project apply ordering, so true dependency-order assertions
# are not feasible here. Follow-up ticket needed to add structured per-phase
# apply results to apply output, enabling proper AC-5 verification.
Log apply succeeded; dependency-order not structurally verifiable with current output
ELSE
Fail apply failed (rc=${r_apply.rc}) stdout=${r_apply.stdout} stderr=${r_apply.stderr}
END
# ---- Final status — verify terminal state ----
${r_status}= Run CleverAgents Command
... plan status ${plan_id} --format json
... expected_rc=None timeout=60s
Should Be Equal As Integers ${r_status.rc} 0
... plan status failed (rc=${r_status.rc}): ${r_status.stderr}
Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback
Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL
Should Not Be Empty ${r_status.stdout}
Output Should Contain ${r_status} ${plan_id}
# Parse and verify plan is in a terminal state (not intermediate)
${phase}= Safe Parse Json Field ${r_status.stdout} phase
${state}= Safe Parse Json Field ${r_status.stdout} processing_state
Log Final phase=${phase} processing_state=${state}
# At least one field must be non-empty
${state_populated}= Evaluate '${phase}' != '' or '${state}' != ''
Should Be True ${state_populated}
... Plan status should report non-empty phase or processing_state after full lifecycle
# Verify terminal state — phase or processing_state must indicate completion.
# PlanPhase enum: action, strategize, execute, apply (apply is the terminal phase).
# ProcessingState enum: queued, processing, errored, complete, applied, constrained, cancelled.
# Terminal processing states in Apply: applied (success), constrained (cannot proceed),
# cancelled (user/system cancelled). errored is handled separately with a WARN.
# Non-terminal states (queued, processing) may appear if apply is asynchronous.
IF '${phase}' != ''
${is_terminal_phase}= Evaluate '${phase}'.lower() in ('apply',)
Should Be True ${is_terminal_phase}
... Plan should be in terminal phase 'apply' after full lifecycle, got '${phase}'
ELSE
Fail Plan phase is empty after full lifecycle — expected 'apply'
END
IF '${state}' != ''
${is_terminal_state}= Evaluate '${state}'.lower() in ('applied', 'constrained', 'cancelled')
IF '${state}'.lower() == 'errored'
Log Plan reached 'errored' processing_state — apply may have failed WARN
ELSE IF '${state}'.lower() in ('queued', 'processing')
Log Plan in non-terminal state '${state}' after apply — apply may be asynchronous WARN
ELSE
Should Be True ${is_terminal_state}
... Plan should be in terminal processing_state after full lifecycle, got '${state}' (expected: applied, constrained, or cancelled)
END
ELSE
Fail processing_state is empty after full lifecycle with phase='apply'
END
+61 -14
View File
@@ -15,23 +15,70 @@ Suite Teardown WF14 Suite Teardown
Force Tags E2E
*** Keywords ***
WF14 Clean Global Config
[Documentation] Directly remove server.url, server.token, and server.namespace
... from ``~/.cleveragents/config.toml`` using Python/tomlkit,
... bypassing the ``config set`` CLI command entirely. This is
... necessary because ``config set`` does not reliably overwrite an
... existing TOML dotted key when the file already contains that key
... from a previous run (e.g. written by ``server_stubs.robot``).
...
... Handles both nested representation (``[server]\\nurl = "..."``
... parsed as ``{"server": {"url": "..."}}`` by tomllib) and flat
... quoted-key representation (``"server.url" = "..."`` parsed as
... ``{"server.url": "..."}``) so all historic file formats are
... cleaned regardless of how they were written.
...
... Called from both ``WF14 Suite Setup`` (pre-test cleanup, so any
... stale value from a prior run is removed before ``config set``
... runs) and ``WF14 Suite Teardown`` (post-test cleanup, so
... subsequent runs start with a clean slate). Failures are
... silently ignored — a missing config file is a normal first-run
... condition.
${python_exec}= Evaluate sys.executable sys
${py}= Catenate SEPARATOR=\n
... import tomlkit
... from pathlib import Path
... p = Path.home() / ".cleveragents" / "config.toml"
... if not p.exists():
... exit(0)
... doc = tomlkit.loads(p.read_text())
... for section, name in [("server", "url"), ("server", "token"), ("server", "namespace")]:
... if section in doc and hasattr(doc[section], "pop"):
... try:
... doc[section].pop(name, None)
... if len(doc[section]) == 0:
... del doc[section]
... except Exception:
... pass
... for flat_key in ["server.url", "server.token", "server.namespace"]:
... try:
... del doc[flat_key]
... except (KeyError, Exception):
... pass
... p.write_text(tomlkit.dumps(doc))
Run Keyword And Ignore Error
... Run Process ${python_exec} -c ${py} timeout=30s on_timeout=kill
WF14 Suite Setup
[Documentation] E2E Suite Setup plus unique run suffix generation.
[Tags] tdd_issue tdd_issue_4188
E2E Suite Setup
# Generate a unique suffix for entity names to avoid UNIQUE
# constraint collisions on repeated E2E runs or parallel CI.
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
[Documentation] E2E Suite Setup plus unique run suffix generation and
... pre-test global config cleanup.
E2E Suite Setup
# Clean any stale server config from the global config file BEFORE running
# any config set/get tests, so config set always writes into a clean slot.
WF14 Clean Global Config
# Generate a unique suffix for entity names to avoid UNIQUE constraint
# collisions on repeated E2E runs or parallel CI workers.
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
WF14 Suite Teardown
[Documentation] Restore config state and clean up E2E test environment.
# Reset config keys set during the server config test to avoid
# contaminating subsequent test suites.
Run Keyword And Ignore Error Run CleverAgents Command config set server.url ${EMPTY} expected_rc=None
Run Keyword And Ignore Error Run CleverAgents Command config set server.token ${EMPTY} expected_rc=None
Run Keyword And Ignore Error Run CleverAgents Command config set core.namespace local expected_rc=None
E2E Suite Teardown
[Documentation] Remove server config keys from the global config file and
... tear down the E2E environment. Uses direct TOML manipulation
... instead of ``config set`` to reliably remove the server keys
... regardless of their TOML key representation.
WF14 Clean Global Config
E2E Suite Teardown
*** Test Cases ***
WF14 E2E Server Config Setup
+442 -115
View File
@@ -1,146 +1,473 @@
*** Settings ***
Documentation E2E test for Workflow Example 16: Devcontainer-Driven Development
... (supervised automation profile).
Documentation E2E test for Workflow Example 16: Devcontainer-Driven Development
... (supervised automation profile).
...
... Exercises the devcontainer-specific plan lifecycle:
... devcontainer auto-detection during resource registration,
... lazy container build during plan execution, tool invocation
... routing to the container workspace, and apply writing changes
... back to the host filesystem via bind mount.
... Exercises the devcontainer-specific plan lifecycle:
... devcontainer auto-detection during resource registration,
... lazy container build during plan execution, tool invocation
... routing to the container workspace, and apply writing changes
... back to the host filesystem via bind mount.
...
... **Devcontainer-specific assertions** are strict and contribute
... to AC validation. If one or more AC indicators are missing,
... the test records each unmet AC and fails explicitly rather
... than skipping or passing silently.
... **Devcontainer-specific assertions** are strict and contribute
... to AC validation. If one or more AC indicators are missing,
... the test records each unmet AC and fails explicitly rather
... than skipping or passing silently.
...
... The test is tagged ``tdd_expected_fail`` because devcontainer
... features are not yet fully wired. The
... ``tdd_expected_fail_listener`` inverts the failure to a pass
... in CI until all AC indicators are present.
... The test is tagged ``tdd_expected_fail`` because devcontainer
... features are not yet fully wired. The
... ``tdd_expected_fail_listener`` inverts the failure to a pass
... in CI until all AC indicators are present.
...
... Zero mocking — real CLI, real LLM API keys.
Resource common_e2e.resource
Suite Setup WF16 Suite Setup
... Zero mocking — real CLI, real LLM API keys.
Resource common_e2e.resource
Suite Setup WF16 Suite Setup
Suite Teardown E2E Suite Teardown
Force Tags E2E
Force Tags E2E
*** Variables ***
${ACTION_PREFIX} local/wf16-devcontainer-action
${ACTION_PREFIX} local/wf16-devcontainer-action
${RESOURCE_PREFIX} local/wf16-devcontainer-repo
${PROJECT_PREFIX} local/wf16-devcontainer-project
${PROJECT_PREFIX} local/wf16-devcontainer-project
*** Keywords ***
WF16 Suite Setup
[Documentation] E2E Suite Setup plus database initialisation, unique suffix
... generation, and dynamic actor selection for WF16 tests.
[Tags] tdd_issue tdd_issue_4188 tdd_expected_fail
E2E Suite Setup
# Initialise the database so CLI commands work in all tests.
# Use --force because the workspace may already contain an initialised project.
${init}= Run CleverAgents Command init --force --yes
Should Be Equal As Integers ${init.rc} 0
# Generate a unique suffix for resource/project names to avoid UNIQUE
# constraint collisions when tests run in parallel or are re-run
# against the same database (uuid4 provides ~4 billion possibilities).
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
Set Suite Variable ${ACTION_NAME} ${ACTION_PREFIX}-${suffix}
Set Suite Variable ${RESOURCE_NAME} ${RESOURCE_PREFIX}-${suffix}
Set Suite Variable ${PROJECT_NAME} ${PROJECT_PREFIX}-${suffix}
# Probe the OpenAI API to confirm the key is usable before selecting actor.
# Falls back to Anthropic when the key is absent or quota-exhausted.
# Uses gpt-4o-mini for cost optimization — WF16 exercises plan lifecycle
# mechanics (not LLM quality), so a smaller model suffices.
${actor}= Resolve LLM Actor openai_model=openai/gpt-4o-mini
Set Suite Variable ${SELECTED_ACTOR} ${actor}
[Documentation] E2E Suite Setup plus database initialisation, unique suffix
... generation, and dynamic actor selection for WF16 tests.
E2E Suite Setup
# Initialise the database so CLI commands work in all tests.
# Use --force because the workspace may already contain an initialised project.
${init}= Run CleverAgents Command init --force --yes
Should Be Equal As Integers ${init.rc} 0
# Generate a unique suffix for resource/project names to avoid UNIQUE
# constraint collisions when tests run in parallel or are re-run
# against the same database (uuid4 provides ~4 billion possibilities).
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
Set Suite Variable ${ACTION_NAME} ${ACTION_PREFIX}-${suffix}
Set Suite Variable ${RESOURCE_NAME} ${RESOURCE_PREFIX}-${suffix}
Set Suite Variable ${PROJECT_NAME} ${PROJECT_PREFIX}-${suffix}
# Pick an actor that matches available API keys.
# Prefer OpenAI first to reduce Anthropic credit-quota flakiness.
${has_openai}= Evaluate bool(__import__('os').environ.get('OPENAI_API_KEY', ''))
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
IF ${has_openai}
# Use gpt-4o-mini for cost optimization — WF16 exercises plan lifecycle
# mechanics (not LLM quality), so a smaller model suffices.
${actor}= Set Variable openai/gpt-4o-mini
ELSE IF ${has_anthropic}
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
ELSE
${actor}= Set Variable openai/gpt-4o-mini
END
Set Suite Variable ${SELECTED_ACTOR} ${actor}
Create Devcontainer Repo
[Documentation] Create a temp git repo with devcontainer configuration.
... Returns the path to the created repository.
${repo}= Create Temp Git Repo wf16-devcontainer-${RUN_SUFFIX}
Create Directory ${repo}${/}.devcontainer
Create Directory ${repo}${/}src
${devcontainer_json}= Catenate SEPARATOR=\n
... {
... ${SPACE}${SPACE}"name": "wf16-dev",
... ${SPACE}${SPACE}"image": "mcr.microsoft.com/devcontainers/python:3.12@sha256:3de8f04c3748897ff3aa32b11ee18306d6c56556f4d548a276d0ff397b28b9da",
... ${SPACE}${SPACE}"customizations": {
... ${SPACE}${SPACE}${SPACE}${SPACE}"vscode": {
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}"extensions": ["ms-python.python"]
... ${SPACE}${SPACE}${SPACE}${SPACE}}
... ${SPACE}${SPACE}}
... }
Create File ${repo}${/}.devcontainer${/}devcontainer.json ${devcontainer_json}
${app_content}= Catenate SEPARATOR=\n
... """Main application module."""
... ${EMPTY}
... ${EMPTY}
... def main():
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Entry point."""
... ${SPACE}${SPACE}${SPACE}${SPACE}print("Hello from devcontainer app")
... ${SPACE}${SPACE}${SPACE}${SPACE}return 0
Create File ${repo}${/}src${/}app.py ${app_content}
${test_content}= Catenate SEPARATOR=\n
... """Tests for the application."""
... from src.app import main
... ${EMPTY}
... ${EMPTY}
... def test_main():
... ${SPACE}${SPACE}${SPACE}${SPACE}assert main() == 0
Create File ${repo}${/}src${/}test_app.py ${test_content}
Create File ${repo}${/}src${/}__init__.py \n
Create File ${repo}${/}requirements.txt pytest>=7.0\n
${r_add}= Run Process git add . cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc})
${r_commit}= Run Process git commit -m Initial devcontainer project cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc})
RETURN ${repo}
[Documentation] Create a temp git repo with devcontainer configuration.
... Returns the path to the created repository.
${repo}= Create Temp Git Repo wf16-devcontainer-${RUN_SUFFIX}
Create Directory ${repo}${/}.devcontainer
Create Directory ${repo}${/}src
${devcontainer_json}= Catenate SEPARATOR=\n
... {
... ${SPACE}${SPACE}"name": "wf16-dev",
... ${SPACE}${SPACE}"image": "mcr.microsoft.com/devcontainers/python:3.12@sha256:3de8f04c3748897ff3aa32b11ee18306d6c56556f4d548a276d0ff397b28b9da",
... ${SPACE}${SPACE}"customizations": {
... ${SPACE}${SPACE}${SPACE}${SPACE}"vscode": {
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}"extensions": ["ms-python.python"]
... ${SPACE}${SPACE}${SPACE}${SPACE}}
... ${SPACE}${SPACE}}
... }
Create File ${repo}${/}.devcontainer${/}devcontainer.json ${devcontainer_json}
${app_content}= Catenate SEPARATOR=\n
... """Main application module."""
... ${EMPTY}
... ${EMPTY}
... def main():
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Entry point."""
... ${SPACE}${SPACE}${SPACE}${SPACE}print("Hello from devcontainer app")
... ${SPACE}${SPACE}${SPACE}${SPACE}return 0
Create File ${repo}${/}src${/}app.py ${app_content}
${test_content}= Catenate SEPARATOR=\n
... """Tests for the application."""
... from src.app import main
... ${EMPTY}
... ${EMPTY}
... def test_main():
... ${SPACE}${SPACE}${SPACE}${SPACE}assert main() == 0
Create File ${repo}${/}src${/}test_app.py ${test_content}
Create File ${repo}${/}src${/}__init__.py \n
Create File ${repo}${/}requirements.txt pytest>=7.0\n
${r_add}= Run Process git add . cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_add.rc} 0 msg=git add failed (rc=${r_add.rc})
${r_commit}= Run Process git commit -m Initial devcontainer project cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${r_commit.rc} 0 msg=git commit failed (rc=${r_commit.rc})
RETURN ${repo}
WF16 Test Teardown
[Documentation] Log diagnostic context on failure for debugging.
... Captures plan status when a plan ID is available.
${plan_id}= Get Variable Value ${WF16_PLAN_ID} ${EMPTY}
IF '${plan_id}' != ''
${status} ${result}= Run Keyword And Ignore Error
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
IF '${status}' == 'PASS'
Log Teardown plan status: ${result.stdout} WARN
END
END
[Documentation] Log diagnostic context on failure for debugging.
... Captures plan status when a plan ID is available.
${plan_id}= Get Variable Value ${WF16_PLAN_ID} ${EMPTY}
IF '${plan_id}' != ''
${status} ${result}= Run Keyword And Ignore Error
... Run CleverAgents Command plan status ${plan_id} --format json expected_rc=None timeout=30s
IF '${status}' == 'PASS'
Log Teardown plan status: ${result.stdout} WARN
END
END
*** Test Cases ***
WF16 Devcontainer Driven Development Supervised Profile
[Documentation] Supervised-profile workflow exercising devcontainer-specific
... behaviors: auto-detection during resource registration,
... lazy container build during execution, tool invocation
... routing to the container workspace, and apply writing
... changes back to the host filesystem via bind mount.
...
... Devcontainer-specific assertions are enforced via
... explicit AC checks. Missing AC indicators are collected
... and reported as explicit test failures.
...
... Tagged ``tdd_expected_fail`` because devcontainer features
... are not yet fully wired; the listener inverts the failure
... to a CI pass until all AC indicators are present.
[Tags] tdd_issue tdd_issue_4188 tdd_expected_fail tdd_issue_1208
# Timeout budget: 35 minutes test-level timeout. Per-step timeouts sum
# higher in theory, but retries and conditional paths are mutually
# exclusive — realistic worst-case is well under 35 minutes.
[Timeout] 35 minutes
[Teardown] WF16 Test Teardown
Skip If No LLM Keys
${ac_checks_missing}= Evaluate []
Set Test Variable ${WF16_PLAN_ID} ${EMPTY}
[Documentation] Supervised-profile workflow exercising devcontainer-specific
... behaviors: auto-detection during resource registration,
... lazy container build during execution, tool invocation
... routing to the container workspace, and apply writing
... changes back to the host filesystem via bind mount.
...
... Devcontainer-specific assertions are enforced via
... explicit AC checks. Missing AC indicators are collected
... and reported as explicit test failures.
...
... Tagged ``tdd_expected_fail`` because devcontainer features
... are not yet fully wired; the listener inverts the failure
... to a CI pass until all AC indicators are present.
[Tags] tdd_expected_fail tdd_issue tdd_issue_1208
# Timeout budget: 35 minutes test-level timeout. Per-step timeouts sum
# higher in theory, but retries and conditional paths are mutually
# exclusive — realistic worst-case is well under 35 minutes.
[Timeout] 35 minutes
[Teardown] WF16 Test Teardown
Skip If No LLM Keys
${ac_checks_missing}= Evaluate []
Set Test Variable ${WF16_PLAN_ID} ${EMPTY}
# ---- Create fixture repo with devcontainer ----
${repo}= Create Devcontainer Repo
${branch_result}= Run Process git rev-parse --abbrev-ref HEAD cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${branch_result.rc} 0 msg=git rev-parse failed (rc=${branch_result.rc})
${branch}= Strip String ${branch_result.stdout}
Log Fixture repo created at ${repo} on branch ${branch}
# ---- Register resource (AC-3: devcontainer auto-detection) ----
${r_res}= Run CleverAgents Command
... resource add git-checkout ${RESOURCE_NAME}
... --path ${repo} --branch ${branch}
Should Not Contain ${r_res.stdout} Traceback
Should Not Contain ${r_res.stderr} Traceback
Should Not Contain ${r_res.stdout} INTERNAL
Should Not Contain ${r_res.stderr} INTERNAL
Output Should Contain ${r_res} ${RESOURCE_NAME}
# AC-3: Verify devcontainer auto-detection indicators from resource add.
# Spec Example 16 shows "devcontainer" plus "detected (not built)".
${res_combined}= Set Variable ${r_res.stdout}\n${r_res.stderr}
${res_lower}= Evaluate ($res_combined).lower()
${has_devcontainer}= Evaluate 'devcontainer' in $res_lower
${has_detected}= Evaluate 'detected' in $res_lower
${has_not_built}= Evaluate 'not built' in $res_lower
${has_ac3}= Evaluate $has_devcontainer and $has_detected and $has_not_built
IF ${has_ac3}
Log AC-3 verified: resource output shows devcontainer detected (not built) state
ELSE
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-3: resource add output missing explicit detected (not built) devcontainer state']
Log AC-3 unmet: missing explicit detected (not built) devcontainer state in resource output. AC-4, AC-5, and AC-6 are dependent on AC-3 and are likely to also fail. WARN
END
Log Resource registered: ${RESOURCE_NAME}
# ---- Create project ----
${r_proj}= Run CleverAgents Command
... project create ${PROJECT_NAME}
... --resource ${RESOURCE_NAME}
Should Not Contain ${r_proj.stdout} Traceback
Should Not Contain ${r_proj.stderr} Traceback
Should Not Contain ${r_proj.stdout} INTERNAL
Should Not Contain ${r_proj.stderr} INTERNAL
Output Should Contain ${r_proj} ${PROJECT_NAME}
Log Project created: ${PROJECT_NAME}
# ---- Create action (dynamic actor selection) ----
${action_yaml}= Catenate SEPARATOR=\n
... name: ${ACTION_NAME}
... description: Implement feature in devcontainer-enabled project
... definition_of_done: Feature implemented with tests passing
... strategy_actor: ${SELECTED_ACTOR}
... execution_actor: ${SELECTED_ACTOR}
... reusable: true
... read_only: false
${action_path}= Set Variable ${SUITE_HOME}${/}wf16_action.yaml
Create File ${action_path} ${action_yaml}
${r_action}= Run CleverAgents Command
... action create --config ${action_path}
Should Not Contain ${r_action.stdout} Traceback
Should Not Contain ${r_action.stderr} Traceback
Should Not Contain ${r_action.stdout} INTERNAL
Should Not Contain ${r_action.stderr} INTERNAL
Output Should Contain ${r_action} ${ACTION_NAME}
Log Action created: ${ACTION_NAME} with actor ${SELECTED_ACTOR}
# ---- Plan use with supervised automation profile ----
${r_use}= Run CleverAgents Command
... plan use ${ACTION_NAME} ${PROJECT_NAME}
... --automation-profile supervised
... --format json
... expected_rc=None timeout=180s
Should Be Equal As Integers ${r_use.rc} 0
... plan use failed (rc=${r_use.rc}); stderr redacted from assertion message
Should Not Contain ${r_use.stdout} Traceback
Should Not Contain ${r_use.stderr} Traceback
Should Not Contain ${r_use.stdout} INTERNAL
Should Not Contain ${r_use.stderr} INTERNAL
# Parse plan ID from structured JSON first to avoid selecting the wrong
# ULID when output contains multiple identifiers. Only use regex fallback
# if JSON parsing fails and the fallback result is unambiguous.
Output Should Contain ${r_use} plan_id
${plan_id}= Safe Parse Json Field ${r_use.stdout} plan_id
${plan_id}= Set Variable If $plan_id is None ${EMPTY} ${plan_id}
IF '${plan_id}' != ''
${plan_id}= Strip String ${plan_id}
Should Match Regexp ${plan_id} (?i)^[0-9A-HJKMNP-TV-Z]{26}$
ELSE
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-HJKMNP-TV-Z]{26} flags=IGNORECASE
Should Not Be Empty ${plan_ids} msg=Expected plan ID in plan use output
${unique_plan_ids}= Evaluate list(dict.fromkeys($plan_ids))
${plan_id_count}= Evaluate len($unique_plan_ids)
Should Be Equal As Integers ${plan_id_count} 1
... msg=Ambiguous ULID fallback while extracting plan ID: ${unique_plan_ids}
${plan_id}= Set Variable ${unique_plan_ids}[0]
END
# Verify automation profile was accepted via JSON output
${resolved_profile}= Safe Parse Json Field ${r_use.stdout} automation_profile
${resolved_profile}= Set Variable If $resolved_profile is None ${EMPTY} ${resolved_profile}
IF '${resolved_profile}' != ''
Should Be Equal As Strings ${resolved_profile} supervised
ELSE
# Fallback: check combined output for "supervised" profile name
Output Should Contain ${r_use} supervised
END
Set Test Variable ${WF16_PLAN_ID} ${plan_id}
Log Plan created: ${plan_id} with supervised profile
# ---- Strategize + Execute ----
# The first plan execute call advances through all pending phases
# (Strategize → Execute) in a single invocation.
${r_strat_first}= Run CleverAgents Command
... plan execute ${plan_id}
... --format json
... expected_rc=None timeout=180s
${r_strat}= Set Variable ${r_strat_first}
IF ${r_strat_first.rc} != 0
Log First execute returned rc=${r_strat_first.rc}; retrying once WARN
${r_strat_retry}= Run CleverAgents Command
... plan execute ${plan_id}
... --format json
... expected_rc=None timeout=300s
IF ${r_strat_retry.rc} != 0
Fail WF16 execute instability after retry (first pass): rc1=${r_strat_first.rc}, rc2=${r_strat_retry.rc} (stderr redacted)
ELSE
${r_strat}= Set Variable ${r_strat_retry}
END
END
Should Not Contain ${r_strat.stdout} Traceback
Should Not Contain ${r_strat.stderr} Traceback
Should Not Contain ${r_strat.stdout} INTERNAL
Should Not Contain ${r_strat.stderr} INTERNAL
${strat_first_combined}= Set Variable ${r_strat_first.stdout}\n${r_strat_first.stderr}
${strat_first_lower}= Evaluate ($strat_first_combined).lower()
Log First execute completed: rc=${r_strat.rc}
# AC-4: Verify lazy devcontainer build indicators are present in the
# first execute call (proof of lazy build trigger on first execution).
${has_build_first}= Evaluate 'building' in $strat_first_lower and 'devcontainer' in $strat_first_lower
IF ${has_build_first}
Log AC-4 verified: first execute output shows lazy devcontainer build
ELSE
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-4: first execute output missing lazy devcontainer build indicator']
Log AC-4 unmet: first execute output missing lazy devcontainer build indicator WARN
END
# Determine whether a second execute call is needed; only re-execute when
# the first execute did not reach a ready-for-apply state.
${r_status_after_first}= Run CleverAgents Command
... plan status ${plan_id}
... --format json
... expected_rc=None timeout=180s
Should Be Equal As Integers ${r_status_after_first.rc} 0
... plan status after first execute failed (rc=${r_status_after_first.rc}); stderr redacted from assertion message
${phase_after_first}= Safe Parse Json Field ${r_status_after_first.stdout} phase
${phase_after_first}= Set Variable If $phase_after_first is None ${EMPTY} ${phase_after_first}
${state_after_first}= Safe Parse Json Field ${r_status_after_first.stdout} processing_state
${state_after_first}= Set Variable If $state_after_first is None ${EMPTY} ${state_after_first}
${phase_after_first_norm}= Evaluate $phase_after_first.strip().lower()
${state_after_first_norm}= Evaluate $state_after_first.strip().lower()
${is_apply_phase}= Evaluate $phase_after_first_norm == 'apply'
${is_execute_complete}= Evaluate $phase_after_first_norm == 'execute' and $state_after_first_norm in ['complete', 'completed']
${is_already_applied}= Evaluate $state_after_first_norm == 'applied'
${ready_for_apply}= Evaluate $is_apply_phase or $is_execute_complete or $is_already_applied
${needs_second_execute}= Evaluate not $ready_for_apply
# ---- Defensive re-execute (AC-4: lazy container build, AC-5: container routing) ----
# The first call typically completes both Strategize and Execute phases.
# This second call is a defensive re-check — it is a no-op if both
# phases already completed, but ensures execution finishes if the first
# call only advanced through Strategize.
${r_exec}= Set Variable ${NONE}
IF ${needs_second_execute}
${r_exec}= Run CleverAgents Command
... plan execute ${plan_id}
... --format json
... expected_rc=None timeout=300s
IF ${r_exec.rc} != 0
Log Conditional re-check execute returned rc=${r_exec.rc}; retrying once WARN
${r_exec_retry}= Run CleverAgents Command
... plan execute ${plan_id}
... --format json
... expected_rc=None timeout=300s
IF ${r_exec_retry.rc} != 0
Fail WF16 execute instability after retry (conditional re-check): rc1=${r_exec.rc}, rc2=${r_exec_retry.rc} (stderr redacted)
ELSE
${r_exec}= Set Variable ${r_exec_retry}
END
END
Should Not Contain ${r_exec.stdout} Traceback
Should Not Contain ${r_exec.stderr} Traceback
Should Not Contain ${r_exec.stdout} INTERNAL
Should Not Contain ${r_exec.stderr} INTERNAL
Log Conditional second execute completed: rc=${r_exec.rc}
ELSE
Log Second execute skipped: first execute already reached ready-for-apply state
END
# Combine output from both execute calls — devcontainer indicators may
# appear in either invocation depending on which call runs each phase.
IF ${needs_second_execute}
${exec_combined}= Set Variable ${r_strat.stdout}\n${r_strat.stderr}\n${r_exec.stdout}\n${r_exec.stderr}
ELSE
${exec_combined}= Set Variable ${r_strat.stdout}\n${r_strat.stderr}
END
# AC-5: Require concrete routing evidence with both explicit devcontainer
# identity and explicit container workspace path evidence.
${has_dc_resource}= Evaluate bool(__import__('re').search(r'(?im)resource\s*:\s*[^\n]*\(devcontainer-instance\)', $exec_combined))
${has_dc_resolution}= Evaluate bool(__import__('re').search(r'(?im)resolved via\s*:\s*nearest-ancestor devcontainer', $exec_combined))
${has_dc_environment}= Evaluate bool(__import__('re').search(r'(?im)environment\s*:\s*devcontainer\b', $exec_combined))
${has_dc_identity}= Evaluate $has_dc_resource or ($has_dc_resolution and $has_dc_environment)
${has_workspace_header}= Evaluate bool(__import__('re').search(r'(?im)workspace\s*:\s*/workspaces?/', $exec_combined))
${has_workspace_tool_path}= Evaluate bool(__import__('re').search(r'(?im)in\s+container\s+/workspaces?/', $exec_combined))
${has_workspace_context}= Evaluate $has_workspace_header or $has_workspace_tool_path
${has_routing}= Evaluate $has_dc_identity and $has_workspace_context
IF ${has_routing}
Log AC-5 verified: devcontainer-specific routing indicators detected in execute output
ELSE
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-5: execute output missing devcontainer-specific routing indicator']
Log AC-5 unmet: missing devcontainer-specific routing indicator in execute output WARN
END
IF ${needs_second_execute}
Log Execute completed: rc=${r_exec.rc}
ELSE
Log Execute completed in first pass (no second execute needed)
END
# ---- Diff (verify non-empty changeset) ----
${r_diff}= Run CleverAgents Command
... plan diff ${plan_id}
... --format json
... expected_rc=None timeout=180s
Should Be Equal As Integers ${r_diff.rc} 0
... plan diff failed (rc=${r_diff.rc}); stderr redacted from assertion message
Should Not Contain ${r_diff.stdout} Traceback
Should Not Contain ${r_diff.stderr} Traceback
Should Not Contain ${r_diff.stdout} INTERNAL
Should Not Contain ${r_diff.stderr} INTERNAL
Should Not Be Empty ${r_diff.stdout} msg=Plan diff produced no output — expected a changeset
Log Diff completed with non-empty output
# ---- Apply (AC-6: host filesystem write verification) ----
# Capture HEAD SHA before apply to detect new commits written by apply
${pre_apply_head}= Run Process git rev-parse HEAD cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${pre_apply_head.rc} 0 msg=git rev-parse HEAD failed before apply (rc=${pre_apply_head.rc})
${head_before}= Strip String ${pre_apply_head.stdout}
${pre_apply_status}= Run Process git status --porcelain cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${pre_apply_status.rc} 0 msg=git status --porcelain failed before apply (rc=${pre_apply_status.rc})
${worktree_before}= Strip String ${pre_apply_status.stdout}
Should Be Empty ${worktree_before} msg=Fixture repository must be clean before apply
Log HEAD before apply: ${head_before}
# Use ``plan apply`` so the plan drives through all three Apply
# sub-transitions synchronously:
# Execute/complete → Apply/queued → Apply/processing → Apply/applied.
# ``plan apply`` with a plan ID calls ``_lifecycle_apply_with_id``
# which completes the full transition to the terminal Apply/applied
# state.
${r_apply}= Run CleverAgents Command
... plan apply --yes --format json ${plan_id}
... expected_rc=None timeout=180s
Should Be Equal As Integers ${r_apply.rc} 0
... plan apply failed (rc=${r_apply.rc}); stderr redacted from assertion message
Should Not Contain ${r_apply.stdout} Traceback
Should Not Contain ${r_apply.stderr} Traceback
Should Not Contain ${r_apply.stdout} INTERNAL
Should Not Contain ${r_apply.stderr} INTERNAL
# Verify apply output contains the plan ID (meaningful verification
# that apply processed the correct plan).
Output Should Contain ${r_apply} ${plan_id}
# Verify the apply phase in JSON output (parse success + phase presence
# are mandatory before phase assertion).
${apply_phase}= Safe Parse Json Field ${r_apply.stdout} phase
${apply_phase}= Set Variable If $apply_phase is None ${EMPTY} ${apply_phase}
Should Not Be Empty ${apply_phase} msg=Could not parse non-empty phase from plan apply JSON
Should Contain ${apply_phase.lower()} apply Plan phase should indicate apply after plan apply
${post_apply_head}= Run Process git rev-parse HEAD cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${post_apply_head.rc} 0 msg=git rev-parse HEAD failed after apply (rc=${post_apply_head.rc})
${head_after}= Strip String ${post_apply_head.stdout}
${post_apply_status}= Run Process git status --porcelain cwd=${repo} timeout=60s on_timeout=kill
Should Be Equal As Integers ${post_apply_status.rc} 0 msg=git status --porcelain failed after apply (rc=${post_apply_status.rc})
${worktree_after}= Strip String ${post_apply_status.stdout}
Log HEAD after apply: ${head_after}
${apply_combined}= Set Variable ${r_apply.stdout}\n${r_apply.stderr}
${apply_lower}= Evaluate ($apply_combined).lower()
${has_bind_mount_mechanism}= Evaluate 'bind mount' in $apply_lower or 'bind-mount' in $apply_lower or 'bind_mount' in $apply_lower
${has_head_advance}= Evaluate $head_before != $head_after
${has_worktree_changes}= Evaluate bool($worktree_after.strip())
${has_host_mutation}= Evaluate $has_head_advance or $has_worktree_changes
${has_ac6}= Evaluate $has_bind_mount_mechanism and $has_host_mutation
IF ${has_ac6}
IF ${has_worktree_changes}
Log AC-6 host mutation evidence (git status --porcelain): ${worktree_after}
END
Log AC-6 verified: bind-mount mechanism signal and concrete host mutation both detected
ELSE
${ac_checks_missing}= Evaluate $ac_checks_missing + ['AC-6: missing bind-mount mechanism signal and/or host mutation evidence after apply']
Log AC-6 unmet: missing bind-mount mechanism signal and/or host mutation evidence after apply WARN
END
Log Apply completed: plan transitioned to apply phase
# ---- Status (terminal state verification) ----
${r_status}= Run CleverAgents Command
... plan status ${plan_id}
... --format json
... expected_rc=None timeout=180s
Should Be Equal As Integers ${r_status.rc} 0
... plan status failed (rc=${r_status.rc}); stderr redacted from assertion message
Should Not Contain ${r_status.stdout} Traceback
Should Not Contain ${r_status.stderr} Traceback
Should Not Contain ${r_status.stdout} INTERNAL
Should Not Contain ${r_status.stderr} INTERNAL
Should Not Be Empty ${r_status.stdout}
Output Should Contain ${r_status} ${plan_id}
# Verify success-only terminal state after apply.
${status_phase}= Safe Parse Json Field ${r_status.stdout} phase
${status_phase}= Set Variable If $status_phase is None ${EMPTY} ${status_phase}
Should Not Be Empty ${status_phase} msg=Could not parse non-empty phase from plan status JSON
Should Contain ${status_phase.lower()} apply
... Plan phase should remain apply after successful plan apply
${status_state}= Safe Parse Json Field ${r_status.stdout} processing_state
${status_state}= Set Variable If $status_state is None ${EMPTY} ${status_state}
IF '${status_state}' != ''
Log Final processing_state: ${status_state}
${state_lower}= Evaluate ($status_state).lower()
Should Be Equal As Strings ${state_lower} applied
... Expected success-only terminal state 'applied' but found: ${status_state}
ELSE
Fail Could not parse processing_state from plan status JSON
END
${missing_ac_count}= Evaluate len($ac_checks_missing)
IF ${missing_ac_count} > 0
${missing_ac_summary}= Evaluate '; '.join($ac_checks_missing)
Fail WF16 AC verification incomplete: ${missing_ac_summary}
END
Log Plan status verified: ${plan_id} in state ${status_state}
+245 -88
View File
@@ -1,116 +1,273 @@
*** Settings ***
Documentation E2E test for Workflow Example 18: Container with Remote Repo
... Clone (trusted profile).
Documentation E2E test for Workflow Example 18: Container with Remote Repo
... Clone (trusted profile).
...
... Exercises resource registration with container-instance and
... --clone-into flag for remote repo clone, two-step project
... creation and linking, action with trusted automation profile,
... plan-level execution environment with fallback priority, and
... full plan lifecycle including apply with container commit/push.
... Exercises resource registration with container-instance and
... --clone-into flag for remote repo clone, two-step project
... creation and linking, action with trusted automation profile,
... plan-level execution environment with fallback priority, and
... full plan lifecycle including apply with container commit/push.
...
... Zero mocking — real CLI, real LLM API keys.
Resource common_e2e.resource
Suite Setup WF18 Suite Setup
... Zero mocking — real CLI, real LLM API keys.
Resource common_e2e.resource
Suite Setup WF18 Suite Setup
Suite Teardown E2E Suite Teardown
Force Tags E2E
Force Tags E2E
*** Variables ***
${ACTION_PREFIX} local/wf18-clone-action
${ACTION_PREFIX} local/wf18-clone-action
${RESOURCE_PREFIX} local/wf18-clone-res
${PROJECT_PREFIX} local/wf18-clone-proj
${PROJECT_PREFIX} local/wf18-clone-proj
*** Keywords ***
WF18 Suite Setup
[Documentation] E2E Suite Setup plus unique suffix generation and dynamic
... actor selection for WF18 tests.
[Tags] tdd_issue tdd_issue_4188
E2E Suite Setup
# Initialise the workspace so CLI commands work.
${init}= Run CleverAgents Command init --force --yes
Should Be Equal As Integers ${init.rc} 0
# Generate a unique suffix for resource/project names to avoid UNIQUE
# constraint collisions on repeated E2E runs or parallel CI.
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
# Pick an actor that matches the available API key.
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
IF ${has_anthropic}
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
ELSE
${actor}= Set Variable openai/gpt-4o
END
Set Suite Variable ${LLM_ACTOR} ${actor}
# Compute unique action name using the run suffix for parallel CI safety.
${action_name}= Set Variable ${ACTION_PREFIX}-${RUN_SUFFIX}
Set Suite Variable ${ACTION_NAME} ${action_name}
[Documentation] E2E Suite Setup plus unique suffix generation and dynamic
... actor selection for WF18 tests.
E2E Suite Setup
# Initialise the workspace so CLI commands work.
${init}= Run CleverAgents Command init --force --yes
Should Be Equal As Integers ${init.rc} 0
# Generate a unique suffix for resource/project names to avoid UNIQUE
# constraint collisions on repeated E2E runs or parallel CI.
${suffix}= Evaluate __import__('uuid').uuid4().hex[:12]
Set Suite Variable ${RUN_SUFFIX} ${suffix}
# Pick an actor that matches the available API key.
${has_anthropic}= Evaluate bool(__import__('os').environ.get('ANTHROPIC_API_KEY', ''))
IF ${has_anthropic}
${actor}= Set Variable anthropic/claude-sonnet-4-20250514
ELSE
${actor}= Set Variable openai/gpt-4o
END
Set Suite Variable ${LLM_ACTOR} ${actor}
# Compute unique action name using the run suffix for parallel CI safety.
${action_name}= Set Variable ${ACTION_PREFIX}-${RUN_SUFFIX}
Set Suite Variable ${ACTION_NAME} ${action_name}
Create Remote Clone Repo
[Documentation] Create a temp git repo simulating a project that would
... be cloned from a remote repository into a container.
${repo}= Create Temp Git Repo wf18-remote-clone-${RUN_SUFFIX}
Create Directory ${repo}${/}src
Create Directory ${repo}${/}deploy
${app_content}= Catenate SEPARATOR=\n
... """Application for remote deployment via container clone."""
... ${EMPTY}
... ${EMPTY}
... class DeploymentManager:
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Manages deployment lifecycle."""
... ${EMPTY}
... ${SPACE}${SPACE}${SPACE}${SPACE}def __init__(self, repo_url, branch="main"):
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.repo_url = repo_url
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.branch = branch
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.deployed = False
... ${EMPTY}
... ${SPACE}${SPACE}${SPACE}${SPACE}def deploy(self):
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}"""Execute deployment."""
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.deployed = True
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"status": "deployed", "branch": self.branch}
Create File ${repo}${/}src${/}deploy_manager.py ${app_content}
${deploy_config}= Catenate SEPARATOR=\n
... # Deployment configuration
... CONTAINER_IMAGE=python:3.12-slim
... CLONE_DEPTH=1
... DEPLOY_TIMEOUT=300
... HEALTH_CHECK_INTERVAL=10
Create File ${repo}${/}deploy${/}config.env ${deploy_config}
# NOTE: This Dockerfile is fixture data only — never built by this test.
# It simulates a project that would be cloned into a container.
${dockerfile}= Catenate SEPARATOR=\n
... FROM python:3.12-slim
... RUN apt-get update && apt-get install -y git
... WORKDIR /workspace
... ARG REPO_URL
... RUN git clone --depth 1 \${REPO_URL} .
... CMD ["python", "-m", "src.deploy_manager"]
Create File ${repo}${/}Dockerfile ${dockerfile}
Create File ${repo}${/}src${/}__init__.py \n
${git_add}= Run Process git add . cwd=${repo}
Should Be Equal As Integers ${git_add.rc} 0 msg=Fixture git add failed: ${git_add.stderr}
${git_commit}= Run Process git commit -m Initial remote-clone container project cwd=${repo}
Should Be Equal As Integers ${git_commit.rc} 0 msg=Fixture git commit failed: ${git_commit.stderr}
RETURN ${repo}
[Documentation] Create a temp git repo simulating a project that would
... be cloned from a remote repository into a container.
${repo}= Create Temp Git Repo wf18-remote-clone-${RUN_SUFFIX}
Create Directory ${repo}${/}src
Create Directory ${repo}${/}deploy
${app_content}= Catenate SEPARATOR=\n
... """Application for remote deployment via container clone."""
... ${EMPTY}
... ${EMPTY}
... class DeploymentManager:
... ${SPACE}${SPACE}${SPACE}${SPACE}"""Manages deployment lifecycle."""
... ${EMPTY}
... ${SPACE}${SPACE}${SPACE}${SPACE}def __init__(self, repo_url, branch="main"):
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.repo_url = repo_url
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.branch = branch
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.deployed = False
... ${EMPTY}
... ${SPACE}${SPACE}${SPACE}${SPACE}def deploy(self):
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}"""Execute deployment."""
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}self.deployed = True
... ${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}return {"status": "deployed", "branch": self.branch}
Create File ${repo}${/}src${/}deploy_manager.py ${app_content}
${deploy_config}= Catenate SEPARATOR=\n
... # Deployment configuration
... CONTAINER_IMAGE=python:3.12-slim
... CLONE_DEPTH=1
... DEPLOY_TIMEOUT=300
... HEALTH_CHECK_INTERVAL=10
Create File ${repo}${/}deploy${/}config.env ${deploy_config}
# NOTE: This Dockerfile is fixture data only — never built by this test.
# It simulates a project that would be cloned into a container.
${dockerfile}= Catenate SEPARATOR=\n
... FROM python:3.12-slim
... RUN apt-get update && apt-get install -y git
... WORKDIR /workspace
... ARG REPO_URL
... RUN git clone --depth 1 \${REPO_URL} .
... CMD ["python", "-m", "src.deploy_manager"]
Create File ${repo}${/}Dockerfile ${dockerfile}
Create File ${repo}${/}src${/}__init__.py \n
${git_add}= Run Process git add . cwd=${repo}
Should Be Equal As Integers ${git_add.rc} 0
... Fixture git add failed: ${git_add.stderr}
${git_commit}= Run Process git commit -m Initial remote-clone container project cwd=${repo}
Should Be Equal As Integers ${git_commit.rc} 0
... Fixture git commit failed: ${git_commit.stderr}
RETURN ${repo}
*** Test Cases ***
WF18 Container With Remote Repo Clone Trusted Profile
[Documentation] Trusted-profile workflow: container-instance resource with
... --clone-into for remote repo clone, two-step project setup,
... plan-level execution environment with fallback priority, and
... full plan lifecycle including apply with container commit/push.
[Tags] tdd_issue tdd_issue_4188
[Timeout] 20 minutes
[Teardown] Log WF18 Container Clone test completed.
Skip If No LLM Keys
[Documentation] Trusted-profile workflow: container-instance resource with
... --clone-into for remote repo clone, two-step project setup,
... plan-level execution environment with fallback priority, and
... full plan lifecycle including apply with container commit/push.
[Timeout] 20 minutes
[Teardown] Log WF18 Container Clone test completed.
Skip If No LLM Keys
# ---- Create fixture repo ----
${repo}= Create Remote Clone Repo
# ---- Register container-instance resource with --clone-into ----
${resource_name}= Set Variable ${RESOURCE_PREFIX}-${RUN_SUFFIX}
${r_resource}= Run CleverAgents Command
... resource add container-instance ${resource_name}
... --image python:3.12-slim
... --clone-into ${repo}:/workspace
... --format plain
Should Not Contain ${r_resource.stdout}${r_resource.stderr} Traceback
Should Not Contain ${r_resource.stdout}${r_resource.stderr} INTERNAL
Output Should Contain ${r_resource} ${resource_name}
Output Should Contain ${r_resource} container-instance
# ---- Create project (two-step: create then link-resource per spec) ----
${project_name}= Set Variable ${PROJECT_PREFIX}-${RUN_SUFFIX}
${r_project_create}= Run CleverAgents Command
... project create ${project_name}
... --format plain
Should Not Contain ${r_project_create.stdout}${r_project_create.stderr} Traceback
Should Not Contain ${r_project_create.stdout}${r_project_create.stderr} INTERNAL
Output Should Contain ${r_project_create} ${project_name}
${r_link}= Run CleverAgents Command
... project link-resource ${project_name} ${resource_name}
... --format plain
Should Not Contain ${r_link.stdout}${r_link.stderr} Traceback
Should Not Contain ${r_link.stdout}${r_link.stderr} INTERNAL
Output Should Contain ${r_link} ${resource_name}
# ---- Create action with dynamic actor selection ----
${action_yaml}= Catenate SEPARATOR=\n
... name: ${ACTION_NAME}
... description: Implement container-based remote deployment with repo cloning
... definition_of_done: Deployment pipeline configured and validated
... strategy_actor: ${LLM_ACTOR}
... execution_actor: ${LLM_ACTOR}
${action_path}= Set Variable ${SUITE_HOME}${/}wf18_action.yaml
Create File ${action_path} ${action_yaml}
${r_action}= Run CleverAgents Command
... action create --config ${action_path}
... --format plain
Should Not Contain ${r_action.stdout}${r_action.stderr} Traceback
Should Not Contain ${r_action.stdout}${r_action.stderr} INTERNAL
Output Should Contain ${r_action} ${ACTION_NAME}
# ---- Plan use with trusted profile, execution environment, and fallback priority ----
# NOTE: The spec Example 18 shows --execution-environment cloud/build-env
# (a resource name). The current CLI implementation validates against the
# ExecutionEnvironment enum which only accepts "host" or "container".
# Resource-name-based resolution is future work; we use "container" here to
# exercise the plan-level fallback routing path (precedence level 4).
${r_use}= Run CleverAgents Command
... plan use ${ACTION_NAME} ${project_name}
... --automation-profile trusted
... --execution-environment container
... --execution-env-priority fallback
... --format plain
... timeout=120s
Should Not Contain ${r_use.stdout}${r_use.stderr} Traceback
Should Not Contain ${r_use.stdout}${r_use.stderr} INTERNAL
Should Not Be Empty ${r_use.stdout} Plan use produced no output
${plan_ids}= Get Regexp Matches ${r_use.stdout} [0-9A-HJKMNP-TV-Z]{26}
Should Not Be Empty ${plan_ids} msg=Expected ULID plan ID in plan use output
${plan_id}= Set Variable ${plan_ids}[0]
Log Plan ID: ${plan_id}
# ---- Strategize ----
# The plan lifecycle uses two sequential ``plan execute`` calls:
# the first advances the plan through the Strategize phase, and
# the second advances it through the Execute phase.
${r_strat}= Run CleverAgents Command
... plan execute ${plan_id}
... --format plain
... timeout=180s
Should Not Contain ${r_strat.stdout}${r_strat.stderr} Traceback
Should Not Contain ${r_strat.stdout}${r_strat.stderr} INTERNAL
Should Not Be Empty ${r_strat.stdout} Strategize produced no output
Output Should Contain ${r_strat} ${plan_id}
# ---- Execute ----
${r_exec}= Run CleverAgents Command
... plan execute ${plan_id}
... --format plain
... timeout=300s
Should Not Contain ${r_exec.stdout}${r_exec.stderr} Traceback
Should Not Contain ${r_exec.stdout}${r_exec.stderr} INTERNAL
Should Not Be Empty ${r_exec.stdout} Execute produced no output
Output Should Contain ${r_exec} ${plan_id}
# Verify container start and remote repo clone evidence (AC-4).
# Soft assertion: LLM output is non-deterministic so we warn rather than fail.
${exec_combined}= Set Variable ${r_exec.stdout}\n${r_exec.stderr}
${exec_lower}= Evaluate ($exec_combined).lower()
${has_container_evidence}= Evaluate 'container' in $exec_lower or 'clone' in $exec_lower or 'workspace' in $exec_lower or 'starting' in $exec_lower
Log Container/clone evidence in execute output: ${has_container_evidence}
Run Keyword And Warn On Failure Should Be True ${has_container_evidence}
... Execute output should contain evidence of container start or clone operation
# Verify execution environment resolution — plan fallback, precedence level 4 (AC-5).
${has_env_resolution}= Evaluate 'fallback' in $exec_lower or 'execution' in $exec_lower or 'environment' in $exec_lower or 'precedence' in $exec_lower or 'resolved' in $exec_lower
Log Execution environment resolution evidence: ${has_env_resolution}
Run Keyword And Warn On Failure Should Be True ${has_env_resolution}
... Execute output should contain execution environment resolution evidence (fallback/precedence)
# ---- Diff ----
${r_diff}= Run CleverAgents Command
... plan diff ${plan_id}
... --format plain
... timeout=60s
Should Not Contain ${r_diff.stdout}${r_diff.stderr} Traceback
Should Not Contain ${r_diff.stdout}${r_diff.stderr} INTERNAL
Should Not Be Empty ${r_diff.stdout} Plan diff produced no output
# ---- Capture pre-apply commit count ----
${pre_log}= Run Process git log --oneline cwd=${repo}
Should Be Equal As Integers ${pre_log.rc} 0
... Pre-apply git log failed: ${pre_log.stderr}
${pre_commit_count}= Get Line Count ${pre_log.stdout}
Log Pre-apply commit count: ${pre_commit_count}
# ---- Apply ----
${r_apply}= Run CleverAgents Command
... plan apply --yes ${plan_id}
... --format plain
... timeout=120s
Should Not Contain ${r_apply.stdout}${r_apply.stderr} Traceback
Should Not Contain ${r_apply.stdout}${r_apply.stderr} INTERNAL
Should Not Be Empty ${r_apply.stdout} Apply produced no output
Output Should Contain ${r_apply} ${plan_id}
# Verify apply output references container commit and push (AC-6).
# Soft assertion: LLM output is non-deterministic so we warn rather than fail.
${apply_combined}= Set Variable ${r_apply.stdout}\n${r_apply.stderr}
${apply_lower}= Evaluate ($apply_combined).lower()
${has_apply_evidence}= Evaluate 'commit' in $apply_lower or 'push' in $apply_lower or 'applied' in $apply_lower or 'container' in $apply_lower or 'origin' in $apply_lower
Log Apply container commit/push evidence: ${has_apply_evidence}
Run Keyword And Warn On Failure Should Be True ${has_apply_evidence}
... Apply output should contain evidence of commit/push from container
# ---- Status — verify terminal state ----
${r_status}= Run CleverAgents Command
... plan status ${plan_id}
... --format plain
... timeout=60s
Should Not Contain ${r_status.stdout}${r_status.stderr} Traceback
Should Not Contain ${r_status.stdout}${r_status.stderr} INTERNAL
Should Not Be Empty ${r_status.stdout} Plan status produced no output
Output Should Contain ${r_status} ${plan_id}
# Verify plan reached a terminal state (applied phase)
${status_lower}= Evaluate ($r_status.stdout).lower()
${is_terminal}= Evaluate 'applied' in $status_lower or 'complete' in $status_lower or 'done' in $status_lower or 'finished' in $status_lower or 'apply' in $status_lower
Should Be True ${is_terminal}
... Plan status should indicate a terminal/applied state after apply
# ---- Verify repo state after apply ----
${post_log}= Run Process git log --oneline cwd=${repo}
Should Be Equal As Integers ${post_log.rc} 0
... Post-apply git log failed: ${post_log.stderr}
Log Git log after apply: ${post_log.stdout}
${post_commit_count}= Get Line Count ${post_log.stdout}
# Apply may or may not create new commits depending on whether the LLM
# generated file changes. Log the delta for diagnostics but only
# hard-assert that the repo is still valid (at least the fixture commits).
${delta}= Evaluate ${post_commit_count} - ${pre_commit_count}
Log Commit count delta after apply: ${delta} (pre=${pre_commit_count}, post=${post_commit_count})
Should Be True ${post_commit_count} >= ${pre_commit_count}
... Repo commit count should not decrease after apply (pre=${pre_commit_count}, post=${post_commit_count})
# Verify fixture files are still present in the repo after the full lifecycle.
File Should Exist ${repo}${/}src${/}deploy_manager.py
File Should Exist ${repo}${/}Dockerfile
+4 -3
View File
@@ -180,9 +180,10 @@ def _artifacts_output() -> None:
# Combine: return value or captured stdout
effective = output or buf.getvalue().strip()
parsed = json.loads(effective)
assert parsed["changeset_id"] == "robot-cs-001"
assert "sandbox-robot-001" in parsed["sandbox_refs"]
assert len(parsed["files_changed"]) == 2
print(parsed)
assert parsed["data"]["changeset_id"] == "robot-cs-001"
assert "sandbox-robot-001" in parsed["data"]["sandbox_refs"]
assert len(parsed["data"]["files_changed"]) == 2
print("artifacts-output-ok")
+67
View File
@@ -0,0 +1,67 @@
*** Settings ***
Documentation Integration tests for plan diff and artifacts output.
... Tests D0b.apply features: diff rendering, artifacts summary,
... empty changeset guard, apply summary persistence, and merge failure handling.
Resource ${CURDIR}/common.resource
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
*** Variables ***
${HELPER_SCRIPT} robot/helper_plan_diff_artifacts.py
*** Test Cases ***
Plan Diff Renders Changeset Summary
[Documentation] Verify plan diff shows file changes grouped by path
[Tags] diff plan changeset
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} diff-output cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} diff-output-ok
Plan Artifacts Shows Changeset Metadata
[Documentation] Verify plan artifacts shows changeset ID, sandbox refs, and file list
[Tags] artifacts plan changeset
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} artifacts-output cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} artifacts-output-ok
Empty Changeset Guard Blocks Apply
[Documentation] Verify apply is blocked when changeset is empty
[Tags] guard plan apply
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} empty-guard cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} empty-guard-ok
Empty Changeset Guard Allows With Flag
[Documentation] Verify apply proceeds when --allow-empty is set
[Tags] guard plan apply
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} empty-guard-allow cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} empty-guard-allow-ok
Apply Summary Persistence
[Documentation] Verify apply summary metadata is persisted into plan
[Tags] apply plan metadata
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} apply-summary cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} apply-summary-ok
Merge Failure Handling
[Documentation] Verify merge failure sets plan to errored with conflict details
[Tags] merge plan error
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} merge-failure cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} merge-failure-ok
Diff JSON Format Output
[Documentation] Verify diff JSON format includes entries and summary
[Tags] diff plan json
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} diff-json cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} diff-json-ok
Diff YAML Format Output
[Documentation] Verify diff YAML format includes changeset metadata
[Tags] diff plan yaml
${result}= Run Process ${PYTHON} ${HELPER_SCRIPT} diff-yaml cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} diff-yaml-ok
+56 -66
View File
@@ -4,95 +4,85 @@ Library String
*** Test Cases ***
TUI Headless Startup Check Returns JSON
[Tags] tdd_issue tdd_issue_4193 tdd_issue_4297
${result}= Run Process ${PYTHON} -m cleveragents tui --headless shell=False stderr=STDOUT env:CLEVERAGENTS_DATABASE_URL=sqlite:///:memory:
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} textual_available
Should Contain ${result.stdout} default_persona
TUI Headless Works When Shell Disabled
[Tags] tdd_issue tdd_issue_4193 tdd_issue_4297
${result}= Run Process ${PYTHON} -m cleveragents tui --headless shell=False stderr=STDOUT env:CLEVERAGENTS_DATABASE_URL=sqlite:///:memory: env:CLEVERAGENTS_DISABLE_SHELL_MODE=1
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} textual_available
Should Contain ${result.stdout} default_persona
TUI Input Mode Router And Prompt Widget Behavior
[Tags] tdd_issue tdd_issue_4193 tdd_issue_4297 tdd_expected_fail
${script}= Catenate SEPARATOR=\n
... from cleveragents.tui.input.modes import InputModeRouter
... from cleveragents.tui.widgets.prompt import PromptInput
... prompt = PromptInput()
... prompt.value = "hello"
... prompt.action_submit()
... assert prompt.value == "", f"Expected cleared prompt after submit, got: {prompt.value!r}"
... print("prompt-widget-ok")
${result}= Run Process ${PYTHON} -c ${script} shell=False
... payload = prompt.consume_text()
... assert payload.text == "hello"
... assert prompt.value == ""
... router = InputModeRouter(command_handler=lambda raw: f"ok:{raw}")
... command_result = router.process("/help")
... assert command_result.mode.value == "command"
... assert command_result.command_result == "ok:help"
... normal_result = router.process("inspect @README.md")
... assert normal_result.mode.value == "normal"
... print("router-widget-ok")
${result}= Run Process ${PYTHON} -c ${script} shell=False stderr=STDOUT
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} prompt-widget-ok
Should Contain ${result.stdout} router-widget-ok
TUI Chat History Widget Behavior
[Tags] tdd_issue tdd_issue_4193 tdd_issue_4297 tdd_expected_fail
${script}= Catenate SEPARATOR=\n
... from cleveragents.tui.widgets.history import ChatHistory
... from cleveragents.tui.events import MessageEvent, MessageType
... from textual.app import App
... import asyncio
... class TestApp(App):
... async def on_mount(self):
... hist = ChatHistory()
... await self.mount(hist)
... msg_ev = MessageEvent(msg_type=MessageType.USER, content="Test message")
... await hist.post_message_event(msg_ev)
... # Just verify it doesn't crash
... self.exit(0)
... app = TestApp()
... asyncio.run(app.run_async())
... print("chat-history-ok")
${result}= Run Process ${PYTHON} -c ${script} shell=False
TUI Headless Includes Router Help Payload
${result}= Run Process ${PYTHON} -m cleveragents tui --headless shell=False stderr=STDOUT env:CLEVERAGENTS_DATABASE_URL=sqlite:///:memory:
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} chat-history-ok
Should Contain ${result.stdout} "help"
Should Contain ${result.stdout} /persona
TUI Screen Stack And Overlay Behavior
[Tags] tdd_issue tdd_issue_4193 tdd_issue_4297 tdd_expected_fail
Slash Command Overlay Renders Descriptions
[Documentation] Verify SlashCommandOverlay.set_commands renders descriptions alongside names
${script}= Catenate SEPARATOR=\n
... from textual.app import App
... from textual.screen import Screen
... from cleveragents.tui.screens.examples import ExamplesScreen
... import asyncio
... class TestApp(App):
... async def on_mount(self):
... # Push examples screen as overlay
... examples = ExamplesScreen()
... examples.can_replace = False # Makes it an overlay
... await self.push_screen(examples)
... # Verify stack depth
... assert len(self.screen_stack) == 2, "Expected 2 screens after overlay push"
... # Pop overlay
... await self.pop_screen()
... assert len(self.screen_stack) == 1, "Expected 1 screen after overlay pop"
... self.exit(0)
... app = TestApp()
... asyncio.run(app.run_async())
... print("screen-stack-ok")
${result}= Run Process ${PYTHON} -c ${script} shell=False
... import importlib
... from unittest.mock import patch
... import cleveragents.tui.widgets.slash_command_overlay as _sco_mod
... from cleveragents.tui.slash_catalog import SLASH_COMMAND_SPECS, slash_command_specs
... with patch("importlib.import_module", side_effect=ImportError("no textual")):
... ${SPACE}${SPACE}${SPACE}${SPACE}importlib.reload(_sco_mod)
... overlay = _sco_mod.SlashCommandOverlay()
... overlay.set_commands("", slash_command_specs())
... text = overlay._text
... assert "${SPACE}${SPACE}/session:create" in text, f"Missing /session:create in: {text}"
... assert "Create a new session tab" in text, f"Missing description in: {text}"
... overlay2 = _sco_mod.SlashCommandOverlay()
... overlay2.set_commands("settings", slash_command_specs())
... text2 = overlay2._text
... assert "${SPACE}${SPACE}/settings" in text2, f"Missing /settings in: {text2}"
... assert "Open settings" in text2, f"Missing settings description in: {text2}"
... print("slash-overlay-descriptions-ok")
${result}= Run Process ${PYTHON} -c ${script} shell=False stderr=STDOUT
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} screen-stack-ok
Should Contain ${result.stdout} slash-overlay-descriptions-ok
TUI Shell Mode Detection
[Tags] tdd_issue tdd_issue_4193 tdd_issue_4297 tdd_expected_fail
TUI Help Panel Context Switching
${script}= Catenate SEPARATOR=\n
... import os
... # Test shell mode enabled (default)
... os.environ.pop('CLEVERAGENTS_DISABLE_SHELL_MODE', None)
... from cleveragents.config import get_config
... cfg = get_config()
... assert cfg.enable_shell_mode is True, f"Expected shell mode enabled, got: {cfg.enable_shell_mode}"
... # Test shell mode disabled
... os.environ['CLEVERAGENTS_DISABLE_SHELL_MODE'] = '1'
... cfg = get_config()
... assert cfg.enable_shell_mode is False, f"Expected shell mode disabled, got: {cfg.enable_shell_mode}"
... print("shell-mode-detection-ok")
${result}= Run Process ${PYTHON} -c ${script} shell=False
... from types import SimpleNamespace
... from features.steps import tui_app_coverage_steps as steps
... context = SimpleNamespace(add_cleanup=lambda fn: None)
... steps._install_mock_textual(context)
... steps.step_create_mock_deps(context)
... steps.step_instantiate_app(context)
... steps.step_call_on_mount(context)
... steps.step_set_prompt_text(context, "/persona list")
... steps.step_call_action_help(context)
... steps.step_help_panel_contains(context, "Help: Slash Commands")
... steps.step_set_prompt_text(context, "!ls")
... steps.step_call_action_help(context)
... steps.step_help_panel_contains(context, "Help: Shell Mode")
... steps._restore_modules(context)
... steps._cleanup_tmpdir(context)
... print("tui-help-panel-ok")
${result}= Run Process ${PYTHON} -c ${script} shell=False stderr=STDOUT
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} shell-mode-detection-ok
Should Contain ${result.stdout} tui-help-panel-ok
@@ -40,6 +40,8 @@ from cleveragents.core.exceptions import (
ValidationError,
)
from cleveragents.domain.models.core.change import (
ChangeEntry,
ChangeOperation,
SpecChangeSet,
)
from cleveragents.domain.models.core.correction import CorrectionAttemptRecord
@@ -1040,20 +1042,67 @@ class PlanApplyService:
def _resolve_changeset(self, plan: Plan) -> SpecChangeSet | None:
"""Resolve the ChangeSet for a plan.
Checks the changeset store first (if available), then builds
an empty SpecChangeSet stub if the plan has a changeset_id but
no store is configured.
Resolution order:
1. Changeset store (if wired e.g. in server mode or tests).
2. Reconstruct from ``plan.error_details["changeset_entries_json"]``
serialised by ``PlanExecutor._run_execute_with_stub()``.
3. Empty ``SpecChangeSet`` stub (changeset ID known but no entries).
"""
if plan.changeset_id is None:
return None
# Try store lookup
# Priority 1: explicit store lookup
if self._changeset_store is not None:
stored = self._changeset_store.get(plan.changeset_id)
if stored is not None:
return stored
# Fallback: empty changeset with the known ID
# Priority 2: reconstruct from entries persisted in plan metadata.
# PlanExecutor._run_execute_with_stub() serialises lightweight
# ChangeSetEntry objects here so that plan diff works without a
# dedicated changeset_store being wired into the CLI.
if isinstance(plan.error_details, dict):
entries_json = plan.error_details.get("changeset_entries_json")
if entries_json:
try:
_op_map: dict[str, ChangeOperation] = {
"create": ChangeOperation.CREATE,
"modify": ChangeOperation.MODIFY,
"delete": ChangeOperation.DELETE,
"rename": ChangeOperation.RENAME,
"move": ChangeOperation.RENAME,
}
entries_data: list[dict[str, Any]] = json.loads(entries_json)
domain_entries: list[ChangeEntry] = [
ChangeEntry(
plan_id=plan.identity.plan_id,
resource_id=str(
e.get("resource_id") or plan.identity.plan_id
),
tool_name=str(e.get("tool_name") or "llm_execute"),
operation=_op_map.get(
str(e.get("operation", "modify")).lower(),
ChangeOperation.MODIFY,
),
path=str(e["path"]),
before_hash=e.get("before_hash"),
after_hash=e.get("after_hash"),
)
for e in entries_data
]
return SpecChangeSet(
changeset_id=plan.changeset_id,
plan_id=plan.identity.plan_id,
entries=domain_entries,
)
except Exception:
self._logger.debug(
"Failed to reconstruct SpecChangeSet from plan metadata",
plan_id=plan.identity.plan_id,
exc_info=True,
)
# Priority 3: empty stub so callers know the plan has a changeset ID
return SpecChangeSet(
changeset_id=plan.changeset_id,
plan_id=plan.identity.plan_id,
@@ -1048,6 +1048,26 @@ class PlanExecutor:
"sandbox_refs_count": str(len(result.sandbox_refs)),
"mode": "stub",
}
# Persist lightweight changeset entries so that plan diff can
# reconstruct a SpecChangeSet without a changeset_store being
# wired into PlanApplyService. The JSON is stored in
# error_details (which already acts as a general metadata bag)
# and read back by PlanApplyService._resolve_changeset().
if result.changeset.entries:
entries_data = [
{
"operation": e.operation,
"path": e.path,
"resource_id": e.resource_id,
"tool_name": e.tool_name,
"before_hash": e.before_hash,
"after_hash": e.after_hash,
}
for e in result.changeset.entries
]
plan.error_details["changeset_entries_json"] = json.dumps(
entries_data, default=str
)
plan.timestamps.updated_at = datetime.now(tz=UTC)
# Spawn and execute child subplans from spawn decisions
File diff suppressed because it is too large Load Diff