Address review-driven fixes across actor schema, preflight guardrails, docs/examples,
and Behave/Robot coverage: unify preflight warning behavior with shared role-warning logic,
resolve actor-name to config payloads in production preflight flow, harden response_format
validation/coercion edge cases, extract duplicated helper logic, and expand negative-path
test coverage. Also fix cross-scenario patcher leakage in step modules to eliminate
full-run-only coverage failures.
- Add `role_hint` and `response_format` support to actor schema.
- Add non-fatal estimation-role compatibility warnings in actor registration CLI flows.
- Add preflight warning path when `estimation` actor is missing `response_format`.
- Add `examples/actors/estimator.yaml` and update actor examples documentation/tests.
- Update integration helper expectations (m1/m2/m3/m6) for missing provider config in local test env.
ISSUES CLOSED: #650
Add 3 deferred virtual resource types (remote, submodule, symlink) with
equivalence metadata for physical-to-virtual resource linking.
Depends on: #662 (child_types reference types introduced by #662)
- Type definitions extracted to _resource_registry_virtual_deferred.py
for consistency with _resource_registry_virtual.py (#329)
- YAML configs with equivalence criteria per spec, spec reference comments
- Bootstrap registration via BUILTIN_TYPES spread, hidden from
resource add scaffolding (user_addable: false)
- Equivalence structural validation in ResourceTypeSpec model validator:
criteria must be a non-empty list of non-empty strings; virtual types
must have sandbox_strategy=none, user_addable=false, handler=None,
all capabilities false
- Behave tests (52 scenarios), Robot tests (7), ASV benchmarks
- DB roundtrip tests verifying virtual types survive bootstrap persistence
- Negative tests: missing equivalence/name/kind, manual add rejection
for all 3 virtual types (register_resource guard), invalid criteria
elements (non-string, empty string)
ISSUES CLOSED: #331
_get_plan_executor() created a second PlanLifecycleService Factory
instance with its own in-memory _plans cache. After the executor's
run_strategize() advanced the plan to execute/queued (via
auto_progress), the CLI handler's separate service instance returned
stale strategize/queued state from its cache, causing spurious
"Plan is not in an executable state" errors.
Fix: _get_plan_executor() now accepts an optional lifecycle_service
parameter; the plan execute handler passes its own service instance so
both share the same cache.
Also addressed review feedback:
- Improved type safety: lifecycle_service parameter typed as
PlanLifecycleService | None instead of Any | None.
- Added BDD regression test verifying the lifecycle service is shared
between the CLI handler and the executor.
- Updated reference documentation to reflect the type annotation change.
ISSUES CLOSED: #1026
start_strategize() built its action_registry from the in-memory _actions
dict only, so fresh CLI processes (e.g. `plan execute` after a separate
`plan use`) failed with PreflightRejection: "Action not found in
registry". The PreflightRejection (extending bare Exception, not
CleverAgentsError) escaped the CLI error handler, producing an opaque
"Error [500] INTERNAL: An unexpected error occurred" message.
Additionally, `plan execute` only transitioned the plan to
execute/queued without running the execute phase, leaving the plan stuck
and making `lifecycle-apply` fail with PlanNotReadyError.
Changes:
- start_strategize() loads the plan's action from the persistence layer
(via get_action()) before building the preflight action_registry.
- plan execute CLI catches PreflightRejection for user-friendly errors.
- plan execute CLI runs the execute phase inline via PlanExecutor so
the plan progresses through execute/queued -> execute/complete.
- lifecycle-apply CLI handles plans already auto-progressed to
apply/queued by complete_execute()'s auto_progress() call.
ISSUES CLOSED: #746
Implemented Robot Framework E2E test suite for M6 autonomy hardening
acceptance criteria. Tests exercise the real CleverAgents CLI with zero
mocking, covering session lifecycle, automation profiles, project setup,
plan lifecycle via A2A facade, guard enforcement, and a full autonomy
acceptance flow. LLM-dependent tests use Skip If No LLM Keys for
graceful degradation when API keys are unavailable.
Hardened shared E2E keywords (common_e2e.resource):
- Safe JSON parsing with rfind-bounded extraction and error wrapping (C1, M1)
- Multi-object fallback: last-line reverse scan for multi-JSON output (M5)
- Moved Safe Parse Json Field to common_e2e.resource for reuse (L1)
- Migrated deprecated Run Keyword If to IF/ELSE blocks (M4)
- Added cwd parameter to Run CleverAgents Command (M6)
- API key protection via inline evaluation instead of RF variables (S1)
- Git return-code assertions in Create Temp Git Repo (L1)
- Removed unused Collections library import (L2)
- Warning log on directory removal failure instead of silent ignore (L6)
Hardened m6_acceptance.robot:
- Force Tags E2E instead of per-test [Tags] (L3)
- Per-test [Teardown] for resource cleanup, including Init test (M2, M9, L5)
- Collision-safe uuid4 hex suffix instead of randint (L7)
- Initialized session_id to EMPTY before test body for safe teardown (L2)
- ELSE branches with WARN log on conditional assertions (M3, M4, M6)
- Guard enforcement checks specific automation-profile fields (M5)
- Strengthened ci assertions with case-sensitive matching and JSON
field parsing for config get verification (M9)
- Eliminated Python code injection via string interpolation (H2)
- Added assertions to Full Flow Apply Step keyword (H4)
- Removed redundant config set in Guard test (M10)
- Accurate CHANGELOG entry describing actual test scope (C2)
Post-review fixes applied:
- Verify all 8 built-in profiles (manual, review, supervised, cautious,
trusted, auto, ci, full-auto) instead of 4 (M1)
- Session delete confirms removal via re-list (M2)
- Full Flow Apply Step verifies plan phase transition (M3)
- Plan execute output asserts plan_id presence + phase parsing (M4)
- Safe Parse Json Field gains two-strategy approach: outer-bracket
extraction then last-line reverse scan fallback (M5)
- Removed redundant automation profile reset in Config test (L1)
- JSON-quoted assertions ("ci", "auto") prevent false-positive
substring matches on short profile names (L3)
New E2E tests covering remaining acceptance criteria:
- Guard enforcement with custom profile: registers a profile with
explicit denylist, budget cap, and tool-call limits, then verifies
all guard fields via automation-profile show (AC-4 / H1)
- Profile precedence resolution: sets global profile to "review",
creates plan with --automation-profile trusted, asserts plan output
shows "trusted" not "review" (AC-5 / C2)
- Event queue via plan lifecycle transitions: creates plan, captures
initial state, executes, verifies state transition proving domain
event bus delivered and processed events (AC-3 / C1)
- Hierarchical decomposition via plan tree: creates plan with full-auto
profile, executes, runs plan tree --format json, verifies decision
nodes and children structure (AC-6 / C3)
Robot Framework uses dots as hierarchy separators in suite names
(e.g. "E2E.M6 Acceptance"). The E2E Suite Setup keyword replaced
spaces with underscores but preserved dots, producing directory names
like "E2E.M6_Acceptance". The CLI init command derives the project
name from Path.cwd().name, and Project.validate_name() rejects dots
("Name must be alphanumeric with hyphens, underscores, or spaces").
Added a second Replace String call to convert dots to underscores so
the sanitized suite name passes Project name validation.
Also increased subprocess timeouts in m3_e2e_verification.robot (60s->120s)
and m4_e2e_verification.robot (30s->120s) to prevent flaky CI failures from
Python startup overhead.
The plan lifecycle E2E tests call "plan use local/code-review" but
actions are user-defined entities that must be explicitly registered
via "action create --config <yaml>" before use. Without the action,
plan use failed with "Action 'local/code-review' not found" and all
LLM-dependent tests skipped.
Added action registration to M6 Suite Setup that dynamically selects
the actor matching the available API key (anthropic/claude-sonnet-4
when ANTHROPIC_API_KEY is set, openai/gpt-4o otherwise) and creates
the action YAML inline.
ISSUES CLOSED: #746
The `plan use` CLI set the --automation-profile, actor, and
execution-environment overrides on the in-memory Plan object after
use_action() had already persisted it to the database. Subsequent CLI
invocations (separate processes) loaded the plan from the database
without the overrides, so `plan execute` could never see the automation
profile.
Additionally, `plan execute` required the plan to be in
Strategize/complete state, but nothing ran the strategize phase between
`plan use` and `plan execute`. The auto_strategize field existed on
the AutomationProfile model but was never implemented — plans created
with auto_strategize=0.0 (ci, full-auto, trusted, etc.) stayed in
Strategize/queued indefinitely.
Changes:
- Call service._commit_plan(plan) after applying post-creation overrides
in the `plan use` CLI command so that automation-profile, actor, and
execution-environment changes survive across process boundaries.
- In the `plan execute` CLI command, detect plans still in
Strategize/queued and run the strategize phase inline via
PlanExecutor.run_strategize() before transitioning to Execute.
After inline strategize, if auto_progress already advanced the plan
to Execute (e.g. when auto_execute=0.0), skip the explicit
execute_plan() call and report the current state.
- Widen the auto-select query to include both QUEUED and COMPLETE plans
(QUEUED plans are now eligible because strategize will run inline).
- Update the "no plans ready" test to use an empty plan list (the old
test returned a QUEUED plan which is now eligible).
- Add BDD scenarios covering inline strategize, auto-progress handling,
and automation-profile persistence.
Refs: #746
list_plans() only read from the in-memory self._plans dict, but the
DI container creates PlanLifecycleService via providers.Factory (a new
instance per call), so every CLI invocation started with an empty dict.
Plans created by "plan use" were persisted to the database via
UnitOfWork.transaction() but "plan lifecycle-list" could never find
them because it never queried the database.
Added a database query path to list_plans() that calls
LifecyclePlanRepository.list_all() when persistence is enabled,
mirroring the existing database fallback in get_plan(). Also added
the list_all() method to LifecyclePlanRepository.
Refs: #746
The format_output() function returned a string that callers passed to
Rich console.print(), which wraps long lines at terminal width. This
injected literal newline characters into JSON string values (e.g. in
definition_of_done fields), producing invalid JSON that downstream
parsers could not decode (JSONDecodeError: Invalid control character).
For machine-readable formats (json, yaml, plain), format_output() now
writes the rendered output directly to sys.stdout and returns an empty
string. This preserves the exact serialization from json.dumps/
yaml.dump without Rich text processing artifacts.
Refs: #746
The automation-profile CLI commands used an _InMemoryProfileRepository
(a Python dict) that lost all data between CLI process invocations.
Profiles created with "automation-profile add" were invisible to
subsequent "automation-profile show" or "list" calls because each CLI
command is a separate process with a fresh empty dict.
Changes:
- Replaced _InMemoryProfileRepository with the real
AutomationProfileRepository from the infrastructure layer, wired
via the DI container following the same pattern as tool.py and
session.py.
- Added auto_commit support to AutomationProfileRepository (matching
the existing SessionRepository pattern) so that CLI commands running
outside a UnitOfWork commit each operation automatically.
- Added safety_json and guards_json Text columns to the
automation_profiles table (Alembic migration m6_005) for full-fidelity
round-trip of the AutomationGuard and SafetyProfile sub-models.
Previously, guards and several safety fields (max_cost_per_plan,
max_retries_per_step, etc.) were silently dropped on persistence.
- Updated _from_domain, _to_domain, and _update_row to serialize and
deserialize the full guard and safety sub-models via JSON, with
backward-compatible fallback to legacy scalar columns.
Refs: #746
The re-export of make_mock_scenario from features/mocks/__init__.py
caused ASV benchmark discovery to fail because tdd_test_helpers imports
behave.model.Status, which is unavailable in the ASV benchmark
virtualenv. All callers already import via the full module path
(features.mocks.tdd_test_helpers), so the re-export was unnecessary.
ISSUES CLOSED: #628
Implements the three-tag TDD bug-capture system in Robot Framework via a
Listener v3 module, paralleling the Behave implementation. Tests tagged
tdd_expected_fail that fail have their result inverted to PASS (bug still
exists); tests that unexpectedly pass are inverted to FAIL with guidance.
Addresses all 15 findings from code review (PR !673, reviewer hamza.khyari):
P2 fixes:
- Added idempotency guard (_processed_tests set) to prevent double-inversion
when the listener is loaded twice in the same process.
- Rewrote normal-test-unaffected check to run alongside a tdd_expected_fail
fixture in a single Robot invocation, proving the listener is loaded and
selectively applies rather than being a tautological pass.
P3 fixes:
- Added output.xml existence guard with clear diagnostics in _run_fixture.
- Documented intentional use of data.tags (static definition) vs result.tags
(runtime-modifiable) in end_test docstring.
- Added SKIP status test fixture and integration test case.
- Added message content assertion in cmd_expected_fail_inverted.
- Tightened substring assertions to match specific error text.
- Added tdd_expected_fail-alone fixture (both companions missing).
- Added close() hook to clear _validation_errors and _processed_tests.
- Simplified _run_fixture return type to tuple[str, str].
- Changed listener path resolution from CWD-relative to __file__-relative
in noxfile.py (integration_tests, slow_integration_tests, e2e_tests).
P4 fixes:
- Added __all__ declaration to helper module.
- Changed module docstring from "mirroring" to "paralleling".
- Added comment documenting accepted XML parsing risk (self-generated XML).
Additional fixes:
- Increased M4 E2E plan-tree test timeout from 30s to 120s (pre-existing
timeout failure unrelated to this feature).
Quality gates (post-rebase onto latest master):
- nox -s lint: PASS
- nox -s typecheck: PASS (0 errors)
- nox -s unit_tests: PASS (10,700 scenarios)
- nox -s integration_tests: PASS (1,505 tests)
- nox -s coverage_report: PASS (97.9% >= 97% threshold)
- nox -s benchmark: PASS
- nox -s docs: PASS
- nox -s build: PASS
- nox -s security_scan: PASS
- nox -s dead_code: PASS
ISSUES CLOSED: #628
TDD expected-fail tests proving bug #822 exists:
CheckpointService.rollback_to_checkpoint() returns a successful
RollbackResult but does not execute git reset --hard. Files modified
after the checkpoint remain unchanged after rollback.
Also fixes Robot Framework timeout robustness across the entire test
suite: all Run Process calls now use on_timeout=kill (prevents
SIGTERM-induced -15 exit codes under CI load) and timeouts increased
to 120s (prevents premature kills during heavy parallel execution).
ISSUES CLOSED: #839
Write Behave scenario and Robot Framework test proving that
SubplanService.spawn() only creates metadata (SubplanStatus records
and SpawnMetadata) without creating real child Plan domain objects
or triggering lifecycle progression. Tests are tagged
@tdd_expected_fail so CI passes via result inversion.
ISSUES CLOSED: #838
The `plan execute` CLI command only performed phase transitions
(Strategize → Execute) without ever invoking the `PlanExecutor` to
drive the strategize or execute actors. `PlanExecutor.__init__`
unconditionally created `StrategizeStubActor()` and
`ExecuteStubActor()` which parse text locally and return empty
changesets — no real LLM call was made.
Added `_get_plan_executor()` helper that resolves `ProviderRegistry`
from the DI container and constructs `LLMStrategizeActor` /
`LLMExecuteActor` for real LLM invocations via LangChain. Updated
`execute_plan` CLI command to detect plan phase/state and
automatically run the appropriate actor:
- Strategize/queued → run strategize actor → transition to Execute
- Strategize/complete → phase transition only (backward compat)
- Execute/queued → run execute actor → mark complete
New `llm_actors.py` module provides `LLMStrategizeActor` (task
decomposition into numbered steps) and `LLMExecuteActor` (code
generation with FILE: blocks). Both resolve `provider/model` actor
names (e.g. `openai/gpt-4`) to live LangChain LLM instances.
`PlanExecutor.__init__` now accepts optional `strategize_actor` and
`execute_actor` parameters, falling back to stubs when None.
Existing mock-based BDD tests remain backward-compatible via duck-
typing fallback (MagicMock.phase is not a PlanPhase, so the legacy
`service.execute_plan()` path is taken).
Includes Behave BDD scenarios testing custom actor injection into
PlanExecutor.
ISSUES CLOSED: #960
The `action create` CLI command was the only action subcommand missing
the `--format`/`-f` parameter. All other action subcommands (`list`,
`show`, `archive`) already accepted `--format` and routed through
`_print_action()`. Running `action create --config action.yaml
--format plain` failed with a Typer unrecognized-option error.
Added the `fmt` parameter (with `--format`/`-f` aliases, defaulting
to `rich`) to the `create()` function signature and passed it through
to the existing `_print_action()` helper which already handles all
output formats. Added Behave BDD scenarios for `--format plain` and
`--format json` to `action_cli_spec_alignment.feature`.
ISSUES CLOSED: #959
Add Robot Framework E2E test suite robot/e2e/m2_acceptance.robot exercising
M2 acceptance criteria with zero mocking. Test creates a temp git repo with
sample project files, registers a custom actor via CLI, sets up resource and
project, creates an action referencing the actor, and runs the full plan
lifecycle (use → execute strategize → execute → diff → apply). Validates
actor YAML compilation, skill registry, tool lifecycle, and LLM integration
through real CLI invocations with real provider API keys. Uses flexible
structural assertions and expected_rc=None for LLM-dependent commands.
ISSUES CLOSED: #742
Fixed 5 bugs preventing the M1 E2E acceptance test from passing:
1. _get_lifecycle_service() in action.py and plan.py bypassed the DI
container, creating PlanLifecycleService without UnitOfWork. All
plan/action data was in-memory only and lost between subprocess
calls. Now uses container.plan_lifecycle_service() for DB persistence.
2. `plan execute` CLI only called service.execute_plan() (a pure state
transition) without running PlanExecutor phase processing. Rewrote
to detect the plan's current phase/state and dispatch synchronously:
Strategize/queued → run_strategize(), Strategize/complete → transition
+ run_execute(), Execute/queued → run_execute().
3. `plan apply` CLI had no plan_id argument. Added optional positional
plan_id with _lifecycle_apply_with_id() that drives the plan through
Apply/queued → Apply/processing → Apply/applied.
4. Preflight guardrail in start_strategize() built action_registry from
the in-memory _actions dict only. Added get_action(plan.action_name)
call to load the action from DB into cache before the guardrail check.
5. Robot Framework Create File syntax used continuation lines producing
9 arguments instead of 1. Fixed to use Catenate SEPARATOR=\n then
pass single variable to Create File. Also fixed --branch main to
--branch master (git init default).
update mocks for execute_plan CLI changes across unit and integration tests
The new execute_plan() command calls _get_plan_executor() and
service.get_plan(plan_id) for phase/state detection. Existing tests
only mocked _get_lifecycle_service, so MagicMock defaults caused
phase/state comparisons to fail.
Changes across 14 files:
- Patch _get_plan_executor in all test setups that invoke the CLI
execute command (Behave step files + Robot helper scripts)
- Set service.get_plan.return_value to real Plan objects with correct
phase/state so the execute_plan dispatch logic works
- Fix error-path tests to use STRATEGIZE/COMPLETE plans so the error
side_effects are actually reached
- Fix "Multiple plans eligible" → "Multiple plans ready" message text
to match existing test expectations
increase Robot Framework subprocess timeouts for CI resource contention
Three integration tests were timing out in CI due to resource contention
when pabot runs multiple test suites in parallel. All three pass locally
and the timeouts were simply too tight for constrained CI environments.
- tdd_session_create_di.robot: 30s → 90s (DI container init + DB setup)
- database_integration.robot: 60s → 120s (Run Python Script keyword)
- m3_e2e_verification.robot: 60s → 120s (correction-live-revert spawns
3 sequential CLI subprocesses with full container initialization)
ISSUES CLOSED: #789
Added Robot Framework E2E test suite for M1 milestone acceptance criteria.
Tests the complete plan lifecycle (action create → resource add → project
create → plan use → plan execute strategize → plan execute → plan diff →
plan apply) with real LLM API keys and no mocking.
Key implementation details:
- Uses openai/gpt-4o-mini as strategy/execution actor (cost-effective)
- Simple definition_of_done: "Create a file called HELLO.md"
- Creates isolated temp git repo via Create Temp Git Repo keyword
- Extracts plan ID via ULID regex from plain-text output
- Uses expected_rc=None for LLM-dependent steps (execute, diff, apply)
to handle non-deterministic LLM behavior gracefully
- Flexible structural assertions: checks rc, output presence, git log
- Skips gracefully when no LLM API keys (ANTHROPIC/OPENAI) are set
- Tagged [E2E] so it runs only in nox -s e2e_tests session
ISSUES CLOSED: #741
Updated CI pipeline to inject ANTHROPIC_API_KEY and OPENAI_API_KEY
secrets as environment variables during Robot Framework integration
test execution. Added CI setup documentation.
ISSUES CLOSED: #701
- Replace `rc == 0 or rc == 1` with strict `rc == 0` in resource type
list test so failures are no longer silently accepted
- Capture and assert the return code of the Suite Setup database schema
creation to fail fast if the setup itself is broken
apply strict RC checks to resource_cli.robot
- Capture return value of Run Process in Suite Setup and assert rc==0
- Replace tolerant RC check (rc==0 or rc==1) with strict Should Be Equal
As Integers check for resource type list test case
broaden exception handling in resource CLI commands
Add catch-all `except Exception` handler after each `except
CleverAgentsError` block in all 14 resource CLI command handlers.
This ensures unexpected exceptions (e.g. sqlalchemy.exc.OperationalError)
are caught and displayed gracefully instead of producing raw tracebacks.
re-raise typer.Abort/Exit in broad exception handlers
The `except Exception` handlers added in the previous commit
inadvertently caught typer.Abort and typer.Exit, which are
subclasses of Exception (via click.exceptions). This turned
successful CLI exits into aborts and double-handled already-caught
errors, breaking integration tests that rely on normal typer exit
behaviour.
Add an isinstance guard to re-raise typer.Abort and typer.Exit
before the catch-all handler runs.
Fixes#896
- Remove silent retry logic in Run Python Script that masked intermittent
failures by retrying on empty stdout + non-zero RC
- Change output filter from 'in' to 'startswith' so only lines beginning
with log prefixes are filtered, not lines containing them mid-string
- Remove [error from the filter list so error lines are never silently
discarded from test output
Fixes#898
Replace Run Keyword And Return Status soft checks with direct assertions
in three test cases:
- Docstring check now uses Should Contain directly instead of logging
WARN, so missing docstrings cause a hard test failure.
- Settings file (config/settings.py) existence is now a hard File Should
Exist assertion instead of a conditional that silently skips all
checks when the file is absent.
- Exceptions file (core/exceptions.py) existence is now a hard File
Should Exist assertion instead of a conditional that silently skips
the exception hierarchy checks when the file is absent.
aditya
hamza.khyari
freemo
CoreRasurae
brent.edwards
hurui200320