Merge branch 'master' into test/e2e-m5-acceptance
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 19s
CI / quality (pull_request) Successful in 30s
CI / build (pull_request) Successful in 30s
CI / security (pull_request) Successful in 45s
CI / typecheck (pull_request) Successful in 46s
CI / unit_tests (pull_request) Successful in 3m8s
CI / integration_tests (pull_request) Successful in 3m34s
CI / docker (pull_request) Successful in 57s
CI / e2e_tests (pull_request) Successful in 5m24s
CI / coverage (pull_request) Successful in 6m56s
CI / benchmark-regression (pull_request) Failing after 31m11s

This commit is contained in:
2026-03-19 06:45:20 +00:00
committed by Forgejo
9 changed files with 1248 additions and 0 deletions
+9
View File
@@ -2,6 +2,15 @@
## Unreleased
- Added TDD bug-capture tests for bug #967`plan execute` phase processing.
Tests exercise the CLI orchestration layer via CliRunner (Behave) and
replicated CLI logic (Robot) to verify that `plan execute` correctly
handles plans in Strategize/QUEUED state by running `run_strategize()`
before transitioning. Includes four Behave scenarios and four Robot
integration test cases covering CLI execute from QUEUED, full lifecycle
orchestration, positive control, and auto-discovery of QUEUED plans.
(`features/tdd_plan_execute_phase_processing.feature`,
`robot/tdd_plan_execute_phase_processing.robot`) (#977)
- Added built-in deferred virtual resource types: `remote`, `submodule`, and
`symlink` with equivalence metadata rules for cross-repo and cross-layer
identity tracking. Registry bootstrap includes deferred virtual types but
@@ -0,0 +1,226 @@
"""Step definitions for TDD Bug #783 — init --yes should not require user input.
These steps exercise the *real* ``agents init --yes`` code path while
simulating a TTY environment by replacing the migration runner's
``_default_prompt_for_migration`` with a function that declines
migration (simulating a user on a real terminal providing no input).
The ``@tdd_expected_fail`` tag on the scenarios inverts the result:
these tests **pass** CI while the bug is present and will **fail** once
the bug is fixed (signalling that the tag should be removed).
Root cause
~~~~~~~~~~
``init_command()`` in ``project.py`` receives the ``--yes`` flag but
does not forward it to the migration runner. The migration runner's
``_default_prompt_for_migration()`` calls ``typer.confirm()`` when a
TTY is detected, blocking the command for user input even though
``--yes`` was specified. In a non-TTY environment (like CliRunner),
the prompt is silently skipped — the bug only surfaces in real terminal
sessions. We replace the prompt function to reproduce the TTY code
path.
Mock strategy
~~~~~~~~~~~~~
CliRunner replaces ``sys.stdin`` during ``invoke()``, so patching
``sys.stdin.isatty()`` alone is ineffective. Instead we:
1. Patch ``sys.stdin`` on the real ``sys`` module (documents intent
and is the correct target per the project's established pattern in
``features/steps/migration_runner_steps.py``).
2. Patch ``MigrationRunner._default_prompt_for_migration`` with a
replacement that returns ``False`` — simulating a TTY prompt where
the user declines migration. This ensures the prompt code path is
exercised deterministically regardless of CliRunner's stdin
replacement.
3. Set ``CLEVERAGENTS_DATABASE_URL`` to a path inside our tmp directory
whose filename does **not** match the ``before_scenario`` template-DB
prefixes (``cleveragents_*``, ``test_*``, ``db.*``). This forces the
migration runner's ``init_or_upgrade`` through the real Alembic path
instead of the test fast-path that copies a pre-migrated template.
With bug #783 present, ``require_confirmation=True`` is hardcoded in
``UnitOfWork._ensure_database_initialized()``, so the prompt fires and
returns ``False``, causing ``MigrationNotApprovedError``. After the
fix, ``--yes`` should bypass the prompt entirely.
"""
from __future__ import annotations
import os
import shutil
import sys
import tempfile
from unittest.mock import MagicMock, patch
from behave import given, then, when
from behave.runner import Context
from typer.testing import CliRunner
from cleveragents.application.container import reset_container
from cleveragents.cli.commands.project import app as project_app
from cleveragents.infrastructure.database.migration_runner import MigrationRunner
runner = CliRunner()
def _tty_prompt_declines(_message: str) -> bool:
"""Simulate a TTY migration prompt that declines (no user input).
Replaces ``_default_prompt_for_migration`` to reproduce the TTY
code path without depending on ``sys.stdin.isatty()``, which
CliRunner overrides during ``invoke()``.
"""
return False
# ---------------------------------------------------------------------------
# Given steps
# ---------------------------------------------------------------------------
@given("a fresh environment for tdd-init-yes-no-input")
def step_fresh_environment(context: Context) -> None:
"""Set up a fresh temporary environment with no existing database.
Creates a temporary directory to serve as the working directory so
that ``agents init`` creates a fresh ``.cleveragents`` directory.
Disables auto-apply environment variables so the real prompt
behaviour is exercised.
"""
context.tdd_init_tmpdir = tempfile.mkdtemp(prefix="tdd_init_yes_")
# Save env vars that would bypass the prompt or redirect the
# database, and remove them. ``CLEVERAGENTS_DATABASE_URL`` and
# ``CLEVERAGENTS_TEST_DATABASE_URL`` are set by ``before_scenario``
# to pre-migrated scenario databases — removing them forces the
# container to derive a fresh DB path from ``CLEVERAGENTS_HOME``.
context.tdd_init_saved_env = {
k: os.environ.pop(k, None)
for k in (
"CLEVERAGENTS_AUTO_APPLY_MIGRATIONS",
"CI",
"BEHAVE_TESTING",
"CLEVERAGENTS_DATABASE_URL",
"CLEVERAGENTS_TEST_DATABASE_URL",
)
}
# Register cleanup to restore env vars and remove tmpdir even if
# subsequent steps are never reached (e.g. hook errors).
saved_env = context.tdd_init_saved_env
tmpdir = context.tdd_init_tmpdir
def _cleanup() -> None:
for key, val in saved_env.items():
if val is not None:
os.environ[key] = val
else:
os.environ.pop(key, None)
shutil.rmtree(tmpdir, ignore_errors=True)
reset_container()
context.add_cleanup(_cleanup)
# ---------------------------------------------------------------------------
# When steps
# ---------------------------------------------------------------------------
@when("I invoke agents init --yes with no stdin")
def step_invoke_init_yes_no_stdin(context: Context) -> None:
"""Invoke ``agents init --yes`` simulating a TTY with no real input.
Uses ``CliRunner`` with ``input=""`` to provide no input. Patches
``sys.stdin`` on the real ``sys`` module for intent documentation
and replaces ``MigrationRunner._default_prompt_for_migration`` with
a function that returns ``False`` (simulating a TTY prompt where the
user declines). See module docstring for the full mock strategy.
"""
# Create a mock stdin that reports as a TTY but has no data
mock_stdin = MagicMock(spec=sys.stdin)
mock_stdin.isatty.return_value = True
# Reset the global container to ensure a fresh UoW is created
# using our env var overrides (no stale CLEVERAGENTS_DATABASE_URL).
reset_container()
# Use a DB path whose filename (``init_yes_test.sqlite``) does
# NOT start with the template-fast-path prefixes
# (``cleveragents_*``, ``test_*``, ``db.*``) so the real Alembic
# migration path runs instead of copying a pre-migrated template.
# Do NOT pre-create directories — ``init_command`` expects a
# fresh path without an existing ``.cleveragents`` directory.
tdd_db_path = os.path.join(
context.tdd_init_tmpdir, ".cleveragents", "init_yes_test.sqlite"
)
tdd_db_url = f"sqlite:///{tdd_db_path}"
with patch.dict(
os.environ,
{
"CLEVERAGENTS_HOME": context.tdd_init_tmpdir,
"CLEVERAGENTS_DATABASE_URL": tdd_db_url,
},
):
# Remove auto-apply env vars inside the patched environment
for key in (
"CLEVERAGENTS_AUTO_APPLY_MIGRATIONS",
"CI",
"BEHAVE_TESTING",
):
os.environ.pop(key, None)
# Patch sys.stdin on the real sys module (correct target per
# project convention) and replace the prompt function to
# simulate a TTY decline. See module docstring for why
# both patches are needed.
with (
patch.object(sys, "stdin", mock_stdin),
patch.object(
MigrationRunner,
"_default_prompt_for_migration",
staticmethod(_tty_prompt_declines),
),
):
context.tdd_init_result = runner.invoke(
project_app,
["init", "--yes", "--path", context.tdd_init_tmpdir],
input="",
)
# ---------------------------------------------------------------------------
# Then steps
# ---------------------------------------------------------------------------
@then("the tdd-init-yes-no-input command should exit successfully")
def step_exit_success(context: Context) -> None:
"""Assert the init command exited with code 0."""
result = context.tdd_init_result
assert result is not None, "init --yes was not invoked"
actual = result.exit_code
assert actual == 0, f"Expected exit code 0, got {actual}.\nOutput:\n{result.output}"
@then('the tdd-init-yes-no-input output should not contain "{text}"')
def step_output_not_contains(context: Context, text: str) -> None:
"""Assert the given text does NOT appear in the command output."""
result = context.tdd_init_result
assert result is not None, "init --yes was not invoked"
output = result.output
assert text.lower() not in output.lower(), (
f"Did NOT expect '{text}' in output but found it:\n{output}"
)
@then('the tdd-init-yes-no-input output should contain "{text}"')
def step_output_contains(context: Context, text: str) -> None:
"""Assert the given text appears in the command output."""
result = context.tdd_init_result
assert result is not None, "init --yes was not invoked"
output = result.output
assert text.lower() in output.lower(), (
f"Expected '{text}' in output but did not find it:\n{output}"
)
@@ -0,0 +1,391 @@
"""Step definitions for TDD Bug #967 — plan execute phase processing.
These steps exercise the **CLI orchestration layer** (the ``execute_plan``
command handler in ``plan.py``) via Typer's ``CliRunner`` to verify that
the ``plan execute`` command correctly handles plans in Strategize/QUEUED
state.
Bug #967: The ``plan execute`` CLI command originally only called
``service.execute_plan(plan_id)``, which is a state transition only
(Strategize/COMPLETE → Execute/QUEUED). It did not construct a
``PlanExecutor`` or call ``run_strategize()`` / ``run_execute()``. When a
plan was in Strategize/QUEUED state (immediately after ``plan use``), the
command failed because ``execute_plan()`` requires Strategize/COMPLETE.
The fix added phase-aware orchestration to the CLI handler: when a plan is
in Strategize/QUEUED, it calls ``PlanExecutor.run_strategize()`` before
transitioning. The auto-discovery filter was also updated to include
Strategize/QUEUED plans.
Scenarios 1, 2, and 4 exercise the CLI handler (the code path that was
fixed). Since the fix is already in the codebase, the ``@tdd_expected_fail``
tag has been removed and these tests serve as permanent regression guards.
The "positive control" scenario (Scenario 3) demonstrates that the proper
service-level orchestration path (``run_strategize`` → ``execute_plan``)
works correctly.
This test was written to capture bug #967 per ticket #977.
"""
from __future__ import annotations
from datetime import datetime
from unittest.mock import MagicMock, patch
from behave import given, then, when
from behave.runner import Context
from typer.testing import CliRunner
from cleveragents.application.services.plan_executor import PlanExecutor
from cleveragents.application.services.plan_lifecycle_service import (
PlanLifecycleService,
PlanNotReadyError,
)
from cleveragents.cli.commands.plan import app as plan_app
from cleveragents.config.settings import Settings
from cleveragents.core.exceptions import PlanError
from cleveragents.domain.models.core.plan import (
NamespacedName,
Plan,
PlanIdentity,
PlanPhase,
PlanTimestamps,
ProcessingState,
ProjectLink,
)
def _make_cli_plan(
*,
plan_id: str = "01ARZ3NDEKTSV4RRFFQ69G5967",
phase: PlanPhase = PlanPhase.STRATEGIZE,
state: ProcessingState = ProcessingState.QUEUED,
) -> Plan:
"""Build a real ``Plan`` domain object for CLI tests.
Uses the Plan domain model directly (not a MagicMock) so that the CLI
handler's ``isinstance`` checks and attribute accesses work correctly.
"""
return Plan(
identity=PlanIdentity(plan_id=plan_id),
namespaced_name=NamespacedName(namespace="local", name="tdd-967-plan"),
action_name="local/tdd-967-action",
description="TDD plan for bug #967",
phase=phase,
processing_state=state,
project_links=[ProjectLink(project_name="proj-1")],
strategy_actor="openai/gpt-4",
execution_actor="openai/gpt-4",
timestamps=PlanTimestamps(created_at=datetime.now(), updated_at=datetime.now()),
)
def _create_service_and_queued_plan() -> tuple[PlanLifecycleService, str]:
"""Create a real in-memory PlanLifecycleService and a plan in Strategize/QUEUED.
Returns a tuple of (service, plan_id). The plan is created via the normal
``create_action`` → ``use_action`` flow, which puts the plan in
Strategize/QUEUED — the state that triggers bug #967.
"""
settings: Settings = Settings()
service: PlanLifecycleService = PlanLifecycleService(settings=settings)
action = service.create_action(
name="local/tdd-967-action",
description="TDD action for bug #967",
definition_of_done="Complete the test plan",
strategy_actor="local/stub-strategize",
execution_actor="local/stub-execute",
)
plan: Plan = service.use_action(action_name=str(action.namespaced_name))
return service, plan.identity.plan_id
# ---------------------------------------------------------------------------
# CLI-level test setup (used by Scenarios 1, 2, 4)
# ---------------------------------------------------------------------------
_PLAN_ID: str = "01ARZ3NDEKTSV4RRFFQ69G5967"
@given("a CLI runner and mocked services for bug 967")
def step_cli_runner_and_mocked_services(context: Context) -> None:
"""Set up a CliRunner and mock service/executor for CLI-level testing."""
context.runner_967 = CliRunner()
context.mock_service_967 = MagicMock()
context.mock_executor_967 = MagicMock()
@given("a plan in Strategize/QUEUED state for bug 967")
def step_plan_in_strategize_queued(context: Context) -> None:
"""Configure the mocked service to return a plan in Strategize/QUEUED.
The mock ``get_plan`` returns the QUEUED plan for the initial checks,
then returns an Execute/QUEUED plan after strategize runs (simulating
auto_progress), and finally an Execute/COMPLETE plan after run_execute.
"""
queued_plan: Plan = _make_cli_plan(
plan_id=_PLAN_ID,
phase=PlanPhase.STRATEGIZE,
state=ProcessingState.QUEUED,
)
execute_queued_plan: Plan = _make_cli_plan(
plan_id=_PLAN_ID,
phase=PlanPhase.EXECUTE,
state=ProcessingState.QUEUED,
)
execute_complete_plan: Plan = _make_cli_plan(
plan_id=_PLAN_ID,
phase=PlanPhase.EXECUTE,
state=ProcessingState.COMPLETE,
)
# get_plan call sequence in the CLI execute_plan handler:
# 1. pre_plan (read-only check)
# 2. current_plan (phase detection)
# 3. re-fetch after run_strategize (auto_progress moved to Execute)
# 4. re-fetch for inline execute check
# 5. re-fetch after run_execute
context.mock_service_967.get_plan.side_effect = [
queued_plan,
queued_plan,
execute_queued_plan,
execute_queued_plan,
execute_complete_plan,
]
context.queued_plan_967 = queued_plan
context.execute_complete_plan_967 = execute_complete_plan
# ---------------------------------------------------------------------------
# Scenario 1: CLI execute command handles plan in Strategize/QUEUED state
# Scenario 2: CLI execute command orchestrates full lifecycle for QUEUED plan
# ---------------------------------------------------------------------------
@when("I invoke the plan execute CLI command for the QUEUED plan for bug 967")
def step_invoke_cli_execute(context: Context) -> None:
"""Invoke the ``plan execute`` CLI command via CliRunner.
This exercises the CLI orchestration layer — the actual code path that
was buggy in #967. The handler should detect Strategize/QUEUED and
call ``run_strategize`` before transitioning to Execute.
"""
with (
patch(
"cleveragents.cli.commands.plan._get_lifecycle_service",
return_value=context.mock_service_967,
),
patch(
"cleveragents.cli.commands.plan._get_plan_executor",
return_value=context.mock_executor_967,
),
):
context.cli_result_967 = context.runner_967.invoke(
plan_app, ["execute", _PLAN_ID]
)
@then("the CLI should succeed and the plan should reach Execute phase for bug 967")
def step_cli_succeed_execute_phase(context: Context) -> None:
"""Assert the CLI command completed without error.
Bug #967 regression guard: if the CLI orchestration is broken, this
assertion fails because the command would abort with an error.
"""
result = context.cli_result_967
assert result.exit_code == 0, (
f"Bug #967: CLI 'plan execute' failed with exit code {result.exit_code}. "
f"The CLI should handle Strategize/QUEUED plans by running "
f"run_strategize() before transitioning. Output:\n{result.output}"
)
assert "Execute" in result.output or "execute" in result.output, (
f"Bug #967: CLI output does not mention Execute phase. Output:\n{result.output}"
)
@then("the executor should have run strategize for the plan for bug 967")
def step_executor_ran_strategize(context: Context) -> None:
"""Assert the CLI handler invoked run_strategize on the executor.
Bug #967 regression guard: the CLI handler must call
``executor.run_strategize(plan_id)`` for plans in Strategize/QUEUED.
"""
context.mock_executor_967.run_strategize.assert_called_once_with(_PLAN_ID)
@then("the plan should have completed execute phase processing via CLI for bug 967")
def step_plan_completed_execute_via_cli(context: Context) -> None:
"""Assert the CLI handler also ran the execute phase.
Bug #967 regression guard: after strategize, the CLI should run
``executor.run_execute(plan_id)`` to complete the Execute phase.
"""
context.mock_executor_967.run_execute.assert_called_once_with(_PLAN_ID)
# ---------------------------------------------------------------------------
# Scenario 3: Positive control — proper orchestration works
# ---------------------------------------------------------------------------
@given("a real plan executor with a plan in Strategize/QUEUED for bug 967")
def step_real_executor_queued_plan(context: Context) -> None:
"""Set up a real PlanExecutor and PlanLifecycleService with a QUEUED plan."""
service, plan_id = _create_service_and_queued_plan()
context.service_967 = service
context.plan_id_967 = plan_id
context.executor_967 = PlanExecutor(lifecycle_service=service)
@when("I run the proper orchestration of strategize then execute for bug 967")
def step_proper_orchestration(context: Context) -> None:
"""Run the correct orchestration: run_strategize → execute_plan → verify.
This positive control demonstrates that when the caller properly
orchestrates the lifecycle (as the fixed CLI does), a QUEUED plan
can be taken through to Execute phase.
"""
service: PlanLifecycleService = context.service_967
executor: PlanExecutor = context.executor_967
plan_id: str = context.plan_id_967
context.orchestration_error_967 = None
context.orchestrated_plan_967 = None
try:
# Step 1: Run strategize to completion (QUEUED → PROCESSING → COMPLETE)
executor.run_strategize(plan_id)
# Step 2: Re-fetch the plan — auto_progress in complete_strategize
# may have already advanced the plan to Execute phase.
plan: Plan = service.get_plan(plan_id)
if plan.phase == PlanPhase.EXECUTE:
# auto_progress already moved it
context.orchestrated_plan_967 = plan
elif (
plan.phase == PlanPhase.STRATEGIZE
and plan.state == ProcessingState.COMPLETE
):
# Transition to Execute phase
context.orchestrated_plan_967 = service.execute_plan(plan_id)
else:
context.orchestration_error_967 = PlanError(
f"Unexpected plan state after strategize: "
f"{plan.phase.value}/{plan.state.value}"
)
except (PlanError, PlanNotReadyError) as exc:
context.orchestration_error_967 = exc
@then("the plan should be in Execute/QUEUED state via orchestration for bug 967")
def step_plan_in_execute_via_orchestration(context: Context) -> None:
"""Assert the properly orchestrated plan reached Execute phase with QUEUED state.
This is a positive control: when the orchestration is done correctly,
the plan should be in Execute phase. This scenario is NOT tagged
``@tdd_expected_fail`` and must always pass.
"""
if context.orchestration_error_967 is not None:
raise AssertionError(
f"Proper orchestration failed unexpectedly: "
f"{context.orchestration_error_967}"
)
plan: Plan | None = context.orchestrated_plan_967
assert plan is not None, "Orchestration returned no plan"
assert plan.phase == PlanPhase.EXECUTE, (
f"Expected Execute phase, got {plan.phase.value}"
)
assert plan.state == ProcessingState.QUEUED, (
f"Expected QUEUED state, got {plan.state.value}"
)
# ---------------------------------------------------------------------------
# Scenario 4: CLI auto-discovery finds plans in Strategize/QUEUED state
# ---------------------------------------------------------------------------
@given(
"a single plan in Strategize/QUEUED state eligible for auto-discovery for bug 967"
)
def step_single_queued_plan_auto_discovery(context: Context) -> None:
"""Configure the mocked service for auto-discovery with a QUEUED plan.
The CLI auto-discovery (no plan_id) calls ``service.list_plans()`` and
filters for eligible plans. Bug #967 had a filter that only accepted
Strategize/COMPLETE, missing QUEUED plans. The fix includes QUEUED.
"""
queued_plan: Plan = _make_cli_plan(
plan_id=_PLAN_ID,
phase=PlanPhase.STRATEGIZE,
state=ProcessingState.QUEUED,
)
execute_queued_plan: Plan = _make_cli_plan(
plan_id=_PLAN_ID,
phase=PlanPhase.EXECUTE,
state=ProcessingState.QUEUED,
)
execute_complete_plan: Plan = _make_cli_plan(
plan_id=_PLAN_ID,
phase=PlanPhase.EXECUTE,
state=ProcessingState.COMPLETE,
)
# list_plans is called twice: once for STRATEGIZE, once for EXECUTE
context.mock_service_967.list_plans.side_effect = [
[queued_plan], # Strategize phase plans
[], # Execute phase plans (none)
]
# get_plan call sequence after auto-discovery selects the plan:
# 1. pre_plan (read-only check)
# 2. current_plan (phase detection)
# 3. re-fetch after run_strategize
# 4. re-fetch for inline execute check
# 5. re-fetch after run_execute
context.mock_service_967.get_plan.side_effect = [
queued_plan,
queued_plan,
execute_queued_plan,
execute_queued_plan,
execute_complete_plan,
]
@when("I invoke the plan execute CLI command without a plan id for bug 967")
def step_invoke_cli_execute_no_plan_id(context: Context) -> None:
"""Invoke ``plan execute`` without a plan ID to trigger auto-discovery.
Bug #967 regression guard: the CLI auto-discovery filter must include
Strategize/QUEUED plans, not just Strategize/COMPLETE.
"""
with (
patch(
"cleveragents.cli.commands.plan._get_lifecycle_service",
return_value=context.mock_service_967,
),
patch(
"cleveragents.cli.commands.plan._get_plan_executor",
return_value=context.mock_executor_967,
),
):
context.cli_result_967 = context.runner_967.invoke(plan_app, ["execute"])
@then("the CLI should succeed and auto-discover the QUEUED plan for bug 967")
def step_cli_auto_discovery_succeeds(context: Context) -> None:
"""Assert the CLI auto-discovered and processed the Strategize/QUEUED plan.
Bug #967 regression guard: if the auto-discovery filter excludes QUEUED
plans, the CLI would abort with "No plans ready for execution."
"""
result = context.cli_result_967
assert result.exit_code == 0, (
"Bug #967: CLI 'plan execute' (no plan_id) failed with exit code "
f"{result.exit_code}. The auto-discovery filter should include "
f"Strategize/QUEUED plans. Output:\n{result.output}"
)
# Verify the executor was actually invoked (plan was found and processed)
context.mock_executor_967.run_strategize.assert_called_once_with(_PLAN_ID)
+28
View File
@@ -0,0 +1,28 @@
@tdd_expected_fail @tdd_bug @tdd_bug_783
Feature: TDD Bug #783 — agents init --yes should not require user input
As a developer
I want to verify that `agents init --yes` completes without any user
input prompts
So that the bug is captured and will be caught by a regression test
The root cause is that `init_command()` in project.py receives the
`--yes` flag but does not pass it through to the migration runner.
The migration runner's `_default_prompt_for_migration` uses
`typer.confirm()` when a TTY is detected, causing the command to block
for user input even when `--yes` was specified.
Scenario: Init with --yes completes without stdin
Given a fresh environment for tdd-init-yes-no-input
When I invoke agents init --yes with no stdin
Then the tdd-init-yes-no-input command should exit successfully
And the tdd-init-yes-no-input output should not contain "Apply migrations now"
# Note: While the bug is present, the first assertion (exit code 0) will
# fail and Behave will skip subsequent steps. This means the "contains
# Initialized" assertion is only evaluated after the bug is fixed, at
# which point this scenario provides distinct post-fix regression value.
Scenario: Init with --yes does not prompt for migration approval
Given a fresh environment for tdd-init-yes-no-input
When I invoke agents init --yes with no stdin
Then the tdd-init-yes-no-input command should exit successfully
And the tdd-init-yes-no-input output should contain "Initialized"
@@ -0,0 +1,42 @@
@tdd_bug @tdd_bug_967 @mock_only
Feature: TDD Bug #967 — plan execute only transitions state, does not run strategize or execute phase processing
As a developer
I want to verify that the plan execute CLI command handles plans in Strategize/QUEUED state
by running strategize phase processing before transitioning to Execute
So that the bug is captured and will be caught by a regression test
Bug #967: The plan execute CLI command originally only called
service.execute_plan(plan_id), which is a state transition only
(Strategize/COMPLETE Execute/QUEUED). It did not construct a
PlanExecutor or call run_strategize() / run_execute(). When a plan
was in Strategize/QUEUED state (immediately after plan use), the command
failed because execute_plan() requires Strategize/COMPLETE.
These tests exercise the CLI orchestration layer (the execute_plan command
handler in plan.py) via CliRunner to verify the bug is fixed. The fix
added phase-aware orchestration: when a plan is in Strategize/QUEUED,
the CLI runs PlanExecutor.run_strategize() before transitioning.
Scenario: CLI execute command handles plan in Strategize/QUEUED state
Given a CLI runner and mocked services for bug 967
And a plan in Strategize/QUEUED state for bug 967
When I invoke the plan execute CLI command for the QUEUED plan for bug 967
Then the CLI should succeed and the plan should reach Execute phase for bug 967
Scenario: CLI execute command orchestrates full lifecycle for QUEUED plan
Given a CLI runner and mocked services for bug 967
And a plan in Strategize/QUEUED state for bug 967
When I invoke the plan execute CLI command for the QUEUED plan for bug 967
Then the executor should have run strategize for the plan for bug 967
And the plan should have completed execute phase processing via CLI for bug 967
Scenario: Positive control — proper orchestration transitions QUEUED plan to Execute
Given a real plan executor with a plan in Strategize/QUEUED for bug 967
When I run the proper orchestration of strategize then execute for bug 967
Then the plan should be in Execute/QUEUED state via orchestration for bug 967
Scenario: CLI auto-discovery finds plans in Strategize/QUEUED state
Given a CLI runner and mocked services for bug 967
And a single plan in Strategize/QUEUED state eligible for auto-discovery for bug 967
When I invoke the plan execute CLI command without a plan id for bug 967
Then the CLI should succeed and auto-discover the QUEUED plan for bug 967
+242
View File
@@ -0,0 +1,242 @@
"""Helper script for tdd_init_yes_no_input.robot smoke tests.
Each subcommand exercises the real ``agents init --yes`` code path
simulating a TTY environment with no real stdin input to reproduce
bug #783. The helper reports the **real** outcome: it exits 0 and
prints the sentinel when the operation succeeds (bug is fixed), and
exits 1 when the bug is still present. The
``tdd_expected_fail_listener`` on the Robot side handles pass/fail
inversion while the bug remains open.
Mock strategy
~~~~~~~~~~~~~
CliRunner replaces ``sys.stdin`` during ``invoke()``, so patching
``sys.stdin.isatty()`` alone is ineffective. Instead we:
1. Patch ``sys.stdin`` on the real ``sys`` module (documents intent
and is the correct mock target per the project convention).
2. Patch ``MigrationRunner._default_prompt_for_migration`` with a
replacement that returns ``False`` simulating a TTY prompt where
the user declines migration.
3. Set ``CLEVERAGENTS_DATABASE_URL`` to a path whose filename does
**not** match the ``before_scenario`` template-DB prefixes so the
real Alembic migration path runs.
With bug #783, ``require_confirmation=True`` is hardcoded, the prompt
fires and returns ``False``, causing ``MigrationNotApprovedError``.
After the fix, ``--yes`` should bypass the prompt entirely.
"""
from __future__ import annotations
import os
import shutil
import sys
import tempfile
from collections.abc import Callable
from pathlib import Path
from unittest.mock import MagicMock, patch
# Ensure local source tree is importable
_SRC = str(Path(__file__).resolve().parents[1] / "src")
if _SRC not in sys.path:
sys.path.insert(0, _SRC)
from typer.testing import CliRunner # noqa: E402
from cleveragents.cli.commands.project import app as project_app # noqa: E402
from cleveragents.infrastructure.database.migration_runner import ( # noqa: E402
MigrationRunner,
)
runner = CliRunner()
def _tty_prompt_declines(_message: str) -> bool:
"""Simulate a TTY migration prompt that declines (no user input).
Replaces ``_default_prompt_for_migration`` to reproduce the TTY
code path without depending on ``sys.stdin.isatty()``, which
CliRunner overrides during ``invoke()``.
"""
return False
# ---------------------------------------------------------------------------
# Subcommands
# ---------------------------------------------------------------------------
def init_no_stdin() -> None:
"""Invoke ``agents init --yes`` simulating a TTY with no real input.
Patches ``sys.stdin`` on the real ``sys`` module for intent
documentation and replaces ``_default_prompt_for_migration`` with
a function that returns ``False`` (simulating a TTY prompt that
declines). See module docstring for the full mock strategy.
Exits 0 with sentinel when the command completes successfully
(bug is fixed). Exits 1 when the command fails (bug still
present).
"""
tmpdir = tempfile.mkdtemp(prefix="tdd_init_yes_robot_")
try:
# Remove env vars that would bypass the prompt
env_save: dict[str, str | None] = {}
for key in (
"CLEVERAGENTS_AUTO_APPLY_MIGRATIONS",
"CI",
"BEHAVE_TESTING",
"CLEVERAGENTS_HOME",
"CLEVERAGENTS_DATABASE_URL",
"CLEVERAGENTS_TEST_DATABASE_URL",
):
env_save[key] = os.environ.pop(key, None)
os.environ["CLEVERAGENTS_HOME"] = tmpdir
# Use a DB path whose filename does NOT start with the
# template-fast-path prefixes (``cleveragents_*``, ``test_*``,
# ``db.*``) so the real Alembic migration path runs.
tdd_db_path = os.path.join(tmpdir, ".cleveragents", "init_yes_test.sqlite")
os.environ["CLEVERAGENTS_DATABASE_URL"] = f"sqlite:///{tdd_db_path}"
# Create a mock stdin that reports as a TTY
mock_stdin = MagicMock(spec=sys.stdin)
mock_stdin.isatty.return_value = True
# Patch sys.stdin on the real sys module and replace the
# prompt function to simulate a TTY decline.
with (
patch.object(sys, "stdin", mock_stdin),
patch.object(
MigrationRunner,
"_default_prompt_for_migration",
staticmethod(_tty_prompt_declines),
),
):
result = runner.invoke(
project_app,
["init", "--yes", "--path", tmpdir],
input="",
)
# Restore env vars
for key, val in env_save.items():
if val is not None:
os.environ[key] = val
else:
os.environ.pop(key, None)
if result.exit_code != 0:
print(
f"init --yes failed with exit code {result.exit_code}",
file=sys.stderr,
)
print(f"Output: {result.output}", file=sys.stderr)
sys.exit(1)
# Bug fixed — command succeeded without input.
print("tdd-init-yes-no-input-ok")
finally:
# Clean up tmpdir
shutil.rmtree(tmpdir, ignore_errors=True)
def init_no_prompt() -> None:
"""Invoke ``agents init --yes`` and verify no migration prompt appears.
Uses the same mock strategy as ``init_no_stdin``. See module
docstring for details.
Exits 0 with sentinel when the output does not contain the
migration prompt text (bug is fixed). Exits 1 when the prompt
text appears or the command fails (bug still present).
"""
tmpdir = tempfile.mkdtemp(prefix="tdd_init_yes_robot_")
try:
# Remove env vars that would bypass the prompt
env_save: dict[str, str | None] = {}
for key in (
"CLEVERAGENTS_AUTO_APPLY_MIGRATIONS",
"CI",
"BEHAVE_TESTING",
"CLEVERAGENTS_HOME",
"CLEVERAGENTS_DATABASE_URL",
"CLEVERAGENTS_TEST_DATABASE_URL",
):
env_save[key] = os.environ.pop(key, None)
os.environ["CLEVERAGENTS_HOME"] = tmpdir
# Use a DB path whose filename does NOT start with the
# template-fast-path prefixes.
tdd_db_path = os.path.join(tmpdir, ".cleveragents", "init_yes_test.sqlite")
os.environ["CLEVERAGENTS_DATABASE_URL"] = f"sqlite:///{tdd_db_path}"
# Create a mock stdin that reports as a TTY
mock_stdin = MagicMock(spec=sys.stdin)
mock_stdin.isatty.return_value = True
# Patch sys.stdin on the real sys module and replace the
# prompt function to simulate a TTY decline.
with (
patch.object(sys, "stdin", mock_stdin),
patch.object(
MigrationRunner,
"_default_prompt_for_migration",
staticmethod(_tty_prompt_declines),
),
):
result = runner.invoke(
project_app,
["init", "--yes", "--path", tmpdir],
input="",
)
# Restore env vars
for key, val in env_save.items():
if val is not None:
os.environ[key] = val
else:
os.environ.pop(key, None)
if result.exit_code != 0:
print(
f"init --yes failed with exit code {result.exit_code}",
file=sys.stderr,
)
print(f"Output: {result.output}", file=sys.stderr)
sys.exit(1)
# Check that no migration prompt text appears
output = result.output.lower()
if "apply migrations now" in output:
print(
"Bug present: migration prompt appeared in output",
file=sys.stderr,
)
sys.exit(1)
# Bug fixed — no prompt appeared.
print("tdd-init-yes-no-prompt-ok")
finally:
# Clean up tmpdir
shutil.rmtree(tmpdir, ignore_errors=True)
# ---------------------------------------------------------------------------
# Dispatcher
# ---------------------------------------------------------------------------
_COMMANDS: dict[str, Callable[[], None]] = {
"init-no-stdin": init_no_stdin,
"init-no-prompt": init_no_prompt,
}
if __name__ == "__main__":
if len(sys.argv) < 2 or sys.argv[1] not in _COMMANDS:
print(f"Usage: {sys.argv[0]} <{'|'.join(_COMMANDS)}>", file=sys.stderr)
sys.exit(1)
cmd = _COMMANDS[sys.argv[1]]
cmd()
@@ -0,0 +1,217 @@
"""Helper script for tdd_plan_execute_phase_processing.robot smoke tests.
Each subcommand exercises the **CLI orchestration layer** by replicating the
logic of the ``execute_plan`` CLI handler in ``plan.py``. The helper uses
a real ``PlanLifecycleService`` (in-memory mode) and ``PlanExecutor`` (with
stub actors) to verify that the orchestration correctly handles plans in
Strategize/QUEUED state.
Bug #967: The ``plan execute`` CLI command originally only called
``service.execute_plan(plan_id)``, which is a state transition only
(Strategize/COMPLETE Execute/QUEUED). It did not construct a
``PlanExecutor`` or call ``run_strategize()`` / ``run_execute()``. The fix
added phase-aware orchestration; these tests verify it works.
This test was written to capture bug #967 per ticket #977. The bug fix is
already in the codebase, so these tests serve as permanent regression guards.
"""
from __future__ import annotations
import sys
from collections.abc import Callable
from pathlib import Path
from typing import NoReturn
_ROOT: Path = Path(__file__).resolve().parents[1]
_SRC: str = str(_ROOT / "src")
if _SRC not in sys.path:
sys.path.insert(0, _SRC)
from cleveragents.application.services.plan_executor import PlanExecutor # noqa: E402
from cleveragents.application.services.plan_lifecycle_service import ( # noqa: E402
PlanLifecycleService,
)
from cleveragents.config.settings import Settings # noqa: E402
from cleveragents.domain.models.core.plan import ( # noqa: E402
Plan,
PlanPhase,
ProcessingState,
)
def _fail(msg: str) -> NoReturn:
"""Print an error message to stderr and exit with code 1."""
print(msg, file=sys.stderr)
sys.exit(1)
def _create_service_and_queued_plan() -> tuple[PlanLifecycleService, str]:
"""Create a real in-memory PlanLifecycleService and a plan in Strategize/QUEUED."""
service: PlanLifecycleService = PlanLifecycleService(settings=Settings())
action = service.create_action(
name="local/tdd-967-robot-action",
description="TDD action for bug #967 (Robot)",
definition_of_done="Complete the test plan",
strategy_actor="local/stub-strategize",
execution_actor="local/stub-execute",
)
plan: Plan = service.use_action(action_name=str(action.namespaced_name))
return service, plan.identity.plan_id
def _cli_execute_from_queued() -> None:
"""Test the CLI orchestration path for a plan in Strategize/QUEUED.
Replicates the ``execute_plan`` CLI handler logic: detect the plan is
in Strategize/QUEUED, run strategize via ``PlanExecutor``, then
transition to Execute phase.
"""
service, plan_id = _create_service_and_queued_plan()
executor: PlanExecutor = PlanExecutor(lifecycle_service=service)
# Replicate CLI orchestration: detect phase, run strategize if QUEUED
plan: Plan = service.get_plan(plan_id)
if plan.phase == PlanPhase.STRATEGIZE and plan.state == ProcessingState.QUEUED:
executor.run_strategize(plan_id)
plan = service.get_plan(plan_id)
# Transition to Execute if still in Strategize/COMPLETE
if plan.phase == PlanPhase.STRATEGIZE and plan.state == ProcessingState.COMPLETE:
plan = service.execute_plan(plan_id)
elif plan.phase != PlanPhase.EXECUTE:
_fail(
f"Bug #967: CLI orchestration failed. Plan in unexpected state "
f"after strategize: {plan.phase.value}/{plan.state.value}"
)
if plan.phase != PlanPhase.EXECUTE:
_fail(f"Bug #967: Expected Execute phase, got {plan.phase.value}")
print("tdd-cli-execute-from-queued-ok")
def _cli_full_orchestration() -> None:
"""Test the CLI orchestration runs both strategize and execute phases.
Replicates the full ``execute_plan`` CLI handler: strategize transition
run_execute.
"""
service, plan_id = _create_service_and_queued_plan()
executor: PlanExecutor = PlanExecutor(lifecycle_service=service)
# Step 1: CLI detects Strategize/QUEUED and runs strategize
plan: Plan = service.get_plan(plan_id)
if plan.phase == PlanPhase.STRATEGIZE and plan.state in (
ProcessingState.QUEUED,
ProcessingState.PROCESSING,
):
executor.run_strategize(plan_id)
plan = service.get_plan(plan_id)
# Step 2: Transition to Execute if needed
if plan.phase == PlanPhase.STRATEGIZE and plan.state == ProcessingState.COMPLETE:
plan = service.execute_plan(plan_id)
elif plan.phase != PlanPhase.EXECUTE:
_fail(
f"Bug #967: Unexpected state after strategize: "
f"{plan.phase.value}/{plan.state.value}"
)
# Step 3: Run execute phase
plan = service.get_plan(plan_id)
if plan.phase == PlanPhase.EXECUTE and plan.state == ProcessingState.QUEUED:
executor.run_execute(plan_id)
plan = service.get_plan(plan_id)
if plan.phase != PlanPhase.EXECUTE:
_fail(f"Bug #967: Expected Execute phase, got {plan.phase.value}")
if plan.state not in (ProcessingState.COMPLETE, ProcessingState.QUEUED):
_fail(f"Bug #967: Expected COMPLETE or QUEUED state, got {plan.state.value}")
print("tdd-cli-full-orchestration-ok")
def _proper_orchestration() -> None:
"""Positive control: proper service-level orchestration works on a QUEUED plan."""
service, plan_id = _create_service_and_queued_plan()
executor: PlanExecutor = PlanExecutor(lifecycle_service=service)
# Run strategize first (QUEUED → PROCESSING → COMPLETE)
executor.run_strategize(plan_id)
# Re-fetch — auto_progress may have moved it to Execute
plan: Plan = service.get_plan(plan_id)
if plan.phase == PlanPhase.STRATEGIZE and plan.state == ProcessingState.COMPLETE:
plan = service.execute_plan(plan_id)
elif plan.phase != PlanPhase.EXECUTE:
_fail(
f"Bug #967: Unexpected state after orchestration: "
f"{plan.phase.value}/{plan.state.value}"
)
plan = service.get_plan(plan_id)
if plan.phase != PlanPhase.EXECUTE:
_fail(f"Bug #967: Expected Execute phase, got {plan.phase.value}")
if plan.state != ProcessingState.QUEUED:
_fail(f"Bug #967: Expected QUEUED state, got {plan.state.value}")
print("tdd-proper-orchestration-ok")
def _cli_auto_discovery() -> None:
"""Test the CLI auto-discovery filter includes Strategize/QUEUED plans.
Replicates the CLI auto-discovery logic from the ``execute_plan`` handler:
list plans and filter for eligible states including QUEUED.
"""
service: PlanLifecycleService = PlanLifecycleService(settings=Settings())
# Create an action
action = service.create_action(
name="local/tdd-967-discovery",
description="TDD action for auto-discovery (Robot)",
definition_of_done="Test auto-discovery",
strategy_actor="local/stub-strategize",
execution_actor="local/stub-execute",
)
action_name: str = str(action.namespaced_name)
# Plan 1: Strategize/QUEUED (fresh from use_action)
plan_queued: Plan = service.use_action(action_name=action_name)
queued_id: str = plan_queued.identity.plan_id
# Apply the PRODUCTION auto-discovery filter (the fixed version):
# includes both QUEUED and COMPLETE Strategize plans
plans_strat: list[Plan] = service.list_plans(phase=PlanPhase.STRATEGIZE)
plans_exec: list[Plan] = service.list_plans(phase=PlanPhase.EXECUTE)
eligible: list[Plan] = [
p
for p in plans_strat
if p.state in (ProcessingState.QUEUED, ProcessingState.COMPLETE)
] + [p for p in plans_exec if p.state == ProcessingState.QUEUED]
queued_in_eligible: list[Plan] = [
p for p in eligible if p.identity.plan_id == queued_id
]
if len(queued_in_eligible) == 0:
_fail(
"Bug #967: Auto-discovery filter does not find Strategize/QUEUED "
f"plans. Eligible count: {len(eligible)}, QUEUED plan: {queued_id}"
)
print("tdd-cli-auto-discovery-ok")
_COMMANDS: dict[str, Callable[[], None]] = {
"cli-execute-from-queued": _cli_execute_from_queued,
"cli-full-orchestration": _cli_full_orchestration,
"proper-orchestration": _proper_orchestration,
"cli-auto-discovery": _cli_auto_discovery,
}
if __name__ == "__main__":
if len(sys.argv) < 2 or sys.argv[1] not in _COMMANDS:
print(
f"Usage: {sys.argv[0]} <{'|'.join(_COMMANDS)}>",
file=sys.stderr,
)
sys.exit(1)
cmd: Callable[[], None] = _COMMANDS[sys.argv[1]]
cmd()
+33
View File
@@ -0,0 +1,33 @@
*** Settings ***
Documentation TDD Bug #783 — agents init --yes should not require user input
... Integration smoke tests verifying that the ``init --yes`` command
... completes without blocking for user input. Bug #783 reports that
... the migration runner still prompts for approval even with ``--yes``.
Resource ${CURDIR}/common.resource
Suite Setup Setup Test Environment auto_apply_migrations=${FALSE}
Suite Teardown Cleanup Test Environment
*** Variables ***
${HELPER} ${CURDIR}/helper_tdd_init_yes_no_input.py
*** Test Cases ***
TDD Init Yes No Input Completes Without Stdin
[Documentation] Verify that ``init --yes`` exits successfully without any
... stdin input. The helper runs the real CLI init command in
... a fresh directory with stdin closed.
[Tags] tdd_bug tdd_bug_783 tdd_expected_fail
${result}= Run Process ${PYTHON} ${HELPER} init-no-stdin cwd=${WORKSPACE} timeout=120s on_timeout=kill
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} tdd-init-yes-no-input-ok
TDD Init Yes No Migration Prompt
[Documentation] Verify that ``init --yes`` output does not contain the
... migration approval prompt text.
[Tags] tdd_bug tdd_bug_783 tdd_expected_fail
${result}= Run Process ${PYTHON} ${HELPER} init-no-prompt cwd=${WORKSPACE} timeout=120s on_timeout=kill
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} tdd-init-yes-no-prompt-ok
@@ -0,0 +1,60 @@
*** Settings ***
Documentation TDD Bug #967 — plan execute only transitions state, does not run
... strategize or execute phase processing. Integration smoke tests
... exercising the CLI orchestration layer via real
... PlanLifecycleService (in-memory) and PlanExecutor (stub actors)
... to verify that a plan in Strategize/QUEUED state can be taken
... through to Execute phase. The bug fix is already in the
... codebase; these tests serve as permanent regression guards.
Resource ${CURDIR}/common.resource
Suite Setup Setup Test Environment With Database Isolation
Suite Teardown Cleanup Test Environment
*** Variables ***
${HELPER} ${CURDIR}/helper_tdd_plan_execute_phase_processing.py
*** Test Cases ***
TDD CLI Execute Plan From Strategize QUEUED State
[Documentation] Verify that the CLI execute_plan handler correctly
... orchestrates run_strategize before transitioning when
... a plan is in Strategize/QUEUED state.
... Bug #967: originally the CLI only called service.execute_plan().
[Tags] tdd_bug tdd_bug_967
${result}= Run Process ${PYTHON} ${HELPER} cli-execute-from-queued cwd=${WORKSPACE} timeout=30s on_timeout=kill
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} tdd-cli-execute-from-queued-ok
TDD CLI Execute Orchestrates Full Lifecycle
[Documentation] Verify that the CLI execute_plan handler calls both
... run_strategize and run_execute for a QUEUED plan.
... Bug #967: the CLI skipped PlanExecutor orchestration.
[Tags] tdd_bug tdd_bug_967
${result}= Run Process ${PYTHON} ${HELPER} cli-full-orchestration cwd=${WORKSPACE} timeout=30s on_timeout=kill
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} tdd-cli-full-orchestration-ok
TDD Positive Control Proper Orchestration
[Documentation] Positive control: verify that the proper service-level
... orchestration path (run_strategize then execute_plan)
... works correctly.
[Tags] tdd_bug tdd_bug_967
${result}= Run Process ${PYTHON} ${HELPER} proper-orchestration cwd=${WORKSPACE} timeout=30s on_timeout=kill
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} tdd-proper-orchestration-ok
TDD CLI Auto Discovery Includes QUEUED Plans
[Documentation] Verify that the CLI auto-discovery filter includes plans
... in Strategize/QUEUED state, not just Strategize/COMPLETE.
... Bug #967: the original filter excluded QUEUED plans.
[Tags] tdd_bug tdd_bug_967
${result}= Run Process ${PYTHON} ${HELPER} cli-auto-discovery cwd=${WORKSPACE} timeout=30s on_timeout=kill
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} tdd-cli-auto-discovery-ok