test(resource): add failing tests for built-in fs-directory type bootstrap (#537) #567

Merged
brent.edwards merged 2 commits from feature/m3-test-resource-bootstrap-fs into master 2026-03-07 02:31:45 +00:00
4 changed files with 319 additions and 0 deletions
+5
View File
@@ -2,6 +2,11 @@
## Unreleased
- Added TDD-style Behave BDD tests for the built-in `fs-directory` resource type
bootstrap. Three scenarios: one failing TDD test reproducing bug #523 (no bootstrap
called during init), and two regression tests verifying `bootstrap_builtin_types()`
seeds correct data and `agents resource add fs-directory` succeeds. Includes Robot
Framework regression tests. (#537)
- Added TDD-style failing Behave BDD tests for the missing `agents init --yes` flag.
Five scenarios: four TDD-failing tests (exit code, prompt suppression, `-y` alias,
output summary) and one regression guard for interactive mode. Includes Robot
@@ -0,0 +1,44 @@
# These tests target bug #523 and are expected to fail until the fix is applied.
# bootstrap_builtin_types() is never called during initialization, so the
# built-in fs-directory resource type is not seeded into the database.
Feature: Built-in fs-directory Resource Type Bootstrap
As a CleverAgents user
I want the built-in fs-directory resource type to be available after initialization
So that I can run "agents resource add fs-directory" without "Resource type not found"
# Bug reproduction: init path does NOT call bootstrap
# This scenario is the TDD failing test. It creates a registry WITHOUT
# calling bootstrap_builtin_types() and asserts fs-directory exists.
Outdated
Review

Minor: The scenarios in PR #566 use @tdd @bug522 tags, which allow selective execution (e.g., behave --tags=@tdd to run only TDD tests, or behave --tags=~@tdd to skip them). These scenarios don't have equivalent tags.

Consider adding @tdd @bug523 tags for consistency:

  @tdd @bug523
  Scenario: After initialization fs-directory type exists in the registry
Minor: The scenarios in PR #566 use `@tdd @bug522` tags, which allow selective execution (e.g., `behave --tags=@tdd` to run only TDD tests, or `behave --tags=~@tdd` to skip them). These scenarios don't have equivalent tags. Consider adding `@tdd @bug523` tags for consistency: ```gherkin @tdd @bug523 Scenario: After initialization fs-directory type exists in the registry ```
# It will fail until the fix integrates bootstrap into the init path.
#
# NOTE FOR FIX AUTHOR: If the chosen fix is to call bootstrap inside
# init_command() / initialize_project() rather than in
# ResourceRegistryService.__init__(), you will need to update the Given
# step to exercise the init path instead of constructing a bare service.
@tdd @bug523 @wip
Scenario: fs-directory type exists after init without explicit bootstrap call
Given a fresh in-memory resource registry without bootstrap
When I query the fs bootstrap resource type registry for "fs-directory"
Then the bootstrap fs resource type "fs-directory" should exist
# ── Regression: bootstrap function itself works correctly ──
@tdd @bug523
Scenario: After initialization fs-directory type exists in the registry
Given a fresh in-memory resource registry with bootstrap
When I query the fs bootstrap resource type registry for "fs-directory"
Then the bootstrap fs resource type "fs-directory" should exist
And the bootstrap fs resource type kind should be "physical"
And the bootstrap fs resource type sandbox_strategy should be "copy_on_write"
# ── CLI add command ────────────────────────────────────────
@tdd @bug523
Scenario: resource add fs-directory succeeds after bootstrap
Given a fresh in-memory resource registry with bootstrap
When I run resource add for type "fs-directory" named "local/test" with path "/tmp/test"
Then the resource add command should succeed
And the resource add output should contain "Added resource"
And the resource add output should not contain "Resource type not found"
@@ -0,0 +1,207 @@
"""Step definitions for resource_type_bootstrap_fs.feature.
These tests target bug #523. The first scenario (without bootstrap) is a
genuine TDD failing test: it creates a fresh registry *without* calling
``bootstrap_builtin_types()`` and asserts ``fs-directory`` exists — which
fails until the init path is fixed to call bootstrap.
The remaining scenarios verify that ``bootstrap_builtin_types()`` itself
seeds the correct data and that the CLI ``resource add`` path works once
the types are present.
"""
from __future__ import annotations
from io import StringIO
from typing import Any
from unittest.mock import patch
from behave import given, then, when
from behave.runner import Context
from rich.console import Console
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from cleveragents.application.services.resource_registry_service import (
ResourceRegistryService,
)
from cleveragents.core.exceptions import NotFoundError
from cleveragents.infrastructure.database.models import Base
# ── Helpers ──────────────────────────────────────────────────
_PATCH_SERVICE = "cleveragents.cli.commands.resource._get_registry_service"
_PATCH_CONSOLE = "cleveragents.cli.commands.resource.console"
Outdated
Review

F2 -- HIGH: This explicit call means the tests pass now, contradicting the "expected to fail" claim.

The issue (#537), PR description, changelog, and commit messages all state these tests are expected to fail until bug #523 is fixed. But bootstrap_builtin_types() itself works correctly -- the bug is that agents init never calls it. By manually calling it here, the tests will pass immediately.

Consider adding a separate scenario that exercises the actual agents init path (without this manual call) to genuinely reproduce the bug.

**F2 -- HIGH: This explicit call means the tests pass now, contradicting the "expected to fail" claim.** The issue (#537), PR description, changelog, and commit messages all state these tests are _expected to fail_ until bug #523 is fixed. But `bootstrap_builtin_types()` itself works correctly -- the bug is that `agents init` never calls it. By manually calling it here, the tests will pass immediately. Consider adding a separate scenario that exercises the actual `agents init` path (without this manual call) to genuinely reproduce the bug.
def _make_service(context: Context, *, run_bootstrap: bool) -> ResourceRegistryService:
"""Create an in-memory ResourceRegistryService.
When *run_bootstrap* is ``True`` the built-in types are seeded; when
``False`` the registry is left empty (reproducing the bug).
"""
engine = create_engine("sqlite:///:memory:", echo=False)
Outdated
Review

Minor: The manual monkey-patching approach (_patch_service / _unpatch_service) works, but unittest.mock.patch would be more robust and is the pattern used in PR #566's step definitions. With manual patching, if an exception is raised between _patch_service() and the finally block calling _unpatch_service(), the module state could be left dirty.

The current code does use try/finally which mitigates this, so it's not broken — just noting the consistency difference with the companion PRs.

Minor: The manual monkey-patching approach (`_patch_service` / `_unpatch_service`) works, but `unittest.mock.patch` would be more robust and is the pattern used in PR #566's step definitions. With manual patching, if an exception is raised between `_patch_service()` and the `finally` block calling `_unpatch_service()`, the module state could be left dirty. The current code does use `try/finally` which mitigates this, so it's not broken — just noting the consistency difference with the companion PRs.
Base.metadata.create_all(engine)
factory = sessionmaker(bind=engine, expire_on_commit=False)
service = ResourceRegistryService(session_factory=factory)
if run_bootstrap:
service.bootstrap_builtin_types()
Outdated
Review

F3 -- MEDIUM: Inconsistent mocking pattern.

The codebase convention (used in 90+ test files) is unittest.mock.patch:

_PATCH_TARGET = "cleveragents.cli.commands.resource._get_registry_service"
with patch(_PATCH_TARGET, return_value=mock_service): ...

Direct attribute assignment is not thread-safe and lacks the automatic rollback guarantees that patch() provides. Consider refactoring to match the established pattern.

**F3 -- MEDIUM: Inconsistent mocking pattern.** The codebase convention (used in 90+ test files) is `unittest.mock.patch`: ```python _PATCH_TARGET = "cleveragents.cli.commands.resource._get_registry_service" with patch(_PATCH_TARGET, return_value=mock_service): ... ``` Direct attribute assignment is not thread-safe and lacks the automatic rollback guarantees that `patch()` provides. Consider refactoring to match the established pattern.
context.bootstrap_fs_service = service
return service
def _capture_output(func: Any, *args: Any, **kwargs: Any) -> tuple[str, bool]:
"""Run a CLI function capturing its console output and success status."""
buf = StringIO()
console = Console(
file=buf, width=200, no_color=True, highlight=False, force_terminal=False
)
failed = False
failure_reason = ""
with patch(_PATCH_CONSOLE, console):
try:
func(*args, **kwargs)
except SystemExit as exc:
# Defensive: currently resource_add raises typer.Abort (a
# RuntimeError subclass) on failure, not SystemExit. This
# handler guards against future changes where the function
# is invoked through the Typer CLI runner, which converts
# exceptions into SystemExit.
if exc.code not in (None, 0):
failed = True
failure_reason = f"SystemExit: {exc.code}"
except Exception as exc:
failed = True
# typer.Abort() carries no message; str(exc) is empty.
Outdated
Review

Substantive: _capture_output catches both SystemExit and Exception but discards the actual exception. When failed=True, the only diagnostic available is whatever was written to the console buffer before the error — the exception message/traceback itself is lost.

This will make debugging harder for the #523 fix author when their first attempt doesn't quite work. Consider capturing the exception:

failed = False
failure_reason = ""
try:
    func(*args, **kwargs)
except SystemExit as exc:
    failed = True
    failure_reason = f"SystemExit: {exc.code}"
except Exception as exc:
    failed = True
    failure_reason = f"{type(exc).__name__}: {exc}"
finally:
    resource_mod.console = orig_console

return buf.getvalue() + failure_reason, failed

That way assertion messages in the Then steps will include the actual error.

Substantive: `_capture_output` catches both `SystemExit` and `Exception` but discards the actual exception. When `failed=True`, the only diagnostic available is whatever was written to the console buffer before the error — the exception message/traceback itself is lost. This will make debugging harder for the #523 fix author when their first attempt doesn't quite work. Consider capturing the exception: ```python failed = False failure_reason = "" try: func(*args, **kwargs) except SystemExit as exc: failed = True failure_reason = f"SystemExit: {exc.code}" except Exception as exc: failed = True failure_reason = f"{type(exc).__name__}: {exc}" finally: resource_mod.console = orig_console return buf.getvalue() + failure_reason, failed ``` That way assertion messages in the Then steps will include the actual error.
# Fall back to the class name so failure_reason is not blank.
reason = str(exc) if str(exc) else "(no message)"
failure_reason = f"{type(exc).__name__}: {reason}"
output = buf.getvalue()
if failure_reason:
output = output + "\n" + failure_reason
return output, failed
Outdated
Review

F5 -- LOW: No separator between output and failure_reason.

If buf.getvalue() has content, the failure reason is concatenated directly without a newline, producing hard-to-read diagnostics like:

[red]Resource type not found:[/red] fs-directorySystemExit: 1

Consider: output = output + "\n" + failure_reason

**F5 -- LOW: No separator between output and failure_reason.** If `buf.getvalue()` has content, the failure reason is concatenated directly without a newline, producing hard-to-read diagnostics like: ``` [red]Resource type not found:[/red] fs-directorySystemExit: 1 ``` Consider: `output = output + "\n" + failure_reason`
# ── Given steps ──────────────────────────────────────────────
@given("a fresh in-memory resource registry with bootstrap")
def step_fresh_registry_with_bootstrap(context: Context) -> None:
"""Set up a fresh in-memory database and run bootstrap_builtin_types."""
_make_service(context, run_bootstrap=True)
context.bootstrap_fs_output = ""
context.bootstrap_fs_failed = False
@given("a fresh in-memory resource registry without bootstrap")
def step_fresh_registry_without_bootstrap(context: Context) -> None:
"""Set up a fresh in-memory database WITHOUT running bootstrap.
This reproduces bug #523: the registry is empty because
``bootstrap_builtin_types()`` is never called during initialization.
"""
_make_service(context, run_bootstrap=False)
context.bootstrap_fs_output = ""
context.bootstrap_fs_failed = False
# ── When steps ───────────────────────────────────────────────
@when('I query the fs bootstrap resource type registry for "{name}"')
def step_query_registry_for_type(context: Context, name: str) -> None:
"""Query the registry for a specific resource type by name."""
service: ResourceRegistryService = context.bootstrap_fs_service
try:
context.bootstrap_fs_type_spec = service.show_type(name)
context.bootstrap_fs_type_found = True
except NotFoundError:
Outdated
Review

F4 -- MEDIUM: Missing image parameter.

The resource_add() signature includes image: str | None = None but this call omits it. While Python fills the default, explicitly passing image=None documents intent and guards against future signature changes.

**F4 -- MEDIUM: Missing `image` parameter.** The `resource_add()` signature includes `image: str | None = None` but this call omits it. While Python fills the default, explicitly passing `image=None` documents intent and guards against future signature changes.
context.bootstrap_fs_type_spec = None
context.bootstrap_fs_type_found = False
@when('I run resource add for type "{type_name}" named "{name}" with path "{path}"')
def step_run_resource_add_fs(
context: Context, type_name: str, name: str, path: str
) -> None:
"""Run resource add command via the CLI function."""
from cleveragents.cli.commands.resource import resource_add
with patch(
_PATCH_SERVICE,
return_value=context.bootstrap_fs_service,
):
output, failed = _capture_output(
resource_add,
type_name=type_name,
Outdated
Review

Note: The companion PR #568 (git-checkout bootstrap) defines similar steps but with different matcher text for the same domain concepts:

  • This PR: the resource type "{name}" should exist

  • PR #568: the resource type "{name}" should be present in the registry

  • This PR: the resource type kind should be "{kind}"

  • PR #568: the resource type "{name}" should have kind "{kind}"

No runtime conflict since the matchers are distinct, but when both merge the step vocabulary will be inconsistent for the same domain. If there's an opportunity to align them (either in this PR or #568), it would make the test suite more uniform. Low priority.

Note: The companion PR #568 (git-checkout bootstrap) defines similar steps but with different matcher text for the same domain concepts: - This PR: `the resource type "{name}" should exist` - PR #568: `the resource type "{name}" should be present in the registry` - This PR: `the resource type kind should be "{kind}"` - PR #568: `the resource type "{name}" should have kind "{kind}"` No runtime conflict since the matchers are distinct, but when both merge the step vocabulary will be inconsistent for the same domain. If there's an opportunity to align them (either in this PR or #568), it would make the test suite more uniform. Low priority.
name=name,
path=path,
branch=None,
description=None,
image=None,
read_only=False,
fmt="rich",
)
context.bootstrap_fs_output = output
context.bootstrap_fs_failed = failed
# ── Then steps ───────────────────────────────────────────────
@then('the bootstrap fs resource type "{name}" should exist')
def step_type_should_exist(context: Context, name: str) -> None:
"""Assert the resource type was found in the registry."""
assert context.bootstrap_fs_type_found is True, (
Outdated
Review

F1 -- CRITICAL: Duplicate Behave step pattern.

This exact pattern 'the resource type sandbox_strategy should be "{strategy}"' is already defined in resource_type_model_steps.py:471. Behave uses a global step registry -- the last file loaded alphabetically wins, silently breaking whichever feature file gets the wrong implementation.

The two implementations also diverge: this one does string comparison, the other does SandboxStrategy(strategy) enum comparison.

Fix: Rename to a unique pattern, e.g. 'the bootstrap fs resource type sandbox_strategy should be "{strategy}"' and update the .feature file to match.

**F1 -- CRITICAL: Duplicate Behave step pattern.** This exact pattern `'the resource type sandbox_strategy should be "{strategy}"'` is already defined in `resource_type_model_steps.py:471`. Behave uses a global step registry -- the last file loaded alphabetically wins, silently breaking whichever feature file gets the wrong implementation. The two implementations also diverge: this one does string comparison, the other does `SandboxStrategy(strategy)` enum comparison. **Fix:** Rename to a unique pattern, e.g. `'the bootstrap fs resource type sandbox_strategy should be "{strategy}"'` and update the `.feature` file to match.
f"Expected resource type '{name}' to exist in the registry after "
f"bootstrap, but it was not found."
)
@then('the bootstrap fs resource type kind should be "{kind}"')
def step_type_kind(context: Context, kind: str) -> None:
"""Assert the resource type has the expected kind."""
spec = context.bootstrap_fs_type_spec
actual = spec.resource_kind
actual_str = actual.value if hasattr(actual, "value") else str(actual)
assert actual_str == kind, f"Expected kind '{kind}', got '{actual_str}'"
@then('the bootstrap fs resource type sandbox_strategy should be "{strategy}"')
def step_type_sandbox(context: Context, strategy: str) -> None:
"""Assert the resource type has the expected sandbox strategy."""
spec = context.bootstrap_fs_type_spec
actual = spec.sandbox_strategy
actual_str = actual.value if hasattr(actual, "value") else str(actual)
assert actual_str == strategy, (
f"Expected sandbox_strategy '{strategy}', got '{actual_str}'"
)
@then("the resource add command should succeed")
def step_resource_add_should_succeed(context: Context) -> None:
"""Assert the resource add command did not fail."""
assert context.bootstrap_fs_failed is False, (
f"Expected 'resource add' to succeed, but it failed. "
f"Output: {context.bootstrap_fs_output!r}"
)
@then('the resource add output should contain "{text}"')
def step_fs_output_should_contain(context: Context, text: str) -> None:
"""Assert the output contains the given text."""
assert text in context.bootstrap_fs_output, (
f"Expected output to contain '{text}', but it did not. "
f"Full output: {context.bootstrap_fs_output!r}"
)
@then('the resource add output should not contain "{text}"')
def step_output_should_not_contain(context: Context, text: str) -> None:
"""Assert the output does not contain the given text."""
assert text not in context.bootstrap_fs_output, (
f"Output unexpectedly contained '{text}'. "
f"Full output: {context.bootstrap_fs_output!r}"
)
+63
View File
@@ -0,0 +1,63 @@
*** Settings ***
Documentation Regression tests for built-in fs-directory type bootstrap (bug #523).
... These tests verify that bootstrap_builtin_types() correctly seeds
... fs-directory into the registry and that resource registration succeeds
... after bootstrap. Both tests call bootstrap explicitly and should pass.
Library Process
Library OperatingSystem
Resource ${CURDIR}/common.resource
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
*** Test Cases ***
Bootstrap Seeds Fs Directory Type Into Registry
[Documentation] Verify that bootstrap_builtin_types() seeds fs-directory into the
... database so that show_type("fs-directory") succeeds.
Outdated
Review

F6 -- LOW: Large inline Python scripts are fragile and hard to maintain.

These 15+ line Python scripts embedded as Catenate strings have no syntax highlighting, no linting, and produce unhelpful tracebacks (no line numbers). The ${SPACE} indentation is also fragile.

Consider extracting to standalone .py files in robot/scripts/ and invoking via Run Process ${PYTHON} ${script_path}.

**F6 -- LOW: Large inline Python scripts are fragile and hard to maintain.** These 15+ line Python scripts embedded as `Catenate` strings have no syntax highlighting, no linting, and produce unhelpful tracebacks (no line numbers). The `${SPACE}` indentation is also fragile. Consider extracting to standalone `.py` files in `robot/scripts/` and invoking via `Run Process ${PYTHON} ${script_path}`.
${script}= Catenate SEPARATOR=\n
... from sqlalchemy import create_engine
... from sqlalchemy.orm import sessionmaker
... from cleveragents.infrastructure.database.models import Base
... from cleveragents.application.services.resource_registry_service import ResourceRegistryService
... engine = create_engine("sqlite:///:memory:", echo=False)
... Base.metadata.create_all(engine)
... factory = sessionmaker(bind=engine, expire_on_commit=False)
... service = ResourceRegistryService(session_factory=factory)
... registered = service.bootstrap_builtin_types()
... assert "fs-directory" in registered, f"fs-directory not in registered: {registered}"
... spec = service.show_type("fs-directory")
... assert spec.name == "fs-directory", f"name mismatch: {spec.name}"
... kind = spec.resource_kind.value if hasattr(spec.resource_kind, "value") else str(spec.resource_kind)
... assert kind == "physical", f"kind: {kind}"
... strategy = spec.sandbox_strategy.value if hasattr(spec.sandbox_strategy, "value") else str(spec.sandbox_strategy)
... assert strategy == "copy_on_write", f"strategy: {strategy}"
... print("fs-directory bootstrap validated successfully")
${result}= Run Process ${PYTHON} -c ${script} timeout=60s
Should Be Equal As Integers ${result.rc} 0 fs-directory bootstrap failed: ${result.stderr}
Should Contain ${result.stdout} fs-directory bootstrap validated successfully
Resource Add Fs Directory Succeeds After Bootstrap
[Documentation] Verify that after bootstrap, registering an fs-directory resource
... instance succeeds (no "Resource type not found" error).
${script}= Catenate SEPARATOR=\n
Outdated
Review

Nit: Both Robot test cases repeat the engine/session/service setup boilerplate (lines 17-24 and 42-50). Could extract a shared Robot keyword or a Python helper script to reduce duplication. Not blocking — the inline scripts are readable and consistent with other Robot tests in the project.

Nit: Both Robot test cases repeat the engine/session/service setup boilerplate (lines 17-24 and 42-50). Could extract a shared Robot keyword or a Python helper script to reduce duplication. Not blocking — the inline scripts are readable and consistent with other Robot tests in the project.
... from sqlalchemy import create_engine
... from sqlalchemy.orm import sessionmaker
... from cleveragents.infrastructure.database.models import Base
... from cleveragents.application.services.resource_registry_service import ResourceRegistryService
... engine = create_engine("sqlite:///:memory:", echo=False)
... Base.metadata.create_all(engine)
... factory = sessionmaker(bind=engine, expire_on_commit=False)
... service = ResourceRegistryService(session_factory=factory)
... service.bootstrap_builtin_types()
... resource = service.register_resource(
... ${SPACE}${SPACE}${SPACE}${SPACE}type_name="fs-directory",
... ${SPACE}${SPACE}${SPACE}${SPACE}name="local/test",
... ${SPACE}${SPACE}${SPACE}${SPACE}location="/tmp/test",
... ${SPACE}${SPACE}${SPACE}${SPACE}description="Test fs-directory resource",
... )
... assert resource.resource_type_name == "fs-directory", f"type: {resource.resource_type_name}"
... assert resource.name == "local/test", f"name: {resource.name}"
... print("resource add fs-directory succeeded after bootstrap")
${result}= Run Process ${PYTHON} -c ${script} timeout=60s
Should Be Equal As Integers ${result.rc} 0 resource add fs-directory failed: ${result.stderr}
Should Contain ${result.stdout} resource add fs-directory succeeded after bootstrap