test(resource): add failing tests for built-in fs-directory type bootstrap (#537) #567
@@ -2,6 +2,11 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Added TDD-style Behave BDD tests for the built-in `fs-directory` resource type
|
||||
bootstrap. Three scenarios: one failing TDD test reproducing bug #523 (no bootstrap
|
||||
called during init), and two regression tests verifying `bootstrap_builtin_types()`
|
||||
seeds correct data and `agents resource add fs-directory` succeeds. Includes Robot
|
||||
Framework regression tests. (#537)
|
||||
- Added TDD-style failing Behave BDD tests for the missing `agents init --yes` flag.
|
||||
Five scenarios: four TDD-failing tests (exit code, prompt suppression, `-y` alias,
|
||||
output summary) and one regression guard for interactive mode. Includes Robot
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
# These tests target bug #523 and are expected to fail until the fix is applied.
|
||||
# bootstrap_builtin_types() is never called during initialization, so the
|
||||
# built-in fs-directory resource type is not seeded into the database.
|
||||
|
||||
Feature: Built-in fs-directory Resource Type Bootstrap
|
||||
As a CleverAgents user
|
||||
I want the built-in fs-directory resource type to be available after initialization
|
||||
So that I can run "agents resource add fs-directory" without "Resource type not found"
|
||||
|
||||
# ── Bug reproduction: init path does NOT call bootstrap ────
|
||||
# This scenario is the TDD failing test. It creates a registry WITHOUT
|
||||
# calling bootstrap_builtin_types() and asserts fs-directory exists.
|
||||
|
|
||||
# It will fail until the fix integrates bootstrap into the init path.
|
||||
#
|
||||
# NOTE FOR FIX AUTHOR: If the chosen fix is to call bootstrap inside
|
||||
# init_command() / initialize_project() rather than in
|
||||
# ResourceRegistryService.__init__(), you will need to update the Given
|
||||
# step to exercise the init path instead of constructing a bare service.
|
||||
|
||||
@tdd @bug523 @wip
|
||||
Scenario: fs-directory type exists after init without explicit bootstrap call
|
||||
Given a fresh in-memory resource registry without bootstrap
|
||||
When I query the fs bootstrap resource type registry for "fs-directory"
|
||||
Then the bootstrap fs resource type "fs-directory" should exist
|
||||
|
||||
# ── Regression: bootstrap function itself works correctly ──
|
||||
|
||||
@tdd @bug523
|
||||
Scenario: After initialization fs-directory type exists in the registry
|
||||
Given a fresh in-memory resource registry with bootstrap
|
||||
When I query the fs bootstrap resource type registry for "fs-directory"
|
||||
Then the bootstrap fs resource type "fs-directory" should exist
|
||||
And the bootstrap fs resource type kind should be "physical"
|
||||
And the bootstrap fs resource type sandbox_strategy should be "copy_on_write"
|
||||
|
||||
# ── CLI add command ────────────────────────────────────────
|
||||
|
||||
@tdd @bug523
|
||||
Scenario: resource add fs-directory succeeds after bootstrap
|
||||
Given a fresh in-memory resource registry with bootstrap
|
||||
When I run resource add for type "fs-directory" named "local/test" with path "/tmp/test"
|
||||
Then the resource add command should succeed
|
||||
And the resource add output should contain "Added resource"
|
||||
And the resource add output should not contain "Resource type not found"
|
||||
@@ -0,0 +1,207 @@
|
||||
"""Step definitions for resource_type_bootstrap_fs.feature.
|
||||
|
||||
These tests target bug #523. The first scenario (without bootstrap) is a
|
||||
genuine TDD failing test: it creates a fresh registry *without* calling
|
||||
``bootstrap_builtin_types()`` and asserts ``fs-directory`` exists — which
|
||||
fails until the init path is fixed to call bootstrap.
|
||||
|
||||
The remaining scenarios verify that ``bootstrap_builtin_types()`` itself
|
||||
seeds the correct data and that the CLI ``resource add`` path works once
|
||||
the types are present.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from io import StringIO
|
||||
from typing import Any
|
||||
from unittest.mock import patch
|
||||
|
||||
from behave import given, then, when
|
||||
from behave.runner import Context
|
||||
from rich.console import Console
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from cleveragents.application.services.resource_registry_service import (
|
||||
ResourceRegistryService,
|
||||
)
|
||||
from cleveragents.core.exceptions import NotFoundError
|
||||
from cleveragents.infrastructure.database.models import Base
|
||||
|
||||
# ── Helpers ──────────────────────────────────────────────────
|
||||
|
||||
_PATCH_SERVICE = "cleveragents.cli.commands.resource._get_registry_service"
|
||||
_PATCH_CONSOLE = "cleveragents.cli.commands.resource.console"
|
||||
|
CoreRasurae
commented
F2 -- HIGH: This explicit call means the tests pass now, contradicting the "expected to fail" claim. The issue (#537), PR description, changelog, and commit messages all state these tests are expected to fail until bug #523 is fixed. But Consider adding a separate scenario that exercises the actual **F2 -- HIGH: This explicit call means the tests pass now, contradicting the "expected to fail" claim.**
The issue (#537), PR description, changelog, and commit messages all state these tests are _expected to fail_ until bug #523 is fixed. But `bootstrap_builtin_types()` itself works correctly -- the bug is that `agents init` never calls it. By manually calling it here, the tests will pass immediately.
Consider adding a separate scenario that exercises the actual `agents init` path (without this manual call) to genuinely reproduce the bug.
|
||||
|
||||
|
||||
def _make_service(context: Context, *, run_bootstrap: bool) -> ResourceRegistryService:
|
||||
"""Create an in-memory ResourceRegistryService.
|
||||
|
||||
When *run_bootstrap* is ``True`` the built-in types are seeded; when
|
||||
``False`` the registry is left empty (reproducing the bug).
|
||||
"""
|
||||
engine = create_engine("sqlite:///:memory:", echo=False)
|
||||
|
brent.edwards
commented
Minor: The manual monkey-patching approach ( The current code does use Minor: The manual monkey-patching approach (`_patch_service` / `_unpatch_service`) works, but `unittest.mock.patch` would be more robust and is the pattern used in PR #566's step definitions. With manual patching, if an exception is raised between `_patch_service()` and the `finally` block calling `_unpatch_service()`, the module state could be left dirty.
The current code does use `try/finally` which mitigates this, so it's not broken — just noting the consistency difference with the companion PRs.
|
||||
Base.metadata.create_all(engine)
|
||||
factory = sessionmaker(bind=engine, expire_on_commit=False)
|
||||
service = ResourceRegistryService(session_factory=factory)
|
||||
if run_bootstrap:
|
||||
service.bootstrap_builtin_types()
|
||||
|
CoreRasurae
commented
F3 -- MEDIUM: Inconsistent mocking pattern. The codebase convention (used in 90+ test files) is Direct attribute assignment is not thread-safe and lacks the automatic rollback guarantees that **F3 -- MEDIUM: Inconsistent mocking pattern.**
The codebase convention (used in 90+ test files) is `unittest.mock.patch`:
```python
_PATCH_TARGET = "cleveragents.cli.commands.resource._get_registry_service"
with patch(_PATCH_TARGET, return_value=mock_service): ...
```
Direct attribute assignment is not thread-safe and lacks the automatic rollback guarantees that `patch()` provides. Consider refactoring to match the established pattern.
|
||||
context.bootstrap_fs_service = service
|
||||
return service
|
||||
|
||||
|
||||
def _capture_output(func: Any, *args: Any, **kwargs: Any) -> tuple[str, bool]:
|
||||
"""Run a CLI function capturing its console output and success status."""
|
||||
buf = StringIO()
|
||||
console = Console(
|
||||
file=buf, width=200, no_color=True, highlight=False, force_terminal=False
|
||||
)
|
||||
|
||||
failed = False
|
||||
failure_reason = ""
|
||||
with patch(_PATCH_CONSOLE, console):
|
||||
try:
|
||||
func(*args, **kwargs)
|
||||
except SystemExit as exc:
|
||||
# Defensive: currently resource_add raises typer.Abort (a
|
||||
# RuntimeError subclass) on failure, not SystemExit. This
|
||||
# handler guards against future changes where the function
|
||||
# is invoked through the Typer CLI runner, which converts
|
||||
# exceptions into SystemExit.
|
||||
if exc.code not in (None, 0):
|
||||
failed = True
|
||||
failure_reason = f"SystemExit: {exc.code}"
|
||||
except Exception as exc:
|
||||
failed = True
|
||||
# typer.Abort() carries no message; str(exc) is empty.
|
||||
|
brent.edwards
commented
Substantive: This will make debugging harder for the #523 fix author when their first attempt doesn't quite work. Consider capturing the exception: That way assertion messages in the Then steps will include the actual error. Substantive: `_capture_output` catches both `SystemExit` and `Exception` but discards the actual exception. When `failed=True`, the only diagnostic available is whatever was written to the console buffer before the error — the exception message/traceback itself is lost.
This will make debugging harder for the #523 fix author when their first attempt doesn't quite work. Consider capturing the exception:
```python
failed = False
failure_reason = ""
try:
func(*args, **kwargs)
except SystemExit as exc:
failed = True
failure_reason = f"SystemExit: {exc.code}"
except Exception as exc:
failed = True
failure_reason = f"{type(exc).__name__}: {exc}"
finally:
resource_mod.console = orig_console
return buf.getvalue() + failure_reason, failed
```
That way assertion messages in the Then steps will include the actual error.
|
||||
# Fall back to the class name so failure_reason is not blank.
|
||||
reason = str(exc) if str(exc) else "(no message)"
|
||||
failure_reason = f"{type(exc).__name__}: {reason}"
|
||||
|
||||
output = buf.getvalue()
|
||||
if failure_reason:
|
||||
output = output + "\n" + failure_reason
|
||||
return output, failed
|
||||
|
||||
|
||||
|
CoreRasurae
commented
F5 -- LOW: No separator between output and failure_reason. If Consider: **F5 -- LOW: No separator between output and failure_reason.**
If `buf.getvalue()` has content, the failure reason is concatenated directly without a newline, producing hard-to-read diagnostics like:
```
[red]Resource type not found:[/red] fs-directorySystemExit: 1
```
Consider: `output = output + "\n" + failure_reason`
|
||||
# ── Given steps ──────────────────────────────────────────────
|
||||
|
||||
|
||||
@given("a fresh in-memory resource registry with bootstrap")
|
||||
def step_fresh_registry_with_bootstrap(context: Context) -> None:
|
||||
"""Set up a fresh in-memory database and run bootstrap_builtin_types."""
|
||||
_make_service(context, run_bootstrap=True)
|
||||
context.bootstrap_fs_output = ""
|
||||
context.bootstrap_fs_failed = False
|
||||
|
||||
|
||||
@given("a fresh in-memory resource registry without bootstrap")
|
||||
def step_fresh_registry_without_bootstrap(context: Context) -> None:
|
||||
"""Set up a fresh in-memory database WITHOUT running bootstrap.
|
||||
|
||||
This reproduces bug #523: the registry is empty because
|
||||
``bootstrap_builtin_types()`` is never called during initialization.
|
||||
"""
|
||||
_make_service(context, run_bootstrap=False)
|
||||
context.bootstrap_fs_output = ""
|
||||
context.bootstrap_fs_failed = False
|
||||
|
||||
|
||||
# ── When steps ───────────────────────────────────────────────
|
||||
|
||||
|
||||
@when('I query the fs bootstrap resource type registry for "{name}"')
|
||||
def step_query_registry_for_type(context: Context, name: str) -> None:
|
||||
"""Query the registry for a specific resource type by name."""
|
||||
service: ResourceRegistryService = context.bootstrap_fs_service
|
||||
try:
|
||||
context.bootstrap_fs_type_spec = service.show_type(name)
|
||||
context.bootstrap_fs_type_found = True
|
||||
except NotFoundError:
|
||||
|
CoreRasurae
commented
F4 -- MEDIUM: Missing The **F4 -- MEDIUM: Missing `image` parameter.**
The `resource_add()` signature includes `image: str | None = None` but this call omits it. While Python fills the default, explicitly passing `image=None` documents intent and guards against future signature changes.
|
||||
context.bootstrap_fs_type_spec = None
|
||||
context.bootstrap_fs_type_found = False
|
||||
|
||||
|
||||
@when('I run resource add for type "{type_name}" named "{name}" with path "{path}"')
|
||||
def step_run_resource_add_fs(
|
||||
context: Context, type_name: str, name: str, path: str
|
||||
) -> None:
|
||||
"""Run resource add command via the CLI function."""
|
||||
from cleveragents.cli.commands.resource import resource_add
|
||||
|
||||
with patch(
|
||||
_PATCH_SERVICE,
|
||||
return_value=context.bootstrap_fs_service,
|
||||
):
|
||||
output, failed = _capture_output(
|
||||
resource_add,
|
||||
type_name=type_name,
|
||||
|
brent.edwards
commented
Note: The companion PR #568 (git-checkout bootstrap) defines similar steps but with different matcher text for the same domain concepts:
No runtime conflict since the matchers are distinct, but when both merge the step vocabulary will be inconsistent for the same domain. If there's an opportunity to align them (either in this PR or #568), it would make the test suite more uniform. Low priority. Note: The companion PR #568 (git-checkout bootstrap) defines similar steps but with different matcher text for the same domain concepts:
- This PR: `the resource type "{name}" should exist`
- PR #568: `the resource type "{name}" should be present in the registry`
- This PR: `the resource type kind should be "{kind}"`
- PR #568: `the resource type "{name}" should have kind "{kind}"`
No runtime conflict since the matchers are distinct, but when both merge the step vocabulary will be inconsistent for the same domain. If there's an opportunity to align them (either in this PR or #568), it would make the test suite more uniform. Low priority.
|
||||
name=name,
|
||||
path=path,
|
||||
branch=None,
|
||||
description=None,
|
||||
image=None,
|
||||
read_only=False,
|
||||
fmt="rich",
|
||||
)
|
||||
context.bootstrap_fs_output = output
|
||||
context.bootstrap_fs_failed = failed
|
||||
|
||||
|
||||
# ── Then steps ───────────────────────────────────────────────
|
||||
|
||||
|
||||
@then('the bootstrap fs resource type "{name}" should exist')
|
||||
def step_type_should_exist(context: Context, name: str) -> None:
|
||||
"""Assert the resource type was found in the registry."""
|
||||
assert context.bootstrap_fs_type_found is True, (
|
||||
|
CoreRasurae
commented
F1 -- CRITICAL: Duplicate Behave step pattern. This exact pattern The two implementations also diverge: this one does string comparison, the other does Fix: Rename to a unique pattern, e.g. **F1 -- CRITICAL: Duplicate Behave step pattern.**
This exact pattern `'the resource type sandbox_strategy should be "{strategy}"'` is already defined in `resource_type_model_steps.py:471`. Behave uses a global step registry -- the last file loaded alphabetically wins, silently breaking whichever feature file gets the wrong implementation.
The two implementations also diverge: this one does string comparison, the other does `SandboxStrategy(strategy)` enum comparison.
**Fix:** Rename to a unique pattern, e.g. `'the bootstrap fs resource type sandbox_strategy should be "{strategy}"'` and update the `.feature` file to match.
|
||||
f"Expected resource type '{name}' to exist in the registry after "
|
||||
f"bootstrap, but it was not found."
|
||||
)
|
||||
|
||||
|
||||
@then('the bootstrap fs resource type kind should be "{kind}"')
|
||||
def step_type_kind(context: Context, kind: str) -> None:
|
||||
"""Assert the resource type has the expected kind."""
|
||||
spec = context.bootstrap_fs_type_spec
|
||||
actual = spec.resource_kind
|
||||
actual_str = actual.value if hasattr(actual, "value") else str(actual)
|
||||
assert actual_str == kind, f"Expected kind '{kind}', got '{actual_str}'"
|
||||
|
||||
|
||||
@then('the bootstrap fs resource type sandbox_strategy should be "{strategy}"')
|
||||
def step_type_sandbox(context: Context, strategy: str) -> None:
|
||||
"""Assert the resource type has the expected sandbox strategy."""
|
||||
spec = context.bootstrap_fs_type_spec
|
||||
actual = spec.sandbox_strategy
|
||||
actual_str = actual.value if hasattr(actual, "value") else str(actual)
|
||||
assert actual_str == strategy, (
|
||||
f"Expected sandbox_strategy '{strategy}', got '{actual_str}'"
|
||||
)
|
||||
|
||||
|
||||
@then("the resource add command should succeed")
|
||||
def step_resource_add_should_succeed(context: Context) -> None:
|
||||
"""Assert the resource add command did not fail."""
|
||||
assert context.bootstrap_fs_failed is False, (
|
||||
f"Expected 'resource add' to succeed, but it failed. "
|
||||
f"Output: {context.bootstrap_fs_output!r}"
|
||||
)
|
||||
|
||||
|
||||
@then('the resource add output should contain "{text}"')
|
||||
def step_fs_output_should_contain(context: Context, text: str) -> None:
|
||||
"""Assert the output contains the given text."""
|
||||
assert text in context.bootstrap_fs_output, (
|
||||
f"Expected output to contain '{text}', but it did not. "
|
||||
f"Full output: {context.bootstrap_fs_output!r}"
|
||||
)
|
||||
|
||||
|
||||
@then('the resource add output should not contain "{text}"')
|
||||
def step_output_should_not_contain(context: Context, text: str) -> None:
|
||||
"""Assert the output does not contain the given text."""
|
||||
assert text not in context.bootstrap_fs_output, (
|
||||
f"Output unexpectedly contained '{text}'. "
|
||||
f"Full output: {context.bootstrap_fs_output!r}"
|
||||
)
|
||||
@@ -0,0 +1,63 @@
|
||||
*** Settings ***
|
||||
Documentation Regression tests for built-in fs-directory type bootstrap (bug #523).
|
||||
... These tests verify that bootstrap_builtin_types() correctly seeds
|
||||
... fs-directory into the registry and that resource registration succeeds
|
||||
... after bootstrap. Both tests call bootstrap explicitly and should pass.
|
||||
Library Process
|
||||
Library OperatingSystem
|
||||
Resource ${CURDIR}/common.resource
|
||||
|
||||
Suite Setup Setup Test Environment
|
||||
Suite Teardown Cleanup Test Environment
|
||||
|
||||
*** Test Cases ***
|
||||
Bootstrap Seeds Fs Directory Type Into Registry
|
||||
[Documentation] Verify that bootstrap_builtin_types() seeds fs-directory into the
|
||||
... database so that show_type("fs-directory") succeeds.
|
||||
|
CoreRasurae
commented
F6 -- LOW: Large inline Python scripts are fragile and hard to maintain. These 15+ line Python scripts embedded as Consider extracting to standalone **F6 -- LOW: Large inline Python scripts are fragile and hard to maintain.**
These 15+ line Python scripts embedded as `Catenate` strings have no syntax highlighting, no linting, and produce unhelpful tracebacks (no line numbers). The `${SPACE}` indentation is also fragile.
Consider extracting to standalone `.py` files in `robot/scripts/` and invoking via `Run Process ${PYTHON} ${script_path}`.
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
... from sqlalchemy import create_engine
|
||||
... from sqlalchemy.orm import sessionmaker
|
||||
... from cleveragents.infrastructure.database.models import Base
|
||||
... from cleveragents.application.services.resource_registry_service import ResourceRegistryService
|
||||
... engine = create_engine("sqlite:///:memory:", echo=False)
|
||||
... Base.metadata.create_all(engine)
|
||||
... factory = sessionmaker(bind=engine, expire_on_commit=False)
|
||||
... service = ResourceRegistryService(session_factory=factory)
|
||||
... registered = service.bootstrap_builtin_types()
|
||||
... assert "fs-directory" in registered, f"fs-directory not in registered: {registered}"
|
||||
... spec = service.show_type("fs-directory")
|
||||
... assert spec.name == "fs-directory", f"name mismatch: {spec.name}"
|
||||
... kind = spec.resource_kind.value if hasattr(spec.resource_kind, "value") else str(spec.resource_kind)
|
||||
... assert kind == "physical", f"kind: {kind}"
|
||||
... strategy = spec.sandbox_strategy.value if hasattr(spec.sandbox_strategy, "value") else str(spec.sandbox_strategy)
|
||||
... assert strategy == "copy_on_write", f"strategy: {strategy}"
|
||||
... print("fs-directory bootstrap validated successfully")
|
||||
${result}= Run Process ${PYTHON} -c ${script} timeout=60s
|
||||
Should Be Equal As Integers ${result.rc} 0 fs-directory bootstrap failed: ${result.stderr}
|
||||
Should Contain ${result.stdout} fs-directory bootstrap validated successfully
|
||||
|
||||
Resource Add Fs Directory Succeeds After Bootstrap
|
||||
[Documentation] Verify that after bootstrap, registering an fs-directory resource
|
||||
... instance succeeds (no "Resource type not found" error).
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
|
brent.edwards
commented
Nit: Both Robot test cases repeat the engine/session/service setup boilerplate (lines 17-24 and 42-50). Could extract a shared Robot keyword or a Python helper script to reduce duplication. Not blocking — the inline scripts are readable and consistent with other Robot tests in the project. Nit: Both Robot test cases repeat the engine/session/service setup boilerplate (lines 17-24 and 42-50). Could extract a shared Robot keyword or a Python helper script to reduce duplication. Not blocking — the inline scripts are readable and consistent with other Robot tests in the project.
|
||||
... from sqlalchemy import create_engine
|
||||
... from sqlalchemy.orm import sessionmaker
|
||||
... from cleveragents.infrastructure.database.models import Base
|
||||
... from cleveragents.application.services.resource_registry_service import ResourceRegistryService
|
||||
... engine = create_engine("sqlite:///:memory:", echo=False)
|
||||
... Base.metadata.create_all(engine)
|
||||
... factory = sessionmaker(bind=engine, expire_on_commit=False)
|
||||
... service = ResourceRegistryService(session_factory=factory)
|
||||
... service.bootstrap_builtin_types()
|
||||
... resource = service.register_resource(
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type_name="fs-directory",
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}name="local/test",
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}location="/tmp/test",
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description="Test fs-directory resource",
|
||||
... )
|
||||
... assert resource.resource_type_name == "fs-directory", f"type: {resource.resource_type_name}"
|
||||
... assert resource.name == "local/test", f"name: {resource.name}"
|
||||
... print("resource add fs-directory succeeded after bootstrap")
|
||||
${result}= Run Process ${PYTHON} -c ${script} timeout=60s
|
||||
Should Be Equal As Integers ${result.rc} 0 resource add fs-directory failed: ${result.stderr}
|
||||
Should Contain ${result.stdout} resource add fs-directory succeeded after bootstrap
|
||||
Minor: The scenarios in PR #566 use
@tdd @bug522tags, which allow selective execution (e.g.,behave --tags=@tddto run only TDD tests, orbehave --tags=~@tddto skip them). These scenarios don't have equivalent tags.Consider adding
@tdd @bug523tags for consistency: