test(resource): add failing tests for built-in git-checkout type bootstrap (#524) #568

Merged
brent.edwards merged 12 commits from feature/m3-test-resource-bootstrap-git into master 2026-03-07 02:52:51 +00:00
4 changed files with 327 additions and 0 deletions
+5
View File
@@ -2,6 +2,11 @@
## Unreleased
- Added TDD-style Behave BDD tests for the built-in `git-checkout` resource type
bootstrap. Three scenarios: one failing TDD test reproducing bug #524 (no bootstrap
called during init), and two regression tests verifying `bootstrap_builtin_types()`
seeds correct data and `agents resource add git-checkout` succeeds. Includes Robot
Framework regression tests. (#553)
- Added TDD-style Behave BDD tests for the built-in `fs-directory` resource type
bootstrap. Three scenarios: one failing TDD test reproducing bug #523 (no bootstrap
called during init), and two regression tests verifying `bootstrap_builtin_types()`
@@ -0,0 +1,44 @@
# These tests target bug #524 and are expected to fail until the fix is applied.
# bootstrap_builtin_types() is never called during initialization, so the
# git-checkout built-in resource type is missing from the registry at runtime.
Feature: Built-in git-checkout type bootstrap on initialization
As a CleverAgents user
I want the built-in git-checkout resource type to be available after initialization
So that I can run "agents resource add git-checkout" without "Resource type not found"
# Bug reproduction: init path does NOT call bootstrap
# This scenario is the TDD failing test. It creates a registry WITHOUT
# calling bootstrap_builtin_types() and asserts git-checkout exists.
Outdated
Review

Minor: The scenarios in PR #566 use @tdd @bug522 tags, which allow selective execution (e.g., behave --tags=@tdd to run only TDD tests, or --tags=~@tdd to skip them). This PR and the companion PR #567 don't have scenario-level tags.

Consider adding @tdd @bug524 for consistency:

  @tdd @bug524
  Scenario: After initialization the git-checkout type exists in the resource type registry
Minor: The scenarios in PR #566 use `@tdd @bug522` tags, which allow selective execution (e.g., `behave --tags=@tdd` to run only TDD tests, or `--tags=~@tdd` to skip them). This PR and the companion PR #567 don't have scenario-level tags. Consider adding `@tdd @bug524` for consistency: ```gherkin @tdd @bug524 Scenario: After initialization the git-checkout type exists in the resource type registry ```
# It will fail until the fix integrates bootstrap into the init path.
#
# NOTE FOR FIX AUTHOR: If the chosen fix is to call bootstrap inside
# init_command() / initialize_project() rather than in
# ResourceRegistryService.__init__(), you will need to update the Given
# step to exercise the init path instead of constructing a bare service.
@tdd @bug524 @wip
Scenario: git-checkout type is missing when bootstrap is not called during init
Given a bootstrap-git fresh in-memory resource registry without bootstrap
When I query the bootstrap-git resource type registry for "git-checkout"
Then the bootstrap-git resource type "git-checkout" should exist
# ── Regression: bootstrap function itself works correctly ──
@tdd @bug524
Scenario: After initialization the git-checkout type exists in the resource type registry
Given a bootstrap-git fresh in-memory resource registry with bootstrap
When I query the bootstrap-git resource type registry for "git-checkout"
Then the bootstrap-git resource type "git-checkout" should exist
And the bootstrap-git resource type "git-checkout" should have kind "physical"
And the bootstrap-git resource type "git-checkout" should have sandbox_strategy "git_worktree"
And the bootstrap-git resource type "git-checkout" should be user_addable
And the bootstrap-git registered types should include "git-checkout"
# ── CLI resource add succeeds ──────────────────────────────
@tdd @bug524
Scenario: agents resource add git-checkout succeeds without Resource type not found error
Given a bootstrap-git fresh in-memory resource registry with bootstrap
When I run bootstrap-git resource add for type "git-checkout" named "local/test" with path "/tmp/repo" and branch "main"
Then the bootstrap-git resource add command should succeed
And the bootstrap-git resource add output should not contain "Resource type not found"
@@ -0,0 +1,211 @@
"""Step definitions for resource_type_bootstrap_git.feature.
These tests target bug #524. The first scenario (without bootstrap) is a
genuine TDD failing test: it creates a fresh registry *without* calling
``bootstrap_builtin_types()`` and asserts ``git-checkout`` exists — which
fails until the init path is fixed to call bootstrap.
The remaining scenarios verify that ``bootstrap_builtin_types()`` itself
seeds the correct data and that the CLI ``resource add`` path works once
the types are present.
"""
from __future__ import annotations
from io import StringIO
from typing import Any
from unittest.mock import patch
from behave import given, then, when # type: ignore[import-untyped]
from behave.runner import Context # type: ignore[import-untyped]
from rich.console import Console
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
Review

Nit: runner is created as a module-level singleton, while PR #566's step file creates a new CliRunner() per step invocation. Both approaches are correct (CliRunner is stateless), just noting the style difference across the companion PRs.

Nit: `runner` is created as a module-level singleton, while PR #566's step file creates a new `CliRunner()` per step invocation. Both approaches are correct (CliRunner is stateless), just noting the style difference across the companion PRs.
from cleveragents.application.services.resource_registry_service import (
ResourceRegistryService,
)
from cleveragents.core.exceptions import NotFoundError
from cleveragents.infrastructure.database.models import Base
# ── Helpers ──────────────────────────────────────────────────
_PATCH_SERVICE = "cleveragents.cli.commands.resource._get_registry_service"
_PATCH_CONSOLE = "cleveragents.cli.commands.resource.console"
def _make_service(context: Context, *, run_bootstrap: bool) -> ResourceRegistryService:
"""Create an in-memory ResourceRegistryService.
When *run_bootstrap* is ``True`` the built-in types are seeded; when
``False`` the registry is left empty (reproducing the bug).
"""
engine = create_engine("sqlite:///:memory:", echo=False)
Base.metadata.create_all(engine)
factory = sessionmaker(bind=engine, expire_on_commit=False)
service = ResourceRegistryService(session_factory=factory)
if run_bootstrap:
context.bootstrap_git_registered = service.bootstrap_builtin_types() # type: ignore[attr-defined]
context.bootstrap_git_service = service # type: ignore[attr-defined]
return service
def _capture_output(func: Any, *args: Any, **kwargs: Any) -> tuple[str, bool]:
"""Run a CLI function capturing its console output and success status."""
buf = StringIO()
console = Console(
file=buf, width=200, no_color=True, highlight=False, force_terminal=False
)
failed = False
failure_reason = ""
with patch(_PATCH_CONSOLE, console):
try:
func(*args, **kwargs)
except SystemExit as exc:
if exc.code not in (None, 0):
failed = True
failure_reason = f"SystemExit: {exc.code}"
except Exception as exc:
failed = True
failure_reason = f"{type(exc).__name__}: {exc}"
output = buf.getvalue()
if failure_reason:
output = output + "\n" + failure_reason
return output, failed
# ── Given steps ──────────────────────────────────────────────
@given("a bootstrap-git fresh in-memory resource registry with bootstrap")
def step_fresh_registry_with_bootstrap(context: Context) -> None:
"""Set up a fresh in-memory database and run bootstrap_builtin_types."""
_make_service(context, run_bootstrap=True)
context.bootstrap_git_output = "" # type: ignore[attr-defined]
context.bootstrap_git_failed = False # type: ignore[attr-defined]
@given("a bootstrap-git fresh in-memory resource registry without bootstrap")
def step_fresh_registry_without_bootstrap(context: Context) -> None:
"""Set up a fresh in-memory database WITHOUT running bootstrap.
This reproduces bug #524: the registry is empty because
``bootstrap_builtin_types()`` is never called during initialization.
"""
_make_service(context, run_bootstrap=False)
context.bootstrap_git_output = "" # type: ignore[attr-defined]
context.bootstrap_git_failed = False # type: ignore[attr-defined]
# ── When steps ───────────────────────────────────────────────
@when('I query the bootstrap-git resource type registry for "{name}"')
Outdated
Review

Note: The step matcher text here differs from the companion PR #567 for the same domain concepts:

Concept This PR (#568) PR #567
Type existence should be present in the registry should exist
Kind check "{name}" should have kind "{kind}" kind should be "{kind}"
Strategy check "{name}" should have sandbox_strategy sandbox_strategy should be

No runtime conflict since the matchers are distinct, and I'd argue this PR's patterns are better (parameterized by {name}, so they generalize to any type). But when both merge, the step vocabulary will be inconsistent for the same domain.

If there's an opportunity to align #567 to match this PR's patterns, it would make the test suite more uniform. Low priority — could be done in a follow-up.

Note: The step matcher text here differs from the companion PR #567 for the same domain concepts: | Concept | This PR (#568) | PR #567 | |---|---|---| | Type existence | `should be present in the registry` | `should exist` | | Kind check | `"{name}" should have kind "{kind}"` | `kind should be "{kind}"` | | Strategy check | `"{name}" should have sandbox_strategy` | `sandbox_strategy should be` | No runtime conflict since the matchers are distinct, and I'd argue this PR's patterns are better (parameterized by `{name}`, so they generalize to any type). But when both merge, the step vocabulary will be inconsistent for the same domain. If there's an opportunity to align #567 to match this PR's patterns, it would make the test suite more uniform. Low priority — could be done in a follow-up.
def step_query_registry_for_type(context: Context, name: str) -> None:
"""Query the registry for a specific resource type by name."""
service: ResourceRegistryService = context.bootstrap_git_service # type: ignore[attr-defined]
try:
context.bootstrap_git_type_spec = service.show_type(name) # type: ignore[attr-defined]
context.bootstrap_git_type_found = True # type: ignore[attr-defined]
except NotFoundError:
context.bootstrap_git_type_spec = None # type: ignore[attr-defined]
context.bootstrap_git_type_found = False # type: ignore[attr-defined]
@when(
'I run bootstrap-git resource add for type "{type_name}" named "{name}" with path "{path}" and branch "{branch}"'
)
def step_run_resource_add_git(
context: Context, type_name: str, name: str, path: str, branch: str
) -> None:
"""Run resource add command via the CLI function."""
from cleveragents.cli.commands.resource import resource_add
with patch(
_PATCH_SERVICE,
return_value=context.bootstrap_git_service, # type: ignore[attr-defined]
):
output, failed = _capture_output(
resource_add,
type_name=type_name,
name=name,
path=path,
branch=branch,
description=None,
image=None,
read_only=False,
fmt="rich",
)
context.bootstrap_git_output = output # type: ignore[attr-defined]
context.bootstrap_git_failed = failed # type: ignore[attr-defined]
# ── Then steps ───────────────────────────────────────────────
@then('the bootstrap-git resource type "{name}" should exist')
def step_type_should_exist(context: Context, name: str) -> None:
"""Assert the resource type was found in the registry."""
assert context.bootstrap_git_type_found is True, ( # type: ignore[attr-defined]
f"Expected resource type '{name}' to exist in the registry after "
f"bootstrap, but it was not found. This is bug #524: "
f"bootstrap_builtin_types() is never called during initialization."
)
@then('the bootstrap-git resource type "{name}" should have kind "{kind}"')
def step_type_kind(context: Context, name: str, kind: str) -> None:
"""Assert the resource type has the expected kind."""
spec = context.bootstrap_git_type_spec # type: ignore[attr-defined]
actual = spec.resource_kind.value
assert actual == kind, f"Expected kind '{kind}', got '{actual}'"
@then(
'the bootstrap-git resource type "{name}" should have sandbox_strategy "{strategy}"'
)
def step_type_sandbox(context: Context, name: str, strategy: str) -> None:
"""Assert the resource type has the expected sandbox strategy."""
spec = context.bootstrap_git_type_spec # type: ignore[attr-defined]
actual = spec.sandbox_strategy.value
assert actual == strategy, f"Expected sandbox_strategy '{strategy}', got '{actual}'"
@then('the bootstrap-git resource type "{name}" should be user_addable')
def step_type_user_addable(context: Context, name: str) -> None:
"""Assert the resource type is user-addable."""
spec = context.bootstrap_git_type_spec # type: ignore[attr-defined]
assert spec.user_addable is True, (
f"Expected user_addable=True, got {spec.user_addable}"
)
@then('the bootstrap-git registered types should include "{name}"')
def step_registered_includes(context: Context, name: str) -> None:
"""Assert the bootstrap return value includes the named type."""
registered = context.bootstrap_git_registered # type: ignore[attr-defined]
assert name in registered, (
f"Expected '{name}' in bootstrap_builtin_types() return value, "
f"got: {registered}"
)
@then("the bootstrap-git resource add command should succeed")
def step_resource_add_should_succeed(context: Context) -> None:
"""Assert the resource add command did not fail."""
assert context.bootstrap_git_failed is not True, ( # type: ignore[attr-defined]
f"Expected 'resource add' to succeed, but it failed. "
f"Output: {context.bootstrap_git_output!r}. "
f"This is bug #524: bootstrap_builtin_types() is never called."
)
@then('the bootstrap-git resource add output should not contain "{text}"')
def step_output_should_not_contain(context: Context, text: str) -> None:
"""Assert the output does not contain the given text."""
assert text not in context.bootstrap_git_output, ( # type: ignore[attr-defined]
f"Output unexpectedly contained '{text}'. "
f"Full output: {context.bootstrap_git_output!r}"
)
+67
View File
@@ -0,0 +1,67 @@
*** Settings ***
Documentation Regression tests for built-in git-checkout type bootstrap (bug #524).
... These tests verify that bootstrap_builtin_types() correctly seeds
... git-checkout into the registry and that resource registration succeeds
... after bootstrap. Both tests call bootstrap explicitly and should pass.
Library Process
Library OperatingSystem
Resource ${CURDIR}/common.resource
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
*** Test Cases ***
Resource Add Git Checkout Should Not Fail With Type Not Found
[Documentation] Verify that after bootstrap, registering a git-checkout resource
... instance succeeds (no "Resource type not found" error).
${script}= Catenate SEPARATOR=\n
Outdated
Review

Note: This test case runs agents resource add git-checkout via subprocess against the real CLI. Unlike the second test case (which uses an in-memory SQLite script), this depends on the system having a configured/initialized environment. If CI doesn't have agents init run first, the subprocess may fail for a different reason (missing database) than the intended one (missing type).

The second test case's approach (in-memory script) is more hermetic and self-contained. Not blocking since this is expected to fail anyway in the TDD phase, but when the fix is applied the CI environment setup will need to ensure agents init runs before this Robot suite.

Note: This test case runs `agents resource add git-checkout` via subprocess against the real CLI. Unlike the second test case (which uses an in-memory SQLite script), this depends on the system having a configured/initialized environment. If CI doesn't have `agents init` run first, the subprocess may fail for a different reason (missing database) than the intended one (missing type). The second test case's approach (in-memory script) is more hermetic and self-contained. Not blocking since this is expected to fail anyway in the TDD phase, but when the fix is applied the CI environment setup will need to ensure `agents init` runs before this Robot suite.
... from sqlalchemy import create_engine
... from sqlalchemy.orm import sessionmaker
... from cleveragents.infrastructure.database.models import Base
... from cleveragents.application.services.resource_registry_service import ResourceRegistryService
... engine = create_engine("sqlite:///:memory:", echo=False)
... Base.metadata.create_all(engine)
... factory = sessionmaker(bind=engine, expire_on_commit=False)
... service = ResourceRegistryService(session_factory=factory)
... service.bootstrap_builtin_types()
... resource = service.register_resource(
... ${SPACE}${SPACE}${SPACE}${SPACE}type_name="git-checkout",
... ${SPACE}${SPACE}${SPACE}${SPACE}name="local/test",
... ${SPACE}${SPACE}${SPACE}${SPACE}location="/tmp/repo",
... ${SPACE}${SPACE}${SPACE}${SPACE}description="Test git-checkout resource",
... ${SPACE}${SPACE}${SPACE}${SPACE}properties={"branch": "main"},
... )
... assert resource.resource_type_name == "git-checkout", f"type: {resource.resource_type_name}"
... assert resource.name == "local/test", f"name: {resource.name}"
... print("resource add git-checkout succeeded after bootstrap")
${result}= Run Process ${PYTHON} -c ${script} timeout=60s
Should Be Equal As Integers ${result.rc} 0
... msg=resource add git-checkout failed: ${result.stderr}
Should Contain ${result.stdout} resource add git-checkout succeeded after bootstrap
Git Checkout Type Exists After Bootstrap
[Documentation] Directly call bootstrap_builtin_types() and verify git-checkout
... is present in the type registry.
${script}= Catenate SEPARATOR=\n
... from sqlalchemy import create_engine
... from sqlalchemy.orm import sessionmaker
... from cleveragents.infrastructure.database.models import Base
... from cleveragents.application.services.resource_registry_service import ResourceRegistryService
... engine = create_engine("sqlite:///:memory:", echo=False)
... Base.metadata.create_all(engine)
... factory = sessionmaker(bind=engine, expire_on_commit=False)
... service = ResourceRegistryService(session_factory=factory)
... registered = service.bootstrap_builtin_types()
... assert "git-checkout" in registered, f"git-checkout not in registered: {registered}"
... spec = service.show_type("git-checkout")
... assert spec.name == "git-checkout", f"name mismatch: {spec.name}"
... rk = spec.resource_kind.value if hasattr(spec.resource_kind, "value") else str(spec.resource_kind)
... assert rk == "physical", f"kind mismatch: {rk}"
... ss = spec.sandbox_strategy.value if hasattr(spec.sandbox_strategy, "value") else str(spec.sandbox_strategy)
... assert ss == "git_worktree", f"sandbox mismatch: {ss}"
... assert spec.user_addable is True, f"user_addable: {spec.user_addable}"
... print("git-checkout bootstrap validated successfully")
${result}= Run Process ${PYTHON} -c ${script} timeout=60s
Should Be Equal As Integers ${result.rc} 0
... msg=Bootstrap validation failed: ${result.stderr}
Should Contain ${result.stdout} git-checkout bootstrap validated successfully