test(resource): add failing tests for built-in git-checkout type bootstrap (#524) #568
@@ -2,6 +2,11 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Added TDD-style Behave BDD tests for the built-in `git-checkout` resource type
|
||||
bootstrap. Three scenarios: one failing TDD test reproducing bug #524 (no bootstrap
|
||||
called during init), and two regression tests verifying `bootstrap_builtin_types()`
|
||||
seeds correct data and `agents resource add git-checkout` succeeds. Includes Robot
|
||||
Framework regression tests. (#553)
|
||||
- Added TDD-style Behave BDD tests for the built-in `fs-directory` resource type
|
||||
bootstrap. Three scenarios: one failing TDD test reproducing bug #523 (no bootstrap
|
||||
called during init), and two regression tests verifying `bootstrap_builtin_types()`
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
# These tests target bug #524 and are expected to fail until the fix is applied.
|
||||
# bootstrap_builtin_types() is never called during initialization, so the
|
||||
# git-checkout built-in resource type is missing from the registry at runtime.
|
||||
Feature: Built-in git-checkout type bootstrap on initialization
|
||||
As a CleverAgents user
|
||||
I want the built-in git-checkout resource type to be available after initialization
|
||||
So that I can run "agents resource add git-checkout" without "Resource type not found"
|
||||
|
||||
# ── Bug reproduction: init path does NOT call bootstrap ────
|
||||
# This scenario is the TDD failing test. It creates a registry WITHOUT
|
||||
# calling bootstrap_builtin_types() and asserts git-checkout exists.
|
||||
|
|
||||
# It will fail until the fix integrates bootstrap into the init path.
|
||||
#
|
||||
# NOTE FOR FIX AUTHOR: If the chosen fix is to call bootstrap inside
|
||||
# init_command() / initialize_project() rather than in
|
||||
# ResourceRegistryService.__init__(), you will need to update the Given
|
||||
# step to exercise the init path instead of constructing a bare service.
|
||||
|
||||
@tdd @bug524 @wip
|
||||
Scenario: git-checkout type is missing when bootstrap is not called during init
|
||||
Given a bootstrap-git fresh in-memory resource registry without bootstrap
|
||||
When I query the bootstrap-git resource type registry for "git-checkout"
|
||||
Then the bootstrap-git resource type "git-checkout" should exist
|
||||
|
||||
# ── Regression: bootstrap function itself works correctly ──
|
||||
|
||||
@tdd @bug524
|
||||
Scenario: After initialization the git-checkout type exists in the resource type registry
|
||||
Given a bootstrap-git fresh in-memory resource registry with bootstrap
|
||||
When I query the bootstrap-git resource type registry for "git-checkout"
|
||||
Then the bootstrap-git resource type "git-checkout" should exist
|
||||
And the bootstrap-git resource type "git-checkout" should have kind "physical"
|
||||
And the bootstrap-git resource type "git-checkout" should have sandbox_strategy "git_worktree"
|
||||
And the bootstrap-git resource type "git-checkout" should be user_addable
|
||||
And the bootstrap-git registered types should include "git-checkout"
|
||||
|
||||
# ── CLI resource add succeeds ──────────────────────────────
|
||||
|
||||
@tdd @bug524
|
||||
Scenario: agents resource add git-checkout succeeds without Resource type not found error
|
||||
Given a bootstrap-git fresh in-memory resource registry with bootstrap
|
||||
When I run bootstrap-git resource add for type "git-checkout" named "local/test" with path "/tmp/repo" and branch "main"
|
||||
Then the bootstrap-git resource add command should succeed
|
||||
And the bootstrap-git resource add output should not contain "Resource type not found"
|
||||
@@ -0,0 +1,211 @@
|
||||||||||||||||
"""Step definitions for resource_type_bootstrap_git.feature.
|
||||||||||||||||
|
||||||||||||||||
These tests target bug #524. The first scenario (without bootstrap) is a
|
||||||||||||||||
genuine TDD failing test: it creates a fresh registry *without* calling
|
||||||||||||||||
``bootstrap_builtin_types()`` and asserts ``git-checkout`` exists — which
|
||||||||||||||||
fails until the init path is fixed to call bootstrap.
|
||||||||||||||||
|
||||||||||||||||
The remaining scenarios verify that ``bootstrap_builtin_types()`` itself
|
||||||||||||||||
seeds the correct data and that the CLI ``resource add`` path works once
|
||||||||||||||||
the types are present.
|
||||||||||||||||
"""
|
||||||||||||||||
|
||||||||||||||||
from __future__ import annotations
|
||||||||||||||||
|
||||||||||||||||
from io import StringIO
|
||||||||||||||||
from typing import Any
|
||||||||||||||||
from unittest.mock import patch
|
||||||||||||||||
|
||||||||||||||||
from behave import given, then, when # type: ignore[import-untyped]
|
||||||||||||||||
from behave.runner import Context # type: ignore[import-untyped]
|
||||||||||||||||
from rich.console import Console
|
||||||||||||||||
from sqlalchemy import create_engine
|
||||||||||||||||
from sqlalchemy.orm import sessionmaker
|
||||||||||||||||
|
brent.edwards
commented
Nit: Nit: `runner` is created as a module-level singleton, while PR #566's step file creates a new `CliRunner()` per step invocation. Both approaches are correct (CliRunner is stateless), just noting the style difference across the companion PRs.
|
||||||||||||||||
|
||||||||||||||||
from cleveragents.application.services.resource_registry_service import (
|
||||||||||||||||
ResourceRegistryService,
|
||||||||||||||||
)
|
||||||||||||||||
from cleveragents.core.exceptions import NotFoundError
|
||||||||||||||||
from cleveragents.infrastructure.database.models import Base
|
||||||||||||||||
|
||||||||||||||||
# ── Helpers ──────────────────────────────────────────────────
|
||||||||||||||||
|
||||||||||||||||
_PATCH_SERVICE = "cleveragents.cli.commands.resource._get_registry_service"
|
||||||||||||||||
_PATCH_CONSOLE = "cleveragents.cli.commands.resource.console"
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
def _make_service(context: Context, *, run_bootstrap: bool) -> ResourceRegistryService:
|
||||||||||||||||
"""Create an in-memory ResourceRegistryService.
|
||||||||||||||||
|
||||||||||||||||
When *run_bootstrap* is ``True`` the built-in types are seeded; when
|
||||||||||||||||
``False`` the registry is left empty (reproducing the bug).
|
||||||||||||||||
"""
|
||||||||||||||||
engine = create_engine("sqlite:///:memory:", echo=False)
|
||||||||||||||||
Base.metadata.create_all(engine)
|
||||||||||||||||
factory = sessionmaker(bind=engine, expire_on_commit=False)
|
||||||||||||||||
service = ResourceRegistryService(session_factory=factory)
|
||||||||||||||||
if run_bootstrap:
|
||||||||||||||||
context.bootstrap_git_registered = service.bootstrap_builtin_types() # type: ignore[attr-defined]
|
||||||||||||||||
context.bootstrap_git_service = service # type: ignore[attr-defined]
|
||||||||||||||||
return service
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
def _capture_output(func: Any, *args: Any, **kwargs: Any) -> tuple[str, bool]:
|
||||||||||||||||
"""Run a CLI function capturing its console output and success status."""
|
||||||||||||||||
buf = StringIO()
|
||||||||||||||||
console = Console(
|
||||||||||||||||
file=buf, width=200, no_color=True, highlight=False, force_terminal=False
|
||||||||||||||||
)
|
||||||||||||||||
|
||||||||||||||||
failed = False
|
||||||||||||||||
failure_reason = ""
|
||||||||||||||||
with patch(_PATCH_CONSOLE, console):
|
||||||||||||||||
try:
|
||||||||||||||||
func(*args, **kwargs)
|
||||||||||||||||
except SystemExit as exc:
|
||||||||||||||||
if exc.code not in (None, 0):
|
||||||||||||||||
failed = True
|
||||||||||||||||
failure_reason = f"SystemExit: {exc.code}"
|
||||||||||||||||
except Exception as exc:
|
||||||||||||||||
failed = True
|
||||||||||||||||
failure_reason = f"{type(exc).__name__}: {exc}"
|
||||||||||||||||
|
||||||||||||||||
output = buf.getvalue()
|
||||||||||||||||
if failure_reason:
|
||||||||||||||||
output = output + "\n" + failure_reason
|
||||||||||||||||
return output, failed
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
# ── Given steps ──────────────────────────────────────────────
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
@given("a bootstrap-git fresh in-memory resource registry with bootstrap")
|
||||||||||||||||
def step_fresh_registry_with_bootstrap(context: Context) -> None:
|
||||||||||||||||
"""Set up a fresh in-memory database and run bootstrap_builtin_types."""
|
||||||||||||||||
_make_service(context, run_bootstrap=True)
|
||||||||||||||||
context.bootstrap_git_output = "" # type: ignore[attr-defined]
|
||||||||||||||||
context.bootstrap_git_failed = False # type: ignore[attr-defined]
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
@given("a bootstrap-git fresh in-memory resource registry without bootstrap")
|
||||||||||||||||
def step_fresh_registry_without_bootstrap(context: Context) -> None:
|
||||||||||||||||
"""Set up a fresh in-memory database WITHOUT running bootstrap.
|
||||||||||||||||
|
||||||||||||||||
This reproduces bug #524: the registry is empty because
|
||||||||||||||||
``bootstrap_builtin_types()`` is never called during initialization.
|
||||||||||||||||
"""
|
||||||||||||||||
_make_service(context, run_bootstrap=False)
|
||||||||||||||||
context.bootstrap_git_output = "" # type: ignore[attr-defined]
|
||||||||||||||||
context.bootstrap_git_failed = False # type: ignore[attr-defined]
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
# ── When steps ───────────────────────────────────────────────
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
@when('I query the bootstrap-git resource type registry for "{name}"')
|
||||||||||||||||
|
brent.edwards
commented
Note: The step matcher text here differs from the companion PR #567 for the same domain concepts:
No runtime conflict since the matchers are distinct, and I'd argue this PR's patterns are better (parameterized by If there's an opportunity to align #567 to match this PR's patterns, it would make the test suite more uniform. Low priority — could be done in a follow-up. Note: The step matcher text here differs from the companion PR #567 for the same domain concepts:
| Concept | This PR (#568) | PR #567 |
|---|---|---|
| Type existence | `should be present in the registry` | `should exist` |
| Kind check | `"{name}" should have kind "{kind}"` | `kind should be "{kind}"` |
| Strategy check | `"{name}" should have sandbox_strategy` | `sandbox_strategy should be` |
No runtime conflict since the matchers are distinct, and I'd argue this PR's patterns are better (parameterized by `{name}`, so they generalize to any type). But when both merge, the step vocabulary will be inconsistent for the same domain.
If there's an opportunity to align #567 to match this PR's patterns, it would make the test suite more uniform. Low priority — could be done in a follow-up.
|
||||||||||||||||
def step_query_registry_for_type(context: Context, name: str) -> None:
|
||||||||||||||||
"""Query the registry for a specific resource type by name."""
|
||||||||||||||||
service: ResourceRegistryService = context.bootstrap_git_service # type: ignore[attr-defined]
|
||||||||||||||||
try:
|
||||||||||||||||
context.bootstrap_git_type_spec = service.show_type(name) # type: ignore[attr-defined]
|
||||||||||||||||
context.bootstrap_git_type_found = True # type: ignore[attr-defined]
|
||||||||||||||||
except NotFoundError:
|
||||||||||||||||
context.bootstrap_git_type_spec = None # type: ignore[attr-defined]
|
||||||||||||||||
context.bootstrap_git_type_found = False # type: ignore[attr-defined]
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
@when(
|
||||||||||||||||
'I run bootstrap-git resource add for type "{type_name}" named "{name}" with path "{path}" and branch "{branch}"'
|
||||||||||||||||
)
|
||||||||||||||||
def step_run_resource_add_git(
|
||||||||||||||||
context: Context, type_name: str, name: str, path: str, branch: str
|
||||||||||||||||
) -> None:
|
||||||||||||||||
"""Run resource add command via the CLI function."""
|
||||||||||||||||
from cleveragents.cli.commands.resource import resource_add
|
||||||||||||||||
|
||||||||||||||||
with patch(
|
||||||||||||||||
_PATCH_SERVICE,
|
||||||||||||||||
return_value=context.bootstrap_git_service, # type: ignore[attr-defined]
|
||||||||||||||||
):
|
||||||||||||||||
output, failed = _capture_output(
|
||||||||||||||||
resource_add,
|
||||||||||||||||
type_name=type_name,
|
||||||||||||||||
name=name,
|
||||||||||||||||
path=path,
|
||||||||||||||||
branch=branch,
|
||||||||||||||||
description=None,
|
||||||||||||||||
image=None,
|
||||||||||||||||
read_only=False,
|
||||||||||||||||
fmt="rich",
|
||||||||||||||||
)
|
||||||||||||||||
context.bootstrap_git_output = output # type: ignore[attr-defined]
|
||||||||||||||||
context.bootstrap_git_failed = failed # type: ignore[attr-defined]
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
# ── Then steps ───────────────────────────────────────────────
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
@then('the bootstrap-git resource type "{name}" should exist')
|
||||||||||||||||
def step_type_should_exist(context: Context, name: str) -> None:
|
||||||||||||||||
"""Assert the resource type was found in the registry."""
|
||||||||||||||||
assert context.bootstrap_git_type_found is True, ( # type: ignore[attr-defined]
|
||||||||||||||||
f"Expected resource type '{name}' to exist in the registry after "
|
||||||||||||||||
f"bootstrap, but it was not found. This is bug #524: "
|
||||||||||||||||
f"bootstrap_builtin_types() is never called during initialization."
|
||||||||||||||||
)
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
@then('the bootstrap-git resource type "{name}" should have kind "{kind}"')
|
||||||||||||||||
def step_type_kind(context: Context, name: str, kind: str) -> None:
|
||||||||||||||||
"""Assert the resource type has the expected kind."""
|
||||||||||||||||
spec = context.bootstrap_git_type_spec # type: ignore[attr-defined]
|
||||||||||||||||
actual = spec.resource_kind.value
|
||||||||||||||||
assert actual == kind, f"Expected kind '{kind}', got '{actual}'"
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
@then(
|
||||||||||||||||
'the bootstrap-git resource type "{name}" should have sandbox_strategy "{strategy}"'
|
||||||||||||||||
)
|
||||||||||||||||
def step_type_sandbox(context: Context, name: str, strategy: str) -> None:
|
||||||||||||||||
"""Assert the resource type has the expected sandbox strategy."""
|
||||||||||||||||
spec = context.bootstrap_git_type_spec # type: ignore[attr-defined]
|
||||||||||||||||
actual = spec.sandbox_strategy.value
|
||||||||||||||||
assert actual == strategy, f"Expected sandbox_strategy '{strategy}', got '{actual}'"
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
@then('the bootstrap-git resource type "{name}" should be user_addable')
|
||||||||||||||||
def step_type_user_addable(context: Context, name: str) -> None:
|
||||||||||||||||
"""Assert the resource type is user-addable."""
|
||||||||||||||||
spec = context.bootstrap_git_type_spec # type: ignore[attr-defined]
|
||||||||||||||||
assert spec.user_addable is True, (
|
||||||||||||||||
f"Expected user_addable=True, got {spec.user_addable}"
|
||||||||||||||||
)
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
@then('the bootstrap-git registered types should include "{name}"')
|
||||||||||||||||
def step_registered_includes(context: Context, name: str) -> None:
|
||||||||||||||||
"""Assert the bootstrap return value includes the named type."""
|
||||||||||||||||
registered = context.bootstrap_git_registered # type: ignore[attr-defined]
|
||||||||||||||||
assert name in registered, (
|
||||||||||||||||
f"Expected '{name}' in bootstrap_builtin_types() return value, "
|
||||||||||||||||
f"got: {registered}"
|
||||||||||||||||
)
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
@then("the bootstrap-git resource add command should succeed")
|
||||||||||||||||
def step_resource_add_should_succeed(context: Context) -> None:
|
||||||||||||||||
"""Assert the resource add command did not fail."""
|
||||||||||||||||
assert context.bootstrap_git_failed is not True, ( # type: ignore[attr-defined]
|
||||||||||||||||
f"Expected 'resource add' to succeed, but it failed. "
|
||||||||||||||||
f"Output: {context.bootstrap_git_output!r}. "
|
||||||||||||||||
f"This is bug #524: bootstrap_builtin_types() is never called."
|
||||||||||||||||
)
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
@then('the bootstrap-git resource add output should not contain "{text}"')
|
||||||||||||||||
def step_output_should_not_contain(context: Context, text: str) -> None:
|
||||||||||||||||
"""Assert the output does not contain the given text."""
|
||||||||||||||||
assert text not in context.bootstrap_git_output, ( # type: ignore[attr-defined]
|
||||||||||||||||
f"Output unexpectedly contained '{text}'. "
|
||||||||||||||||
f"Full output: {context.bootstrap_git_output!r}"
|
||||||||||||||||
)
|
||||||||||||||||
@@ -0,0 +1,67 @@
|
||||
*** Settings ***
|
||||
Documentation Regression tests for built-in git-checkout type bootstrap (bug #524).
|
||||
... These tests verify that bootstrap_builtin_types() correctly seeds
|
||||
... git-checkout into the registry and that resource registration succeeds
|
||||
... after bootstrap. Both tests call bootstrap explicitly and should pass.
|
||||
Library Process
|
||||
Library OperatingSystem
|
||||
Resource ${CURDIR}/common.resource
|
||||
|
||||
Suite Setup Setup Test Environment
|
||||
Suite Teardown Cleanup Test Environment
|
||||
|
||||
*** Test Cases ***
|
||||
Resource Add Git Checkout Should Not Fail With Type Not Found
|
||||
[Documentation] Verify that after bootstrap, registering a git-checkout resource
|
||||
... instance succeeds (no "Resource type not found" error).
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
|
brent.edwards
commented
Note: This test case runs The second test case's approach (in-memory script) is more hermetic and self-contained. Not blocking since this is expected to fail anyway in the TDD phase, but when the fix is applied the CI environment setup will need to ensure Note: This test case runs `agents resource add git-checkout` via subprocess against the real CLI. Unlike the second test case (which uses an in-memory SQLite script), this depends on the system having a configured/initialized environment. If CI doesn't have `agents init` run first, the subprocess may fail for a different reason (missing database) than the intended one (missing type).
The second test case's approach (in-memory script) is more hermetic and self-contained. Not blocking since this is expected to fail anyway in the TDD phase, but when the fix is applied the CI environment setup will need to ensure `agents init` runs before this Robot suite.
|
||||
... from sqlalchemy import create_engine
|
||||
... from sqlalchemy.orm import sessionmaker
|
||||
... from cleveragents.infrastructure.database.models import Base
|
||||
... from cleveragents.application.services.resource_registry_service import ResourceRegistryService
|
||||
... engine = create_engine("sqlite:///:memory:", echo=False)
|
||||
... Base.metadata.create_all(engine)
|
||||
... factory = sessionmaker(bind=engine, expire_on_commit=False)
|
||||
... service = ResourceRegistryService(session_factory=factory)
|
||||
... service.bootstrap_builtin_types()
|
||||
... resource = service.register_resource(
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}type_name="git-checkout",
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}name="local/test",
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}location="/tmp/repo",
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}description="Test git-checkout resource",
|
||||
... ${SPACE}${SPACE}${SPACE}${SPACE}properties={"branch": "main"},
|
||||
... )
|
||||
... assert resource.resource_type_name == "git-checkout", f"type: {resource.resource_type_name}"
|
||||
... assert resource.name == "local/test", f"name: {resource.name}"
|
||||
... print("resource add git-checkout succeeded after bootstrap")
|
||||
${result}= Run Process ${PYTHON} -c ${script} timeout=60s
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
... msg=resource add git-checkout failed: ${result.stderr}
|
||||
Should Contain ${result.stdout} resource add git-checkout succeeded after bootstrap
|
||||
|
||||
Git Checkout Type Exists After Bootstrap
|
||||
[Documentation] Directly call bootstrap_builtin_types() and verify git-checkout
|
||||
... is present in the type registry.
|
||||
${script}= Catenate SEPARATOR=\n
|
||||
... from sqlalchemy import create_engine
|
||||
... from sqlalchemy.orm import sessionmaker
|
||||
... from cleveragents.infrastructure.database.models import Base
|
||||
... from cleveragents.application.services.resource_registry_service import ResourceRegistryService
|
||||
... engine = create_engine("sqlite:///:memory:", echo=False)
|
||||
... Base.metadata.create_all(engine)
|
||||
... factory = sessionmaker(bind=engine, expire_on_commit=False)
|
||||
... service = ResourceRegistryService(session_factory=factory)
|
||||
... registered = service.bootstrap_builtin_types()
|
||||
... assert "git-checkout" in registered, f"git-checkout not in registered: {registered}"
|
||||
... spec = service.show_type("git-checkout")
|
||||
... assert spec.name == "git-checkout", f"name mismatch: {spec.name}"
|
||||
... rk = spec.resource_kind.value if hasattr(spec.resource_kind, "value") else str(spec.resource_kind)
|
||||
... assert rk == "physical", f"kind mismatch: {rk}"
|
||||
... ss = spec.sandbox_strategy.value if hasattr(spec.sandbox_strategy, "value") else str(spec.sandbox_strategy)
|
||||
... assert ss == "git_worktree", f"sandbox mismatch: {ss}"
|
||||
... assert spec.user_addable is True, f"user_addable: {spec.user_addable}"
|
||||
... print("git-checkout bootstrap validated successfully")
|
||||
${result}= Run Process ${PYTHON} -c ${script} timeout=60s
|
||||
Should Be Equal As Integers ${result.rc} 0
|
||||
... msg=Bootstrap validation failed: ${result.stderr}
|
||||
Should Contain ${result.stdout} git-checkout bootstrap validated successfully
|
||||
Minor: The scenarios in PR #566 use
@tdd @bug522tags, which allow selective execution (e.g.,behave --tags=@tddto run only TDD tests, or--tags=~@tddto skip them). This PR and the companion PR #567 don't have scenario-level tags.Consider adding
@tdd @bug524for consistency: