forked from HAL9000/cleveragents-core
efacfd61c3
## Summary Adds TDD bug-capture tests proving that CLI commands (`resource add`, `project create`) fail in a fresh environment without explicit `agents init`, even when `CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true` is set. Bug #1023 reports that `CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true` triggers schema migrations on an existing database but does NOT create the database file or its parent directory structure, causing `sqlite3.OperationalError: unable to open database file`. ## Changes ### Behave Tests - `features/tdd_e2e_implicit_init.feature` — Two scenarios tagged `@tdd_expected_fail @tdd_bug @tdd_bug_1023`: - `resource add` in a fresh environment without prior init - `project create` in a fresh environment without prior init - Both scenarios include output content assertions - `features/steps/tdd_e2e_implicit_init_steps.py` — Step definitions using Typer's CliRunner with real CLI invocation in a pristine temp directory. - Env isolation: saves/removes `CLEVERAGENTS_AUTO_APPLY_MIGRATIONS`, `CLEVERAGENTS_DATABASE_URL`, `CLEVERAGENTS_TEST_DATABASE_URL`, `CLEVERAGENTS_HOME`, `BEHAVE_TESTING`, `CLEVERAGENTS_TEMPLATE_DB`, `CLEVERAGENTS_TESTING_USE_MOCK_AI` - CliRunner invoked without `catch_exceptions=False` to ensure proper `@tdd_expected_fail` inversion ### Robot Tests - `robot/tdd_e2e_implicit_init.robot` — Two test cases tagged `tdd_bug tdd_bug_1023 tdd_expected_fail` (listener inverts results while bug is unfixed). - `robot/helper_tdd_e2e_implicit_init.py` — Helper script exercising CLI in a fresh environment. - Env isolation includes `ROBOT_TESTING`, `CLEVERAGENTS_TEMPLATE_DB`, `CLEVERAGENTS_TESTING_USE_MOCK_AI` - Env restoration in `finally` block ensures cleanup even on exceptions - CliRunner invoked without `catch_exceptions=False` for correct inversion ### Changelog - Updated `CHANGELOG.md` with entry for this TDD test addition. ## Quality Gates | Gate | Result | |------|--------| | `nox -s lint` | PASS | | `nox -s typecheck` | PASS (0 errors) | | `nox -s unit_tests` | PASS (462 features, 12232 scenarios, 0 failed) | | `nox -s integration_tests` | PASS (pre-existing failures only; our suite correct) | | `nox -s e2e_tests` | N/A (TDD test only, no E2E suite changes) | | `nox -s coverage_report` | PASS (98% coverage, threshold 97%) | Closes #1033 Reviewed-on: cleveragents/cleveragents-core#1113 Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com> Co-authored-by: Brent Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent Edwards <brent.edwards@cleverthis.com>
174 lines
6.7 KiB
Python
174 lines
6.7 KiB
Python
"""Step definitions for TDD Bug #1023 — implicit init requirement.
|
|
|
|
These steps exercise real CLI commands in a fresh environment where
|
|
``agents init`` has NOT been explicitly run. The environment variable
|
|
``CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true`` is set, which *should*
|
|
cause the database to be created implicitly.
|
|
|
|
Bug #1023 reports that ``CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true``
|
|
triggers schema migrations on an existing database but does NOT create
|
|
the database file or its parent directory structure. As a result, any
|
|
CLI command that touches the database (e.g., ``resource add``,
|
|
``project create``) fails with:
|
|
|
|
sqlite3.OperationalError: unable to open database file
|
|
|
|
The assertions here expect the commands to succeed (exit code 0),
|
|
proving implicit init works and preventing regressions.
|
|
|
|
Mock strategy
|
|
~~~~~~~~~~~~~
|
|
No mocks are needed — we exercise the real CLI via Typer's CliRunner
|
|
in a pristine temporary directory. We only manipulate environment
|
|
variables to create a fresh, uninitialised CLEVERAGENTS_HOME. We
|
|
remove ``CLEVERAGENTS_DATABASE_URL``, ``CLEVERAGENTS_TEST_DATABASE_URL``,
|
|
``BEHAVE_TESTING``, and ``CLEVERAGENTS_TEMPLATE_DB`` so the container
|
|
derives the DB path from ``CLEVERAGENTS_HOME``, migration prompts are
|
|
not auto-approved by the testing guard, and the template-DB fast-path
|
|
does not silently create the database for us.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import shutil
|
|
import tempfile
|
|
|
|
from behave import given, then, when
|
|
from behave.runner import Context
|
|
from typer.testing import CliRunner
|
|
|
|
from cleveragents.application.container import reset_container
|
|
from cleveragents.cli.main import app
|
|
|
|
runner = CliRunner()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Given steps
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given("a fresh isolated environment for tdd-implicit-init")
|
|
def step_fresh_environment(context: Context) -> None:
|
|
"""Set up a fresh temporary environment with no existing database.
|
|
|
|
Creates a temporary directory to serve as ``CLEVERAGENTS_HOME``.
|
|
Does NOT run ``agents init`` — the entire point of this TDD test is
|
|
to verify that CLI commands work without it.
|
|
|
|
Removes ``BEHAVE_TESTING`` so the migration runner's auto-approve
|
|
guard does not mask the real code path.
|
|
"""
|
|
tmpdir = tempfile.mkdtemp(prefix="tdd_implicit_init_")
|
|
context.tdd_implicit_init_tmpdir = tmpdir
|
|
|
|
# Save and remove env vars that point to pre-migrated databases,
|
|
# template-DB fast-paths, or testing guards that auto-approve
|
|
# migration prompts. CLEVERAGENTS_TEMPLATE_DB is particularly
|
|
# important: when set, the Behave template-DB patch copies a
|
|
# pre-migrated database, silently bypassing the directory-creation
|
|
# bug this test is designed to capture.
|
|
saved_env: dict[str, str | None] = {
|
|
k: os.environ.pop(k, None)
|
|
for k in (
|
|
"CLEVERAGENTS_AUTO_APPLY_MIGRATIONS",
|
|
"CLEVERAGENTS_DATABASE_URL",
|
|
"CLEVERAGENTS_TEST_DATABASE_URL",
|
|
"CLEVERAGENTS_HOME",
|
|
"BEHAVE_TESTING",
|
|
"CLEVERAGENTS_TEMPLATE_DB",
|
|
"CLEVERAGENTS_TESTING_USE_MOCK_AI",
|
|
)
|
|
}
|
|
context.tdd_implicit_init_saved_env = saved_env
|
|
|
|
# Point CLEVERAGENTS_HOME at our pristine temp directory.
|
|
os.environ["CLEVERAGENTS_HOME"] = tmpdir
|
|
|
|
# Force an isolated SQLite database path under this scenario's temp
|
|
# directory. This avoids leaking state from repository-level local
|
|
# database files created by prior test runs.
|
|
db_url = f"sqlite:///{tmpdir}/db/cleveragents.db"
|
|
os.environ["CLEVERAGENTS_DATABASE_URL"] = db_url
|
|
os.environ["CLEVERAGENTS_TEST_DATABASE_URL"] = db_url
|
|
|
|
# Reset the DI container so it picks up our env overrides.
|
|
reset_container()
|
|
|
|
def _cleanup() -> None:
|
|
for key, val in saved_env.items():
|
|
if val is not None:
|
|
os.environ[key] = val
|
|
else:
|
|
os.environ.pop(key, None)
|
|
reset_container()
|
|
shutil.rmtree(tmpdir, ignore_errors=True)
|
|
|
|
context.add_cleanup(_cleanup)
|
|
|
|
|
|
@given("CLEVERAGENTS_AUTO_APPLY_MIGRATIONS is set to true")
|
|
def step_set_auto_apply_migrations(context: Context) -> None:
|
|
"""Ensure CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true is in the environment.
|
|
|
|
This is the env var that should, in theory, trigger implicit
|
|
database creation and migration. Bug #1023 reports it does not
|
|
create the directory structure.
|
|
"""
|
|
os.environ["CLEVERAGENTS_AUTO_APPLY_MIGRATIONS"] = "true"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# When steps
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when('I run the CLI command "{command}" without prior init')
|
|
def step_run_cli_command_without_init(context: Context, command: str) -> None:
|
|
"""Invoke a CLI command via CliRunner without running ``agents init`` first.
|
|
|
|
The command string is split into tokens and passed to the main CLI
|
|
app. ``CLEVERAGENTS_HOME`` is set to the pristine temp directory
|
|
and ``CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true`` is in the
|
|
environment.
|
|
|
|
Exceptions are caught by CliRunner (the default behaviour) so that
|
|
the result object always carries an ``exit_code`` and ``output``.
|
|
This is essential for ``@tdd_expected_fail`` inversion — if
|
|
exceptions propagated as non-AssertionError, the inversion guard
|
|
in ``apply_tdd_inversion`` would skip inversion and the scenario
|
|
would be reported as a hard failure instead of an expected failure.
|
|
"""
|
|
# Reset container again to ensure a clean UoW derivation.
|
|
reset_container()
|
|
|
|
args = command.split()
|
|
context.tdd_implicit_init_result = runner.invoke(app, args)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Then steps
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@then("the tdd-implicit-init command should exit with code {code:d}")
|
|
def step_exit_code(context: Context, code: int) -> None:
|
|
"""Assert the CLI command exited with the expected return code."""
|
|
result = context.tdd_implicit_init_result
|
|
assert result is not None, "CLI command was not invoked"
|
|
actual = result.exit_code
|
|
assert actual == code, (
|
|
f"Expected exit code {code}, got {actual}.\nOutput:\n{result.output}"
|
|
)
|
|
|
|
|
|
@then('the tdd-implicit-init command output should contain "{text}"')
|
|
def step_output_contains(context: Context, text: str) -> None:
|
|
"""Assert the CLI command output contains the expected text."""
|
|
result = context.tdd_implicit_init_result
|
|
assert result is not None, "CLI command was not invoked"
|
|
assert text in result.output, (
|
|
f"Expected output to contain {text!r}.\nActual output:\n{result.output}"
|
|
)
|