26632f79e9
CI / build (push) Successful in 18s
CI / helm (push) Successful in 21s
CI / lint (push) Successful in 3m17s
CI / quality (push) Successful in 3m41s
CI / typecheck (push) Successful in 4m3s
CI / security (push) Successful in 4m4s
CI / benchmark-regression (push) Has been skipped
CI / integration_tests (push) Successful in 7m7s
CI / unit_tests (push) Successful in 7m15s
CI / docker (push) Successful in 1m31s
CI / e2e_tests (push) Successful in 12m39s
CI / coverage (push) Successful in 11m32s
CI / status-check (push) Successful in 1s
CI / benchmark-publish (push) Successful in 29m36s
## Summary This PR adds TDD bug-capture tests for bug #1079 (`project context set` missing `--execution-env-priority` flag). Per the Bug Fix Workflow in CONTRIBUTING.md, the first step in fixing any bug is to write a test that proves the bug exists. ### What was done - **Behave unit tests** (6 scenarios in `features/project_context_set_exec_env_priority.feature`): - `project context set --execution-env-priority override` with `--execution-environment` should succeed and persist - `project context set --execution-env-priority fallback` with `--execution-environment` should succeed and persist - `--execution-env-priority` without `--execution-environment` should be rejected - Default to `fallback` when only `--execution-environment` is specified - Invalid `--execution-env-priority` value should be rejected - `project context show` should reflect persisted `execution_env_priority` in JSON output - **Robot integration tests** (3 test cases in `robot/project_context_set_exec_env_priority.robot`): - Override acceptance with persistence verification - Fallback acceptance with persistence verification - Full round-trip persistence check All tests are tagged `@tdd_bug @tdd_bug_1079 @tdd_expected_fail`. The underlying assertions fail (confirming the bug exists — `--execution-env-priority` is "No such option"), and the `@tdd_expected_fail` tag inverts the result so CI passes. ### Bug confirmed The `context_set()` function in `cleveragents.cli.commands.project_context` does not accept `execution_env_priority` as a parameter. The specification (§Execution Environment Routing, precedence table) requires this flag at precedence level 2 for project-level execution environment priority control. ### Quality gates | Gate | Result | |------|--------| | `nox -s lint` | PASS | | `nox -s typecheck` | PASS (0 errors) | | `nox -s unit_tests` | PASS (12236 scenarios, 0 failed) | | `nox -s integration_tests` | My 3 tests pass (13 pre-existing failures) | | `nox -s coverage_report` | PASS (98%, threshold 97%) | Closes #1100 Reviewed-on: #1130 Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com> Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
189 lines
6.5 KiB
Python
189 lines
6.5 KiB
Python
"""Helper script for tdd_e2e_implicit_init.robot smoke tests.
|
|
|
|
Each subcommand exercises a real CLI command in a fresh environment
|
|
where ``agents init`` has NOT been run, to reproduce bug #1023.
|
|
|
|
Bug #1023 reports that ``CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true``
|
|
triggers schema migrations on an existing database but does NOT create
|
|
the database file or its parent directory structure. As a result, any
|
|
CLI command that touches the database fails with:
|
|
|
|
sqlite3.OperationalError: unable to open database file
|
|
|
|
The helper reports the **real** outcome: it exits 0 and prints the
|
|
sentinel when the operation succeeds (bug is fixed), and exits 1 when
|
|
the bug is still present. The ``tdd_expected_fail_listener`` on the
|
|
Robot side handles pass/fail inversion while the bug remains open.
|
|
|
|
Mock strategy
|
|
~~~~~~~~~~~~~
|
|
No mocks are needed — we exercise the real CLI via Typer's CliRunner
|
|
in a pristine temporary directory. We only manipulate environment
|
|
variables to create a fresh, uninitialised CLEVERAGENTS_HOME and set
|
|
``CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import shutil
|
|
import sys
|
|
import tempfile
|
|
from collections.abc import Callable
|
|
from pathlib import Path
|
|
|
|
# Ensure local source tree is importable
|
|
_SRC = str(Path(__file__).resolve().parents[1] / "src")
|
|
if _SRC not in sys.path:
|
|
sys.path.insert(0, _SRC)
|
|
|
|
from typer.testing import CliRunner # noqa: E402
|
|
|
|
from cleveragents.application.container import reset_container # noqa: E402
|
|
from cleveragents.cli.main import app # noqa: E402
|
|
from cleveragents.config.settings import Settings # noqa: E402
|
|
|
|
runner = CliRunner()
|
|
|
|
|
|
def _run_cli_without_init(args: list[str], sentinel: str) -> None:
|
|
"""Run a CLI command in a fresh environment without prior init.
|
|
|
|
Creates a pristine temp directory, sets CLEVERAGENTS_HOME to it,
|
|
enables CLEVERAGENTS_AUTO_APPLY_MIGRATIONS, and invokes the CLI.
|
|
Exits 0 with sentinel on success; exits 1 on failure.
|
|
|
|
Environment variables are always restored in the ``finally`` block
|
|
so subsequent tests are not polluted even if the CLI invocation
|
|
raises an unexpected exception.
|
|
|
|
Exceptions are caught by CliRunner (the default behaviour) so that
|
|
the result object always carries an ``exit_code`` and ``output``.
|
|
Using ``catch_exceptions=False`` would cause non-AssertionError
|
|
exceptions to propagate, preventing the ``tdd_expected_fail``
|
|
listener from inverting the result.
|
|
|
|
Parameters
|
|
----------
|
|
args:
|
|
CLI arguments to pass (e.g., ``["resource", "add", ...]``).
|
|
sentinel:
|
|
Sentinel string to print on success.
|
|
"""
|
|
tmpdir = tempfile.mkdtemp(prefix="tdd_implicit_init_robot_")
|
|
env_save: dict[str, str | None] = {}
|
|
try:
|
|
# Save and remove env vars that point to pre-migrated databases,
|
|
# template-DB fast-paths, or testing guards that auto-approve
|
|
# migration prompts. CLEVERAGENTS_TEMPLATE_DB is particularly
|
|
# important: when set, the Behave template-DB patch copies a
|
|
# pre-migrated database, silently bypassing the directory-
|
|
# creation bug this test is designed to capture.
|
|
for key in (
|
|
"CLEVERAGENTS_AUTO_APPLY_MIGRATIONS",
|
|
"CI",
|
|
"BEHAVE_TESTING",
|
|
"ROBOT_TESTING",
|
|
"CLEVERAGENTS_HOME",
|
|
"CLEVERAGENTS_DATABASE_URL",
|
|
"CLEVERAGENTS_TEST_DATABASE_URL",
|
|
"CLEVERAGENTS_TEMPLATE_DB",
|
|
"CLEVERAGENTS_TESTING_USE_MOCK_AI",
|
|
):
|
|
env_save[key] = os.environ.pop(key, None)
|
|
|
|
os.environ["CLEVERAGENTS_HOME"] = tmpdir
|
|
os.environ["CLEVERAGENTS_AUTO_APPLY_MIGRATIONS"] = "true"
|
|
db_path = Path(tmpdir) / "cleveragents.db"
|
|
test_db_path = Path(tmpdir) / "cleveragents_test.db"
|
|
os.environ["CLEVERAGENTS_DATABASE_URL"] = f"sqlite:///{db_path}"
|
|
os.environ["CLEVERAGENTS_TEST_DATABASE_URL"] = f"sqlite:///{test_db_path}"
|
|
|
|
# Reset DI container so it picks up our env overrides.
|
|
Settings.reset()
|
|
reset_container()
|
|
|
|
result = runner.invoke(app, args)
|
|
|
|
if result.exit_code != 0:
|
|
print(
|
|
f"CLI command failed with exit code {result.exit_code}",
|
|
file=sys.stderr,
|
|
)
|
|
print(f"Output: {result.output}", file=sys.stderr)
|
|
sys.exit(1)
|
|
|
|
# Bug fixed — command succeeded without explicit init.
|
|
print(sentinel)
|
|
finally:
|
|
# Always restore env vars and clean up the temp directory,
|
|
# even if runner.invoke() or assertions raise.
|
|
for key, val in env_save.items():
|
|
if val is not None:
|
|
os.environ[key] = val
|
|
else:
|
|
os.environ.pop(key, None)
|
|
Settings.reset()
|
|
reset_container()
|
|
shutil.rmtree(tmpdir, ignore_errors=True)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Subcommands
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def resource_add_no_init() -> None:
|
|
"""Invoke ``resource add`` without prior ``agents init``.
|
|
|
|
Exercises the ``resource add git-checkout`` command in a fresh
|
|
environment. If the database directory structure is not
|
|
auto-created, this will fail with OperationalError.
|
|
"""
|
|
_run_cli_without_init(
|
|
[
|
|
"resource",
|
|
"add",
|
|
"git-checkout",
|
|
"local/tdd-test-resource",
|
|
"--path",
|
|
".",
|
|
"--branch",
|
|
"main",
|
|
],
|
|
"tdd-resource-add-no-init-ok",
|
|
)
|
|
|
|
|
|
def project_create_no_init() -> None:
|
|
"""Invoke ``project create`` without prior ``agents init``.
|
|
|
|
Exercises the ``project create`` command in a fresh environment.
|
|
If the database directory structure is not auto-created, this
|
|
will fail with OperationalError.
|
|
"""
|
|
_run_cli_without_init(
|
|
["project", "create", "local/tdd-test-project"],
|
|
"tdd-project-create-no-init-ok",
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Dispatcher
|
|
# ---------------------------------------------------------------------------
|
|
|
|
_COMMANDS: dict[str, Callable[[], None]] = {
|
|
"resource-add-no-init": resource_add_no_init,
|
|
"project-create-no-init": project_create_no_init,
|
|
}
|
|
|
|
if __name__ == "__main__":
|
|
if len(sys.argv) < 2 or sys.argv[1] not in _COMMANDS:
|
|
print(
|
|
f"Usage: {sys.argv[0]} <{'|'.join(_COMMANDS)}>",
|
|
file=sys.stderr,
|
|
)
|
|
sys.exit(1)
|
|
cmd = _COMMANDS[sys.argv[1]]
|
|
cmd()
|