Files
cleveragents-core/robot/helper_tdd_e2e_implicit_init.py
brent.edwards 26632f79e9
CI / build (push) Successful in 18s
CI / helm (push) Successful in 21s
CI / lint (push) Successful in 3m17s
CI / quality (push) Successful in 3m41s
CI / typecheck (push) Successful in 4m3s
CI / security (push) Successful in 4m4s
CI / benchmark-regression (push) Has been skipped
CI / integration_tests (push) Successful in 7m7s
CI / unit_tests (push) Successful in 7m15s
CI / docker (push) Successful in 1m31s
CI / e2e_tests (push) Successful in 12m39s
CI / coverage (push) Successful in 11m32s
CI / status-check (push) Successful in 1s
CI / benchmark-publish (push) Successful in 29m36s
test: add TDD bug-capture test for #1079 — project context set missing flag (#1130)
## Summary

This PR adds TDD bug-capture tests for bug #1079 (`project context set` missing `--execution-env-priority` flag). Per the Bug Fix Workflow in CONTRIBUTING.md, the first step in fixing any bug is to write a test that proves the bug exists.

### What was done

- **Behave unit tests** (6 scenarios in `features/project_context_set_exec_env_priority.feature`):
  - `project context set --execution-env-priority override` with `--execution-environment` should succeed and persist
  - `project context set --execution-env-priority fallback` with `--execution-environment` should succeed and persist
  - `--execution-env-priority` without `--execution-environment` should be rejected
  - Default to `fallback` when only `--execution-environment` is specified
  - Invalid `--execution-env-priority` value should be rejected
  - `project context show` should reflect persisted `execution_env_priority` in JSON output

- **Robot integration tests** (3 test cases in `robot/project_context_set_exec_env_priority.robot`):
  - Override acceptance with persistence verification
  - Fallback acceptance with persistence verification
  - Full round-trip persistence check

All tests are tagged `@tdd_bug @tdd_bug_1079 @tdd_expected_fail`. The underlying assertions fail (confirming the bug exists — `--execution-env-priority` is "No such option"), and the `@tdd_expected_fail` tag inverts the result so CI passes.

### Bug confirmed

The `context_set()` function in `cleveragents.cli.commands.project_context` does not accept `execution_env_priority` as a parameter. The specification (§Execution Environment Routing, precedence table) requires this flag at precedence level 2 for project-level execution environment priority control.

### Quality gates

| Gate | Result |
|------|--------|
| `nox -s lint` | PASS |
| `nox -s typecheck` | PASS (0 errors) |
| `nox -s unit_tests` | PASS (12236 scenarios, 0 failed) |
| `nox -s integration_tests` | My 3 tests pass (13 pre-existing failures) |
| `nox -s coverage_report` | PASS (98%, threshold 97%) |

Closes #1100

Reviewed-on: #1130
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
2026-03-28 02:14:55 +00:00

189 lines
6.5 KiB
Python

"""Helper script for tdd_e2e_implicit_init.robot smoke tests.
Each subcommand exercises a real CLI command in a fresh environment
where ``agents init`` has NOT been run, to reproduce bug #1023.
Bug #1023 reports that ``CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true``
triggers schema migrations on an existing database but does NOT create
the database file or its parent directory structure. As a result, any
CLI command that touches the database fails with:
sqlite3.OperationalError: unable to open database file
The helper reports the **real** outcome: it exits 0 and prints the
sentinel when the operation succeeds (bug is fixed), and exits 1 when
the bug is still present. The ``tdd_expected_fail_listener`` on the
Robot side handles pass/fail inversion while the bug remains open.
Mock strategy
~~~~~~~~~~~~~
No mocks are needed — we exercise the real CLI via Typer's CliRunner
in a pristine temporary directory. We only manipulate environment
variables to create a fresh, uninitialised CLEVERAGENTS_HOME and set
``CLEVERAGENTS_AUTO_APPLY_MIGRATIONS=true``.
"""
from __future__ import annotations
import os
import shutil
import sys
import tempfile
from collections.abc import Callable
from pathlib import Path
# Ensure local source tree is importable
_SRC = str(Path(__file__).resolve().parents[1] / "src")
if _SRC not in sys.path:
sys.path.insert(0, _SRC)
from typer.testing import CliRunner # noqa: E402
from cleveragents.application.container import reset_container # noqa: E402
from cleveragents.cli.main import app # noqa: E402
from cleveragents.config.settings import Settings # noqa: E402
runner = CliRunner()
def _run_cli_without_init(args: list[str], sentinel: str) -> None:
"""Run a CLI command in a fresh environment without prior init.
Creates a pristine temp directory, sets CLEVERAGENTS_HOME to it,
enables CLEVERAGENTS_AUTO_APPLY_MIGRATIONS, and invokes the CLI.
Exits 0 with sentinel on success; exits 1 on failure.
Environment variables are always restored in the ``finally`` block
so subsequent tests are not polluted even if the CLI invocation
raises an unexpected exception.
Exceptions are caught by CliRunner (the default behaviour) so that
the result object always carries an ``exit_code`` and ``output``.
Using ``catch_exceptions=False`` would cause non-AssertionError
exceptions to propagate, preventing the ``tdd_expected_fail``
listener from inverting the result.
Parameters
----------
args:
CLI arguments to pass (e.g., ``["resource", "add", ...]``).
sentinel:
Sentinel string to print on success.
"""
tmpdir = tempfile.mkdtemp(prefix="tdd_implicit_init_robot_")
env_save: dict[str, str | None] = {}
try:
# Save and remove env vars that point to pre-migrated databases,
# template-DB fast-paths, or testing guards that auto-approve
# migration prompts. CLEVERAGENTS_TEMPLATE_DB is particularly
# important: when set, the Behave template-DB patch copies a
# pre-migrated database, silently bypassing the directory-
# creation bug this test is designed to capture.
for key in (
"CLEVERAGENTS_AUTO_APPLY_MIGRATIONS",
"CI",
"BEHAVE_TESTING",
"ROBOT_TESTING",
"CLEVERAGENTS_HOME",
"CLEVERAGENTS_DATABASE_URL",
"CLEVERAGENTS_TEST_DATABASE_URL",
"CLEVERAGENTS_TEMPLATE_DB",
"CLEVERAGENTS_TESTING_USE_MOCK_AI",
):
env_save[key] = os.environ.pop(key, None)
os.environ["CLEVERAGENTS_HOME"] = tmpdir
os.environ["CLEVERAGENTS_AUTO_APPLY_MIGRATIONS"] = "true"
db_path = Path(tmpdir) / "cleveragents.db"
test_db_path = Path(tmpdir) / "cleveragents_test.db"
os.environ["CLEVERAGENTS_DATABASE_URL"] = f"sqlite:///{db_path}"
os.environ["CLEVERAGENTS_TEST_DATABASE_URL"] = f"sqlite:///{test_db_path}"
# Reset DI container so it picks up our env overrides.
Settings.reset()
reset_container()
result = runner.invoke(app, args)
if result.exit_code != 0:
print(
f"CLI command failed with exit code {result.exit_code}",
file=sys.stderr,
)
print(f"Output: {result.output}", file=sys.stderr)
sys.exit(1)
# Bug fixed — command succeeded without explicit init.
print(sentinel)
finally:
# Always restore env vars and clean up the temp directory,
# even if runner.invoke() or assertions raise.
for key, val in env_save.items():
if val is not None:
os.environ[key] = val
else:
os.environ.pop(key, None)
Settings.reset()
reset_container()
shutil.rmtree(tmpdir, ignore_errors=True)
# ---------------------------------------------------------------------------
# Subcommands
# ---------------------------------------------------------------------------
def resource_add_no_init() -> None:
"""Invoke ``resource add`` without prior ``agents init``.
Exercises the ``resource add git-checkout`` command in a fresh
environment. If the database directory structure is not
auto-created, this will fail with OperationalError.
"""
_run_cli_without_init(
[
"resource",
"add",
"git-checkout",
"local/tdd-test-resource",
"--path",
".",
"--branch",
"main",
],
"tdd-resource-add-no-init-ok",
)
def project_create_no_init() -> None:
"""Invoke ``project create`` without prior ``agents init``.
Exercises the ``project create`` command in a fresh environment.
If the database directory structure is not auto-created, this
will fail with OperationalError.
"""
_run_cli_without_init(
["project", "create", "local/tdd-test-project"],
"tdd-project-create-no-init-ok",
)
# ---------------------------------------------------------------------------
# Dispatcher
# ---------------------------------------------------------------------------
_COMMANDS: dict[str, Callable[[], None]] = {
"resource-add-no-init": resource_add_no_init,
"project-create-no-init": project_create_no_init,
}
if __name__ == "__main__":
if len(sys.argv) < 2 or sys.argv[1] not in _COMMANDS:
print(
f"Usage: {sys.argv[0]} <{'|'.join(_COMMANDS)}>",
file=sys.stderr,
)
sys.exit(1)
cmd = _COMMANDS[sys.argv[1]]
cmd()