cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

test: add TDD bug-capture test for #1079 — project context set missing flag

Browse Source
This commit is contained in:
Brent Edwards
2026-03-26 20:03:26 +00:00
parent b46a0e6102
commit 2589562595
5 changed files with 657 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+7
View File
@@ -22,6 +22,13 @@
`resource add` and `project create` succeed in a fresh environment without
prior init. Tests use `@tdd_expected_fail` until the bug fix is merged.
(#1033)
- Added TDD bug-capture tests for bug #1079 — `project context set` missing
`--execution-env-priority` flag. Six Behave BDD scenarios exercise the CLI
path (override, fallback, rejection without `--execution-environment`, default
value, invalid value, and round-trip via `context show`). Three Robot
Framework integration tests cover override acceptance, fallback acceptance,
and full persistence round-trip. Tests use `@tdd_expected_fail` until the
fix is merged. (#1100)
- Added TDD bug-capture tests for bug #1076 — `use_action()` does not
propagate `automation_profile` to Plan. Three Behave BDD scenarios
(`@tdd_bug @tdd_bug_1076 @tdd_expected_fail`) verify the full precedence
features/project_context_set_exec_env_priority.feature
+51
View File
@@ -0,0 +1,51 @@
Feature: Project context set --execution-env-priority flag (Bug #1079)
As a developer
I want to set execution environment priority when using project context set
So that I can control project-level execution environment precedence per spec §Execution Environment Routing
# This feature captures bug #1079: the --execution-env-priority flag is missing
# from the project context set command. The spec requires it at precedence level 2.
# These tests use @tdd_expected_fail because the flag is not yet implemented;
# the tag will be removed when the bug fix in #1079 is merged.
Background:
Given a bug 1079 in-memory database is initialized
And a project "local/bug1079-app" exists for bug 1079
@tdd_bug @tdd_bug_1079 @tdd_expected_fail
Scenario: Bug #1079 - project context set accepts --execution-env-priority override
When I run bug 1079 context set on "local/bug1079-app" with execution_environment "host" and execution_env_priority "override"
Then the bug 1079 command should succeed
And the stored bug 1079 execution_env_priority should be "override"
@tdd_bug @tdd_bug_1079 @tdd_expected_fail
Scenario: Bug #1079 - project context set accepts --execution-env-priority fallback
When I run bug 1079 context set on "local/bug1079-app" with execution_environment "host" and execution_env_priority "fallback"
Then the bug 1079 command should succeed
And the stored bug 1079 execution_env_priority should be "fallback"
@tdd_bug @tdd_bug_1079 @tdd_expected_fail
Scenario: Bug #1079 - project context set rejects --execution-env-priority without --execution-environment
When I run bug 1079 context set on "local/bug1079-app" with execution_env_priority "override" but no execution_environment
Then the bug 1079 command should fail
And the bug 1079 output should contain "--execution-env-priority requires --execution-environment"
@tdd_bug @tdd_bug_1079 @tdd_expected_fail
Scenario: Bug #1079 - project context set defaults execution-env-priority to fallback when not specified
When I run bug 1079 context set on "local/bug1079-app" with execution_environment "host" but no execution_env_priority
Then the bug 1079 command should succeed
And the stored bug 1079 execution_env_priority should be "fallback"
@tdd_bug @tdd_bug_1079 @tdd_expected_fail
Scenario: Bug #1079 - project context set rejects invalid --execution-env-priority value
When I run bug 1079 context set on "local/bug1079-app" with execution_environment "host" and execution_env_priority "invalid-value"
Then the bug 1079 command should fail
And the bug 1079 output should contain "Invalid execution env priority"
@tdd_bug @tdd_bug_1079 @tdd_expected_fail
Scenario: Bug #1079 - project context show reflects persisted execution-env-priority
When I run bug 1079 context set on "local/bug1079-app" with execution_environment "host" and execution_env_priority "override"
Then the bug 1079 command should succeed
When I run bug 1079 context show on "local/bug1079-app" with format "json"
Then the bug 1079 command should succeed
And the bug 1079 json output should include "execution_env_priority" as "override"
features/steps/project_context_set_exec_env_priority_steps.py
+289
View File
@@ -0,0 +1,289 @@
"""Step definitions for project_context_set_exec_env_priority.feature.
TDD bug-capture test for bug #1079: the ``--execution-env-priority`` flag
is missing from the ``project context set`` command.
These steps use @tdd_expected_fail because the flag is not yet implemented.
The underlying assertions will fail (proving the bug exists), and the
expected-fail tag inverts the result so CI passes. When the fix is merged
in #1079, the @tdd_expected_fail tag will be removed and the tests will
run normally.
See CONTRIBUTING.md > Bug Fix Workflow > TDD Bug Test Tags.
"""
from __future__ import annotations
import json as _json
from typing import Any
from unittest.mock import MagicMock, patch
from behave import given, then, when
from sqlalchemy import create_engine, text
from sqlalchemy.orm import Session, sessionmaker
from typer.testing import CliRunner
from cleveragents.cli.commands.project_context import app as project_context_app
# ------------------------------------------------------------------
# Session wrapper — prevents the production ``close()`` from
# destroying our shared in-memory session.
# ------------------------------------------------------------------
class _SafeSession:
"""Thin wrapper preventing close() from destroying shared in-memory session."""
def __init__(self, real: Session) -> None:
object.__setattr__(self, "_real", real)
def close(self) -> None:
"""Reset session state without closing the underlying connection."""
real: Session = object.__getattribute__(self, "_real")
real.rollback()
def __getattr__(self, name: str) -> Any:
return getattr(object.__getattribute__(self, "_real"), name)
def __setattr__(self, name: str, value: Any) -> None:
setattr(object.__getattribute__(self, "_real"), name, value)
def _make_session_factory(context: Any) -> tuple[Any, Any]:
from cleveragents.infrastructure.database.models import Base
engine = create_engine(
"sqlite:///:memory:",
echo=False,
connect_args={"check_same_thread": False},
)
Base.metadata.create_all(engine)
real = sessionmaker(
bind=engine, expire_on_commit=False, autoflush=True, autocommit=False
)()
wrapper = _SafeSession(real)
return engine, lambda: wrapper
# ------------------------------------------------------------------
# Mock container
# ------------------------------------------------------------------
def _mock_container(context: Any) -> MagicMock:
from cleveragents.application.services.context_tiers import ContextTierService
mc = MagicMock()
mc.namespaced_project_repo.return_value = context.bug1079_project_repo
mc.session_factory.return_value = context.bug1079_session_factory
if not hasattr(context, "bug1079_tier_service"):
context.bug1079_tier_service = ContextTierService()
mc.context_tier_service.return_value = context.bug1079_tier_service
return mc
def _invoke_set(context: Any, cli_args: list[str]) -> None:
"""Invoke ``project context set`` via the Typer CliRunner.
Uses the CLI interface so that unknown flags (like the missing
``--execution-env-priority``) produce a non-zero exit code and
error output rather than a TypeError. This ensures the assertion
failure is an AssertionError (``exit_code == 0`` fails), which the
``@tdd_expected_fail`` mechanism correctly inverts.
"""
mc = _mock_container(context)
runner = CliRunner()
with patch(
"cleveragents.application.container.get_container",
return_value=mc,
):
context.bug1079_result = runner.invoke(project_context_app, cli_args)
context.bug1079_exit_code = context.bug1079_result.exit_code
context.bug1079_output = context.bug1079_result.output
def _load_raw_blob(context: Any, ns: str) -> dict[str, Any] | None:
session = context.bug1079_session_factory()
row = session.execute(
text("SELECT context_policy_json FROM ns_projects WHERE namespaced_name = :ns"),
{"ns": ns},
).fetchone()
if row is None or row[0] is None:
return None
return _json.loads(row[0]) # type: ignore[no-any-return]
# ------------------------------------------------------------------
# Background
# ------------------------------------------------------------------
@given("a bug 1079 in-memory database is initialized")
def step_bug1079_init_db(context: Any) -> None:
from cleveragents.infrastructure.database.repositories import (
NamespacedProjectRepository,
)
engine, sf = _make_session_factory(context)
context.bug1079_engine = engine
context.bug1079_session_factory = sf
context.bug1079_project_repo = NamespacedProjectRepository(session_factory=sf)
context.bug1079_output = ""
context.bug1079_exit_code = 0
context.add_cleanup(engine.dispose)
@given('a project "{name}" exists for bug 1079')
def step_bug1079_create_project(context: Any, name: str) -> None:
from cleveragents.domain.models.core.project import (
NamespacedProject,
parse_namespaced_name,
)
parsed = parse_namespaced_name(name)
proj = NamespacedProject(name=parsed.name, namespace=parsed.namespace)
context.bug1079_project_repo.create(proj)
# ------------------------------------------------------------------
# When steps
# ------------------------------------------------------------------
@when(
'I run bug 1079 context set on "{project}" '
'with execution_environment "{env}" and execution_env_priority "{priority}"'
)
def step_bug1079_set_env_and_priority(
context: Any, project: str, env: str, priority: str
) -> None:
"""Run project context set with both --execution-environment and --execution-env-priority.
This exercises the code path that bug #1079 describes as missing.
The ``--execution-env-priority`` flag does not exist on ``context_set`` yet,
so the CLI runner will reject it with a "No such option" error — proving the bug.
"""
_invoke_set(
context,
[
"set",
project,
"--execution-environment",
env,
"--execution-env-priority",
priority,
],
)
@when(
'I run bug 1079 context set on "{project}" '
'with execution_env_priority "{priority}" but no execution_environment'
)
def step_bug1079_set_priority_no_env(context: Any, project: str, priority: str) -> None:
"""Run project context set with --execution-env-priority but no --execution-environment.
When the bug is fixed, the command should reject this combination with a
clear error message, mirroring the plan use validation behaviour.
"""
_invoke_set(
context,
[
"set",
project,
"--execution-env-priority",
priority,
],
)
@when(
'I run bug 1079 context set on "{project}" '
'with execution_environment "{env}" but no execution_env_priority'
)
def step_bug1079_set_env_no_priority(context: Any, project: str, env: str) -> None:
"""Run project context set with --execution-environment but no --execution-env-priority.
When the bug is fixed, the command should default the priority to "fallback".
"""
_invoke_set(
context,
[
"set",
project,
"--execution-environment",
env,
],
)
@when('I run bug 1079 context show on "{project}" with format "{fmt}"')
def step_bug1079_show(context: Any, project: str, fmt: str) -> None:
"""Run project context show to check persisted state."""
mc = _mock_container(context)
runner = CliRunner()
with patch(
"cleveragents.application.container.get_container",
return_value=mc,
):
context.bug1079_result = runner.invoke(
project_context_app,
["show", project, "--format", fmt],
)
context.bug1079_exit_code = context.bug1079_result.exit_code
context.bug1079_output = context.bug1079_result.output
# ------------------------------------------------------------------
# Then steps
# ------------------------------------------------------------------
@then("the bug 1079 command should succeed")
def step_bug1079_ok(context: Any) -> None:
assert context.bug1079_exit_code == 0, (
f"Expected exit 0, got {context.bug1079_exit_code}. "
f"Output: {context.bug1079_output}"
)
@then("the bug 1079 command should fail")
def step_bug1079_fail(context: Any) -> None:
assert context.bug1079_exit_code != 0, (
f"Expected non-zero exit code, got {context.bug1079_exit_code}. "
f"Output: {context.bug1079_output}"
)
@then('the stored bug 1079 execution_env_priority should be "{priority}"')
def step_bug1079_stored_priority(context: Any, priority: str) -> None:
"""Assert execution_env_priority was persisted in the project context JSON.
When the bug is fixed, the ``context_set`` command will persist this
value alongside the execution_environment. Currently the flag does
not exist, so the blob will either be None or lack the key — causing
this assertion to fail (as expected for a TDD bug-capture test).
"""
blob = _load_raw_blob(context, "local/bug1079-app")
assert blob is not None, "No policy blob stored — context set did not persist data"
stored = blob.get("execution_env_priority")
assert stored == priority, (
f"Expected execution_env_priority='{priority}', got {stored!r}. "
f"Full blob: {blob}"
)
@then('the bug 1079 output should contain "{text}"')
def step_bug1079_output_contains(context: Any, text: str) -> None:
assert text in context.bug1079_output, (
f"Expected output to contain '{text}'. Output: {context.bug1079_output!r}"
)
@then('the bug 1079 json output should include "{key}" as "{value}"')
def step_bug1079_json_includes(context: Any, key: str, value: str) -> None:
data = _json.loads(context.bug1079_output)
assert key in data, (
f"Expected JSON to contain key '{key}'. Keys: {list(data.keys())}"
)
assert data[key] == value, f"Expected {key}='{value}', got {data[key]!r}"
robot/helper_project_context_set_exec_env_priority.py
+272
View File
@@ -0,0 +1,272 @@
"""Helper script for project_context_set_exec_env_priority.robot smoke tests.
TDD bug-capture helper for bug #1079: the ``--execution-env-priority`` flag
is missing from the ``project context set`` command.
Each subcommand is a self-contained check that prints a sentinel on success.
These tests are tagged with [tdd_expected_fail] in the .robot file because
the underlying assertions fail (proving the bug exists).
See CONTRIBUTING.md > Bug Fix Workflow > TDD Bug Test Tags.
"""
from __future__ import annotations
import json as _json
import sys
from pathlib import Path
from typing import Any
from unittest.mock import MagicMock, patch
# Ensure the local source tree takes priority over any installed copy.
_SRC = str(Path(__file__).resolve().parents[1] / "src")
if _SRC not in sys.path:
sys.path.insert(0, _SRC)
from sqlalchemy import create_engine, text # noqa: E402
from sqlalchemy.orm import sessionmaker # noqa: E402
from typer.testing import CliRunner # noqa: E402
from cleveragents.application.services.context_tiers import ( # noqa: E402
ContextTierService,
)
from cleveragents.cli.commands.project_context import ( # noqa: E402
app as project_context_app,
)
from cleveragents.infrastructure.database.models import Base # noqa: E402
from cleveragents.infrastructure.database.repositories import ( # noqa: E402
NamespacedProjectRepository,
)
runner = CliRunner()
# ------------------------------------------------------------------
# In-memory DB setup
# ------------------------------------------------------------------
class _SafeSession:
"""Thin wrapper preventing close() from destroying shared in-memory session."""
def __init__(self, real: Any) -> None:
object.__setattr__(self, "_real", real)
def close(self) -> None:
real: Any = object.__getattribute__(self, "_real")
real.rollback()
def __getattr__(self, name: str) -> Any:
return getattr(object.__getattribute__(self, "_real"), name)
def __setattr__(self, name: str, value: Any) -> None:
setattr(object.__getattribute__(self, "_real"), name, value)
def _setup_db() -> tuple[Any, Any, NamespacedProjectRepository]:
"""Set up in-memory DB, create project, return (engine, session_factory, repo)."""
from cleveragents.domain.models.core.project import (
NamespacedProject,
parse_namespaced_name,
)
engine = create_engine(
"sqlite:///:memory:",
echo=False,
connect_args={"check_same_thread": False},
)
Base.metadata.create_all(engine)
real = sessionmaker(
bind=engine, expire_on_commit=False, autoflush=True, autocommit=False
)()
wrapper = _SafeSession(real)
sf = lambda: wrapper # noqa: E731
repo = NamespacedProjectRepository(session_factory=sf)
parsed = parse_namespaced_name("local/robot-bug1079")
proj = NamespacedProject(name=parsed.name, namespace=parsed.namespace)
repo.create(proj)
return engine, sf, repo
def _run_context_set(
sf: Any, repo: NamespacedProjectRepository, cli_args: list[str]
) -> tuple[int, str]:
"""Run ``project context set`` via CliRunner, return (exit_code, output).
Uses the CLI interface so that unknown flags (like the missing
``--execution-env-priority``) produce a non-zero exit code and
error output rather than a TypeError.
"""
mc = MagicMock()
mc.namespaced_project_repo.return_value = repo
mc.session_factory.return_value = sf
mc.context_tier_service.return_value = ContextTierService()
with patch(
"cleveragents.application.container.get_container",
return_value=mc,
):
result = runner.invoke(project_context_app, cli_args)
return result.exit_code, result.output
def _load_raw_blob(sf: Any, ns: str) -> dict[str, Any] | None:
session = sf()
row = session.execute(
text("SELECT context_policy_json FROM ns_projects WHERE namespaced_name = :ns"),
{"ns": ns},
).fetchone()
if row is None or row[0] is None:
return None
return _json.loads(row[0])
# ------------------------------------------------------------------
# Subcommands
# ------------------------------------------------------------------
def set_env_priority_override() -> None:
"""Verify project context set accepts --execution-env-priority override."""
engine, sf, repo = _setup_db()
try:
exit_code, output = _run_context_set(
sf,
repo,
[
"set",
"local/robot-bug1079",
"--execution-environment",
"host",
"--execution-env-priority",
"override",
],
)
if exit_code == 0:
blob = _load_raw_blob(sf, "local/robot-bug1079")
if blob and blob.get("execution_env_priority") == "override":
print("project-context-set-env-priority-override-ok")
else:
print(
f"FAIL: blob={blob!r}",
file=sys.stderr,
)
sys.exit(1)
else:
print(
f"FAIL: exit code {exit_code}, output: {output}",
file=sys.stderr,
)
sys.exit(1)
finally:
engine.dispose()
def set_env_priority_fallback() -> None:
"""Verify project context set accepts --execution-env-priority fallback."""
engine, sf, repo = _setup_db()
try:
exit_code, output = _run_context_set(
sf,
repo,
[
"set",
"local/robot-bug1079",
"--execution-environment",
"host",
"--execution-env-priority",
"fallback",
],
)
if exit_code == 0:
blob = _load_raw_blob(sf, "local/robot-bug1079")
if blob and blob.get("execution_env_priority") == "fallback":
print("project-context-set-env-priority-fallback-ok")
else:
print(
f"FAIL: blob={blob!r}",
file=sys.stderr,
)
sys.exit(1)
else:
print(
f"FAIL: exit code {exit_code}, output: {output}",
file=sys.stderr,
)
sys.exit(1)
finally:
engine.dispose()
def set_env_priority_persists() -> None:
"""Verify execution_env_priority is persisted and retrievable."""
engine, sf, repo = _setup_db()
try:
exit_code, output = _run_context_set(
sf,
repo,
[
"set",
"local/robot-bug1079",
"--execution-environment",
"host",
"--execution-env-priority",
"override",
],
)
if exit_code != 0:
print(
f"FAIL: set exit code {exit_code}, output: {output}",
file=sys.stderr,
)
sys.exit(1)
blob = _load_raw_blob(sf, "local/robot-bug1079")
if blob is None:
print("FAIL: no blob stored", file=sys.stderr)
sys.exit(1)
got = blob.get("execution_env_priority")
if got != "override":
print(
f"FAIL: expected 'override', got {got!r}",
file=sys.stderr,
)
sys.exit(1)
if blob.get("execution_environment") != "host":
print(
f"FAIL: expected 'host', got {blob.get('execution_environment')!r}",
file=sys.stderr,
)
sys.exit(1)
print("project-context-set-env-priority-persists-ok")
finally:
engine.dispose()
# ------------------------------------------------------------------
# Main dispatcher
# ------------------------------------------------------------------
_COMMANDS = {
"set-env-priority-override": set_env_priority_override,
"set-env-priority-fallback": set_env_priority_fallback,
"set-env-priority-persists": set_env_priority_persists,
}
def main() -> None:
if len(sys.argv) < 2 or sys.argv[1] not in _COMMANDS:
print(
f"Usage: {sys.argv[0]} <{'|'.join(_COMMANDS)}>",
file=sys.stderr,
)
sys.exit(2)
_COMMANDS[sys.argv[1]]()
if __name__ == "__main__":
main()
robot/project_context_set_exec_env_priority.robot
+38
View File
@@ -0,0 +1,38 @@
*** Settings ***
Documentation Smoke tests for project context set --execution-env-priority flag (Bug #1079)
...
... This test captures bug #1079: the ``--execution-env-priority`` flag
... is missing from the ``project context set`` command. Tagged as
... [tdd_bug] [tdd_bug_1079] [tdd_expected_fail] so that the test
... passes CI while the bug is unfixed. See CONTRIBUTING.md > Bug Fix Workflow.
Resource ${CURDIR}/common.resource
Suite Setup Setup Test Environment
Suite Teardown Cleanup Test Environment
Force Tags tdd_bug tdd_bug_1079 tdd_expected_fail
*** Variables ***
${HELPER} ${CURDIR}/helper_project_context_set_exec_env_priority.py
*** Test Cases ***
Project Context Set Accepts Execution Env Priority Override
[Documentation] Verify that ``project context set --execution-env-priority override``
... works with ``--execution-environment``. Currently fails due to bug #1079.
${result}= Run Process ${PYTHON} ${HELPER} set-env-priority-override cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} project-context-set-env-priority-override-ok
Project Context Set Accepts Execution Env Priority Fallback
[Documentation] Verify that ``project context set --execution-env-priority fallback``
... works with ``--execution-environment``. Currently fails due to bug #1079.
${result}= Run Process ${PYTHON} ${HELPER} set-env-priority-fallback cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} project-context-set-env-priority-fallback-ok
Project Context Set Persists Execution Env Priority
[Documentation] Verify that execution_env_priority is persisted and retrievable
... after ``project context set``. Currently fails due to bug #1079.
${result}= Run Process ${PYTHON} ${HELPER} set-env-priority-persists cwd=${WORKSPACE}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} project-context-set-env-priority-persists-ok