test(plan-tree): add failing BDD scenario proving corrected nodes not visually marked #8671

Merged
HAL9000 merged 3 commits from test/plan-tree-correction-visual-tdd into master 2026-06-02 22:56:04 +00:00
4 changed files with 185 additions and 1 deletions
+6 -1
View File
@@ -497,7 +497,12 @@ _ALL_DATA_COLUMNS + ") " "SELECT " + _ALL_DATA_COLUMNS + " FROM v3_plans"`.
- **Quick Start Guide** (PR #9245): Added `docs/quickstart.md` with an end-to-end quick start guide covering prerequisites, installation, project creation, resource registration, plan/apply workflow, and troubleshooting. Updated `mkdocs.yml` navigation to include the Quick Start page.
- **Plan checkpoint management CLI commands** (#8683): Added `agents plan checkpoint-list <plan-id>` and `agents plan checkpoint-delete <checkpoint-id>` commands. Listing output now highlights checkpoint ID, type, created timestamp, reason, phase, and decision linkage with a concise field summary footer across rich/table/json/yaml formats. Deletion supports batch IDs, interactive confirmation (skip with `--yes`), and structured JSON/YAML responses for automation-friendly scripting.
- **Invariant Remove CLI Command** (#8530): Implemented `agents invariant remove <id>` command that soft-deletes an invariant by ID. The command displays a confirmation prompt before removal (bypassable with `--yes`/`-y`), outputs the removed invariant ID on success, and shows a clear error message when the invariant ID does not exist. Supports `--format` flag for JSON and YAML output. Full BDD test coverage and Robot Framework integration tests included.
- **Invariant Remove CLI Command** (#8530): Implemented `agents invariant remove <id>` command that soft-deletes an invariant by ID. The command displays a confirmation prompt before removal (bypassable with `--yes`/ `-y`), outputs the removed invariant ID on success, and shows a clear error message when the invariant ID does not exist. Supports `--format` flag for JSON and YAML output. Full BDD test coverage and Robot Framework integration tests included.
- **TDD: plan tree does not visually mark corrected nodes** (#8576): Added a failing
BDD scenario proving that corrected nodes (decisions with `is_correction=True`) are
not visually distinguished in the `agents plan tree` output. The scenario is tagged
`@tdd_expected_fail` and will pass (by inversion) until the underlying gap described
in Spec Requirement #7 is fixed.
- **TDD: MCPToolAdapter.infer_resource_slots() TypeError with null properties** (#10470):
Added a TDD issue-capture Behave scenario that reproduces the bug where
`MCPToolAdapter.infer_resource_slots()` raises `TypeError` when the input schema
+1
View File
@@ -68,3 +68,4 @@ Below are some of the specific details of various contributions.
* HAL 9000 has contributed the plan correct JSON output envelope fix (PR #8662 / issue #8584): restructured `agents plan correct --format json` output to nest correction fields under `data.correction` and pass `command="plan correct"` to `format_output`, producing the spec-required CLI envelope. Added three BDD scenarios validating `data.correction.mode` (revert and append modes) and the `command` field.
* HAL 9000 has contributed BDD feature file tag coverage improvements (#9124 / pr #9183): added required `@a2a`, `@session`, and `@cli` Gherkin tags to 30 feature files (8 A2A, 7 session, 15 CLI) to enable selective tag-based test filtering via `behave --tags=a2a,session,cli`.
* HAL 9000 has contributed the plan tree JSON `decision_id` fix (#9096): updated `step_tree_json_valid` in features/steps/plan_explain_steps.py to correctly handle the {"data": [...]} envelope structure produced by format_output, and removed @tdd_expected_fail from the @tdd_issue_4254 scenario so it runs as a permanent regression guard.
* HAL 9000 has contributed the TDD scenario for plan tree correction visual marking (PR #8671 / issue #8576): added a failing BDD scenario proving that corrected nodes (decisions with is_correction=True) are not visually distinguished in the plan tree output, formalizing Spec Requirement #7 as an executable specification.
@@ -0,0 +1,155 @@
"""Step definitions for tdd_plan_tree_correction_visual.feature.
Tests that corrected nodes (decisions with is_correction=True) are visually
marked in the plan tree output. This is a TDD scenario documenting the gap
where the current implementation does NOT visually distinguish corrected nodes.
"""
from __future__ import annotations
from behave import given, then, when
from behave.runner import Context
from ulid import ULID
from cleveragents.cli.commands.plan import build_decision_tree
Outdated
Review

BLOCKER — CI / unit_tests Failing

This step imports build_decision_tree from cleveragents.cli.commands.plan. If this import triggers a runtime error (e.g., a missing dependency or side effect in the large CLI module), the step will fail with a non-AssertionError exception — bypassing the @tdd_expected_fail inversion mechanism in environment.py, which only inverts AssertionError.

To diagnose:

  1. Check CI logs for run 18747, job 4 — identify the exact exception type and stack trace
  2. Run nox -s unit_tests -- features/tdd_plan_tree_correction_visual.feature locally
  3. If it is an ImportError or AttributeError, trace the import chain and fix the root cause

Once the exact failure is known and resolved, CI should pass because the scenario logic itself is sound — _node_dict() has no label key, so str(corrected_node.get("label", "")) returns "", has_marker is False, and AssertionError is raised (which IS inverted to a pass).


Automated by CleverAgents Bot
Supervisor: PR Review | Agent: pr-review-worker

**BLOCKER — CI / unit_tests Failing** This step imports `build_decision_tree` from `cleveragents.cli.commands.plan`. If this import triggers a runtime error (e.g., a missing dependency or side effect in the large CLI module), the step will fail with a non-`AssertionError` exception — bypassing the `@tdd_expected_fail` inversion mechanism in `environment.py`, which only inverts `AssertionError`. To diagnose: 1. Check CI logs for run 18747, job 4 — identify the exact exception type and stack trace 2. Run `nox -s unit_tests -- features/tdd_plan_tree_correction_visual.feature` locally 3. If it is an `ImportError` or `AttributeError`, trace the import chain and fix the root cause Once the exact failure is known and resolved, CI should pass because the scenario logic itself is sound — `_node_dict()` has no `label` key, so `str(corrected_node.get("label", ""))` returns `""`, `has_marker` is `False`, and `AssertionError` is raised (which IS inverted to a pass). --- Automated by CleverAgents Bot Supervisor: PR Review | Agent: pr-review-worker
Outdated
Review

BLOCKER — CI / unit_tests Failing: Likely Non-AssertionError Exception Preventing TDD Inversion

This module-level import:

from cleveragents.cli.commands.plan import build_decision_tree

pulls in src/cleveragents/cli/commands/plan.py, a ~5000-line CLI module with heavy module-level initialization (get_container, SQLAlchemyError, A2aRequest, GitWorktreeSandbox, etc.). In the Behave CI test environment, this import may raise a non-AssertionError exception (e.g., ImportError, ModuleNotFoundError, RuntimeError).

The apply_tdd_inversion function in features/environment.py explicitly skips inversion for any exception that is not an AssertionError:

if not isinstance(step.exception, AssertionError):
    # ... warning logged, returns failed — NOT inverted
    return failed

This means a module-level import failure causes the test to report as a hard CI failure instead of an expected-fail pass.

Diagnosis:

  1. Check CI logs for run 18747, job 4 — find the exact exception type and stack trace
  2. Run nox -s unit_tests -- features/tdd_plan_tree_correction_visual.feature locally to reproduce

Suggested fix — defer the import inside the When step:

@when("I build the decision tree with default options")
def step_build_tree_default(context: Context) -> None:
    """Build the decision tree from the test decisions."""
    from cleveragents.cli.commands.plan import build_decision_tree  # deferred
    context.tdd_tree_data = build_decision_tree(
        context.tdd_decisions,
        show_superseded=False,
        max_depth=0,
    )

A deferred import ensures the exception (if any) occurs inside the step and provides a better diagnostic. Longer-term, consider moving build_decision_tree to a lighter domain-layer module (e.g., cleveragents.domain.services.plan_tree) so test files can import it without loading the entire CLI module.


Automated by CleverAgents Bot
Supervisor: PR Review | Agent: pr-review-worker

**BLOCKER — CI / unit_tests Failing: Likely Non-`AssertionError` Exception Preventing TDD Inversion** This module-level import: ```python from cleveragents.cli.commands.plan import build_decision_tree ``` pulls in `src/cleveragents/cli/commands/plan.py`, a ~5000-line CLI module with heavy module-level initialization (`get_container`, `SQLAlchemyError`, `A2aRequest`, `GitWorktreeSandbox`, etc.). In the Behave CI test environment, this import may raise a non-`AssertionError` exception (e.g., `ImportError`, `ModuleNotFoundError`, `RuntimeError`). The `apply_tdd_inversion` function in `features/environment.py` **explicitly skips inversion** for any exception that is not an `AssertionError`: ```python if not isinstance(step.exception, AssertionError): # ... warning logged, returns failed — NOT inverted return failed ``` This means a module-level import failure causes the test to report as a hard CI failure instead of an expected-fail pass. **Diagnosis:** 1. Check CI logs for run 18747, job 4 — find the exact exception type and stack trace 2. Run `nox -s unit_tests -- features/tdd_plan_tree_correction_visual.feature` locally to reproduce **Suggested fix — defer the import inside the When step:** ```python @when("I build the decision tree with default options") def step_build_tree_default(context: Context) -> None: """Build the decision tree from the test decisions.""" from cleveragents.cli.commands.plan import build_decision_tree # deferred context.tdd_tree_data = build_decision_tree( context.tdd_decisions, show_superseded=False, max_depth=0, ) ``` A deferred import ensures the exception (if any) occurs inside the step and provides a better diagnostic. Longer-term, consider moving `build_decision_tree` to a lighter domain-layer module (e.g., `cleveragents.domain.services.plan_tree`) so test files can import it without loading the entire CLI module. --- Automated by CleverAgents Bot Supervisor: PR Review | Agent: pr-review-worker
from cleveragents.domain.models.core.decision import Decision, DecisionType
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
_PLAN_ID = str(ULID())
# ---------------------------------------------------------------------------
# Given steps
# ---------------------------------------------------------------------------
@given("a set of test decisions with a corrected decision")
def step_decisions_with_correction(context: Context) -> None:
"""Create a set of decisions where one is marked as a correction."""
root_id = str(ULID())
corrected_id = str(ULID())
corrects_id = str(ULID())
context.tdd_decisions = [
# Root decision
Decision(
decision_id=root_id,
plan_id=_PLAN_ID,
sequence_number=0,
decision_type=DecisionType.PROMPT_DEFINITION,
question="What to build?",
chosen_option="REST API",
),
# Original decision that will be corrected
Decision(
decision_id=corrects_id,
plan_id=_PLAN_ID,
parent_decision_id=root_id,
sequence_number=1,
decision_type=DecisionType.STRATEGY_CHOICE,
question="Which framework?",
chosen_option="Django",
),
# Corrected decision (is_correction=True)
Decision(
decision_id=corrected_id,
plan_id=_PLAN_ID,
parent_decision_id=root_id,
sequence_number=2,
decision_type=DecisionType.STRATEGY_CHOICE,
question="Which framework?",
chosen_option="FastAPI",
is_correction=True,
corrects_decision_id=corrects_id,
),
]
context.tdd_corrected_id = corrected_id
# ---------------------------------------------------------------------------
# When steps
# ---------------------------------------------------------------------------
@when("I build the correction TDD test decision tree")
def step_build_tree_default(context: Context) -> None:
"""Build the decision tree from the test decisions."""
context.tdd_tree_data = build_decision_tree(
context.tdd_decisions,
show_superseded=False,
max_depth=0,
)
# ---------------------------------------------------------------------------
# Then steps
# ---------------------------------------------------------------------------
@then("the tree should include a visual marker for the corrected decision")
def step_tree_has_correction_marker(context: Context) -> None:
"""Verify that the tree data includes a marker for the corrected decision."""
tree_nodes: list[dict[str, object]] = context.tdd_tree_data
# The tree_data is a list of root-level node dicts.
# We need to search through it to find the corrected decision node
# and verify it has a visual marker.
def find_corrected_node(
nodes: list[dict[str, object]],
) -> dict[str, object] | None:
"""Recursively search for the corrected decision node."""
for node in nodes:
if node.get("decision_id") == context.tdd_corrected_id:
return node
children = node.get("children", [])
assert isinstance(children, list)
result = find_corrected_node(children)
if result is not None:
return result
return None
corrected_node = find_corrected_node(tree_nodes)
assert corrected_node is not None, (
f"Corrected decision {context.tdd_corrected_id} not found in tree"
)
# Store the node for the next assertion
context.tdd_corrected_node = corrected_node
@then(
'the corrected decision label should contain "[corrected]" or "" or similar marker'
)
def step_corrected_label_has_marker(context: Context) -> None:
"""Verify that the corrected decision's label contains a visual marker.
This assertion is expected to FAIL (via @tdd_expected_fail) because
build_decision_tree does not currently include a 'label' key in its
node dicts, and even if it did, it does not add visual markers for
corrected decisions (is_correction=True). This TDD scenario documents
the gap described in Spec Requirement #7.
"""
corrected_node = context.tdd_corrected_node
# The label should be in the node's label field.
# build_decision_tree currently does NOT include a 'label' key in its
# node dicts -- this assertion will fail with AssertionError, proving
# the bug exists (Spec Requirement #7: corrected nodes must be visually
# marked in the plan tree output).
label = str(corrected_node.get("label", ""))
# Check for common visual markers for corrections
markers = ["[corrected]", "\u270e", "[correction]", "\u270f", "[fix]", "[amended]"]
has_marker = any(marker in label for marker in markers)
assert has_marker, (
f"Corrected decision label '{label}' does not contain any visual marker. "
f"Expected one of: {markers}. "
f"build_decision_tree does not currently add visual markers for "
f"decisions with is_correction=True (Spec Requirement #7 gap)."
)
@@ -0,0 +1,23 @@
@tdd_expected_fail
@tdd_issue
@tdd_issue_8576
Feature: TDD: plan tree visually marks corrected nodes
Spec Requirement #7 states that `agents plan tree` should visually
distinguish corrected nodes (decisions with is_correction=True).
This TDD scenario documents the gap: the current implementation does NOT
visually mark corrected nodes in the tree output. The tree renderer builds
node labels without checking decision.is_correction.
Expected behavior: Corrected nodes should be marked with a visual indicator
such as [corrected], , or similar annotation.
# ------------------------------------------------------------------
# Scenario: Corrected node is visually marked in tree output
# ------------------------------------------------------------------
Scenario: Corrected decision is visually marked in plan tree
Given a set of test decisions with a corrected decision
When I build the correction TDD test decision tree
Then the tree should include a visual marker for the corrected decision
And the corrected decision label should contain "[corrected]" or "" or similar marker