cleveragents/cleveragents-core

Fork 3

TDD: Write failing test for #1038 — validation add missing --required flag #1102

New issue

Closed

opened 2026-03-22 16:30:11 +00:00 by freemo · 2 comments

freemo commented

2026-03-22 16:30:11 +00:00

Owner

Metadata

Commit Message: test: add TDD bug-capture test for #1038 — validation add --required flag
Branch: tdd/m5-validation-required-flag

Background and Context

This is the TDD counterpart to bug #1038. Per the project's Test-Driven Development workflow for bugs (see CONTRIBUTING.md > Bug Fix Workflow), the first step in fixing any bug is to write a test that captures the buggy behavior. The test is tagged with @tdd_bug, @tdd_bug_1038, and @tdd_expected_fail so that it passes CI while the bug is still unfixed. Once the fix is implemented in #1038, the @tdd_expected_fail tag will be removed and the test will run normally.

See #1038 for full bug details.

Expected Behavior

A new test exists that:

Captures the exact failure described in #1038.
Is tagged with @tdd_bug, @tdd_bug_1038, and @tdd_expected_fail.
Passes CI via the expected-failure mechanism (the underlying assertion fails, confirming the bug exists, but the tag inversion causes the test to pass).
Would fail CI if the bug were fixed without removing the @tdd_expected_fail tag.

Acceptance Criteria

A test is written that captures the bug behavior described in #1038.
The test is tagged with @tdd_bug, @tdd_bug_1038, and @tdd_expected_fail.
The @tdd_expected_fail tag causes the test to pass CI (the underlying assertion fails as expected, proving the bug exists).
The test is specific enough that it will pass normally (without the tag) only when the bug is genuinely fixed.
Tag validation rules pass: @tdd_bug_1038 has corresponding @tdd_bug, and @tdd_expected_fail has both.
A pull request is opened from the branch to master, CI passes, and the PR is merged through the normal merge process.

Definition of Done

This issue is complete when:

All subtasks below are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the test and what bug behavior it captures.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, CI passes, and the PR is merged before this issue is marked done.

Subtasks

Code: Analyze bug #1038 to identify the exact failure condition, including the inputs, state, and code path that trigger the bug.
Code: Determine the appropriate test type (Behave unit test, Robot integration test, or both) and file location for the reproducing test.
Tests (Behave): Write a Behave scenario in features/ that captures the bug. Tag the scenario with @tdd_bug, @tdd_bug_1038, and @tdd_expected_fail. The scenario must exercise the specific code path that triggers the bug and assert the correct expected behavior (which currently fails due to the bug). Name the scenario descriptively to indicate it is a bug regression test.
Tests (Robot): Add a Robot test in robot/ with equivalent tags. Four test cases in robot/tdd_validation_required_flag.robot exercise the same CLI paths as the Behave scenarios via robot/helper_tdd_validation_required_flag.py.
Docs: Add a comment in the test file explaining this test captures bug #1038 and uses @tdd_expected_fail until the fix is merged.
Quality: Verify CI passes with the tagged test. Confirm the underlying assertion fails for the correct reason.
Quality: Verify tag validation rules pass.
Quality: Verify coverage >=97% via nox -s coverage_report. If coverage is <97% then review the current unit test coverage report at build/coverage.xml and use it to write new Behave based unit tests to improve code coverage.
Quality: Run nox (all default sessions), fix any errors if needed ensuring nox passes across entire code base.

## Metadata - **Commit Message**: `test: add TDD bug-capture test for #1038 — validation add --required flag` - **Branch**: `tdd/m5-validation-required-flag` ## Background and Context This is the TDD counterpart to bug #1038. Per the project's Test-Driven Development workflow for bugs (see `CONTRIBUTING.md` > Bug Fix Workflow), the first step in fixing any bug is to write a test that captures the buggy behavior. The test is tagged with `@tdd_bug`, `@tdd_bug_1038`, and `@tdd_expected_fail` so that it passes CI while the bug is still unfixed. Once the fix is implemented in #1038, the `@tdd_expected_fail` tag will be removed and the test will run normally. See #1038 for full bug details. ## Expected Behavior A new test exists that: 1. Captures the exact failure described in #1038. 2. Is tagged with `@tdd_bug`, `@tdd_bug_1038`, and `@tdd_expected_fail`. 3. Passes CI via the expected-failure mechanism (the underlying assertion fails, confirming the bug exists, but the tag inversion causes the test to pass). 4. Would fail CI if the bug were fixed without removing the `@tdd_expected_fail` tag. ## Acceptance Criteria - [x] A test is written that captures the bug behavior described in #1038. - [x] The test is tagged with `@tdd_bug`, `@tdd_bug_1038`, and `@tdd_expected_fail`. - [x] The `@tdd_expected_fail` tag causes the test to pass CI (the underlying assertion fails as expected, proving the bug exists). - [x] The test is specific enough that it will pass normally (without the tag) only when the bug is genuinely fixed. - [x] Tag validation rules pass: `@tdd_bug_1038` has corresponding `@tdd_bug`, and `@tdd_expected_fail` has both. - [ ] A pull request is opened from the branch to `master`, CI passes, and the PR is merged through the normal merge process. ## Definition of Done This issue is complete when: - All subtasks below are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the test and what bug behavior it captures. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, CI passes, and the PR is **merged** before this issue is marked done. ## Subtasks - [x] Code: Analyze bug #1038 to identify the exact failure condition, including the inputs, state, and code path that trigger the bug. - [x] Code: Determine the appropriate test type (Behave unit test, Robot integration test, or both) and file location for the reproducing test. - [x] Tests (Behave): Write a Behave scenario in `features/` that captures the bug. Tag the scenario with `@tdd_bug`, `@tdd_bug_1038`, and `@tdd_expected_fail`. The scenario must exercise the specific code path that triggers the bug and assert the correct expected behavior (which currently fails due to the bug). Name the scenario descriptively to indicate it is a bug regression test. - [x] Tests (Robot): Add a Robot test in `robot/` with equivalent tags. Four test cases in `robot/tdd_validation_required_flag.robot` exercise the same CLI paths as the Behave scenarios via `robot/helper_tdd_validation_required_flag.py`. - [x] Docs: Add a comment in the test file explaining this test captures bug #1038 and uses `@tdd_expected_fail` until the fix is merged. - [x] Quality: Verify CI passes with the tagged test. Confirm the underlying assertion fails for the correct reason. - [x] Quality: Verify tag validation rules pass. - [x] Quality: Verify coverage >=97% via `nox -s coverage_report`. If coverage is <97% then review the current unit test coverage report at `build/coverage.xml` and use it to write new Behave based unit tests to improve code coverage. - [x] Quality: Run `nox` (all default sessions), fix any errors if needed ensuring nox passes across **entire** code base.

freemo added this to the v3.5.0 milestone

2026-03-22 16:30:11 +00:00

freemo added the

labels

2026-03-22 16:30:11 +00:00

hurui200320 was assigned by freemo

2026-03-22 16:30:11 +00:00

freemo added a new dependency

2026-03-22 16:32:55 +00:00

#1038 --required flag in agents validation add

freemo referenced this issue

2026-03-22 16:32:55 +00:00

--required flag in agents validation add #1038

freemo referenced this issue

2026-03-22 16:39:30 +00:00

--required flag in agents validation add #1038

freemo added the

State

Verified

label

2026-03-22 16:54:45 +00:00

hurui200320 was unassigned by freemo

2026-03-23 03:30:08 +00:00

brent.edwards was assigned by freemo

2026-03-23 03:30:08 +00:00

brent.edwards added

and removed

labels

2026-03-23 04:18:00 +00:00

brent.edwards referenced this issue from a commit

2026-03-23 12:35:37 +00:00

test: add TDD bug-capture test for #1038 — validation add --required flag

brent.edwards commented

2026-03-23 12:36:09 +00:00

Member

Implementation Notes

Bug Analysis

The add command in cleveragents.cli.commands.validation (function add()) defines only --config, --update, and --format options. It does not define --required or --informational flags. The spec (docs/specification.md §22334) explicitly requires these flags. Passing --required triggers Typer's NoSuchOption error with exit code 2.

Test Design

Type: Behave unit test — purely CLI argument parsing issue, no integration behavior.
Feature: features/tdd_validation_add_required_flag.feature — 3 scenarios tagged @tdd_expected_fail @tdd_bug @tdd_bug_1038
Steps: features/steps/tdd_validation_add_required_flag_steps.py — uses typer.testing.CliRunner with mocked _get_tool_registry_service
Robot: N/A — failure occurs at Typer option parsing before any service layer code is reached.

Quality Gates

Gate	Result
`nox -s lint`	✅ Pass
`nox -s typecheck`	✅ Pass (0 errors)
`nox -s unit_tests`	✅ All 3 new scenarios pass via `@tdd_expected_fail`
`nox -s coverage_report`	✅ 98% (above 97% threshold)
Tag validation	✅ All tags comply

## Implementation Notes ### Bug Analysis The `add` command in `cleveragents.cli.commands.validation` (function `add()`) defines only `--config`, `--update`, and `--format` options. It does **not** define `--required` or `--informational` flags. The spec (`docs/specification.md` §22334) explicitly requires these flags. Passing `--required` triggers Typer's `NoSuchOption` error with exit code 2. ### Test Design - **Type:** Behave unit test — purely CLI argument parsing issue, no integration behavior. - **Feature:** `features/tdd_validation_add_required_flag.feature` — 3 scenarios tagged `@tdd_expected_fail @tdd_bug @tdd_bug_1038` - **Steps:** `features/steps/tdd_validation_add_required_flag_steps.py` — uses `typer.testing.CliRunner` with mocked `_get_tool_registry_service` - **Robot:** N/A — failure occurs at Typer option parsing before any service layer code is reached. ### Quality Gates | Gate | Result | |------|--------| | `nox -s lint` | ✅ Pass | | `nox -s typecheck` | ✅ Pass (0 errors) | | `nox -s unit_tests` | ✅ All 3 new scenarios pass via `@tdd_expected_fail` | | `nox -s coverage_report` | ✅ 98% (above 97% threshold) | | Tag validation | ✅ All tags comply |

~~brent.edwards referenced this issue 2026-03-23 12:36:22 +00:00~~

test: add TDD bug-capture test for #1038 — validation add --required flag #1133

brent.edwards added

and removed

labels

2026-03-23 12:38:05 +00:00

brent.edwards referenced this issue from a commit

2026-03-23 17:14:04 +00:00

test: add TDD bug-capture test for #1038 — validation add --required flag

brent.edwards referenced this issue from a pull request that will close it,

2026-03-23 17:14:23 +00:00

test: add TDD bug-capture test for #1038 — validation add --required flag #1133

brent.edwards referenced this issue from a commit

2026-03-23 18:05:53 +00:00

test: add TDD bug-capture test for #1038 — validation add --required flag

brent.edwards referenced this issue from a commit

2026-03-25 01:46:59 +00:00

test: add TDD bug-capture test for #1038 — validation add --required flag

brent.edwards referenced this issue from a commit

2026-03-25 21:58:23 +00:00

test: add TDD bug-capture test for #1038 — validation add --required flag

brent.edwards commented

2026-03-25 21:59:49 +00:00

Member

Implementation Note: Robot Framework tests added per PR #1133 review

Review Comment Addressed

freemo's review on PR #1133 (Review ID 2707) identified a blocking issue: missing Robot Framework integration tests. This was the only TDD bug-capture PR in the current batch without .robot + helper coverage.

What Was Changed

Added two new files to the commit:

robot/tdd_validation_required_flag.robot — Four Robot Framework test cases exercising the same CLI paths as the Behave scenarios:
- TDD Validation Add Required Flag Accepted
- TDD Validation Add Informational Flag Accepted
- TDD Validation Add Required Flag Overrides YAML Config
- TDD Validation Add Informational Flag Overrides YAML Config
robot/helper_tdd_validation_required_flag.py — Helper script with four subcommands (check-required, check-informational, check-required-overrides-config, check-informational-overrides-config). Uses typer.testing.CliRunner with a mocked _get_tool_registry_service to invoke the validation CLI.

Design Decisions

Pattern: Followed the established pattern from helper_tdd_plan_apply_yes_flag.py — the helper reports the real outcome (exit 0 + sentinel on bug fixed, exit 1 on bug present) and the tdd_expected_fail_listener on the Robot side handles pass/fail inversion.
Override scenarios include false-positive guards: The check-required-overrides-config and check-informational-overrides-config subcommands verify both CLI output and the register_tool call args to prevent false positives from mock configuration.
Tags: All test cases tagged tdd_bug, tdd_bug_1038, tdd_expected_fail — matching the Behave scenarios.
Suite setup: Uses Setup Test Environment (no database isolation needed since this is an in-process CLI test with mocked services).

Quality Gate Results

nox -e lint: ✅ Pass
nox -e typecheck: ✅ Pass (0 errors)
nox -e unit_tests: ✅ Pass (473 features, 12455 scenarios, 0 failures)
nox -e integration_tests: ✅ All 4 new Robot tests pass. Pre-existing failures only (Container Resolve Crash timeout, Resource CLI timeout — unrelated to this PR).
nox -e coverage_report: ✅ 98% (>= 97% threshold)

Additional Work

Rebased branch onto latest origin/master (34c2acc3)
Updated ticket subtask "Tests (Robot)" from N/A to completed
Amended commit and force-pushed

## Implementation Note: Robot Framework tests added per PR #1133 review ### Review Comment Addressed freemo's review on PR #1133 (Review ID 2707) identified a blocking issue: missing Robot Framework integration tests. This was the only TDD bug-capture PR in the current batch without `.robot` + helper coverage. ### What Was Changed Added two new files to the commit: 1. **`robot/tdd_validation_required_flag.robot`** — Four Robot Framework test cases exercising the same CLI paths as the Behave scenarios: - `TDD Validation Add Required Flag Accepted` - `TDD Validation Add Informational Flag Accepted` - `TDD Validation Add Required Flag Overrides YAML Config` - `TDD Validation Add Informational Flag Overrides YAML Config` 2. **`robot/helper_tdd_validation_required_flag.py`** — Helper script with four subcommands (`check-required`, `check-informational`, `check-required-overrides-config`, `check-informational-overrides-config`). Uses `typer.testing.CliRunner` with a mocked `_get_tool_registry_service` to invoke the validation CLI. ### Design Decisions - **Pattern**: Followed the established pattern from `helper_tdd_plan_apply_yes_flag.py` — the helper reports the real outcome (exit 0 + sentinel on bug fixed, exit 1 on bug present) and the `tdd_expected_fail_listener` on the Robot side handles pass/fail inversion. - **Override scenarios include false-positive guards**: The `check-required-overrides-config` and `check-informational-overrides-config` subcommands verify both CLI output and the `register_tool` call args to prevent false positives from mock configuration. - **Tags**: All test cases tagged `tdd_bug`, `tdd_bug_1038`, `tdd_expected_fail` — matching the Behave scenarios. - **Suite setup**: Uses `Setup Test Environment` (no database isolation needed since this is an in-process CLI test with mocked services). ### Quality Gate Results - `nox -e lint`: ✅ Pass - `nox -e typecheck`: ✅ Pass (0 errors) - `nox -e unit_tests`: ✅ Pass (473 features, 12455 scenarios, 0 failures) - `nox -e integration_tests`: ✅ All 4 new Robot tests pass. Pre-existing failures only (Container Resolve Crash timeout, Resource CLI timeout — unrelated to this PR). - `nox -e coverage_report`: ✅ 98% (>= 97% threshold) ### Additional Work - Rebased branch onto latest `origin/master` (`34c2acc3`) - Updated ticket subtask "Tests (Robot)" from N/A to completed - Amended commit and force-pushed

brent.edwards referenced this issue from a commit

2026-03-28 02:41:01 +00:00

test: add TDD bug-capture test for #1038 — validation add --required flag

brent.edwards referenced this issue from a commit

2026-03-28 05:55:08 +00:00

test: add TDD bug-capture test for #1038 — validation add --required flag

brent.edwards closed this issue

2026-03-28 18:09:28 +00:00

brent.edwards referenced this issue from a commit

2026-03-28 18:09:28 +00:00

test: add TDD bug-capture test for #1038 — validation add --required flag (#1133)

brent.edwards referenced this issue