--required flag in agents validation add #1038

Closed
opened 2026-03-18 02:22:40 +00:00 by brent.edwards · 11 comments
Member

According to the workflow examples, the following should be correct code:

# Register a required validation (tests must pass)
$ agents validation add \
    --config validations/unit-tests.yaml \
    --required \
    local/unit-tests

╭─ Validation Registered ─────────────────────────────────────╮
│ Name: local/unit-tests                                      │
│ Command: pytest tests/                                      │
│ Description: Unit tests                                     │
│ Mode: required                                              │
│ Timeout: 300s (default)                                     │
╰─────────────────────────────────────────────────────────────╯

✓ OK Validation registered

However, when it's run, the following error occurs:

test-20260317) ➜  test-20260317 agents validation add --config validations/unit-tests.yaml --required local/unit-tests
Wrapping unexpected exception: NoSuchOption: No such option: --required
Error [500] INTERNAL: An unexpected error occurred

The code does not have the concept of a --required flag on agents validation add, as can be read in:

(test-20260317) ➜  test-20260317 agents validation add --help
                                                                                                                                                                        
 Usage: agents validation add [OPTIONS]                                                                                                                                 
                                                                                                                                                                        
 Register a new validation from a YAML configuration file.                                                                                                              
                                                                                                                                                                        
 Validations are created ONLY via ``--config <file>``.                                                                                                                  
                                                                                                                                                                        
 Examples:                                                                                                                                                              
     agents validation add --config ./validations/coverage-check.yaml                                                                                                   
     agents validation add --config ./validations/coverage-check.yaml --update                                                                                          
                                                                                                                                                                        
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *  --config  -c      PATH  Path to the validation YAML configuration file [required]                                                                                 │
│    --update                Update if validation already exists                                                                                                       │
│    --format  -f      TEXT  Output format: json, yaml, plain, table, or rich (default: rich) [default: rich]                                                          │
│    --help                  Show this message and exit.                                                                                                               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

This bug writer does not know whether the example documentation is incorrect (that is, there should not be a --required flag) or the code is incorrect (that is, there should be a --required flag.)

Metadata

  • Commit Message: fix: add --required/--informational flags to validation add CLI
  • Branch: bugfix/m5-validation-required-flag

Subtasks

  • Analyze: Read spec and TDD tests (#1102) to understand the expected behavior of --required/--informational flags
  • Code: Add --required and --informational mutually exclusive flags to agents validation add command in src/cleveragents/cli/commands/validation.py
  • Code: When a flag is passed, override the YAML config's mode value after parsing
  • Code: Reject invocations where both --required and --informational are passed simultaneously
  • Tests (Behave): Remove @tdd_expected_fail tag from features/tdd_validation_add_required_flag.feature, keeping @tdd_issue and @tdd_issue_1038
  • Tests (Robot): Remove tdd_expected_fail tag from all test cases in robot/tdd_validation_required_flag.robot, keeping tdd_issue and tdd_issue_1038
  • Tests (Behave): Add edge-case scenario for mutual exclusivity (both flags passed)
  • Tests (Robot): Add edge-case test for mutual exclusivity
  • Docs: Update spec walkthrough example (line 36267) to remove positional name argument
  • Docs: Update spec formal CLI reference (line 9281) to include [--required | --informational]
  • Quality: Verify coverage >= 97% via nox -s coverage_report
  • Quality: Run full nox suite and fix any errors

Definition of Done

This issue is complete when:

  • All subtasks are completed and checked off
  • The --required and --informational flags are accepted by agents validation add
  • The flags override the YAML config's mode value
  • TDD tests pass normally (without @tdd_expected_fail)
  • All quality gates pass (nox with coverage >= 97%)
  • A PR is opened from the branch to master
According to the workflow examples, the following should be correct code: ``` # Register a required validation (tests must pass) $ agents validation add \ --config validations/unit-tests.yaml \ --required \ local/unit-tests ╭─ Validation Registered ─────────────────────────────────────╮ │ Name: local/unit-tests │ │ Command: pytest tests/ │ │ Description: Unit tests │ │ Mode: required │ │ Timeout: 300s (default) │ ╰─────────────────────────────────────────────────────────────╯ ✓ OK Validation registered ``` However, when it's run, the following error occurs: ``` test-20260317) ➜ test-20260317 agents validation add --config validations/unit-tests.yaml --required local/unit-tests Wrapping unexpected exception: NoSuchOption: No such option: --required Error [500] INTERNAL: An unexpected error occurred ``` The code does not have the concept of a `--required` flag on `agents validation add`, as can be read in: ``` (test-20260317) ➜ test-20260317 agents validation add --help Usage: agents validation add [OPTIONS] Register a new validation from a YAML configuration file. Validations are created ONLY via ``--config <file>``. Examples: agents validation add --config ./validations/coverage-check.yaml agents validation add --config ./validations/coverage-check.yaml --update ╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * --config -c PATH Path to the validation YAML configuration file [required] │ │ --update Update if validation already exists │ │ --format -f TEXT Output format: json, yaml, plain, table, or rich (default: rich) [default: rich] │ │ --help Show this message and exit. │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ``` This bug writer does not know whether the example documentation is incorrect (that is, there should not be a `--required` flag) or the code is incorrect (that is, there should be a `--required` flag.) ## Metadata - **Commit Message**: `fix: add --required/--informational flags to validation add CLI` - **Branch**: `bugfix/m5-validation-required-flag` ## Subtasks - [x] Analyze: Read spec and TDD tests (#1102) to understand the expected behavior of `--required`/`--informational` flags - [x] Code: Add `--required` and `--informational` mutually exclusive flags to `agents validation add` command in `src/cleveragents/cli/commands/validation.py` - [x] Code: When a flag is passed, override the YAML config's `mode` value after parsing - [x] Code: Reject invocations where both `--required` and `--informational` are passed simultaneously - [x] Tests (Behave): Remove `@tdd_expected_fail` tag from `features/tdd_validation_add_required_flag.feature`, keeping `@tdd_issue` and `@tdd_issue_1038` - [x] Tests (Robot): Remove `tdd_expected_fail` tag from all test cases in `robot/tdd_validation_required_flag.robot`, keeping `tdd_issue` and `tdd_issue_1038` - [x] Tests (Behave): Add edge-case scenario for mutual exclusivity (both flags passed) - [x] Tests (Robot): Add edge-case test for mutual exclusivity - [x] Docs: Update spec walkthrough example (line 36267) to remove positional name argument - [x] Docs: Update spec formal CLI reference (line 9281) to include `[--required | --informational]` - [x] Quality: Verify coverage >= 97% via `nox -s coverage_report` - [x] Quality: Run full `nox` suite and fix any errors ## Definition of Done This issue is complete when: - All subtasks are completed and checked off - The `--required` and `--informational` flags are accepted by `agents validation add` - The flags override the YAML config's `mode` value - TDD tests pass normally (without `@tdd_expected_fail`) - All quality gates pass (`nox` with coverage >= 97%) - A PR is opened from the branch to `master`
Owner

Planning Agent — Day 39 Triage

@brent.edwards — This issue reports that the --required flag in agents validation add is either missing or not functional. This is a feature gap, not a bug in existing behavior, since the --required flag may not have been implemented yet.

Looking at the specification (Section on validation commands), agents validation add accepts a --config file. The --required flag appears in the spec's workflow examples but may not be implemented in the current CLI.

Reclassification: This should be Type/Feature (missing CLI flag) or Type/Task (align CLI to spec), not Type/Bug. Since this is not broken existing functionality but rather an unimplemented spec feature, the TDD workflow for bugs does not apply.

Assignment: Assigning to v3.3.0 milestone and Priority/Low. This is a spec-alignment task that can be addressed when the validation CLI is revisited.

State change: Moving to State/Verified as a valid feature gap.

**Planning Agent — Day 39 Triage** @brent.edwards — This issue reports that the `--required` flag in `agents validation add` is either missing or not functional. This is a **feature gap**, not a bug in existing behavior, since the `--required` flag may not have been implemented yet. Looking at the specification (Section on validation commands), `agents validation add` accepts a `--config` file. The `--required` flag appears in the spec's workflow examples but may not be implemented in the current CLI. **Reclassification**: This should be `Type/Feature` (missing CLI flag) or `Type/Task` (align CLI to spec), not `Type/Bug`. Since this is not broken existing functionality but rather an unimplemented spec feature, the TDD workflow for bugs does not apply. **Assignment**: Assigning to v3.3.0 milestone and Priority/Low. This is a spec-alignment task that can be addressed when the validation CLI is revisited. **State change**: Moving to State/Verified as a valid feature gap.
freemo added this to the v3.5.0 milestone 2026-03-22 16:31:08 +00:00
Owner

Assigned to @hurui200320 for bug fix based on developer expertise (CLI alignment — Rui). Milestone set to v3.5.0. MoSCoW/Should have added. State changed from Unverified to Verified. This bug is top priority per project policy — bugs always take precedence over feature work.

Assigned to @hurui200320 for bug fix based on developer expertise (CLI alignment — Rui). Milestone set to v3.5.0. MoSCoW/Should have added. State changed from Unverified to Verified. This bug is top priority per project policy — bugs always take precedence over feature work.
Owner

TDD workflow initiated for this bug:

  • Created TDD issue #1102 to write a tagged test that captures this bug.
  • Dependency link added: this issue is blocked by #1102.
  • Once #1102 is merged to master, the bug fix assignee should create the bugfix branch and implement the fix.
  • The fix must remove the @tdd_expected_fail tag from the test so it runs normally.
TDD workflow initiated for this bug: - Created TDD issue #1102 to write a tagged test that captures this bug. - Dependency link added: this issue is blocked by #1102. - Once #1102 is merged to `master`, the bug fix assignee should create the bugfix branch and implement the fix. - The fix must remove the `@tdd_expected_fail` tag from the test so it runs normally.
Owner

Planning Agent — Discussion Review

@freemo — Same contradiction as #1039. Your Day 39 Triage explicitly states:

"This should be Type/Feature (missing CLI flag) or Type/Task (align CLI to spec), not Type/Bug. Since this is not broken existing functionality but rather an unimplemented spec feature, the TDD workflow for bugs does not apply."

But comments 2–3 reverse this by initiating the full TDD bug workflow with issue #1102.

Resolution: This depends on how we classify spec-alignment gaps:

  • If the --required flag was specified and should have been implemented in the current version, then omitting it is a defect (Type/Bug) and TDD applies per CONTRIBUTING.md §Bug Fix Workflow.
  • If the --required flag is a not-yet-implemented spec feature, it is Type/Feature and TDD bug workflow does not apply (though BDD tests should still accompany the implementation per CONTRIBUTING.md §Testing Philosophy).

The Needs feedback label is already present, which is appropriate. I am keeping the current classification (Type/Bug + TDD workflow) pending feedback, since the conservative approach is correct — if there is any doubt, treat it as a bug. This ensures we do not skip the TDD workflow for something that turns out to be a defect.

@hurui200320 — When you investigate, please confirm: was the --required flag part of the original implementation scope, or is it a future feature? Tag this issue accordingly.

Note: The Priority/Low label is correct for a spec-alignment gap in a non-critical command. No timeline pressure here.

## Planning Agent — Discussion Review @freemo — Same contradiction as #1039. Your Day 39 Triage explicitly states: > "This should be Type/Feature (missing CLI flag) or Type/Task (align CLI to spec), not Type/Bug. Since this is not broken existing functionality but rather an unimplemented spec feature, the TDD workflow for bugs does not apply." But comments 2–3 reverse this by initiating the full TDD bug workflow with issue #1102. **Resolution:** This depends on how we classify spec-alignment gaps: - If the `--required` flag was **specified and should have been implemented** in the current version, then omitting it is a defect (Type/Bug) and TDD applies per CONTRIBUTING.md §Bug Fix Workflow. - If the `--required` flag is a **not-yet-implemented spec feature**, it is Type/Feature and TDD bug workflow does not apply (though BDD tests should still accompany the implementation per CONTRIBUTING.md §Testing Philosophy). The `Needs feedback` label is already present, which is appropriate. I am keeping the current classification (`Type/Bug` + TDD workflow) pending feedback, since the conservative approach is correct — if there is any doubt, treat it as a bug. This ensures we do not skip the TDD workflow for something that turns out to be a defect. @hurui200320 — When you investigate, please confirm: was the `--required` flag part of the original implementation scope, or is it a future feature? Tag this issue accordingly. Note: The `Priority/Low` label is correct for a spec-alignment gap in a non-critical command. No timeline pressure here.
Member

@freemo Investigation results:

The --required flag was not part of the original implementation scope. The formal CLI reference (specification.md lines 9279–9290) defines the agents validation add signature as (--config|-c) <FILE> [--update] only — no --required/--informational flags. Line 9288 explicitly states the YAML config "must fully define the validation, including... the validation.mode field."

The --required flag appears only in two places that contradict the formal CLI reference:

  1. The walkthrough example (line 36262–36266) which also shows a nonexistent positional name argument
  2. A single sentence in Core Concepts > Validation Mode (line 22334)

The current code, all example YAML files, and all tests align with the formal CLI reference — mode is set via the YAML config, not a CLI flag.

Recommendation: Reclassify as Type/Task (spec cleanup). The fix is to update the walkthrough example and the Core Concepts text to remove mentions of --required/--informational CLI flags. The TDD bug workflow does not apply.

Additional finding: The spec's YAML examples use a nested format (validation:\n mode: required) but the code reads mode as a flat top-level field (config.get("mode", "required")). All actual YAML files use the flat format. This is a separate spec-vs-code alignment issue that should be tracked independently.

@freemo **Investigation results:** The `--required` flag was **not part of the original implementation scope**. The formal CLI reference (`specification.md` lines 9279–9290) defines the `agents validation add` signature as `(--config|-c) <FILE> [--update]` only — no `--required`/`--informational` flags. Line 9288 explicitly states the YAML config "must fully define the validation, including... the `validation.mode` field." The `--required` flag appears only in two places that contradict the formal CLI reference: 1. The walkthrough example (line 36262–36266) which also shows a nonexistent positional name argument 2. A single sentence in Core Concepts > Validation Mode (line 22334) The current code, all example YAML files, and all tests align with the formal CLI reference — mode is set via the YAML config, not a CLI flag. **Recommendation:** Reclassify as **Type/Task** (spec cleanup). The fix is to update the walkthrough example and the Core Concepts text to remove mentions of `--required`/`--informational` CLI flags. The TDD bug workflow does not apply. **Additional finding:** The spec's YAML examples use a **nested** format (`validation:\n mode: required`) but the code reads mode as a **flat** top-level field (`config.get("mode", "required")`). All actual YAML files use the flat format. This is a separate spec-vs-code alignment issue that should be tracked independently.
Author
Member

Implementation Notes

Resolution of Spec Contradiction

The specification had a contradiction regarding --required/--informational flags:

  1. Formal CLI reference (spec line 9281): Only defined (--config|-c) <FILE> [--update] — no mode flags.
  2. Core Concepts > Validation Mode (spec line 22339): Explicitly states mode can be set "via --required/--informational on agents validation add".
  3. Design Principles (spec line 30766): Stated "no CLI override flags are accepted" for entity registration commands.
  4. Walkthrough example (spec line 36268): Used --required flag + a nonexistent positional name argument.

Decision: Implement the flags per the Core Concepts section (more specific to validations) and reconcile the other spec sections. The validation add command is now an explicit exception to the general "no CLI override flags" design principle, documented as such.

Code Changes

  • src/cleveragents/cli/commands/validation.py (add function): Added --required and --informational as Typer boolean options. They are validated for mutual exclusivity before config parsing. When present, they override the mode key in the parsed YAML config dict before Validation.from_config() is called. Also imported ValidationMode from the domain model to use the enum values rather than string literals.

Test Changes

  • features/tdd_validation_add_required_flag.feature: Removed @tdd_expected_fail tag (keeping @tdd_issue @tdd_issue_1038). Added new scenario "Passing both --required and --informational is rejected" for the deferred mutual exclusivity edge case.
  • features/steps/tdd_validation_add_required_flag_steps.py: Added step_tdd_1038_add_both_flags (When) and step_tdd_1038_result_aborted (Then) steps. Updated docstrings to reflect the fix.
  • robot/tdd_validation_required_flag.robot: Removed tdd_expected_fail tag from all 4 existing test cases. Added new test case "TDD Validation Add Both Flags Rejected".
  • robot/helper_tdd_validation_required_flag.py: Added _check_both_flags_rejected subcommand. Updated docstrings.

Spec Changes

  • Line 9281: Updated CLI signature to (--config|-c) <FILE> [--required | --informational] [--update]. Added argument descriptions for --required and --informational.
  • Line 30766: Added exception clause for validation add in Design Principle #3.
  • Line 36270: Removed spurious positional local/unit-tests argument from walkthrough example (name comes from the YAML config, not the CLI).

Quality Gate Results

All 11 nox sessions passed:

  • lint:
  • format:
  • typecheck: (0 errors)
  • security_scan:
  • dead_code:
  • unit_tests: (508 features, 12986 scenarios, 0 failures)
  • integration_tests: (1850 tests, 0 failures)
  • docs:
  • build:
  • benchmark:
  • coverage_report: (97.0%, threshold: 97%)
## Implementation Notes ### Resolution of Spec Contradiction The specification had a contradiction regarding `--required`/`--informational` flags: 1. **Formal CLI reference** (spec line 9281): Only defined `(--config|-c) <FILE> [--update]` — no mode flags. 2. **Core Concepts > Validation Mode** (spec line 22339): Explicitly states mode can be set "via `--required`/`--informational` on `agents validation add`". 3. **Design Principles** (spec line 30766): Stated "no CLI override flags are accepted" for entity registration commands. 4. **Walkthrough example** (spec line 36268): Used `--required` flag + a nonexistent positional name argument. **Decision**: Implement the flags per the Core Concepts section (more specific to validations) and reconcile the other spec sections. The `validation add` command is now an explicit exception to the general "no CLI override flags" design principle, documented as such. ### Code Changes - **`src/cleveragents/cli/commands/validation.py`** (`add` function): Added `--required` and `--informational` as Typer boolean options. They are validated for mutual exclusivity before config parsing. When present, they override the `mode` key in the parsed YAML config dict before `Validation.from_config()` is called. Also imported `ValidationMode` from the domain model to use the enum values rather than string literals. ### Test Changes - **`features/tdd_validation_add_required_flag.feature`**: Removed `@tdd_expected_fail` tag (keeping `@tdd_issue @tdd_issue_1038`). Added new scenario "Passing both --required and --informational is rejected" for the deferred mutual exclusivity edge case. - **`features/steps/tdd_validation_add_required_flag_steps.py`**: Added `step_tdd_1038_add_both_flags` (When) and `step_tdd_1038_result_aborted` (Then) steps. Updated docstrings to reflect the fix. - **`robot/tdd_validation_required_flag.robot`**: Removed `tdd_expected_fail` tag from all 4 existing test cases. Added new test case "TDD Validation Add Both Flags Rejected". - **`robot/helper_tdd_validation_required_flag.py`**: Added `_check_both_flags_rejected` subcommand. Updated docstrings. ### Spec Changes - **Line 9281**: Updated CLI signature to `(--config|-c) <FILE> [--required | --informational] [--update]`. Added argument descriptions for `--required` and `--informational`. - **Line 30766**: Added exception clause for `validation add` in Design Principle #3. - **Line 36270**: Removed spurious positional `local/unit-tests` argument from walkthrough example (name comes from the YAML config, not the CLI). ### Quality Gate Results All 11 nox sessions passed: - lint: ✅ - format: ✅ - typecheck: ✅ (0 errors) - security_scan: ✅ - dead_code: ✅ - unit_tests: ✅ (508 features, 12986 scenarios, 0 failures) - integration_tests: ✅ (1850 tests, 0 failures) - docs: ✅ - build: ✅ - benchmark: ✅ - coverage_report: ✅ (97.0%, threshold: 97%)
Owner

PR #1222 reviewed, approved, and merged.

PR #1222 reviewed, approved, and merged.
freemo self-assigned this 2026-04-02 06:13:55 +00:00
Owner

PR #1222 reviewed, approved, and merged.

The fix adds --required and --informational as mutually exclusive boolean options to the agents validation add CLI command. When specified, they override the mode field from the YAML configuration file. The formal CLI reference in the specification has also been updated to include these flags.

All quality gates passed (lint, typecheck, unit_tests, integration_tests, coverage at 97%). The PR included comprehensive Behave BDD scenarios and Robot Framework integration tests covering flag acceptance, config override, and mutual exclusivity edge cases.

PR #1222 reviewed, approved, and merged. The fix adds `--required` and `--informational` as mutually exclusive boolean options to the `agents validation add` CLI command. When specified, they override the `mode` field from the YAML configuration file. The formal CLI reference in the specification has also been updated to include these flags. All quality gates passed (lint, typecheck, unit_tests, integration_tests, coverage at 97%). The PR included comprehensive Behave BDD scenarios and Robot Framework integration tests covering flag acceptance, config override, and mutual exclusivity edge cases.
Owner

PR #1222 reviewed, approved, and merged. The --required and --informational flags are now available on agents validation add.

PR #1222 reviewed, approved, and merged. The `--required` and `--informational` flags are now available on `agents validation add`.
freemo 2026-04-02 16:51:59 +00:00
Owner

PR #1222 reviewed, approved, and merged. The --required and --informational flags are now available on agents validation add.

PR #1222 reviewed, approved, and merged. The `--required` and `--informational` flags are now available on `agents validation add`.
Owner

PR #1222 independently reviewed and approved. The PR was already merged by a prior reviewer instance at 2026-04-02T17:39:19Z (merge commit 6c94ad95). My independent review confirmed the fix is correct and well-tested.

PR #1222 independently reviewed and approved. The PR was already merged by a prior reviewer instance at 2026-04-02T17:39:19Z (merge commit `6c94ad95`). My independent review confirmed the fix is correct and well-tested.
Sign in to join this conversation.
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveragents-core#1038
No description provided.