cleveragents/cleveragents-core

Fork 3

UAT: `Validation.from_config` ignores `validation.mode` nested YAML key — spec-documented YAML format silently falls back to `required` mode #5113

New issue

Open

opened 2026-04-09 01:04:20 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-09 01:04:20 +00:00

Owner

Bug Report

Feature Area: Validation and Invariants — Validation Tool Subtype / YAML Configuration
Tested By: UAT worker (uat-pool-1), feature area: Validation and Invariants
Severity: High — users following the spec's documented YAML format cannot set informational mode; it silently defaults to required

What Was Tested

Code-level analysis of Validation.from_config in:

src/cleveragents/domain/models/core/tool.py

Cross-referenced against the spec's documented YAML format in:

docs/specification.md §Validation Configuration Files (line 34312)
docs/specification.md §Tool Wrapping (line 22758)

Expected Behavior (from spec)

The spec (§Validation Configuration Files, line 34317) states:

A Validation YAML file uses the same format as a Tool YAML file, with an additional validation block containing the validation-specific metadata.

The spec's table (line 34329) documents the field as:

Field	Type	Default	Description
`validation.mode`	string	`"required"`	`"required"` or `"informational"`

The spec provides multiple YAML examples showing mode nested under a validation: key:

Example 1 (§Tool Wrapping, line 22776):

name: local/tests-pass
description: "Validate that all unit tests pass (wraps local/run-tests)"
wraps: local/run-tests
transform: |
  def transform(tool_output):
      ...

validation:
  mode: required    ← mode is nested under "validation:"

timeout: 600

Example 2 (§Validation Configuration Files, line 34376):

name: local/run-tests
description: "Run unit tests with coverage and report pass/fail"
source: custom
code: |
  ...

validation:
  mode: required    ← mode is nested under "validation:"

input_schema:
  ...

Actual Behavior

Validation.from_config reads mode from the top-level config dict, not from the nested validation: key:

# src/cleveragents/domain/models/core/tool.py, line 689
return cls(
    ...
    mode=ValidationMode(config.get("mode", "required")),  # ← reads top-level "mode"
    ...
)

When a user writes a YAML file following the spec's documented format with validation:\n mode: informational, the from_config method:

Looks for config["mode"] at the top level — not found
Falls back to the default "required"
Silently ignores the validation.mode setting

Concrete Example

# user writes this YAML following the spec:
name: local/bundle-size-check
description: "Check bundle size (advisory)"
source: custom
code: |
  def run(inputs):
      return {"passed": True, "message": "Bundle size OK"}

validation:
  mode: informational   ← user intends informational mode

config = yaml.safe_load(yaml_content)
validation = Validation.from_config(config)
print(validation.mode)  # → ValidationMode.REQUIRED  ← WRONG, should be INFORMATIONAL

The mode is silently forced to required because config.get("mode", "required") finds no top-level mode key.

Impact

Users cannot set informational mode using the spec's documented YAML format — all validations become required regardless of intent.
Silent failure — no error is raised; the wrong mode is applied without warning.
Spec compliance: The spec documents validation.mode as the canonical field name. The implementation reads mode (top-level), which is undocumented.
Gradual promotion pattern broken: The spec's recommended pattern of starting validations as informational and promoting to required (§Validation Mode, line 22492) cannot be implemented via YAML files.

Code Location

src/cleveragents/domain/models/core/tool.py:
- Line 689: mode=ValidationMode(config.get("mode", "required")) — should read from config.get("validation", {}).get("mode", "required")

Steps to Reproduce

import yaml
from cleveragents.domain.models.core.tool import Validation, ValidationMode

yaml_content = """
name: local/bundle-size-check
description: "Check bundle size (advisory)"
source: custom
code: |
  def run(inputs):
      return {"passed": True}

validation:
  mode: informational
"""

config = yaml.safe_load(yaml_content)
v = Validation.from_config(config)
assert v.mode == ValidationMode.INFORMATIONAL  # FAILS — mode is REQUIRED

Fix Required

In Validation.from_config, read mode from the nested validation: key, with fallback to top-level mode for backward compatibility:

# Read mode from nested "validation:" block (spec format), with fallback to top-level
validation_block = config.get("validation", {})
mode_str = validation_block.get("mode") or config.get("mode", "required")
mode = ValidationMode(mode_str)

This preserves backward compatibility with any existing YAML files that use the top-level mode key while also supporting the spec-documented validation.mode format.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: Validation and Invariants — Validation Tool Subtype / YAML Configuration **Tested By**: UAT worker (uat-pool-1), feature area: Validation and Invariants **Severity**: High — users following the spec's documented YAML format cannot set `informational` mode; it silently defaults to `required` --- ## What Was Tested Code-level analysis of `Validation.from_config` in: - `src/cleveragents/domain/models/core/tool.py` Cross-referenced against the spec's documented YAML format in: - `docs/specification.md` §Validation Configuration Files (line 34312) - `docs/specification.md` §Tool Wrapping (line 22758) ## Expected Behavior (from spec) The spec (§Validation Configuration Files, line 34317) states: > A Validation YAML file uses the **same format** as a Tool YAML file, with an additional `validation` block containing the validation-specific metadata. The spec's table (line 34329) documents the field as: | Field | Type | Default | Description | |-------|------|---------|-------------| | `validation.mode` | string | `"required"` | `"required"` or `"informational"` | The spec provides multiple YAML examples showing `mode` nested under a `validation:` key: **Example 1** (§Tool Wrapping, line 22776): ```yaml name: local/tests-pass description: "Validate that all unit tests pass (wraps local/run-tests)" wraps: local/run-tests transform: | def transform(tool_output): ... validation: mode: required ← mode is nested under "validation:" timeout: 600 ``` **Example 2** (§Validation Configuration Files, line 34376): ```yaml name: local/run-tests description: "Run unit tests with coverage and report pass/fail" source: custom code: | ... validation: mode: required ← mode is nested under "validation:" input_schema: ... ``` ## Actual Behavior `Validation.from_config` reads `mode` from the **top-level** config dict, not from the nested `validation:` key: ```python # src/cleveragents/domain/models/core/tool.py, line 689 return cls( ... mode=ValidationMode(config.get("mode", "required")), # ← reads top-level "mode" ... ) ``` When a user writes a YAML file following the spec's documented format with `validation:\n mode: informational`, the `from_config` method: 1. Looks for `config["mode"]` at the top level — not found 2. Falls back to the default `"required"` 3. **Silently ignores** the `validation.mode` setting ### Concrete Example ```yaml # user writes this YAML following the spec: name: local/bundle-size-check description: "Check bundle size (advisory)" source: custom code: | def run(inputs): return {"passed": True, "message": "Bundle size OK"} validation: mode: informational ← user intends informational mode ``` ```python config = yaml.safe_load(yaml_content) validation = Validation.from_config(config) print(validation.mode) # → ValidationMode.REQUIRED ← WRONG, should be INFORMATIONAL ``` The `mode` is silently forced to `required` because `config.get("mode", "required")` finds no top-level `mode` key. ## Impact 1. **Users cannot set `informational` mode** using the spec's documented YAML format — all validations become `required` regardless of intent. 2. **Silent failure** — no error is raised; the wrong mode is applied without warning. 3. **Spec compliance**: The spec documents `validation.mode` as the canonical field name. The implementation reads `mode` (top-level), which is undocumented. 4. **Gradual promotion pattern broken**: The spec's recommended pattern of starting validations as `informational` and promoting to `required` (§Validation Mode, line 22492) cannot be implemented via YAML files. ## Code Location - `src/cleveragents/domain/models/core/tool.py`: - Line 689: `mode=ValidationMode(config.get("mode", "required"))` — should read from `config.get("validation", {}).get("mode", "required")` ## Steps to Reproduce ```python import yaml from cleveragents.domain.models.core.tool import Validation, ValidationMode yaml_content = """ name: local/bundle-size-check description: "Check bundle size (advisory)" source: custom code: | def run(inputs): return {"passed": True} validation: mode: informational """ config = yaml.safe_load(yaml_content) v = Validation.from_config(config) assert v.mode == ValidationMode.INFORMATIONAL # FAILS — mode is REQUIRED ``` ## Fix Required In `Validation.from_config`, read `mode` from the nested `validation:` key, with fallback to top-level `mode` for backward compatibility: ```python # Read mode from nested "validation:" block (spec format), with fallback to top-level validation_block = config.get("validation", {}) mode_str = validation_block.get("mode") or config.get("mode", "required") mode = ValidationMode(mode_str) ``` This preserves backward compatibility with any existing YAML files that use the top-level `mode` key while also supporting the spec-documented `validation.mode` format. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 01:04:24 +00:00

HAL9000 added this to the v3.2.0 milestone

2026-04-09 01:10:59 +00:00

HAL9000 commented

2026-04-09 01:11:11 +00:00

Author

Owner

Issue triaged by project owner: Verified as valid spec compliance bug. Priority: Medium. Milestone: v3.2.0.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner

Issue triaged by project owner: Verified as valid spec compliance bug. Priority: Medium. Milestone: v3.2.0. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner

HAL9000 added a new dependency

2026-04-09 01:16:54 +00:00

#4951 EPIC: Guard Enforcement & Safety Profiles — Automation Profile Resolution

HAL9000 referenced this issue

2026-04-09 01:17:55 +00:00

[AUTO-EPIC] Epic Planning Health Report (Cycle 1) #4985

HAL9000 referenced this issue

2026-04-09 02:56:11 +00:00

[AUTO-UAT-POOL] UAT Testing Report (Cycle 4) #5160

HAL9000 referenced this issue

2026-04-09 03:09:06 +00:00

[AUTO-UAT-POOL] UAT Testing Report (Cycle 5) #5200