UAT: Validation model does not enforce mandatory `passed` boolean field in output_schema — violates spec Validation contract #3671

New issue

Open

opened 2026-04-05 21:20:09 +00:00 by freemo · 0 comments

freemo commented

2026-04-05 21:20:09 +00:00

Owner

Metadata

Branch: fix/validation-output-schema-passed-field-required
Commit Message: fix(domain/validation): enforce mandatory passed boolean in Validation output_schema
Milestone: None (Backlog)
Parent Epic: #392 (Epic: Actor YAML & Compiler)

Backlog note: This issue was discovered during autonomous operation
on milestone v3.7.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

What Was Tested

Reviewed the Validation domain model in src/cleveragents/domain/models/core/tool.py and compared its output schema enforcement against the specification's Validation contract.

Expected Behavior (from spec)

Per docs/specification.md (Glossary, Tools & Skills section):

Validation: A Tool subtype adding: a mode (required | informational) controlling whether failure blocks execution; a structured JSON return with mandatory passed boolean, optional data, and optional message. Always read-only (writes = false, checkpointable = false).

The spec explicitly requires that a Validation's output schema MUST include a mandatory passed boolean field. This is the core semantic contract of a Validation — it must return a pass/fail result.

Actual Behavior (from code)

In src/cleveragents/domain/models/core/tool.py, the Validation class:

Correctly enforces read_only=True, writes=False, checkpointable=False via _enforce_validation_constraints
Correctly enforces tool_type=ToolType.VALIDATION
Does NOT validate that output_schema contains a passed boolean field

A Validation can be created with an output_schema that has no passed field, or with output_schema=None, without any validation error:

from cleveragents.domain.models.core.tool import Validation, ToolSource
# This should fail but succeeds — no 'passed' field in output_schema:
v = Validation(
    name="local/my-validation",
    description="A validation without passed field",
    source=ToolSource.BUILTIN,
    output_schema={"type": "object", "properties": {"result": {"type": "string"}}},
    # Missing required 'passed' boolean field in output_schema
)

Impact

Spec non-compliance: Validations registered without a passed field in their output schema will silently fail at runtime when the execution layer tries to read result["passed"].
Silent failures: The plan execution engine expects validation_result["passed"] to be a boolean. If absent, this causes a KeyError or TypeError at runtime rather than a clear validation error at registration time.
Misleading tool registry: Tools registered as validations that don't conform to the validation contract pollute the registry.

Code Location

src/cleveragents/domain/models/core/tool.py — Validation class, _enforce_validation_constraints model validator

Suggested Fix

Add a model validator to Validation that checks the output_schema contains a passed boolean:

@model_validator(mode="after")
def _validate_output_schema_has_passed(self) -> Validation:
    """Enforce that output_schema contains a mandatory 'passed' boolean field."""
    if self.output_schema is not None:
        props = self.output_schema.get("properties", {})
        if "passed" not in props:
            raise ValueError(
                "Validation output_schema must include a 'passed' boolean field "
                "(spec requirement: structured JSON return with mandatory passed boolean)"
            )
        passed_type = props["passed"].get("type")
        if passed_type != "boolean":
            raise ValueError(
                f"Validation output_schema 'passed' field must be type 'boolean', "
                f"got '{passed_type}'"
            )
    return self

Subtasks

Add _validate_output_schema_has_passed model validator to the Validation class in src/cleveragents/domain/models/core/tool.py
Validator must check that output_schema.properties contains a passed key of type boolean
Validator must raise a clear ValueError with a spec-referencing message when the field is missing or wrong type
Update all existing tests that construct Validation instances to include a valid output_schema with a passed boolean field
Add a new Behave scenario verifying Validation rejects output_schema without a passed boolean field
Run nox -e typecheck and nox -e unit_tests to confirm no regressions

Definition of Done

Validation model validates that output_schema contains a passed boolean field when output_schema is provided
Validation model raises a clear ValueError when output_schema is missing the passed field
Existing tests updated to include valid output_schema with passed field
New behave test scenario verifies Validation rejects output_schema without passed boolean
All nox stages pass
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/validation-output-schema-passed-field-required` - **Commit Message**: `fix(domain/validation): enforce mandatory passed boolean in Validation output_schema` - **Milestone**: None (Backlog) - **Parent Epic**: #392 (Epic: Actor YAML & Compiler) > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.7.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. ## What Was Tested Reviewed the `Validation` domain model in `src/cleveragents/domain/models/core/tool.py` and compared its output schema enforcement against the specification's Validation contract. ## Expected Behavior (from spec) Per `docs/specification.md` (Glossary, Tools & Skills section): > **Validation**: A Tool subtype adding: a `mode` (`required` | `informational`) controlling whether failure blocks execution; a **structured JSON return with mandatory `passed` boolean**, optional `data`, and optional `message`. Always read-only (`writes = false`, `checkpointable = false`). The spec explicitly requires that a Validation's output schema MUST include a mandatory `passed` boolean field. This is the core semantic contract of a Validation — it must return a pass/fail result. ## Actual Behavior (from code) In `src/cleveragents/domain/models/core/tool.py`, the `Validation` class: 1. Correctly enforces `read_only=True`, `writes=False`, `checkpointable=False` via `_enforce_validation_constraints` 2. Correctly enforces `tool_type=ToolType.VALIDATION` 3. **Does NOT validate** that `output_schema` contains a `passed` boolean field A `Validation` can be created with an `output_schema` that has no `passed` field, or with `output_schema=None`, without any validation error: ```python from cleveragents.domain.models.core.tool import Validation, ToolSource # This should fail but succeeds — no 'passed' field in output_schema: v = Validation( name="local/my-validation", description="A validation without passed field", source=ToolSource.BUILTIN, output_schema={"type": "object", "properties": {"result": {"type": "string"}}}, # Missing required 'passed' boolean field in output_schema ) ``` ## Impact 1. **Spec non-compliance**: Validations registered without a `passed` field in their output schema will silently fail at runtime when the execution layer tries to read `result["passed"]`. 2. **Silent failures**: The plan execution engine expects `validation_result["passed"]` to be a boolean. If absent, this causes a `KeyError` or `TypeError` at runtime rather than a clear validation error at registration time. 3. **Misleading tool registry**: Tools registered as validations that don't conform to the validation contract pollute the registry. ## Code Location - `src/cleveragents/domain/models/core/tool.py` — `Validation` class, `_enforce_validation_constraints` model validator ## Suggested Fix Add a model validator to `Validation` that checks the `output_schema` contains a `passed` boolean: ```python @model_validator(mode="after") def _validate_output_schema_has_passed(self) -> Validation: """Enforce that output_schema contains a mandatory 'passed' boolean field.""" if self.output_schema is not None: props = self.output_schema.get("properties", {}) if "passed" not in props: raise ValueError( "Validation output_schema must include a 'passed' boolean field " "(spec requirement: structured JSON return with mandatory passed boolean)" ) passed_type = props["passed"].get("type") if passed_type != "boolean": raise ValueError( f"Validation output_schema 'passed' field must be type 'boolean', " f"got '{passed_type}'" ) return self ``` ## Subtasks - [ ] Add `_validate_output_schema_has_passed` model validator to the `Validation` class in `src/cleveragents/domain/models/core/tool.py` - [ ] Validator must check that `output_schema.properties` contains a `passed` key of type `boolean` - [ ] Validator must raise a clear `ValueError` with a spec-referencing message when the field is missing or wrong type - [ ] Update all existing tests that construct `Validation` instances to include a valid `output_schema` with a `passed` boolean field - [ ] Add a new Behave scenario verifying `Validation` rejects `output_schema` without a `passed` boolean field - [ ] Run `nox -e typecheck` and `nox -e unit_tests` to confirm no regressions ## Definition of Done - [ ] `Validation` model validates that `output_schema` contains a `passed` boolean field when `output_schema` is provided - [ ] `Validation` model raises a clear `ValueError` when `output_schema` is missing the `passed` field - [ ] Existing tests updated to include valid `output_schema` with `passed` field - [ ] New behave test scenario verifies Validation rejects output_schema without `passed` boolean - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added a new dependency

2026-04-05 21:20:22 +00:00

#392 Epic: Actor YAML & Compiler

freemo referenced this issue

2026-04-05 21:20:57 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 21:24:08 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 21:29:54 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377