cleveragents/cleveragents-core

Fork 3

Tool registry does not validate `input_schema`/`output_schema` against JSON Schema meta-schema (spec violation) #8449

New issue

Open

opened 2026-04-13 19:08:53 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-13 19:08:53 +00:00

Owner

Metadata

Commit message: Build: Reinforced label enforcement, and ensure implementation workers dont continue work on a mergable PR.
Branch: master

Background and Context

The specification states:

"The tool registry must validate tool inputs/outputs using JSON Schema (jsonschema>=4.20.0)"

In src/cleveragents/domain/models/core/tool.py, the Tool model stores input_schema and output_schema as raw dict[str, Any] | None fields:

input_schema: dict[str, Any] | None = Field(
    default=None, description="JSON Schema for tool inputs"
)
output_schema: dict[str, Any] | None = Field(
    default=None, description="JSON Schema for tool outputs"
)

In Tool.from_config(), these fields are passed through without any JSON Schema meta-schema validation:

return cls(
    ...
    input_schema=config.get("input_schema"),
    output_schema=config.get("output_schema"),
    ...
)

Similarly, in src/cleveragents/cli/commands/tool.py, the add() command calls Tool.from_config(config_dict) and then service.register_tool(tool) without any JSON Schema validation of the schema fields.

This means a tool can be registered with an invalid input_schema or output_schema (e.g., {"type": "invalid_type"} or a non-object value), which will only fail at execution time rather than at registration time.

Current Behavior

Tools can be registered with syntactically invalid or semantically incorrect input_schema and output_schema values. No validation against the JSON Schema meta-schema (Draft 4, 7, or 2020-12) is performed at registration time. Invalid schemas are silently stored and only cause errors at tool execution time.

Expected Behavior

Per the specification:

When input_schema or output_schema is provided, it must be validated against the JSON Schema meta-schema using jsonschema>=4.20.0 at registration time.
If the schema is invalid, Tool.from_config() or ToolRegistryService.register_tool() must raise a ValidationError with a clear message identifying which schema is invalid and why.
Valid JSON Schema documents (including empty {} which matches anything) must be accepted.

Acceptance Criteria

Tool.from_config() validates input_schema and output_schema against the JSON Schema meta-schema using jsonschema>=4.20.0 when these fields are provided.
An invalid input_schema or output_schema raises ValidationError with a descriptive message.
A valid JSON Schema (including {}, {"type": "object"}, etc.) is accepted without error.
None values for input_schema/output_schema are accepted (schema is optional).
Behave BDD scenarios cover: valid schema accepted, invalid schema rejected, None accepted.
nox passes (all sessions).
Coverage remains ≥ 97%.

Subtasks

Add jsonschema>=4.20.0 to project dependencies if not already present
Add a @model_validator or @field_validator in Tool that validates input_schema and output_schema against the JSON Schema meta-schema
Raise ValidationError (from cleveragents.core.exceptions) with a clear message on invalid schemas
Add Behave BDD scenarios for valid schema, invalid schema, and None schema
Run nox and confirm all sessions pass

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message is feat(tool-registry): validate input_schema and output_schema against JSON Schema meta-schema, followed by a blank line, then additional details, and a footer ISSUES CLOSED: #<this issue number>.
The commit is pushed to a branch and submitted as a pull request to master, reviewed, and merged.
All nox stages pass and coverage ≥ 97%.

Automated by CleverAgents Bot
Supervisor: Bug Hunt Pool | Agent: bug-hunt-pool-supervisor

## Metadata - **Commit message**: `Build: Reinforced label enforcement, and ensure implementation workers dont continue work on a mergable PR.` - **Branch**: `master` ## Background and Context The specification states: > "The tool registry must validate tool inputs/outputs using JSON Schema (jsonschema>=4.20.0)" In `src/cleveragents/domain/models/core/tool.py`, the `Tool` model stores `input_schema` and `output_schema` as raw `dict[str, Any] | None` fields: ```python input_schema: dict[str, Any] | None = Field( default=None, description="JSON Schema for tool inputs" ) output_schema: dict[str, Any] | None = Field( default=None, description="JSON Schema for tool outputs" ) ``` In `Tool.from_config()`, these fields are passed through without any JSON Schema meta-schema validation: ```python return cls( ... input_schema=config.get("input_schema"), output_schema=config.get("output_schema"), ... ) ``` Similarly, in `src/cleveragents/cli/commands/tool.py`, the `add()` command calls `Tool.from_config(config_dict)` and then `service.register_tool(tool)` without any JSON Schema validation of the schema fields. This means a tool can be registered with an invalid `input_schema` or `output_schema` (e.g., `{"type": "invalid_type"}` or a non-object value), which will only fail at execution time rather than at registration time. ## Current Behavior Tools can be registered with syntactically invalid or semantically incorrect `input_schema` and `output_schema` values. No validation against the JSON Schema meta-schema (Draft 4, 7, or 2020-12) is performed at registration time. Invalid schemas are silently stored and only cause errors at tool execution time. ## Expected Behavior Per the specification: - When `input_schema` or `output_schema` is provided, it must be validated against the JSON Schema meta-schema using `jsonschema>=4.20.0` at registration time. - If the schema is invalid, `Tool.from_config()` or `ToolRegistryService.register_tool()` must raise a `ValidationError` with a clear message identifying which schema is invalid and why. - Valid JSON Schema documents (including empty `{}` which matches anything) must be accepted. ## Acceptance Criteria - [ ] `Tool.from_config()` validates `input_schema` and `output_schema` against the JSON Schema meta-schema using `jsonschema>=4.20.0` when these fields are provided. - [ ] An invalid `input_schema` or `output_schema` raises `ValidationError` with a descriptive message. - [ ] A valid JSON Schema (including `{}`, `{"type": "object"}`, etc.) is accepted without error. - [ ] `None` values for `input_schema`/`output_schema` are accepted (schema is optional). - [ ] Behave BDD scenarios cover: valid schema accepted, invalid schema rejected, None accepted. - [ ] `nox` passes (all sessions). - [ ] Coverage remains ≥ 97%. ## Subtasks - [ ] Add `jsonschema>=4.20.0` to project dependencies if not already present - [ ] Add a `@model_validator` or `@field_validator` in `Tool` that validates `input_schema` and `output_schema` against the JSON Schema meta-schema - [ ] Raise `ValidationError` (from `cleveragents.core.exceptions`) with a clear message on invalid schemas - [ ] Add Behave BDD scenarios for valid schema, invalid schema, and None schema - [ ] Run `nox` and confirm all sessions pass ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message is `feat(tool-registry): validate input_schema and output_schema against JSON Schema meta-schema`, followed by a blank line, then additional details, and a footer `ISSUES CLOSED: #<this issue number>`. - The commit is pushed to a branch and submitted as a pull request to `master`, reviewed, and merged. - All nox stages pass and coverage ≥ 97%. --- **Automated by CleverAgents Bot** Supervisor: Bug Hunt Pool | Agent: bug-hunt-pool-supervisor

HAL9000 added the

labels

2026-04-13 19:10:31 +00:00

HAL9000 added this to the v3.6.0 milestone

2026-04-13 19:14:34 +00:00

HAL9000 referenced this issue

2026-04-13 19:16:26 +00:00

[AUTO-SPEC] Status: Spec Update Supervisor — Cycle 1 #8411

HAL9000 referenced this issue

2026-04-13 20:04:10 +00:00

[AUTO-ARCH] Status: Cycle 2 — Architecture Supervision #8472

HAL9000 added

and removed

labels

2026-04-13 20:37:13 +00:00

HAL9000 commented

2026-04-13 20:37:49 +00:00

Author

Owner

[AUTO-OWNR-5] Triage Decision

Status: ✅ Verified

MoSCoW: Should Have
Priority: Medium

Rationale: The specification explicitly requires that the tool registry validate input_schema and output_schema using jsonschema>=4.20.0. Currently, invalid schemas are silently stored and only fail at execution time, which is a spec violation and a correctness issue. This is a real bug that degrades developer experience and can cause hard-to-diagnose runtime failures. Classified as Should Have for v3.6.0 because it is an important correctness improvement but does not block core agent execution features — invalid schemas would only surface when a tool with a bad schema is actually invoked.

Next Steps: Add a @model_validator or @field_validator in the Tool model to validate input_schema and output_schema against the JSON Schema meta-schema using jsonschema>=4.20.0. Raise ValidationError with a descriptive message on invalid schemas. Add Behave BDD scenarios covering valid schema, invalid schema, and None schema. Run nox and confirm all sessions pass with coverage ≥ 97%.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

## [AUTO-OWNR-5] Triage Decision **Status**: ✅ Verified **MoSCoW**: Should Have **Priority**: Medium **Rationale**: The specification explicitly requires that the tool registry validate `input_schema` and `output_schema` using `jsonschema>=4.20.0`. Currently, invalid schemas are silently stored and only fail at execution time, which is a spec violation and a correctness issue. This is a real bug that degrades developer experience and can cause hard-to-diagnose runtime failures. Classified as **Should Have** for v3.6.0 because it is an important correctness improvement but does not block core agent execution features — invalid schemas would only surface when a tool with a bad schema is actually invoked. **Next Steps**: Add a `@model_validator` or `@field_validator` in the `Tool` model to validate `input_schema` and `output_schema` against the JSON Schema meta-schema using `jsonschema>=4.20.0`. Raise `ValidationError` with a descriptive message on invalid schemas. Add Behave BDD scenarios covering valid schema, invalid schema, and `None` schema. Run `nox` and confirm all sessions pass with coverage ≥ 97%. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 referenced this issue

2026-04-18 21:27:14 +00:00

Implementation Pool Supervisor - Cycle 24 Status #10628