cleveragents/cleveragents-core

Fork 3

UAT: `agents validation attach` accepts plain tools (tool_type="tool") — spec requires rejection #3482

New issue

Open

opened 2026-04-05 18:32:29 +00:00 by freemo · 1 comment

freemo commented

2026-04-05 18:32:29 +00:00

Owner

Metadata

Branch: fix/m3-validation-attach-tool-type-check
Commit Message: fix(validation): enforce tool_type check in attach_validation to reject plain tools
Milestone: v3.2.0
Parent Epic: #357

Background and Context

The specification (docs/specification.md line 22322) explicitly requires that agents validation attach rejects any name that resolves to a plain Tool (tool_type="tool") rather than a Validation (tool_type="validation"). This type safety guarantee is enforced by the type discriminator stored in the Tool Registry, where each entry is tagged as either tool or validation.

"a plain Tool cannot be used where a Validation is specifically required. For example, agents validation attach rejects a name that resolves to a plain Tool rather than a Validation. This distinction is enforced by the type discriminator stored in the Tool Registry: each entry is tagged as either tool or validation."
— docs/specification.md line 22322

Current Behavior

ToolRegistryService.attach_validation() (located in src/cleveragents/application/services/tool_registry_service.py, lines ~155–185) only checks whether the named tool exists (raising NotFoundError if not found). It does not inspect the tool_type field after the existence check. As a result, a plain tool registered with tool_type="tool" can be silently attached as a validation attachment, bypassing the type safety guarantee.

Steps to Reproduce:

Register a plain tool: agents tool add --config plain-tool.yaml (where plain-tool.yaml has tool_type="tool")
Attempt to attach it as a validation: agents validation attach my-resource local/my-plain-tool
Observe: The attachment succeeds instead of failing with an error

Expected Behavior

agents validation attach <RESOURCE> <PLAIN_TOOL_NAME> should fail with a clear error message:

Error: 'local/my-plain-tool' is a plain tool, not a validation. Use `agents validation add` to register a validation first.

Impact

Critical — violates the type safety contract of the validation subsystem. Actors and the plan lifecycle may attempt to run plain tools as validations, expecting a passed boolean return that plain tools don't provide, leading to silent failures or incorrect validation results. The mode field also defaults to "required" since plain tools have no mode attribute, compounding the issue.

Code Location

File: src/cleveragents/application/services/tool_registry_service.py
Method: attach_validation() (lines ~155–185)

Required fix — add a tool_type check immediately after the existence check:

# After: existing = self._tool_repo.get_by_name(validation_name)
# Add:
tool_type = existing.get("tool_type") if isinstance(existing, dict) else getattr(existing, "tool_type", None)
if str(tool_type) != "validation":
    raise ValidationError(
        f"'{validation_name}' is a plain tool, not a validation. "
        "Use `agents validation add` to register a validation."
    )

Subtasks

Add tool_type guard in ToolRegistryService.attach_validation() after existence check
Raise ValidationError (or appropriate domain error) with a clear, user-facing message when tool_type != "validation"
Tests (Behave): Add scenario — agents validation attach with a plain tool name returns an error
Tests (Behave): Add scenario — agents validation attach with a valid validation name still succeeds
Tests (unit): Add unit test for attach_validation() rejecting tool_type="tool" entries
Verify coverage >= 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage >= 97%.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/m3-validation-attach-tool-type-check` - **Commit Message**: `fix(validation): enforce tool_type check in attach_validation to reject plain tools` - **Milestone**: v3.2.0 - **Parent Epic**: #357 ## Background and Context The specification (docs/specification.md line 22322) explicitly requires that `agents validation attach` rejects any name that resolves to a plain Tool (`tool_type="tool"`) rather than a Validation (`tool_type="validation"`). This type safety guarantee is enforced by the `type` discriminator stored in the Tool Registry, where each entry is tagged as either `tool` or `validation`. > "a plain Tool cannot be used where a Validation is specifically required. For example, `agents validation attach` rejects a name that resolves to a plain Tool rather than a Validation. This distinction is enforced by the `type` discriminator stored in the Tool Registry: each entry is tagged as either `tool` or `validation`." > — docs/specification.md line 22322 ## Current Behavior `ToolRegistryService.attach_validation()` (located in `src/cleveragents/application/services/tool_registry_service.py`, lines ~155–185) only checks whether the named tool exists (raising `NotFoundError` if not found). It does **not** inspect the `tool_type` field after the existence check. As a result, a plain tool registered with `tool_type="tool"` can be silently attached as a validation attachment, bypassing the type safety guarantee. **Steps to Reproduce:** 1. Register a plain tool: `agents tool add --config plain-tool.yaml` (where `plain-tool.yaml` has `tool_type="tool"`) 2. Attempt to attach it as a validation: `agents validation attach my-resource local/my-plain-tool` 3. **Observe:** The attachment succeeds instead of failing with an error ## Expected Behavior `agents validation attach <RESOURCE> <PLAIN_TOOL_NAME>` should fail with a clear error message: ``` Error: 'local/my-plain-tool' is a plain tool, not a validation. Use `agents validation add` to register a validation first. ``` ## Impact **Critical** — violates the type safety contract of the validation subsystem. Actors and the plan lifecycle may attempt to run plain tools as validations, expecting a `passed` boolean return that plain tools don't provide, leading to silent failures or incorrect validation results. The `mode` field also defaults to `"required"` since plain tools have no `mode` attribute, compounding the issue. ## Code Location - **File**: `src/cleveragents/application/services/tool_registry_service.py` - **Method**: `attach_validation()` (lines ~155–185) **Required fix** — add a `tool_type` check immediately after the existence check: ```python # After: existing = self._tool_repo.get_by_name(validation_name) # Add: tool_type = existing.get("tool_type") if isinstance(existing, dict) else getattr(existing, "tool_type", None) if str(tool_type) != "validation": raise ValidationError( f"'{validation_name}' is a plain tool, not a validation. " "Use `agents validation add` to register a validation." ) ``` ## Subtasks - [ ] Add `tool_type` guard in `ToolRegistryService.attach_validation()` after existence check - [ ] Raise `ValidationError` (or appropriate domain error) with a clear, user-facing message when `tool_type != "validation"` - [ ] Tests (Behave): Add scenario — `agents validation attach` with a plain tool name returns an error - [ ] Tests (Behave): Add scenario — `agents validation attach` with a valid validation name still succeeds - [ ] Tests (unit): Add unit test for `attach_validation()` rejecting `tool_type="tool"` entries - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage >= 97%. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added this to the v3.2.0 milestone

2026-04-05 18:32:34 +00:00

freemo added a new dependency

2026-04-05 18:32:42 +00:00

#357 Epic: Decisions + Validations + Invariants (M3)

freemo referenced this issue

2026-04-05 18:39:51 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 18:45:07 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 18:52:57 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 19:05:02 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 19:33:25 +00:00

UAT: ValidationPipeline not integrated into plan lifecycle — validations never execute during plan phases #3487

freemo referenced this issue

2026-04-05 19:33:28 +00:00

UAT: ApplyValidationGate uses DefaultValidationRunner stub — apply-phase validation does not execute registered validation tools #3490

freemo commented

2026-04-05 19:33:30 +00:00

Author

Owner

Issue verified and triaged:

Priority: Critical — violates the type safety contract of the validation subsystem. Plain tools can be silently attached as validations.
Milestone: v3.2.0 (already assigned)
Story Points: 2 (S) — small, focused fix: add a tool_type guard in attach_validation().
Parent Epic: #357 (already linked)
Dependency: Part of the validation subsystem fix cluster with #3487 and #3490.
Next step: This issue is now ready for implementation.

Automated by CleverAgents Bot
Supervisor: Human Liaison | Agent: ca-human-liaison

Issue verified and triaged: - **Priority**: Critical — violates the type safety contract of the validation subsystem. Plain tools can be silently attached as validations. - **Milestone**: v3.2.0 (already assigned) - **Story Points**: 2 (S) — small, focused fix: add a tool_type guard in attach_validation(). - **Parent Epic**: #357 (already linked) - **Dependency**: Part of the validation subsystem fix cluster with #3487 and #3490. - **Next step**: This issue is now ready for implementation. --- **Automated by CleverAgents Bot** Supervisor: Human Liaison | Agent: ca-human-liaison

freemo referenced this issue

2026-04-05 19:49:58 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo removed this from the v3.2.0 milestone

2026-04-06 23:48:46 +00:00