UAT: `ToolCallRouter._check_is_validation()` uses name-based heuristic — any read-only tool with "valid" in its name is misclassified as a validation tool #6129

New issue

Open

opened 2026-04-09 15:18:40 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-09 15:18:40 +00:00

Owner

Bug Report

Feature Area: Tool Configuration / Actor Routing
Milestone Scope: v3.4.0 (tool routing)
Severity: Non-critical (incorrect metadata, potential false positives)

What Was Tested

The ToolCallRouter._check_is_validation() method in src/cleveragents/tool/router.py and how it classifies tools as validation tools.

Expected Behavior (from spec)

Validation tools should be identified by their explicit tool_type metadata (e.g., ToolType.validation from the domain model), not by a name-based heuristic. The spec defines validation tools as tools registered with tool_type="validation" via agents tool add.

Actual Behavior

The _check_is_validation() static method uses a heuristic that combines capability flags AND name matching:

@staticmethod
def _check_is_validation(spec: ToolSpec) -> bool:
    """Check if a ToolSpec represents a validation tool."""
    cap = spec.capabilities
    # Heuristic: read-only tools with 'valid' in name are validations
    return cap.read_only and not cap.writes and "valid" in spec.name.lower()

Problems with this heuristic:

False positives: Any read-only tool with "valid" in its name (e.g., local/validate-schema, local/is-valid-json, mcp/validate-input) will be classified as a validation tool, even if it was registered as a regular tool.
False negatives: A validation tool named local/check-invariants (no "valid" in name) will NOT be classified as a validation tool even if it was explicitly registered with tool_type="validation".
Inconsistency: The ToolType enum in src/cleveragents/domain/models/core/tool.py has a validation value, and agents tool add supports registering tools with tool_type: validation in YAML. But ToolCallRouter ignores this explicit metadata and uses name-based inference instead.

Steps to Reproduce

# my-validator.yaml
name: local/validate-input
description: Validates user input format
source: custom
code: |
  def run(inputs):
      return {"valid": True}

agents tool add --config my-validator.yaml

When this tool is called via ToolCallRouter.route(), is_validation=True will be set in the result, and validation_passed will be read from the output's "passed" key — even though this is a regular tool, not a validation tool.

Code Location

src/cleveragents/tool/router.py — ToolCallRouter._check_is_validation() static method.

Fix

Use the explicit tool_type metadata from the ToolSpec instead of name-based heuristics:

@staticmethod
def _check_is_validation(spec: ToolSpec) -> bool:
    """Check if a ToolSpec represents a validation tool."""
    # Use explicit tool_type metadata, not name heuristics
    return getattr(spec, 'tool_type', None) == 'validation'

Or check the source metadata if tool_type is stored there.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: Tool Configuration / Actor Routing **Milestone Scope**: v3.4.0 (tool routing) **Severity**: Non-critical (incorrect metadata, potential false positives) --- ## What Was Tested The `ToolCallRouter._check_is_validation()` method in `src/cleveragents/tool/router.py` and how it classifies tools as validation tools. ## Expected Behavior (from spec) Validation tools should be identified by their explicit `tool_type` metadata (e.g., `ToolType.validation` from the domain model), not by a name-based heuristic. The spec defines validation tools as tools registered with `tool_type="validation"` via `agents tool add`. ## Actual Behavior The `_check_is_validation()` static method uses a heuristic that combines capability flags AND name matching: ```python @staticmethod def _check_is_validation(spec: ToolSpec) -> bool: """Check if a ToolSpec represents a validation tool.""" cap = spec.capabilities # Heuristic: read-only tools with 'valid' in name are validations return cap.read_only and not cap.writes and "valid" in spec.name.lower() ``` **Problems with this heuristic:** 1. **False positives**: Any read-only tool with "valid" in its name (e.g., `local/validate-schema`, `local/is-valid-json`, `mcp/validate-input`) will be classified as a validation tool, even if it was registered as a regular tool. 2. **False negatives**: A validation tool named `local/check-invariants` (no "valid" in name) will NOT be classified as a validation tool even if it was explicitly registered with `tool_type="validation"`. 3. **Inconsistency**: The `ToolType` enum in `src/cleveragents/domain/models/core/tool.py` has a `validation` value, and `agents tool add` supports registering tools with `tool_type: validation` in YAML. But `ToolCallRouter` ignores this explicit metadata and uses name-based inference instead. ## Steps to Reproduce Register a read-only tool with "valid" in the name: ```yaml # my-validator.yaml name: local/validate-input description: Validates user input format source: custom code: | def run(inputs): return {"valid": True} ``` ```bash agents tool add --config my-validator.yaml ``` When this tool is called via `ToolCallRouter.route()`, `is_validation=True` will be set in the result, and `validation_passed` will be read from the output's `"passed"` key — even though this is a regular tool, not a validation tool. ## Code Location `src/cleveragents/tool/router.py` — `ToolCallRouter._check_is_validation()` static method. ## Fix Use the explicit `tool_type` metadata from the `ToolSpec` instead of name-based heuristics: ```python @staticmethod def _check_is_validation(spec: ToolSpec) -> bool: """Check if a ToolSpec represents a validation tool.""" # Use explicit tool_type metadata, not name heuristics return getattr(spec, 'tool_type', None) == 'validation' ``` Or check the source metadata if `tool_type` is stored there. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 15:26:47 +00:00

HAL9000 referenced this issue

2026-04-09 15:39:15 +00:00

[AUTO-UAT-POOL] UAT Testing Report (Cycle 1) #5750

HAL9000 added the

State