UAT: ToolCallRouter._check_is_validation uses name heuristic instead of tool_type field — validation tools with non-"valid" names are not detected #3715

New issue

Open

opened 2026-04-05 22:17:19 +00:00 by freemo · 0 comments

freemo commented

2026-04-05 22:17:19 +00:00

Owner

Metadata

Branch: fix/tool-router-check-is-validation
Commit Message: fix(tool): use tool_type field instead of name heuristic in ToolCallRouter._check_is_validation
Milestone: Backlog
Parent Epic: #397

Background and Context

ToolCallRouter._check_is_validation() in src/cleveragents/tool/router.py uses a name-based heuristic to determine if a ToolSpec represents a validation tool:

@staticmethod
def _check_is_validation(spec: ToolSpec) -> bool:
    """Check if a ToolSpec represents a validation tool.

    Uses the metadata dict to determine if tool_type is 'validation'.
    """
    # Check capabilities for validation pattern: read_only=True, writes=False
    # Also check if the name contains 'validation' pattern
    cap = spec.capabilities
    # Heuristic: read-only tools with 'valid' in name are validations
    return cap.read_only and not cap.writes and "valid" in spec.name.lower()

However, ToolSpec already has a dedicated tool_type: Literal["tool", "validation"] field (defined in src/cleveragents/tool/runtime.py) that explicitly discriminates between tools and validations. The heuristic is incorrect per spec.

Current Behavior

The heuristic cap.read_only and not cap.writes and "valid" in spec.name.lower() produces two classes of errors:

False Negatives — Validation tools NOT detected as validations:

core/schema-check — no "valid" in name → is_validation=False (wrong)
core/lint-check — no "valid" in name → is_validation=False (wrong)
core/coverage-gate — no "valid" in name → is_validation=False (wrong)

False Positives — Regular read-only tools incorrectly detected as validations:

core/validate-path — "valid" in name, read-only → is_validation=True (wrong if it's a regular tool)

Impact: When is_validation=False for an actual validation tool:

NormalizedToolCallResult.is_validation is False (incorrect)
NormalizedToolCallResult.validation_passed is None (incorrect — should reflect pass/fail)
NormalizedToolCallResult.validation_mode is None (incorrect — should be "required" or "informational")
Callers that check result.is_validation to decide whether to block execution will not block on validation failures

Expected Behavior

_check_is_validation should use the tool_type field:

@staticmethod
def _check_is_validation(spec: ToolSpec) -> bool:
    return spec.tool_type == "validation"

This is consistent with the spec's discriminator pattern and the ToolType.VALIDATION enum value.

Code Locations

src/cleveragents/tool/router.py lines ~895-901 (_check_is_validation method)
src/cleveragents/tool/runtime.py line ~50 (tool_type: Literal["tool", "validation"] field on ToolSpec)

Acceptance Criteria

ToolCallRouter._check_is_validation() uses spec.tool_type == "validation" instead of the name heuristic
Validation tools with names not containing "valid" (e.g., core/schema-check) are correctly identified as validations
Regular read-only tools with "valid" in the name are not falsely identified as validations
NormalizedToolCallResult.is_validation, validation_passed, and validation_mode are correctly populated for all validation tools
Existing tests for ToolCallRouter pass
New tests cover validation detection for tools with names not containing "valid"

Subtasks

Update _check_is_validation() in src/cleveragents/tool/router.py to use spec.tool_type == "validation"
Add Behave scenarios for validation detection with non-"valid" tool names
Verify nox -e unit_tests passes
Verify nox -e typecheck passes

Definition of Done

All subtasks completed
A Git commit is created with the exact first line: fix(tool): use tool_type field instead of name heuristic in ToolCallRouter._check_is_validation
The commit is pushed to branch fix/tool-router-check-is-validation
A Pull Request has been submitted, reviewed, and merged
All nox stages pass
Coverage >= 97%

Backlog note: This issue was discovered during autonomous UAT testing.
It does not block milestone completion and has been placed in the backlog
for human review and future milestone assignment.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Metadata - **Branch**: `fix/tool-router-check-is-validation` - **Commit Message**: `fix(tool): use tool_type field instead of name heuristic in ToolCallRouter._check_is_validation` - **Milestone**: Backlog - **Parent Epic**: #397 ## Background and Context `ToolCallRouter._check_is_validation()` in `src/cleveragents/tool/router.py` uses a name-based heuristic to determine if a `ToolSpec` represents a validation tool: ```python @staticmethod def _check_is_validation(spec: ToolSpec) -> bool: """Check if a ToolSpec represents a validation tool. Uses the metadata dict to determine if tool_type is 'validation'. """ # Check capabilities for validation pattern: read_only=True, writes=False # Also check if the name contains 'validation' pattern cap = spec.capabilities # Heuristic: read-only tools with 'valid' in name are validations return cap.read_only and not cap.writes and "valid" in spec.name.lower() ``` However, `ToolSpec` already has a dedicated `tool_type: Literal["tool", "validation"]` field (defined in `src/cleveragents/tool/runtime.py`) that explicitly discriminates between tools and validations. The heuristic is incorrect per spec. ## Current Behavior The heuristic `cap.read_only and not cap.writes and "valid" in spec.name.lower()` produces two classes of errors: **False Negatives** — Validation tools NOT detected as validations: - `core/schema-check` — no "valid" in name → `is_validation=False` (wrong) - `core/lint-check` — no "valid" in name → `is_validation=False` (wrong) - `core/coverage-gate` — no "valid" in name → `is_validation=False` (wrong) **False Positives** — Regular read-only tools incorrectly detected as validations: - `core/validate-path` — "valid" in name, read-only → `is_validation=True` (wrong if it's a regular tool) **Impact**: When `is_validation=False` for an actual validation tool: - `NormalizedToolCallResult.is_validation` is `False` (incorrect) - `NormalizedToolCallResult.validation_passed` is `None` (incorrect — should reflect pass/fail) - `NormalizedToolCallResult.validation_mode` is `None` (incorrect — should be "required" or "informational") - Callers that check `result.is_validation` to decide whether to block execution will not block on validation failures ## Expected Behavior `_check_is_validation` should use the `tool_type` field: ```python @staticmethod def _check_is_validation(spec: ToolSpec) -> bool: return spec.tool_type == "validation" ``` This is consistent with the spec's discriminator pattern and the `ToolType.VALIDATION` enum value. ## Code Locations - `src/cleveragents/tool/router.py` lines ~895-901 (`_check_is_validation` method) - `src/cleveragents/tool/runtime.py` line ~50 (`tool_type: Literal["tool", "validation"]` field on `ToolSpec`) ## Acceptance Criteria - [ ] `ToolCallRouter._check_is_validation()` uses `spec.tool_type == "validation"` instead of the name heuristic - [ ] Validation tools with names not containing "valid" (e.g., `core/schema-check`) are correctly identified as validations - [ ] Regular read-only tools with "valid" in the name are not falsely identified as validations - [ ] `NormalizedToolCallResult.is_validation`, `validation_passed`, and `validation_mode` are correctly populated for all validation tools - [ ] Existing tests for `ToolCallRouter` pass - [ ] New tests cover validation detection for tools with names not containing "valid" ## Subtasks - [ ] Update `_check_is_validation()` in `src/cleveragents/tool/router.py` to use `spec.tool_type == "validation"` - [ ] Add Behave scenarios for validation detection with non-"valid" tool names - [ ] Verify `nox -e unit_tests` passes - [ ] Verify `nox -e typecheck` passes ## Definition of Done - [ ] All subtasks completed - [ ] A Git commit is created with the exact first line: `fix(tool): use tool_type field instead of name heuristic in ToolCallRouter._check_is_validation` - [ ] The commit is pushed to branch `fix/tool-router-check-is-validation` - [ ] A Pull Request has been submitted, reviewed, and merged - [ ] All nox stages pass - [ ] Coverage >= 97% > **Backlog note:** This issue was discovered during autonomous UAT testing. > It does not block milestone completion and has been placed in the backlog > for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo referenced this issue

2026-04-05 22:18:21 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 (Resumed) #3654

freemo referenced this issue

2026-04-05 22:20:24 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 (Resumed) #3654

freemo referenced this issue

2026-04-05 22:20:38 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 (Resumed) #3654

freemo referenced this issue

2026-04-05 22:22:14 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 (Resumed) #3654