cleveragents-core/docs/showcase/cli-tools/tool-and-validation-management.md

Managing Tools and Validations in CleverAgents

Overview

CleverAgents provides a unified tool registry where you can register, inspect, and manage two kinds of callable operations: tools (general-purpose functions) and validations (pass/fail quality gates). This example walks through the complete lifecycle — registering tools from YAML config files, listing them with filters, inspecting details, and removing them — all from the CLI.

Prerequisites

• CleverAgents installed (pip install cleveragents)
• Python 3.13 or higher

What You'll Learn

• How to register a custom tool from a YAML configuration file
• How to register validations — both standalone (custom code) and wrapped (delegating to an existing tool)
• How to list all tools and validations with optional type/namespace filters
• How to inspect a specific tool to see its capability flags, resource slots, and validation mode
• How to remove a tool from the registry
• How to use --format json for machine-readable output in scripts

---

Step-by-Step Walkthrough

Step 1: Register a custom tool from YAML

Create a YAML file describing your tool (see examples/tools/custom-tool.yaml for the full schema):

# examples/tools/custom-tool.yaml
name: local/line-counter
description: Count lines in a file
source: custom
code: |
  import pathlib
  path = pathlib.Path(inputs["file_path"])
  count = len(path.read_text().splitlines())
  return {"line_count": count}

input_schema:
  type: object
  required:
    - file_path
  properties:
    file_path:
      type: string
      description: Path to the file to count lines in

output_schema:
  type: object
  properties:
    line_count:
      type: integer

capability:
  read_only: true
  writes: false
  idempotent: true

resource_slots:
  - name: target_file
    resource_type: fs-mount
    access: read_only
    description: File system containing the target file
    binding: contextual

timeout: 60

Register it with:

$ agents tool add --config examples/tools/custom-tool.yaml

Expected Output:

╭─────────── Tool Registered ───────────╮
│ Name: local/line-counter              │
│ Namespace: local                      │
│ Short Name: line-counter              │
│ Description: Count lines in a file    │
│ Source: custom                        │
│ Type: tool                            │
│ Timeout: 60s                          │
│ Capability:                           │
│   read_only: True                     │
│   writes: False                       │
│   checkpointable: False               │
│   idempotent: True                    │
│   unsafe: False                       │
│   human_approval_required: False      │
│ Resource Slots:                       │
│   - target_file (fs-mount, read_only) │
│ Lifecycle:                            │
│   (none)                              │
╰───────────────────────────────────────╯

What's Happening:

The tool add command reads the YAML file, validates the schema, and persists the tool definition to the local tool registry (SQLite database). The capability block is especially important — it tells the plan executor whether the tool is safe to run without human approval, whether it can be checkpointed, and whether it has side effects.

The namespaced name local/line-counter uses the local namespace, which is the conventional namespace for project-specific tools. Built-in tools use provider namespaces like devops/ or qa/.

---

Step 2: Register a required validation (custom code)

Validations are pass/fail quality gates. A required validation must pass before a plan step can proceed. Register one from YAML:

# examples/validations/required-validation.yaml
name: qa/coverage-check
description: Verify test coverage meets the required threshold
source: custom
mode: required
code: |
  import json
  report = json.loads(inputs["coverage_json"])
  total = report.get("totals", {}).get("percent_covered", 0)
  passed = total >= inputs.get("threshold", 80)
  return {
    "passed": passed,
    "data": {"coverage_percent": total},
    "message": f"Coverage {total}% {'meets' if passed else 'below'} threshold"
  }

$ agents validation add --config examples/validations/required-validation.yaml

Expected Output:

╭──────────────────── Validation Registered ─────────────────────╮
│ Name: qa/coverage-check                                        │
│ Description: Verify test coverage meets the required threshold │
│ Source: custom                                                 │
│ Mode: required                                                 │
╰────────────────────────────────────────────────────────────────╯

What's Happening:

The validation is stored in the same tool registry as regular tools, but with tool_type = validation. The mode: required field means this validation blocks plan execution if it fails. The inline code block is a Python snippet that receives inputs and must return a dict with a passed boolean.

---

Step 3: Register a wrapped validation

A wrapped validation delegates execution to an existing tool and transforms its output into a pass/fail result:

# examples/validations/wrapped-validation.yaml
name: qa/lint-check
description: Validate that linting passes by wrapping the lint tool
wraps: devops/run-linter
mode: required
transform: |
  def transform(result):
    errors = result.get("errors", [])
    passed = len(errors) == 0
    return {
      "passed": passed,
      "data": {"error_count": len(errors), "errors": errors[:5]},
      "message": f"{len(errors)} lint errors found" if errors else "No lint errors"
    }

$ agents validation add --config examples/validations/wrapped-validation.yaml

Expected Output:

╭─────────────────────────── Validation Registered ────────────────────────────╮
│ Name: qa/lint-check                                                          │
│ Description: Validate that linting passes by wrapping the lint tool          │
│ Source: wrapped                                                              │
│ Mode: required                                                               │
│ Wraps: devops/run-linter                                                     │
│ Transform: def transform(result):                                            │
│   errors = result.get("errors", [])                                          │
│   passed = len(errors) == 0                                                  │
│   return {                                                                   │
│     "passed": passed,                                                        │
│     "data": {"error_count": len(errors), "errors": errors[:5]},              │
│     "message": f"{len(errors)} lint errors found" if errors else "No lint    │
│ errors"                                                                      │
│   }                                                                          │
╰──────────────────────────────────────────────────────────────────────────────╯

What's Happening:

The wraps field points to an existing tool (devops/run-linter). When the validation runs, CleverAgents calls that tool and passes the result through the transform function. This lets you reuse existing tools as quality gates without duplicating logic.

---

Step 4: List all tools and validations

$ agents tool list

Expected Output:

                                Tools (3 total)
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name              ┃ Type       ┃ Source  ┃ Description             ┃ Timeout ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ local/line-counter│ tool       │ custom  │ Count lines in a file   │      60 │
│ qa/coverage-check │ validation │ custom  │ Verify test coverage    │     120 │
│                   │            │         │ meets the requir...     │         │
│ qa/lint-check     │ validation │ wrapped │ Validate that linting   │     180 │
│                   │            │         │ passes by wrapp...      │         │
└───────────────────┴────────────┴─────────┴─────────────────────────┴─────────┘

What's Happening:

tool list shows both tools and validations in a single unified view. The Type column distinguishes them. The Source column shows how each was defined: custom (inline Python code), wrapped (delegates to another tool), or mcp (delegates to an MCP server tool).

---

Step 5: Filter by type

Show only validations:

$ agents tool list --type validation

Expected Output:

                                Tools (2 total)
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name              ┃ Type       ┃ Source  ┃ Description             ┃ Timeout ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ qa/coverage-check │ validation │ custom  │ Verify test coverage    │     120 │
│                   │            │         │ meets the requir...     │         │
│ qa/lint-check     │ validation │ wrapped │ Validate that linting   │     180 │
│                   │            │         │ passes by wrapp...      │         │
└───────────────────┴────────────┴─────────┴─────────────────────────┴─────────┘

Show only tools (not validations):

$ agents tool list --type tool

Expected Output:

                              Tools (1 total)
┏━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name            ┃ Type ┃ Source ┃ Description                 ┃ Timeout ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ local/line-counter│ tool │ custom │ Count lines in a file     │      60 │
└─────────────────┴──────┴────────┴─────────────────────────────┴─────────┘

Filter by namespace:

$ agents tool list --namespace qa

Expected Output:

                                Tools (2 total)
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name              ┃ Type       ┃ Source  ┃ Description             ┃ Timeout ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ qa/coverage-check │ validation │ custom  │ Verify test coverage    │     120 │
│ qa/lint-check     │ validation │ wrapped │ Validate that linting   │     180 │
└───────────────────┴────────────┴─────────┴─────────────────────────┴─────────┘

What's Happening:

The --type flag accepts tool or validation. The --namespace flag filters by the prefix before the / in the tool name. You can also pass a regex pattern as a positional argument: agents tool list "line-.*".

---

Step 6: Inspect a specific tool

$ agents tool show local/line-counter

Expected Output:

╭─────────── Tool Details ───────────╮
│ Name: local/line-counter           │
│ Namespace: local                   │
│ Short Name: line-counter           │
│ Description: Count lines in a file │
│ Source: custom                     │
│ Type: tool                         │
│ Timeout: 60s                       │
│ Capability:                        │
│   (default)                        │
│ Resource Slots:                    │
│   (none)                           │
│ Lifecycle:                         │
│   (none)                           │
╰────────────────────────────────────╯

Inspect a validation to see its mode:

$ agents tool show qa/coverage-check

Expected Output:

╭───────────────────────── Tool Details ─────────────────────────╮
│ Name: qa/coverage-check                                        │
│ Namespace: qa                                                  │
│ Short Name: coverage-check                                     │
│ Description: Verify test coverage meets the required threshold │
│ Source: custom                                                 │
│ Type: validation                                               │
│ Timeout: 120s                                                  │
│ Capability:                                                    │
│   (default)                                                    │
│ Resource Slots:                                                │
│   (none)                                                       │
│ Lifecycle:                                                     │
│   (none)                                                       │
│ Validation Mode: required                                      │
╰────────────────────────────────────────────────────────────────╯

What's Happening:

tool show works for both tools and validations — use the full namespaced name (namespace/short-name). For validations, the panel includes an extra Validation Mode field showing required or informational.

Why do capabilities show as (default)? The detail view collapses the default capability flags and omits resource slot bindings when they match the defaults stored in the registry. The values you set during registration are still persisted and enforced when the tool runs, even though the summary view shows (default) / (none).

---

Step 7: Get machine-readable JSON output

$ agents tool list --format json

Expected Output:

{
  "command": "",
  "status": "ok",
  "exit_code": 0,
  "data": [
    {
      "name": "local/line-counter",
      "description": "Count lines in a file",
      "source": "custom",
      "tool_type": "tool",
      "namespace": "local",
      "short_name": "line-counter",
      "capability": {},
      "timeout": 60
    },
    {
      "name": "qa/coverage-check",
      "description": "Verify test coverage meets the required threshold",
      "source": "custom",
      "tool_type": "validation",
      "namespace": "qa",
      "short_name": "coverage-check",
      "capability": {},
      "timeout": 120
    },
    {
      "name": "qa/lint-check",
      "description": "Validate that linting passes by wrapping the lint tool",
      "source": "wrapped",
      "tool_type": "validation",
      "namespace": "qa",
      "short_name": "lint-check",
      "capability": {},
      "timeout": 180
    }
  ],
  "timing": {
    "duration_ms": 0
  },
  "messages": [
    {
      "level": "ok",
      "text": "ok"
    }
  ]
}

Show a single tool in JSON:

$ agents tool show local/line-counter --format json

Expected Output:

{
  "command": "",
  "status": "ok",
  "exit_code": 0,
  "data": {
    "name": "local/line-counter",
    "description": "Count lines in a file",
    "source": "custom",
    "tool_type": "tool",
    "namespace": "local",
    "short_name": "line-counter",
    "capability": {},
    "timeout": 60
  },
  "timing": {
    "duration_ms": 0
  },
  "messages": [
    {
      "level": "ok",
      "text": "ok"
    }
  ]
}

What's Happening:

All tool commands support --format json (or -f json). The response uses the standard CleverAgents JSON envelope: data holds the payload, status is "ok" on success, and timing.duration_ms shows elapsed time. For list commands, data is an array; for show, data is a single object.

---

Step 8: Update an existing tool or validation

If you modify the YAML and want to re-register without removing first:

$ agents validation add --config examples/validations/required-validation.yaml --update

Expected Output:

╭──────────────────── Validation Updated ────────────────────────╮
│ Name: qa/coverage-check                                        │
│ Description: Verify test coverage meets the required threshold │
│ Source: custom                                                 │
│ Mode: required                                                 │
╰────────────────────────────────────────────────────────────────╯

What's Happening:

Without --update, re-registering an existing name raises an error. The --update flag performs an upsert — it updates the existing record in place. The same flag is available on tool add --update.

---

Step 9: Remove a tool

$ agents tool remove --yes local/line-counter

Expected Output:

Removed tool: local/line-counter

What's Happening:

tool remove soft-deletes the tool from the registry. The --yes flag skips the interactive confirmation prompt, which is useful in scripts. Without --yes, the CLI will ask you to confirm before deleting.

---

Scripting with JSON Output

Because all commands support --format json, you can integrate tool management into CI/CD pipelines:

# Check if a specific tool is registered
$ agents tool list --format json | jq -r '.data[] | select(.name == "qa/coverage-check") | .name'
qa/coverage-check

# List all validation names
$ agents tool list --type validation --format json | jq -r '.data[].name'
qa/coverage-check
qa/lint-check

# Check the timeout for a tool
$ agents tool show local/line-counter --format json | jq '.data.timeout'
60

# Count registered tools
$ agents tool list --format json | jq '.data | length'
3

# List all tools in the qa namespace
$ agents tool list --namespace qa --format json | jq -r '.data[].name'
qa/coverage-check
qa/lint-check

---

Tool vs Validation: When to Use Each

Aspect	Tool	Validation
Purpose	General-purpose callable operation	Pass/fail quality gate
Return value	Any structured data	Must include passed: bool
Modes	N/A	required (blocks) or informational (advisory)
Source types	custom, mcp	custom, wrapped
Used in plans	As action steps	As pre/post conditions
Namespace convention	local/, devops/	qa/

---

Complete Interaction Log

Click to see the full verified command sequence

# 1. Register a custom tool
$ agents tool add --config examples/tools/custom-tool.yaml
╭─────────── Tool Registered ───────────╮
│ Name: local/line-counter              │
│ Source: custom                        │
│ Type: tool                            │
│ Timeout: 60s                          │
│ Capability: read_only: True, ...      │
╰───────────────────────────────────────╯

# 2. Register a required validation (custom code)
$ agents validation add --config examples/validations/required-validation.yaml
╭──────────────────── Validation Registered ─────────────────────╮
│ Name: qa/coverage-check                                        │
│ Mode: required                                                 │
╰────────────────────────────────────────────────────────────────╯

# 3. Register a wrapped validation
$ agents validation add --config examples/validations/wrapped-validation.yaml
╭─────────────────────────── Validation Registered ────────────────────────────╮
│ Name: qa/lint-check                                                          │
│ Source: wrapped                                                              │
│ Wraps: devops/run-linter                                                     │
╰──────────────────────────────────────────────────────────────────────────────╯

# 4. List all (tools + validations unified)
$ agents tool list
                                Tools (3 total)
┃ Name              ┃ Type       ┃ Source  ┃ Description             ┃ Timeout ┃
│ local/line-counter│ tool       │ custom  │ Count lines in a file   │      60 │
│ qa/coverage-check │ validation │ custom  │ Verify test coverage... │     120 │
│ qa/lint-check     │ validation │ wrapped │ Validate that linting...│     180 │

# 5. Filter by type
$ agents tool list --type validation
# → shows only qa/coverage-check and qa/lint-check

$ agents tool list --type tool
# → shows only local/line-counter

# 6. Filter by namespace
$ agents tool list --namespace qa
# → shows qa/coverage-check and qa/lint-check

# 7. Inspect a tool
$ agents tool show local/line-counter
# → shows capability flags, resource slots, lifecycle

# 8. Inspect a validation (shows Validation Mode)
$ agents tool show qa/coverage-check
# → shows Validation Mode: required

# 9. JSON output for scripting
$ agents tool list --format json
# → {"status": "ok", "data": [...], "timing": {...}}

$ agents tool show local/line-counter --format json
# → {"status": "ok", "data": {...}, "timing": {...}}

# 10. Update without removing
$ agents validation add --config examples/validations/required-validation.yaml --update
# → Validation Updated panel

# 11. Remove a tool
$ agents tool remove --yes local/line-counter
# → Removed tool: local/line-counter

---

Key Takeaways

• Tools and validations share a unified registry — tool list shows both; use --type tool or --type validation to filter.
• Namespaced names (namespace/short-name) uniquely identify every tool. Use local/ for project-specific tools and qa/ for quality gates.
• Validations have a mode — required blocks plan execution on failure; informational is advisory only.
• Wrapped validations reuse existing tools as quality gates via a transform function — no code duplication needed.
• --update flag enables safe re-registration when you modify a YAML config without needing to remove and re-add.
• --format json works on all tool commands — use it to integrate tool management into CI/CD scripts with jq.

Try It Yourself

Now that you've seen the full tool and validation lifecycle, try these variations:

• Register the MCP tool example: agents tool add --config examples/tools/mcp-tool.yaml
• Filter by source: agents tool list --source wrapped
• Regex filter: agents tool list "qa/.*"
• CI check: agents tool list --format json | jq '.data | map(select(.tool_type == "validation")) | length'
• Override mode at registration: agents validation add --config examples/validations/required-validation.yaml --informational

Related Examples

• See cli-tools/output-format-flags.md for a deep dive into all six output formats (json, yaml, plain, table, rich, color)
• See docs/showcase/cli-tools/ for more CLI tool examples

---

This example was automatically generated and verified by the CleverAgents UAT system. Feature area: Tool and validation management | Test cycle: 1 | Generated: 2026-04-07