cleveragents-core/docs/showcase/cli-tools/tool-and-validation-management.md

Managing Tools and Validations in CleverAgents

Overview

CleverAgents provides a unified tool registry where you can register, inspect, and manage two kinds of callable operations: tools (general-purpose functions) and validations (pass/fail quality gates). This example walks through the complete lifecycle — registering tools from YAML config files, listing them with filters, inspecting details, and removing them — all from the CLI.

Prerequisites

• CleverAgents installed (pip install cleveragents)
• Python 3.13 or higher

What You'll Learn

• How to register a custom tool from a YAML configuration file
• How to register validations — both standalone (custom code) and wrapped (delegating to an existing tool)
• How to list all tools and validations with optional type/namespace filters
• How to inspect a specific tool to see its capability flags, resource slots, and validation mode
• How to remove a tool from the registry
• How to use --format json for machine-readable output in scripts

---

Step-by-Step Walkthrough

Step 1: Register a custom tool from YAML

Create a YAML file describing your tool (see examples/tools/custom-tool.yaml for the full schema):

# examples/tools/custom-tool.yaml
name: local/line-counter
description: Count lines in a file
source: custom
code: |
  import pathlib
  path = pathlib.Path(inputs["file_path"])
  count = len(path.read_text().splitlines())
  return {"line_count": count}

input_schema:
  type: object
  required:
    - file_path
  properties:
    file_path:
      type: string
      description: Path to the file to count lines in

output_schema:
  type: object
  properties:
    line_count:
      type: integer

capability:
  read_only: true
  writes: false
  idempotent: true

resource_slots:
  - name: target_file
    resource_type: fs-mount
    access: read_only
    description: File system containing the target file
    binding: contextual

timeout: 60

Register it with:

$ agents tool add --config examples/tools/custom-tool.yaml

Expected Output:

╭─────────── Tool Registered ───────────╮
│ Name: local/line-counter              │
│ Namespace: local                      │
│ Short Name: line-counter              │
│ Description: Count lines in a file    │
│ Source: custom                        │
│ Type: tool                            │
│ Timeout: 60s                          │
│ Capability:                           │
│   read_only: True                     │
│   writes: False                       │
│   checkpointable: False               │
│   idempotent: True                    │
│   unsafe: False                       │
│   human_approval_required: False      │
│ Resource Slots:                       │
│   - target_file (fs-mount, read_only) │
│ Lifecycle:                            │
│   (none)                              │
╰───────────────────────────────────────╯

What's Happening:

The tool add command reads the YAML file, validates the schema, and persists the tool definition to the local tool registry (SQLite database). The capability block is especially important — it tells the plan executor whether the tool is safe to run without human approval, whether it can be checkpointed, and whether it has side effects.

The namespaced name local/line-counter uses the local namespace, which is the conventional namespace for project-specific tools. Built-in tools use provider namespaces like devops/ or qa/.

---

Step 2: Register a required validation (custom code)

Validations are pass/fail quality gates. A required validation must pass before a plan step can proceed. Register one from YAML:

# examples/validations/required-validation.yaml
name: qa/coverage-check
description: Verify test coverage meets the required threshold
source: custom
mode: required
code: |
  import json
  report = json.loads(inputs["coverage_json"])
  total = report.get("totals", {}).get("percent_covered", 0)
  passed = total >= inputs.get("threshold", 80)
  return {
    "passed": passed,
    "data": {"coverage_percent": total},
    "message": f"Coverage {total}% {'meets' if passed else 'below'} threshold"
  }

$ agents validation add --config examples/validations/required-validation.yaml

Expected Output:

╭──────────────────── Validation Registered ─────────────────────╮
│ Name: qa/coverage-check                                        │
│ Description: Verify test coverage meets the required threshold │
│ Source: custom                                                 │
│ Mode: required                                                 │
╰────────────────────────────────────────────────────────────────╯

What's Happening:

The validation is stored in the same tool registry as regular tools, but with tool_type = validation. The mode: required field means this validation blocks plan execution if it fails. The inline code block is a Python snippet that receives inputs and must return a dict with a passed boolean.

---

Step 3: Register a wrapped validation

A wrapped validation delegates execution to an existing tool and transforms its output into a pass/fail result:

# examples/validations/wrapped-validation.yaml
name: qa/lint-check
description: Validate that linting passes by wrapping the lint tool
wraps: devops/run-linter
mode: required
transform: |
  def transform(result):
    errors = result.get("errors", [])
    passed = len(errors) == 0
    return {
      "passed": passed,
      "data": {"error_count": len(errors), "errors": errors[:5]},
      "message": f"{len(errors)} lint errors found" if errors else "No lint errors"
    }

$ agents validation add --config examples/validations/wrapped-validation.yaml

Expected Output:

╭─────────────────────────── Validation Registered ────────────────────────────╮
│ Name: qa/lint-check                                                          │
│ Description: Validate that linting passes by wrapping the lint tool          │
│ Source: wrapped                                                              │
│ Mode: required                                                               │
│ Wraps: devops/run-linter                                                     │
│ Transform: def transform(result):                                            │
│   errors = result.get("errors", [])                                          │
│   passed = len(errors) == 0                                                  │
│   return {                                                                   │
│     "passed": passed,                                                        │
│     "data": {"error_count": len(errors), "errors": errors[:5]},              │
│     "message": f"{len(errors)} lint errors found" if errors else "No lint    │
│ errors"                                                                      │
│   }                                                                          │
╰──────────────────────────────────────────────────────────────────────────────╯

What's Happening:

The wraps field points to an existing tool (devops/run-linter). When the validation runs, CleverAgents calls that tool and passes the result through the transform function. This lets you reuse existing tools as quality gates without duplicating logic.

---

Step 4: List all tools and validations

$ agents tool list

Expected Output:

                                Tools (3 total)
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name              ┃ Type       ┃ Source  ┃ Description             ┃ Timeout ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ local/line-counter│ tool       │ custom  │ Count lines in a file   │      60 │
│ qa/coverage-check │ validation │ custom  │ Verify test coverage    │     120 │
│                   │            │         │ meets the requir...     │         │
│ qa/lint-check     │ validation │ wrapped │ Validate that linting   │     180 │
│                   │            │         │ passes by wrapp...      │         │
└───────────────────┴────────────┴─────────┴─────────────────────────┴─────────┘

What's Happening:

tool list shows both tools and validations in a single unified view. The Type column distinguishes them. The Source column shows how each was defined: custom (inline Python code), wrapped (delegates to another tool), or mcp (delegates to an MCP server tool).

---

Step 5: Filter by type

Show only validations:

$ agents tool list --type validation

Expected Output:

                                Tools (2 total)
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name              ┃ Type       ┃ Source  ┃ Description             ┃ Timeout ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ qa/coverage-check │ validation │ custom  │ Verify test coverage    │     120 │
│                   │            │         │ meets the requir...     │         │
│ qa/lint-check     │ validation │ wrapped │ Validate that linting   │     180 │
│                   │            │         │ passes by wrapp...      │         │
└───────────────────┴────────────┴─────────┴─────────────────────────┴─────────┘

Show only tools (not validations):

$ agents tool list --type tool

Expected Output:

                              Tools (1 total)
┏━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name            ┃ Type ┃ Source ┃ Description                 ┃ Timeout ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ local/line-counter│ tool │ custom │ Count lines in a file     │      60 │
└─────────────────┴──────┴────────┴─────────────────────────────┴─────────┘

Filter by namespace:

$ agents tool list --namespace qa

Expected Output:

                                Tools (2 total)
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name              ┃ Type       ┃ Source  ┃ Description             ┃ Timeout ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ qa/coverage-check │ validation │ custom  │ Verify test coverage    │     120 │
│ qa/lint-check     │ validation │ wrapped │ Validate that linting   │     180 │
└───────────────────┴────────────┴─────────┴─────────────────────────┴─────────┘

What's Happening:

The --type flag accepts tool or validation. The --namespace flag filters by the prefix before the / in the tool name. You can also pass a regex pattern as a positional argument: agents tool list "line-.*".

---

Step 6: Inspect a specific tool

$ agents tool show local/line-counter

Expected Output:

╭─────────── Tool Details ───────────╮
│ Name: local/line-counter           │
│ Namespace: local                   │
│ Short Name: line-counter           │
│ Description: Count lines in a file │
│ Source: custom                     │
│ Type: tool                         │
│ Timeout: 60s                       │
│ Capability:                        │
│   (default)                        │
│ Resource Slots:                    │
│   (none)                           │
│ Lifecycle:                         │
│   (none)                           │
╰────────────────────────────────────╯

Inspect a validation to see its mode:

$ agents tool show qa/coverage-check

Expected Output:

╭───────────────────────── Tool Details ─────────────────────────╮
│ Name: qa/coverage-check                                        │
│ Namespace: qa                                                  │
│ Short Name: coverage-check                                     │
│ Description: Verify test coverage meets the required threshold │
│ Source: custom                                                 │
│ Type: validation                                               │
│ Timeout: 120s                                                  │
│ Capability:                                                    │
│   (default)                                                    │
│ Resource Slots:                                                │
│   (none)                                                       │
│ Lifecycle:                                                     │
│   (none)                                                       │
│ Validation Mode: required                                      │
╰────────────────────────────────────────────────────────────────╯

What's Happening:

tool show works for both tools and validations — use the full namespaced name (namespace/short-name). For validations, the panel includes an extra Validation Mode field showing required or informational.

---

Step 7: Get machine-readable JSON output

$ agents tool list --format json

Expected Output:

{
  "command": "",
  "status": "ok",
  "exit_code": 0,
  "data": [
    {
      "name": "local/line-counter",
      "description": "Count lines in a file",
      "source": "custom",
      "tool_type": "tool",
      "namespace": "local",
      "short_name": "line-counter",
      "capability": {},
      "timeout": 60
    },
    {
      "name": "qa/coverage-check",
      "description": "Verify test coverage meets the required threshold",
      "source": "custom",
      "tool_type": "validation",
      "namespace": "qa",
      "short_name": "coverage-check",
      "capability": {},
      "timeout": 120
    },
    {
      "name": "qa/lint-check",
      "description": "Validate that linting passes by wrapping the lint tool",
      "source": "wrapped",
      "tool_type": "validation",
      "namespace": "qa",
      "short_name": "lint-check",
      "capability": {},
      "timeout": 180
    }
  ],
  "timing": {
    "duration_ms": 0
  },
  "messages": [
    {
      "level": "ok",
      "text": "ok"
    }
  ]
}

Show a single tool in JSON:

$ agents tool show local/line-counter --format json

Expected Output:

{
  "command": "",
  "status": "ok",
  "exit_code": 0,
  "data": {
    "name": "local/line-counter",
    "description": "Count lines in a file",
    "source": "custom",
    "tool_type": "tool",
    "namespace": "local",
    "short_name": "line-counter",
    "capability": {},
    "timeout": 60
  },
  "timing": {
    "duration_ms": 0
  },
  "messages": [
    {
      "level": "ok",
      "text": "ok"
    }
  ]
}

What's Happening:

All tool commands support --format json (or -f json). The response uses the standard CleverAgents JSON envelope: data holds the payload, status is "ok" on success, and timing.duration_ms shows elapsed time. For list commands, data is an array; for show, data is a single object.

---

Step 8: Update an existing tool or validation

If you modify the YAML and want to re-register without removing first:

$ agents validation add --config examples/validations/required-validation.yaml --update

Expected Output:

╭──────────────────── Validation Updated ────────────────────────╮
│ Name: qa/coverage-check                                        │
│ Description: Verify test coverage meets the required threshold │
│ Source: custom                                                 │
│ Mode: required                                                 │
╰────────────────────────────────────────────────────────────────╯

What's Happening:

Without --update, re-registering an existing name raises an error. The --update flag performs an upsert — it updates the existing record in place. The same flag is available on tool add --update.

---

Step 9: Remove a tool

$ agents tool remove --yes local/line-counter

Expected Output:

Removed tool: local/line-counter

What's Happening:

tool remove soft-deletes the tool from the registry. The --yes flag skips the interactive confirmation prompt, which is useful in scripts. Without --yes, the CLI will ask you to confirm before deleting.

---

Scripting with JSON Output

Because all commands support --format json, you can integrate tool management into CI/CD pipelines:

# Check if a specific tool is registered
$ agents tool list --format json | jq -r '.data[] | select(.name == "qa/coverage-check") | .name'
qa/coverage-check

# List all validation names
$ agents tool list --type validation --format json | jq -r '.data[].name'
qa/coverage-check
qa/lint-check

# Check the timeout for a tool
$ agents tool show local/line-counter --format json | jq '.data.timeout'
60

# Count registered tools
$ agents tool list --format json | jq '.data | length'
3

# List all tools in the qa namespace
$ agents tool list --namespace qa --format json | jq -r '.data[].name'
qa/coverage-check
qa/lint-check

---

Tool vs Validation: When to Use Each

Aspect	Tool	Validation
Purpose	General-purpose callable operation	Pass/fail quality gate
Return value	Any structured data	Must include passed: bool
Modes	N/A	required (blocks) or informational (advisory)
Source types	custom, mcp	custom, wrapped
Used in plans	As action steps	As pre/post conditions
Namespace convention	local/, devops/	qa/

---

Complete Interaction Log

Click to see the full verified command sequence

# 1. Register a custom tool
$ agents tool add --config examples/tools/custom-tool.yaml
╭─────────── Tool Registered ───────────╮
│ Name: local/line-counter              │
│ Source: custom                        │
│ Type: tool                            │
│ Timeout: 60s                          │
│ Capability: read_only: True, ...      │
╰───────────────────────────────────────╯

# 2. Register a required validation (custom code)
$ agents validation add --config examples/validations/required-validation.yaml
╭──────────────────── Validation Registered ─────────────────────╮
│ Name: qa/coverage-check                                        │
│ Mode: required                                                 │
╰────────────────────────────────────────────────────────────────╯

# 3. Register a wrapped validation
$ agents validation add --config examples/validations/wrapped-validation.yaml
╭─────────────────────────── Validation Registered ────────────────────────────╮
│ Name: qa/lint-check                                                          │
│ Source: wrapped                                                              │
│ Wraps: devops/run-linter                                                     │
╰──────────────────────────────────────────────────────────────────────────────╯

# 4. List all (tools + validations unified)
$ agents tool list
                                Tools (3 total)
┃ Name              ┃ Type       ┃ Source  ┃ Description             ┃ Timeout ┃
│ local/line-counter│ tool       │ custom  │ Count lines in a file   │      60 │
│ qa/coverage-check │ validation │ custom  │ Verify test coverage... │     120 │
│ qa/lint-check     │ validation │ wrapped │ Validate that linting...│     180 │

# 5. Filter by type
$ agents tool list --type validation
# → shows only qa/coverage-check and qa/lint-check

$ agents tool list --type tool
# → shows only local/line-counter

# 6. Filter by namespace
$ agents tool list --namespace qa
# → shows qa/coverage-check and qa/lint-check

# 7. Inspect a tool
$ agents tool show local/line-counter
# → shows capability flags, resource slots, lifecycle

# 8. Inspect a validation (shows Validation Mode)
$ agents tool show qa/coverage-check
# → shows Validation Mode: required

# 9. JSON output for scripting
$ agents tool list --format json
# → {"status": "ok", "data": [...], "timing": {...}}

$ agents tool show local/line-counter --format json
# → {"status": "ok", "data": {...}, "timing": {...}}

# 10. Update without removing
$ agents validation add --config examples/validations/required-validation.yaml --update
# → Validation Updated panel

# 11. Remove a tool
$ agents tool remove --yes local/line-counter
# → Removed tool: local/line-counter

---

Key Takeaways

• Tools and validations share a unified registry — tool list shows both; use --type tool or --type validation to filter.
• Namespaced names (namespace/short-name) uniquely identify every tool. Use local/ for project-specific tools and qa/ for quality gates.
• Validations have a mode — required blocks plan execution on failure; informational is advisory only.
• Wrapped validations reuse existing tools as quality gates via a transform function — no code duplication needed.
• --update flag enables safe re-registration when you modify a YAML config without needing to remove and re-add.
• --format json works on all tool commands — use it to integrate tool management into CI/CD scripts with jq.

Try It Yourself

Now that you've seen the full tool and validation lifecycle, try these variations:

• Register the MCP tool example: agents tool add --config examples/tools/mcp-tool.yaml
• Filter by source: agents tool list --source wrapped
• Regex filter: agents tool list "qa/.*"
• CI check: agents tool list --format json | jq '.data | map(select(.tool_type == "validation")) | length'
• Override mode at registration: agents validation add --config examples/validations/required-validation.yaml --informational

Related Examples

• See cli-tools/output-format-flags.md for a deep dive into all six output formats (json, yaml, plain, table, rich, color)
• See docs/showcase/cli-tools/ for more CLI tool examples

---

This example was automatically generated and verified by the CleverAgents UAT system. Feature area: Tool and validation management | Test cycle: 1 | Generated: 2026-04-07