cleveragents/cleveragents-core

Fork 3

UAT: `ActorLoader.discover()` silently ignores unresolved tool references — broken actors are loaded without error #4821

New issue

Open

opened 2026-04-08 19:44:30 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-08 19:44:30 +00:00

Owner

Bug Report

Feature Area: Actor System — Actor loader, actor lifecycle
Severity: Medium — actors with broken tool references are silently loaded; tool failures only surface at runtime
Found by: UAT tester instance uat-tester-actor-system
Spec reference: docs/specification.md §Pre-flight Checks (line 28717), §Actor Definition Fields (line 20429)

What Was Tested

Code-level analysis of src/cleveragents/actor/loader.py — ActorLoader._resolve_tools() (lines 290–302) and discover() (lines 95–222) against the spec's tool reference requirements.

Expected Behavior (from spec)

The spec (line 28717) states in the pre-flight checks:

Skill and tool existence: Verify that all skills and tools referenced by the action's actor configuration exist in the registry. This includes both named tools and tools referenced via skills (transitively resolved).

The spec (line 20429) defines the tools field:

tools | list | Yes (TOOL) | List of tool references (strings) and/or inline tool definitions.

When an actor YAML references a tool by name (e.g., files/read_file), that tool must be registered in the Tool Registry. If it is not, the actor configuration is invalid and should fail to load.

Actual Behavior (from code)

ActorLoader._resolve_tools() at src/cleveragents/actor/loader.py:290–302:

def _resolve_tools(self, config: ActorConfigSchema) -> None:
    if self._tool_registry is None:
        return
    for tool_ref in config.tools:
        if isinstance(tool_ref, str):
            spec = self._tool_registry.get(tool_ref)
            if spec is None:
                msg = (
                    f"Unresolved tool reference '{tool_ref}' "
                    f"in actor '{config.name}'"
                )
                logger.warning(msg)         # ← Only a WARNING
                self._warnings.append(msg)  # ← Stored as warning, not error

When a tool reference cannot be resolved:

A WARNING log message is emitted
The warning is stored in self._warnings
The actor is still loaded successfully — discover() does not fail
The actor with broken tool references is returned in the list of loaded actors

This means:

An actor YAML with tools: [files/read_file, nonexistent/tool] will load without error
The nonexistent/tool reference is silently ignored
At runtime, when the actor tries to invoke nonexistent/tool, it will fail with a confusing error

Additionally, ActorLoader is constructed with tool_registry=None in many places, which means _resolve_tools() is a no-op and tool references are never validated at all.

Steps to Reproduce

from pathlib import Path
from cleveragents.actor.loader import ActorLoader
from cleveragents.tool.registry import ToolRegistry

# Create a tool registry with no tools registered
tool_registry = ToolRegistry()

# Create a loader with the empty registry
loader = ActorLoader(
    search_roots=[Path("/path/to/actors")],
    tool_registry=tool_registry,
)

# Actor YAML contains: tools: [files/read_file, nonexistent/tool]
actors = loader.discover()
# No error raised — actors loaded successfully despite broken tool references
# loader.warnings contains the warning messages
print(loader.warnings)  # ["Unresolved tool reference 'nonexistent/tool' in actor 'local/my-actor'"]

Code Location

src/cleveragents/actor/loader.py:290–302 — ActorLoader._resolve_tools() (warning only, no error)
src/cleveragents/actor/loader.py:95–222 — ActorLoader.discover() (does not fail on tool resolution warnings)
src/cleveragents/application/container.py — ActorLoader construction (check if tool_registry is always provided)

Expected Fix

_resolve_tools() should either:

Option A (strict): Raise a ValidationError when a tool reference cannot be resolved, causing discover() to fail for that actor (consistent with how duplicate actors are handled):

def _resolve_tools(self, config: ActorConfigSchema) -> None:
    if self._tool_registry is None:
        return
    for tool_ref in config.tools:
        if isinstance(tool_ref, str):
            spec = self._tool_registry.get(tool_ref)
            if spec is None:
                raise ValidationError(
                    f"Unresolved tool reference '{tool_ref}' "
                    f"in actor '{config.name}'. "
                    f"Register the tool first with 'agents tool add'."
                )

Option B (lenient with clear warning): Keep as warning but ensure the warning is surfaced prominently in CLI output and the actor is marked as "degraded" so users know it has broken dependencies.

Option A is preferred as it aligns with the spec's pre-flight check requirements and prevents silent failures at runtime.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area:** Actor System — Actor loader, actor lifecycle **Severity:** Medium — actors with broken tool references are silently loaded; tool failures only surface at runtime **Found by:** UAT tester instance `uat-tester-actor-system` **Spec reference:** `docs/specification.md` §Pre-flight Checks (line 28717), §Actor Definition Fields (line 20429) --- ### What Was Tested Code-level analysis of `src/cleveragents/actor/loader.py` — `ActorLoader._resolve_tools()` (lines 290–302) and `discover()` (lines 95–222) against the spec's tool reference requirements. ### Expected Behavior (from spec) The spec (line 28717) states in the pre-flight checks: > **Skill and tool existence**: Verify that all skills and tools referenced by the action's actor configuration exist in the registry. This includes both named tools and tools referenced via skills (transitively resolved). The spec (line 20429) defines the `tools` field: > `tools` | list | Yes (TOOL) | List of tool references (strings) and/or inline tool definitions. When an actor YAML references a tool by name (e.g., `files/read_file`), that tool must be registered in the Tool Registry. If it is not, the actor configuration is invalid and should fail to load. ### Actual Behavior (from code) `ActorLoader._resolve_tools()` at `src/cleveragents/actor/loader.py:290–302`: ```python def _resolve_tools(self, config: ActorConfigSchema) -> None: if self._tool_registry is None: return for tool_ref in config.tools: if isinstance(tool_ref, str): spec = self._tool_registry.get(tool_ref) if spec is None: msg = ( f"Unresolved tool reference '{tool_ref}' " f"in actor '{config.name}'" ) logger.warning(msg) # ← Only a WARNING self._warnings.append(msg) # ← Stored as warning, not error ``` When a tool reference cannot be resolved: 1. A `WARNING` log message is emitted 2. The warning is stored in `self._warnings` 3. **The actor is still loaded successfully** — `discover()` does not fail 4. The actor with broken tool references is returned in the list of loaded actors This means: - An actor YAML with `tools: [files/read_file, nonexistent/tool]` will load without error - The `nonexistent/tool` reference is silently ignored - At runtime, when the actor tries to invoke `nonexistent/tool`, it will fail with a confusing error Additionally, `ActorLoader` is constructed with `tool_registry=None` in many places, which means `_resolve_tools()` is a no-op and tool references are **never validated at all**. ### Steps to Reproduce ```python from pathlib import Path from cleveragents.actor.loader import ActorLoader from cleveragents.tool.registry import ToolRegistry # Create a tool registry with no tools registered tool_registry = ToolRegistry() # Create a loader with the empty registry loader = ActorLoader( search_roots=[Path("/path/to/actors")], tool_registry=tool_registry, ) # Actor YAML contains: tools: [files/read_file, nonexistent/tool] actors = loader.discover() # No error raised — actors loaded successfully despite broken tool references # loader.warnings contains the warning messages print(loader.warnings) # ["Unresolved tool reference 'nonexistent/tool' in actor 'local/my-actor'"] ``` ### Code Location - `src/cleveragents/actor/loader.py:290–302` — `ActorLoader._resolve_tools()` (warning only, no error) - `src/cleveragents/actor/loader.py:95–222` — `ActorLoader.discover()` (does not fail on tool resolution warnings) - `src/cleveragents/application/container.py` — `ActorLoader` construction (check if `tool_registry` is always provided) ### Expected Fix `_resolve_tools()` should either: **Option A (strict)**: Raise a `ValidationError` when a tool reference cannot be resolved, causing `discover()` to fail for that actor (consistent with how duplicate actors are handled): ```python def _resolve_tools(self, config: ActorConfigSchema) -> None: if self._tool_registry is None: return for tool_ref in config.tools: if isinstance(tool_ref, str): spec = self._tool_registry.get(tool_ref) if spec is None: raise ValidationError( f"Unresolved tool reference '{tool_ref}' " f"in actor '{config.name}'. " f"Register the tool first with 'agents tool add'." ) ``` **Option B (lenient with clear warning)**: Keep as warning but ensure the warning is surfaced prominently in CLI output and the actor is marked as "degraded" so users know it has broken dependencies. Option A is preferred as it aligns with the spec's pre-flight check requirements and prevents silent failures at runtime. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 referenced this issue

2026-04-08 19:46:08 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 referenced this issue

2026-04-08 19:59:07 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 referenced this issue

2026-04-08 20:12:39 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 referenced this issue

2026-04-08 20:16:04 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 referenced this issue

2026-04-08 20:22:25 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 added the

labels

2026-04-08 23:00:06 +00:00

HAL9000 commented

2026-04-09 03:03:00 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: Medium — spec compliance bug identified by UAT testing
Story Points: 3 (M) — targeted fix to align implementation with spec
MoSCoW: Must Have — spec compliance is required for correct system behavior

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium — spec compliance bug identified by UAT testing - **Story Points**: 3 (M) — targeted fix to align implementation with spec - **MoSCoW**: Must Have — spec compliance is required for correct system behavior --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner

HAL9000 added

and removed

labels

2026-04-09 03:03:02 +00:00

HAL9000 added this to the v3.5.0 milestone