CHANGELOG.md

@@ -29,6 +29,10 @@ and this project adheres to [Clever Semantic Versioning](https://www.w3.org/subm
 
 ### Fixed
 
+- **LLM Agent Synthesis-Round Tool Name Validation (issue #63)** (`llm.py`): When the model hallucinates a tool name not present in the declared tools list during the synthesis round, the invalid call is now rejected with a `ToolMessage` error stating `"Tool '<name>' is not available."` instead of crashing or silently discarding the response. A `_declared_names` set is built once per synthesis round from `self._lc_tools` for O(1) lookup.
+
+- **ToolAgent Tool Name Dispatch Bug (issue #63)** (`tool.py`): `_execute_tool()` was checking `tool_name not in self.tools` where `self.tools` is a heterogeneous list of strings and dicts — a string tool name like `"file_read"` would never match the list because the list contains strings, not dicts with that name. Fixed by extracting `string_tool_names` and checking against that list instead.
+
 - **LLMAgent Tool-Calling Pipeline (issue #59)** (`llm.py`, `tool.py`, `llm_tools.py`, `llm_imports.py`, `nodes.py`): The LLM agent tool-calling path was broken in multiple ways — tools from agent config were not passed to the LLM, tool execution was single-pass (the model could not see results or make follow-up calls), tool errors were discarded, `shell` and `python_exec` tools were never registered in `builtin_tools`, and `create_subprocess_exec` was used for shell commands preventing pipes and redirections.
 
   **Multi-turn tool-call loop:** `process_message()` now passes tools to the LLM via `ainvoke(tools=...)` and enters a configurable multi-turn loop (default 20 rounds, minimum 1, overridable via `tool_max_rounds` config or `TOOL_MAX_ROUNDS` env var; values ≤0 are clamped to 1). The stuck-model synthesis-prompt round counts as an additional invocation beyond the configured limit. After each batch of tool calls, `ToolMessage`s with results are appended to the conversation and the LLM is re-invoked, enabling multi-step reasoning chains. **Streaming path (`stream_message`) does not support tool calling** — when an LLM agent is configured with tools and invoked via the streaming API, tools are not passed to the model and the multi-turn loop is not engaged.

features/llm_agent_tool_calling.feature

@@ -345,3 +345,13 @@ Feature: LLMAgent Tool Calling
     And I set a mock chat model that gets stuck then returns tool_calls in synthesis
     And I tool_calling process a message "Get stuck"
     Then the tool_calling result should contain "Synthesized answer"
+
+  # ── Synthesis-round tool name validation (fix for issue #63) ────────
+
+  Scenario: Synthesis round rejects hallucinated tool name not in declared tools
+    Given I have an LLM agent configuration with tools [{"name": "echo"}]
+    And I set tool_max_rounds to 2
+    When I create the LLM agent
+    And I set a mock chat model that gets stuck then returns hallucinated tool name in synthesis
+    And I tool_calling process a message "Get stuck"
+    Then the synthesis messages should contain a ToolMessage with content "Tool 'nonexistent_tool' is not available."

features/steps/llm_agent_tool_calling_steps.py

@@ -1003,3 +1003,63 @@ def step_mock_stuck_with_synth_tools(context):
     mock_model = Mock(spec=BaseChatModel)
     mock_model.ainvoke = _mock_ainvoke
     context.llm_agent.chat_model = mock_model
+
+
+@when(
+    "I set a mock chat model that gets stuck then returns hallucinated tool name in synthesis"
+)
+def step_mock_stuck_with_synth_invalid_tool(context):
+    call_counter = [0]
+    context._synthesis_messages_captured = None
+
+    async def _mock_ainvoke(messages, **invoke_kwargs):
+        call_counter[0] += 1
+        if call_counter[0] <= 2:
+            return _create_mock_response(
+                "",
+                tool_calls=[
+                    {
+                        "id": f"call_stuck_{call_counter[0]}",
+                        "type": "function",
+                        "function": {
+                            "name": "echo",
+                            "arguments": '{"text": "test"}',
+                        },
+                    }
+                ],
+            )
+        if call_counter[0] == 3:
+            return _create_mock_response(
+                "Here is the synthesis",
+                tool_calls=[
+                    {
+                        "id": "call_synth_invalid",
+                        "type": "function",
+                        "function": {
+                            "name": "nonexistent_tool",
+                            "arguments": "{}",
+                        },
+                    }
+                ],
+            )
+        context._synthesis_messages_captured = list(messages)
+        return _create_mock_response("Final answer")
+
+    mock_model = Mock(spec=BaseChatModel)
+    mock_model.ainvoke = _mock_ainvoke
+    context.llm_agent.chat_model = mock_model
+
+
+@then('the synthesis messages should contain a ToolMessage with content "{text}"')
+def step_synthesis_messages_have_toolmessage(context, text):
+    messages = context._synthesis_messages_captured
+    assert messages is not None, (
+        "Synthesis messages were not captured - the mock may not have "
+        "reached the final ainvoke call"
+    )
+    tool_msgs = [m for m in messages if isinstance(m, ToolMessage)]
+    found = any(str(m.content) == text for m in tool_msgs)
+    assert found, (
+        f"No ToolMessage found with exact content {text!r}. "
+        f"ToolMessages: {[str(m.content) for m in tool_msgs]}"
+    )

features/tool_agent.feature

@@ -719,3 +719,72 @@ Feature: ToolAgent Functionality
       {"tool": "file_write", "args": {"file": "test.txt", "content": "test", "mode": "invalid"}}
       """
     Then tool execution should fail with message containing "Invalid mode"
+
+  # ── ToolAgent dict-style tool name dispatch fix (issue #63) ──────────
+
+  Scenario: Tool agent with dict-style tools rejects unauthorized tool
+    Given a ToolAgent is configured with name "dict_tool_agent" and config
+      """
+      {
+        "tools": [{"name": "echo"}]
+      }
+      """
+    When I create the ToolAgent
+    And I process a JSON message with the ToolAgent:
+      """
+      {"tool": "file_read", "args": {"file": "test.txt"}}
+      """
+    Then tool execution should fail with message containing "not in allowed tools list"
+
+  Scenario: Tool agent with dict-style tools accepts authorized tool
+    Given a ToolAgent is configured with name "dict_echo_agent" and config
+      """
+      {
+        "tools": [{"name": "echo"}]
+      }
+      """
+    When I create the ToolAgent
+    And I process a JSON message with the ToolAgent:
+      """
+      {"tool": "echo", "args": {"text": "dict works"}}
+      """
+    Then the result should be "dict works"
+
+  Scenario: Tool agent with mixed string and dict tools rejects unauthorized
+    Given a ToolAgent is configured with name "mixed_tool_agent" and config
+      """
+      {
+        "tools": ["echo", {"name": "math"}]
+      }
+      """
+    When I create the ToolAgent
+    And I process a JSON message with the ToolAgent:
+      """
+      {"tool": "file_read", "args": {"file": "test.txt"}}
+      """
+    Then tool execution should fail with message containing "not in allowed tools list"
+
+  Scenario: Tool agent with mixed string and dict tools accepts authorized string tool
+    Given a ToolAgent is configured with name "mixed_echo_agent" and config
+      """
+      {
+        "tools": ["echo", {"name": "math"}]
+      }
+      """
+    When I create the ToolAgent
+    And I process a message with the ToolAgent: "echo works with mixed"
+    Then the result should be "works with mixed"
+
+  Scenario: Tool agent with mixed string and dict tools accepts authorized dict tool
+    Given a ToolAgent is configured with name "mixed_math_agent" and config
+      """
+      {
+        "tools": ["echo", {"name": "math"}]
+      }
+      """
+    When I create the ToolAgent
+    And I process a JSON message with the ToolAgent:
+      """
+      {"tool": "math", "args": {"expression": "40+2"}}
+      """
+    Then the result should be "42"

src/cleveractors/agents/llm.py

@@ -752,6 +752,10 @@ class LLMAgent(AgentWithMemory):
 
             _budget_exhausted: bool = False
 
+            _declared_names = {
+                t.get("function", {}).get("name", "") for t in (self._lc_tools or [])
+            }
+
             for _tool_round in range(_TOOL_MAX_ROUNDS):
                 if _tool_round > 0 and not _has_tools:
                     break
@@ -861,6 +865,14 @@ class LLMAgent(AgentWithMemory):
                                         if parent_unsafe
                                         else None
                                     )
+                                    if tool_name not in _declared_names:
+                                        messages.append(
+                                            ToolMessage(
+                                                content=f"Tool '{tool_name}' is not available.",
+                                                tool_call_id=call_id,
+                                            )
+                                        )
+                                        break
                                     raw_out = await t_agent.process_message(
                                         json.dumps({"tool": tool_name, "args": args})
                                         if isinstance(args, dict)
@@ -946,6 +958,7 @@ class LLMAgent(AgentWithMemory):
                             )
                         )
                         continue
+
                     args: dict[str, Any] = {}
                     if isinstance(arguments_raw, str):
                         try:
@@ -967,6 +980,16 @@ class LLMAgent(AgentWithMemory):
                             "exec_python": self.config.get("exec_python", False),
                             "timeout": self.config.get("timeout", 1),
                         }
+
+                        if tool_name not in _declared_names:
+                            messages.append(
+                                ToolMessage(
+                                    content=f"Tool '{tool_name}' is not available.",
+                                    tool_call_id=call_id,
+                                )
+                            )
+                            continue
+
                         agent = _TA(
                             name=f"_tc_{call_id}",
                             config=tool_config,
@@ -1054,6 +1077,7 @@ class LLMAgent(AgentWithMemory):
                 # write files or perform other last-minute operations
                 # requested by the synthesis prompt.
                 synth_tool_calls: object = getattr(response, "tool_calls", None)
+
                 if (
                     isinstance(synth_tool_calls, list)
                     and synth_tool_calls
@@ -1077,6 +1101,16 @@ class LLMAgent(AgentWithMemory):
                                 )
                             )
                             continue
+
+                        if tool_name not in _declared_names:
+                            messages.append(
+                                ToolMessage(
+                                    content=f"Tool '{tool_name}' is not available.",
+                                    tool_call_id=call_id,
+                                )
+                            )
+                            continue
+
                         args: dict[str, Any] = {}
                         if isinstance(arguments_raw, str):
                             try:

src/cleveractors/agents/tool.py

@@ -91,6 +91,7 @@ class ToolAgent(Agent):
 
     def _validate_tools(self) -> None:
         """Validate the configured tools."""
+
         for tool in self.tools:
             if isinstance(tool, str):
                 if tool not in self.builtin_tools and not self.allow_shell:
@@ -196,6 +197,7 @@ class ToolAgent(Agent):
                     "input_data": message,
                     "message": message,
                 }
+
                 result = await self._execute_tool(
                     tool_config["name"], tool_args, effective_context
                 )
@@ -251,9 +253,10 @@ class ToolAgent(Agent):
     ) -> Any:
         """Execute a specific tool."""
         # Check if tool is in the allowed list first
+        string_tool_names = [t for t in self.tools if isinstance(t, str)]
         dict_tool_names = [t.get("name") for t in self.tools if isinstance(t, dict)]
 
-        if tool_name not in self.tools and tool_name not in dict_tool_names:
+        if tool_name not in string_tool_names and tool_name not in dict_tool_names:
             logger.error("Tool '%s' not in allowed list for %s", tool_name, self.name)
             raise ExecutionError(f"Tool '{tool_name}' not in allowed tools list")