LLMAgent does not implement tool calling — tools configs silently ignored #59

Closed
opened 2026-06-19 19:19:09 +00:00 by CoreRasurae · 1 comment
Member

Metadata

Field Value
Commit Message feat(agents): implement multi-turn tool-call loop and sandbox improvements
Branch bugfix/m2-llm-agent-tools-calling

Background

The porting-actor-tools-graph.yaml graph defines 8+ LLM agents (source-classifier, ui-configuration, source-code-analysis, etc.) with tools: and capabilities: fields declaring which tools each agent should use:

agents:
  source-classifier:
    type: llm
    config:
      tools:
        - file_read
        - file_write
      unsafe_mode: true

When a user runs the graph via test_app.py, the LLM receives only the system prompt text as instructions to use tools, but no actual tool definitions are passed to the LLM API. The response is generated without any tool calling capability, causing the LLM to hallucinate file paths and output fake tool call text instead of real tool invocations.

Current behavior

From llm_requests.log, every agent invocation:

  • Does NOT include a tools parameter in the call to LangChain's ainvoke() or astream()
  • The LLM responds with fabricated file paths (e.g., WindowsFormsApplications2/MainForm.cs) that do not exist — this is hallucination, not tool output
  • Example from log line 5: the response says file_read: ./WindowsFormsApplications2/MainForm.cs but this file does not exist anywhere on disk. The real folder is WindowsFormsApplication2

Expected behavior

  1. When an agent config specifies tools: [file_read, file_write], those tool definitions should be converted to LangChain-compatible tool objects and passed to the LLM API via the tools parameter in ainvoke(tools=...) / astream(tools=...)
  2. The LLM should respond with structured tool calls (JSON function calling format), not plain text describing what it would do
  3. A tool execution loop in the agent should receive tool results and feed them back into conversation until no more tools are pending
  4. After tool execution, the final tool result should be returned as the agent output

Root cause analysis

The LLMAgent class (cleveractors/agents/llm.py) has three gaps:

  1. process_message() (~line 471) calls self.chat_model.ainvoke(messages) with no tools parameter — LangChain's ChatModel supports tools for function calling but it is never used
  2. stream_message() (~line 796) calls self.chat_model.astream(lc_messages) — also no tools parameter
  3. The YAML config fields tools:, capabilities:, unsafe_mode:, allow_shell: are stored in self.config but never read by any method of LLMAgent

This issue exists across ALL code paths that invoke LLM agents:

  • runtime_dispatch.py:_execute_llm() — no tool extraction from config
  • langgraph/pure_graph.py / nodes.py:_execute_agent() — no tools passed to agent
  • Graph-type TOOLS nodes (_execute_tool()) execute only a stub returning fake results like {"result": "Executed file_read"}

The library has:

  • A dedicated ToolAgent (type=tool) that can directly execute tools via JSON input, but this requires the caller to send a structured tool request — it is NOT connected to LLMs
  • An LLMAgent (type=llm) that calls LLMs with no tool definitions whatsoever
  • No code that converts tools: [file_read] from YAML config into LangChain-compatible tool objects

Additional issues discovered during implementation

While building the core tool-calling pipeline, the following additional gaps, bugs, and missing safeguards were discovered — all of which were absolutely required for tool calls to be functional and safe:

1. Tool schemas were missing — LLMs received no parameter guidance

The LLM receives tool descriptions via the OpenAI function-calling API. Without proper schemas and guidance, models repeatedly made the same mistakes: reading large files without truncation (causing context overflow), calling file_read with shell commands like grep, and using open() in the Python sandbox when file_read/file_write should be used instead.

Resolution: Created llm_tools.py with normalize_tool_entry() and _BUILTIN_TOOL_SCHEMAS — a registry of OpenAI-compatible parameter schemas for all 9 built-in tools (echo, math, json_parse, http_request, file_read, file_write, progress_bar, shell, python_exec). Each schema includes targeted LLM guidance: file_read requires max_chars for files >10KB; file_write clarifies response text is NOT persisted; shell directs the LLM toward file_read; python_exec lists available builtins and instructs use of file_read/file_write instead of open().

2. Tool execution errors were silently discarded

When a tool raised ExecutionError or ConfigurationError, the error details were swallowed and the LLM only received "Tool error: <name>". The model had no way to self-correct because it never saw what went wrong.

Resolution: Tool error messages are now included in the ToolMessage sent back to the LLM, enabling the model to see actual error text (e.g. "File not found: ... Try listing the parent directory") and self-correct.

3. ToolAgent config was hardcoded — parent agent settings ignored

Transient ToolAgent instances created per tool call were always created with hardcoded unsafe_mode: true regardless of the parent LLM agent's config. The timeout from the parent's config was also not propagated.

Resolution: Config keys unsafe_mode, allow_shell, exec_python, and timeout from the LLM agent are now propagated to each transient ToolAgent.

4. LLM provider errors were silently swallowed

All LLM provider errors (e.g. BadRequestError from context overflow) and processing errors were mapped to a generic "LLM processing failed" with no actionable information. The actual error details were only available in a debug-level log most users never see.

Resolution: The actual error message is now preserved in the ExecutionError"LLM processing failed: The input or tool call is too long..." — giving callers and downstream agents actionable information.

5. Models got stuck in endless tool-call mode

Some models never stop making tool calls on their own — they keep calling file_read over and over even after gathering all needed data, eventually exhausting the round limit and producing empty output.

Resolution: When the tool loop exhausts rounds but the model produced no meaningful content, a final synthesizing prompt is injected: "You have finished gathering information. Now produce your final answer based on all the data collected. Do NOT make any more tool calls." The LLM is re-invoked one final time.

6. shell and python_exec tools were never registered

The shell and python_exec tool methods existed on ToolAgent but were never added to builtin_tools. LLM agents with allow_shell: true or exec_python: true in their config could not invoke these tools — the ToolAgent would never find them.

Resolution: Added registration: self.builtin_tools["shell"] = self._shell_tool when allow_shell is enabled, and self.builtin_tools["python_exec"] = self._python_exec_tool when exec_python is enabled.

7. Shell commands could not use pipes or redirections

_execute_shell_command used asyncio.create_subprocess_exec which passes arguments as a list to exec() — this does not support shell constructs like pipes (|), &&, or redirections (> / <). The LLM could not use piped commands like find ./dir -name '*.cs' | sort.

Resolution: Switched to asyncio.create_subprocess_shell so the command string is passed to the system shell, enabling piped/chained/redirected constructs.

8. file_read lacked directory listing — LLMs could not navigate

The LLM needed to navigate directory structures but file_read only understood file paths. When given a directory, it silently failed.

Resolution: file_read now detects directories and returns a formatted listing with file sizes and type indicators (\ud83d\udcc1 for directories, \ud83d\udcc4 for files).

9. file_read lacked max_chars — context overflow inevitable

The LLM would call file_read on entire large files (362KB+) causing context overflow (262K token limit) and request failure. There was no mechanism to limit what was read.

Resolution: Added optional max_chars integer parameter. When set, content is truncated and the output header includes TRUNCATED to N chars. Schemas enforce this via "CRITICAL: Always use max_chars..." and "REQUIRED for files >10KB" guidance.

10. LLMs confused file_read with shell commands

Models frequently passed shell commands (e.g. "grep -n pattern file | head") as file paths to file_read, causing confusing FileNotFoundError.

Resolution: file_read now detects shell command patterns (pipe |, ;, &&, ||, >, < operators, or first word matching known shell commands) and returns an explicit error redirecting to the shell tool.

11. file_read gave no recovery path on FileNotFoundError

When a file was not found, the LLM received a generic error with no guidance on how to recover — it would often try the same path again or hallucinate the content.

Resolution: File-not-found errors now include the parent directory path and instruct the LLM: "Try listing the parent directory with file_read '<parent>' to see available files."

12. python_exec sandbox suppressed stdout output

sys.stdout was not redirected during sandbox execution (only stderr was). When the LLM used print() in sandbox code, the output was invisible — the function appeared to do nothing.

Resolution: sys.stdout is now redirected to io.StringIO() and captured output is included in the result.

13. python_exec sandbox gave unhelpful NameError messages

When sandbox code referenced an undefined name like open or exec, the raw NameError("name 'open' is not defined") gave no guidance on what to use instead.

Resolution: NameError is now caught and the error message extracts the name, providing actionable guidance: "'open' is not available in the Python sandbox. Use the file_read and file_write tools for file operations, and the shell tool for system commands."

14. Empty/whitespace-only messages polluted conversation history

When an LLM agent produced empty output ("\n"), the graph state stored it as a valid {"role": "assistant", "content": "\n"} message. In a 27-agent pipeline, each passing empty messages forward, the noise compounds dramatically and pollutes every downstream LLM request.

Resolution: _prepare_conversation_history() in nodes.py now skips messages whose content is empty or whitespace-only. Only messages with meaningful content are included in conversation history.

15. Tool metadata and capability discovery was missing

get_capabilities() did not report tool-calling support, and get_metadata() had no visibility into whether tools were configured on an agent.

Resolution: get_capabilities() now returns "tool-calling" when tools are configured. get_metadata() includes tools_configured and tool_count.

Files changed

File Additions Deletions
src/cleveractors/agents/llm_tools.py (new) 307 0
src/cleveractors/agents/llm.py 193 57
src/cleveractors/agents/tool.py 172 50
src/cleveractors/agents/llm_imports.py 3 0
src/cleveractors/langgraph/nodes.py 2 0

Subtasks

  • Add ToolMessage import to LangChain globals (llm_imports.py)
  • Create llm_tools.py module with normalize_tool_entry() and _BUILTIN_TOOL_SCHEMAS for all 9 built-in tools
  • Read tools from agent config at __init__ and convert to LangChain-compatible format via normalize_tool_entry()
  • Inject tools parameter into LLMAgent.process_message() calls to ainvoke(tools=...)
  • Add multi-turn tool execution loop: call LLM -> execute tools via ToolAgent -> append ToolMessages -> repeat until no more tool_calls
  • Propagate unsafe_mode, allow_shell, exec_python, timeout from LLM agent config to transient ToolAgent instances
  • Include actual error details in ToolMessages for LLM self-correction (no more discarded errors)
  • Preserve LLM provider error details in ExecutionError messages (no more generic "LLM processing failed")
  • Add stuck-model recovery: inject synthesizing prompt when model exhausts rounds with empty output
  • Register shell and python_exec tools in builtin_tools when allow_shell / exec_python is enabled
  • Fix _execute_shell_command to use create_subprocess_shell (enables pipes, &&, redirections)
  • Add directory listing to file_read tool (formatted output with file sizes and type indicators)
  • Add max_chars truncation parameter to file_read (prevents context overflow on large files)
  • Add shell command detection to file_read (redirects grep/find/ls to shell tool)
  • Add file-not-found recovery hints to file_read (suggests listing parent directory)
  • Capture stdout in python_exec sandbox (print() output now visible to LLM)
  • Add NameError guidance to python_exec sandbox (redirects open()/exec() to proper tools)
  • Filter empty/whitespace-only messages from conversation history in nodes.py
  • Add "tool-calling" to get_capabilities() and tool metadata to get_metadata()
  • Add tests: BDD unit tests for tool calling loop, tool schemas, file_read enhancements, sandbox, empty message filtering
  • Add tests: Robot Framework integration tests against real LangChain server for tool calling
  • Verify end-to-end: run graph with tool-using agents -> tool calls execute and return real results

Definition of Done

This issue is complete when:

  • All subtasks above are completed and checked off.
  • A Git commit is created where the first line matches the Commit Message in Metadata exactly.
  • The commit is pushed to the branch matching the Branch in Metadata exactly.
  • The commit is submitted as a PR to master, reviewed, and merged.
## Metadata | Field | Value | |-------|-------| | Commit Message | feat(agents): implement multi-turn tool-call loop and sandbox improvements | | Branch | bugfix/m2-llm-agent-tools-calling | ## Background The `porting-actor-tools-graph.yaml` graph defines 8+ LLM agents (source-classifier, ui-configuration, source-code-analysis, etc.) with `tools:` and `capabilities:` fields declaring which tools each agent should use: ```yaml agents: source-classifier: type: llm config: tools: - file_read - file_write unsafe_mode: true ``` When a user runs the graph via `test_app.py`, the LLM receives only the system prompt text as instructions to use tools, but **no actual tool definitions are passed to the LLM API**. The response is generated without any tool calling capability, causing the LLM to hallucinate file paths and output fake tool call text instead of real tool invocations. ## Current behavior From `llm_requests.log`, every agent invocation: - **Does NOT include a `tools` parameter** in the call to LangChain's `ainvoke()` or `astream()` - The LLM responds with fabricated file paths (e.g., `WindowsFormsApplications2/MainForm.cs`) that do not exist — this is **hallucination**, not tool output - Example from log line 5: the response says ` file_read: ./WindowsFormsApplications2/MainForm.cs ` but this file does not exist anywhere on disk. The real folder is `WindowsFormsApplication2` ## Expected behavior 1. When an agent config specifies `tools: [file_read, file_write]`, those tool definitions should be converted to LangChain-compatible tool objects and passed to the LLM API via the `tools` parameter in `ainvoke(tools=...)` / `astream(tools=...)` 2. The LLM should respond with structured tool calls (JSON function calling format), not plain text describing what it would do 3. A tool execution loop in the agent should receive tool results and feed them back into conversation until no more tools are pending 4. After tool execution, the final tool result should be returned as the agent output ## Root cause analysis The `LLMAgent` class (`cleveractors/agents/llm.py`) has three gaps: 1. **`process_message()`** (~line 471) calls `self.chat_model.ainvoke(messages)` with no `tools` parameter — LangChain's ChatModel supports `tools` for function calling but it is never used 2. **`stream_message()`** (~line 796) calls `self.chat_model.astream(lc_messages)` — also no `tools` parameter 3. The YAML config fields `tools:`, `capabilities:`, `unsafe_mode:`, `allow_shell:` are stored in `self.config` but **never read** by any method of `LLMAgent` This issue exists across ALL code paths that invoke LLM agents: - `runtime_dispatch.py:_execute_llm()` — no tool extraction from config - `langgraph/pure_graph.py` / `nodes.py:_execute_agent()` — no tools passed to agent - Graph-type TOOLS nodes (`_execute_tool()`) execute only a stub returning fake results like `{"result": "Executed file_read"}` The library has: - A dedicated **ToolAgent** (type=`tool`) that can directly execute tools via JSON input, but this requires the caller to send a structured tool request — it is NOT connected to LLMs - An **LLMAgent** (type=`llm`) that calls LLMs with no tool definitions whatsoever - **No code** that converts `tools: [file_read]` from YAML config into LangChain-compatible tool objects ## Additional issues discovered during implementation While building the core tool-calling pipeline, the following additional gaps, bugs, and missing safeguards were discovered — all of which were absolutely required for tool calls to be functional and safe: ### 1. Tool schemas were missing — LLMs received no parameter guidance The LLM receives tool descriptions via the OpenAI function-calling API. Without proper schemas and guidance, models repeatedly made the same mistakes: reading large files without truncation (causing context overflow), calling `file_read` with shell commands like `grep`, and using `open()` in the Python sandbox when `file_read`/`file_write` should be used instead. **Resolution**: Created `llm_tools.py` with `normalize_tool_entry()` and `_BUILTIN_TOOL_SCHEMAS` — a registry of OpenAI-compatible parameter schemas for all 9 built-in tools (`echo`, `math`, `json_parse`, `http_request`, `file_read`, `file_write`, `progress_bar`, `shell`, `python_exec`). Each schema includes targeted LLM guidance: `file_read` requires `max_chars` for files >10KB; `file_write` clarifies response text is NOT persisted; `shell` directs the LLM toward `file_read`; `python_exec` lists available builtins and instructs use of `file_read`/`file_write` instead of `open()`. ### 2. Tool execution errors were silently discarded When a tool raised `ExecutionError` or `ConfigurationError`, the error details were swallowed and the LLM only received `"Tool error: <name>"`. The model had no way to self-correct because it never saw what went wrong. **Resolution**: Tool error messages are now included in the `ToolMessage` sent back to the LLM, enabling the model to see actual error text (e.g. `"File not found: ... Try listing the parent directory"`) and self-correct. ### 3. ToolAgent config was hardcoded — parent agent settings ignored Transient `ToolAgent` instances created per tool call were always created with hardcoded `unsafe_mode: true` regardless of the parent LLM agent's config. The `timeout` from the parent's config was also not propagated. **Resolution**: Config keys `unsafe_mode`, `allow_shell`, `exec_python`, and `timeout` from the LLM agent are now propagated to each transient `ToolAgent`. ### 4. LLM provider errors were silently swallowed All LLM provider errors (e.g. `BadRequestError` from context overflow) and processing errors were mapped to a generic `"LLM processing failed"` with no actionable information. The actual error details were only available in a debug-level log most users never see. **Resolution**: The actual error message is now preserved in the `ExecutionError` — `"LLM processing failed: The input or tool call is too long..."` — giving callers and downstream agents actionable information. ### 5. Models got stuck in endless tool-call mode Some models never stop making tool calls on their own — they keep calling `file_read` over and over even after gathering all needed data, eventually exhausting the round limit and producing empty output. **Resolution**: When the tool loop exhausts rounds but the model produced no meaningful content, a final synthesizing prompt is injected: *"You have finished gathering information. Now produce your final answer based on all the data collected. Do NOT make any more tool calls."* The LLM is re-invoked one final time. ### 6. `shell` and `python_exec` tools were never registered The `shell` and `python_exec` tool methods existed on `ToolAgent` but were never added to `builtin_tools`. LLM agents with `allow_shell: true` or `exec_python: true` in their config could not invoke these tools — the ToolAgent would never find them. **Resolution**: Added registration: `self.builtin_tools["shell"] = self._shell_tool` when `allow_shell` is enabled, and `self.builtin_tools["python_exec"] = self._python_exec_tool` when `exec_python` is enabled. ### 7. Shell commands could not use pipes or redirections `_execute_shell_command` used `asyncio.create_subprocess_exec` which passes arguments as a list to `exec()` — this does not support shell constructs like pipes (`|`), `&&`, or redirections (`>` / `<`). The LLM could not use piped commands like `find ./dir -name '*.cs' | sort`. **Resolution**: Switched to `asyncio.create_subprocess_shell` so the command string is passed to the system shell, enabling piped/chained/redirected constructs. ### 8. `file_read` lacked directory listing — LLMs could not navigate The LLM needed to navigate directory structures but `file_read` only understood file paths. When given a directory, it silently failed. **Resolution**: `file_read` now detects directories and returns a formatted listing with file sizes and type indicators (`\ud83d\udcc1` for directories, `\ud83d\udcc4` for files). ### 9. `file_read` lacked max_chars — context overflow inevitable The LLM would call `file_read` on entire large files (362KB+) causing context overflow (262K token limit) and request failure. There was no mechanism to limit what was read. **Resolution**: Added optional `max_chars` integer parameter. When set, content is truncated and the output header includes `TRUNCATED to N chars`. Schemas enforce this via `"CRITICAL: Always use max_chars..."` and `"REQUIRED for files >10KB"` guidance. ### 10. LLMs confused `file_read` with shell commands Models frequently passed shell commands (e.g. `"grep -n pattern file | head"`) as file paths to `file_read`, causing confusing FileNotFoundError. **Resolution**: `file_read` now detects shell command patterns (pipe `|`, `;`, `&&`, `||`, `>`, `<` operators, or first word matching known shell commands) and returns an explicit error redirecting to the shell tool. ### 11. `file_read` gave no recovery path on FileNotFoundError When a file was not found, the LLM received a generic error with no guidance on how to recover — it would often try the same path again or hallucinate the content. **Resolution**: File-not-found errors now include the parent directory path and instruct the LLM: `"Try listing the parent directory with file_read '<parent>' to see available files."` ### 12. `python_exec` sandbox suppressed stdout output `sys.stdout` was not redirected during sandbox execution (only `stderr` was). When the LLM used `print()` in sandbox code, the output was invisible — the function appeared to do nothing. **Resolution**: `sys.stdout` is now redirected to `io.StringIO()` and captured output is included in the result. ### 13. `python_exec` sandbox gave unhelpful NameError messages When sandbox code referenced an undefined name like `open` or `exec`, the raw `NameError("name 'open' is not defined")` gave no guidance on what to use instead. **Resolution**: `NameError` is now caught and the error message extracts the name, providing actionable guidance: `"'open' is not available in the Python sandbox. Use the file_read and file_write tools for file operations, and the shell tool for system commands."` ### 14. Empty/whitespace-only messages polluted conversation history When an LLM agent produced empty output (`"\n"`), the graph state stored it as a valid `{"role": "assistant", "content": "\n"}` message. In a 27-agent pipeline, each passing empty messages forward, the noise compounds dramatically and pollutes every downstream LLM request. **Resolution**: `_prepare_conversation_history()` in `nodes.py` now skips messages whose content is empty or whitespace-only. Only messages with meaningful content are included in conversation history. ### 15. Tool metadata and capability discovery was missing `get_capabilities()` did not report tool-calling support, and `get_metadata()` had no visibility into whether tools were configured on an agent. **Resolution**: `get_capabilities()` now returns `"tool-calling"` when tools are configured. `get_metadata()` includes `tools_configured` and `tool_count`. ## Files changed | File | Additions | Deletions | |------|-----------|-----------| | `src/cleveractors/agents/llm_tools.py` (new) | 307 | 0 | | `src/cleveractors/agents/llm.py` | 193 | 57 | | `src/cleveractors/agents/tool.py` | 172 | 50 | | `src/cleveractors/agents/llm_imports.py` | 3 | 0 | | `src/cleveractors/langgraph/nodes.py` | 2 | 0 | ## Subtasks - [x] Add `ToolMessage` import to LangChain globals (`llm_imports.py`) - [x] Create `llm_tools.py` module with `normalize_tool_entry()` and `_BUILTIN_TOOL_SCHEMAS` for all 9 built-in tools - [x] Read tools from agent config at `__init__` and convert to LangChain-compatible format via `normalize_tool_entry()` - [x] Inject `tools` parameter into `LLMAgent.process_message()` calls to `ainvoke(tools=...)` - [x] Add multi-turn tool execution loop: call LLM -> execute tools via ToolAgent -> append ToolMessages -> repeat until no more tool_calls - [x] Propagate `unsafe_mode`, `allow_shell`, `exec_python`, `timeout` from LLM agent config to transient ToolAgent instances - [x] Include actual error details in ToolMessages for LLM self-correction (no more discarded errors) - [x] Preserve LLM provider error details in ExecutionError messages (no more generic "LLM processing failed") - [x] Add stuck-model recovery: inject synthesizing prompt when model exhausts rounds with empty output - [x] Register `shell` and `python_exec` tools in `builtin_tools` when `allow_shell` / `exec_python` is enabled - [x] Fix `_execute_shell_command` to use `create_subprocess_shell` (enables pipes, &&, redirections) - [x] Add directory listing to `file_read` tool (formatted output with file sizes and type indicators) - [x] Add `max_chars` truncation parameter to `file_read` (prevents context overflow on large files) - [x] Add shell command detection to `file_read` (redirects grep/find/ls to shell tool) - [x] Add file-not-found recovery hints to `file_read` (suggests listing parent directory) - [x] Capture `stdout` in `python_exec` sandbox (print() output now visible to LLM) - [x] Add `NameError` guidance to `python_exec` sandbox (redirects open()/exec() to proper tools) - [x] Filter empty/whitespace-only messages from conversation history in `nodes.py` - [x] Add `"tool-calling"` to `get_capabilities()` and tool metadata to `get_metadata()` - [x] Add tests: BDD unit tests for tool calling loop, tool schemas, file_read enhancements, sandbox, empty message filtering - [x] Add tests: Robot Framework integration tests against real LangChain server for tool calling - [x] Verify end-to-end: run graph with tool-using agents -> tool calls execute and return real results ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the first line matches the Commit Message in Metadata exactly. - The commit is pushed to the branch matching the Branch in Metadata exactly. - The commit is submitted as a PR to master, reviewed, and merged.
CoreRasurae added this to the v2.1.0 milestone 2026-06-19 19:24:00 +00:00
Author
Member

PR #60 implements this fix and is linked for review.

PR details:

  • Branch: feature/m2-llm-agent-tool-calling
  • PR URL: #60
PR #60 implements this fix and is linked for review. PR details: - Branch: feature/m2-llm-agent-tool-calling - PR URL: https://git.cleverthis.com/cleveragents/cleveractors-core/pulls/60
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveractors-core#59
No description provided.