LLMAgent does not implement tool calling — tools configs silently ignored #59
Labels
No labels
auto/blocked-by-deps
auto/ci-timeout
auto/claimed-implementer
auto/claimed-merge
auto/claimed-reviewer
auto/driver-down
auto/invariant-violation
auto/last-attempt-tier-0
auto/last-attempt-tier-1
auto/last-attempt-tier-2
auto/last-attempt-tier-min
Automation Tracking
auto/needs-conflict-resolution
auto/needs-implementer
auto/postmortem
auto/ready-to-merge
auto/restart-throttled
auto/revert
auto/sentinel
auto/stale-inactivity
auto/unstable
Blocked
Bounty
$100
Bounty
$1000
Bounty
$10000
Bounty
$20
Bounty
$2000
Bounty
$250
Bounty
$50
Bounty
$500
Bounty
$5000
Bounty
$750
MoSCoW
Could have
MoSCoW
Must have
MoSCoW
Should have
Needs Feedback
Points
1
Points
13
Points
2
Points
21
Points
3
Points
34
Points
5
Points
55
Points
8
Points
88
Priority
Backlog
Priority
CI Blocker
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Signed-off: Owner
Signed-off: Scrum Master
Signed-off: Tech Lead
Spike
State
Completed
State
Duplicate
State
In Progress
State
In Review
State
Paused
State
Unverified
State
Verified
State
Wont Do
Type
Automation
Type
Bug
Type
Discussion
Type
Documentation
Type
Epic
Type
Feature
Type
Legendary
Type
Refactor
Type
Support
Type
Task
Type
Testing
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Blocks
Depends on
#22 Epic: Package Registry Client — Support Package Registry Standard v1.0.0
cleveragents/cleveractors-core
#60 feat(agents): implement multi-turn tool-call loop and sandbox improvements
cleveragents/cleveractors-core
Reference
cleveragents/cleveractors-core#59
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Metadata
Background
The
porting-actor-tools-graph.yamlgraph defines 8+ LLM agents (source-classifier, ui-configuration, source-code-analysis, etc.) withtools:andcapabilities:fields declaring which tools each agent should use:When a user runs the graph via
test_app.py, the LLM receives only the system prompt text as instructions to use tools, but no actual tool definitions are passed to the LLM API. The response is generated without any tool calling capability, causing the LLM to hallucinate file paths and output fake tool call text instead of real tool invocations.Current behavior
From
llm_requests.log, every agent invocation:toolsparameter in the call to LangChain'sainvoke()orastream()WindowsFormsApplications2/MainForm.cs) that do not exist — this is hallucination, not tool outputfile_read: ./WindowsFormsApplications2/MainForm.csbut this file does not exist anywhere on disk. The real folder isWindowsFormsApplication2Expected behavior
tools: [file_read, file_write], those tool definitions should be converted to LangChain-compatible tool objects and passed to the LLM API via thetoolsparameter inainvoke(tools=...)/astream(tools=...)Root cause analysis
The
LLMAgentclass (cleveractors/agents/llm.py) has three gaps:process_message()(~line 471) callsself.chat_model.ainvoke(messages)with notoolsparameter — LangChain's ChatModel supportstoolsfor function calling but it is never usedstream_message()(~line 796) callsself.chat_model.astream(lc_messages)— also notoolsparametertools:,capabilities:,unsafe_mode:,allow_shell:are stored inself.configbut never read by any method ofLLMAgentThis issue exists across ALL code paths that invoke LLM agents:
runtime_dispatch.py:_execute_llm()— no tool extraction from configlanggraph/pure_graph.py/nodes.py:_execute_agent()— no tools passed to agent_execute_tool()) execute only a stub returning fake results like{"result": "Executed file_read"}The library has:
tool) that can directly execute tools via JSON input, but this requires the caller to send a structured tool request — it is NOT connected to LLMsllm) that calls LLMs with no tool definitions whatsoevertools: [file_read]from YAML config into LangChain-compatible tool objectsAdditional issues discovered during implementation
While building the core tool-calling pipeline, the following additional gaps, bugs, and missing safeguards were discovered — all of which were absolutely required for tool calls to be functional and safe:
1. Tool schemas were missing — LLMs received no parameter guidance
The LLM receives tool descriptions via the OpenAI function-calling API. Without proper schemas and guidance, models repeatedly made the same mistakes: reading large files without truncation (causing context overflow), calling
file_readwith shell commands likegrep, and usingopen()in the Python sandbox whenfile_read/file_writeshould be used instead.Resolution: Created
llm_tools.pywithnormalize_tool_entry()and_BUILTIN_TOOL_SCHEMAS— a registry of OpenAI-compatible parameter schemas for all 9 built-in tools (echo,math,json_parse,http_request,file_read,file_write,progress_bar,shell,python_exec). Each schema includes targeted LLM guidance:file_readrequiresmax_charsfor files >10KB;file_writeclarifies response text is NOT persisted;shelldirects the LLM towardfile_read;python_execlists available builtins and instructs use offile_read/file_writeinstead ofopen().2. Tool execution errors were silently discarded
When a tool raised
ExecutionErrororConfigurationError, the error details were swallowed and the LLM only received"Tool error: <name>". The model had no way to self-correct because it never saw what went wrong.Resolution: Tool error messages are now included in the
ToolMessagesent back to the LLM, enabling the model to see actual error text (e.g."File not found: ... Try listing the parent directory") and self-correct.3. ToolAgent config was hardcoded — parent agent settings ignored
Transient
ToolAgentinstances created per tool call were always created with hardcodedunsafe_mode: trueregardless of the parent LLM agent's config. Thetimeoutfrom the parent's config was also not propagated.Resolution: Config keys
unsafe_mode,allow_shell,exec_python, andtimeoutfrom the LLM agent are now propagated to each transientToolAgent.4. LLM provider errors were silently swallowed
All LLM provider errors (e.g.
BadRequestErrorfrom context overflow) and processing errors were mapped to a generic"LLM processing failed"with no actionable information. The actual error details were only available in a debug-level log most users never see.Resolution: The actual error message is now preserved in the
ExecutionError—"LLM processing failed: The input or tool call is too long..."— giving callers and downstream agents actionable information.5. Models got stuck in endless tool-call mode
Some models never stop making tool calls on their own — they keep calling
file_readover and over even after gathering all needed data, eventually exhausting the round limit and producing empty output.Resolution: When the tool loop exhausts rounds but the model produced no meaningful content, a final synthesizing prompt is injected: "You have finished gathering information. Now produce your final answer based on all the data collected. Do NOT make any more tool calls." The LLM is re-invoked one final time.
6.
shellandpython_exectools were never registeredThe
shellandpython_exectool methods existed onToolAgentbut were never added tobuiltin_tools. LLM agents withallow_shell: trueorexec_python: truein their config could not invoke these tools — the ToolAgent would never find them.Resolution: Added registration:
self.builtin_tools["shell"] = self._shell_toolwhenallow_shellis enabled, andself.builtin_tools["python_exec"] = self._python_exec_toolwhenexec_pythonis enabled.7. Shell commands could not use pipes or redirections
_execute_shell_commandusedasyncio.create_subprocess_execwhich passes arguments as a list toexec()— this does not support shell constructs like pipes (|),&&, or redirections (>/<). The LLM could not use piped commands likefind ./dir -name '*.cs' | sort.Resolution: Switched to
asyncio.create_subprocess_shellso the command string is passed to the system shell, enabling piped/chained/redirected constructs.8.
file_readlacked directory listing — LLMs could not navigateThe LLM needed to navigate directory structures but
file_readonly understood file paths. When given a directory, it silently failed.Resolution:
file_readnow detects directories and returns a formatted listing with file sizes and type indicators (\ud83d\udcc1for directories,\ud83d\udcc4for files).9.
file_readlacked max_chars — context overflow inevitableThe LLM would call
file_readon entire large files (362KB+) causing context overflow (262K token limit) and request failure. There was no mechanism to limit what was read.Resolution: Added optional
max_charsinteger parameter. When set, content is truncated and the output header includesTRUNCATED to N chars. Schemas enforce this via"CRITICAL: Always use max_chars..."and"REQUIRED for files >10KB"guidance.10. LLMs confused
file_readwith shell commandsModels frequently passed shell commands (e.g.
"grep -n pattern file | head") as file paths tofile_read, causing confusing FileNotFoundError.Resolution:
file_readnow detects shell command patterns (pipe|,;,&&,||,>,<operators, or first word matching known shell commands) and returns an explicit error redirecting to the shell tool.11.
file_readgave no recovery path on FileNotFoundErrorWhen a file was not found, the LLM received a generic error with no guidance on how to recover — it would often try the same path again or hallucinate the content.
Resolution: File-not-found errors now include the parent directory path and instruct the LLM:
"Try listing the parent directory with file_read '<parent>' to see available files."12.
python_execsandbox suppressed stdout outputsys.stdoutwas not redirected during sandbox execution (onlystderrwas). When the LLM usedprint()in sandbox code, the output was invisible — the function appeared to do nothing.Resolution:
sys.stdoutis now redirected toio.StringIO()and captured output is included in the result.13.
python_execsandbox gave unhelpful NameError messagesWhen sandbox code referenced an undefined name like
openorexec, the rawNameError("name 'open' is not defined")gave no guidance on what to use instead.Resolution:
NameErroris now caught and the error message extracts the name, providing actionable guidance:"'open' is not available in the Python sandbox. Use the file_read and file_write tools for file operations, and the shell tool for system commands."14. Empty/whitespace-only messages polluted conversation history
When an LLM agent produced empty output (
"\n"), the graph state stored it as a valid{"role": "assistant", "content": "\n"}message. In a 27-agent pipeline, each passing empty messages forward, the noise compounds dramatically and pollutes every downstream LLM request.Resolution:
_prepare_conversation_history()innodes.pynow skips messages whose content is empty or whitespace-only. Only messages with meaningful content are included in conversation history.15. Tool metadata and capability discovery was missing
get_capabilities()did not report tool-calling support, andget_metadata()had no visibility into whether tools were configured on an agent.Resolution:
get_capabilities()now returns"tool-calling"when tools are configured.get_metadata()includestools_configuredandtool_count.Files changed
src/cleveractors/agents/llm_tools.py(new)src/cleveractors/agents/llm.pysrc/cleveractors/agents/tool.pysrc/cleveractors/agents/llm_imports.pysrc/cleveractors/langgraph/nodes.pySubtasks
ToolMessageimport to LangChain globals (llm_imports.py)llm_tools.pymodule withnormalize_tool_entry()and_BUILTIN_TOOL_SCHEMASfor all 9 built-in tools__init__and convert to LangChain-compatible format vianormalize_tool_entry()toolsparameter intoLLMAgent.process_message()calls toainvoke(tools=...)unsafe_mode,allow_shell,exec_python,timeoutfrom LLM agent config to transient ToolAgent instancesshellandpython_exectools inbuiltin_toolswhenallow_shell/exec_pythonis enabled_execute_shell_commandto usecreate_subprocess_shell(enables pipes, &&, redirections)file_readtool (formatted output with file sizes and type indicators)max_charstruncation parameter tofile_read(prevents context overflow on large files)file_read(redirects grep/find/ls to shell tool)file_read(suggests listing parent directory)stdoutinpython_execsandbox (print() output now visible to LLM)NameErrorguidance topython_execsandbox (redirects open()/exec() to proper tools)nodes.py"tool-calling"toget_capabilities()and tool metadata toget_metadata()Definition of Done
This issue is complete when:
PR #60 implements this fix and is linked for review.
PR details:
type: toolgraph node execution by wiringNodeConfig.tooltoToolAgent._execute_tool#75