LLMAgent does not implement tool calling — tools configs silently ignored #59
Closed
opened 2026-06-19 19:19:09 +00:00 by CoreRasurae
·
1 comment
No Branch/Tag Specified
Labels
Clear labels
auto/blocked-by-deps
auto/ci-timeout
auto/claimed-implementer
auto/claimed-merge
auto/claimed-reviewer
auto/driver-down
auto/invariant-violation
auto/last-attempt-tier-0
auto/last-attempt-tier-1
auto/last-attempt-tier-2
auto/last-attempt-tier-min
Automation Tracking
auto/needs-conflict-resolution
auto/needs-implementer
auto/postmortem
auto/ready-to-merge
auto/restart-throttled
auto/revert
auto/sentinel
auto/stale-inactivity
auto/unstable
Blocked
Needs Feedback
Signed-off: Owner
Signed-off: Scrum Master
Signed-off: Tech Lead
Spike
PR blocked by an open issue dependency. Operator must close the dep (or remove the dependency link) before the merge driver can act. Auto-cleared by merge_drive when no open deps remain.
Most recent merge cycle hit CI timeout. Driver excludes this PR while last merge_cycle row is < 30 min old; label persists thereafter as visible history.
Currently being processed by an implementer worker.
Currently being processed by the merge driver.
Currently being processed by a reviewer worker.
Merge driver heartbeat stale; pipeline halted. Closed automatically on next clean tick.
Detected master commit violating the strict merge invariant. Tracked as an issue (not a PR label); kept here for label completeness.
In-cycle escalation: most recent attempt ran at the Tier 0 slot (`tier-0`). Slot's model defined in .opencode/models/tiers.yaml.
In-cycle escalation: most recent attempt ran at the Tier 1 slot (`tier-1`). Slot's model defined in .opencode/models/tiers.yaml.
In-cycle escalation: most recent attempt ran at the Tier 2 slot (`tier-2`). Slot's model defined in .opencode/models/tiers.yaml. Gated behind IMPLEMENTER_ESCALATION_TIER2_ENABLED.
In-cycle escalation: most recent attempt ran at the Tier -1 slot (`tier-min`). Slot's model defined in .opencode/models/tiers.yaml. Suffix is ``-min`` (not ``--1``) so the Forgejo UI reads naturally.
Tracking issues used by the AI Automation system for agents to communicate and report.
Rebase conflict needs LLM conflict-resolver.
Failing CI needs implementer attention.
Documenting a driver incident or rollback.
Reviewer has APPROVED this PR and no later REQUEST_CHANGES is outstanding. The merge driver requires this label to even consider a PR for merging. Set by the reviewer worker on APPROVE; cleared on REQUEST_CHANGES.
Train repeatedly lost master-tempo races. Driver excludes via merge_cycle until cooldown elapses; label persists as visible history.
Revert PR backing out an invariant violation. Fast-tracked through the merge driver.
Sentinel PR duplicated from upstream into a personal fork by tools/duplicate_prs_to_fork.py for pipeline testing. Lives only in the fork; the canonical pipeline never sees it.
No implementer activity for N days. Flagged for human review. Auto-cleared on next push to head branch.
Repeatedly fails on current master (>= 3 ci-fail-on-rebased-sha releases in 12 h). Excluded from driver until human triage.
A ticket in a blocked state and unable to complete until some other task is completed first.
Bounty
$100
A bounty of $100 for any open-source contributor who provides a MR that solves this issue
Bounty
$1000
A bounty of $1000 for any open-source contributor who provides a MR that solves this issue
Bounty
$10000
A bounty of $10000 for any open-source contributor who provides a MR that solves this issue
Bounty
$20
A bounty of $20 for any open-source contributor who provides a MR that solves this issue
Bounty
$2000
A bounty of $2000 for any open-source contributor who provides a MR that solves this issue
Bounty
$250
A bounty of $250 for any open-source contributor who provides a MR that solves this issue
Bounty
$50
A bounty of $50 for any open-source contributor who provides a MR that solves this issue
Bounty
$500
A bounty of $500 for any open-source contributor who provides a MR that solves this issue
Bounty
$5000
A bounty of $5000 for any open-source contributor who provides a MR that solves this issue
Bounty
$750
A bounty of $750 for any open-source contributor who provides a MR that solves this issue
MoSCoW
Could have
Could have feature in order to satisfy the epic/legendary.
MoSCoW
Must have
Must have feature in order to satisfy the epic/legendary.
MoSCoW
Should have
Should have feature in order to satisfy the epic/legendary.
There are questions in the ticket that can not be completed until the project owner provides clarity.
Points
1
1 man-hours worth of work for an expert with no learning curve.
Points
13
13 man-hours worth of work for an expert with no learning curve.
Points
2
2 man-hours worth of work for an expert with no learning curve.
Points
21
21 man-hours worth of work for an expert with no learning curve.
Points
3
3 man-hours worth of work for an expert with no learning curve.
Points
34
34 man-hours worth of work for an expert with no learning curve.
Points
5
5 man-hours worth of work for an expert with no learning curve.
Points
55
55 man-hours worth of work for an expert with no learning curve.
Points
8
8 man-hours worth of work for an expert with no learning curve.
Points
88
88 man-hours worth of work for an expert with no learning curve.
Priority
Backlog
This ticket has backlogged priority and is not to be worked on yet
Priority
CI Blocker
Critical priority issue that blocks CI/CD pipeline and prevents PR merges
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.
When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.
When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.
A ticket for learning a tool or technology that is needed to be able to do future planning and design.
State
Completed
The ticket has been fully implemented, completed, and merged with the source code. This label should only be applied once a ticket is closed.
State
Duplicate
A ticket that represents the same content as an existing ticket.
State
In Progress
A ticket that is actively being developed.
State
In Review
A ticket that has had some code completed to implement but is waiting to pass peer review and is not yet merged in.
State
Paused
This ticket's work started but wasn't finished. It's on hold (likely in a feature branch) and will be resumed later, either due to a blocker or a delay.
State
Unverified
All new tickets start in this state. A developer may set it to show the ticket is unverified. This means we haven't agreed to work on it. It will either move to a verified state or be closed as wontdo.
State
Verified
The issue has been verified by a developer as legitimate. It will be worked on and verified tickets are now considered part of the backlog.
State
Wont Do
This ticket has been decided it wont be done. This may mean the bug has been determined to not be real (cant verify) or the feature is one we have decided we dont want to adopt.
Type
Automation
Any edits or discussion about the AI automated coding system.
Type
Bug
Something that doesnt work as intended.
Type
Discussion
Anytime a ticket represents a discussion about a subject and doesnt fall into one of the other categories.
Type
Documentation
An error or improvement needed in the documentation.
Type
Epic
Any first tier epic. That is, an epic which contains only issues as children and will not have sub-epics.
Type
Feature
Some new functionality not present.
Type
Legendary
A type of Epic which will contain other Epics.
Type
Refactor
A code change that restructures existing code without changing its external behavior.
Type
Support
Someone needs help using the project.
Type
Task
A generic task that doesnt fit into the other type categories.
Type
Testing
Work exclusively focusing on fixing or expanding testing.
Projects
Clear projects
No project
Assignees
aditya (Aditya Chhabra)
aleenaumair (Aleena Umair)
brent.edwards (Brent Edwards)
CoreRasurae (Luis Mendes)
drew (Drew Morris)
eugen.thaci (Eugen Thaci)
freemo (Jeffrey Phillips Freeman)
HAL9000 (HAL 9000)
HAL9001 (HAL9001)
hamza.khyari (Hamza Khyari)
hurui200320 (Rui Hu)
justin.morris
khird (Kyle Hird)
org.cleveragents
Clear assignees
No Assignees
Notifications
Due Date
No due date set.
Blocks
Depends on
#22 Epic: Package Registry Client — Support Package Registry Standard v1.0.0
cleveragents/cleveractors-core
#60 feat(agents): implement multi-turn tool-call loop and sandbox improvements
cleveragents/cleveractors-core
Reference: cleveragents/cleveractors-core#59
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Metadata
Background
The
porting-actor-tools-graph.yamlgraph defines 8+ LLM agents (source-classifier, ui-configuration, source-code-analysis, etc.) withtools:andcapabilities:fields declaring which tools each agent should use:When a user runs the graph via
test_app.py, the LLM receives only the system prompt text as instructions to use tools, but no actual tool definitions are passed to the LLM API. The response is generated without any tool calling capability, causing the LLM to hallucinate file paths and output fake tool call text instead of real tool invocations.Current behavior
From
llm_requests.log, every agent invocation:toolsparameter in the call to LangChain'sainvoke()orastream()WindowsFormsApplications2/MainForm.cs) that do not exist — this is hallucination, not tool outputfile_read: ./WindowsFormsApplications2/MainForm.csbut this file does not exist anywhere on disk. The real folder isWindowsFormsApplication2Expected behavior
tools: [file_read, file_write], those tool definitions should be converted to LangChain-compatible tool objects and passed to the LLM API via thetoolsparameter inainvoke(tools=...)/astream(tools=...)Root cause analysis
The
LLMAgentclass (cleveractors/agents/llm.py) has three gaps:process_message()(~line 471) callsself.chat_model.ainvoke(messages)with notoolsparameter — LangChain's ChatModel supportstoolsfor function calling but it is never usedstream_message()(~line 796) callsself.chat_model.astream(lc_messages)— also notoolsparametertools:,capabilities:,unsafe_mode:,allow_shell:are stored inself.configbut never read by any method ofLLMAgentThis issue exists across ALL code paths that invoke LLM agents:
runtime_dispatch.py:_execute_llm()— no tool extraction from configlanggraph/pure_graph.py/nodes.py:_execute_agent()— no tools passed to agent_execute_tool()) execute only a stub returning fake results like{"result": "Executed file_read"}The library has:
tool) that can directly execute tools via JSON input, but this requires the caller to send a structured tool request — it is NOT connected to LLMsllm) that calls LLMs with no tool definitions whatsoevertools: [file_read]from YAML config into LangChain-compatible tool objectsAdditional issues discovered during implementation
While building the core tool-calling pipeline, the following additional gaps, bugs, and missing safeguards were discovered — all of which were absolutely required for tool calls to be functional and safe:
1. Tool schemas were missing — LLMs received no parameter guidance
The LLM receives tool descriptions via the OpenAI function-calling API. Without proper schemas and guidance, models repeatedly made the same mistakes: reading large files without truncation (causing context overflow), calling
file_readwith shell commands likegrep, and usingopen()in the Python sandbox whenfile_read/file_writeshould be used instead.Resolution: Created
llm_tools.pywithnormalize_tool_entry()and_BUILTIN_TOOL_SCHEMAS— a registry of OpenAI-compatible parameter schemas for all 9 built-in tools (echo,math,json_parse,http_request,file_read,file_write,progress_bar,shell,python_exec). Each schema includes targeted LLM guidance:file_readrequiresmax_charsfor files >10KB;file_writeclarifies response text is NOT persisted;shelldirects the LLM towardfile_read;python_execlists available builtins and instructs use offile_read/file_writeinstead ofopen().2. Tool execution errors were silently discarded
When a tool raised
ExecutionErrororConfigurationError, the error details were swallowed and the LLM only received"Tool error: <name>". The model had no way to self-correct because it never saw what went wrong.Resolution: Tool error messages are now included in the
ToolMessagesent back to the LLM, enabling the model to see actual error text (e.g."File not found: ... Try listing the parent directory") and self-correct.3. ToolAgent config was hardcoded — parent agent settings ignored
Transient
ToolAgentinstances created per tool call were always created with hardcodedunsafe_mode: trueregardless of the parent LLM agent's config. Thetimeoutfrom the parent's config was also not propagated.Resolution: Config keys
unsafe_mode,allow_shell,exec_python, andtimeoutfrom the LLM agent are now propagated to each transientToolAgent.4. LLM provider errors were silently swallowed
All LLM provider errors (e.g.
BadRequestErrorfrom context overflow) and processing errors were mapped to a generic"LLM processing failed"with no actionable information. The actual error details were only available in a debug-level log most users never see.Resolution: The actual error message is now preserved in the
ExecutionError—"LLM processing failed: The input or tool call is too long..."— giving callers and downstream agents actionable information.5. Models got stuck in endless tool-call mode
Some models never stop making tool calls on their own — they keep calling
file_readover and over even after gathering all needed data, eventually exhausting the round limit and producing empty output.Resolution: When the tool loop exhausts rounds but the model produced no meaningful content, a final synthesizing prompt is injected: "You have finished gathering information. Now produce your final answer based on all the data collected. Do NOT make any more tool calls." The LLM is re-invoked one final time.
6.
shellandpython_exectools were never registeredThe
shellandpython_exectool methods existed onToolAgentbut were never added tobuiltin_tools. LLM agents withallow_shell: trueorexec_python: truein their config could not invoke these tools — the ToolAgent would never find them.Resolution: Added registration:
self.builtin_tools["shell"] = self._shell_toolwhenallow_shellis enabled, andself.builtin_tools["python_exec"] = self._python_exec_toolwhenexec_pythonis enabled.7. Shell commands could not use pipes or redirections
_execute_shell_commandusedasyncio.create_subprocess_execwhich passes arguments as a list toexec()— this does not support shell constructs like pipes (|),&&, or redirections (>/<). The LLM could not use piped commands likefind ./dir -name '*.cs' | sort.Resolution: Switched to
asyncio.create_subprocess_shellso the command string is passed to the system shell, enabling piped/chained/redirected constructs.8.
file_readlacked directory listing — LLMs could not navigateThe LLM needed to navigate directory structures but
file_readonly understood file paths. When given a directory, it silently failed.Resolution:
file_readnow detects directories and returns a formatted listing with file sizes and type indicators (\ud83d\udcc1for directories,\ud83d\udcc4for files).9.
file_readlacked max_chars — context overflow inevitableThe LLM would call
file_readon entire large files (362KB+) causing context overflow (262K token limit) and request failure. There was no mechanism to limit what was read.Resolution: Added optional
max_charsinteger parameter. When set, content is truncated and the output header includesTRUNCATED to N chars. Schemas enforce this via"CRITICAL: Always use max_chars..."and"REQUIRED for files >10KB"guidance.10. LLMs confused
file_readwith shell commandsModels frequently passed shell commands (e.g.
"grep -n pattern file | head") as file paths tofile_read, causing confusing FileNotFoundError.Resolution:
file_readnow detects shell command patterns (pipe|,;,&&,||,>,<operators, or first word matching known shell commands) and returns an explicit error redirecting to the shell tool.11.
file_readgave no recovery path on FileNotFoundErrorWhen a file was not found, the LLM received a generic error with no guidance on how to recover — it would often try the same path again or hallucinate the content.
Resolution: File-not-found errors now include the parent directory path and instruct the LLM:
"Try listing the parent directory with file_read '<parent>' to see available files."12.
python_execsandbox suppressed stdout outputsys.stdoutwas not redirected during sandbox execution (onlystderrwas). When the LLM usedprint()in sandbox code, the output was invisible — the function appeared to do nothing.Resolution:
sys.stdoutis now redirected toio.StringIO()and captured output is included in the result.13.
python_execsandbox gave unhelpful NameError messagesWhen sandbox code referenced an undefined name like
openorexec, the rawNameError("name 'open' is not defined")gave no guidance on what to use instead.Resolution:
NameErroris now caught and the error message extracts the name, providing actionable guidance:"'open' is not available in the Python sandbox. Use the file_read and file_write tools for file operations, and the shell tool for system commands."14. Empty/whitespace-only messages polluted conversation history
When an LLM agent produced empty output (
"\n"), the graph state stored it as a valid{"role": "assistant", "content": "\n"}message. In a 27-agent pipeline, each passing empty messages forward, the noise compounds dramatically and pollutes every downstream LLM request.Resolution:
_prepare_conversation_history()innodes.pynow skips messages whose content is empty or whitespace-only. Only messages with meaningful content are included in conversation history.15. Tool metadata and capability discovery was missing
get_capabilities()did not report tool-calling support, andget_metadata()had no visibility into whether tools were configured on an agent.Resolution:
get_capabilities()now returns"tool-calling"when tools are configured.get_metadata()includestools_configuredandtool_count.Files changed
src/cleveractors/agents/llm_tools.py(new)src/cleveractors/agents/llm.pysrc/cleveractors/agents/tool.pysrc/cleveractors/agents/llm_imports.pysrc/cleveractors/langgraph/nodes.pySubtasks
ToolMessageimport to LangChain globals (llm_imports.py)llm_tools.pymodule withnormalize_tool_entry()and_BUILTIN_TOOL_SCHEMASfor all 9 built-in tools__init__and convert to LangChain-compatible format vianormalize_tool_entry()toolsparameter intoLLMAgent.process_message()calls toainvoke(tools=...)unsafe_mode,allow_shell,exec_python,timeoutfrom LLM agent config to transient ToolAgent instancesshellandpython_exectools inbuiltin_toolswhenallow_shell/exec_pythonis enabled_execute_shell_commandto usecreate_subprocess_shell(enables pipes, &&, redirections)file_readtool (formatted output with file sizes and type indicators)max_charstruncation parameter tofile_read(prevents context overflow on large files)file_read(redirects grep/find/ls to shell tool)file_read(suggests listing parent directory)stdoutinpython_execsandbox (print() output now visible to LLM)NameErrorguidance topython_execsandbox (redirects open()/exec() to proper tools)nodes.py"tool-calling"toget_capabilities()and tool metadata toget_metadata()Definition of Done
This issue is complete when:
PR #60 implements this fix and is linked for review.
PR details: