forked from cleveragents/cleveragents-core
hurui200320
0c5724c2f6
Implements the full tool-calling path for `agents actor run --skill` so that LLM
actors can actually invoke tools through the ToolCallingRuntime loop when a skill
is attached.
Core changes:
- reactive/tool_caller.py (new): ToolCallingLLMCaller implements the LLMCaller protocol;
binds tool schemas via bind_tools(), threads SystemMessage+HumanMessage on first call
and AIMessage+ToolMessages on subsequent calls, extracts tool calls from LangChain
responses following the LangChainSessionCaller pattern.
- reactive/tool_caller.py: bidirectional tool name encoding via uppercase sentinels
(_encode_tool_name / _decode_tool_name) to make CleverAgents namespaced tool names
("builtin/file-read", "server:local/tool") compatible with Anthropic's tool name
pattern ^[a-zA-Z0-9_-]{1,128}$. Uses _C_ for ":" and _S_ for "/" — uppercase
sentinels are safe because valid CleverAgents tool names forbid uppercase letters.
Encoding applied in _resolve_llm() before bind_tools(); decoding applied in invoke()
when extracting tool calls from the LLM response.
- reactive/tool_agent.py (new): ToolCallingAgent builds a per-run local ToolRegistry from
resolved skill tool entries by looking up names in the shared builtin_registry; drives
ToolCallingRuntime.run_tool_loop(); exposes last_result for tool_calls surfacing.
- reactive/application.py: (ST-1) _make_agent_instance() now always merges skill tools
instead of silently dropping them when actor has no base tools list; routes
tools+llm→ToolCallingAgent, tools+non-llm→SimpleToolAgent, no-tools+llm→SimpleLLMAgent;
(ST-4) _builtin_registry created at startup with register_file_tools/git/subplan;
(ST-6) _tally_tool_calls() + last_run_tool_calls property.
- reactive/graph_executor.py: (ST-5) ToolCallingAgent added to isinstance check in
_invoke_agent() so context dict is forwarded for Jinja2 rendering.
- cli/commands/actor_run.py: prints "Tool Calls: {n}" when > 0.
Test fixes:
- features/steps/actor_cli_run_steps.py: _make_app() sets last_run_tool_calls=0 to avoid
MagicMock>int TypeError in Python 3.13.
- features/steps/actor_run_signature_resolve_steps.py: same fix.
- robot/helper_actor_run_signature.py: same fix.
- features/reactive_application_coverage_boost.feature: updated scenario to verify new
correct behavior (LLM+skills → ToolCallingAgent, not silently kept as SimpleLLMAgent).
BDD coverage: 34 scenarios in features/actor_run_tool_calling.feature covering
tool call success, multi-turn loop, no-skill regression, silent-drop fix, LLMCaller
internals, _build_tool_registry edge cases, last_run_tool_calls tallying,
tool name encoding/decoding, and LLM response decoding.
ISSUES CLOSED: #11211
275 lines
13 KiB
Gherkin
275 lines
13 KiB
Gherkin
@coverage @coverage_actor_run_tool_calling
|
|
Feature: Actor run tool-calling via ToolCallingRuntime
|
|
When a skill is attached to `agents actor run` via `--skill`, the actor
|
|
should perform real LLM tool calls through ToolCallingRuntime.
|
|
|
|
Background:
|
|
Given a minimal ToolCallingAgent fixture
|
|
|
|
# ---------- A: single tool call succeeds ----------
|
|
|
|
Scenario: Tool call succeeds — mock LLM returns read_file call
|
|
Given a mock LLM caller that makes one read_file tool call
|
|
When ToolCallingAgent.process is called with prompt "Review src/auth.py"
|
|
Then the tool_calls count is 1
|
|
And the final response is non-empty
|
|
|
|
# ---------- B: multi-turn tool loop ----------
|
|
|
|
Scenario: Multi-turn tool loop — mock LLM makes two consecutive tool calls
|
|
Given a mock LLM caller that makes 2 consecutive tool calls before finishing
|
|
When ToolCallingAgent.process is called with prompt "Analyse two files"
|
|
Then the tool_calls count is 2
|
|
And the loop ran for at least 3 iterations
|
|
|
|
# ---------- C: no-skill plain LLM regression ----------
|
|
|
|
Scenario: No-skill plain LLM — SimpleLLMAgent used, no tool schemas sent
|
|
Given a SimpleLLMAgent with a mock LLM
|
|
When SimpleLLMAgent.process is called with a simple prompt
|
|
Then SimpleLLMAgent returns the mocked response
|
|
And no tool schemas were sent to the LLM
|
|
|
|
# ---------- D: skill tools not silently dropped ----------
|
|
|
|
Scenario: Skill tools not silently dropped when actor has no base tools list
|
|
Given an actor config with no base tools list
|
|
And resolved skill tool entries for the actor
|
|
When _make_agent_instance is called for that actor
|
|
Then the result is a ToolCallingAgent instance
|
|
|
|
# ---------- E: ToolCallingLLMCaller first-call system prompt ----------
|
|
|
|
Scenario: ToolCallingLLMCaller sends system prompt on first call
|
|
Given a ToolCallingLLMCaller with an actor config containing a system_prompt
|
|
When invoke is called for the first time with a user prompt
|
|
Then the accumulated messages include a SystemMessage
|
|
And the accumulated messages include a HumanMessage
|
|
|
|
# ---------- F: ToolCallingLLMCaller subsequent call appends tool results ----------
|
|
|
|
Scenario: ToolCallingLLMCaller appends tool results on subsequent call
|
|
Given a ToolCallingLLMCaller with an actor config containing a system_prompt
|
|
And a prior first invoke has been made
|
|
When invoke is called again with tool_results
|
|
Then a ToolMessage is appended to the accumulated messages
|
|
|
|
# ---------- G: GraphExecutor passes context to ToolCallingAgent ----------
|
|
|
|
Scenario: GraphExecutor._invoke_agent passes context dict to ToolCallingAgent
|
|
Given a ToolCallingAgent spy that captures its process() arguments
|
|
When GraphExecutor._invoke_agent is called with that agent and a context dict
|
|
Then the context dict was forwarded to ToolCallingAgent.process
|
|
|
|
# ---------- H: last_run_tool_calls property reflects run results ----------
|
|
|
|
Scenario: ReactiveCleverAgentsApp.last_run_tool_calls reflects tool calls
|
|
Given a ReactiveCleverAgentsApp with a registered ToolCallingAgent that has last_result
|
|
When _tally_tool_calls is called on the app
|
|
Then last_run_tool_calls equals the number of tool call history entries
|
|
|
|
# ---------- I: ToolCallingLLMCaller _render_prompt ----------
|
|
|
|
Scenario: ToolCallingLLMCaller._render_prompt handles empty template
|
|
Given a fresh ToolCallingLLMCaller
|
|
When _render_prompt is called with an empty string template
|
|
Then _render_prompt returns an empty string
|
|
|
|
Scenario: ToolCallingLLMCaller._render_prompt renders Jinja2 template
|
|
Given a fresh ToolCallingLLMCaller
|
|
When _render_prompt is called with a Jinja2 template and context
|
|
Then _render_prompt returns the rendered string
|
|
|
|
Scenario: ToolCallingLLMCaller._render_prompt handles rendering exception
|
|
Given a fresh ToolCallingLLMCaller
|
|
When _render_prompt is called with a template that raises during rendering
|
|
Then _render_prompt returns the original template
|
|
|
|
# ---------- J: ToolCallingLLMCaller _resolve_llm with kwargs ----------
|
|
|
|
Scenario: ToolCallingLLMCaller._resolve_llm passes temperature max_tokens max_retries to provider
|
|
Given a ToolCallingLLMCaller with temperature max_tokens max_retries in actor_config
|
|
When _resolve_llm is called with tool_schemas
|
|
Then the provider registry create_llm was called with the config kwargs
|
|
|
|
Scenario: ToolCallingLLMCaller._resolve_llm skips bind_tools when no schemas
|
|
Given a ToolCallingLLMCaller with temperature max_tokens max_retries in actor_config
|
|
When _resolve_llm is called without tool_schemas
|
|
Then bind_tools was not called on the LLM
|
|
|
|
Scenario: ToolCallingLLMCaller._resolve_llm falls back when bind_tools raises
|
|
Given a ToolCallingLLMCaller with a provider that raises on bind_tools
|
|
When _resolve_llm is called with tool_schemas for a failing bind
|
|
Then the plain LLM is used without tool binding
|
|
|
|
# ---------- K: ToolCallingLLMCaller invoke no system prompt ----------
|
|
|
|
Scenario: ToolCallingLLMCaller.invoke without system prompt only adds HumanMessage
|
|
Given a ToolCallingLLMCaller with an actor config without system_prompt
|
|
When invoke is called for the first time with a user prompt
|
|
Then the accumulated messages include a HumanMessage
|
|
And no SystemMessage was added
|
|
|
|
# ---------- L: ToolCallingLLMCaller invoke list content ----------
|
|
|
|
Scenario: ToolCallingLLMCaller.invoke handles list content in response
|
|
Given a ToolCallingLLMCaller with an actor config without system_prompt
|
|
When invoke is called and the LLM returns list content
|
|
Then the response content joins the list parts
|
|
|
|
# ---------- M: ToolCallingLLMCaller invoke failed tool result ----------
|
|
|
|
Scenario: ToolCallingLLMCaller.invoke uses error field on failed tool result
|
|
Given a ToolCallingLLMCaller with an actor config containing a system_prompt
|
|
And a prior first invoke has been made
|
|
When invoke is called with a failed tool_result
|
|
Then a ToolMessage with the error text is appended
|
|
|
|
# ---------- N: ToolCallingAgent._build_tool_registry edge cases ----------
|
|
|
|
Scenario: ToolCallingAgent._build_tool_registry skips empty tool name
|
|
Given a ToolCallingAgent with an entry that has an empty name
|
|
When _build_tool_registry is called
|
|
Then the local registry is empty
|
|
|
|
Scenario: ToolCallingAgent._build_tool_registry skips duplicate names
|
|
Given a ToolCallingAgent with two entries for the same builtin tool
|
|
When _build_tool_registry is called
|
|
Then the local registry has exactly one tool registered
|
|
|
|
Scenario: ToolCallingAgent._build_tool_registry warns when tool not in builtin registry
|
|
Given a ToolCallingAgent with an entry not found in builtin registry
|
|
When _build_tool_registry is called
|
|
Then the local registry is empty
|
|
|
|
# ---------- O: ToolCallingAgent.process_message_sync ----------
|
|
|
|
Scenario: ToolCallingAgent.process_message_sync delegates to process
|
|
Given a ToolCallingAgent with no tools and a mock process method
|
|
When process_message_sync is called
|
|
Then the result is the same as calling process directly
|
|
|
|
# ---------- P: ToolCallingLLMCaller.invoke returns tool calls from response ----------
|
|
|
|
Scenario: ToolCallingLLMCaller.invoke extracts tool calls from LLM response
|
|
Given a ToolCallingLLMCaller with an actor config without system_prompt
|
|
When invoke is called and the LLM returns a response with tool call dicts
|
|
Then the LLMResponse contains the extracted tool calls
|
|
|
|
# ---------- Q: _resolve_provider_format maps provider to correct format ----------
|
|
|
|
Scenario: _resolve_provider_format returns ANTHROPIC for anthropic provider
|
|
Given an actor config with provider "anthropic"
|
|
When _resolve_provider_format is called with that actor config
|
|
Then _resolve_provider_format returns ProviderFormat.ANTHROPIC
|
|
|
|
Scenario: _resolve_provider_format returns OPENAI for openai provider
|
|
Given an actor config with provider "openai"
|
|
When _resolve_provider_format is called with that actor config
|
|
Then _resolve_provider_format returns ProviderFormat.OPENAI
|
|
|
|
Scenario: _resolve_provider_format returns OPENAI for unknown provider
|
|
Given an actor config with provider "google"
|
|
When _resolve_provider_format is called with that actor config
|
|
Then _resolve_provider_format returns ProviderFormat.OPENAI
|
|
|
|
Scenario: _resolve_provider_format returns OPENAI for empty actor config
|
|
Given an empty actor config
|
|
When _resolve_provider_format is called with that actor config
|
|
Then _resolve_provider_format returns ProviderFormat.OPENAI
|
|
|
|
Scenario: _resolve_provider_format returns OPENAI for None actor config
|
|
When _resolve_provider_format is called with actor_config None
|
|
Then _resolve_provider_format returns ProviderFormat.OPENAI
|
|
|
|
# ---------- R: ToolCallingAgent.process passes provider_format to runtime ----------
|
|
|
|
Scenario: ToolCallingAgent.process passes provider_format to ToolCallingRuntime
|
|
Given a ToolCallingAgent with an anthropic actor config and a file-read tool entry
|
|
When ToolCallingAgent.process is called with prompt "Read my file"
|
|
Then ToolCallingRuntime was constructed with ProviderFormat.ANTHROPIC
|
|
|
|
# ---------- S: Tool name encoding/decoding ----------
|
|
|
|
Scenario: _encode_tool_name replaces colon and slash with uppercase sentinels
|
|
When _encode_tool_name is called with "server:namespace/tool"
|
|
Then _encode_tool_name returns "server_C_namespace_S_tool"
|
|
|
|
Scenario: _encode_tool_name is idempotent on already-clean names
|
|
When _encode_tool_name is called with "my-clean_tool"
|
|
Then _encode_tool_name returns "my-clean_tool"
|
|
|
|
Scenario: _decode_tool_name restores colon and slash from sentinels
|
|
When _decode_tool_name is called with "server_C_namespace_S_tool"
|
|
Then _decode_tool_name returns "server:namespace/tool"
|
|
|
|
Scenario: _decode_tool_name is idempotent on clean names
|
|
When _decode_tool_name is called with "my-clean_tool"
|
|
Then _decode_tool_name returns "my-clean_tool"
|
|
|
|
Scenario: encode-decode round-trip preserves original name
|
|
Given a tool name "local/my__tool"
|
|
When the tool name is encoded then decoded
|
|
Then the round-trip name equals "local/my__tool"
|
|
|
|
Scenario: encode-decode round-trip for server-qualified name
|
|
Given a tool name "server.example:builtin/git-status"
|
|
When the tool name is encoded then decoded
|
|
Then the round-trip name equals "server.example:builtin/git-status"
|
|
|
|
# ---------- T: _resolve_llm encodes tool names in schemas ----------
|
|
|
|
Scenario: _resolve_llm encodes tool names before calling bind_tools
|
|
Given a ToolCallingLLMCaller with default actor config
|
|
When _resolve_llm is called with a schema containing a namespaced tool name
|
|
Then bind_tools was called with the encoded tool name
|
|
|
|
# ---------- U: invoke decodes tool names from LLM response ----------
|
|
|
|
Scenario: invoke decodes tool names when extracting tool calls from LLM response
|
|
Given a ToolCallingLLMCaller with an actor config without system_prompt
|
|
When invoke is called and the LLM returns a response with encoded tool call names
|
|
Then the LLMResponse contains the decoded tool calls
|
|
|
|
# ---------- S: Tool name encoding/decoding ----------
|
|
|
|
Scenario: _encode_tool_name replaces colon and slash with uppercase sentinels
|
|
When _encode_tool_name is called with "server:namespace/tool"
|
|
Then _encode_tool_name returns "server_C_namespace_S_tool"
|
|
|
|
Scenario: _encode_tool_name is idempotent on already-clean names
|
|
When _encode_tool_name is called with "my-clean_tool"
|
|
Then _encode_tool_name returns "my-clean_tool"
|
|
|
|
Scenario: _decode_tool_name restores colon and slash from sentinels
|
|
When _decode_tool_name is called with "server_C_namespace_S_tool"
|
|
Then _decode_tool_name returns "server:namespace/tool"
|
|
|
|
Scenario: _decode_tool_name is idempotent on clean names
|
|
When _decode_tool_name is called with "my-clean_tool"
|
|
Then _decode_tool_name returns "my-clean_tool"
|
|
|
|
Scenario: encode-decode round-trip preserves original name
|
|
Given a tool name "local/my__tool"
|
|
When the tool name is encoded then decoded
|
|
Then the round-trip name equals "local/my__tool"
|
|
|
|
Scenario: encode-decode round-trip for server-qualified name
|
|
Given a tool name "server.example:builtin/git-status"
|
|
When the tool name is encoded then decoded
|
|
Then the round-trip name equals "server.example:builtin/git-status"
|
|
|
|
# ---------- T: _resolve_llm encodes tool names in schemas ----------
|
|
|
|
Scenario: _resolve_llm encodes tool names before calling bind_tools
|
|
Given a ToolCallingLLMCaller with default actor config
|
|
When _resolve_llm is called with a schema containing a namespaced tool name
|
|
Then bind_tools was called with the encoded tool name
|
|
|
|
# ---------- U: invoke decodes tool names from LLM response ----------
|
|
|
|
Scenario: invoke decodes tool names when extracting tool calls from LLM response
|
|
Given a ToolCallingLLMCaller with an actor config without system_prompt
|
|
When invoke is called and the LLM returns a response with encoded tool call names
|
|
Then the LLMResponse contains the decoded tool calls
|