hurui200320
91691750ed
CI / push-validation (pull_request) Successful in 35s
CI / helm (pull_request) Successful in 42s
CI / build (pull_request) Successful in 47s
CI / lint (pull_request) Successful in 1m10s
CI / typecheck (pull_request) Successful in 1m11s
CI / quality (pull_request) Successful in 1m10s
CI / security (pull_request) Successful in 1m37s
CI / integration_tests (pull_request) Failing after 4m9s
CI / unit_tests (pull_request) Successful in 4m59s
CI / docker (pull_request) Successful in 1m25s
CI / coverage (pull_request) Successful in 11m7s
CI / status-check (pull_request) Failing after 3s
Port the options-block merging logic from SimpleLLMAgent._resolve_llm()
(stream_router.py, added in PR #11225 / commit b3851693) to the parallel
resolution path ToolCallingLLMCaller._resolve_llm() in tool_caller.py.
Without this fix, running an actor with --skill would ignore the options
block in the actor YAML, causing openai_api_base and openai_api_key to be
silently dropped. The provider registry then fell back to OPENAI_API_KEY
from the environment, routing requests to api.openai.com instead of the
configured local llama-swap/llama.cpp backend (HTTP 401).
Changes:
- tool_caller.ToolCallingLLMCaller._resolve_llm: read actor_config.options;
extract openai_api_key -> __api_key_sentinel; forward allowed keys
(openai_api_base, timeout, top_p, frequency_penalty, presence_penalty);
reject reserved keys (provider_type, model_id) and unknown keys with
logger.warning, matching SimpleLLMAgent behaviour exactly.
- features/actor_run_tool_calling.feature: add five BDD regression
scenarios (section V, tagged @tdd_issue @tdd_issue_11243) covering
api_base+key forwarding, allowed extra keys, reserved key rejection,
unknown key rejection, and top-level precedence over options.
- features/steps/actor_run_tool_calling_steps.py: add corresponding step
definitions for all five scenarios.
All nox quality gates pass (lint, typecheck, unit_tests 15822/0 fail,
integration_tests 1999/0 fail, coverage_report 97%).
ISSUES CLOSED: #11243
308 lines
15 KiB
Gherkin
308 lines
15 KiB
Gherkin
@coverage @coverage_actor_run_tool_calling
|
|
Feature: Actor run tool-calling via ToolCallingRuntime
|
|
When a skill is attached to `agents actor run` via `--skill`, the actor
|
|
should perform real LLM tool calls through ToolCallingRuntime.
|
|
|
|
Background:
|
|
Given a minimal ToolCallingAgent fixture
|
|
|
|
# ---------- A: single tool call succeeds ----------
|
|
|
|
Scenario: Tool call succeeds — mock LLM returns read_file call
|
|
Given a mock LLM caller that makes one read_file tool call
|
|
When ToolCallingAgent.process is called with prompt "Review src/auth.py"
|
|
Then the tool_calls count is 1
|
|
And the final response is non-empty
|
|
|
|
# ---------- B: multi-turn tool loop ----------
|
|
|
|
Scenario: Multi-turn tool loop — mock LLM makes two consecutive tool calls
|
|
Given a mock LLM caller that makes 2 consecutive tool calls before finishing
|
|
When ToolCallingAgent.process is called with prompt "Analyse two files"
|
|
Then the tool_calls count is 2
|
|
And the loop ran for at least 3 iterations
|
|
|
|
# ---------- C: no-skill plain LLM regression ----------
|
|
|
|
Scenario: No-skill plain LLM — SimpleLLMAgent used, no tool schemas sent
|
|
Given a SimpleLLMAgent with a mock LLM
|
|
When SimpleLLMAgent.process is called with a simple prompt
|
|
Then SimpleLLMAgent returns the mocked response
|
|
And no tool schemas were sent to the LLM
|
|
|
|
# ---------- D: skill tools not silently dropped ----------
|
|
|
|
Scenario: Skill tools not silently dropped when actor has no base tools list
|
|
Given an actor config with no base tools list
|
|
And resolved skill tool entries for the actor
|
|
When _make_agent_instance is called for that actor
|
|
Then the result is a ToolCallingAgent instance
|
|
|
|
# ---------- E: ToolCallingLLMCaller first-call system prompt ----------
|
|
|
|
Scenario: ToolCallingLLMCaller sends system prompt on first call
|
|
Given a ToolCallingLLMCaller with an actor config containing a system_prompt
|
|
When invoke is called for the first time with a user prompt
|
|
Then the accumulated messages include a SystemMessage
|
|
And the accumulated messages include a HumanMessage
|
|
|
|
# ---------- F: ToolCallingLLMCaller subsequent call appends tool results ----------
|
|
|
|
Scenario: ToolCallingLLMCaller appends tool results on subsequent call
|
|
Given a ToolCallingLLMCaller with an actor config containing a system_prompt
|
|
And a prior first invoke has been made
|
|
When invoke is called again with tool_results
|
|
Then a ToolMessage is appended to the accumulated messages
|
|
|
|
# ---------- G: GraphExecutor passes context to ToolCallingAgent ----------
|
|
|
|
Scenario: GraphExecutor._invoke_agent passes context dict to ToolCallingAgent
|
|
Given a ToolCallingAgent spy that captures its process() arguments
|
|
When GraphExecutor._invoke_agent is called with that agent and a context dict
|
|
Then the context dict was forwarded to ToolCallingAgent.process
|
|
|
|
# ---------- H: last_run_tool_calls property reflects run results ----------
|
|
|
|
Scenario: ReactiveCleverAgentsApp.last_run_tool_calls reflects tool calls
|
|
Given a ReactiveCleverAgentsApp with a registered ToolCallingAgent that has last_result
|
|
When _tally_tool_calls is called on the app
|
|
Then last_run_tool_calls equals the number of tool call history entries
|
|
|
|
# ---------- I: ToolCallingLLMCaller _render_prompt ----------
|
|
|
|
Scenario: ToolCallingLLMCaller._render_prompt handles empty template
|
|
Given a fresh ToolCallingLLMCaller
|
|
When _render_prompt is called with an empty string template
|
|
Then _render_prompt returns an empty string
|
|
|
|
Scenario: ToolCallingLLMCaller._render_prompt renders Jinja2 template
|
|
Given a fresh ToolCallingLLMCaller
|
|
When _render_prompt is called with a Jinja2 template and context
|
|
Then _render_prompt returns the rendered string
|
|
|
|
Scenario: ToolCallingLLMCaller._render_prompt handles rendering exception
|
|
Given a fresh ToolCallingLLMCaller
|
|
When _render_prompt is called with a template that raises during rendering
|
|
Then _render_prompt returns the original template
|
|
|
|
# ---------- J: ToolCallingLLMCaller _resolve_llm with kwargs ----------
|
|
|
|
Scenario: ToolCallingLLMCaller._resolve_llm passes temperature max_tokens max_retries to provider
|
|
Given a ToolCallingLLMCaller with temperature max_tokens max_retries in actor_config
|
|
When _resolve_llm is called with tool_schemas
|
|
Then the provider registry create_llm was called with the config kwargs
|
|
|
|
Scenario: ToolCallingLLMCaller._resolve_llm skips bind_tools when no schemas
|
|
Given a ToolCallingLLMCaller with temperature max_tokens max_retries in actor_config
|
|
When _resolve_llm is called without tool_schemas
|
|
Then bind_tools was not called on the LLM
|
|
|
|
Scenario: ToolCallingLLMCaller._resolve_llm falls back when bind_tools raises
|
|
Given a ToolCallingLLMCaller with a provider that raises on bind_tools
|
|
When _resolve_llm is called with tool_schemas for a failing bind
|
|
Then the plain LLM is used without tool binding
|
|
|
|
# ---------- K: ToolCallingLLMCaller invoke no system prompt ----------
|
|
|
|
Scenario: ToolCallingLLMCaller.invoke without system prompt only adds HumanMessage
|
|
Given a ToolCallingLLMCaller with an actor config without system_prompt
|
|
When invoke is called for the first time with a user prompt
|
|
Then the accumulated messages include a HumanMessage
|
|
And no SystemMessage was added
|
|
|
|
# ---------- L: ToolCallingLLMCaller invoke list content ----------
|
|
|
|
Scenario: ToolCallingLLMCaller.invoke handles list content in response
|
|
Given a ToolCallingLLMCaller with an actor config without system_prompt
|
|
When invoke is called and the LLM returns list content
|
|
Then the response content joins the list parts
|
|
|
|
# ---------- M: ToolCallingLLMCaller invoke failed tool result ----------
|
|
|
|
Scenario: ToolCallingLLMCaller.invoke uses error field on failed tool result
|
|
Given a ToolCallingLLMCaller with an actor config containing a system_prompt
|
|
And a prior first invoke has been made
|
|
When invoke is called with a failed tool_result
|
|
Then a ToolMessage with the error text is appended
|
|
|
|
# ---------- N: ToolCallingAgent._build_tool_registry edge cases ----------
|
|
|
|
Scenario: ToolCallingAgent._build_tool_registry skips empty tool name
|
|
Given a ToolCallingAgent with an entry that has an empty name
|
|
When _build_tool_registry is called
|
|
Then the local registry is empty
|
|
|
|
Scenario: ToolCallingAgent._build_tool_registry skips duplicate names
|
|
Given a ToolCallingAgent with two entries for the same builtin tool
|
|
When _build_tool_registry is called
|
|
Then the local registry has exactly one tool registered
|
|
|
|
Scenario: ToolCallingAgent._build_tool_registry warns when tool not in builtin registry
|
|
Given a ToolCallingAgent with an entry not found in builtin registry
|
|
When _build_tool_registry is called
|
|
Then the local registry is empty
|
|
|
|
# ---------- O: ToolCallingAgent.process_message_sync ----------
|
|
|
|
Scenario: ToolCallingAgent.process_message_sync delegates to process
|
|
Given a ToolCallingAgent with no tools and a mock process method
|
|
When process_message_sync is called
|
|
Then the result is the same as calling process directly
|
|
|
|
# ---------- P: ToolCallingLLMCaller.invoke returns tool calls from response ----------
|
|
|
|
Scenario: ToolCallingLLMCaller.invoke extracts tool calls from LLM response
|
|
Given a ToolCallingLLMCaller with an actor config without system_prompt
|
|
When invoke is called and the LLM returns a response with tool call dicts
|
|
Then the LLMResponse contains the extracted tool calls
|
|
|
|
# ---------- Q: _resolve_provider_format maps provider to correct format ----------
|
|
|
|
Scenario: _resolve_provider_format returns ANTHROPIC for anthropic provider
|
|
Given an actor config with provider "anthropic"
|
|
When _resolve_provider_format is called with that actor config
|
|
Then _resolve_provider_format returns ProviderFormat.ANTHROPIC
|
|
|
|
Scenario: _resolve_provider_format returns OPENAI for openai provider
|
|
Given an actor config with provider "openai"
|
|
When _resolve_provider_format is called with that actor config
|
|
Then _resolve_provider_format returns ProviderFormat.OPENAI
|
|
|
|
Scenario: _resolve_provider_format returns OPENAI for unknown provider
|
|
Given an actor config with provider "google"
|
|
When _resolve_provider_format is called with that actor config
|
|
Then _resolve_provider_format returns ProviderFormat.OPENAI
|
|
|
|
Scenario: _resolve_provider_format returns OPENAI for empty actor config
|
|
Given an empty actor config
|
|
When _resolve_provider_format is called with that actor config
|
|
Then _resolve_provider_format returns ProviderFormat.OPENAI
|
|
|
|
Scenario: _resolve_provider_format returns OPENAI for None actor config
|
|
When _resolve_provider_format is called with actor_config None
|
|
Then _resolve_provider_format returns ProviderFormat.OPENAI
|
|
|
|
# ---------- R: ToolCallingAgent.process passes provider_format to runtime ----------
|
|
|
|
Scenario: ToolCallingAgent.process passes provider_format to ToolCallingRuntime
|
|
Given a ToolCallingAgent with an anthropic actor config and a file-read tool entry
|
|
When ToolCallingAgent.process is called with prompt "Read my file"
|
|
Then ToolCallingRuntime was constructed with ProviderFormat.ANTHROPIC
|
|
|
|
# ---------- S: Tool name encoding/decoding ----------
|
|
|
|
Scenario: _encode_tool_name replaces colon and slash with uppercase sentinels
|
|
When _encode_tool_name is called with "server:namespace/tool"
|
|
Then _encode_tool_name returns "server_C_namespace_S_tool"
|
|
|
|
Scenario: _encode_tool_name is idempotent on already-clean names
|
|
When _encode_tool_name is called with "my-clean_tool"
|
|
Then _encode_tool_name returns "my-clean_tool"
|
|
|
|
Scenario: _decode_tool_name restores colon and slash from sentinels
|
|
When _decode_tool_name is called with "server_C_namespace_S_tool"
|
|
Then _decode_tool_name returns "server:namespace/tool"
|
|
|
|
Scenario: _decode_tool_name is idempotent on clean names
|
|
When _decode_tool_name is called with "my-clean_tool"
|
|
Then _decode_tool_name returns "my-clean_tool"
|
|
|
|
Scenario: encode-decode round-trip preserves original name
|
|
Given a tool name "local/my__tool"
|
|
When the tool name is encoded then decoded
|
|
Then the round-trip name equals "local/my__tool"
|
|
|
|
Scenario: encode-decode round-trip for server-qualified name
|
|
Given a tool name "server.example:builtin/git-status"
|
|
When the tool name is encoded then decoded
|
|
Then the round-trip name equals "server.example:builtin/git-status"
|
|
|
|
# ---------- T: _resolve_llm encodes tool names in schemas ----------
|
|
|
|
Scenario: _resolve_llm encodes tool names before calling bind_tools
|
|
Given a ToolCallingLLMCaller with default actor config
|
|
When _resolve_llm is called with a schema containing a namespaced tool name
|
|
Then bind_tools was called with the encoded tool name
|
|
|
|
# ---------- U: invoke decodes tool names from LLM response ----------
|
|
|
|
Scenario: invoke decodes tool names when extracting tool calls from LLM response
|
|
Given a ToolCallingLLMCaller with an actor config without system_prompt
|
|
When invoke is called and the LLM returns a response with encoded tool call names
|
|
Then the LLMResponse contains the decoded tool calls
|
|
|
|
# ---------- S: Tool name encoding/decoding ----------
|
|
|
|
Scenario: _encode_tool_name replaces colon and slash with uppercase sentinels
|
|
When _encode_tool_name is called with "server:namespace/tool"
|
|
Then _encode_tool_name returns "server_C_namespace_S_tool"
|
|
|
|
Scenario: _encode_tool_name is idempotent on already-clean names
|
|
When _encode_tool_name is called with "my-clean_tool"
|
|
Then _encode_tool_name returns "my-clean_tool"
|
|
|
|
Scenario: _decode_tool_name restores colon and slash from sentinels
|
|
When _decode_tool_name is called with "server_C_namespace_S_tool"
|
|
Then _decode_tool_name returns "server:namespace/tool"
|
|
|
|
Scenario: _decode_tool_name is idempotent on clean names
|
|
When _decode_tool_name is called with "my-clean_tool"
|
|
Then _decode_tool_name returns "my-clean_tool"
|
|
|
|
Scenario: encode-decode round-trip preserves original name
|
|
Given a tool name "local/my__tool"
|
|
When the tool name is encoded then decoded
|
|
Then the round-trip name equals "local/my__tool"
|
|
|
|
Scenario: encode-decode round-trip for server-qualified name
|
|
Given a tool name "server.example:builtin/git-status"
|
|
When the tool name is encoded then decoded
|
|
Then the round-trip name equals "server.example:builtin/git-status"
|
|
|
|
# ---------- T: _resolve_llm encodes tool names in schemas ----------
|
|
|
|
Scenario: _resolve_llm encodes tool names before calling bind_tools
|
|
Given a ToolCallingLLMCaller with default actor config
|
|
When _resolve_llm is called with a schema containing a namespaced tool name
|
|
Then bind_tools was called with the encoded tool name
|
|
|
|
# ---------- U: invoke decodes tool names from LLM response ----------
|
|
|
|
Scenario: invoke decodes tool names when extracting tool calls from LLM response
|
|
Given a ToolCallingLLMCaller with an actor config without system_prompt
|
|
When invoke is called and the LLM returns a response with encoded tool call names
|
|
Then the LLMResponse contains the decoded tool calls
|
|
|
|
# ---------- V: ToolCallingLLMCaller options block forwarding (#11243) ----------
|
|
|
|
@tdd_issue @tdd_issue_11243
|
|
Scenario: ToolCallingLLMCaller forwards openai_api_base and openai_api_key from options block
|
|
Given a ToolCallingLLMCaller with actor config options containing openai_api_base and openai_api_key
|
|
When _resolve_llm is called on the ToolCallingLLMCaller with empty tool_schemas
|
|
Then create_llm was called with openai_api_base forwarded from options
|
|
And create_llm was called with __api_key_sentinel extracted from options openai_api_key
|
|
|
|
@tdd_issue @tdd_issue_11243
|
|
Scenario: ToolCallingLLMCaller forwards allowed options keys to create_llm
|
|
Given a ToolCallingLLMCaller with actor config options containing allowed extra keys
|
|
When _resolve_llm is called on the ToolCallingLLMCaller with empty tool_schemas
|
|
Then create_llm was called with the allowed options keys forwarded
|
|
|
|
@tdd_issue @tdd_issue_11243
|
|
Scenario: ToolCallingLLMCaller rejects reserved keys in options block with a warning
|
|
Given a ToolCallingLLMCaller with actor config options containing reserved key provider_type
|
|
When _resolve_llm is called on the ToolCallingLLMCaller with empty tool_schemas
|
|
Then create_llm was NOT called with the reserved key provider_type
|
|
|
|
@tdd_issue @tdd_issue_11243
|
|
Scenario: ToolCallingLLMCaller rejects unknown keys in options block with a warning
|
|
Given a ToolCallingLLMCaller with actor config options containing unknown key foo_bar
|
|
When _resolve_llm is called on the ToolCallingLLMCaller with empty tool_schemas
|
|
Then create_llm was NOT called with the unknown key foo_bar
|
|
|
|
@tdd_issue @tdd_issue_11243
|
|
Scenario: ToolCallingLLMCaller top-level config takes precedence over options block
|
|
Given a ToolCallingLLMCaller with actor config with top-level temperature and options temperature
|
|
When _resolve_llm is called on the ToolCallingLLMCaller with empty tool_schemas
|
|
Then create_llm was called with the top-level temperature value
|