cleveragents/cleveragents-core

Fork 3

Bug: ToolCallingLLMCaller ignores actor options block, causing wrong LLM provider when skill is used #11243

New issue

Closed

opened 2026-05-18 07:11:52 +00:00 by hurui200320 · 2 comments

hurui200320 commented

2026-05-18 07:11:52 +00:00

Member

Metadata

Commit Message: fix(reactive): forward actor options block in ToolCallingLLMCaller._resolve_llm
Branch: bugfix/m7-tool-calling-llm-options

Background

PR #11225 introduced support for custom provider options in actor configuration (e.g. openai_api_base and openai_api_key to redirect an OpenAI-protocol actor to a local llama-swap/llama.cpp backend). This fix was applied to SimpleLLMAgent._resolve_llm() in stream_router.py.

However, PR #11211 (merged after #11225) introduced a parallel LLM resolution path — ToolCallingLLMCaller._resolve_llm() in tool_caller.py — which is used whenever an actor is run with tools attached (i.e. via --skill). This new class was written from scratch and does not include the options-block forwarding logic that was added to SimpleLLMAgent.

Current Behavior

Running an actor without --skill correctly routes to the local backend:

uv run agents actor run llama-cpp-gemma4 "Hi"
# → works fine, llama-swap captures the request

Running the same actor with --skill sends the request to OpenAI instead:

uv run agents actor run llama-cpp-gemma4 "Hi" --skill local/built-in-tools
# Unexpected error: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-...

Root Cause

When --skill is passed, the actor has tools attached and _make_agent_instance() creates a ToolCallingAgent instead of a SimpleLLMAgent. ToolCallingAgent.process() delegates LLM resolution to ToolCallingLLMCaller._resolve_llm() in src/cleveragents/reactive/tool_caller.py.

That method reads temperature, max_tokens, and max_retries from the actor config, but never reads the options block. As a result:

openai_api_base is never forwarded to ChatOpenAI(base_url=...)
openai_api_key is never extracted as __api_key_sentinel
The provider registry falls back to reading OPENAI_API_KEY from the environment and sends the request to api.openai.com

The fix exists in SimpleLLMAgent._resolve_llm() (added by PR #11225 / commit b3851693) but was never ported to ToolCallingLLMCaller._resolve_llm().

Expected Behavior

Running an actor with --skill should use the same LLM backend as running without --skill. The options block in the actor config (including openai_api_base and openai_api_key) must be respected regardless of whether tools are attached.

Reproduction Steps

Create an actor with provider: openai, a custom openai_api_base pointing to a local llama-swap instance, and openai_api_key: none
Run uv run agents actor run <actor-name> "Hi" — succeeds, uses local backend
Run uv run agents actor run <actor-name> "Hi" --skill <any-skill> — fails with OpenAI 401

Acceptance Criteria

ToolCallingLLMCaller._resolve_llm() reads the options block from the actor config
openai_api_base from options is forwarded to the LLM client as base_url
openai_api_key from options is extracted as __api_key_sentinel (overrides env var)
Other allowed options (timeout, top_p, frequency_penalty, presence_penalty) are also forwarded
Reserved keys (provider_type, model_id) are rejected with a warning
Unknown keys are rejected with a warning
Running an actor with --skill and a custom openai_api_base routes to the correct local backend
BDD regression test added with @tdd_issue + @tdd_issue_N tags

Supporting Information

Affected file: src/cleveragents/reactive/tool_caller.py, ToolCallingLLMCaller._resolve_llm() (~line 117)
Reference implementation: src/cleveragents/reactive/stream_router.py, SimpleLLMAgent._resolve_llm() (~line 214)
Introduced by: PR #11211 (ToolCallingAgent / skill support)
Fix exists in: PR #11225 (options block for SimpleLLMAgent) — needs porting

Subtasks

Port options-block merging logic from SimpleLLMAgent._resolve_llm() to ToolCallingLLMCaller._resolve_llm()
Write BDD scenario reproducing the bug (with @tdd_issue, @tdd_issue_N, no @tdd_expected_fail since this is fix+TDD in one PR)
Verify fix: actor with custom openai_api_base + --skill routes to local backend
Ensure all quality gates pass (nox)

Definition of Done

ToolCallingLLMCaller._resolve_llm() forwards the full options block identically to SimpleLLMAgent._resolve_llm()
BDD regression test present and passing
nox full suite green with coverage ≥ 97%
PR merged and this issue closed

## Metadata - **Commit Message:** `fix(reactive): forward actor options block in ToolCallingLLMCaller._resolve_llm` - **Branch:** `bugfix/m7-tool-calling-llm-options` ## Background PR #11225 introduced support for custom provider options in actor configuration (e.g. `openai_api_base` and `openai_api_key` to redirect an OpenAI-protocol actor to a local llama-swap/llama.cpp backend). This fix was applied to `SimpleLLMAgent._resolve_llm()` in `stream_router.py`. However, PR #11211 (merged after #11225) introduced a parallel LLM resolution path — `ToolCallingLLMCaller._resolve_llm()` in `tool_caller.py` — which is used whenever an actor is run with tools attached (i.e. via `--skill`). This new class was written from scratch and **does not include the options-block forwarding logic** that was added to `SimpleLLMAgent`. ## Current Behavior Running an actor **without** `--skill` correctly routes to the local backend: ``` uv run agents actor run llama-cpp-gemma4 "Hi" # → works fine, llama-swap captures the request ``` Running the **same actor with** `--skill` sends the request to OpenAI instead: ``` uv run agents actor run llama-cpp-gemma4 "Hi" --skill local/built-in-tools # Unexpected error: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-... ``` ## Root Cause When `--skill` is passed, the actor has tools attached and `_make_agent_instance()` creates a `ToolCallingAgent` instead of a `SimpleLLMAgent`. `ToolCallingAgent.process()` delegates LLM resolution to `ToolCallingLLMCaller._resolve_llm()` in `src/cleveragents/reactive/tool_caller.py`. That method reads `temperature`, `max_tokens`, and `max_retries` from the actor config, but **never reads the `options` block**. As a result: - `openai_api_base` is never forwarded to `ChatOpenAI(base_url=...)` - `openai_api_key` is never extracted as `__api_key_sentinel` - The provider registry falls back to reading `OPENAI_API_KEY` from the environment and sends the request to `api.openai.com` The fix exists in `SimpleLLMAgent._resolve_llm()` (added by PR #11225 / commit `b3851693`) but was never ported to `ToolCallingLLMCaller._resolve_llm()`. ## Expected Behavior Running an actor with `--skill` should use the same LLM backend as running without `--skill`. The `options` block in the actor config (including `openai_api_base` and `openai_api_key`) must be respected regardless of whether tools are attached. ## Reproduction Steps 1. Create an actor with `provider: openai`, a custom `openai_api_base` pointing to a local llama-swap instance, and `openai_api_key: none` 2. Run `uv run agents actor run <actor-name> "Hi"` — succeeds, uses local backend 3. Run `uv run agents actor run <actor-name> "Hi" --skill <any-skill>` — fails with OpenAI 401 ## Acceptance Criteria - [x] `ToolCallingLLMCaller._resolve_llm()` reads the `options` block from the actor config - [x] `openai_api_base` from `options` is forwarded to the LLM client as `base_url` - [x] `openai_api_key` from `options` is extracted as `__api_key_sentinel` (overrides env var) - [x] Other allowed options (`timeout`, `top_p`, `frequency_penalty`, `presence_penalty`) are also forwarded - [x] Reserved keys (`provider_type`, `model_id`) are rejected with a warning - [x] Unknown keys are rejected with a warning - [x] Running an actor with `--skill` and a custom `openai_api_base` routes to the correct local backend - [x] BDD regression test added with `@tdd_issue` + `@tdd_issue_N` tags ## Supporting Information - **Affected file:** `src/cleveragents/reactive/tool_caller.py`, `ToolCallingLLMCaller._resolve_llm()` (~line 117) - **Reference implementation:** `src/cleveragents/reactive/stream_router.py`, `SimpleLLMAgent._resolve_llm()` (~line 214) - **Introduced by:** PR #11211 (ToolCallingAgent / skill support) - **Fix exists in:** PR #11225 (options block for SimpleLLMAgent) — needs porting ## Subtasks - [x] Port options-block merging logic from `SimpleLLMAgent._resolve_llm()` to `ToolCallingLLMCaller._resolve_llm()` - [x] Write BDD scenario reproducing the bug (with `@tdd_issue`, `@tdd_issue_N`, no `@tdd_expected_fail` since this is fix+TDD in one PR) - [x] Verify fix: actor with custom `openai_api_base` + `--skill` routes to local backend - [x] Ensure all quality gates pass (`nox`) ## Definition of Done - `ToolCallingLLMCaller._resolve_llm()` forwards the full `options` block identically to `SimpleLLMAgent._resolve_llm()` - BDD regression test present and passing - `nox` full suite green with coverage ≥ 97% - PR merged and this issue closed

hurui200320 added the

labels

2026-05-18 07:12:20 +00:00

hurui200320 added this to the v3.6.0 milestone

2026-05-18 07:12:28 +00:00

hurui200320 added a new dependency

2026-05-18 07:13:00 +00:00

#8369 Epic: Additional LLM Backends & Provider Integrations (M7)

hurui200320 added

and removed

labels

2026-05-18 07:13:26 +00:00

hurui200320 self-assigned this

2026-05-18 07:13:51 +00:00

hurui200320 added

and removed

labels

2026-05-18 07:24:44 +00:00

hurui200320 commented

2026-05-18 07:26:48 +00:00

Author

Member

Note: This ticket does not have a TDD counterpart. I will implement both TDD and fix in the same PR.

hurui200320 commented

2026-05-18 08:44:36 +00:00

Author

Member

Implementation Notes

Root Cause Confirmed

As described in the issue, ToolCallingLLMCaller._resolve_llm() in cleveragents.reactive.tool_caller was missing the options-block forwarding logic that exists in SimpleLLMAgent._resolve_llm() in cleveragents.reactive.stream_router. The reference implementation was added by PR #11225 / commit b3851693 but was not ported when ToolCallingLLMCaller was introduced in PR #11211.

Fix Applied

File: src/cleveragents/reactive/tool_caller.py
Method: ToolCallingLLMCaller._resolve_llm()

Ported the full options-block merging logic from SimpleLLMAgent._resolve_llm(), placed after the existing top-level config extraction (temperature, max_tokens, max_retries). This preserves the precedence rule: top-level config keys win over identically-named options block keys.

Key behaviours now identical to SimpleLLMAgent:

openai_api_key in options is extracted and forwarded as __api_key_sentinel (removed from options dict before the loop to avoid double-processing)
openai_api_base, timeout, top_p, frequency_penalty, presence_penalty from options are forwarded to create_llm() if not already set by top-level keys
Reserved keys (provider_type, model_id) are rejected with a logger.warning call (not logger_sr.warning — the module-level logger is logger in tool_caller.py)
Unknown keys are rejected with a logger.warning call citing allowed keys

TDD Approach

Since this is a bug ticket with no prior TDD ticket, both tests and fix are included in the same PR (no @tdd_expected_fail tags).

Feature file: features/actor_run_tool_calling.feature
Steps file: features/steps/actor_run_tool_calling_steps.py

Five new BDD scenarios added under section # ---------- V: ToolCallingLLMCaller options block forwarding (#11243) ----------:

V1: openai_api_base and openai_api_key from options forwarded correctly
V2: Allowed extra keys (timeout, top_p, frequency_penalty, presence_penalty) forwarded
V3: Reserved key provider_type rejected (call succeeds without TypeError)
V4: Unknown key foo_bar not forwarded to create_llm()
V5: Top-level temperature takes precedence over options.temperature

All 5 new scenarios are tagged @tdd_issue @tdd_issue_11243 and pass after the fix.

Quality Gates

nox -e lint: ✅ PASS
nox -e typecheck: ✅ PASS (0 errors, 3 pre-existing warnings from optional deps)
nox -e unit_tests: ✅ PASS (701 features, 15822 scenarios, 0 failures)
nox -e integration_tests: ✅ PASS (1999 tests, 1999 passed)
nox -e coverage_report: ✅ PASS (97% coverage, threshold met)

## Implementation Notes ### Root Cause Confirmed As described in the issue, `ToolCallingLLMCaller._resolve_llm()` in `cleveragents.reactive.tool_caller` was missing the options-block forwarding logic that exists in `SimpleLLMAgent._resolve_llm()` in `cleveragents.reactive.stream_router`. The reference implementation was added by PR #11225 / commit `b3851693` but was not ported when `ToolCallingLLMCaller` was introduced in PR #11211. ### Fix Applied **File:** `src/cleveragents/reactive/tool_caller.py` **Method:** `ToolCallingLLMCaller._resolve_llm()` Ported the full options-block merging logic from `SimpleLLMAgent._resolve_llm()`, placed after the existing top-level config extraction (temperature, max_tokens, max_retries). This preserves the precedence rule: top-level config keys win over identically-named options block keys. Key behaviours now identical to `SimpleLLMAgent`: 1. `openai_api_key` in options is extracted and forwarded as `__api_key_sentinel` (removed from options dict before the loop to avoid double-processing) 2. `openai_api_base`, `timeout`, `top_p`, `frequency_penalty`, `presence_penalty` from options are forwarded to `create_llm()` if not already set by top-level keys 3. Reserved keys (`provider_type`, `model_id`) are rejected with a `logger.warning` call (not `logger_sr.warning` — the module-level logger is `logger` in `tool_caller.py`) 4. Unknown keys are rejected with a `logger.warning` call citing allowed keys ### TDD Approach Since this is a bug ticket with no prior TDD ticket, both tests and fix are included in the same PR (no `@tdd_expected_fail` tags). **Feature file:** `features/actor_run_tool_calling.feature` **Steps file:** `features/steps/actor_run_tool_calling_steps.py` Five new BDD scenarios added under section `# ---------- V: ToolCallingLLMCaller options block forwarding (#11243) ----------`: - `V1`: `openai_api_base` and `openai_api_key` from options forwarded correctly - `V2`: Allowed extra keys (`timeout`, `top_p`, `frequency_penalty`, `presence_penalty`) forwarded - `V3`: Reserved key `provider_type` rejected (call succeeds without TypeError) - `V4`: Unknown key `foo_bar` not forwarded to `create_llm()` - `V5`: Top-level `temperature` takes precedence over `options.temperature` All 5 new scenarios are tagged `@tdd_issue @tdd_issue_11243` and pass after the fix. ### Quality Gates - `nox -e lint`: ✅ PASS - `nox -e typecheck`: ✅ PASS (0 errors, 3 pre-existing warnings from optional deps) - `nox -e unit_tests`: ✅ PASS (701 features, 15822 scenarios, 0 failures) - `nox -e integration_tests`: ✅ PASS (1999 tests, 1999 passed) - `nox -e coverage_report`: ✅ PASS (97% coverage, threshold met)

hurui200320 referenced this issue from a commit

2026-05-18 08:45:01 +00:00

fix(reactive): forward actor options block in ToolCallingLLMCaller._resolve_llm

hurui200320 referenced this issue from a pull request that will close it,

2026-05-18 08:45:32 +00:00

fix(reactive): forward actor options block in ToolCallingLLMCaller._resolve_llm #11244

hurui200320 added a new dependency

2026-05-18 08:46:35 +00:00

#11244 fix(reactive): forward actor options block in ToolCallingLLMCaller._resolve_llm

hurui200320 added

State

In Review