cleveragents/cleveractors-core

Fork

You've already forked cleveractors-core

Code Issues 13 Pull requests 2 Projects Releases Packages Wiki Activity Actions

Add tool-call support to LLMAgent.stream_message #67

New issue

Closed

opened 2026-07-02 06:42:57 +00:00 by hurui200320 · 1 comment

hurui200320 commented

2026-07-02 06:42:57 +00:00

Member

Copy link

Metadata

Commit Message: feat(agents): add tool-call support to LLMAgent.stream_message
Branch: feature/m2-llm-agent-stream-tool-calls

Background

LLMAgent exposes two execution paths: process_message() (blocking, uses ainvoke()) and stream_message() (streaming, uses astream()). These two paths have diverged in capability: process_message() has a fully working multi-turn tool-call loop — it passes tools to the LLM, detects tool_calls on every response, dispatches to ephemeral ToolAgent instances, appends ToolMessages, re-invokes up to tool_max_rounds, handles token-budget exhaustion, runs a synthesis prompt when the model is stuck in tool-only mode, and supports tool-output pruning. stream_message() has none of this.

The fix is not to duplicate the tool-call logic into stream_message() — that would split one authoritative implementation into two and create a permanent maintenance burden. Instead, the tool-call loop should be extracted from process_message() into a shared private helper, _execute_tool_loop(), that both paths call. process_message() is refactored to delegate to the helper (no change in observable behaviour). stream_message() calls the same helper for tool-using actors (yielding the complete response as a single content chunk) and retains its existing astream() path unchanged for actors that declare no tools.

The single-chunk behaviour for tool-using actors is an accepted trade-off: when tools are involved, the user already waits for tool execution to complete before the final answer begins, so yielding that answer as a single chunk rather than token-by-token has minimal UX impact. The alternative — calling the LLM a second time with astream() after the helper has already produced the final response with ainvoke() — would double the LLM cost of every tool-using request and is not acceptable in a production billing context.

The cleveragents-webapp has a temporary router-side workaround in PR #328 (actor_config_has_tools + fallback to executor.execute() in _execute_via_cleveractors_stream and _generate_actor_sse) that silently degrades to non-streaming for tool-declaring actors. That workaround can be removed from the webapp once this issue is resolved.

Current Behaviour

Calling executor.execute_stream(message) on an actor whose LLM agent declares tools yields an empty response or plain-text hallucination. stream_message() calls self.chat_model.astream(lc_messages) without passing tools, so the LLM receives no tool schemas and emits no tool calls. The stream completes immediately with no useful output.

Additionally, process_message() contains the entire multi-turn tool-call loop inline (~200 lines), with no separation between the "tool orchestration" concern and the "final text retrieval" concern, making the method hard to unit-test in isolation and impossible to reuse from other callers.

Expected Behaviour

A new private async method _execute_tool_loop() on LLMAgent encapsulates the complete tool-call orchestration: passing tools to the LLM, running the multi-turn loop via ainvoke(), dispatching tool calls through ephemeral ToolAgent instances, accumulating token counts across all rounds, handling token-budget exhaustion and the synthesis prompt, and running the tool-output pruning pass. It returns a result object containing the final AIMessage response and the accumulated (prompt_tokens, completion_tokens).

process_message() delegates to _execute_tool_loop() for the tool-call rounds and extracts str(final_response.content) from the result. Observable behaviour is identical to today.

stream_message() branches on whether the actor declares tools:

No tools configured: the existing astream() path runs unchanged — real token-by-token streaming is preserved.
Tools configured: _execute_tool_loop() runs all rounds via ainvoke(); stream_message() then yields the final response text as a single content chunk and sets _last_token_usage / last_token_usage_var from the accumulated token counts returned by the helper.

All billing-integrity guarantees (populating last_token_usage_var and the _captured_prompt / _captured_completion sentinels on exception paths) are met for both callers of _execute_tool_loop().

Acceptance Criteria

executor.execute_stream(message) on a tool-declaring single-LLM actor yields at least one non-empty string token and executor.last_result reports non-zero accumulated token counts covering all tool-call rounds.
When the LLM requires two tool-call rounds before the final answer, execute_stream yields the final answer and executor.last_result.prompt_tokens equals the sum across all three invocations (two tool rounds + final).
When tool_max_rounds is exhausted, the generator terminates cleanly and executor.last_result is populated.
When the token budget is exceeded mid-loop, the synthesis flow inside the helper runs and the generator yields the synthesis answer.
When the model is stuck in tool-only mode (empty final text), the synthesis prompt flow inside the helper runs and the generator yields the synthesis answer.
stream_message() on an actor with no tools configured behaves identically to the current implementation — real astream() token-by-token streaming, same token counts.
process_message() behaviour is unchanged for both tool-using and non-tool actors.
On any exception path (tool dispatch error, ConfigurationError, LangChain error), executor.last_result is populated with accumulated token counts before the exception propagates.
_execute_tool_loop() has dedicated Behave coverage for each of its internal branches, independently of process_message() and stream_message().
nox -s typecheck reports zero errors and zero warnings on all changed files.
nox -s coverage_report reports ≥ 97% coverage.

Supporting Information

Root file: src/cleveractors/agents/llm.py.
The tool-call loop to extract lives in process_message() (lines ~831–1303). It is tightly coupled to several local variables (_accumulated_prompt, _captured_prompt, _budget_exhausted, messages, response); the extraction must design a clean return type (a private dataclass or named tuple) that surfaces all state needed by both callers without leaking internal implementation details.
Billing-integrity sentinels (_captured_prompt / _captured_completion) in process_message() and stream_message() must be updated to read accumulated counts from the helper's return value rather than re-reading local variables.
cleveragents-webapp workaround to remove after fix: actor_config_has_tools + stream_fallback_to_non_stream branch in PR cleveragents/cleveragents-webapp#328 (src/cleverthis/api/routes/v1_chat.py and src/cleverthis/services/chat_stream.py).
Effort estimate: ~4–5 developer-days (helper extraction + process_message() refactor ~2d, stream_message() integration ~0.5d, Behave scenarios ~1.5d, Robot test ~0.5d, coverage + PR ~0.5d).

Subtasks

Design the return type for _execute_tool_loop(): a private dataclass (e.g. _ToolLoopResult) carrying final_response, accumulated_prompt, accumulated_completion, and any flags needed by callers (budget_exhausted, synthesis_was_run).
Extract _execute_tool_loop() from process_message(): move the multi-turn ainvoke() loop, per-tool ToolAgent dispatch, token accumulation, budget exhaustion path, synthesis prompt flow, and tool-output pruning pass into the new method. _execute_tool_loop() must populate its result correctly on both success and exception paths.
Refactor process_message() to call _execute_tool_loop() and derive response_text from result.final_response.content. All existing tests must continue to pass without modification.
Update stream_message() to call _execute_tool_loop() when self._lc_tools is not None, yield str(result.final_response.content) as a single content chunk, and set _last_token_usage / last_token_usage_var from result.accumulated_prompt / result.accumulated_completion. The existing no-tools astream() branch must remain untouched.
Update the _captured_prompt / _captured_completion billing-integrity sentinels in both callers to correctly reflect accumulated counts from the helper on exception paths.
Tests (Behave) — _execute_tool_loop() internals: basic single tool round; multi-round (two consecutive tool-call rounds); tool_max_rounds exhaustion; token accumulation spanning all rounds; token-budget exhaustion (synthesis path runs); model stuck in tool-only mode (synthesis prompt flow); tool-output pruning (pass runs and token costs included); tool dispatch error (ExecutionError captured as ToolMessage, loop continues); exception path (partial token counts preserved in result).
Tests (Behave) — stream_message() integration: tool-using actor yields one non-empty chunk and last_result token counts equal helper's accumulated totals; no-tools actor behaviour unchanged (real astream tokens).
Tests (Behave) — process_message() regression: confirm existing tool-calling scenarios continue to pass after refactor (no new scenarios required if existing coverage is sufficient).
Tests (Robot): add at least one integration test exercising execute_stream() on an actor with a tool config (stub provider; no real API key required).
Run nox -s coverage_report and confirm ≥ 97%.
Run nox (all default sessions) and confirm all pass with zero errors.

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A single Git commit is created whose first line matches the Commit Message in the Metadata section exactly.
The commit is pushed to the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged.

## Metadata - **Commit Message:** `feat(agents): add tool-call support to LLMAgent.stream_message` - **Branch:** `feature/m2-llm-agent-stream-tool-calls` ## Background `LLMAgent` exposes two execution paths: `process_message()` (blocking, uses `ainvoke()`) and `stream_message()` (streaming, uses `astream()`). These two paths have diverged in capability: `process_message()` has a fully working multi-turn tool-call loop — it passes `tools` to the LLM, detects `tool_calls` on every response, dispatches to ephemeral `ToolAgent` instances, appends `ToolMessage`s, re-invokes up to `tool_max_rounds`, handles token-budget exhaustion, runs a synthesis prompt when the model is stuck in tool-only mode, and supports tool-output pruning. `stream_message()` has none of this. The fix is **not** to duplicate the tool-call logic into `stream_message()` — that would split one authoritative implementation into two and create a permanent maintenance burden. Instead, the tool-call loop should be extracted from `process_message()` into a shared private helper, `_execute_tool_loop()`, that both paths call. `process_message()` is refactored to delegate to the helper (no change in observable behaviour). `stream_message()` calls the same helper for tool-using actors (yielding the complete response as a single content chunk) and retains its existing `astream()` path unchanged for actors that declare no tools. The single-chunk behaviour for tool-using actors is an accepted trade-off: when tools are involved, the user already waits for tool execution to complete before the final answer begins, so yielding that answer as a single chunk rather than token-by-token has minimal UX impact. The alternative — calling the LLM a second time with `astream()` after the helper has already produced the final response with `ainvoke()` — would double the LLM cost of every tool-using request and is not acceptable in a production billing context. The cleveragents-webapp has a temporary router-side workaround in PR #328 (`actor_config_has_tools` + fallback to `executor.execute()` in `_execute_via_cleveractors_stream` and `_generate_actor_sse`) that silently degrades to non-streaming for tool-declaring actors. That workaround can be removed from the webapp once this issue is resolved. ## Current Behaviour Calling `executor.execute_stream(message)` on an actor whose LLM agent declares `tools` yields an empty response or plain-text hallucination. `stream_message()` calls `self.chat_model.astream(lc_messages)` without passing `tools`, so the LLM receives no tool schemas and emits no tool calls. The stream completes immediately with no useful output. Additionally, `process_message()` contains the entire multi-turn tool-call loop inline (~200 lines), with no separation between the "tool orchestration" concern and the "final text retrieval" concern, making the method hard to unit-test in isolation and impossible to reuse from other callers. ## Expected Behaviour A new private async method `_execute_tool_loop()` on `LLMAgent` encapsulates the complete tool-call orchestration: passing `tools` to the LLM, running the multi-turn loop via `ainvoke()`, dispatching tool calls through ephemeral `ToolAgent` instances, accumulating token counts across all rounds, handling token-budget exhaustion and the synthesis prompt, and running the tool-output pruning pass. It returns a result object containing the final `AIMessage` response and the accumulated `(prompt_tokens, completion_tokens)`. `process_message()` delegates to `_execute_tool_loop()` for the tool-call rounds and extracts `str(final_response.content)` from the result. Observable behaviour is identical to today. `stream_message()` branches on whether the actor declares tools: - **No tools configured:** the existing `astream()` path runs unchanged — real token-by-token streaming is preserved. - **Tools configured:** `_execute_tool_loop()` runs all rounds via `ainvoke()`; `stream_message()` then yields the final response text as a single content chunk and sets `_last_token_usage` / `last_token_usage_var` from the accumulated token counts returned by the helper. All billing-integrity guarantees (populating `last_token_usage_var` and the `_captured_prompt` / `_captured_completion` sentinels on exception paths) are met for both callers of `_execute_tool_loop()`. ## Acceptance Criteria 1. `executor.execute_stream(message)` on a tool-declaring single-LLM actor yields at least one non-empty string token and `executor.last_result` reports non-zero accumulated token counts covering all tool-call rounds. 2. When the LLM requires two tool-call rounds before the final answer, `execute_stream` yields the final answer and `executor.last_result.prompt_tokens` equals the sum across all three invocations (two tool rounds + final). 3. When `tool_max_rounds` is exhausted, the generator terminates cleanly and `executor.last_result` is populated. 4. When the token budget is exceeded mid-loop, the synthesis flow inside the helper runs and the generator yields the synthesis answer. 5. When the model is stuck in tool-only mode (empty final text), the synthesis prompt flow inside the helper runs and the generator yields the synthesis answer. 6. `stream_message()` on an actor with **no** tools configured behaves identically to the current implementation — real `astream()` token-by-token streaming, same token counts. 7. `process_message()` behaviour is unchanged for both tool-using and non-tool actors. 8. On any exception path (tool dispatch error, `ConfigurationError`, LangChain error), `executor.last_result` is populated with accumulated token counts before the exception propagates. 9. `_execute_tool_loop()` has dedicated Behave coverage for each of its internal branches, independently of `process_message()` and `stream_message()`. 10. `nox -s typecheck` reports zero errors and zero warnings on all changed files. 11. `nox -s coverage_report` reports ≥ 97% coverage. ## Supporting Information - Root file: `src/cleveractors/agents/llm.py`. - The tool-call loop to extract lives in `process_message()` (lines ~831–1303). It is tightly coupled to several local variables (`_accumulated_prompt`, `_captured_prompt`, `_budget_exhausted`, `messages`, `response`); the extraction must design a clean return type (a private dataclass or named tuple) that surfaces all state needed by both callers without leaking internal implementation details. - Billing-integrity sentinels (`_captured_prompt` / `_captured_completion`) in `process_message()` and `stream_message()` must be updated to read accumulated counts from the helper's return value rather than re-reading local variables. - cleveragents-webapp workaround to remove after fix: `actor_config_has_tools` + `stream_fallback_to_non_stream` branch in PR cleveragents/cleveragents-webapp#328 (`src/cleverthis/api/routes/v1_chat.py` and `src/cleverthis/services/chat_stream.py`). - Effort estimate: ~4–5 developer-days (helper extraction + `process_message()` refactor ~2d, `stream_message()` integration ~0.5d, Behave scenarios ~1.5d, Robot test ~0.5d, coverage + PR ~0.5d). ## Subtasks - [x] Design the return type for `_execute_tool_loop()`: a private dataclass (e.g. `_ToolLoopResult`) carrying `final_response`, `accumulated_prompt`, `accumulated_completion`, and any flags needed by callers (`budget_exhausted`, `synthesis_was_run`). - [x] Extract `_execute_tool_loop()` from `process_message()`: move the multi-turn `ainvoke()` loop, per-tool `ToolAgent` dispatch, token accumulation, budget exhaustion path, synthesis prompt flow, and tool-output pruning pass into the new method. `_execute_tool_loop()` must populate its result correctly on both success and exception paths. - [x] Refactor `process_message()` to call `_execute_tool_loop()` and derive `response_text` from `result.final_response.content`. All existing tests must continue to pass without modification. - [x] Update `stream_message()` to call `_execute_tool_loop()` when `self._lc_tools is not None`, yield `str(result.final_response.content)` as a single content chunk, and set `_last_token_usage` / `last_token_usage_var` from `result.accumulated_prompt` / `result.accumulated_completion`. The existing no-tools `astream()` branch must remain untouched. - [x] Update the `_captured_prompt` / `_captured_completion` billing-integrity sentinels in both callers to correctly reflect accumulated counts from the helper on exception paths. - [x] Tests (Behave) — `_execute_tool_loop()` internals: basic single tool round; multi-round (two consecutive tool-call rounds); `tool_max_rounds` exhaustion; token accumulation spanning all rounds; token-budget exhaustion (synthesis path runs); model stuck in tool-only mode (synthesis prompt flow); tool-output pruning (pass runs and token costs included); tool dispatch error (`ExecutionError` captured as `ToolMessage`, loop continues); exception path (partial token counts preserved in result). - [x] Tests (Behave) — `stream_message()` integration: tool-using actor yields one non-empty chunk and `last_result` token counts equal helper's accumulated totals; no-tools actor behaviour unchanged (real astream tokens). - [x] Tests (Behave) — `process_message()` regression: confirm existing tool-calling scenarios continue to pass after refactor (no new scenarios required if existing coverage is sufficient). - [x] Tests (Robot): add at least one integration test exercising `execute_stream()` on an actor with a tool config (stub provider; no real API key required). - [x] Run `nox -s coverage_report` and confirm ≥ 97%. - [x] Run `nox` (all default sessions) and confirm all pass with zero errors. ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A single Git commit is created whose **first line** matches the Commit Message in the Metadata section exactly. - The commit is pushed to the branch matching the Branch in Metadata exactly. - The commit is submitted as a pull request to `master`, reviewed, and merged.

hurui200320 added the

labels

2026-07-02 06:43:23 +00:00

hurui200320 added this to the v2.1.0 milestone

2026-07-02 06:43:23 +00:00

hurui200320 added

and removed

labels

2026-07-02 06:44:49 +00:00

hurui200320 self-assigned this

2026-07-02 06:44:50 +00:00

hurui200320 added

and removed

labels

2026-07-02 06:47:39 +00:00

hurui200320 referenced this issue from a commit

2026-07-02 07:47:47 +00:00

feat(agents): add tool-call support to LLMAgent.stream_message

hurui200320 commented

2026-07-02 07:49:10 +00:00

Author

Member

Copy link

Implementation Notes — commit `306f114`

Design decisions

_ToolLoopError for billing integrity across the extraction boundary

The key challenge in extracting _execute_tool_loop() was preserving billing integrity: the callers (process_message and stream_message) need to know the partial accumulated token counts if the helper raises before returning. Since Python has no output parameters, I defined a private _ToolLoopError exception that wraps the original exception and carries accumulated_prompt, accumulated_completion, and any_invocation_made. Callers catch this, set their _captured_prompt/_captured_completion sentinels, then re-raise cause from None. The outer exception handlers (except ConfigurationError, except LangChainException, except Exception) then process the re-raised original exception normally. This preserves all billing-integrity guarantees from the original implementation.

Single-chunk output for stream_message() with tools

When tools are configured, stream_message() awaits _execute_tool_loop() and yields the complete final response as one chunk. The alternative (calling astream() again after ainvoke() produced the final answer) would double LLM cost for every tool-using request. The accepted trade-off is documented in the issue background and in the commit message. Real token-by-token astream() is preserved unchanged for no-tool actors.

messages list modified in-place

_execute_tool_loop() takes messages: list[Any] and appends AIMessage + ToolMessage objects as the conversation progresses. This mirrors the original inline behavior. Both callers build the initial [SystemMessage, HumanMessage] list and pass it to the helper, which grows it through tool rounds.

tool_max_rounds parsing moved into the helper

Previously tool_max_rounds was parsed at the top of process_message() before the if _has_tools check (so it would raise ConfigurationError for invalid values even for no-tool agents). After extraction, parsing happens inside _execute_tool_loop(), which is only called for tool-configured agents. This is a minor behavior change (no-tool agents with invalid tool_max_rounds no longer raise at call time), but there are no tests for that case and it's not a specified behavior.

Key code locations

_ToolLoopResult: src/cleveractors/agents/llm.py, module level, after ChatOpenAI globals
_ToolLoopError: same file, module level, after _ToolLoopResult
_execute_tool_loop(): LLMAgent._execute_tool_loop, after _run_pruning_pass
process_message() delegation: LLMAgent.process_message, ~15-line block replacing 490+ lines
stream_message() tools branch: LLMAgent.stream_message, before the existing astream() loop

Tests summary

features/llm_agent_tool_loop.feature + features/steps/llm_agent_tool_loop_steps.py: 10 Behave scenarios covering _execute_tool_loop() independently
features/llm_agent_stream_tool_calls.feature + features/steps/llm_agent_stream_tool_calls_steps.py: 7 Behave scenarios for stream_message() with tools
robot/llm_tool_calling.robot: 2 new Robot integration tests (stream loop + multi-round token accumulation)
All 63 existing llm_agent_tool_calling.feature scenarios pass unmodified

Quality gates (commit `306f114`)

format ✓ lint ✓ typecheck ✓ security_scan ✓ unit_tests ✓ integration_tests ✓
coverage_report ✓ (97.04%)

Follow-up note (out of scope)

The webapp workaround actor_config_has_tools + stream_fallback_to_non_stream in cleveragents/cleveragents-webapp#328 can now be removed. This is webapp work tracked in the issue description under Supporting Information.

## Implementation Notes — commit 306f114 ### Design decisions **`_ToolLoopError` for billing integrity across the extraction boundary** The key challenge in extracting `_execute_tool_loop()` was preserving billing integrity: the callers (`process_message` and `stream_message`) need to know the partial accumulated token counts if the helper raises before returning. Since Python has no output parameters, I defined a private `_ToolLoopError` exception that wraps the original exception and carries `accumulated_prompt`, `accumulated_completion`, and `any_invocation_made`. Callers catch this, set their `_captured_prompt`/`_captured_completion` sentinels, then re-raise `cause from None`. The outer exception handlers (`except ConfigurationError`, `except LangChainException`, `except Exception`) then process the re-raised original exception normally. This preserves all billing-integrity guarantees from the original implementation. **Single-chunk output for `stream_message()` with tools** When tools are configured, `stream_message()` awaits `_execute_tool_loop()` and yields the complete final response as one chunk. The alternative (calling `astream()` again after `ainvoke()` produced the final answer) would double LLM cost for every tool-using request. The accepted trade-off is documented in the issue background and in the commit message. Real token-by-token `astream()` is preserved unchanged for no-tool actors. **`messages` list modified in-place** `_execute_tool_loop()` takes `messages: list[Any]` and appends `AIMessage` + `ToolMessage` objects as the conversation progresses. This mirrors the original inline behavior. Both callers build the initial `[SystemMessage, HumanMessage]` list and pass it to the helper, which grows it through tool rounds. **`tool_max_rounds` parsing moved into the helper** Previously `tool_max_rounds` was parsed at the top of `process_message()` before the `if _has_tools` check (so it would raise `ConfigurationError` for invalid values even for no-tool agents). After extraction, parsing happens inside `_execute_tool_loop()`, which is only called for tool-configured agents. This is a minor behavior change (no-tool agents with invalid `tool_max_rounds` no longer raise at call time), but there are no tests for that case and it's not a specified behavior. ### Key code locations - `_ToolLoopResult`: `src/cleveractors/agents/llm.py`, module level, after `ChatOpenAI` globals - `_ToolLoopError`: same file, module level, after `_ToolLoopResult` - `_execute_tool_loop()`: `LLMAgent._execute_tool_loop`, after `_run_pruning_pass` - `process_message()` delegation: `LLMAgent.process_message`, ~15-line block replacing 490+ lines - `stream_message()` tools branch: `LLMAgent.stream_message`, before the existing `astream()` loop ### Tests summary - `features/llm_agent_tool_loop.feature` + `features/steps/llm_agent_tool_loop_steps.py`: 10 Behave scenarios covering `_execute_tool_loop()` independently - `features/llm_agent_stream_tool_calls.feature` + `features/steps/llm_agent_stream_tool_calls_steps.py`: 7 Behave scenarios for `stream_message()` with tools - `robot/llm_tool_calling.robot`: 2 new Robot integration tests (stream loop + multi-round token accumulation) - All 63 existing `llm_agent_tool_calling.feature` scenarios pass unmodified ### Quality gates (commit 306f114) - format ✓ lint ✓ typecheck ✓ security_scan ✓ unit_tests ✓ integration_tests ✓ - coverage_report ✓ (97.04%) ### Follow-up note (out of scope) The webapp workaround `actor_config_has_tools` + `stream_fallback_to_non_stream` in `cleveragents/cleveragents-webapp#328` can now be removed. This is webapp work tracked in the issue description under Supporting Information.

hurui200320 referenced this issue from a pull request that will close it,

2026-07-02 07:49:36 +00:00

feat(agents): add tool-call support to LLMAgent.stream_message #68

hurui200320 added a new dependency

2026-07-02 07:49:41 +00:00

#68 feat(agents): add tool-call support to LLMAgent.stream_message

hurui200320 added

and removed