feat(agents): implement multi-turn tool-call loop and sandbox improvements #60

Merged
CoreRasurae merged 2 commits from feature/m2-llm-agent-tool-calling into master 2026-06-23 11:59:40 +00:00
34 changed files with 2559 additions and 73 deletions
+30
View File
@@ -7,8 +7,38 @@ and this project adheres to [Clever Semantic Versioning](https://www.w3.org/subm
## [Unreleased]
### Added
- **Tool Schema Descriptions (`llm_tools.py`) (issue #59)** (`cleveractors.agents.llm_tools`): New module providing `normalize_tool_entry()` and `_BUILTIN_TOOL_SCHEMAS` — a registry of OpenAI function-calling parameter schemas for all built-in tools (`echo`, `math`, `json_parse`, `http_request`, `file_read`, `file_write`, `progress_bar`, `shell`, `python_exec`). Each schema includes targeted LLM guidance: `file_read` requires `max_chars` for files >10KB to prevent context overflow; `file_write` clarifies that response text is NOT persisted to disk; `shell` directs the LLM toward `file_read` for reading files; `python_exec` lists available sandbox builtins and instructs the LLM to use `file_read`/`file_write` instead of `open()`. String tool names in agent config (e.g. `"file_read"`) are normalized to full parameter schemas so the LLM knows about parameters like `max_chars`. Config dicts with a `"name"` key are also normalized; already-formatted OpenAI tool dicts pass through unchanged.
- **Directory Listing and `max_chars` Truncation in `file_read` Tool (issue #59)** (`tool.py`): When given a directory path, `file_read` now returns a formatted listing with file sizes and type indicators (`📁` for directories, `📄` for files). New optional `max_chars` integer parameter truncates file content to the specified limit, preventing models from reading multi-hundred-KB files and overflowing the 262K token context window. Output header includes `TRUNCATED to N chars` when truncation applies. Shell command detection rejects paths containing pipe/redirect operators or starting with known shell commands (`grep`, `ls`, `find`, etc.), returning an explicit error redirecting to the shell tool. File-not-found errors now include the parent directory path and instruct the LLM to list that directory for recovery.
- **`stdout` Capture and `NameError` Guidance in `python_exec` Sandbox (issue #59)** (`tool.py`): `sys.stdout` is now redirected to `io.StringIO()` during sandbox execution, so `print()` calls produce visible output included in the result. When sandbox code references an undefined name (e.g. `open` or `exec`), the error message extracts the name and provides actionable guidance directing the LLM to use `file_read`/`file_write` tools instead.
- **Tool-Calling Capability Discovery (issue #59)** (`llm.py`): `get_capabilities()` now returns `"tool-calling"` when tools are configured on the agent. `get_metadata()` includes `tools_configured` (boolean) and `tool_count` (integer) for introspection.
### Fixed
- **LLMAgent Tool-Calling Pipeline (issue #59)** (`llm.py`, `tool.py`, `llm_tools.py`, `llm_imports.py`, `nodes.py`): The LLM agent tool-calling path was broken in multiple ways — tools from agent config were not passed to the LLM, tool execution was single-pass (the model could not see results or make follow-up calls), tool errors were discarded, `shell` and `python_exec` tools were never registered in `builtin_tools`, and `create_subprocess_exec` was used for shell commands preventing pipes and redirections.
**Multi-turn tool-call loop:** `process_message()` now passes tools to the LLM via `ainvoke(tools=...)` and enters a configurable multi-turn loop (default 20 rounds, minimum 1, overridable via `tool_max_rounds` config or `TOOL_MAX_ROUNDS` env var; values ≤0 are clamped to 1). The stuck-model synthesis-prompt round counts as an additional invocation beyond the configured limit. After each batch of tool calls, `ToolMessage`s with results are appended to the conversation and the LLM is re-invoked, enabling multi-step reasoning chains. **Streaming path (`stream_message`) does not support tool calling** — when an LLM agent is configured with tools and invoked via the streaming API, tools are not passed to the model and the multi-turn loop is not engaged.
**Tool config propagation:** Config keys `unsafe_mode`, `allow_shell`, `exec_python`, and `timeout` from the LLM agent are now propagated to transient `ToolAgent` instances created per tool call.
**Tool error propagation:** `ExecutionError` and `ConfigurationError` raised by tools are now included in the `ToolMessage` sent back to the LLM (previously discarded), enabling the model to self-correct.
**Stuck-model recovery:** When the tool loop exhausts rounds but the model produces no meaningful content, a final prompt is injected instructing the model to synthesize output without making more tool calls.
**Error message preservation:** LLM provider errors (e.g. `BadRequestError` from context overflow) and processing errors now include the actual error details in the `ExecutionError` message instead of the generic `"LLM processing failed"`.
**Tool registration:** `shell` and `python_exec` tools are now registered in `builtin_tools` when `allow_shell` or `exec_python` is enabled in the agent config.
**Shell command execution:** `_execute_shell_command` now uses `asyncio.create_subprocess_shell` instead of `create_subprocess_exec`, enabling piped/chained/redirected shell constructs. Both `stdout` and `stderr` are captured via `subprocess.PIPE`.
**Empty message filtering:** `_prepare_conversation_history()` in `nodes.py` now skips messages whose content is empty or whitespace-only, preventing empty messages from accumulating and polluting downstream conversation history across multi-agent pipelines.
**Module:** `src/cleveractors/agents/llm_tools.py` (new), `src/cleveractors/agents/llm.py`, `src/cleveractors/agents/llm_imports.py`, `src/cleveractors/agents/tool.py`, `src/cleveractors/langgraph/nodes.py`. 5 source files changed, 675 insertions, 50 deletions.
- **LLMAgent.cleanup() no longer closes shared cached httpx clients (issue #57)**: `LLMAgent.cleanup()` previously iterated over hard-coded provider SDK client attributes (`root_async_client`, `root_client`, `_async_client`, `_client`) and called `close()` on each. Recent versions of `langchain-anthropic` and `langchain-openai` cache their default httpx clients via module-level `lru_cache` functions; closing those clients poisoned the cache so every subsequent `ChatAnthropic`/`ChatOpenAI` instance in the same process received the same closed httpx client and failed with a connection error. `cleanup()` now only releases the agent's own reference to the chat model (`self._chat_model = None`). The removed `_KNOWN_CLIENT_ATTRS` class variable has been deleted. Six regression BDD scenarios (tagged `@tdd_issue_57`) covering all four provider SDK client attribute paths (Anthropic `_async_client`/`_client`, OpenAI `root_async_client`/`root_client`) guard against future regressions.
- **Registry Resolution Bugfixes (PR Review rui.hu)**: Fixed `TemplateRegistry._instantiate_from_registry_ref` hardcoding `package_type="template"` instead of threading the template type, which broke all non-template registry resolutions. `_try_parse_registry_ref` now filters to `ReferenceType.REGISTRY` only, preserving the "local templates unaffected" acceptance criterion. `_original_reference` in resolved results now stores the verbatim original reference string, not the cache-key with type suffix. `application.py` template-type mapping extended to cover all 8 types (previously only AGENT/GRAPH/STREAM — the other 5 silently misclassified as STREAM). Duplicated `_instantiate_from_registry_ref`/`_try_parse_registry_ref` logic extracted into shared `_resolve_registry_ref` helper in `base.py`. `RegistryClient` now URL-encodes namespace/name path components to prevent injection. HTTPS enforcement with `allow_insecure` flag per spec §12.2. `ReferenceResolver.close_all()` closes clients under the async lock to prevent concurrent client leak. Async lock lazily initialised to avoid deprecation warning. Added `plural_name` property to `TemplateType`.
@@ -0,0 +1,257 @@
# ADR-2030: LLM Agent Tool Calling — Specification Extensions
**Status:** accepted
**Date:** 2026-06-22
**Author:** CoreRasurae
**Issue:** #59 — LLMAgent does not implement tool calling
---
## Context
Issue #59 reported that `LLMAgent` did not implement tool calling —
`tools:` declarations in agent YAML config were stored in `self.config`
but never read, never passed to the LLM, and never executed. The LLM
hallucinated tool outputs instead of producing structured tool calls.
During implementation, additional gaps were discovered beyond the core
tool-calling pipeline. Each gap required a decision about whether to
extend the specification or remain within existing spec boundaries.
The Actor Configuration Standard (§4.5.1) defines seven built-in
tools (`echo`, `math`, `json_parse`, `http_request`, `file_read`,
`file_write`, `progress_bar`) with fixed argument sets and output
formats. §1.3 permits compliant implementations to "provide additional
agent types, node types, operators, conditions, or routing match types
beyond those defined here." §4.5.1 allows additional tool arguments
to be handled (as opposed to silently ignored).
This ADR documents the decisions made for each specification extension
introduced by the tool-calling implementation.
---
## Decision
### D-1: Tool Schema Registry (`llm_tools.py`)
**What:** A new module `cleveractors.agents.llm_tools` providing
`_BUILTIN_TOOL_SCHEMAS` (a registry of OpenAI function-calling
parameter schemas for all built-in tools) and `normalize_tool_entry()`
(a function that converts raw tool config entries into
`{"type": "function", "function": {...}}` format).
**Spec relationship:** Tool schema descriptions are LLM-facing hints,
not part of the normative tool contract defined in §4.5.1. They extend
the spec without conflicting with any MUST/SHALL requirement. Each
schema includes targeted LLM guidance (e.g. `file_read` requires
`max_chars` for files >10KB, `python_exec` lists available sandbox
builtins).
**Modules:**
- `cleveractors.agents.llm_tools._BUILTIN_TOOL_SCHEMAS`
- `cleveractors.agents.llm_tools.normalize_tool_entry`
### D-2: `file_read` Directory Listing
**What:** When given a directory path, `file_read` returns a formatted
listing with file sizes and type indicators (`📁` for directories,
`📄` for files) instead of failing silently.
**Spec relationship:** §4.5.1 defines `file_read` for reading file
contents only. Directory listing is an extension. Per §1.3, compliant
implementations MAY provide additional features beyond those defined
in the normative specification.
**Output format:** `[FILE_READ_SUCCESS]📁 Directory: <path>` with
one entry per line: ` <marker> <name> (<size> bytes)`.
### D-3: `file_read` `max_chars` Truncation
**What:** New optional `max_chars` integer parameter. When set, file
content is truncated to the specified limit. The output header
includes `TRUNCATED to N chars` when truncation applies.
**Spec relationship:** §4.5.1 allows additional arguments to be handled
by tool implementations. `max_chars` extends the argument set without
removing or altering any existing mandated parameter. This prevents
models from reading multi-hundred-KB files and overflowing the 262K
token context window.
### D-4: `file_read` Shell Command Detection and File-Not-Found Recovery
**What:**
- **Shell command detection:** Before opening a file, `file_read`
checks if the path contains shell operators (`|`, `;`, `&&`, `||`,
`>`, `<`) or starts with known shell commands (`grep`, `ls`, `find`,
etc.). If detected, it returns an explicit error redirecting to the
shell tool.
- **File-not-found recovery:** When a file is not found, the error
message includes the parent directory path and instructs the LLM to
list that directory for recovery.
**Spec relationship:** These are error-quality improvements. §4.5.1
does not prescribe error message formats, and §1.3 allows quality
improvements that do not alter normative behavior. The mandated
behavior (reading existing files, returning error on failure) is
preserved — only the error message quality is enhanced.
### D-5: `python_exec` Sandbox `stdout` Capture
**What:** `sys.stdout` is now redirected to `io.StringIO()` during
sandbox execution, and captured output is included in the result.
Previously only `stderr` was redirected.
**Spec relationship:** §13.2.1 defines the sandbox `__builtins__` dict
and security model. `stdout` capture is an implementation detail that
improves tool behavior without changing the sandbox security model.
The `__builtins__` dict remains strictly compliant — no additional
builtins were added. `print()` is already in the allowed builtins
list; capturing its output makes it useful.
### D-6: `python_exec` Sandbox `NameError` Guidance
**What:** When sandbox code references an undefined name (e.g. `open`
or `exec`), the error message now extracts the name and provides
actionable guidance directing the LLM to use `file_read` and
`file_write` tools for file operations and the `shell` tool for
system commands.
**Spec relationship:** §13.2.3 requires error messages for prohibited
operations. This enhancement improves error messages without exposing
prohibited facilities. The sandbox security model is unchanged — the
offending name is never made available; only the name itself is
reported back for guidance.
### D-7: `shell` and `python_exec` Tool Registration
**What:** The `shell` and `python_exec` tool methods existed on
`ToolAgent` but were never added to `builtin_tools`. They are now
registered when `allow_shell` or `exec_python` is enabled in the
agent config.
**Spec relationship:** The `shell` and `python_exec` tools are
implementation extensions. They are not listed in §4.5.1's built-in
tools table, but §1.3 permits compliant implementations to "provide
additional agent types, node types, operators, conditions, or routing
match types beyond those defined here."
These tools are gated behind explicit opt-in flags (`allow_shell`,
`exec_python`) that default to `false`, ensuring security-by-default.
### D-8: Empty Message Filtering in Conversation History
**What:** `_prepare_conversation_history()` in `cleveractors.langgraph.nodes`
now skips messages whose content is empty or whitespace-only. Only
messages with meaningful content are included in conversation history
passed to downstream agents.
**Spec relationship:** §6.3.1 defines the graph state structure
(messages, metadata, etc.) but does not mandate that empty-content
messages must be preserved. Filtering improves the quality of
conversation history without changing any externally observable
behavior defined by the specification. The filtering is applied
during history preparation, not during state mutation — the raw
graph state is untouched.
---
## Consequences
### Positive
- **LLM self-correction:** Tool errors, shell command misrouting, and
file-not-found failures now provide actionable recovery paths that
the LLM can act on, reducing the need for human intervention.
- **Context window safety:** `max_chars` truncation plus schema-level
`CRITICAL` warnings prevent the most common cause of request failure
(context overflow from reading large files).
- **Security-by-default:** `shell` and `python_exec` are opt-in, gated
behind explicit agent config flags.
- **No spec violations:** Every extension is justified by §1.3
(extensibility clause), §4.5.1 (additional arguments permitted), or
falls within implementation-quality territory not normatively
constrained by the specification.
- **Backward compatibility:** All new arguments are optional. Existing
YAML configs that do not use these extensions continue to work
unchanged.
### Negative / Risks
- **Schema drift:** `_BUILTIN_TOOL_SCHEMAS` must be kept in sync with
actual tool implementations. If a tool's parameter set changes but
the schema is not updated, the LLM may call the tool with wrong
arguments.
- **Output format stability:** The directory listing format and
`TRUNCATED` header are not part of the normative spec. If downstream
code parses `file_read` output, format changes could break parsing.
This is mitigated by the `[FILE_READ_SUCCESS]` prefix remaining
stable.
- **Increased module surface:** `llm_tools.py` adds a new module to the
`cleveractors.agents` package. If LLM tool calling is later moved to
a dedicated subsystem (e.g. a `ToolCallingService`), this module may
need to be relocated or reorganized.
### Follow-up Required
- **ADR-2030 compliance validation:** Add BDD scenarios that verify
each extension's spec compliance boundary (e.g. verify that no
prohibited `__builtins__` entry was added to the sandbox).
- **Schema synchronization test:** Add a test that verifies every
entry in `_BUILTIN_TOOL_SCHEMAS` corresponds to a registered tool
in `ToolAgent.builtin_tools`, and vice versa.
- **Specification update:** If `shell`, `python_exec`, or `file_read`
directory listing graduate to normative status, update §4.5.1 of
the Actor Configuration Standard via a separate ADR.
---
## Alternatives Considered
### A-1: Keep all tools within §4.5.1 and reject extensions
**Rejected because:** §4.5.1's tool set was designed for direct human
invocation, not LLM-driven tool calling. Without directory listing,
`max_chars`, and sandbox guidance, the LLM cannot self-navigate a
codebase, cannot read large files safely, and cannot self-correct
from common mistakes. The seven §4.5.1 tools are insufficient for
autonomous multi-turn tool use.
### A-2: Add all extensions to the normative specification
**Rejected because:** The extensions have not yet been battle-tested
at scale. Directory listing format, `max_chars` default values, and
sandbox guidance wording are likely to evolve as we observe real LLM
behavior. Committing them to the normative spec now would require
a deprecation cycle for any future adjustment.
### A-3: Handle `max_chars` as a separate `head`/`tail` tool
**Rejected because:** The LLM already knows about `file_read` from
schema descriptions. Adding a separate truncation tool would require
the LLM to learn a new tool name and decide when to use it, increasing
the cognitive load on the model and the likelihood of incorrect tool
selection. Integrating truncation into `file_read` as an optional
parameter keeps the tool surface simple.
### A-4: Reject shell command detection in `file_read`
**Rejected because:** Without detection, the LLM's `FileNotFoundError`
for a path like `"grep -n pattern file | head"` is confusing and
offers no recovery path. The LLM cannot distinguish between "file
doesn't exist" and "I accidentally passed a shell command." Detection
and redirection give the model the information it needs to correct
its own mistake.
### A-5: Filter empty messages at the graph orchestration level instead of in `nodes.py`
**Rejected because:** Filtering in `_prepare_conversation_history()`
is the single choke point through which all conversation history
passes before being sent to an agent. Filtering at the graph
orchestration level (e.g. in `pure_graph.py`) would require changes
at every call site that builds history. The node-level approach
applies the fix globally with minimal code change and zero risk of
missing a call site.
+12 -1
View File
@@ -59,7 +59,6 @@ Feature: Registry Cache Coverage
Scenario: CacheFactory with invalid max_size raises ValueError
When I try to create a CacheFactory with max_size=0
Then a ValueError should be raised by the cache factory
Scenario: CacheFactory with invalid ttl raises ValueError
When I try to create a CacheFactory with ttl=-1.0
Then a ValueError should be raised by the cache factory
@@ -118,3 +117,15 @@ Feature: Registry Cache Coverage
When I call resolve_package with type=actor ns=bench name=pkg3 version=v1.0.0
Then the resolve result should contain package_id
Then the cache stats evictions should be 1
Scenario: Resolve package TTL expiry triggers re-fetch
When I create a RegistryCache with ttl 0.01 and mock client
When I call resolve_package with type=actor ns=bench name=pkg version=v1.0.0
When I wait for TTL to expire
When I call resolve_package with type=actor ns=bench name=pkg version=v1.0.0
Then the cache stats misses should be 2
Scenario: Cache content validation returns False when canonicalizer raises TypeError
When I create a RegistryCache with validation enabled and mock client
When I trigger SHA-1 validation with a TypeError-raising canonicalizer
Then the validation should return False
+82
View File
@@ -0,0 +1,82 @@
Feature: LLMAgent Tool Calling
Scenario: LLMAgent reads tools from config at init
Given I have an LLM agent configuration with tools [{"name": "echo"}, {"name": "math"}]
When I create the LLM agent
Then the agent should store tool definitions in _lc_tools with count 2
Scenario: LLMAgent without tools has empty _lc_tools
Given I have a basic LLM agent configuration (no tools)
And that config does NOT include a tools field
When I create the LLM agent
Then the agent _lc_tools should be None
Scenario: LLMAgent get_capabilities includes tool-calling when tools configured
Given I have an LLM agent configuration with tools [{"name": "echo"}]
When I create the LLM agent
And I request the agent capabilities (tool calling)
Then the capabilities should include "tool-calling"
And the capabilities should still include "text-generation"
Scenario: LLMAgent get_capabilities without tool-calling when no tools
Given I have a basic LLM agent configuration
And that config does NOT include a tools field
When I create the LLM agent
And I request the agent capabilities (tool calling)
Then the capabilities should include "text-generation"
And the capabilities should NOT include "tool-calling"
Scenario: LLMAgent process_message passes tools to model when configured
Given I have an LLM agent configuration with tools [{"name": "echo"}]
When I create the LLM agent
And I set a mock chat model that returns a plain text response
And I tool_calling process a message "Hello"
Then the chat_model ainvoke should have been called with a tools keyword
Scenario: LLMAgent process_message without tools does not pass tools to model
Given I have a basic LLM agent configuration
And that config does NOT include a tools field
When I create the LLM agent
And I set a mock chat model that returns a plain text response
And I tool_calling process a message "Hello"
Then the chat_model ainvoke should NOT have been called with a tools keyword
Scenario: LLMAgent _lc_tools passed to model as invoke keyword
Given I have an LLM agent configuration with tools [{"name": "file_read"}]
When I create the LLM agent
And I set a mock chat model that returns a plain text response
And I tool_calling process a message "Read a file"
Then the chat_model ainvoke should have been called with a tools keyword
Scenario: LLMAgent metadata includes tool counts when tools configured
Given I have an LLM agent configuration with tools [{"name": "echo"}, {"name": "math"}]
When I create the LLM agent
And I request the agent metadata (tool calling)
Then the metadata should have "tools_configured" set to true
And the metadata should have "tool_count" equal to 2
Scenario: LLMAgent injects synthesis prompt when model gets stuck in tool-only mode
Given I have an LLM agent configuration with tools [{"name": "echo"}]
And I set tool_max_rounds to 2
When I create the LLM agent
And I set a mock chat model that returns tool_calls with empty content for 2 rounds then text
And I tool_calling process a message "Do something"
Then the synthesis HumanMessage should have been appended to messages
And the final ainvoke should have been called
And the tool_calling result should contain "Synthesized answer"
Scenario: LLMAgent propagates tool execution errors to ToolMessage for LLM self-correction
Given I have an LLM agent configuration with tools [{"name": "echo"}]
And I set tool_max_rounds to 1
When I create the LLM agent
And I set a mock chat model that triggers a tool with ExecutionError "test tool failure"
And I tool_calling process a message "Fail me"
Then the messages list should contain a ToolMessage with content containing _nonexistent_tool_xyz
Scenario: LLMAgent handles empty tool name in malformed tool_calls
Given I have an LLM agent configuration with tools [{"name": "echo"}]
And I set tool_max_rounds to 1
When I create the LLM agent
And I set a mock chat model that returns tool_calls with an empty tool name
And I tool_calling process a message "Call me"
Then the messages list should contain a ToolMessage with content "Tool name is empty"
+6 -1
View File
@@ -82,4 +82,9 @@ Feature: LLM Agent Temperature Override, Cleanup, and Context History
Scenario: Cleanup is a no-op when chat_model has no http clients
Given I setup an LLM agent with a chat model that lacks root_async_client and root_client (llm_gaps)
When I await the cleanup method (llm_gaps)
Then the cleanup should complete without raising errors (llm_gaps)
Then the cleanup should complete without raising errors (llm_gaps)
Scenario: Non-numeric _temperature_override raises ConfigurationError in process_message
Given I setup a test LLM agent for temperature override tests (llm_gaps)
When I call process_message with _temperature_override "bad_string" (llm_gaps)
Then a ConfigurationError mentioning must be a number is raised (llm_gaps)
+34
View File
@@ -0,0 +1,34 @@
Feature: LLM Tools Coverage
As a developer
I want to cover remaining uncovered lines in llm_tools.py
So that coverage reaches the target
Scenario: normalize_tool_entry returns OpenAI-formatted dict as-is
Given an llm_tools coverage test environment
When I call normalize_tool_entry with OpenAI-formatted dict
Then the result should be the same dict unchanged
Scenario: normalize_tool_entry rejects dict missing valid name
Given an llm_tools coverage test environment
When I call normalize_tool_entry with dict missing name
Then a ConfigurationError mentioning non-empty string 'name' is raised
Scenario: normalize_tool_entry rejects dict with empty name
Given an llm_tools coverage test environment
When I call normalize_tool_entry with dict having empty name
Then a ConfigurationError mentioning non-empty string 'name' is raised
Scenario: normalize_tool_entry converts string tool name to OpenAI format
Given an llm_tools coverage test environment
When I call normalize_tool_entry with string "echo"
Then the result should be OpenAI dict with name echo
Scenario: normalize_tool_entry rejects empty string
Given an llm_tools coverage test environment
When I call normalize_tool_entry with empty string
Then a ConfigurationError mentioning non-empty string is raised
Scenario: normalize_tool_entry rejects non-str non-dict type
Given an llm_tools coverage test environment
When I call normalize_tool_entry with integer 42
Then a ConfigurationError mentioning Invalid tool config entry is raised
+23
View File
@@ -36,3 +36,26 @@ Feature: Package Registry Version Resolution Error Branches
Given a reference resolver is created without client and without local store
When resolving an unknown reference type with resolver
Then ResErr: InvalidPackageReferenceError raised containing "Unknown"
Scenario: _pick_latest with empty list raises error
When _pick_latest is called with empty versions list
Then ResErr: InvalidPackageReferenceError raised containing "No versions available to pick"
Scenario: Global alias with no concrete versions raises error
When resolve_version is called with alias "latest" against available versions "v1.x,v2.x"
Then ResErr: InvalidPackageReferenceError raised containing "No concrete versions available"
Scenario: Registry reference missing namespace or name raises error
Given a reference resolver is created without client and without local store
When resolving a registry reference with missing namespace
Then ResErr: InvalidPackageReferenceError raised containing "missing namespace or name"
Scenario: Registry response missing package_id raises error
Given a reference resolver is created with a mock client returning no package_id
When resolving a registry reference with the mock client
Then ResErr: InvalidPackageReferenceError raised containing "Registry response missing"
Scenario: Registry response with invalid package_id raises error
Given a reference resolver is created with a mock client returning invalid package_id
When resolving a registry reference with the mock client
Then ResErr: InvalidPackageReferenceError raised containing "Invalid package_id"
+4
View File
@@ -173,3 +173,7 @@ Feature: Runtime Executor API
Given an Executor for a graph actor using v2.0 actors key instead of agents key
When I execute the graph actor for validation test with message "test actors key"
Then the execution should return an ActorResult
Scenario: runtime_types backward-compatibility shim is importable
When I import from cleaveractors.runtime_types
Then ActorResult and NodeUsage should be importable
@@ -0,0 +1,15 @@
Feature: Runtime Dispatch Normalisation
As a developer
I want graph placeholder node IDs to be normalised to the canonical format
So that multi-actor prefixes produce consistent node identifiers
Background:
Given the runtime dispatch test context is initialised
Scenario: Graph placeholder node_id is normalised to canonical form
When I normalise a graph placeholder node_id "<test_graph:no_llm>"
Then the result should be the canonical "<no_llm>"
Scenario: Regular node_id is returned unchanged by normalise
When I normalise a regular node_id "agent.node_1"
Then the result should be the original node_id
+35
View File
@@ -358,3 +358,38 @@ def step_total_stats_misses(context: Any, expected: int) -> None:
assert context.stats.misses == expected, (
f"Expected total_stats.misses={expected}, got {context.stats.misses}"
)
@when("I create a RegistryCache with ttl {ttl:f} and mock client")
def step_create_cache_short_ttl(context: Any, ttl: float) -> None:
context.cache = RegistryCache(
_create_mock_client(context), ttl=ttl, validate_content=False
)
@when("I wait for TTL to expire")
def step_wait_ttl_expire(context: Any) -> None:
time.sleep(0.1)
@when("I trigger SHA-1 validation with a TypeError-raising canonicalizer")
def step_cache_val_typeerror(context: Any) -> None:
from cleveractors.registry.types import PackageContent, PackageId
cache = context.cache
cache._validate_content_enabled = True
def _bad_compute(content_data, pkg_type):
raise TypeError("bogus type")
cache._canonicalizer.compute_package_id = _bad_compute
pid = PackageId.from_string("pkg_act_0123456789abcdef0123456789abcdef01234567")
content = PackageContent(id=pid, content={"key": "val"})
context.cache_val_result = cache._validate_content(content)
@then("the validation should return False")
def step_cache_val_false(context: Any) -> None:
assert context.cache_val_result is False, (
f"Expected False, got {context.cache_val_result}"
)
+21
View File
@@ -2784,6 +2784,18 @@ async def step_es_stream_message(context: Any) -> None:
_root_logger = logging.getLogger()
_root_logger.addHandler(_handler)
# Also patch _log_no_usage_metadata to record calls directly;
# this is robust even when slipcover wraps the logging machinery
# (coverage_report session).
_cause_records: list[str] = []
_original = agent._log_no_usage_metadata
def _tracking_log(self: Any, cause: str) -> None:
_cause_records.append(cause)
_original(cause)
agent._log_no_usage_metadata = _tracking_log.__get__(agent, type(agent)) # type: ignore[assignment]
try:
tokens: list[str] = []
async for token in agent.stream_message("Hello", None):
@@ -2799,8 +2811,10 @@ async def step_es_stream_message(context: Any) -> None:
context.es_captured_usage_var = (0, 0)
finally:
_root_logger.removeHandler(_handler)
agent._log_no_usage_metadata = _original # type: ignore[method-assign]
context.es_captured_warnings = captured_warnings
context.es_log_causes = _cause_records
@when("I call stream_message with _temperature_override {override} in context (stream)")
@@ -2993,6 +3007,13 @@ def step_es_no_usage_warning_emitted(context: Any) -> None:
_CAUSE_RESPONSE_METADATA_NO_TOKEN_USAGE,
_CAUSE_RESPONSE_METADATA_NOT_DICT,
)
# Primary check: the tracking wrapper recorded calls to _log_no_usage_metadata.
# This is robust across normal behave and slipcover (coverage_report).
_log_causes = getattr(context, "es_log_causes", None)
if _log_causes:
return # at least one cause was recorded — warning was emitted
# Fallback (pre-existing): check the logging handler captured a warning record.
captured = getattr(context, "es_captured_warnings", [])
matching = [
r for r in captured if any(cause in r.getMessage() for cause in _cause_strings)
@@ -0,0 +1,363 @@
"""Step definitions for LLMAgent Tool Calling BDD tests."""
from __future__ import annotations
import asyncio
import json as _json
from typing import TYPE_CHECKING, Any
from unittest.mock import AsyncMock, Mock, patch
from behave import given, then, when
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
if TYPE_CHECKING:
from langchain_core.language_models.chat_models import BaseChatModel
else:
# At runtime we need the actual class for Mock(spec=...)
from langchain_core.language_models.chat_models import BaseChatModel
from cleveractors.agents.llm import LLMAgent
from cleveractors.core.exceptions import ConfigurationError
from cleveractors.templates.renderer import TemplateRenderer
def _create_mock_response(text: str, tool_calls: list[dict[str, Any]] | None = None):
"""Create a mock AIMessage response."""
mock_response = Mock(spec=AIMessage)
mock_response.content = text
if tool_calls is not None:
mock_response.tool_calls = tool_calls
else:
mock_response.tool_calls = None
# Add token usage metadata for billing integrity
mock_response.usage_metadata = {"input_tokens": 10, "output_tokens": 20}
return mock_response
def _make_llm_agent_with_config(config_dict: dict[str, Any]) -> LLMAgent:
"""Create an LLMAgent with a mocked template renderer and pre-injected chat model."""
renderer = Mock(spec=TemplateRenderer)
renderer.render_string.return_value = "mocked system prompt"
agent = LLMAgent(
name=config_dict.get("name", "tool_calling_agent"),
config=config_dict,
template_renderer=renderer,
)
return agent
# ─── Given steps ──────────────────────────────────────────────────────────────
@given("I have an LLM agent configuration with tools {tools_json}")
def step_config_with_tools(context, tools_json):
"""Set up config that includes a tools list."""
parsed = _json.loads(tools_json)
context.llm_config = {
"name": "tool_agent",
"provider": "openai",
"api_key": "test_key",
"tools": parsed,
}
@given("I have a basic LLM agent configuration (no tools)")
def step_basic_llm_config(context):
"""Set up config without tools."""
context.llm_config = {
"name": "basic_agent",
"provider": "openai",
"api_key": "test_key",
}
@given("that config does NOT include a tools field")
def step_no_tools_in_config(context):
"""Ensure the current llm_config has no tools key."""
context.llm_config.pop("tools", None)
# ─── When steps ───────────────────────────────────────────────────────────────
@when("I create the LLM agent")
def step_create_llm_agent(context):
"""Create the LLMAgent."""
context.llm_agent = _make_llm_agent_with_config(context.llm_config)
@when("I set a mock chat model that returns a plain text response")
def step_set_mock_plain_response(context):
"""Inject a mock chat model returning plain-text AIMessage."""
mock_model = Mock(spec=BaseChatModel)
mock_model.ainvoke = AsyncMock(
return_value=_create_mock_response("Plain text response")
)
mock_model.tool_calls = None
context.llm_agent.chat_model = mock_model
@when(
"I set a mock chat model that returns an AIMessage with tool_calls {tool_calls_json}"
)
def step_set_mock_tool_calls_response(context, tool_calls_json):
"""Inject a mock chat model returning AIMessage with tool_calls."""
parsed = _json.loads(tool_calls_json)
mock_model = Mock(spec=BaseChatModel)
mock_model.ainvoke = AsyncMock(
return_value=_create_mock_response(
"Final answer after tools", tool_calls=parsed
)
)
context.llm_agent.chat_model = mock_model
@when("I tool_calling process a message {message_txt}")
def step_process_message(context, message_txt):
"""Process a message through the agent."""
context.result = asyncio.run(context.llm_agent.process_message(message_txt))
@when("I request the agent capabilities (tool calling)")
def step_request_capabilities(context):
"""Get agent capabilities."""
context.capabilities = context.llm_agent.get_capabilities()
@when("I request the agent metadata (tool calling)")
def step_request_metadata(context):
"""Get agent metadata."""
context.metadata = context.llm_agent.get_metadata()
# ─── Then steps ───────────────────────────────────────────────────────────────
@then("the agent should store tool definitions in _lc_tools with count {n:d}")
def step_lctools_count(context, n):
"""Verify _lc_tools was populated with expected count."""
assert context.llm_agent._lc_tools is not None
assert len(context.llm_agent._lc_tools) == n
@then("the agent _lc_tools should be None")
def step_lctools_none(context):
"""Verify _lc_tools is None when no tools config."""
assert context.llm_agent._lc_tools is None
@then('the capabilities should include "{capability}"')
@then('the capabilities should still include "{capability}"')
def step_capability_includes(context, capability):
"""Check that a specific capability is present."""
assert capability in context.capabilities
@then('the capabilities should NOT include "{capability}"')
@then('the capabilities should not include "{capability}"')
def step_capability_not_includes(context, capability):
"""Check that a capability is absent."""
assert capability not in context.capabilities
@then("the chat_model ainvoke should have been called with a tools keyword")
def step_ainvoke_has_tools_kwarg(context):
"""Verify ainvoke was called with the ``tools`` kwarg."""
call = context.llm_agent.chat_model.ainvoke.call_args
if call is None:
raise AssertionError("ainvoke was never called")
kwargs = call.kwargs if hasattr(call, "kwargs") else call[1]
assert "tools" in kwargs, (
f"ainvoke was called but without a ``tools`` kwarg. "
f"call_kwargs={list(kwargs.keys())}"
)
@then("the chat_model ainvoke should NOT have been called with a tools keyword")
def step_ainvoke_no_tools_kwarg(context):
"""Verify ainvoke was NOT passed the ``tools`` kwarg."""
call = context.llm_agent.chat_model.ainvoke.call_args
if call is None:
raise AssertionError("ainvoke was never called")
kwargs = call.kwargs if hasattr(call, "kwargs") else call[1]
assert "tools" not in kwargs
@then("the tool_calling result should contain {expected}")
def step_result_contains(context, expected):
"""Check that the result string contains an expected substring."""
if (
isinstance(expected, str)
and expected.startswith('"')
and expected.endswith('"')
):
expected = expected[1:-1]
assert context.result is not None
assert expected in str(context.result)
@then('the metadata should have "{field}" set to {value}')
def step_metadata_field_value_true(context, field, value):
"""Check a boolean metadata field."""
assert context.metadata is not None
field_value = context.metadata.get(field)
if field_value is None:
raise AssertionError(f"metadata missing key {field!r}")
actual_lower = str(field_value).lower()
expected_lower = value.lower()
assert actual_lower == expected_lower, (
f"metadata.{field} = {field_value!r} != {value!r}"
)
@then('the metadata should have "{field}" equal to {n:d}')
def step_metadata_field_int(context, field, n):
"""Check an integer metadata field."""
assert context.metadata is not None
actual = context.metadata.get(field)
assert actual == n, f"metadata.{field} = {actual!r} != {n}"
# ─── Synthesis-prompt branch ─────────────────────────────────────────────
@given("I set tool_max_rounds to {n:d}")
def step_set_tool_max_rounds(context, n):
context.llm_config["tool_max_rounds"] = n
@when(
"I set a mock chat model that returns tool_calls with empty content "
"for {n:d} rounds then text"
)
def step_set_mock_stuck_synthesis(context, n):
call_counter = [0]
context._synthesis_messages_captured = None
context._synthesis_final_called = False
async def _mock_ainvoke(messages, **invoke_kwargs):
call_counter[0] += 1
if call_counter[0] <= n:
return _create_mock_response(
"",
tool_calls=[
{
"id": f"call_{call_counter[0]}",
"type": "function",
"function": {"name": "echo", "arguments": '{"text": "test"}'},
}
],
)
context._synthesis_final_called = True
context._synthesis_messages_captured = list(messages)
return _create_mock_response("Synthesized answer")
mock_model = Mock(spec=BaseChatModel)
mock_model.ainvoke = _mock_ainvoke
context.llm_agent.chat_model = mock_model
@then("the synthesis HumanMessage should have been appended to messages")
def step_synthesis_human_message_appended(context):
messages = context._synthesis_messages_captured
assert messages is not None, "Synthesis ainvoke was never called"
human_found = any(
isinstance(m, HumanMessage)
and "finished gathering information" in str(m.content)
for m in messages
)
assert human_found, (
f"Synthesis HumanMessage not found in messages: "
f"{[type(m).__name__ for m in messages]}"
)
@then("the final ainvoke should have been called")
def step_final_ainvoke_called(context):
assert context._synthesis_final_called, "Final synthesis ainvoke was not called"
# ─── Tool-error propagation ──────────────────────────────────────────────
@when("I set a mock chat model that triggers a tool with ExecutionError {error_text}")
def step_set_mock_tool_error(context, error_text):
call_counter = [0]
context._tc_messages_captured = None
async def _mock_ainvoke(messages, **invoke_kwargs):
call_counter[0] += 1
if call_counter[0] == 1:
return _create_mock_response(
"",
tool_calls=[
{
"id": "call_err",
"type": "function",
"function": {
"name": "_nonexistent_tool_xyz",
"arguments": "{}",
},
}
],
)
context._tc_messages_captured = list(messages)
return _create_mock_response("Recovered from error")
mock_model = Mock(spec=BaseChatModel)
mock_model.ainvoke = _mock_ainvoke
context.llm_agent.chat_model = mock_model
@then("the messages list should contain a ToolMessage with content containing {text}")
def step_messages_has_toolmessage(context, text):
messages = context._tc_messages_captured
assert messages is not None, "ainvoke messages were not captured"
tool_msgs = [m for m in messages if isinstance(m, ToolMessage)]
found = any(text in str(m.content) for m in tool_msgs)
assert found, (
f"No ToolMessage found with content containing {text!r}. "
f"ToolMessages: {[str(m.content) for m in tool_msgs]}"
)
# ─── Empty tool-name handling ────────────────────────────────────────────
@when("I set a mock chat model that returns tool_calls with an empty tool name")
def step_set_mock_empty_tool_name(context):
call_counter = [0]
context._empty_tool_captured = None
async def _mock_ainvoke(messages, **invoke_kwargs):
call_counter[0] += 1
if call_counter[0] == 1:
return _create_mock_response(
"",
tool_calls=[
{
"id": "empty_001",
"type": "function",
"function": {"name": "", "arguments": "{}"},
}
],
)
context._empty_tool_captured = list(messages)
return _create_mock_response("Final response after empty tool name")
mock_model = Mock(spec=BaseChatModel)
mock_model.ainvoke = _mock_ainvoke
context.llm_agent.chat_model = mock_model
@then('the messages list should contain a ToolMessage with content "{text}"')
def step_messages_has_exact_toolmessage(context, text):
messages = context._empty_tool_captured
assert messages is not None, "ainvoke messages were not captured"
tool_msgs = [m for m in messages if isinstance(m, ToolMessage)]
found = any(str(m.content) == text for m in tool_msgs)
assert found, (
f"No ToolMessage found with exact content {text!r}. "
f"ToolMessages: {[str(m.content) for m in tool_msgs]}"
)
@@ -461,3 +461,25 @@ def step_ml_cleanup_completes_ok(context: Any) -> None:
assert context.ml_cleanup_error is None, (
f"Cleanup should complete without errors, got: {context.ml_cleanup_error}"
)
@when('I call process_message with _temperature_override "{override}" (llm_gaps)')
@async_run_until_complete
async def step_ml_process_bad_temp_override(context: Any, override: str) -> None:
try:
await context.ml_agent.process_message(
"test", context={"_temperature_override": override}
)
context.ml_config_error = None
except Exception as e:
context.ml_config_error = e
@then("a ConfigurationError mentioning {phrase} is raised (llm_gaps)")
def step_ml_config_error_phrase(context: Any, phrase: str) -> None:
assert context.ml_config_error is not None, (
"Expected ConfigurationError but none was raised"
)
assert phrase in str(context.ml_config_error), (
f"Expected error to contain '{phrase}', got: {context.ml_config_error}"
)
@@ -0,0 +1,75 @@
"""Step definitions for llm_tools.py coverage gaps."""
from behave import given, then, when
from cleveractors.agents.llm_tools import normalize_tool_entry
from cleveractors.core.exceptions import ConfigurationError
@given("an llm_tools coverage test environment")
def step_lt_init(context):
context.error = None
context.result = None
@when("I call normalize_tool_entry with OpenAI-formatted dict")
def step_lt_openai_dict(context):
item = {"type": "function", "function": {"name": "test", "description": "desc"}}
context.result = normalize_tool_entry(item)
@when("I call normalize_tool_entry with dict missing name")
def step_lt_dict_no_name(context):
try:
normalize_tool_entry({"description": "no name"})
except ConfigurationError as e:
context.error = str(e)
@when("I call normalize_tool_entry with dict having empty name")
def step_lt_dict_empty_name(context):
try:
normalize_tool_entry({"name": ""})
except ConfigurationError as e:
context.error = str(e)
@when('I call normalize_tool_entry with string "{tool_name}"')
def step_lt_string_name(context, tool_name):
context.result = normalize_tool_entry(tool_name)
@then("the result should be OpenAI dict with name echo")
def step_lt_openai_result_echo(context):
assert context.result["type"] == "function"
assert context.result["function"]["name"] == "echo"
@when("I call normalize_tool_entry with empty string")
def step_lt_empty_string(context):
try:
normalize_tool_entry("")
except ConfigurationError as e:
context.error = str(e)
@when("I call normalize_tool_entry with integer 42")
def step_lt_integer(context):
try:
normalize_tool_entry(42)
except ConfigurationError as e:
context.error = str(e)
@then("the result should be the same dict unchanged")
def step_lt_same_dict(context):
assert context.result == {
"type": "function",
"function": {"name": "test", "description": "desc"},
}
@then("a ConfigurationError mentioning {text} is raised")
def step_lt_config_error(context, text):
assert context.error is not None
assert text in context.error
@@ -129,3 +129,131 @@ def step_assert_error_raised(context: object) -> None:
f"Expected InvalidPackageReferenceError, got {type(context._last_exc)}: "
f"{context._last_exc}"
)
@when("_pick_latest is called with empty versions list")
def step_pick_latest_empty(context: object) -> None:
from cleveractors.registry.resolver import _pick_latest
try:
_pick_latest([])
context._last_exc = None
except InvalidPackageReferenceError as exc:
context._last_exc = exc
@given("a reference resolver is created with a mock client returning no package_id")
def step_resolver_mock_no_package_id(context: object) -> None:
from cleveractors.registry.client import RegistryClient
mock_client = MagicMock(spec=RegistryClient)
async def _resolve_no_id(*_args, **_kwargs):
return {"other": "no package_id here"}
mock_client.resolve_package = _resolve_no_id
resolver = ReferenceResolver(client=mock_client, local_store=None)
context._resolver = resolver
_bootstrap_parse(context)
@given(
"a reference resolver is created with a mock client returning invalid package_id"
)
def step_resolver_mock_invalid_package_id(context: object) -> None:
from cleveractors.registry.client import RegistryClient
mock_client = MagicMock(spec=RegistryClient)
async def _resolve_bad_id(*_args, **_kwargs):
return {"package_id": "not-a-valid-pkg-id"}
mock_client.resolve_package = _resolve_bad_id
resolver = ReferenceResolver(client=mock_client, local_store=None)
context._resolver = resolver
_bootstrap_parse(context)
@when("resolving a registry reference with missing namespace")
def step_resolve_missing_ns(context: object) -> None:
import asyncio
from cleveractors.registry.client import RegistryClient
from cleveractors.registry.types import PackageReference, ReferenceType
ref_obj = PackageReference(
original_reference="test.actor/test@v1.0",
reference_type=ReferenceType.REGISTRY,
namespace="test",
name=None,
) # type: ignore[arg-type]
mock_client = MagicMock(spec=RegistryClient)
resolver = ReferenceResolver(client=mock_client, local_store=None)
async def _resolve() -> None:
original_parse = resolver.parse
def _mock_parse(_ref_str: str) -> PackageReference:
return ref_obj
resolver.parse = _mock_parse # type: ignore[method-assign]
try:
await resolver.resolve("any", package_type="actor")
context._last_exc = None
except InvalidPackageReferenceError as exc:
context._last_exc = exc
finally:
resolver.parse = original_parse # type: ignore[method-assign]
asyncio.run(_resolve())
@when("resolving a registry reference with the mock client")
def step_resolve_with_mock_client(context: object) -> None:
import asyncio
from cleveractors.registry.types import PackageReference, ReferenceType
ref_obj = PackageReference(
original_reference="test.actor/test@v1.0",
reference_type=ReferenceType.REGISTRY,
namespace="test",
name="test",
version="v1.0",
) # type: ignore[arg-type]
async def _resolve() -> None:
try:
await _resolve_with_parsed_ref(context, ref_obj)
except InvalidPackageReferenceError as exc:
context._last_exc = exc
asyncio.run(_resolve())
def _bootstrap_parse(context: object) -> None:
"""Ensure ReferenceResolver.parse is callable as a staticmethod."""
from cleveractors.registry.resolver import ReferenceResolver as RR
if not isinstance(RR.__dict__.get("parse"), staticmethod):
_func = RR.parse
if callable(_func) and not isinstance(_func, staticmethod):
RR.parse = staticmethod(_func)
async def _resolve_with_parsed_ref(context: object, ref_obj: object) -> None:
"""Call resolver with a pre-parsed reference, bypassing parse."""
resolver = context._resolver
resolver._pre_parsed_ref = ref_obj # type: ignore[attr-defined]
original_parse = resolver.parse
def _mock_parse(_ref_str: str) -> object:
return resolver._pre_parsed_ref # type: ignore[attr-defined]
resolver.parse = _mock_parse # type: ignore[method-assign]
try:
await resolver.resolve("any", package_type="actor")
context._last_exc = None
finally:
resolver.parse = original_parse # type: ignore[method-assign]
+14
View File
@@ -222,3 +222,17 @@ def step_assert_no_actors_error(context: Any) -> None:
assert "no actors" in str(context.test_error).lower(), (
f"Expected error to mention 'no actors', got: {context.test_error}"
)
@when("I import from cleaveractors.runtime_types")
def step_import_runtime_types(context):
from cleveractors.runtime_types import ActorResult, NodeUsage
context.rt_actor_result = ActorResult
context.rt_node_usage = NodeUsage
@then("ActorResult and NodeUsage should be importable")
def step_assert_rt_imports(context):
assert context.rt_actor_result is not None
assert context.rt_node_usage is not None
@@ -0,0 +1,36 @@
"""Step definitions for runtime_dispatch.py coverage tests."""
from behave import given, then, when
@given("the runtime dispatch test context is initialised")
def step_rd_init(context):
context.rd_result = None
@when('I normalise a graph placeholder node_id "{node_id}"')
def step_rd_normalise_placeholder(context, node_id):
from cleveractors.runtime_dispatch import _normalize_node_id
context.rd_result = _normalize_node_id(node_id)
@when('I normalise a regular node_id "{node_id}"')
def step_rd_normalise_regular(context, node_id):
from cleveractors.runtime_dispatch import _normalize_node_id
context.rd_result = _normalize_node_id(node_id)
@then('the result should be the canonical "{expected}"')
def step_rd_assert_canonical(context, expected):
assert context.rd_result == expected, (
f"Expected '{expected}', got '{context.rd_result}'"
)
@then("the result should be the original node_id")
def step_rd_assert_original(context):
assert context.rd_result == "agent.node_1", (
f"Expected 'agent.node_1', got '{context.rd_result}'"
)
+23
View File
@@ -1175,3 +1175,26 @@ def step_verify_edge_cases(context):
# Template without parameters section should work
no_params = context.edge_cases["no_params_def"]
assert no_params.parameters == {}
@when("I instantiate a GenericTemplate whose applied vars produce a non-dict result")
def step_gt_nondict_result(context):
from unittest.mock import MagicMock
from cleveractors.templates.base import InstantiationContext, TemplateType
from cleveractors.templates.generic_template import GenericTemplate
gt = GenericTemplate(
name="test", template_type=TemplateType.ACTOR, definition={"k": "{{ v }}"}
)
gt._apply_template_vars = lambda d, p: ["non", "dict", "result"]
ctx = InstantiationContext()
context.gt_result = gt.instantiate(
params={"v": "x"}, registry=MagicMock(), context=ctx
)
@then("the result should be a dict with definition key wrapping the non-dict value")
def step_gt_assert_nondict_wrapped(context):
assert isinstance(context.gt_result, dict)
assert "definition" in context.gt_result
+192 -8
View File
@@ -152,7 +152,7 @@ def step_shell_cmd(context):
m = MagicMock()
m.returncode = 0
m.communicate = AsyncMock(return_value=(b"custom output", b""))
with patch("asyncio.create_subprocess_exec", return_value=m):
with patch("asyncio.create_subprocess_shell", return_value=m):
context.result = _run(a._execute_shell_command("custom_cmd", {}))
@@ -442,6 +442,28 @@ def step_fw_raises(context, text):
assert text in context.error
# ── file_read directory listing OSError ───────────────────────────────
@when("I call file read tool on a directory with unreadable entry")
def step_fr_dir_oserror(context):
a = ToolAgent("tc", {"tools": ["file_read"]})
(context.td / "valid_file.txt").write_text("hello")
with patch("os.path.getsize", side_effect=OSError("Permission denied")):
context.result = _run(
a._file_read_tool({"file": str(context.td)}, context={"_unsafe_mode": True})
)
# ── process_message JSONDecodeError ───────────────────────────────────
@when("I process a message with malformed JSON")
def step_pm_json_error(context):
a = ToolAgent("tc", {"tools": ["echo"]})
context.error = _run(_catch(a.process_message, '{"tool": "echo", "text": bad}'))
# ── shell detailed ───────────────────────────────────────────────────
@@ -451,7 +473,7 @@ def step_sh_cmd_args(context, cmd):
m = MagicMock()
m.returncode = 0
m.communicate = AsyncMock(return_value=(b"hello", b""))
with patch("asyncio.create_subprocess_exec", return_value=m):
with patch("asyncio.create_subprocess_shell", return_value=m):
context.result = _run(a._execute_shell_command(cmd, {"args": ["hello"]}))
@@ -461,7 +483,7 @@ def step_sh_cmd(context, cmd):
m = MagicMock()
m.returncode = 1
m.communicate = AsyncMock(return_value=(b"", b"err"))
with patch("asyncio.create_subprocess_exec", return_value=m):
with patch("asyncio.create_subprocess_shell", return_value=m):
context.error = _run(_catch(a._execute_shell_command, cmd, {}))
@@ -469,11 +491,11 @@ def step_sh_cmd(context, cmd):
def step_sh_timeout(context):
a = ToolAgent("tc", {"tools": ["sleep"], "allow_shell": True})
async def _to(*a, **kw):
raise asyncio.TimeoutError()
with patch("asyncio.create_subprocess_exec", side_effect=_to):
context.error = _run(_catch(a._execute_shell_command, "sleep", {}))
m = MagicMock()
m.communicate = AsyncMock(return_value=(b"", b""))
with patch("asyncio.create_subprocess_shell", return_value=m):
with patch("asyncio.wait_for", side_effect=asyncio.TimeoutError()):
context.error = _run(_catch(a._execute_shell_command, "sleep", {}))
@when("I execute a dangerous shell command {cmd}")
@@ -878,3 +900,165 @@ def step_assert_empty_prefix(context):
def step_caps_fw_only(context):
a = ToolAgent("tc", {"tools": [{"name": "file_write"}]})
context.caps = a.get_capabilities()
# ── python_exec_tool ──────────────────────────────────────────────────
@when("I call python_exec tool without exec_python enabled")
def step_pe_disabled(context):
a = ToolAgent("tc", {"tools": ["python_exec"], "exec_python": True})
a.config["exec_python"] = False
context.error = _run(_catch(a._python_exec_tool, {"code": "x=1"}, None))
@when("I call python_exec tool without code argument")
def step_pe_no_code(context):
a = ToolAgent("tc", {"tools": ["python_exec"], "exec_python": True})
context.error = _run(_catch(a._python_exec_tool, {}, None))
# ── shell_tool ────────────────────────────────────────────────────────
@when("I call shell tool without allow_shell")
def step_sh_disabled(context):
a = ToolAgent("tc", {"tools": ["shell"], "allow_shell": True})
a.allow_shell = False
context.error = _run(_catch(a._shell_tool, {"command": "echo"}, None))
@when("I call shell tool without command argument")
def step_sh_no_cmd(context):
a = ToolAgent("tc", {"tools": ["custom_cmd"], "allow_shell": True})
context.error = _run(_catch(a._shell_tool, {}, None))
# ── python execution details ──────────────────────────────────────────
@when("I execute python code that prints to stdout")
def step_pe_stdout(context):
a = ToolAgent("tc", {"tools": ["python_exec"], "exec_python": True})
context.result = _run(
a._execute_python_code(
"print('hello stdout')",
{},
None,
)
)
@then("the result should contain printed output")
def step_pe_stdout_result(context):
assert "hello stdout" in context.result
@when("I execute python code with undefined variable")
def step_pe_name_error(context):
a = ToolAgent("tc", {"tools": ["python_exec"], "exec_python": True})
context.error = _run(
_catch(
a._execute_python_code,
"x = undefined_var + 1",
{},
None,
)
)
# ── file_read shell command detection ─────────────────────────────────
@when("I call file read tool with a shell command path")
def step_fr_shell_path(context):
a = ToolAgent("tc", {"tools": ["file_read"]})
context.error = _run(_catch(a._file_read_tool, {"file": "ls -la"}, None))
# ── file_read max_chars validation ────────────────────────────────────
@when("I call file read tool with invalid max_chars")
def step_fr_invalid_max(context):
a = ToolAgent("tc", {"tools": ["file_read"]})
fp = context.td / "fr_max.txt"
fp.write_text("sample text")
context.error = _run(
_catch(
a._file_read_tool,
{"file": str(fp), "max_chars": "not_a_number"},
{"_unsafe_mode": True},
)
)
# ── file_read directory listing ───────────────────────────────────────
@when("I call file read tool on an existing directory")
def step_fr_directory(context):
a = ToolAgent("tc", {"tools": ["file_read"]})
(context.td / "somefile.txt").write_text("hello")
context.result = _run(
a._file_read_tool({"file": str(context.td)}, context={"_unsafe_mode": True})
)
@then("the result should contain directory listing header")
def step_fr_dir_header(context):
assert "FILE_READ_SUCCESS" in context.result
assert "Directory:" in context.result
assert "Entries:" in context.result
# ── file_read truncation ──────────────────────────────────────────────
@when("I call file read tool with small max_chars limit")
def step_fr_truncate(context):
a = ToolAgent("tc", {"tools": ["file_read"]})
fp = context.td / "fr_trunc.txt"
fp.write_text("a" * 500)
context.result = _run(
a._file_read_tool(
{"file": str(fp), "max_chars": 10},
context={"_unsafe_mode": True},
)
)
@then("the result should be truncated")
def step_fr_truncated(context):
assert "TRUNCATED" in context.result
# ── file_read general exception ───────────────────────────────────────
@when("I call file read tool causing a read error")
def step_fr_read_error(context):
a = ToolAgent("tc", {"tools": ["file_read"]})
with patch("builtins.open", side_effect=OSError("Permission denied")):
context.error = _run(
_catch(
a._file_read_tool,
{"file": str(context.td / "no_such.txt")},
context={"_unsafe_mode": True},
)
)
# ── file_write absolute path in safe mode ─────────────────────────────
@when("I call file write tool with absolute path in safe mode")
def step_fw_abs_safe(context):
a = ToolAgent("tc", {"tools": ["file_write"]})
context.error = _run(
_catch(
a._validate_file_path_safety,
os.path.join(os.getcwd(), "test_abs_file.txt"),
False,
)
)
@@ -641,3 +641,65 @@ def step_vac_llm_string_config_no_provider(context: Context) -> None:
@given("a config dict with type tool name and string config block (vac)")
def step_vac_tool_string_config(context: Context) -> None:
_set_config_dict(context, {"type": "tool", "name": "test", "config": "not_a_dict"})
@when("I call _compute_subgraph_depth twice for the same subgraph route")
def step_vac_subgraph_depth_cache(context: Context) -> None:
from cleveractors.validation._limits import _compute_subgraph_depth
routes = {
"parent": {"type": "graph", "nodes": {"n": {"type": "llm"}}, "edges": []},
"child": {
"type": "graph",
"nodes": {"n2": {"type": "llm"}},
"subgraph": ["parent"],
"edges": [],
},
}
cache = {}
context._depth1 = _compute_subgraph_depth("child", routes, cache)
context._depth2 = _compute_subgraph_depth("child", routes, cache)
context._cached = "child" in cache
@then("the second call should return the cached depth value")
def step_vac_assert_depth_cached(context: Context) -> None:
assert context._cached, "Expected depth to be cached"
assert context._depth1 == context._depth2, (
f"Depth values differ: {context._depth1} vs {context._depth2}"
)
@when("I call _validate_top_level_keys on a dict missing the agents key")
def step_vac_top_level_missing_agents(context: Context) -> None:
from cleveractors.core.exceptions import ConfigurationError
from cleveractors.validation import _validate_top_level_keys
try:
_validate_top_level_keys({"routes": {}})
except ConfigurationError as e:
context._config_error = str(e)
@then("a ConfigurationError about missing agents key should be raised")
def step_vac_assert_missing_agents(context: Context) -> None:
assert context._config_error is not None, "Expected ConfigurationError"
assert "agents" in context._config_error, (
f"Expected 'agents' in error, got: {context._config_error}"
)
@when("I call _count_total_nodes with a non-dict nodes value in a graph route")
def step_vac_count_total_nondict_nodes(context: Context) -> None:
from cleveractors.core.exceptions import ConfigurationError
from cleveractors.validation._limits import _count_total_nodes
try:
_count_total_nodes({"test": {"type": "graph", "nodes": "not_a_dict"}})
except ConfigurationError as e:
context._config_error = str(e)
@then("a ConfigurationError should be raised about nodes being invalid")
def step_vac_assert_nondict_nodes(context: Context) -> None:
assert context._config_error is not None, "Expected ConfigurationError"
@@ -2130,3 +2130,16 @@ def step_assert_unknown_tag_skipped(context):
# So unknown tags are just passed through without being processed
assert context.result is not None
assert "protected" in context.result
@when("I load YAML content with Jinja2 templates without providing context")
def step_yjl_deferred_no_context(context):
from cleveractors.templates.yaml_jinja_loader import YAMLJinjaLoader
loader = YAMLJinjaLoader()
context.result = loader.load_string("key: {{ value }}")
@then("the result should contain deferred template markers")
def step_yjl_assert_deferred_markers(context):
assert context.result is not None, "Expected deferred template result"
+4
View File
@@ -164,3 +164,7 @@ Feature: Template Parameter Validation and Variable Rendering
When I test error conditions
Then appropriate errors should be raised
And edge cases should be handled correctly
Scenario: GenericTemplate wraps non-dict result in definition key
When I instantiate a GenericTemplate whose applied vars produce a non-dict result
Then the result should be a dict with definition key wrapping the non-dict value
+1 -1
View File
@@ -449,7 +449,7 @@ Feature: ToolAgent Functionality
"""
{"tool": "file_read", "args": {"file": "nonexistent.txt"}}
"""
Then tool execution should fail with message containing "File read failed"
Then tool execution should fail with message containing "File not found"
Scenario: Tool agent file_write tool requires file path and content
Given I am running in unsafe mode
+72 -2
View File
@@ -126,7 +126,7 @@ Feature: Tool Agent Coverage Gaps
Scenario: File read tool raises File read failed for missing file
Given a tool agent coverage test environment
When I call file read tool with a missing file path
Then the file read raises File read failed
Then the file read raises File not found
Scenario: File read tool requires a file path
Given a tool agent coverage test environment
@@ -321,4 +321,74 @@ Feature: Tool Agent Coverage Gaps
Scenario: _execute_python_code writing_stage else None
Given a tool agent coverage test environment
When I execute python code with non-dict context
Then the code should execute successfully
Then the code should execute successfully
Scenario: python_exec tool requires exec_python enabled
Given a tool agent coverage test environment
When I call python_exec tool without exec_python enabled
Then a shell ExecutionError mentioning Python execution is disabled is raised
Scenario: python_exec tool requires code argument
Given a tool agent coverage test environment
When I call python_exec tool without code argument
Then a shell ExecutionError mentioning requires a 'code' argument is raised
Scenario: shell tool requires allow_shell enabled
Given a tool agent coverage test environment
When I call shell tool without allow_shell
Then a shell ExecutionError mentioning disabled is raised
Scenario: shell tool requires command argument
Given a tool agent coverage test environment
When I call shell tool without command argument
Then a shell ExecutionError mentioning requires a 'command' argument is raised
Scenario: _execute_python_code captures stdout fallback
Given a tool agent coverage test environment
When I execute python code that prints to stdout
Then the result should contain printed output
Scenario: _execute_python_code NameError in sandbox
Given a tool agent coverage test environment
When I execute python code with undefined variable
Then a shell ExecutionError mentioning not available in the Python sandbox is raised
Scenario: File read tool detects shell command in file path
Given a tool agent coverage test environment
When I call file read tool with a shell command path
Then the file read raises looks like a shell command
Scenario: File read tool validates max_chars parameter
Given a tool agent coverage test environment
When I call file read tool with invalid max_chars
Then the file read raises must be an integer
Scenario: File read tool lists directory contents
Given a tool agent coverage test environment
When I call file read tool on an existing directory
Then the result should contain directory listing header
Scenario: File read tool handles max_chars truncation
Given a tool agent coverage test environment
When I call file read tool with small max_chars limit
Then the result should be truncated
Scenario: File read tool raises File read failed for read error
Given a tool agent coverage test environment
When I call file read tool causing a read error
Then the file read raises File read failed
Scenario: File write tool blocks absolute path in safe mode
Given a tool agent coverage test environment
When I call file write tool with absolute path in safe mode
Then the file write raises Unsafe file path blocked
Scenario: File read tool directory listing handles OSError for entry size
Given a tool agent coverage test environment
When I call file read tool on a directory with unreadable entry
Then the result should contain directory listing header
Scenario: Process message handles JSONDecodeError
Given a tool agent coverage test environment
When I process a message with malformed JSON
Then an ExecutionError mentioning Invalid JSON is raised
@@ -203,3 +203,15 @@ Feature: Validation Actor Runtime Config Validation
Given a config dict with type tool name and string config block (vac)
When _validate_tool_actor is called (vac)
Then a ConfigurationError should be raised for missing tools (vac)
Scenario: _validate_top_level_keys raises ConfigurationError when agents key is missing
When I call _validate_top_level_keys on a dict missing the agents key
Then a ConfigurationError about missing agents key should be raised
Scenario: _count_total_nodes raises ConfigurationError for non-dict nodes value
When I call _count_total_nodes with a non-dict nodes value in a graph route
Then a ConfigurationError should be raised about nodes being invalid
Scenario: _compute_subgraph_depth returns cached value on cache hit
When I call _compute_subgraph_depth twice for the same subgraph route
Then the second call should return the cached depth value
@@ -263,3 +263,9 @@ Feature: YAMLJinjaLoader and TemplateAwareYAMLParser Template Processing Pipelin
And I have a clean test environment for yaml jinja processing
When I protect template sections with an unknown tag
Then the unknown tag should be replaced with SKIP
Scenario: YAMLJinjaLoader defers rendering when no context is provided
Given the yaml jinja loader system is initialized
And I have a clean test environment for yaml jinja processing
When I load YAML content with Jinja2 templates without providing context
Then the result should contain deferred template markers
+8 -14
View File
@@ -340,9 +340,7 @@ class EmailGraphLib:
)
current = current[key]
def merged_config_key_count(
self, merged: Any, section: str, minimum: int
) -> None:
def merged_config_key_count(self, merged: Any, section: str, minimum: int) -> None:
"""Assert a config section has at least minimum keys."""
assert section in merged, f"Section '{section}' not in merged config"
actual = len(merged[section])
@@ -390,9 +388,7 @@ class EmailGraphLib:
"""Assert the assembled component graph has exactly expected actors."""
config = self.load_all_email_components()
actual = len(config.get("actors", {}))
assert actual == expected, (
f"Expected {expected} actors, got {actual}"
)
assert actual == expected, f"Expected {expected} actors, got {actual}"
# -- namespaced component graph with reference resolver -------------------
@@ -421,7 +417,7 @@ class EmailGraphLib:
for _node_id, node_def in nodes.items():
agent_ref = node_def.get("agent_ref", "")
if agent_ref.startswith("local:"):
ref_name = agent_ref[len("local:"):]
ref_name = agent_ref[len("local:") :]
pkg = store.resolve_package(ref_name)
content = pkg.content
actor_name = content.get("name", ref_name)
@@ -431,11 +427,11 @@ class EmailGraphLib:
}
graph_config["actors"] = actors
for node_def in graph_config.get("routes", {}).get("main", {}).get(
"nodes", {}
).values():
for node_def in (
graph_config.get("routes", {}).get("main", {}).get("nodes", {}).values()
):
if "agent_ref" in node_def and node_def["agent_ref"].startswith("local:"):
ref_name = node_def["agent_ref"][len("local:"):]
ref_name = node_def["agent_ref"][len("local:") :]
pkg = store.resolve_package(ref_name)
node_def["agent"] = pkg.content.get("name", ref_name)
del node_def["agent_ref"]
@@ -501,6 +497,4 @@ class EmailGraphLib:
renderer = TemplateRenderer()
factory = AgentFactory(config=config, template_renderer=renderer)
meta = factory.get_agent_metadata(agent_name)
assert key in meta, (
f"Key '{key}' not in metadata. Keys: {list(meta.keys())}"
)
assert key in meta, f"Key '{key}' not in metadata. Keys: {list(meta.keys())}"
+200
View File
@@ -0,0 +1,200 @@
"""Library for LLM tool-calling integration tests.
Provides keywords that create an Executor with a mock chat model that
returns tool_calls, then execute messages and verify that tools were
invoked correctly without requiring a real LLM API key.
"""
from __future__ import annotations
import asyncio
from typing import Any
from unittest.mock import MagicMock, patch
from langchain_core.messages import AIMessage
from cleveractors.result import ActorResult
from cleveractors.runtime import Executor, create_executor
class ToolCallingTestLib:
"""Keywords for integration-style tests of LLM tool-calling."""
def __init__(self) -> None:
self._executor: Executor | None = None
self._last_result: ActorResult | None = None
self._mock_ainvoke_calls: list[dict[str, Any]] = []
self._patches: list[Any] = []
def _teardown_patches(self) -> None:
for p in self._patches:
p.stop()
self._patches.clear()
def create_executor_with_tool_calling_agent(
self, tool_name: str = "echo", message: str = ""
) -> None:
"""Create an Executor whose LLM agent is configured with a tool.
The chat model is mocked so that ``ainvoke`` returns an AIMessage
with tool_calls. The model first returns a tool-call message,
then a final plain-text answer.
"""
self._teardown_patches()
self._mock_ainvoke_calls = []
final_response_text = "Final answer after tool use"
def _build_mock_model(*args, **kwargs):
mock_model = MagicMock()
mock_model.temperature = 0.7
async def _mock_ainvoke(messages, **invoke_kwargs):
self._mock_ainvoke_calls.append({"kwargs": invoke_kwargs})
call_count = len(self._mock_ainvoke_calls)
if call_count == 1 and invoke_kwargs.get("tools"):
tool_content = message or "hello from tool"
return AIMessage(
content="",
usage_metadata={
"input_tokens": 50,
"output_tokens": 10,
"total_tokens": 60,
},
tool_calls=[
{
"id": "call_001",
"name": tool_name,
"args": {"message": tool_content},
},
],
)
return AIMessage(
content=final_response_text,
usage_metadata={
"input_tokens": 30,
"output_tokens": 15,
"total_tokens": 45,
},
)
async def _mock_astream(messages, **invoke_kwargs):
self._mock_ainvoke_calls.append({"kwargs": invoke_kwargs})
call_count = len(self._mock_ainvoke_calls)
if call_count == 1 and invoke_kwargs.get("tools"):
tool_content = message or "hello from stream"
yield AIMessage(
content="tool_call_response",
usage_metadata={
"input_tokens": 40,
"output_tokens": 10,
"total_tokens": 50,
},
tool_calls=[
{
"id": "call_001",
"name": tool_name,
"args": {"message": tool_content},
},
],
)
else:
yield AIMessage(
content=final_response_text,
usage_metadata={
"input_tokens": 25,
"output_tokens": 12,
"total_tokens": 37,
},
)
mock_model.ainvoke = _mock_ainvoke
mock_model.astream = _mock_astream
return mock_model
patcher = patch(
"cleveractors.agents.llm.build_chat_model",
side_effect=_build_mock_model,
)
patcher.start()
self._patches.append(patcher)
config = {
"type": "llm",
"name": "tool_test_agent",
"provider": "openai",
"model": "gpt-3.5-turbo",
"config": {
"tools": [{"name": tool_name}],
},
}
self._executor = create_executor(
config_dict=config,
credentials={"openai": {"api_key": "mock-key"}},
limits={},
pricing={},
)
async def _execute_async(self, message: str) -> ActorResult:
return await self._executor.execute(message)
def execute_with_message(self, message: str) -> None:
"""Execute the executor with *message* and store the result."""
try:
self._last_result = asyncio.run(self._execute_async(message))
finally:
self._teardown_patches()
def execute_stream_with_message(self, message: str) -> None:
"""Execute via execute_stream and store the last result."""
async def _stream():
chunks = []
async for token in self._executor.execute_stream(message):
chunks.append(token)
return "".join(chunks)
try:
asyncio.run(_stream())
self._last_result = self._executor.last_result
finally:
self._teardown_patches()
def result_is_valid_actor_result(self) -> None:
assert isinstance(self._last_result, ActorResult), (
f"Expected ActorResult, got {type(self._last_result)}"
)
def result_response_contains(self, text: str) -> None:
assert self._last_result is not None
assert text in self._last_result.response, (
f"Expected '{text}' in response, got: {self._last_result.response[:200]}"
)
def result_has_nodes(self) -> None:
assert self._last_result is not None
assert len(self._last_result.nodes) > 0, "Expected at least one node in result"
def result_prompt_tokens_greater_than_zero(self) -> None:
assert self._last_result is not None
assert self._last_result.prompt_tokens > 0, (
f"Expected prompt_tokens > 0, got {self._last_result.prompt_tokens}"
)
def chat_model_was_invoked_with_tools(self) -> None:
for call in self._mock_ainvoke_calls:
if call.get("kwargs", {}).get("tools"):
return
raise AssertionError("chat_model.ainvoke was never called with tools keyword")
def multiple_tool_rounds_occurred(self) -> None:
assert len(self._mock_ainvoke_calls) >= 2, (
f"Expected >= 2 ainvoke calls, got {len(self._mock_ainvoke_calls)}"
)
def tool_call_result_is_in_response(self) -> None:
assert self._last_result is not None
assert len(self._last_result.response.strip()) > 0, (
"Expected non-empty response after tool calls"
)
+52
View File
@@ -0,0 +1,52 @@
*** Settings ***
Documentation LLM tool-calling integration tests.
... Exercises the full pipeline (Executor → LLMAgent → ToolAgent)
... with a mock chat model that returns tool_calls, so no real
... LLM API key is required.
Library ToolCallingTestLib.py
*** Test Cases ***
LLM Agent Receives Tools And Executes Tool Call
[Documentation] Full lifecycle: config with echo tool → mock returns
... tool_calls → ToolAgent executes echo → final response produced.
Create Executor With Tool Calling Agent tool_name=echo message=hello_from_tool
Execute With Message Tell me something using the echo tool
Result Is Valid Actor Result
Result Has Nodes
Chat Model Was Invoked With Tools
Multiple Tool Rounds Occurred
Result Response Contains Final answer after tool use
LLM Agent Tool Calling Works With MATH Tool
[Documentation] MATH tool recognises the expression and computes result.
Create Executor With Tool Calling Agent tool_name=math message=2+2
Execute With Message What is 2+2? Use the math tool.
Result Is Valid Actor Result
Chat Model Was Invoked With Tools
Multiple Tool Rounds Occurred
Result Response Contains Final answer after tool use
LLM Agent With FILE_READ Tool Receives Tool Calls
[Documentation] file_read tool is normalised and passed to the model.
Create Executor With Tool Calling Agent tool_name=file_read message=some_file.txt
Execute With Message Read the file some_file.txt for me
Result Is Valid Actor Result
Chat Model Was Invoked With Tools
Tool Calling Produces Non-Empty Final Response
[Documentation] After tool execution, the final response is populated.
Create Executor With Tool Calling Agent tool_name=echo message=tool_data
Execute With Message Use the echo tool and report back
Result Is Valid Actor Result
Tool Call Result Is In Response
Result Prompt Tokens Greater Than Zero
Streaming Path Works With Tool-Configured Agent
[Documentation] execute_stream produces a valid result with a tool-configured
... agent. Tool calling is not exercised in the streaming path — tools are
... not passed to the model during stream_message. This test verifies the
... streaming path itself works and does not crash with a tool-configured agent.
Create Executor With Tool Calling Agent tool_name=echo message=from_stream
Execute Stream With Message Tell me about yourself
Result Is Valid Actor Result
Result Has Nodes
+261 -20
View File
@@ -24,7 +24,9 @@ Extended Provider Routing (ADR-2028):
from __future__ import annotations
import contextvars
import json
import logging
import os
import threading
from collections.abc import AsyncGenerator
from typing import TYPE_CHECKING, Any, Literal
@@ -35,6 +37,7 @@ if TYPE_CHECKING:
from cleveractors.agents.base import AgentWithMemory
from cleveractors.agents.llm_client import build_chat_model
from cleveractors.agents.llm_imports import populate_langchain_globals
from cleveractors.agents.llm_tools import normalize_tool_entry as _normalize_tool_entry
from cleveractors.core.exceptions import (
AgentCreationError,
ConfigurationError,
@@ -102,6 +105,7 @@ LangChainException: Any = None
AIMessage: Any = None
HumanMessage: Any = None
SystemMessage: Any = None
ToolMessage: Any = None
ChatGoogleGenerativeAI: Any = None
ChatOpenAI: Any = None
@@ -195,6 +199,15 @@ class LLMAgent(AgentWithMemory):
self.max_tokens: int = config.get("max_tokens", DEFAULT_MAX_TOKENS)
self.system_message: str = config.get("system_prompt", DEFAULT_SYSTEM_MESSAGE)
# Tool definitions extracted from agent config. Stored as a list of
# dicts in OpenAI-compatible function-calling format (``{"type":
# "function", "function": {...}}``). Populated during __init__ only
# when ``config["tools"]`` is present — the list is built once so it
# can be reused across multiple calls without re-computation.
self._lc_tools: list[dict[str, Any]] | None = None
if config.get("tools"):
self._lc_tools = [_normalize_tool_entry(t) for t in config["tools"]]
# Per-request credential dict (ADR-2026), validated and shallow-copied.
# Contains {"api_key": "...", "base_url": "..."} for a single provider.
# Stored separately; config is never modified.
@@ -467,10 +480,229 @@ class LLMAgent(AgentWithMemory):
# Add current user message
messages.append(HumanMessage(content=processed_message))
# Call LangChain model
response = await self.chat_model.ainvoke(messages)
# -- Tool calling support (issue #59) -----------------------------------
# When the agent config declares tools, convert them to a LangChain-
# compatible format and pass them to the LLM so it can produce
# structured tool calls instead of hallucinating tool names in plain
# text.
#
# Multi-turn tool-use loop: when the model returns tool_calls,
# execute each tool, append the AIMessage (with tool_calls) plus
# corresponding ToolMessages to the conversation, then re-invoke the
# model. The loop repeats until the model returns a final answer
# (no more tool_calls) or the maximum number of tool rounds is
# reached.
invoke_kwargs: dict[str, Any] = {}
_raw_max_rounds: object = self.config.get(
"tool_max_rounds"
) or os.environ.get("TOOL_MAX_ROUNDS", "20")
try:
_TOOL_MAX_ROUNDS = max(1, int(str(_raw_max_rounds)))
except (TypeError, ValueError) as _mre:
raise ConfigurationError(
f"tool_max_rounds must be an integer, got {_raw_max_rounds!r}"
) from _mre
_has_tools = self._lc_tools is not None and LANGCHAIN_AVAILABLE
if _has_tools:
invoke_kwargs["tools"] = self._lc_tools
for _tool_round in range(_TOOL_MAX_ROUNDS):
if _tool_round > 0 and not _has_tools:
break
response = await self.chat_model.ainvoke(messages, **invoke_kwargs)
response_tool_calls: list[dict[str, Any]] = (
getattr(response, "tool_calls", None) or []
)
if not isinstance(response_tool_calls, list) or not response_tool_calls:
break
if not _has_tools:
break
messages.append(response)
for tc in response_tool_calls:
call_id = tc.get("id", "")
# OpenAI standard: {"function": {"name": "...", "arguments": "..."}}
# HF / non-standard: {"name": "...", "args": {...}}
fn_def = tc.get("function")
if isinstance(fn_def, dict):
tool_name = fn_def.get("name", "")
arguments_raw = fn_def.get("arguments")
else:
tool_name = tc.get("name", "")
arguments_raw = tc.get("args")
if not tool_name:
messages.append(
ToolMessage(
content="Tool name is empty",
tool_call_id=call_id,
)
)
continue
args: dict[str, Any] = {}
if isinstance(arguments_raw, str):
try:
args = json.loads(arguments_raw)
except (json.JSONDecodeError, ValueError):
args = {"_raw": arguments_raw}
elif isinstance(arguments_raw, dict):
args = arguments_raw
try:
from cleveractors.agents.tool import ToolAgent as _TA
# Propagate tool-execution hints from the LLM
# agent's config (§4.5.4). The graph YAML places
# "unsafe_mode", "allow_shell", and "timeout"
# directly on the LLM agent config; we translate
# them into ToolAgent-compatible equivalents.
parent_unsafe = self.config.get("unsafe_mode", False)
tool_config: dict[str, Any] = {
"tools": [{"name": tool_name}],
"safe_mode": not parent_unsafe,
"allow_shell": self.config.get("allow_shell", False),
"exec_python": self.config.get("exec_python", False),
"timeout": self.config.get("timeout", 1),
}
agent = _TA(
name=f"_tc_{call_id}",
config=tool_config,
template_renderer=self.template_renderer,
)
tool_ctx: dict[str, Any] | None = (
{"_unsafe_mode": True} if parent_unsafe else None
)
tool_result = await agent.process_message(
json.dumps({"tool": tool_name, "args": args})
if isinstance(args, dict)
else arguments_raw or "",
context=tool_ctx,
)
messages.append(
ToolMessage(content=tool_result, tool_call_id=call_id)
)
except (ExecutionError, ConfigurationError) as _tool_err:
_tool_err_msg = str(_tool_err)
logger.warning(
"Agent %s: tool call failed for %r (tool=%s): %s",
self.name,
call_id,
tool_name,
_tool_err_msg,
)
messages.append(
ToolMessage(
content=f"Tool '{tool_name}' error: {_tool_err_msg}",
tool_call_id=call_id,
)
)
response_text: str = str(response.content)
# If the tool loop exhausted but the model produced no meaningful
# content (stuck in tool-only mode), ask it to synthesize output
# from all gathered context. This happens with some models that
# never stop making tool calls on their own.
if not response_text.strip() and _has_tools and len(messages) > 2:
messages.append(
HumanMessage(
content=(
"You have finished gathering information. "
"Now produce your final answer based on all the "
"data collected. Do NOT make any more tool calls. "
"If the answer requires writing a file, call "
"file_write in this very response."
)
)
)
response = await self.chat_model.ainvoke(messages, tools=self._lc_tools)
# Allow *one* final tool-call round so the model can
# write files or perform other last-minute operations
# requested by the synthesis prompt.
synth_tool_calls: object = getattr(response, "tool_calls", None)
if (
isinstance(synth_tool_calls, list)
and synth_tool_calls
and _has_tools
):
messages.append(response)
for tc in synth_tool_calls:
call_id = tc.get("id", "")
fn_def = tc.get("function")
if isinstance(fn_def, dict):
tool_name = fn_def.get("name", "")
arguments_raw = fn_def.get("arguments")
else:
tool_name = tc.get("name", "")
arguments_raw = tc.get("args")
if not tool_name:
messages.append(
ToolMessage(
content="Tool name is empty",
tool_call_id=call_id,
)
)
continue
args: dict[str, Any] = {}
if isinstance(arguments_raw, str):
try:
args = json.loads(arguments_raw)
except (json.JSONDecodeError, ValueError):
args = {"_raw": arguments_raw}
elif isinstance(arguments_raw, dict):
args = arguments_raw
try:
from cleveractors.agents.tool import (
ToolAgent as _TA,
)
parent_unsafe = self.config.get("unsafe_mode", False)
tool_config: dict[str, Any] = {
"tools": [{"name": tool_name}],
"safe_mode": not parent_unsafe,
"allow_shell": self.config.get("allow_shell", False),
"exec_python": self.config.get("exec_python", False),
"timeout": self.config.get("timeout", 1),
}
agent = _TA(
name=f"_tc_synth_{call_id}",
config=tool_config,
template_renderer=self.template_renderer,
)
tool_ctx: dict[str, Any] | None = (
{"_unsafe_mode": True} if parent_unsafe else None
)
tool_result = await agent.process_message(
json.dumps({"tool": tool_name, "args": args})
if isinstance(args, dict)
else arguments_raw or "",
context=tool_ctx,
)
messages.append(
ToolMessage(content=tool_result, tool_call_id=call_id)
)
except (ExecutionError, ConfigurationError) as _se:
_se_msg = str(_se)
logger.warning(
"Agent %s: synth tool call failed for %r (tool=%s): %s",
self.name,
call_id,
tool_name,
_se_msg,
)
messages.append(
ToolMessage(
content=f"Tool '{tool_name}' error: {_se_msg}",
tool_call_id=call_id,
)
)
response = await self.chat_model.ainvoke(messages)
response_text = str(response.content)
# Extract real token usage from LangChain response metadata (AC2).
# Primary source: usage_metadata (LangChain standard field).
# Fallback: response_metadata["token_usage"] (provider-specific).
@@ -593,14 +825,14 @@ class LLMAgent(AgentWithMemory):
else:
self._last_token_usage = (0, 0)
last_token_usage_var.set((0, 0))
_err_msg = str(e)
logger.error(
"LLM agent %s LangChain error: %s", self.name, type(e).__name__
)
logger.debug(
"Raw LangChain exception (sanitized): type=%s",
"LLM agent %s LangChain error (%s): %s",
self.name,
type(e).__name__,
_err_msg,
)
raise ExecutionError("LLM processing failed") from None
raise ExecutionError(f"LLM processing failed: {_err_msg}") from None
except Exception as e:
# Billing integrity: if ainvoke() already succeeded (_captured_prompt
# is not None), the LLM provider has already charged the user for
@@ -615,14 +847,14 @@ class LLMAgent(AgentWithMemory):
else:
self._last_token_usage = (0, 0)
last_token_usage_var.set((0, 0))
_err_msg = str(e)
logger.error(
"LLM agent %s processing failed: %s", self.name, type(e).__name__
)
logger.debug(
"Raw processing exception (sanitized): type=%s",
"LLM agent %s processing failed (%s): %s",
self.name,
type(e).__name__,
_err_msg,
)
raise ExecutionError("LLM processing failed") from None
raise ExecutionError(f"LLM processing failed: {_err_msg}") from None
finally:
# Restore original temperature if it was overridden.
# This prevents the override from leaking into subsequent calls
@@ -898,16 +1130,14 @@ class LLMAgent(AgentWithMemory):
else:
self._last_token_usage = (0, 0)
last_token_usage_var.set((0, 0))
_err_msg = str(e)
logger.error(
"LLM agent %s LangChain streaming error: %s",
"LLM agent %s LangChain streaming error (%s): %s",
self.name,
type(e).__name__,
_err_msg,
)
logger.debug(
"Raw LangChain streaming exception (sanitized): type=%s",
type(e).__name__,
)
raise ExecutionError("LLM streaming failed") from None
raise ExecutionError(f"LLM streaming failed: {_err_msg}") from None
except Exception as e: # pylint: disable=broad-exception-caught
# Billing integrity: if astream() already completed
# (_captured_prompt is not None), the LLM provider has already
@@ -922,10 +1152,14 @@ class LLMAgent(AgentWithMemory):
else:
self._last_token_usage = (0, 0)
last_token_usage_var.set((0, 0))
_err_msg = str(e)
logger.error(
"LLM agent %s streaming failed: %s", self.name, type(e).__name__
"LLM agent %s streaming failed (%s): %s",
self.name,
type(e).__name__,
_err_msg,
)
raise ExecutionError("LLM streaming failed") from None
raise ExecutionError(f"LLM streaming failed: {_err_msg}") from None
finally:
# Restore original temperature if it was overridden.
# Mirrors process_message() finally block (spec §4.4.5).
@@ -1056,6 +1290,10 @@ class LLMAgent(AgentWithMemory):
if self.provider.lower() in ["openai", "anthropic", "google"]:
capabilities.append("structured-output")
# Add tool-calling capability when tools are configured.
if self._lc_tools:
capabilities.append("tool-calling")
return capabilities
def get_metadata(self) -> dict[str, Any]:
@@ -1071,6 +1309,9 @@ class LLMAgent(AgentWithMemory):
"langchain_integration": True,
"supports_structured_output": self.provider.lower()
in ["openai", "anthropic", "google"],
"tools_configured": self._lc_tools is not None
and len(self._lc_tools) > 0,
"tool_count": len(self._lc_tools) if self._lc_tools else 0,
}
)
return metadata
+3
View File
@@ -28,6 +28,7 @@ try:
from langchain_core.messages import AIMessage as _AIMessage
from langchain_core.messages import HumanMessage as _HumanMessage
from langchain_core.messages import SystemMessage as _SystemMessage
from langchain_core.messages import ToolMessage as _ToolMessage
from langchain_openai import ChatOpenAI as _ChatOpenAI
_LANGCHAIN_AVAILABLE = True
@@ -37,6 +38,7 @@ except ImportError:
_AIMessage = None
_HumanMessage = None
_SystemMessage = None
_ToolMessage = None
_ChatOpenAI = None
_LANGCHAIN_AVAILABLE = False
@@ -80,6 +82,7 @@ def populate_langchain_globals(target_globals: dict[str, Any]) -> None:
target_globals["AIMessage"] = _AIMessage
target_globals["HumanMessage"] = _HumanMessage
target_globals["SystemMessage"] = _SystemMessage
target_globals["ToolMessage"] = _ToolMessage
target_globals["ChatOpenAI"] = _ChatOpenAI
if _GOOGLE_AVAILABLE:
+316
View File
@@ -0,0 +1,316 @@
"""Tool normalisation for LLMAgent — converts raw config entries to
OpenAI function-calling format (extracted from llm.py).
Following the same modular-extraction pattern as llm_client.py and
llm_providers.py, this module provides the :func:`normalize_tool_entry`
function and the :data:`_BUILTIN_TOOL_SCHEMAS` registry used to supply
descriptions and parameter schemas for built-in tools when they are
referenced by name in an LLM agent's ``tools`` config list.
"""
from __future__ import annotations
from typing import Any
from cleveractors.core.exceptions import ConfigurationError
_BUILTIN_TOOL_SCHEMAS: dict[str, dict[str, Any]] = {
"echo": {
"description": "Echo back the provided text.",
"parameters": {
"type": "object",
"properties": {
"text": {
"type": "string",
"description": "The text to echo back.",
},
"args": {
"type": "array",
"items": {"type": "string"},
"description": "Alternative: list of strings to join and echo.",
},
},
"additionalProperties": False,
},
},
"math": {
"description": "Evaluate a mathematical expression safely.",
"parameters": {
"type": "object",
"properties": {
"expression": {
"type": "string",
"description": "A mathematical expression to evaluate (e.g. '2+2').",
},
"args": {
"type": "array",
"items": {"type": "string"},
"description": "Alternative: first element used as expression.",
},
},
"additionalProperties": False,
},
},
"json_parse": {
"description": "Parse a JSON string and return the structured result.",
"parameters": {
"type": "object",
"properties": {
"json": {
"type": "string",
"description": "The JSON string to parse.",
},
"args": {
"type": "array",
"items": {"type": "string"},
"description": "Alternative: first element used as JSON string.",
},
},
"additionalProperties": False,
},
},
"http_request": {
"description": "Make an HTTP request and return the response.",
"parameters": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "The URL to request.",
},
"method": {
"type": "string",
"description": "HTTP method (GET, POST, PUT, DELETE, etc.). Default: GET.",
"default": "GET",
},
"headers": {
"type": "object",
"description": "Optional HTTP headers as key-value pairs.",
},
"data": {
"type": "object",
"description": "Optional JSON body to send with the request.",
},
},
"required": ["url"],
"additionalProperties": False,
},
},
"file_read": {
"description": "Read a file or list a directory. CRITICAL: Always use max_chars when reading large files (>10KB) or the LLM context will overflow and the request will fail. For .cs/.py source files use max_chars=8000 for classification, max_chars=16000 for analysis, max_chars=32000 only for small files needing full content. For directories, returns formatted listing with file sizes.",
"parameters": {
"type": "object",
"properties": {
"file": {
"type": "string",
"description": "Path to the file or directory. Use '.' or './WindowsFormsApplication2' to list a directory.",
},
"max_chars": {
"type": "integer",
"description": "REQUIRED for files >10KB. Maximum chars to return. Files exceeding this will show 'TRUNCATED'. Use 8000 for scanning, 16000 for analysis. Leaving this unset on a large file WILL cause context overflow and task failure.",
},
"args": {
"type": "array",
"items": {"type": "string"},
"description": "Alternative: first element used as file path.",
},
},
"additionalProperties": False,
},
},
"file_write": {
"description": "Write content to disk. CRITICAL: Text in your response is NOT saved to disk. You MUST call this tool explicitly to persist any file that downstream agents need. Supports write, append, and insert modes. Use mkdir -p via shell first for new directories.",
"parameters": {
"type": "object",
"properties": {
"file": {
"type": "string",
"description": "Path to the file to write.",
},
"content": {
"type": "string",
"description": "Content to write to the file.",
},
"mode": {
"type": "string",
"description": "Write mode: 'w' (overwrite), 'a' (append), or 'insert'.",
"enum": ["w", "a", "insert"],
"default": "w",
},
"position": {
"type": "integer",
"description": "For 'insert' mode, the line number to insert at.",
},
},
"required": ["file", "content"],
"additionalProperties": False,
},
},
"progress_bar": {
"description": "Render or update a progress bar.",
"parameters": {
"type": "object",
"properties": {
"stage": {
"type": "string",
"description": "The current stage or phase name.",
},
"phase": {
"type": "string",
"description": "Alternate name for stage.",
},
"message": {
"type": "string",
"description": "A label or message for the progress bar.",
},
"label": {
"type": "string",
"description": "Alternate name for message.",
},
"total": {
"type": "integer",
"description": "Total number of steps.",
},
"steps": {
"type": "integer",
"description": "Alternate name for total.",
},
"count": {
"type": "integer",
"description": "Alternate name for total.",
},
"current": {
"type": "integer",
"description": "Current step number completed.",
},
"done": {
"type": "integer",
"description": "Alternate name for current.",
},
"remaining": {
"type": "integer",
"description": "Number of steps remaining.",
},
"sections_total": {
"type": "integer",
"description": "Total number of sections.",
},
"sections_completed": {
"type": "integer",
"description": "Number of sections completed.",
},
"completed_sections": {
"type": "integer",
"description": "Alternate name for sections_completed.",
},
},
"additionalProperties": False,
},
},
"shell": {
"description": "Execute a shell command and return stdout. Requires allow_shell. Use for: find, grep, cat. Prefer file_read for reading files (has max_chars to prevent context overflow). When creating files in new directories, always use 'mkdir -p <dir> && touch <path>'.",
"parameters": {
"type": "object",
"properties": {
"command": {
"type": "string",
"description": "Shell command. Chain with && for multi-step: 'mkdir -p dir && touch dir/__init__.py' or 'find . -name \"*.cs\" | sort'.",
},
"args": {
"type": "array",
"items": {"type": "string"},
"description": "Additional command-line arguments.",
},
},
"required": ["command"],
"additionalProperties": False,
},
},
"python_exec": {
"description": "Execute Python code in a sandbox. Requires exec_python. Available builtins: print, len, str, int, float, bool, list, dict, set, tuple, range, enumerate, zip, min, max, sum, abs, round, sorted, reversed, any, all, isinstance, type. Use 'import json' for JSON operations. For file operations use the file_read and file_write tools.",
"parameters": {
"type": "object",
"properties": {
"code": {
"type": "string",
"description": "The Python code to execute.",
},
"args": {
"type": "object",
"description": "Optional variable bindings passed to the code sandbox.",
},
},
"required": ["code"],
"additionalProperties": False,
},
},
}
def normalize_tool_entry(item: object) -> dict[str, Any]:
"""Convert a raw tool config entry into OpenAI function-calling format.
Accepts three input shapes:
* **String** a built-in tool name, e.g. ``"file_read"``.
* **Dict with ``"name"``** a simple tool reference, e.g.
``{"name": "echo"}``. If the ``"name"`` matches a built-in tool,
the tool's parameters schema is included automatically; otherwise a
minimal schema with an empty ``properties`` dict is generated.
* **Already-formatted OpenAI tool** a dict with ``"type"`` and
``"function"`` keys is returned unchanged.
Args:
item: The raw config entry a string, a dict, or an
already-formatted OpenAI tool dict.
Returns:
A dict in ``{"type": "function", "function": {...}}`` format.
Raises:
ConfigurationError: If ``item`` is not a recognised format.
"""
if isinstance(item, dict):
if "type" in item and "function" in item:
return dict(item)
tool_name = item.get("name")
if not isinstance(tool_name, str) or not tool_name:
raise ConfigurationError(
f"Tool dict must contain a non-empty string 'name': {item!r}"
)
schema = _BUILTIN_TOOL_SCHEMAS.get(tool_name, {})
return {
"type": "function",
"function": {
"name": tool_name,
"description": schema.get(
"description", f"Execute the '{tool_name}' tool."
),
"parameters": schema.get(
"parameters",
{"type": "object", "properties": {}},
),
},
}
if isinstance(item, str):
if not item:
raise ConfigurationError(
f"Tool name must be a non-empty string, got {item!r}"
)
schema = _BUILTIN_TOOL_SCHEMAS.get(item, {})
return {
"type": "function",
"function": {
"name": item,
"description": schema.get("description", f"Execute the '{item}' tool."),
"parameters": schema.get(
"parameters",
{"type": "object", "properties": {}},
),
},
}
raise ConfigurationError(
f"Invalid tool config entry: {item!r}. "
"Expected a string tool name, a dict with 'name' key, "
"or an OpenAI-formatted tool dict."
)
+175 -26
View File
@@ -77,6 +77,10 @@ class ToolAgent(Agent):
"file_write": self._file_write_tool,
"progress_bar": self._progress_bar_tool,
}
if self.allow_shell:
self.builtin_tools["shell"] = self._shell_tool
if self.config.get("exec_python", False):
self.builtin_tools["python_exec"] = self._python_exec_tool
# Validate tools
self._validate_tools()
@@ -286,15 +290,19 @@ class ToolAgent(Agent):
f"Dangerous command '{command}' blocked in safe mode"
)
# Build command with arguments
cmd_parts = [command]
if "args" in args:
cmd_parts.extend(str(arg) for arg in args["args"])
# Build the full shell command string. When the caller provides
# positional arguments via ``args["args"]`` they are appended;
# otherwise the entire ``command`` string is used as-is so that
# piped / chained / redirected shell constructs work correctly.
cmd_str = command
if "args" in args and args["args"]:
cmd_str = command + " " + " ".join(str(a) for a in args["args"])
try:
# Execute with timeout
process = await asyncio.create_subprocess_exec(
*cmd_parts, stdout=subprocess.PIPE, stderr=subprocess.PIPE
# Execute with timeout — use subprocess_shell so pipes, &&,
# redirections, etc. are handled by the system shell.
process = await asyncio.create_subprocess_shell(
cmd_str, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
stdout, stderr = await asyncio.wait_for(
@@ -314,6 +322,42 @@ class ToolAgent(Agent):
f"Command '{command}' timed out after {self.timeout} seconds"
) from timeout_err
async def _shell_tool(
self,
args: dict[str, Any],
context: Optional[dict[str, Any]], # pylint: disable=unused-argument
) -> str:
"""Execute an arbitrary shell command.
Requires ``allow_shell`` to be enabled on the agent.
Accepts ``command`` as the executable and optional ``args``
as a list of string arguments.
"""
if not self.allow_shell:
raise ExecutionError("Shell execution is disabled for this agent")
command = args.get("command", "")
if not command:
raise ExecutionError("Shell tool requires a 'command' argument")
return await self._execute_shell_command(command, args)
async def _python_exec_tool(
self,
args: dict[str, Any],
context: Optional[dict[str, Any]],
) -> str:
"""Execute arbitrary Python code in a sandboxed environment.
Requires ``exec_python`` to be enabled on the agent.
Accepts ``code`` (the Python code string) and ``args``
(a dict of variable bindings passed to the code sandbox).
"""
if not self.config.get("exec_python", False):
raise ExecutionError("Python execution is disabled for this agent")
code = args.get("code", "")
if not code:
raise ExecutionError("python_exec tool requires a 'code' argument")
return await self._execute_python_code(code, args, context)
async def _execute_python_code(
self, code: str, args: dict[str, Any], context: Optional[dict[str, Any]]
) -> str:
@@ -395,12 +439,15 @@ class ToolAgent(Agent):
# Redirect stderr to suppress print(..., file=sys.stderr) statements
old_stderr = sys.stderr
sys.stderr = io.StringIO()
try:
exec(code, exec_env, exec_env) # nosec B102: sandboxed inline code execution # nosemgrep
finally:
sys.stderr = old_stderr
else:
old_stdout = sys.stdout
sys.stdout = io.StringIO()
try:
exec(code, exec_env, exec_env) # nosec B102: sandboxed inline code execution # nosemgrep
finally:
stdout_output = sys.stdout.getvalue()
sys.stdout = old_stdout
if should_suppress:
sys.stderr = old_stderr
logger.debug(f"After exec - context: {exec_env.get('context')}")
@@ -423,8 +470,23 @@ class ToolAgent(Agent):
result = exec_env[key]
break
captured = stdout_output.strip()
if captured:
if result is None:
result = captured
else:
result = f"{result}\n\n{captured}"
return str(result) if result is not None else ""
except NameError as e:
_name = str(e).split("'")[1] if "'" in str(e) else str(e)
logger.error("Python code execution failed in %s: %s", self.name, e)
raise ExecutionError(
f"'{_name}' is not available in the Python sandbox. "
f"Use the file_read and file_write tools for file operations, "
f"and the shell tool for system commands."
) from e
except Exception as e:
logger.error("Python code execution failed in %s: %s", self.name, e)
raise ExecutionError(f"Python code execution failed: {str(e)}") from e
@@ -515,7 +577,7 @@ class ToolAgent(Agent):
async def _file_read_tool(
self, args: dict[str, Any], context: Optional[dict[str, Any]]
) -> str:
"""File reading tool."""
"""File reading tool with directory listing support and optional truncation."""
filepath = args.get("file", "")
if "args" in args and args["args"]:
filepath = args["args"][0]
@@ -523,33 +585,120 @@ class ToolAgent(Agent):
if not filepath:
raise ExecutionError("File read tool requires a file path")
# Detect shell commands mistakenly passed as file paths
_shell_indicators = ["|", ";", "&&", "||", ">", "<"]
_shell_commands = {
"ls",
"cat",
"grep",
"head",
"tail",
"find",
"sort",
"wc",
"touch",
"mkdir",
"rm",
"cp",
"mv",
"echo",
"python",
"python3",
"read",
"awk",
"sed",
"xargs",
"cut",
"tr",
"tee",
"env",
"export",
"set",
"unset",
"source",
"chmod",
"chown",
}
_first_word = filepath.strip().split()[0].lower() if filepath.strip() else ""
if (
any(ind in filepath for ind in _shell_indicators)
or _first_word in _shell_commands
):
raise ExecutionError(
f"'{filepath}' looks like a shell command, not a file path. "
f"Use the shell tool instead of file_read for commands like "
f"grep, find, ls, cat. Use file_read only for file/directory paths."
)
if self.safe_mode:
# Always block directory traversal attempts
if ".." in filepath:
raise ExecutionError("Unsafe file path blocked in safe mode")
# Block absolute paths unless in unsafe mode
if filepath.startswith("/") and not (
context and context.get("_unsafe_mode", False)
):
raise ExecutionError("Unsafe file path blocked in safe mode")
max_chars = args.get("max_chars")
if max_chars is not None:
try:
max_chars = int(max_chars)
except (TypeError, ValueError) as exc:
raise ExecutionError(
f"max_chars must be an integer, got {max_chars}"
) from exc
try:
if os.path.isdir(filepath):
entries = sorted(os.listdir(filepath))
result_parts = [f"[FILE_READ_SUCCESS]📁 Directory: {filepath}"]
result_parts.append(f"Entries: {len(entries)}")
for entry in entries:
entry_path = os.path.join(filepath, entry)
marker = "📁" if os.path.isdir(entry_path) else "📄"
try:
size = os.path.getsize(entry_path)
size_str = f" ({size:,} bytes)"
except OSError:
size_str = ""
result_parts.append(f" {marker} {entry}{size_str}")
return "\n".join(result_parts)
with open(filepath, "r", encoding="utf-8") as f:
content = f.read()
# Return content with clean terminal indicator
# The content is still in the response for context/memory
# but we add a prefix for clean terminal display
line_count = content.count("\n") + 1
char_count = len(content)
# Format: Special marker + metadata + full content
# The special marker helps identify this for clean display
return (
line_count = content.count("\n") + 1
truncated = False
if max_chars is not None and char_count > max_chars:
content = content[:max_chars]
truncated = True
header = (
f"[FILE_READ_SUCCESS]📄 File: {filepath} | "
f"Lines: {line_count} | Size: {char_count} chars\n"
f"[FILE_CONTENT_START]\n{content}\n[FILE_CONTENT_END]"
f"Lines: {line_count} | Size: {char_count} chars"
)
except Exception as e:
raise ExecutionError(f"File read failed: {e}") from e
if truncated:
header += f" | TRUNCATED to {max_chars} chars"
return f"{header}\n[FILE_CONTENT_START]\n{content}\n[FILE_CONTENT_END]"
except FileNotFoundError:
parent_dir = os.path.dirname(filepath) or "."
raise ExecutionError(
f"File not found: '{filepath}'. "
f"Try listing the parent directory with file_read '{parent_dir}' "
f"to see available files."
) from None
except OSError as e:
raise ExecutionError(
f"File read failed: {e}. "
f"Check that the path is accessible and you have the required permissions."
) from e
except UnicodeDecodeError as e:
raise ExecutionError(
f"File read failed: cannot decode '{filepath}' as UTF-8. "
f"The file may contain binary content. Use the shell tool "
f"with appropriate encoding commands if binary extraction is needed."
) from e
def _validate_file_write_args(self, filepath: str, content: str) -> None:
"""Validate file write arguments."""
+2
View File
@@ -172,6 +172,8 @@ class Node: # pylint: disable=too-many-instance-attributes
for msg in reversed(messages):
content = str(msg.get("content", ""))
if not content.strip():
continue
total_chars += len(content)
collected.append(deepcopy(msg))