feat(agents): implement multi-turn tool-call loop and sandbox improvements #60
@@ -7,8 +7,38 @@ and this project adheres to [Clever Semantic Versioning](https://www.w3.org/subm
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- **Tool Schema Descriptions (`llm_tools.py`) (issue #59)** (`cleveractors.agents.llm_tools`): New module providing `normalize_tool_entry()` and `_BUILTIN_TOOL_SCHEMAS` — a registry of OpenAI function-calling parameter schemas for all built-in tools (`echo`, `math`, `json_parse`, `http_request`, `file_read`, `file_write`, `progress_bar`, `shell`, `python_exec`). Each schema includes targeted LLM guidance: `file_read` requires `max_chars` for files >10KB to prevent context overflow; `file_write` clarifies that response text is NOT persisted to disk; `shell` directs the LLM toward `file_read` for reading files; `python_exec` lists available sandbox builtins and instructs the LLM to use `file_read`/`file_write` instead of `open()`. String tool names in agent config (e.g. `"file_read"`) are normalized to full parameter schemas so the LLM knows about parameters like `max_chars`. Config dicts with a `"name"` key are also normalized; already-formatted OpenAI tool dicts pass through unchanged.
|
||||
|
||||
- **Directory Listing and `max_chars` Truncation in `file_read` Tool (issue #59)** (`tool.py`): When given a directory path, `file_read` now returns a formatted listing with file sizes and type indicators (`📁` for directories, `📄` for files). New optional `max_chars` integer parameter truncates file content to the specified limit, preventing models from reading multi-hundred-KB files and overflowing the 262K token context window. Output header includes `TRUNCATED to N chars` when truncation applies. Shell command detection rejects paths containing pipe/redirect operators or starting with known shell commands (`grep`, `ls`, `find`, etc.), returning an explicit error redirecting to the shell tool. File-not-found errors now include the parent directory path and instruct the LLM to list that directory for recovery.
|
||||
|
||||
- **`stdout` Capture and `NameError` Guidance in `python_exec` Sandbox (issue #59)** (`tool.py`): `sys.stdout` is now redirected to `io.StringIO()` during sandbox execution, so `print()` calls produce visible output included in the result. When sandbox code references an undefined name (e.g. `open` or `exec`), the error message extracts the name and provides actionable guidance directing the LLM to use `file_read`/`file_write` tools instead.
|
||||
|
||||
- **Tool-Calling Capability Discovery (issue #59)** (`llm.py`): `get_capabilities()` now returns `"tool-calling"` when tools are configured on the agent. `get_metadata()` includes `tools_configured` (boolean) and `tool_count` (integer) for introspection.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **LLMAgent Tool-Calling Pipeline (issue #59)** (`llm.py`, `tool.py`, `llm_tools.py`, `llm_imports.py`, `nodes.py`): The LLM agent tool-calling path was broken in multiple ways — tools from agent config were not passed to the LLM, tool execution was single-pass (the model could not see results or make follow-up calls), tool errors were discarded, `shell` and `python_exec` tools were never registered in `builtin_tools`, and `create_subprocess_exec` was used for shell commands preventing pipes and redirections.
|
||||
|
||||
**Multi-turn tool-call loop:** `process_message()` now passes tools to the LLM via `ainvoke(tools=...)` and enters a configurable multi-turn loop (default 20 rounds, minimum 1, overridable via `tool_max_rounds` config or `TOOL_MAX_ROUNDS` env var; values ≤0 are clamped to 1). The stuck-model synthesis-prompt round counts as an additional invocation beyond the configured limit. After each batch of tool calls, `ToolMessage`s with results are appended to the conversation and the LLM is re-invoked, enabling multi-step reasoning chains. **Streaming path (`stream_message`) does not support tool calling** — when an LLM agent is configured with tools and invoked via the streaming API, tools are not passed to the model and the multi-turn loop is not engaged.
|
||||
|
||||
**Tool config propagation:** Config keys `unsafe_mode`, `allow_shell`, `exec_python`, and `timeout` from the LLM agent are now propagated to transient `ToolAgent` instances created per tool call.
|
||||
|
||||
**Tool error propagation:** `ExecutionError` and `ConfigurationError` raised by tools are now included in the `ToolMessage` sent back to the LLM (previously discarded), enabling the model to self-correct.
|
||||
|
||||
**Stuck-model recovery:** When the tool loop exhausts rounds but the model produces no meaningful content, a final prompt is injected instructing the model to synthesize output without making more tool calls.
|
||||
|
||||
**Error message preservation:** LLM provider errors (e.g. `BadRequestError` from context overflow) and processing errors now include the actual error details in the `ExecutionError` message instead of the generic `"LLM processing failed"`.
|
||||
|
||||
**Tool registration:** `shell` and `python_exec` tools are now registered in `builtin_tools` when `allow_shell` or `exec_python` is enabled in the agent config.
|
||||
|
||||
**Shell command execution:** `_execute_shell_command` now uses `asyncio.create_subprocess_shell` instead of `create_subprocess_exec`, enabling piped/chained/redirected shell constructs. Both `stdout` and `stderr` are captured via `subprocess.PIPE`.
|
||||
|
||||
**Empty message filtering:** `_prepare_conversation_history()` in `nodes.py` now skips messages whose content is empty or whitespace-only, preventing empty messages from accumulating and polluting downstream conversation history across multi-agent pipelines.
|
||||
|
||||
**Module:** `src/cleveractors/agents/llm_tools.py` (new), `src/cleveractors/agents/llm.py`, `src/cleveractors/agents/llm_imports.py`, `src/cleveractors/agents/tool.py`, `src/cleveractors/langgraph/nodes.py`. 5 source files changed, 675 insertions, 50 deletions.
|
||||
|
||||
- **LLMAgent.cleanup() no longer closes shared cached httpx clients (issue #57)**: `LLMAgent.cleanup()` previously iterated over hard-coded provider SDK client attributes (`root_async_client`, `root_client`, `_async_client`, `_client`) and called `close()` on each. Recent versions of `langchain-anthropic` and `langchain-openai` cache their default httpx clients via module-level `lru_cache` functions; closing those clients poisoned the cache so every subsequent `ChatAnthropic`/`ChatOpenAI` instance in the same process received the same closed httpx client and failed with a connection error. `cleanup()` now only releases the agent's own reference to the chat model (`self._chat_model = None`). The removed `_KNOWN_CLIENT_ATTRS` class variable has been deleted. Six regression BDD scenarios (tagged `@tdd_issue_57`) covering all four provider SDK client attribute paths (Anthropic `_async_client`/`_client`, OpenAI `root_async_client`/`root_client`) guard against future regressions.
|
||||
|
||||
- **Registry Resolution Bugfixes (PR Review rui.hu)**: Fixed `TemplateRegistry._instantiate_from_registry_ref` hardcoding `package_type="template"` instead of threading the template type, which broke all non-template registry resolutions. `_try_parse_registry_ref` now filters to `ReferenceType.REGISTRY` only, preserving the "local templates unaffected" acceptance criterion. `_original_reference` in resolved results now stores the verbatim original reference string, not the cache-key with type suffix. `application.py` template-type mapping extended to cover all 8 types (previously only AGENT/GRAPH/STREAM — the other 5 silently misclassified as STREAM). Duplicated `_instantiate_from_registry_ref`/`_try_parse_registry_ref` logic extracted into shared `_resolve_registry_ref` helper in `base.py`. `RegistryClient` now URL-encodes namespace/name path components to prevent injection. HTTPS enforcement with `allow_insecure` flag per spec §12.2. `ReferenceResolver.close_all()` closes clients under the async lock to prevent concurrent client leak. Async lock lazily initialised to avoid deprecation warning. Added `plural_name` property to `TemplateType`.
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
# ADR-2030: LLM Agent Tool Calling — Specification Extensions
|
||||
|
||||
**Status:** accepted
|
||||
|
||||
**Date:** 2026-06-22
|
||||
|
||||
**Author:** CoreRasurae
|
||||
|
||||
**Issue:** #59 — LLMAgent does not implement tool calling
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Issue #59 reported that `LLMAgent` did not implement tool calling —
|
||||
`tools:` declarations in agent YAML config were stored in `self.config`
|
||||
but never read, never passed to the LLM, and never executed. The LLM
|
||||
hallucinated tool outputs instead of producing structured tool calls.
|
||||
|
||||
During implementation, additional gaps were discovered beyond the core
|
||||
tool-calling pipeline. Each gap required a decision about whether to
|
||||
extend the specification or remain within existing spec boundaries.
|
||||
|
||||
The Actor Configuration Standard (§4.5.1) defines seven built-in
|
||||
tools (`echo`, `math`, `json_parse`, `http_request`, `file_read`,
|
||||
`file_write`, `progress_bar`) with fixed argument sets and output
|
||||
formats. §1.3 permits compliant implementations to "provide additional
|
||||
agent types, node types, operators, conditions, or routing match types
|
||||
beyond those defined here." §4.5.1 allows additional tool arguments
|
||||
to be handled (as opposed to silently ignored).
|
||||
|
||||
This ADR documents the decisions made for each specification extension
|
||||
introduced by the tool-calling implementation.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
### D-1: Tool Schema Registry (`llm_tools.py`)
|
||||
|
||||
**What:** A new module `cleveractors.agents.llm_tools` providing
|
||||
`_BUILTIN_TOOL_SCHEMAS` (a registry of OpenAI function-calling
|
||||
parameter schemas for all built-in tools) and `normalize_tool_entry()`
|
||||
(a function that converts raw tool config entries into
|
||||
`{"type": "function", "function": {...}}` format).
|
||||
|
||||
**Spec relationship:** Tool schema descriptions are LLM-facing hints,
|
||||
not part of the normative tool contract defined in §4.5.1. They extend
|
||||
the spec without conflicting with any MUST/SHALL requirement. Each
|
||||
schema includes targeted LLM guidance (e.g. `file_read` requires
|
||||
`max_chars` for files >10KB, `python_exec` lists available sandbox
|
||||
builtins).
|
||||
|
||||
**Modules:**
|
||||
- `cleveractors.agents.llm_tools._BUILTIN_TOOL_SCHEMAS`
|
||||
- `cleveractors.agents.llm_tools.normalize_tool_entry`
|
||||
|
||||
### D-2: `file_read` Directory Listing
|
||||
|
||||
**What:** When given a directory path, `file_read` returns a formatted
|
||||
listing with file sizes and type indicators (`📁` for directories,
|
||||
`📄` for files) instead of failing silently.
|
||||
|
||||
**Spec relationship:** §4.5.1 defines `file_read` for reading file
|
||||
contents only. Directory listing is an extension. Per §1.3, compliant
|
||||
implementations MAY provide additional features beyond those defined
|
||||
in the normative specification.
|
||||
|
||||
**Output format:** `[FILE_READ_SUCCESS]📁 Directory: <path>` with
|
||||
one entry per line: ` <marker> <name> (<size> bytes)`.
|
||||
|
||||
### D-3: `file_read` `max_chars` Truncation
|
||||
|
||||
**What:** New optional `max_chars` integer parameter. When set, file
|
||||
content is truncated to the specified limit. The output header
|
||||
includes `TRUNCATED to N chars` when truncation applies.
|
||||
|
||||
**Spec relationship:** §4.5.1 allows additional arguments to be handled
|
||||
by tool implementations. `max_chars` extends the argument set without
|
||||
removing or altering any existing mandated parameter. This prevents
|
||||
models from reading multi-hundred-KB files and overflowing the 262K
|
||||
token context window.
|
||||
|
||||
### D-4: `file_read` Shell Command Detection and File-Not-Found Recovery
|
||||
|
||||
**What:**
|
||||
- **Shell command detection:** Before opening a file, `file_read`
|
||||
checks if the path contains shell operators (`|`, `;`, `&&`, `||`,
|
||||
`>`, `<`) or starts with known shell commands (`grep`, `ls`, `find`,
|
||||
etc.). If detected, it returns an explicit error redirecting to the
|
||||
shell tool.
|
||||
- **File-not-found recovery:** When a file is not found, the error
|
||||
message includes the parent directory path and instructs the LLM to
|
||||
list that directory for recovery.
|
||||
|
||||
**Spec relationship:** These are error-quality improvements. §4.5.1
|
||||
does not prescribe error message formats, and §1.3 allows quality
|
||||
improvements that do not alter normative behavior. The mandated
|
||||
behavior (reading existing files, returning error on failure) is
|
||||
preserved — only the error message quality is enhanced.
|
||||
|
||||
### D-5: `python_exec` Sandbox `stdout` Capture
|
||||
|
||||
**What:** `sys.stdout` is now redirected to `io.StringIO()` during
|
||||
sandbox execution, and captured output is included in the result.
|
||||
Previously only `stderr` was redirected.
|
||||
|
||||
**Spec relationship:** §13.2.1 defines the sandbox `__builtins__` dict
|
||||
and security model. `stdout` capture is an implementation detail that
|
||||
improves tool behavior without changing the sandbox security model.
|
||||
The `__builtins__` dict remains strictly compliant — no additional
|
||||
builtins were added. `print()` is already in the allowed builtins
|
||||
list; capturing its output makes it useful.
|
||||
|
||||
### D-6: `python_exec` Sandbox `NameError` Guidance
|
||||
|
||||
**What:** When sandbox code references an undefined name (e.g. `open`
|
||||
or `exec`), the error message now extracts the name and provides
|
||||
actionable guidance directing the LLM to use `file_read` and
|
||||
`file_write` tools for file operations and the `shell` tool for
|
||||
system commands.
|
||||
|
||||
**Spec relationship:** §13.2.3 requires error messages for prohibited
|
||||
operations. This enhancement improves error messages without exposing
|
||||
prohibited facilities. The sandbox security model is unchanged — the
|
||||
offending name is never made available; only the name itself is
|
||||
reported back for guidance.
|
||||
|
||||
### D-7: `shell` and `python_exec` Tool Registration
|
||||
|
||||
**What:** The `shell` and `python_exec` tool methods existed on
|
||||
`ToolAgent` but were never added to `builtin_tools`. They are now
|
||||
registered when `allow_shell` or `exec_python` is enabled in the
|
||||
agent config.
|
||||
|
||||
**Spec relationship:** The `shell` and `python_exec` tools are
|
||||
implementation extensions. They are not listed in §4.5.1's built-in
|
||||
tools table, but §1.3 permits compliant implementations to "provide
|
||||
additional agent types, node types, operators, conditions, or routing
|
||||
match types beyond those defined here."
|
||||
|
||||
These tools are gated behind explicit opt-in flags (`allow_shell`,
|
||||
`exec_python`) that default to `false`, ensuring security-by-default.
|
||||
|
||||
### D-8: Empty Message Filtering in Conversation History
|
||||
|
||||
**What:** `_prepare_conversation_history()` in `cleveractors.langgraph.nodes`
|
||||
now skips messages whose content is empty or whitespace-only. Only
|
||||
messages with meaningful content are included in conversation history
|
||||
passed to downstream agents.
|
||||
|
||||
**Spec relationship:** §6.3.1 defines the graph state structure
|
||||
(messages, metadata, etc.) but does not mandate that empty-content
|
||||
messages must be preserved. Filtering improves the quality of
|
||||
conversation history without changing any externally observable
|
||||
behavior defined by the specification. The filtering is applied
|
||||
during history preparation, not during state mutation — the raw
|
||||
graph state is untouched.
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- **LLM self-correction:** Tool errors, shell command misrouting, and
|
||||
file-not-found failures now provide actionable recovery paths that
|
||||
the LLM can act on, reducing the need for human intervention.
|
||||
- **Context window safety:** `max_chars` truncation plus schema-level
|
||||
`CRITICAL` warnings prevent the most common cause of request failure
|
||||
(context overflow from reading large files).
|
||||
- **Security-by-default:** `shell` and `python_exec` are opt-in, gated
|
||||
behind explicit agent config flags.
|
||||
- **No spec violations:** Every extension is justified by §1.3
|
||||
(extensibility clause), §4.5.1 (additional arguments permitted), or
|
||||
falls within implementation-quality territory not normatively
|
||||
constrained by the specification.
|
||||
- **Backward compatibility:** All new arguments are optional. Existing
|
||||
YAML configs that do not use these extensions continue to work
|
||||
unchanged.
|
||||
|
||||
### Negative / Risks
|
||||
|
||||
- **Schema drift:** `_BUILTIN_TOOL_SCHEMAS` must be kept in sync with
|
||||
actual tool implementations. If a tool's parameter set changes but
|
||||
the schema is not updated, the LLM may call the tool with wrong
|
||||
arguments.
|
||||
- **Output format stability:** The directory listing format and
|
||||
`TRUNCATED` header are not part of the normative spec. If downstream
|
||||
code parses `file_read` output, format changes could break parsing.
|
||||
This is mitigated by the `[FILE_READ_SUCCESS]` prefix remaining
|
||||
stable.
|
||||
- **Increased module surface:** `llm_tools.py` adds a new module to the
|
||||
`cleveractors.agents` package. If LLM tool calling is later moved to
|
||||
a dedicated subsystem (e.g. a `ToolCallingService`), this module may
|
||||
need to be relocated or reorganized.
|
||||
|
||||
### Follow-up Required
|
||||
|
||||
- **ADR-2030 compliance validation:** Add BDD scenarios that verify
|
||||
each extension's spec compliance boundary (e.g. verify that no
|
||||
prohibited `__builtins__` entry was added to the sandbox).
|
||||
- **Schema synchronization test:** Add a test that verifies every
|
||||
entry in `_BUILTIN_TOOL_SCHEMAS` corresponds to a registered tool
|
||||
in `ToolAgent.builtin_tools`, and vice versa.
|
||||
- **Specification update:** If `shell`, `python_exec`, or `file_read`
|
||||
directory listing graduate to normative status, update §4.5.1 of
|
||||
the Actor Configuration Standard via a separate ADR.
|
||||
|
||||
---
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### A-1: Keep all tools within §4.5.1 and reject extensions
|
||||
|
||||
**Rejected because:** §4.5.1's tool set was designed for direct human
|
||||
invocation, not LLM-driven tool calling. Without directory listing,
|
||||
`max_chars`, and sandbox guidance, the LLM cannot self-navigate a
|
||||
codebase, cannot read large files safely, and cannot self-correct
|
||||
from common mistakes. The seven §4.5.1 tools are insufficient for
|
||||
autonomous multi-turn tool use.
|
||||
|
||||
### A-2: Add all extensions to the normative specification
|
||||
|
||||
**Rejected because:** The extensions have not yet been battle-tested
|
||||
at scale. Directory listing format, `max_chars` default values, and
|
||||
sandbox guidance wording are likely to evolve as we observe real LLM
|
||||
behavior. Committing them to the normative spec now would require
|
||||
a deprecation cycle for any future adjustment.
|
||||
|
||||
### A-3: Handle `max_chars` as a separate `head`/`tail` tool
|
||||
|
||||
**Rejected because:** The LLM already knows about `file_read` from
|
||||
schema descriptions. Adding a separate truncation tool would require
|
||||
the LLM to learn a new tool name and decide when to use it, increasing
|
||||
the cognitive load on the model and the likelihood of incorrect tool
|
||||
selection. Integrating truncation into `file_read` as an optional
|
||||
parameter keeps the tool surface simple.
|
||||
|
||||
### A-4: Reject shell command detection in `file_read`
|
||||
|
||||
**Rejected because:** Without detection, the LLM's `FileNotFoundError`
|
||||
for a path like `"grep -n pattern file | head"` is confusing and
|
||||
offers no recovery path. The LLM cannot distinguish between "file
|
||||
doesn't exist" and "I accidentally passed a shell command." Detection
|
||||
and redirection give the model the information it needs to correct
|
||||
its own mistake.
|
||||
|
||||
### A-5: Filter empty messages at the graph orchestration level instead of in `nodes.py`
|
||||
|
||||
**Rejected because:** Filtering in `_prepare_conversation_history()`
|
||||
is the single choke point through which all conversation history
|
||||
passes before being sent to an agent. Filtering at the graph
|
||||
orchestration level (e.g. in `pure_graph.py`) would require changes
|
||||
at every call site that builds history. The node-level approach
|
||||
applies the fix globally with minimal code change and zero risk of
|
||||
missing a call site.
|
||||
@@ -59,7 +59,6 @@ Feature: Registry Cache Coverage
|
||||
Scenario: CacheFactory with invalid max_size raises ValueError
|
||||
When I try to create a CacheFactory with max_size=0
|
||||
Then a ValueError should be raised by the cache factory
|
||||
|
||||
Scenario: CacheFactory with invalid ttl raises ValueError
|
||||
When I try to create a CacheFactory with ttl=-1.0
|
||||
Then a ValueError should be raised by the cache factory
|
||||
@@ -118,3 +117,15 @@ Feature: Registry Cache Coverage
|
||||
When I call resolve_package with type=actor ns=bench name=pkg3 version=v1.0.0
|
||||
Then the resolve result should contain package_id
|
||||
Then the cache stats evictions should be 1
|
||||
|
||||
Scenario: Resolve package TTL expiry triggers re-fetch
|
||||
When I create a RegistryCache with ttl 0.01 and mock client
|
||||
When I call resolve_package with type=actor ns=bench name=pkg version=v1.0.0
|
||||
When I wait for TTL to expire
|
||||
When I call resolve_package with type=actor ns=bench name=pkg version=v1.0.0
|
||||
Then the cache stats misses should be 2
|
||||
|
||||
Scenario: Cache content validation returns False when canonicalizer raises TypeError
|
||||
When I create a RegistryCache with validation enabled and mock client
|
||||
When I trigger SHA-1 validation with a TypeError-raising canonicalizer
|
||||
Then the validation should return False
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
Feature: LLMAgent Tool Calling
|
||||
|
||||
Scenario: LLMAgent reads tools from config at init
|
||||
Given I have an LLM agent configuration with tools [{"name": "echo"}, {"name": "math"}]
|
||||
When I create the LLM agent
|
||||
Then the agent should store tool definitions in _lc_tools with count 2
|
||||
|
||||
Scenario: LLMAgent without tools has empty _lc_tools
|
||||
Given I have a basic LLM agent configuration (no tools)
|
||||
And that config does NOT include a tools field
|
||||
When I create the LLM agent
|
||||
Then the agent _lc_tools should be None
|
||||
|
||||
Scenario: LLMAgent get_capabilities includes tool-calling when tools configured
|
||||
Given I have an LLM agent configuration with tools [{"name": "echo"}]
|
||||
When I create the LLM agent
|
||||
And I request the agent capabilities (tool calling)
|
||||
Then the capabilities should include "tool-calling"
|
||||
And the capabilities should still include "text-generation"
|
||||
|
||||
Scenario: LLMAgent get_capabilities without tool-calling when no tools
|
||||
Given I have a basic LLM agent configuration
|
||||
And that config does NOT include a tools field
|
||||
When I create the LLM agent
|
||||
And I request the agent capabilities (tool calling)
|
||||
Then the capabilities should include "text-generation"
|
||||
And the capabilities should NOT include "tool-calling"
|
||||
|
||||
Scenario: LLMAgent process_message passes tools to model when configured
|
||||
Given I have an LLM agent configuration with tools [{"name": "echo"}]
|
||||
When I create the LLM agent
|
||||
And I set a mock chat model that returns a plain text response
|
||||
And I tool_calling process a message "Hello"
|
||||
Then the chat_model ainvoke should have been called with a tools keyword
|
||||
|
||||
Scenario: LLMAgent process_message without tools does not pass tools to model
|
||||
Given I have a basic LLM agent configuration
|
||||
And that config does NOT include a tools field
|
||||
When I create the LLM agent
|
||||
And I set a mock chat model that returns a plain text response
|
||||
And I tool_calling process a message "Hello"
|
||||
Then the chat_model ainvoke should NOT have been called with a tools keyword
|
||||
|
||||
Scenario: LLMAgent _lc_tools passed to model as invoke keyword
|
||||
Given I have an LLM agent configuration with tools [{"name": "file_read"}]
|
||||
When I create the LLM agent
|
||||
And I set a mock chat model that returns a plain text response
|
||||
And I tool_calling process a message "Read a file"
|
||||
Then the chat_model ainvoke should have been called with a tools keyword
|
||||
|
||||
Scenario: LLMAgent metadata includes tool counts when tools configured
|
||||
Given I have an LLM agent configuration with tools [{"name": "echo"}, {"name": "math"}]
|
||||
When I create the LLM agent
|
||||
And I request the agent metadata (tool calling)
|
||||
Then the metadata should have "tools_configured" set to true
|
||||
And the metadata should have "tool_count" equal to 2
|
||||
|
||||
Scenario: LLMAgent injects synthesis prompt when model gets stuck in tool-only mode
|
||||
Given I have an LLM agent configuration with tools [{"name": "echo"}]
|
||||
And I set tool_max_rounds to 2
|
||||
When I create the LLM agent
|
||||
And I set a mock chat model that returns tool_calls with empty content for 2 rounds then text
|
||||
And I tool_calling process a message "Do something"
|
||||
Then the synthesis HumanMessage should have been appended to messages
|
||||
And the final ainvoke should have been called
|
||||
And the tool_calling result should contain "Synthesized answer"
|
||||
|
||||
Scenario: LLMAgent propagates tool execution errors to ToolMessage for LLM self-correction
|
||||
Given I have an LLM agent configuration with tools [{"name": "echo"}]
|
||||
And I set tool_max_rounds to 1
|
||||
When I create the LLM agent
|
||||
And I set a mock chat model that triggers a tool with ExecutionError "test tool failure"
|
||||
And I tool_calling process a message "Fail me"
|
||||
Then the messages list should contain a ToolMessage with content containing _nonexistent_tool_xyz
|
||||
|
||||
Scenario: LLMAgent handles empty tool name in malformed tool_calls
|
||||
Given I have an LLM agent configuration with tools [{"name": "echo"}]
|
||||
And I set tool_max_rounds to 1
|
||||
When I create the LLM agent
|
||||
And I set a mock chat model that returns tool_calls with an empty tool name
|
||||
And I tool_calling process a message "Call me"
|
||||
Then the messages list should contain a ToolMessage with content "Tool name is empty"
|
||||
@@ -82,4 +82,9 @@ Feature: LLM Agent Temperature Override, Cleanup, and Context History
|
||||
Scenario: Cleanup is a no-op when chat_model has no http clients
|
||||
Given I setup an LLM agent with a chat model that lacks root_async_client and root_client (llm_gaps)
|
||||
When I await the cleanup method (llm_gaps)
|
||||
Then the cleanup should complete without raising errors (llm_gaps)
|
||||
Then the cleanup should complete without raising errors (llm_gaps)
|
||||
|
||||
Scenario: Non-numeric _temperature_override raises ConfigurationError in process_message
|
||||
Given I setup a test LLM agent for temperature override tests (llm_gaps)
|
||||
When I call process_message with _temperature_override "bad_string" (llm_gaps)
|
||||
Then a ConfigurationError mentioning must be a number is raised (llm_gaps)
|
||||
@@ -0,0 +1,34 @@
|
||||
Feature: LLM Tools Coverage
|
||||
As a developer
|
||||
I want to cover remaining uncovered lines in llm_tools.py
|
||||
So that coverage reaches the target
|
||||
|
||||
Scenario: normalize_tool_entry returns OpenAI-formatted dict as-is
|
||||
Given an llm_tools coverage test environment
|
||||
When I call normalize_tool_entry with OpenAI-formatted dict
|
||||
Then the result should be the same dict unchanged
|
||||
|
||||
Scenario: normalize_tool_entry rejects dict missing valid name
|
||||
Given an llm_tools coverage test environment
|
||||
When I call normalize_tool_entry with dict missing name
|
||||
Then a ConfigurationError mentioning non-empty string 'name' is raised
|
||||
|
||||
Scenario: normalize_tool_entry rejects dict with empty name
|
||||
Given an llm_tools coverage test environment
|
||||
When I call normalize_tool_entry with dict having empty name
|
||||
Then a ConfigurationError mentioning non-empty string 'name' is raised
|
||||
|
||||
Scenario: normalize_tool_entry converts string tool name to OpenAI format
|
||||
Given an llm_tools coverage test environment
|
||||
When I call normalize_tool_entry with string "echo"
|
||||
Then the result should be OpenAI dict with name echo
|
||||
|
||||
Scenario: normalize_tool_entry rejects empty string
|
||||
Given an llm_tools coverage test environment
|
||||
When I call normalize_tool_entry with empty string
|
||||
Then a ConfigurationError mentioning non-empty string is raised
|
||||
|
||||
Scenario: normalize_tool_entry rejects non-str non-dict type
|
||||
Given an llm_tools coverage test environment
|
||||
When I call normalize_tool_entry with integer 42
|
||||
Then a ConfigurationError mentioning Invalid tool config entry is raised
|
||||
@@ -36,3 +36,26 @@ Feature: Package Registry Version Resolution Error Branches
|
||||
Given a reference resolver is created without client and without local store
|
||||
When resolving an unknown reference type with resolver
|
||||
Then ResErr: InvalidPackageReferenceError raised containing "Unknown"
|
||||
|
||||
Scenario: _pick_latest with empty list raises error
|
||||
When _pick_latest is called with empty versions list
|
||||
Then ResErr: InvalidPackageReferenceError raised containing "No versions available to pick"
|
||||
|
||||
Scenario: Global alias with no concrete versions raises error
|
||||
When resolve_version is called with alias "latest" against available versions "v1.x,v2.x"
|
||||
Then ResErr: InvalidPackageReferenceError raised containing "No concrete versions available"
|
||||
|
||||
Scenario: Registry reference missing namespace or name raises error
|
||||
Given a reference resolver is created without client and without local store
|
||||
When resolving a registry reference with missing namespace
|
||||
Then ResErr: InvalidPackageReferenceError raised containing "missing namespace or name"
|
||||
|
||||
Scenario: Registry response missing package_id raises error
|
||||
Given a reference resolver is created with a mock client returning no package_id
|
||||
When resolving a registry reference with the mock client
|
||||
Then ResErr: InvalidPackageReferenceError raised containing "Registry response missing"
|
||||
|
||||
Scenario: Registry response with invalid package_id raises error
|
||||
Given a reference resolver is created with a mock client returning invalid package_id
|
||||
When resolving a registry reference with the mock client
|
||||
Then ResErr: InvalidPackageReferenceError raised containing "Invalid package_id"
|
||||
|
||||
@@ -173,3 +173,7 @@ Feature: Runtime Executor API
|
||||
Given an Executor for a graph actor using v2.0 actors key instead of agents key
|
||||
When I execute the graph actor for validation test with message "test actors key"
|
||||
Then the execution should return an ActorResult
|
||||
|
||||
Scenario: runtime_types backward-compatibility shim is importable
|
||||
When I import from cleaveractors.runtime_types
|
||||
Then ActorResult and NodeUsage should be importable
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
Feature: Runtime Dispatch Normalisation
|
||||
As a developer
|
||||
I want graph placeholder node IDs to be normalised to the canonical format
|
||||
So that multi-actor prefixes produce consistent node identifiers
|
||||
|
||||
Background:
|
||||
Given the runtime dispatch test context is initialised
|
||||
|
||||
Scenario: Graph placeholder node_id is normalised to canonical form
|
||||
When I normalise a graph placeholder node_id "<test_graph:no_llm>"
|
||||
Then the result should be the canonical "<no_llm>"
|
||||
|
||||
Scenario: Regular node_id is returned unchanged by normalise
|
||||
When I normalise a regular node_id "agent.node_1"
|
||||
Then the result should be the original node_id
|
||||
@@ -358,3 +358,38 @@ def step_total_stats_misses(context: Any, expected: int) -> None:
|
||||
assert context.stats.misses == expected, (
|
||||
f"Expected total_stats.misses={expected}, got {context.stats.misses}"
|
||||
)
|
||||
|
||||
|
||||
@when("I create a RegistryCache with ttl {ttl:f} and mock client")
|
||||
def step_create_cache_short_ttl(context: Any, ttl: float) -> None:
|
||||
context.cache = RegistryCache(
|
||||
_create_mock_client(context), ttl=ttl, validate_content=False
|
||||
)
|
||||
|
||||
|
||||
@when("I wait for TTL to expire")
|
||||
def step_wait_ttl_expire(context: Any) -> None:
|
||||
time.sleep(0.1)
|
||||
|
||||
|
||||
@when("I trigger SHA-1 validation with a TypeError-raising canonicalizer")
|
||||
def step_cache_val_typeerror(context: Any) -> None:
|
||||
from cleveractors.registry.types import PackageContent, PackageId
|
||||
|
||||
cache = context.cache
|
||||
cache._validate_content_enabled = True
|
||||
|
||||
def _bad_compute(content_data, pkg_type):
|
||||
raise TypeError("bogus type")
|
||||
|
||||
cache._canonicalizer.compute_package_id = _bad_compute
|
||||
pid = PackageId.from_string("pkg_act_0123456789abcdef0123456789abcdef01234567")
|
||||
content = PackageContent(id=pid, content={"key": "val"})
|
||||
context.cache_val_result = cache._validate_content(content)
|
||||
|
||||
|
||||
@then("the validation should return False")
|
||||
def step_cache_val_false(context: Any) -> None:
|
||||
assert context.cache_val_result is False, (
|
||||
f"Expected False, got {context.cache_val_result}"
|
||||
)
|
||||
|
||||
@@ -2784,6 +2784,18 @@ async def step_es_stream_message(context: Any) -> None:
|
||||
_root_logger = logging.getLogger()
|
||||
_root_logger.addHandler(_handler)
|
||||
|
||||
# Also patch _log_no_usage_metadata to record calls directly;
|
||||
# this is robust even when slipcover wraps the logging machinery
|
||||
# (coverage_report session).
|
||||
_cause_records: list[str] = []
|
||||
_original = agent._log_no_usage_metadata
|
||||
|
||||
def _tracking_log(self: Any, cause: str) -> None:
|
||||
_cause_records.append(cause)
|
||||
_original(cause)
|
||||
|
||||
agent._log_no_usage_metadata = _tracking_log.__get__(agent, type(agent)) # type: ignore[assignment]
|
||||
|
||||
try:
|
||||
tokens: list[str] = []
|
||||
async for token in agent.stream_message("Hello", None):
|
||||
@@ -2799,8 +2811,10 @@ async def step_es_stream_message(context: Any) -> None:
|
||||
context.es_captured_usage_var = (0, 0)
|
||||
finally:
|
||||
_root_logger.removeHandler(_handler)
|
||||
agent._log_no_usage_metadata = _original # type: ignore[method-assign]
|
||||
|
||||
context.es_captured_warnings = captured_warnings
|
||||
context.es_log_causes = _cause_records
|
||||
|
||||
|
||||
@when("I call stream_message with _temperature_override {override} in context (stream)")
|
||||
@@ -2993,6 +3007,13 @@ def step_es_no_usage_warning_emitted(context: Any) -> None:
|
||||
_CAUSE_RESPONSE_METADATA_NO_TOKEN_USAGE,
|
||||
_CAUSE_RESPONSE_METADATA_NOT_DICT,
|
||||
)
|
||||
# Primary check: the tracking wrapper recorded calls to _log_no_usage_metadata.
|
||||
# This is robust across normal behave and slipcover (coverage_report).
|
||||
_log_causes = getattr(context, "es_log_causes", None)
|
||||
if _log_causes:
|
||||
return # at least one cause was recorded — warning was emitted
|
||||
|
||||
# Fallback (pre-existing): check the logging handler captured a warning record.
|
||||
captured = getattr(context, "es_captured_warnings", [])
|
||||
matching = [
|
||||
r for r in captured if any(cause in r.getMessage() for cause in _cause_strings)
|
||||
|
||||
@@ -0,0 +1,363 @@
|
||||
"""Step definitions for LLMAgent Tool Calling BDD tests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json as _json
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from unittest.mock import AsyncMock, Mock, patch
|
||||
|
||||
from behave import given, then, when
|
||||
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from langchain_core.language_models.chat_models import BaseChatModel
|
||||
else:
|
||||
# At runtime we need the actual class for Mock(spec=...)
|
||||
from langchain_core.language_models.chat_models import BaseChatModel
|
||||
|
||||
from cleveractors.agents.llm import LLMAgent
|
||||
from cleveractors.core.exceptions import ConfigurationError
|
||||
from cleveractors.templates.renderer import TemplateRenderer
|
||||
|
||||
|
||||
def _create_mock_response(text: str, tool_calls: list[dict[str, Any]] | None = None):
|
||||
"""Create a mock AIMessage response."""
|
||||
mock_response = Mock(spec=AIMessage)
|
||||
mock_response.content = text
|
||||
if tool_calls is not None:
|
||||
mock_response.tool_calls = tool_calls
|
||||
else:
|
||||
mock_response.tool_calls = None
|
||||
# Add token usage metadata for billing integrity
|
||||
mock_response.usage_metadata = {"input_tokens": 10, "output_tokens": 20}
|
||||
return mock_response
|
||||
|
||||
|
||||
def _make_llm_agent_with_config(config_dict: dict[str, Any]) -> LLMAgent:
|
||||
"""Create an LLMAgent with a mocked template renderer and pre-injected chat model."""
|
||||
renderer = Mock(spec=TemplateRenderer)
|
||||
renderer.render_string.return_value = "mocked system prompt"
|
||||
|
||||
agent = LLMAgent(
|
||||
name=config_dict.get("name", "tool_calling_agent"),
|
||||
config=config_dict,
|
||||
template_renderer=renderer,
|
||||
)
|
||||
return agent
|
||||
|
||||
|
||||
# ─── Given steps ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@given("I have an LLM agent configuration with tools {tools_json}")
|
||||
def step_config_with_tools(context, tools_json):
|
||||
"""Set up config that includes a tools list."""
|
||||
parsed = _json.loads(tools_json)
|
||||
context.llm_config = {
|
||||
"name": "tool_agent",
|
||||
"provider": "openai",
|
||||
"api_key": "test_key",
|
||||
"tools": parsed,
|
||||
}
|
||||
|
||||
|
||||
@given("I have a basic LLM agent configuration (no tools)")
|
||||
def step_basic_llm_config(context):
|
||||
"""Set up config without tools."""
|
||||
context.llm_config = {
|
||||
"name": "basic_agent",
|
||||
"provider": "openai",
|
||||
"api_key": "test_key",
|
||||
}
|
||||
|
||||
|
||||
@given("that config does NOT include a tools field")
|
||||
def step_no_tools_in_config(context):
|
||||
"""Ensure the current llm_config has no tools key."""
|
||||
context.llm_config.pop("tools", None)
|
||||
|
||||
|
||||
# ─── When steps ───────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@when("I create the LLM agent")
|
||||
def step_create_llm_agent(context):
|
||||
"""Create the LLMAgent."""
|
||||
context.llm_agent = _make_llm_agent_with_config(context.llm_config)
|
||||
|
||||
|
||||
@when("I set a mock chat model that returns a plain text response")
|
||||
def step_set_mock_plain_response(context):
|
||||
"""Inject a mock chat model returning plain-text AIMessage."""
|
||||
mock_model = Mock(spec=BaseChatModel)
|
||||
mock_model.ainvoke = AsyncMock(
|
||||
return_value=_create_mock_response("Plain text response")
|
||||
)
|
||||
mock_model.tool_calls = None
|
||||
context.llm_agent.chat_model = mock_model
|
||||
|
||||
|
||||
@when(
|
||||
"I set a mock chat model that returns an AIMessage with tool_calls {tool_calls_json}"
|
||||
)
|
||||
def step_set_mock_tool_calls_response(context, tool_calls_json):
|
||||
"""Inject a mock chat model returning AIMessage with tool_calls."""
|
||||
parsed = _json.loads(tool_calls_json)
|
||||
mock_model = Mock(spec=BaseChatModel)
|
||||
mock_model.ainvoke = AsyncMock(
|
||||
return_value=_create_mock_response(
|
||||
"Final answer after tools", tool_calls=parsed
|
||||
)
|
||||
)
|
||||
context.llm_agent.chat_model = mock_model
|
||||
|
||||
|
||||
@when("I tool_calling process a message {message_txt}")
|
||||
def step_process_message(context, message_txt):
|
||||
"""Process a message through the agent."""
|
||||
context.result = asyncio.run(context.llm_agent.process_message(message_txt))
|
||||
|
||||
|
||||
@when("I request the agent capabilities (tool calling)")
|
||||
def step_request_capabilities(context):
|
||||
"""Get agent capabilities."""
|
||||
context.capabilities = context.llm_agent.get_capabilities()
|
||||
|
||||
|
||||
@when("I request the agent metadata (tool calling)")
|
||||
def step_request_metadata(context):
|
||||
"""Get agent metadata."""
|
||||
context.metadata = context.llm_agent.get_metadata()
|
||||
|
||||
|
||||
# ─── Then steps ───────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@then("the agent should store tool definitions in _lc_tools with count {n:d}")
|
||||
def step_lctools_count(context, n):
|
||||
"""Verify _lc_tools was populated with expected count."""
|
||||
assert context.llm_agent._lc_tools is not None
|
||||
assert len(context.llm_agent._lc_tools) == n
|
||||
|
||||
|
||||
@then("the agent _lc_tools should be None")
|
||||
def step_lctools_none(context):
|
||||
"""Verify _lc_tools is None when no tools config."""
|
||||
assert context.llm_agent._lc_tools is None
|
||||
|
||||
|
||||
@then('the capabilities should include "{capability}"')
|
||||
@then('the capabilities should still include "{capability}"')
|
||||
def step_capability_includes(context, capability):
|
||||
"""Check that a specific capability is present."""
|
||||
assert capability in context.capabilities
|
||||
|
||||
|
||||
@then('the capabilities should NOT include "{capability}"')
|
||||
@then('the capabilities should not include "{capability}"')
|
||||
def step_capability_not_includes(context, capability):
|
||||
"""Check that a capability is absent."""
|
||||
assert capability not in context.capabilities
|
||||
|
||||
|
||||
@then("the chat_model ainvoke should have been called with a tools keyword")
|
||||
def step_ainvoke_has_tools_kwarg(context):
|
||||
"""Verify ainvoke was called with the ``tools`` kwarg."""
|
||||
call = context.llm_agent.chat_model.ainvoke.call_args
|
||||
if call is None:
|
||||
raise AssertionError("ainvoke was never called")
|
||||
kwargs = call.kwargs if hasattr(call, "kwargs") else call[1]
|
||||
assert "tools" in kwargs, (
|
||||
f"ainvoke was called but without a ``tools`` kwarg. "
|
||||
f"call_kwargs={list(kwargs.keys())}"
|
||||
)
|
||||
|
||||
|
||||
@then("the chat_model ainvoke should NOT have been called with a tools keyword")
|
||||
def step_ainvoke_no_tools_kwarg(context):
|
||||
"""Verify ainvoke was NOT passed the ``tools`` kwarg."""
|
||||
call = context.llm_agent.chat_model.ainvoke.call_args
|
||||
if call is None:
|
||||
raise AssertionError("ainvoke was never called")
|
||||
kwargs = call.kwargs if hasattr(call, "kwargs") else call[1]
|
||||
assert "tools" not in kwargs
|
||||
|
||||
|
||||
@then("the tool_calling result should contain {expected}")
|
||||
def step_result_contains(context, expected):
|
||||
"""Check that the result string contains an expected substring."""
|
||||
if (
|
||||
isinstance(expected, str)
|
||||
and expected.startswith('"')
|
||||
and expected.endswith('"')
|
||||
):
|
||||
expected = expected[1:-1]
|
||||
assert context.result is not None
|
||||
assert expected in str(context.result)
|
||||
|
||||
|
||||
@then('the metadata should have "{field}" set to {value}')
|
||||
def step_metadata_field_value_true(context, field, value):
|
||||
"""Check a boolean metadata field."""
|
||||
assert context.metadata is not None
|
||||
field_value = context.metadata.get(field)
|
||||
if field_value is None:
|
||||
raise AssertionError(f"metadata missing key {field!r}")
|
||||
actual_lower = str(field_value).lower()
|
||||
expected_lower = value.lower()
|
||||
assert actual_lower == expected_lower, (
|
||||
f"metadata.{field} = {field_value!r} != {value!r}"
|
||||
)
|
||||
|
||||
|
||||
@then('the metadata should have "{field}" equal to {n:d}')
|
||||
def step_metadata_field_int(context, field, n):
|
||||
"""Check an integer metadata field."""
|
||||
assert context.metadata is not None
|
||||
actual = context.metadata.get(field)
|
||||
assert actual == n, f"metadata.{field} = {actual!r} != {n}"
|
||||
|
||||
|
||||
# ─── Synthesis-prompt branch ─────────────────────────────────────────────
|
||||
|
||||
|
||||
@given("I set tool_max_rounds to {n:d}")
|
||||
def step_set_tool_max_rounds(context, n):
|
||||
context.llm_config["tool_max_rounds"] = n
|
||||
|
||||
|
||||
@when(
|
||||
"I set a mock chat model that returns tool_calls with empty content "
|
||||
"for {n:d} rounds then text"
|
||||
)
|
||||
def step_set_mock_stuck_synthesis(context, n):
|
||||
call_counter = [0]
|
||||
context._synthesis_messages_captured = None
|
||||
context._synthesis_final_called = False
|
||||
|
||||
async def _mock_ainvoke(messages, **invoke_kwargs):
|
||||
call_counter[0] += 1
|
||||
if call_counter[0] <= n:
|
||||
return _create_mock_response(
|
||||
"",
|
||||
tool_calls=[
|
||||
{
|
||||
"id": f"call_{call_counter[0]}",
|
||||
"type": "function",
|
||||
"function": {"name": "echo", "arguments": '{"text": "test"}'},
|
||||
}
|
||||
],
|
||||
)
|
||||
context._synthesis_final_called = True
|
||||
context._synthesis_messages_captured = list(messages)
|
||||
return _create_mock_response("Synthesized answer")
|
||||
|
||||
mock_model = Mock(spec=BaseChatModel)
|
||||
mock_model.ainvoke = _mock_ainvoke
|
||||
context.llm_agent.chat_model = mock_model
|
||||
|
||||
|
||||
@then("the synthesis HumanMessage should have been appended to messages")
|
||||
def step_synthesis_human_message_appended(context):
|
||||
messages = context._synthesis_messages_captured
|
||||
assert messages is not None, "Synthesis ainvoke was never called"
|
||||
human_found = any(
|
||||
isinstance(m, HumanMessage)
|
||||
and "finished gathering information" in str(m.content)
|
||||
for m in messages
|
||||
)
|
||||
assert human_found, (
|
||||
f"Synthesis HumanMessage not found in messages: "
|
||||
f"{[type(m).__name__ for m in messages]}"
|
||||
)
|
||||
|
||||
|
||||
@then("the final ainvoke should have been called")
|
||||
def step_final_ainvoke_called(context):
|
||||
assert context._synthesis_final_called, "Final synthesis ainvoke was not called"
|
||||
|
||||
|
||||
# ─── Tool-error propagation ──────────────────────────────────────────────
|
||||
|
||||
|
||||
@when("I set a mock chat model that triggers a tool with ExecutionError {error_text}")
|
||||
def step_set_mock_tool_error(context, error_text):
|
||||
call_counter = [0]
|
||||
context._tc_messages_captured = None
|
||||
|
||||
async def _mock_ainvoke(messages, **invoke_kwargs):
|
||||
call_counter[0] += 1
|
||||
if call_counter[0] == 1:
|
||||
return _create_mock_response(
|
||||
"",
|
||||
tool_calls=[
|
||||
{
|
||||
"id": "call_err",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "_nonexistent_tool_xyz",
|
||||
"arguments": "{}",
|
||||
},
|
||||
}
|
||||
],
|
||||
)
|
||||
context._tc_messages_captured = list(messages)
|
||||
return _create_mock_response("Recovered from error")
|
||||
|
||||
mock_model = Mock(spec=BaseChatModel)
|
||||
mock_model.ainvoke = _mock_ainvoke
|
||||
context.llm_agent.chat_model = mock_model
|
||||
|
||||
|
||||
@then("the messages list should contain a ToolMessage with content containing {text}")
|
||||
def step_messages_has_toolmessage(context, text):
|
||||
messages = context._tc_messages_captured
|
||||
assert messages is not None, "ainvoke messages were not captured"
|
||||
tool_msgs = [m for m in messages if isinstance(m, ToolMessage)]
|
||||
found = any(text in str(m.content) for m in tool_msgs)
|
||||
assert found, (
|
||||
f"No ToolMessage found with content containing {text!r}. "
|
||||
f"ToolMessages: {[str(m.content) for m in tool_msgs]}"
|
||||
)
|
||||
|
||||
|
||||
# ─── Empty tool-name handling ────────────────────────────────────────────
|
||||
|
||||
|
||||
@when("I set a mock chat model that returns tool_calls with an empty tool name")
|
||||
def step_set_mock_empty_tool_name(context):
|
||||
call_counter = [0]
|
||||
context._empty_tool_captured = None
|
||||
|
||||
async def _mock_ainvoke(messages, **invoke_kwargs):
|
||||
call_counter[0] += 1
|
||||
if call_counter[0] == 1:
|
||||
return _create_mock_response(
|
||||
"",
|
||||
tool_calls=[
|
||||
{
|
||||
"id": "empty_001",
|
||||
"type": "function",
|
||||
"function": {"name": "", "arguments": "{}"},
|
||||
}
|
||||
],
|
||||
)
|
||||
context._empty_tool_captured = list(messages)
|
||||
return _create_mock_response("Final response after empty tool name")
|
||||
|
||||
mock_model = Mock(spec=BaseChatModel)
|
||||
mock_model.ainvoke = _mock_ainvoke
|
||||
context.llm_agent.chat_model = mock_model
|
||||
|
||||
|
||||
@then('the messages list should contain a ToolMessage with content "{text}"')
|
||||
def step_messages_has_exact_toolmessage(context, text):
|
||||
messages = context._empty_tool_captured
|
||||
assert messages is not None, "ainvoke messages were not captured"
|
||||
tool_msgs = [m for m in messages if isinstance(m, ToolMessage)]
|
||||
found = any(str(m.content) == text for m in tool_msgs)
|
||||
assert found, (
|
||||
f"No ToolMessage found with exact content {text!r}. "
|
||||
f"ToolMessages: {[str(m.content) for m in tool_msgs]}"
|
||||
)
|
||||
@@ -461,3 +461,25 @@ def step_ml_cleanup_completes_ok(context: Any) -> None:
|
||||
assert context.ml_cleanup_error is None, (
|
||||
f"Cleanup should complete without errors, got: {context.ml_cleanup_error}"
|
||||
)
|
||||
|
||||
|
||||
@when('I call process_message with _temperature_override "{override}" (llm_gaps)')
|
||||
@async_run_until_complete
|
||||
async def step_ml_process_bad_temp_override(context: Any, override: str) -> None:
|
||||
try:
|
||||
await context.ml_agent.process_message(
|
||||
"test", context={"_temperature_override": override}
|
||||
)
|
||||
context.ml_config_error = None
|
||||
except Exception as e:
|
||||
context.ml_config_error = e
|
||||
|
||||
|
||||
@then("a ConfigurationError mentioning {phrase} is raised (llm_gaps)")
|
||||
def step_ml_config_error_phrase(context: Any, phrase: str) -> None:
|
||||
assert context.ml_config_error is not None, (
|
||||
"Expected ConfigurationError but none was raised"
|
||||
)
|
||||
assert phrase in str(context.ml_config_error), (
|
||||
f"Expected error to contain '{phrase}', got: {context.ml_config_error}"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Step definitions for llm_tools.py coverage gaps."""
|
||||
|
||||
from behave import given, then, when
|
||||
|
||||
from cleveractors.agents.llm_tools import normalize_tool_entry
|
||||
from cleveractors.core.exceptions import ConfigurationError
|
||||
|
||||
|
||||
@given("an llm_tools coverage test environment")
|
||||
def step_lt_init(context):
|
||||
context.error = None
|
||||
context.result = None
|
||||
|
||||
|
||||
@when("I call normalize_tool_entry with OpenAI-formatted dict")
|
||||
def step_lt_openai_dict(context):
|
||||
item = {"type": "function", "function": {"name": "test", "description": "desc"}}
|
||||
context.result = normalize_tool_entry(item)
|
||||
|
||||
|
||||
@when("I call normalize_tool_entry with dict missing name")
|
||||
def step_lt_dict_no_name(context):
|
||||
try:
|
||||
normalize_tool_entry({"description": "no name"})
|
||||
except ConfigurationError as e:
|
||||
context.error = str(e)
|
||||
|
||||
|
||||
@when("I call normalize_tool_entry with dict having empty name")
|
||||
def step_lt_dict_empty_name(context):
|
||||
try:
|
||||
normalize_tool_entry({"name": ""})
|
||||
except ConfigurationError as e:
|
||||
context.error = str(e)
|
||||
|
||||
|
||||
@when('I call normalize_tool_entry with string "{tool_name}"')
|
||||
def step_lt_string_name(context, tool_name):
|
||||
context.result = normalize_tool_entry(tool_name)
|
||||
|
||||
|
||||
@then("the result should be OpenAI dict with name echo")
|
||||
def step_lt_openai_result_echo(context):
|
||||
assert context.result["type"] == "function"
|
||||
assert context.result["function"]["name"] == "echo"
|
||||
|
||||
|
||||
@when("I call normalize_tool_entry with empty string")
|
||||
def step_lt_empty_string(context):
|
||||
try:
|
||||
normalize_tool_entry("")
|
||||
except ConfigurationError as e:
|
||||
context.error = str(e)
|
||||
|
||||
|
||||
@when("I call normalize_tool_entry with integer 42")
|
||||
def step_lt_integer(context):
|
||||
try:
|
||||
normalize_tool_entry(42)
|
||||
except ConfigurationError as e:
|
||||
context.error = str(e)
|
||||
|
||||
|
||||
@then("the result should be the same dict unchanged")
|
||||
def step_lt_same_dict(context):
|
||||
assert context.result == {
|
||||
"type": "function",
|
||||
"function": {"name": "test", "description": "desc"},
|
||||
}
|
||||
|
||||
|
||||
@then("a ConfigurationError mentioning {text} is raised")
|
||||
def step_lt_config_error(context, text):
|
||||
assert context.error is not None
|
||||
assert text in context.error
|
||||
@@ -129,3 +129,131 @@ def step_assert_error_raised(context: object) -> None:
|
||||
f"Expected InvalidPackageReferenceError, got {type(context._last_exc)}: "
|
||||
f"{context._last_exc}"
|
||||
)
|
||||
|
||||
|
||||
@when("_pick_latest is called with empty versions list")
|
||||
def step_pick_latest_empty(context: object) -> None:
|
||||
from cleveractors.registry.resolver import _pick_latest
|
||||
|
||||
try:
|
||||
_pick_latest([])
|
||||
context._last_exc = None
|
||||
except InvalidPackageReferenceError as exc:
|
||||
context._last_exc = exc
|
||||
|
||||
|
||||
@given("a reference resolver is created with a mock client returning no package_id")
|
||||
def step_resolver_mock_no_package_id(context: object) -> None:
|
||||
from cleveractors.registry.client import RegistryClient
|
||||
|
||||
mock_client = MagicMock(spec=RegistryClient)
|
||||
|
||||
async def _resolve_no_id(*_args, **_kwargs):
|
||||
return {"other": "no package_id here"}
|
||||
|
||||
mock_client.resolve_package = _resolve_no_id
|
||||
resolver = ReferenceResolver(client=mock_client, local_store=None)
|
||||
context._resolver = resolver
|
||||
_bootstrap_parse(context)
|
||||
|
||||
|
||||
@given(
|
||||
"a reference resolver is created with a mock client returning invalid package_id"
|
||||
)
|
||||
def step_resolver_mock_invalid_package_id(context: object) -> None:
|
||||
from cleveractors.registry.client import RegistryClient
|
||||
|
||||
mock_client = MagicMock(spec=RegistryClient)
|
||||
|
||||
async def _resolve_bad_id(*_args, **_kwargs):
|
||||
return {"package_id": "not-a-valid-pkg-id"}
|
||||
|
||||
mock_client.resolve_package = _resolve_bad_id
|
||||
resolver = ReferenceResolver(client=mock_client, local_store=None)
|
||||
context._resolver = resolver
|
||||
_bootstrap_parse(context)
|
||||
|
||||
|
||||
@when("resolving a registry reference with missing namespace")
|
||||
def step_resolve_missing_ns(context: object) -> None:
|
||||
import asyncio
|
||||
|
||||
from cleveractors.registry.client import RegistryClient
|
||||
from cleveractors.registry.types import PackageReference, ReferenceType
|
||||
|
||||
ref_obj = PackageReference(
|
||||
original_reference="test.actor/test@v1.0",
|
||||
reference_type=ReferenceType.REGISTRY,
|
||||
namespace="test",
|
||||
name=None,
|
||||
) # type: ignore[arg-type]
|
||||
|
||||
mock_client = MagicMock(spec=RegistryClient)
|
||||
resolver = ReferenceResolver(client=mock_client, local_store=None)
|
||||
|
||||
async def _resolve() -> None:
|
||||
original_parse = resolver.parse
|
||||
|
||||
def _mock_parse(_ref_str: str) -> PackageReference:
|
||||
return ref_obj
|
||||
|
||||
resolver.parse = _mock_parse # type: ignore[method-assign]
|
||||
try:
|
||||
await resolver.resolve("any", package_type="actor")
|
||||
context._last_exc = None
|
||||
except InvalidPackageReferenceError as exc:
|
||||
context._last_exc = exc
|
||||
finally:
|
||||
resolver.parse = original_parse # type: ignore[method-assign]
|
||||
|
||||
asyncio.run(_resolve())
|
||||
|
||||
|
||||
@when("resolving a registry reference with the mock client")
|
||||
def step_resolve_with_mock_client(context: object) -> None:
|
||||
import asyncio
|
||||
|
||||
from cleveractors.registry.types import PackageReference, ReferenceType
|
||||
|
||||
ref_obj = PackageReference(
|
||||
original_reference="test.actor/test@v1.0",
|
||||
reference_type=ReferenceType.REGISTRY,
|
||||
namespace="test",
|
||||
name="test",
|
||||
version="v1.0",
|
||||
) # type: ignore[arg-type]
|
||||
|
||||
async def _resolve() -> None:
|
||||
try:
|
||||
await _resolve_with_parsed_ref(context, ref_obj)
|
||||
except InvalidPackageReferenceError as exc:
|
||||
context._last_exc = exc
|
||||
|
||||
asyncio.run(_resolve())
|
||||
|
||||
|
||||
def _bootstrap_parse(context: object) -> None:
|
||||
"""Ensure ReferenceResolver.parse is callable as a staticmethod."""
|
||||
from cleveractors.registry.resolver import ReferenceResolver as RR
|
||||
|
||||
if not isinstance(RR.__dict__.get("parse"), staticmethod):
|
||||
_func = RR.parse
|
||||
if callable(_func) and not isinstance(_func, staticmethod):
|
||||
RR.parse = staticmethod(_func)
|
||||
|
||||
|
||||
async def _resolve_with_parsed_ref(context: object, ref_obj: object) -> None:
|
||||
"""Call resolver with a pre-parsed reference, bypassing parse."""
|
||||
resolver = context._resolver
|
||||
resolver._pre_parsed_ref = ref_obj # type: ignore[attr-defined]
|
||||
original_parse = resolver.parse
|
||||
|
||||
def _mock_parse(_ref_str: str) -> object:
|
||||
return resolver._pre_parsed_ref # type: ignore[attr-defined]
|
||||
|
||||
resolver.parse = _mock_parse # type: ignore[method-assign]
|
||||
try:
|
||||
await resolver.resolve("any", package_type="actor")
|
||||
context._last_exc = None
|
||||
finally:
|
||||
resolver.parse = original_parse # type: ignore[method-assign]
|
||||
|
||||
@@ -222,3 +222,17 @@ def step_assert_no_actors_error(context: Any) -> None:
|
||||
assert "no actors" in str(context.test_error).lower(), (
|
||||
f"Expected error to mention 'no actors', got: {context.test_error}"
|
||||
)
|
||||
|
||||
|
||||
@when("I import from cleaveractors.runtime_types")
|
||||
def step_import_runtime_types(context):
|
||||
from cleveractors.runtime_types import ActorResult, NodeUsage
|
||||
|
||||
context.rt_actor_result = ActorResult
|
||||
context.rt_node_usage = NodeUsage
|
||||
|
||||
|
||||
@then("ActorResult and NodeUsage should be importable")
|
||||
def step_assert_rt_imports(context):
|
||||
assert context.rt_actor_result is not None
|
||||
assert context.rt_node_usage is not None
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Step definitions for runtime_dispatch.py coverage tests."""
|
||||
|
||||
from behave import given, then, when
|
||||
|
||||
|
||||
@given("the runtime dispatch test context is initialised")
|
||||
def step_rd_init(context):
|
||||
context.rd_result = None
|
||||
|
||||
|
||||
@when('I normalise a graph placeholder node_id "{node_id}"')
|
||||
def step_rd_normalise_placeholder(context, node_id):
|
||||
from cleveractors.runtime_dispatch import _normalize_node_id
|
||||
|
||||
context.rd_result = _normalize_node_id(node_id)
|
||||
|
||||
|
||||
@when('I normalise a regular node_id "{node_id}"')
|
||||
def step_rd_normalise_regular(context, node_id):
|
||||
from cleveractors.runtime_dispatch import _normalize_node_id
|
||||
|
||||
context.rd_result = _normalize_node_id(node_id)
|
||||
|
||||
|
||||
@then('the result should be the canonical "{expected}"')
|
||||
def step_rd_assert_canonical(context, expected):
|
||||
assert context.rd_result == expected, (
|
||||
f"Expected '{expected}', got '{context.rd_result}'"
|
||||
)
|
||||
|
||||
|
||||
@then("the result should be the original node_id")
|
||||
def step_rd_assert_original(context):
|
||||
assert context.rd_result == "agent.node_1", (
|
||||
f"Expected 'agent.node_1', got '{context.rd_result}'"
|
||||
)
|
||||
@@ -1175,3 +1175,26 @@ def step_verify_edge_cases(context):
|
||||
# Template without parameters section should work
|
||||
no_params = context.edge_cases["no_params_def"]
|
||||
assert no_params.parameters == {}
|
||||
|
||||
|
||||
@when("I instantiate a GenericTemplate whose applied vars produce a non-dict result")
|
||||
def step_gt_nondict_result(context):
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from cleveractors.templates.base import InstantiationContext, TemplateType
|
||||
from cleveractors.templates.generic_template import GenericTemplate
|
||||
|
||||
gt = GenericTemplate(
|
||||
name="test", template_type=TemplateType.ACTOR, definition={"k": "{{ v }}"}
|
||||
)
|
||||
gt._apply_template_vars = lambda d, p: ["non", "dict", "result"]
|
||||
ctx = InstantiationContext()
|
||||
context.gt_result = gt.instantiate(
|
||||
params={"v": "x"}, registry=MagicMock(), context=ctx
|
||||
)
|
||||
|
||||
|
||||
@then("the result should be a dict with definition key wrapping the non-dict value")
|
||||
def step_gt_assert_nondict_wrapped(context):
|
||||
assert isinstance(context.gt_result, dict)
|
||||
assert "definition" in context.gt_result
|
||||
|
||||
@@ -152,7 +152,7 @@ def step_shell_cmd(context):
|
||||
m = MagicMock()
|
||||
m.returncode = 0
|
||||
m.communicate = AsyncMock(return_value=(b"custom output", b""))
|
||||
with patch("asyncio.create_subprocess_exec", return_value=m):
|
||||
with patch("asyncio.create_subprocess_shell", return_value=m):
|
||||
context.result = _run(a._execute_shell_command("custom_cmd", {}))
|
||||
|
||||
|
||||
@@ -442,6 +442,28 @@ def step_fw_raises(context, text):
|
||||
assert text in context.error
|
||||
|
||||
|
||||
# ── file_read directory listing OSError ───────────────────────────────
|
||||
|
||||
|
||||
@when("I call file read tool on a directory with unreadable entry")
|
||||
def step_fr_dir_oserror(context):
|
||||
a = ToolAgent("tc", {"tools": ["file_read"]})
|
||||
(context.td / "valid_file.txt").write_text("hello")
|
||||
with patch("os.path.getsize", side_effect=OSError("Permission denied")):
|
||||
context.result = _run(
|
||||
a._file_read_tool({"file": str(context.td)}, context={"_unsafe_mode": True})
|
||||
)
|
||||
|
||||
|
||||
# ── process_message JSONDecodeError ───────────────────────────────────
|
||||
|
||||
|
||||
@when("I process a message with malformed JSON")
|
||||
def step_pm_json_error(context):
|
||||
a = ToolAgent("tc", {"tools": ["echo"]})
|
||||
context.error = _run(_catch(a.process_message, '{"tool": "echo", "text": bad}'))
|
||||
|
||||
|
||||
# ── shell detailed ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -451,7 +473,7 @@ def step_sh_cmd_args(context, cmd):
|
||||
m = MagicMock()
|
||||
m.returncode = 0
|
||||
m.communicate = AsyncMock(return_value=(b"hello", b""))
|
||||
with patch("asyncio.create_subprocess_exec", return_value=m):
|
||||
with patch("asyncio.create_subprocess_shell", return_value=m):
|
||||
context.result = _run(a._execute_shell_command(cmd, {"args": ["hello"]}))
|
||||
|
||||
|
||||
@@ -461,7 +483,7 @@ def step_sh_cmd(context, cmd):
|
||||
m = MagicMock()
|
||||
m.returncode = 1
|
||||
m.communicate = AsyncMock(return_value=(b"", b"err"))
|
||||
with patch("asyncio.create_subprocess_exec", return_value=m):
|
||||
with patch("asyncio.create_subprocess_shell", return_value=m):
|
||||
context.error = _run(_catch(a._execute_shell_command, cmd, {}))
|
||||
|
||||
|
||||
@@ -469,11 +491,11 @@ def step_sh_cmd(context, cmd):
|
||||
def step_sh_timeout(context):
|
||||
a = ToolAgent("tc", {"tools": ["sleep"], "allow_shell": True})
|
||||
|
||||
async def _to(*a, **kw):
|
||||
raise asyncio.TimeoutError()
|
||||
|
||||
with patch("asyncio.create_subprocess_exec", side_effect=_to):
|
||||
context.error = _run(_catch(a._execute_shell_command, "sleep", {}))
|
||||
m = MagicMock()
|
||||
m.communicate = AsyncMock(return_value=(b"", b""))
|
||||
with patch("asyncio.create_subprocess_shell", return_value=m):
|
||||
with patch("asyncio.wait_for", side_effect=asyncio.TimeoutError()):
|
||||
context.error = _run(_catch(a._execute_shell_command, "sleep", {}))
|
||||
|
||||
|
||||
@when("I execute a dangerous shell command {cmd}")
|
||||
@@ -878,3 +900,165 @@ def step_assert_empty_prefix(context):
|
||||
def step_caps_fw_only(context):
|
||||
a = ToolAgent("tc", {"tools": [{"name": "file_write"}]})
|
||||
context.caps = a.get_capabilities()
|
||||
|
||||
|
||||
# ── python_exec_tool ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
@when("I call python_exec tool without exec_python enabled")
|
||||
def step_pe_disabled(context):
|
||||
a = ToolAgent("tc", {"tools": ["python_exec"], "exec_python": True})
|
||||
a.config["exec_python"] = False
|
||||
context.error = _run(_catch(a._python_exec_tool, {"code": "x=1"}, None))
|
||||
|
||||
|
||||
@when("I call python_exec tool without code argument")
|
||||
def step_pe_no_code(context):
|
||||
a = ToolAgent("tc", {"tools": ["python_exec"], "exec_python": True})
|
||||
context.error = _run(_catch(a._python_exec_tool, {}, None))
|
||||
|
||||
|
||||
# ── shell_tool ────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@when("I call shell tool without allow_shell")
|
||||
def step_sh_disabled(context):
|
||||
a = ToolAgent("tc", {"tools": ["shell"], "allow_shell": True})
|
||||
a.allow_shell = False
|
||||
context.error = _run(_catch(a._shell_tool, {"command": "echo"}, None))
|
||||
|
||||
|
||||
@when("I call shell tool without command argument")
|
||||
def step_sh_no_cmd(context):
|
||||
a = ToolAgent("tc", {"tools": ["custom_cmd"], "allow_shell": True})
|
||||
context.error = _run(_catch(a._shell_tool, {}, None))
|
||||
|
||||
|
||||
# ── python execution details ──────────────────────────────────────────
|
||||
|
||||
|
||||
@when("I execute python code that prints to stdout")
|
||||
def step_pe_stdout(context):
|
||||
a = ToolAgent("tc", {"tools": ["python_exec"], "exec_python": True})
|
||||
context.result = _run(
|
||||
a._execute_python_code(
|
||||
"print('hello stdout')",
|
||||
{},
|
||||
None,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@then("the result should contain printed output")
|
||||
def step_pe_stdout_result(context):
|
||||
assert "hello stdout" in context.result
|
||||
|
||||
|
||||
@when("I execute python code with undefined variable")
|
||||
def step_pe_name_error(context):
|
||||
a = ToolAgent("tc", {"tools": ["python_exec"], "exec_python": True})
|
||||
context.error = _run(
|
||||
_catch(
|
||||
a._execute_python_code,
|
||||
"x = undefined_var + 1",
|
||||
{},
|
||||
None,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
# ── file_read shell command detection ─────────────────────────────────
|
||||
|
||||
|
||||
@when("I call file read tool with a shell command path")
|
||||
def step_fr_shell_path(context):
|
||||
a = ToolAgent("tc", {"tools": ["file_read"]})
|
||||
context.error = _run(_catch(a._file_read_tool, {"file": "ls -la"}, None))
|
||||
|
||||
|
||||
# ── file_read max_chars validation ────────────────────────────────────
|
||||
|
||||
|
||||
@when("I call file read tool with invalid max_chars")
|
||||
def step_fr_invalid_max(context):
|
||||
a = ToolAgent("tc", {"tools": ["file_read"]})
|
||||
fp = context.td / "fr_max.txt"
|
||||
fp.write_text("sample text")
|
||||
context.error = _run(
|
||||
_catch(
|
||||
a._file_read_tool,
|
||||
{"file": str(fp), "max_chars": "not_a_number"},
|
||||
{"_unsafe_mode": True},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
# ── file_read directory listing ───────────────────────────────────────
|
||||
|
||||
|
||||
@when("I call file read tool on an existing directory")
|
||||
def step_fr_directory(context):
|
||||
a = ToolAgent("tc", {"tools": ["file_read"]})
|
||||
(context.td / "somefile.txt").write_text("hello")
|
||||
context.result = _run(
|
||||
a._file_read_tool({"file": str(context.td)}, context={"_unsafe_mode": True})
|
||||
)
|
||||
|
||||
|
||||
@then("the result should contain directory listing header")
|
||||
def step_fr_dir_header(context):
|
||||
assert "FILE_READ_SUCCESS" in context.result
|
||||
assert "Directory:" in context.result
|
||||
assert "Entries:" in context.result
|
||||
|
||||
|
||||
# ── file_read truncation ──────────────────────────────────────────────
|
||||
|
||||
|
||||
@when("I call file read tool with small max_chars limit")
|
||||
def step_fr_truncate(context):
|
||||
a = ToolAgent("tc", {"tools": ["file_read"]})
|
||||
fp = context.td / "fr_trunc.txt"
|
||||
fp.write_text("a" * 500)
|
||||
context.result = _run(
|
||||
a._file_read_tool(
|
||||
{"file": str(fp), "max_chars": 10},
|
||||
context={"_unsafe_mode": True},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@then("the result should be truncated")
|
||||
def step_fr_truncated(context):
|
||||
assert "TRUNCATED" in context.result
|
||||
|
||||
|
||||
# ── file_read general exception ───────────────────────────────────────
|
||||
|
||||
|
||||
@when("I call file read tool causing a read error")
|
||||
def step_fr_read_error(context):
|
||||
a = ToolAgent("tc", {"tools": ["file_read"]})
|
||||
with patch("builtins.open", side_effect=OSError("Permission denied")):
|
||||
context.error = _run(
|
||||
_catch(
|
||||
a._file_read_tool,
|
||||
{"file": str(context.td / "no_such.txt")},
|
||||
context={"_unsafe_mode": True},
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
# ── file_write absolute path in safe mode ─────────────────────────────
|
||||
|
||||
|
||||
@when("I call file write tool with absolute path in safe mode")
|
||||
def step_fw_abs_safe(context):
|
||||
a = ToolAgent("tc", {"tools": ["file_write"]})
|
||||
context.error = _run(
|
||||
_catch(
|
||||
a._validate_file_path_safety,
|
||||
os.path.join(os.getcwd(), "test_abs_file.txt"),
|
||||
False,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -641,3 +641,65 @@ def step_vac_llm_string_config_no_provider(context: Context) -> None:
|
||||
@given("a config dict with type tool name and string config block (vac)")
|
||||
def step_vac_tool_string_config(context: Context) -> None:
|
||||
_set_config_dict(context, {"type": "tool", "name": "test", "config": "not_a_dict"})
|
||||
|
||||
|
||||
@when("I call _compute_subgraph_depth twice for the same subgraph route")
|
||||
def step_vac_subgraph_depth_cache(context: Context) -> None:
|
||||
from cleveractors.validation._limits import _compute_subgraph_depth
|
||||
|
||||
routes = {
|
||||
"parent": {"type": "graph", "nodes": {"n": {"type": "llm"}}, "edges": []},
|
||||
"child": {
|
||||
"type": "graph",
|
||||
"nodes": {"n2": {"type": "llm"}},
|
||||
"subgraph": ["parent"],
|
||||
"edges": [],
|
||||
},
|
||||
}
|
||||
cache = {}
|
||||
context._depth1 = _compute_subgraph_depth("child", routes, cache)
|
||||
context._depth2 = _compute_subgraph_depth("child", routes, cache)
|
||||
context._cached = "child" in cache
|
||||
|
||||
|
||||
@then("the second call should return the cached depth value")
|
||||
def step_vac_assert_depth_cached(context: Context) -> None:
|
||||
assert context._cached, "Expected depth to be cached"
|
||||
assert context._depth1 == context._depth2, (
|
||||
f"Depth values differ: {context._depth1} vs {context._depth2}"
|
||||
)
|
||||
|
||||
|
||||
@when("I call _validate_top_level_keys on a dict missing the agents key")
|
||||
def step_vac_top_level_missing_agents(context: Context) -> None:
|
||||
from cleveractors.core.exceptions import ConfigurationError
|
||||
from cleveractors.validation import _validate_top_level_keys
|
||||
|
||||
try:
|
||||
_validate_top_level_keys({"routes": {}})
|
||||
except ConfigurationError as e:
|
||||
context._config_error = str(e)
|
||||
|
||||
|
||||
@then("a ConfigurationError about missing agents key should be raised")
|
||||
def step_vac_assert_missing_agents(context: Context) -> None:
|
||||
assert context._config_error is not None, "Expected ConfigurationError"
|
||||
assert "agents" in context._config_error, (
|
||||
f"Expected 'agents' in error, got: {context._config_error}"
|
||||
)
|
||||
|
||||
|
||||
@when("I call _count_total_nodes with a non-dict nodes value in a graph route")
|
||||
def step_vac_count_total_nondict_nodes(context: Context) -> None:
|
||||
from cleveractors.core.exceptions import ConfigurationError
|
||||
from cleveractors.validation._limits import _count_total_nodes
|
||||
|
||||
try:
|
||||
_count_total_nodes({"test": {"type": "graph", "nodes": "not_a_dict"}})
|
||||
except ConfigurationError as e:
|
||||
context._config_error = str(e)
|
||||
|
||||
|
||||
@then("a ConfigurationError should be raised about nodes being invalid")
|
||||
def step_vac_assert_nondict_nodes(context: Context) -> None:
|
||||
assert context._config_error is not None, "Expected ConfigurationError"
|
||||
|
||||
@@ -2130,3 +2130,16 @@ def step_assert_unknown_tag_skipped(context):
|
||||
# So unknown tags are just passed through without being processed
|
||||
assert context.result is not None
|
||||
assert "protected" in context.result
|
||||
|
||||
|
||||
@when("I load YAML content with Jinja2 templates without providing context")
|
||||
def step_yjl_deferred_no_context(context):
|
||||
from cleveractors.templates.yaml_jinja_loader import YAMLJinjaLoader
|
||||
|
||||
loader = YAMLJinjaLoader()
|
||||
context.result = loader.load_string("key: {{ value }}")
|
||||
|
||||
|
||||
@then("the result should contain deferred template markers")
|
||||
def step_yjl_assert_deferred_markers(context):
|
||||
assert context.result is not None, "Expected deferred template result"
|
||||
|
||||
@@ -164,3 +164,7 @@ Feature: Template Parameter Validation and Variable Rendering
|
||||
When I test error conditions
|
||||
Then appropriate errors should be raised
|
||||
And edge cases should be handled correctly
|
||||
|
||||
Scenario: GenericTemplate wraps non-dict result in definition key
|
||||
When I instantiate a GenericTemplate whose applied vars produce a non-dict result
|
||||
Then the result should be a dict with definition key wrapping the non-dict value
|
||||
|
||||
@@ -449,7 +449,7 @@ Feature: ToolAgent Functionality
|
||||
"""
|
||||
{"tool": "file_read", "args": {"file": "nonexistent.txt"}}
|
||||
"""
|
||||
Then tool execution should fail with message containing "File read failed"
|
||||
Then tool execution should fail with message containing "File not found"
|
||||
|
||||
Scenario: Tool agent file_write tool requires file path and content
|
||||
Given I am running in unsafe mode
|
||||
|
||||
@@ -126,7 +126,7 @@ Feature: Tool Agent Coverage Gaps
|
||||
Scenario: File read tool raises File read failed for missing file
|
||||
Given a tool agent coverage test environment
|
||||
When I call file read tool with a missing file path
|
||||
Then the file read raises File read failed
|
||||
Then the file read raises File not found
|
||||
|
||||
Scenario: File read tool requires a file path
|
||||
Given a tool agent coverage test environment
|
||||
@@ -321,4 +321,74 @@ Feature: Tool Agent Coverage Gaps
|
||||
Scenario: _execute_python_code writing_stage else None
|
||||
Given a tool agent coverage test environment
|
||||
When I execute python code with non-dict context
|
||||
Then the code should execute successfully
|
||||
Then the code should execute successfully
|
||||
|
||||
Scenario: python_exec tool requires exec_python enabled
|
||||
Given a tool agent coverage test environment
|
||||
When I call python_exec tool without exec_python enabled
|
||||
Then a shell ExecutionError mentioning Python execution is disabled is raised
|
||||
|
||||
Scenario: python_exec tool requires code argument
|
||||
Given a tool agent coverage test environment
|
||||
When I call python_exec tool without code argument
|
||||
Then a shell ExecutionError mentioning requires a 'code' argument is raised
|
||||
|
||||
Scenario: shell tool requires allow_shell enabled
|
||||
Given a tool agent coverage test environment
|
||||
When I call shell tool without allow_shell
|
||||
Then a shell ExecutionError mentioning disabled is raised
|
||||
|
||||
Scenario: shell tool requires command argument
|
||||
Given a tool agent coverage test environment
|
||||
When I call shell tool without command argument
|
||||
Then a shell ExecutionError mentioning requires a 'command' argument is raised
|
||||
|
||||
Scenario: _execute_python_code captures stdout fallback
|
||||
Given a tool agent coverage test environment
|
||||
When I execute python code that prints to stdout
|
||||
Then the result should contain printed output
|
||||
|
||||
Scenario: _execute_python_code NameError in sandbox
|
||||
Given a tool agent coverage test environment
|
||||
When I execute python code with undefined variable
|
||||
Then a shell ExecutionError mentioning not available in the Python sandbox is raised
|
||||
|
||||
Scenario: File read tool detects shell command in file path
|
||||
Given a tool agent coverage test environment
|
||||
When I call file read tool with a shell command path
|
||||
Then the file read raises looks like a shell command
|
||||
|
||||
Scenario: File read tool validates max_chars parameter
|
||||
Given a tool agent coverage test environment
|
||||
When I call file read tool with invalid max_chars
|
||||
Then the file read raises must be an integer
|
||||
|
||||
Scenario: File read tool lists directory contents
|
||||
Given a tool agent coverage test environment
|
||||
When I call file read tool on an existing directory
|
||||
Then the result should contain directory listing header
|
||||
|
||||
Scenario: File read tool handles max_chars truncation
|
||||
Given a tool agent coverage test environment
|
||||
When I call file read tool with small max_chars limit
|
||||
Then the result should be truncated
|
||||
|
||||
Scenario: File read tool raises File read failed for read error
|
||||
Given a tool agent coverage test environment
|
||||
When I call file read tool causing a read error
|
||||
Then the file read raises File read failed
|
||||
|
||||
Scenario: File write tool blocks absolute path in safe mode
|
||||
Given a tool agent coverage test environment
|
||||
When I call file write tool with absolute path in safe mode
|
||||
Then the file write raises Unsafe file path blocked
|
||||
|
||||
Scenario: File read tool directory listing handles OSError for entry size
|
||||
Given a tool agent coverage test environment
|
||||
When I call file read tool on a directory with unreadable entry
|
||||
Then the result should contain directory listing header
|
||||
|
||||
Scenario: Process message handles JSONDecodeError
|
||||
Given a tool agent coverage test environment
|
||||
When I process a message with malformed JSON
|
||||
Then an ExecutionError mentioning Invalid JSON is raised
|
||||
@@ -203,3 +203,15 @@ Feature: Validation Actor Runtime Config Validation
|
||||
Given a config dict with type tool name and string config block (vac)
|
||||
When _validate_tool_actor is called (vac)
|
||||
Then a ConfigurationError should be raised for missing tools (vac)
|
||||
|
||||
Scenario: _validate_top_level_keys raises ConfigurationError when agents key is missing
|
||||
When I call _validate_top_level_keys on a dict missing the agents key
|
||||
Then a ConfigurationError about missing agents key should be raised
|
||||
|
||||
Scenario: _count_total_nodes raises ConfigurationError for non-dict nodes value
|
||||
When I call _count_total_nodes with a non-dict nodes value in a graph route
|
||||
Then a ConfigurationError should be raised about nodes being invalid
|
||||
|
||||
Scenario: _compute_subgraph_depth returns cached value on cache hit
|
||||
When I call _compute_subgraph_depth twice for the same subgraph route
|
||||
Then the second call should return the cached depth value
|
||||
|
||||
@@ -263,3 +263,9 @@ Feature: YAMLJinjaLoader and TemplateAwareYAMLParser Template Processing Pipelin
|
||||
And I have a clean test environment for yaml jinja processing
|
||||
When I protect template sections with an unknown tag
|
||||
Then the unknown tag should be replaced with SKIP
|
||||
|
||||
Scenario: YAMLJinjaLoader defers rendering when no context is provided
|
||||
Given the yaml jinja loader system is initialized
|
||||
And I have a clean test environment for yaml jinja processing
|
||||
When I load YAML content with Jinja2 templates without providing context
|
||||
Then the result should contain deferred template markers
|
||||
|
||||
+8
-14
@@ -340,9 +340,7 @@ class EmailGraphLib:
|
||||
)
|
||||
current = current[key]
|
||||
|
||||
def merged_config_key_count(
|
||||
self, merged: Any, section: str, minimum: int
|
||||
) -> None:
|
||||
def merged_config_key_count(self, merged: Any, section: str, minimum: int) -> None:
|
||||
"""Assert a config section has at least minimum keys."""
|
||||
assert section in merged, f"Section '{section}' not in merged config"
|
||||
actual = len(merged[section])
|
||||
@@ -390,9 +388,7 @@ class EmailGraphLib:
|
||||
"""Assert the assembled component graph has exactly expected actors."""
|
||||
config = self.load_all_email_components()
|
||||
actual = len(config.get("actors", {}))
|
||||
assert actual == expected, (
|
||||
f"Expected {expected} actors, got {actual}"
|
||||
)
|
||||
assert actual == expected, f"Expected {expected} actors, got {actual}"
|
||||
|
||||
# -- namespaced component graph with reference resolver -------------------
|
||||
|
||||
@@ -421,7 +417,7 @@ class EmailGraphLib:
|
||||
for _node_id, node_def in nodes.items():
|
||||
agent_ref = node_def.get("agent_ref", "")
|
||||
if agent_ref.startswith("local:"):
|
||||
ref_name = agent_ref[len("local:"):]
|
||||
ref_name = agent_ref[len("local:") :]
|
||||
pkg = store.resolve_package(ref_name)
|
||||
content = pkg.content
|
||||
actor_name = content.get("name", ref_name)
|
||||
@@ -431,11 +427,11 @@ class EmailGraphLib:
|
||||
}
|
||||
|
||||
graph_config["actors"] = actors
|
||||
for node_def in graph_config.get("routes", {}).get("main", {}).get(
|
||||
"nodes", {}
|
||||
).values():
|
||||
for node_def in (
|
||||
graph_config.get("routes", {}).get("main", {}).get("nodes", {}).values()
|
||||
):
|
||||
if "agent_ref" in node_def and node_def["agent_ref"].startswith("local:"):
|
||||
ref_name = node_def["agent_ref"][len("local:"):]
|
||||
ref_name = node_def["agent_ref"][len("local:") :]
|
||||
pkg = store.resolve_package(ref_name)
|
||||
node_def["agent"] = pkg.content.get("name", ref_name)
|
||||
del node_def["agent_ref"]
|
||||
@@ -501,6 +497,4 @@ class EmailGraphLib:
|
||||
renderer = TemplateRenderer()
|
||||
factory = AgentFactory(config=config, template_renderer=renderer)
|
||||
meta = factory.get_agent_metadata(agent_name)
|
||||
assert key in meta, (
|
||||
f"Key '{key}' not in metadata. Keys: {list(meta.keys())}"
|
||||
)
|
||||
assert key in meta, f"Key '{key}' not in metadata. Keys: {list(meta.keys())}"
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
"""Library for LLM tool-calling integration tests.
|
||||
|
||||
Provides keywords that create an Executor with a mock chat model that
|
||||
returns tool_calls, then execute messages and verify that tools were
|
||||
invoked correctly — without requiring a real LLM API key.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from langchain_core.messages import AIMessage
|
||||
|
||||
from cleveractors.result import ActorResult
|
||||
from cleveractors.runtime import Executor, create_executor
|
||||
|
||||
|
||||
class ToolCallingTestLib:
|
||||
"""Keywords for integration-style tests of LLM tool-calling."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._executor: Executor | None = None
|
||||
self._last_result: ActorResult | None = None
|
||||
self._mock_ainvoke_calls: list[dict[str, Any]] = []
|
||||
self._patches: list[Any] = []
|
||||
|
||||
def _teardown_patches(self) -> None:
|
||||
for p in self._patches:
|
||||
p.stop()
|
||||
self._patches.clear()
|
||||
|
||||
def create_executor_with_tool_calling_agent(
|
||||
self, tool_name: str = "echo", message: str = ""
|
||||
) -> None:
|
||||
"""Create an Executor whose LLM agent is configured with a tool.
|
||||
|
||||
The chat model is mocked so that ``ainvoke`` returns an AIMessage
|
||||
with tool_calls. The model first returns a tool-call message,
|
||||
then a final plain-text answer.
|
||||
"""
|
||||
self._teardown_patches()
|
||||
self._mock_ainvoke_calls = []
|
||||
|
||||
final_response_text = "Final answer after tool use"
|
||||
|
||||
def _build_mock_model(*args, **kwargs):
|
||||
mock_model = MagicMock()
|
||||
mock_model.temperature = 0.7
|
||||
|
||||
async def _mock_ainvoke(messages, **invoke_kwargs):
|
||||
self._mock_ainvoke_calls.append({"kwargs": invoke_kwargs})
|
||||
call_count = len(self._mock_ainvoke_calls)
|
||||
if call_count == 1 and invoke_kwargs.get("tools"):
|
||||
tool_content = message or "hello from tool"
|
||||
return AIMessage(
|
||||
content="",
|
||||
usage_metadata={
|
||||
"input_tokens": 50,
|
||||
"output_tokens": 10,
|
||||
"total_tokens": 60,
|
||||
},
|
||||
tool_calls=[
|
||||
{
|
||||
"id": "call_001",
|
||||
"name": tool_name,
|
||||
"args": {"message": tool_content},
|
||||
},
|
||||
],
|
||||
)
|
||||
return AIMessage(
|
||||
content=final_response_text,
|
||||
usage_metadata={
|
||||
"input_tokens": 30,
|
||||
"output_tokens": 15,
|
||||
"total_tokens": 45,
|
||||
},
|
||||
)
|
||||
|
||||
async def _mock_astream(messages, **invoke_kwargs):
|
||||
self._mock_ainvoke_calls.append({"kwargs": invoke_kwargs})
|
||||
call_count = len(self._mock_ainvoke_calls)
|
||||
if call_count == 1 and invoke_kwargs.get("tools"):
|
||||
tool_content = message or "hello from stream"
|
||||
yield AIMessage(
|
||||
content="tool_call_response",
|
||||
usage_metadata={
|
||||
"input_tokens": 40,
|
||||
"output_tokens": 10,
|
||||
"total_tokens": 50,
|
||||
},
|
||||
tool_calls=[
|
||||
{
|
||||
"id": "call_001",
|
||||
"name": tool_name,
|
||||
"args": {"message": tool_content},
|
||||
},
|
||||
],
|
||||
)
|
||||
else:
|
||||
yield AIMessage(
|
||||
content=final_response_text,
|
||||
usage_metadata={
|
||||
"input_tokens": 25,
|
||||
"output_tokens": 12,
|
||||
"total_tokens": 37,
|
||||
},
|
||||
)
|
||||
|
||||
mock_model.ainvoke = _mock_ainvoke
|
||||
mock_model.astream = _mock_astream
|
||||
return mock_model
|
||||
|
||||
patcher = patch(
|
||||
"cleveractors.agents.llm.build_chat_model",
|
||||
side_effect=_build_mock_model,
|
||||
)
|
||||
patcher.start()
|
||||
self._patches.append(patcher)
|
||||
|
||||
config = {
|
||||
"type": "llm",
|
||||
"name": "tool_test_agent",
|
||||
"provider": "openai",
|
||||
"model": "gpt-3.5-turbo",
|
||||
"config": {
|
||||
"tools": [{"name": tool_name}],
|
||||
},
|
||||
}
|
||||
|
||||
self._executor = create_executor(
|
||||
config_dict=config,
|
||||
credentials={"openai": {"api_key": "mock-key"}},
|
||||
limits={},
|
||||
pricing={},
|
||||
)
|
||||
|
||||
async def _execute_async(self, message: str) -> ActorResult:
|
||||
return await self._executor.execute(message)
|
||||
|
||||
def execute_with_message(self, message: str) -> None:
|
||||
"""Execute the executor with *message* and store the result."""
|
||||
try:
|
||||
self._last_result = asyncio.run(self._execute_async(message))
|
||||
finally:
|
||||
self._teardown_patches()
|
||||
|
||||
def execute_stream_with_message(self, message: str) -> None:
|
||||
"""Execute via execute_stream and store the last result."""
|
||||
|
||||
async def _stream():
|
||||
chunks = []
|
||||
async for token in self._executor.execute_stream(message):
|
||||
chunks.append(token)
|
||||
return "".join(chunks)
|
||||
|
||||
try:
|
||||
asyncio.run(_stream())
|
||||
self._last_result = self._executor.last_result
|
||||
finally:
|
||||
self._teardown_patches()
|
||||
|
||||
def result_is_valid_actor_result(self) -> None:
|
||||
assert isinstance(self._last_result, ActorResult), (
|
||||
f"Expected ActorResult, got {type(self._last_result)}"
|
||||
)
|
||||
|
||||
def result_response_contains(self, text: str) -> None:
|
||||
assert self._last_result is not None
|
||||
assert text in self._last_result.response, (
|
||||
f"Expected '{text}' in response, got: {self._last_result.response[:200]}"
|
||||
)
|
||||
|
||||
def result_has_nodes(self) -> None:
|
||||
assert self._last_result is not None
|
||||
assert len(self._last_result.nodes) > 0, "Expected at least one node in result"
|
||||
|
||||
def result_prompt_tokens_greater_than_zero(self) -> None:
|
||||
assert self._last_result is not None
|
||||
assert self._last_result.prompt_tokens > 0, (
|
||||
f"Expected prompt_tokens > 0, got {self._last_result.prompt_tokens}"
|
||||
)
|
||||
|
||||
def chat_model_was_invoked_with_tools(self) -> None:
|
||||
for call in self._mock_ainvoke_calls:
|
||||
if call.get("kwargs", {}).get("tools"):
|
||||
return
|
||||
raise AssertionError("chat_model.ainvoke was never called with tools keyword")
|
||||
|
||||
def multiple_tool_rounds_occurred(self) -> None:
|
||||
assert len(self._mock_ainvoke_calls) >= 2, (
|
||||
f"Expected >= 2 ainvoke calls, got {len(self._mock_ainvoke_calls)}"
|
||||
)
|
||||
|
||||
def tool_call_result_is_in_response(self) -> None:
|
||||
assert self._last_result is not None
|
||||
assert len(self._last_result.response.strip()) > 0, (
|
||||
"Expected non-empty response after tool calls"
|
||||
)
|
||||
@@ -0,0 +1,52 @@
|
||||
*** Settings ***
|
||||
Documentation LLM tool-calling integration tests.
|
||||
... Exercises the full pipeline (Executor → LLMAgent → ToolAgent)
|
||||
... with a mock chat model that returns tool_calls, so no real
|
||||
... LLM API key is required.
|
||||
Library ToolCallingTestLib.py
|
||||
|
||||
*** Test Cases ***
|
||||
LLM Agent Receives Tools And Executes Tool Call
|
||||
[Documentation] Full lifecycle: config with echo tool → mock returns
|
||||
... tool_calls → ToolAgent executes echo → final response produced.
|
||||
Create Executor With Tool Calling Agent tool_name=echo message=hello_from_tool
|
||||
Execute With Message Tell me something using the echo tool
|
||||
Result Is Valid Actor Result
|
||||
Result Has Nodes
|
||||
Chat Model Was Invoked With Tools
|
||||
Multiple Tool Rounds Occurred
|
||||
Result Response Contains Final answer after tool use
|
||||
|
||||
LLM Agent Tool Calling Works With MATH Tool
|
||||
[Documentation] MATH tool recognises the expression and computes result.
|
||||
Create Executor With Tool Calling Agent tool_name=math message=2+2
|
||||
Execute With Message What is 2+2? Use the math tool.
|
||||
Result Is Valid Actor Result
|
||||
Chat Model Was Invoked With Tools
|
||||
Multiple Tool Rounds Occurred
|
||||
Result Response Contains Final answer after tool use
|
||||
|
||||
LLM Agent With FILE_READ Tool Receives Tool Calls
|
||||
[Documentation] file_read tool is normalised and passed to the model.
|
||||
Create Executor With Tool Calling Agent tool_name=file_read message=some_file.txt
|
||||
Execute With Message Read the file some_file.txt for me
|
||||
Result Is Valid Actor Result
|
||||
Chat Model Was Invoked With Tools
|
||||
|
||||
Tool Calling Produces Non-Empty Final Response
|
||||
[Documentation] After tool execution, the final response is populated.
|
||||
Create Executor With Tool Calling Agent tool_name=echo message=tool_data
|
||||
Execute With Message Use the echo tool and report back
|
||||
Result Is Valid Actor Result
|
||||
Tool Call Result Is In Response
|
||||
Result Prompt Tokens Greater Than Zero
|
||||
|
||||
Streaming Path Works With Tool-Configured Agent
|
||||
[Documentation] execute_stream produces a valid result with a tool-configured
|
||||
... agent. Tool calling is not exercised in the streaming path — tools are
|
||||
... not passed to the model during stream_message. This test verifies the
|
||||
... streaming path itself works and does not crash with a tool-configured agent.
|
||||
Create Executor With Tool Calling Agent tool_name=echo message=from_stream
|
||||
Execute Stream With Message Tell me about yourself
|
||||
Result Is Valid Actor Result
|
||||
Result Has Nodes
|
||||
+261
-20
@@ -24,7 +24,9 @@ Extended Provider Routing (ADR-2028):
|
||||
from __future__ import annotations
|
||||
|
||||
import contextvars
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
from collections.abc import AsyncGenerator
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
@@ -35,6 +37,7 @@ if TYPE_CHECKING:
|
||||
from cleveractors.agents.base import AgentWithMemory
|
||||
from cleveractors.agents.llm_client import build_chat_model
|
||||
from cleveractors.agents.llm_imports import populate_langchain_globals
|
||||
from cleveractors.agents.llm_tools import normalize_tool_entry as _normalize_tool_entry
|
||||
from cleveractors.core.exceptions import (
|
||||
AgentCreationError,
|
||||
ConfigurationError,
|
||||
@@ -102,6 +105,7 @@ LangChainException: Any = None
|
||||
AIMessage: Any = None
|
||||
HumanMessage: Any = None
|
||||
SystemMessage: Any = None
|
||||
ToolMessage: Any = None
|
||||
ChatGoogleGenerativeAI: Any = None
|
||||
ChatOpenAI: Any = None
|
||||
|
||||
@@ -195,6 +199,15 @@ class LLMAgent(AgentWithMemory):
|
||||
self.max_tokens: int = config.get("max_tokens", DEFAULT_MAX_TOKENS)
|
||||
self.system_message: str = config.get("system_prompt", DEFAULT_SYSTEM_MESSAGE)
|
||||
|
||||
# Tool definitions extracted from agent config. Stored as a list of
|
||||
# dicts in OpenAI-compatible function-calling format (``{"type":
|
||||
# "function", "function": {...}}``). Populated during __init__ only
|
||||
# when ``config["tools"]`` is present — the list is built once so it
|
||||
# can be reused across multiple calls without re-computation.
|
||||
self._lc_tools: list[dict[str, Any]] | None = None
|
||||
if config.get("tools"):
|
||||
self._lc_tools = [_normalize_tool_entry(t) for t in config["tools"]]
|
||||
|
||||
# Per-request credential dict (ADR-2026), validated and shallow-copied.
|
||||
# Contains {"api_key": "...", "base_url": "..."} for a single provider.
|
||||
# Stored separately; config is never modified.
|
||||
@@ -467,10 +480,229 @@ class LLMAgent(AgentWithMemory):
|
||||
# Add current user message
|
||||
messages.append(HumanMessage(content=processed_message))
|
||||
|
||||
# Call LangChain model
|
||||
response = await self.chat_model.ainvoke(messages)
|
||||
# -- Tool calling support (issue #59) -----------------------------------
|
||||
# When the agent config declares tools, convert them to a LangChain-
|
||||
# compatible format and pass them to the LLM so it can produce
|
||||
# structured tool calls instead of hallucinating tool names in plain
|
||||
# text.
|
||||
#
|
||||
# Multi-turn tool-use loop: when the model returns tool_calls,
|
||||
# execute each tool, append the AIMessage (with tool_calls) plus
|
||||
# corresponding ToolMessages to the conversation, then re-invoke the
|
||||
# model. The loop repeats until the model returns a final answer
|
||||
# (no more tool_calls) or the maximum number of tool rounds is
|
||||
# reached.
|
||||
invoke_kwargs: dict[str, Any] = {}
|
||||
_raw_max_rounds: object = self.config.get(
|
||||
"tool_max_rounds"
|
||||
) or os.environ.get("TOOL_MAX_ROUNDS", "20")
|
||||
try:
|
||||
_TOOL_MAX_ROUNDS = max(1, int(str(_raw_max_rounds)))
|
||||
except (TypeError, ValueError) as _mre:
|
||||
raise ConfigurationError(
|
||||
f"tool_max_rounds must be an integer, got {_raw_max_rounds!r}"
|
||||
) from _mre
|
||||
_has_tools = self._lc_tools is not None and LANGCHAIN_AVAILABLE
|
||||
if _has_tools:
|
||||
invoke_kwargs["tools"] = self._lc_tools
|
||||
|
||||
for _tool_round in range(_TOOL_MAX_ROUNDS):
|
||||
if _tool_round > 0 and not _has_tools:
|
||||
break
|
||||
|
||||
response = await self.chat_model.ainvoke(messages, **invoke_kwargs)
|
||||
|
||||
response_tool_calls: list[dict[str, Any]] = (
|
||||
getattr(response, "tool_calls", None) or []
|
||||
)
|
||||
if not isinstance(response_tool_calls, list) or not response_tool_calls:
|
||||
break
|
||||
|
||||
if not _has_tools:
|
||||
break
|
||||
|
||||
messages.append(response)
|
||||
|
||||
for tc in response_tool_calls:
|
||||
call_id = tc.get("id", "")
|
||||
# OpenAI standard: {"function": {"name": "...", "arguments": "..."}}
|
||||
# HF / non-standard: {"name": "...", "args": {...}}
|
||||
fn_def = tc.get("function")
|
||||
if isinstance(fn_def, dict):
|
||||
tool_name = fn_def.get("name", "")
|
||||
arguments_raw = fn_def.get("arguments")
|
||||
else:
|
||||
tool_name = tc.get("name", "")
|
||||
arguments_raw = tc.get("args")
|
||||
if not tool_name:
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
content="Tool name is empty",
|
||||
tool_call_id=call_id,
|
||||
)
|
||||
)
|
||||
continue
|
||||
args: dict[str, Any] = {}
|
||||
if isinstance(arguments_raw, str):
|
||||
try:
|
||||
args = json.loads(arguments_raw)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
args = {"_raw": arguments_raw}
|
||||
elif isinstance(arguments_raw, dict):
|
||||
args = arguments_raw
|
||||
try:
|
||||
from cleveractors.agents.tool import ToolAgent as _TA
|
||||
|
||||
# Propagate tool-execution hints from the LLM
|
||||
# agent's config (§4.5.4). The graph YAML places
|
||||
# "unsafe_mode", "allow_shell", and "timeout"
|
||||
# directly on the LLM agent config; we translate
|
||||
# them into ToolAgent-compatible equivalents.
|
||||
parent_unsafe = self.config.get("unsafe_mode", False)
|
||||
tool_config: dict[str, Any] = {
|
||||
"tools": [{"name": tool_name}],
|
||||
"safe_mode": not parent_unsafe,
|
||||
"allow_shell": self.config.get("allow_shell", False),
|
||||
"exec_python": self.config.get("exec_python", False),
|
||||
"timeout": self.config.get("timeout", 1),
|
||||
}
|
||||
|
||||
agent = _TA(
|
||||
name=f"_tc_{call_id}",
|
||||
config=tool_config,
|
||||
template_renderer=self.template_renderer,
|
||||
)
|
||||
tool_ctx: dict[str, Any] | None = (
|
||||
{"_unsafe_mode": True} if parent_unsafe else None
|
||||
)
|
||||
tool_result = await agent.process_message(
|
||||
json.dumps({"tool": tool_name, "args": args})
|
||||
if isinstance(args, dict)
|
||||
else arguments_raw or "",
|
||||
context=tool_ctx,
|
||||
)
|
||||
messages.append(
|
||||
ToolMessage(content=tool_result, tool_call_id=call_id)
|
||||
)
|
||||
except (ExecutionError, ConfigurationError) as _tool_err:
|
||||
_tool_err_msg = str(_tool_err)
|
||||
logger.warning(
|
||||
"Agent %s: tool call failed for %r (tool=%s): %s",
|
||||
self.name,
|
||||
call_id,
|
||||
tool_name,
|
||||
_tool_err_msg,
|
||||
)
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
content=f"Tool '{tool_name}' error: {_tool_err_msg}",
|
||||
tool_call_id=call_id,
|
||||
)
|
||||
)
|
||||
response_text: str = str(response.content)
|
||||
|
||||
# If the tool loop exhausted but the model produced no meaningful
|
||||
# content (stuck in tool-only mode), ask it to synthesize output
|
||||
# from all gathered context. This happens with some models that
|
||||
# never stop making tool calls on their own.
|
||||
if not response_text.strip() and _has_tools and len(messages) > 2:
|
||||
messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"You have finished gathering information. "
|
||||
"Now produce your final answer based on all the "
|
||||
"data collected. Do NOT make any more tool calls. "
|
||||
"If the answer requires writing a file, call "
|
||||
"file_write in this very response."
|
||||
)
|
||||
)
|
||||
)
|
||||
response = await self.chat_model.ainvoke(messages, tools=self._lc_tools)
|
||||
|
||||
# Allow *one* final tool-call round so the model can
|
||||
# write files or perform other last-minute operations
|
||||
# requested by the synthesis prompt.
|
||||
synth_tool_calls: object = getattr(response, "tool_calls", None)
|
||||
if (
|
||||
isinstance(synth_tool_calls, list)
|
||||
and synth_tool_calls
|
||||
and _has_tools
|
||||
):
|
||||
messages.append(response)
|
||||
for tc in synth_tool_calls:
|
||||
call_id = tc.get("id", "")
|
||||
fn_def = tc.get("function")
|
||||
if isinstance(fn_def, dict):
|
||||
tool_name = fn_def.get("name", "")
|
||||
arguments_raw = fn_def.get("arguments")
|
||||
else:
|
||||
tool_name = tc.get("name", "")
|
||||
arguments_raw = tc.get("args")
|
||||
if not tool_name:
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
content="Tool name is empty",
|
||||
tool_call_id=call_id,
|
||||
)
|
||||
)
|
||||
continue
|
||||
args: dict[str, Any] = {}
|
||||
if isinstance(arguments_raw, str):
|
||||
try:
|
||||
args = json.loads(arguments_raw)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
args = {"_raw": arguments_raw}
|
||||
elif isinstance(arguments_raw, dict):
|
||||
args = arguments_raw
|
||||
try:
|
||||
from cleveractors.agents.tool import (
|
||||
ToolAgent as _TA,
|
||||
)
|
||||
|
||||
parent_unsafe = self.config.get("unsafe_mode", False)
|
||||
tool_config: dict[str, Any] = {
|
||||
"tools": [{"name": tool_name}],
|
||||
"safe_mode": not parent_unsafe,
|
||||
"allow_shell": self.config.get("allow_shell", False),
|
||||
"exec_python": self.config.get("exec_python", False),
|
||||
"timeout": self.config.get("timeout", 1),
|
||||
}
|
||||
agent = _TA(
|
||||
name=f"_tc_synth_{call_id}",
|
||||
config=tool_config,
|
||||
template_renderer=self.template_renderer,
|
||||
)
|
||||
tool_ctx: dict[str, Any] | None = (
|
||||
{"_unsafe_mode": True} if parent_unsafe else None
|
||||
)
|
||||
tool_result = await agent.process_message(
|
||||
json.dumps({"tool": tool_name, "args": args})
|
||||
if isinstance(args, dict)
|
||||
else arguments_raw or "",
|
||||
context=tool_ctx,
|
||||
)
|
||||
messages.append(
|
||||
ToolMessage(content=tool_result, tool_call_id=call_id)
|
||||
)
|
||||
except (ExecutionError, ConfigurationError) as _se:
|
||||
_se_msg = str(_se)
|
||||
logger.warning(
|
||||
"Agent %s: synth tool call failed for %r (tool=%s): %s",
|
||||
self.name,
|
||||
call_id,
|
||||
tool_name,
|
||||
_se_msg,
|
||||
)
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
content=f"Tool '{tool_name}' error: {_se_msg}",
|
||||
tool_call_id=call_id,
|
||||
)
|
||||
)
|
||||
response = await self.chat_model.ainvoke(messages)
|
||||
|
||||
response_text = str(response.content)
|
||||
|
||||
# Extract real token usage from LangChain response metadata (AC2).
|
||||
# Primary source: usage_metadata (LangChain standard field).
|
||||
# Fallback: response_metadata["token_usage"] (provider-specific).
|
||||
@@ -593,14 +825,14 @@ class LLMAgent(AgentWithMemory):
|
||||
else:
|
||||
self._last_token_usage = (0, 0)
|
||||
last_token_usage_var.set((0, 0))
|
||||
_err_msg = str(e)
|
||||
logger.error(
|
||||
"LLM agent %s LangChain error: %s", self.name, type(e).__name__
|
||||
)
|
||||
logger.debug(
|
||||
"Raw LangChain exception (sanitized): type=%s",
|
||||
"LLM agent %s LangChain error (%s): %s",
|
||||
self.name,
|
||||
type(e).__name__,
|
||||
_err_msg,
|
||||
)
|
||||
raise ExecutionError("LLM processing failed") from None
|
||||
raise ExecutionError(f"LLM processing failed: {_err_msg}") from None
|
||||
except Exception as e:
|
||||
# Billing integrity: if ainvoke() already succeeded (_captured_prompt
|
||||
# is not None), the LLM provider has already charged the user for
|
||||
@@ -615,14 +847,14 @@ class LLMAgent(AgentWithMemory):
|
||||
else:
|
||||
self._last_token_usage = (0, 0)
|
||||
last_token_usage_var.set((0, 0))
|
||||
_err_msg = str(e)
|
||||
logger.error(
|
||||
"LLM agent %s processing failed: %s", self.name, type(e).__name__
|
||||
)
|
||||
logger.debug(
|
||||
"Raw processing exception (sanitized): type=%s",
|
||||
"LLM agent %s processing failed (%s): %s",
|
||||
self.name,
|
||||
type(e).__name__,
|
||||
_err_msg,
|
||||
)
|
||||
raise ExecutionError("LLM processing failed") from None
|
||||
raise ExecutionError(f"LLM processing failed: {_err_msg}") from None
|
||||
finally:
|
||||
# Restore original temperature if it was overridden.
|
||||
# This prevents the override from leaking into subsequent calls
|
||||
@@ -898,16 +1130,14 @@ class LLMAgent(AgentWithMemory):
|
||||
else:
|
||||
self._last_token_usage = (0, 0)
|
||||
last_token_usage_var.set((0, 0))
|
||||
_err_msg = str(e)
|
||||
logger.error(
|
||||
"LLM agent %s LangChain streaming error: %s",
|
||||
"LLM agent %s LangChain streaming error (%s): %s",
|
||||
self.name,
|
||||
type(e).__name__,
|
||||
_err_msg,
|
||||
)
|
||||
logger.debug(
|
||||
"Raw LangChain streaming exception (sanitized): type=%s",
|
||||
type(e).__name__,
|
||||
)
|
||||
raise ExecutionError("LLM streaming failed") from None
|
||||
raise ExecutionError(f"LLM streaming failed: {_err_msg}") from None
|
||||
except Exception as e: # pylint: disable=broad-exception-caught
|
||||
# Billing integrity: if astream() already completed
|
||||
# (_captured_prompt is not None), the LLM provider has already
|
||||
@@ -922,10 +1152,14 @@ class LLMAgent(AgentWithMemory):
|
||||
else:
|
||||
self._last_token_usage = (0, 0)
|
||||
last_token_usage_var.set((0, 0))
|
||||
_err_msg = str(e)
|
||||
logger.error(
|
||||
"LLM agent %s streaming failed: %s", self.name, type(e).__name__
|
||||
"LLM agent %s streaming failed (%s): %s",
|
||||
self.name,
|
||||
type(e).__name__,
|
||||
_err_msg,
|
||||
)
|
||||
raise ExecutionError("LLM streaming failed") from None
|
||||
raise ExecutionError(f"LLM streaming failed: {_err_msg}") from None
|
||||
finally:
|
||||
# Restore original temperature if it was overridden.
|
||||
# Mirrors process_message() finally block (spec §4.4.5).
|
||||
@@ -1056,6 +1290,10 @@ class LLMAgent(AgentWithMemory):
|
||||
if self.provider.lower() in ["openai", "anthropic", "google"]:
|
||||
capabilities.append("structured-output")
|
||||
|
||||
# Add tool-calling capability when tools are configured.
|
||||
if self._lc_tools:
|
||||
capabilities.append("tool-calling")
|
||||
|
||||
return capabilities
|
||||
|
||||
def get_metadata(self) -> dict[str, Any]:
|
||||
@@ -1071,6 +1309,9 @@ class LLMAgent(AgentWithMemory):
|
||||
"langchain_integration": True,
|
||||
"supports_structured_output": self.provider.lower()
|
||||
in ["openai", "anthropic", "google"],
|
||||
"tools_configured": self._lc_tools is not None
|
||||
and len(self._lc_tools) > 0,
|
||||
"tool_count": len(self._lc_tools) if self._lc_tools else 0,
|
||||
}
|
||||
)
|
||||
return metadata
|
||||
|
||||
@@ -28,6 +28,7 @@ try:
|
||||
from langchain_core.messages import AIMessage as _AIMessage
|
||||
from langchain_core.messages import HumanMessage as _HumanMessage
|
||||
from langchain_core.messages import SystemMessage as _SystemMessage
|
||||
from langchain_core.messages import ToolMessage as _ToolMessage
|
||||
from langchain_openai import ChatOpenAI as _ChatOpenAI
|
||||
|
||||
_LANGCHAIN_AVAILABLE = True
|
||||
@@ -37,6 +38,7 @@ except ImportError:
|
||||
_AIMessage = None
|
||||
_HumanMessage = None
|
||||
_SystemMessage = None
|
||||
_ToolMessage = None
|
||||
_ChatOpenAI = None
|
||||
_LANGCHAIN_AVAILABLE = False
|
||||
|
||||
@@ -80,6 +82,7 @@ def populate_langchain_globals(target_globals: dict[str, Any]) -> None:
|
||||
target_globals["AIMessage"] = _AIMessage
|
||||
target_globals["HumanMessage"] = _HumanMessage
|
||||
target_globals["SystemMessage"] = _SystemMessage
|
||||
target_globals["ToolMessage"] = _ToolMessage
|
||||
target_globals["ChatOpenAI"] = _ChatOpenAI
|
||||
|
||||
if _GOOGLE_AVAILABLE:
|
||||
|
||||
@@ -0,0 +1,316 @@
|
||||
"""Tool normalisation for LLMAgent — converts raw config entries to
|
||||
OpenAI function-calling format (extracted from llm.py).
|
||||
|
||||
Following the same modular-extraction pattern as llm_client.py and
|
||||
llm_providers.py, this module provides the :func:`normalize_tool_entry`
|
||||
function and the :data:`_BUILTIN_TOOL_SCHEMAS` registry used to supply
|
||||
descriptions and parameter schemas for built-in tools when they are
|
||||
referenced by name in an LLM agent's ``tools`` config list.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from cleveractors.core.exceptions import ConfigurationError
|
||||
|
||||
_BUILTIN_TOOL_SCHEMAS: dict[str, dict[str, Any]] = {
|
||||
"echo": {
|
||||
"description": "Echo back the provided text.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"text": {
|
||||
"type": "string",
|
||||
"description": "The text to echo back.",
|
||||
},
|
||||
"args": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Alternative: list of strings to join and echo.",
|
||||
},
|
||||
},
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
"math": {
|
||||
"description": "Evaluate a mathematical expression safely.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"expression": {
|
||||
"type": "string",
|
||||
"description": "A mathematical expression to evaluate (e.g. '2+2').",
|
||||
},
|
||||
"args": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Alternative: first element used as expression.",
|
||||
},
|
||||
},
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
"json_parse": {
|
||||
"description": "Parse a JSON string and return the structured result.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"json": {
|
||||
"type": "string",
|
||||
"description": "The JSON string to parse.",
|
||||
},
|
||||
"args": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Alternative: first element used as JSON string.",
|
||||
},
|
||||
},
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
"http_request": {
|
||||
"description": "Make an HTTP request and return the response.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"url": {
|
||||
"type": "string",
|
||||
"description": "The URL to request.",
|
||||
},
|
||||
"method": {
|
||||
"type": "string",
|
||||
"description": "HTTP method (GET, POST, PUT, DELETE, etc.). Default: GET.",
|
||||
"default": "GET",
|
||||
},
|
||||
"headers": {
|
||||
"type": "object",
|
||||
"description": "Optional HTTP headers as key-value pairs.",
|
||||
},
|
||||
"data": {
|
||||
"type": "object",
|
||||
"description": "Optional JSON body to send with the request.",
|
||||
},
|
||||
},
|
||||
"required": ["url"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
"file_read": {
|
||||
"description": "Read a file or list a directory. CRITICAL: Always use max_chars when reading large files (>10KB) or the LLM context will overflow and the request will fail. For .cs/.py source files use max_chars=8000 for classification, max_chars=16000 for analysis, max_chars=32000 only for small files needing full content. For directories, returns formatted listing with file sizes.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"file": {
|
||||
"type": "string",
|
||||
"description": "Path to the file or directory. Use '.' or './WindowsFormsApplication2' to list a directory.",
|
||||
},
|
||||
"max_chars": {
|
||||
"type": "integer",
|
||||
"description": "REQUIRED for files >10KB. Maximum chars to return. Files exceeding this will show 'TRUNCATED'. Use 8000 for scanning, 16000 for analysis. Leaving this unset on a large file WILL cause context overflow and task failure.",
|
||||
},
|
||||
"args": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Alternative: first element used as file path.",
|
||||
},
|
||||
},
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
"file_write": {
|
||||
"description": "Write content to disk. CRITICAL: Text in your response is NOT saved to disk. You MUST call this tool explicitly to persist any file that downstream agents need. Supports write, append, and insert modes. Use mkdir -p via shell first for new directories.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"file": {
|
||||
"type": "string",
|
||||
"description": "Path to the file to write.",
|
||||
},
|
||||
"content": {
|
||||
"type": "string",
|
||||
"description": "Content to write to the file.",
|
||||
},
|
||||
"mode": {
|
||||
"type": "string",
|
||||
"description": "Write mode: 'w' (overwrite), 'a' (append), or 'insert'.",
|
||||
"enum": ["w", "a", "insert"],
|
||||
"default": "w",
|
||||
},
|
||||
"position": {
|
||||
"type": "integer",
|
||||
"description": "For 'insert' mode, the line number to insert at.",
|
||||
},
|
||||
},
|
||||
"required": ["file", "content"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
"progress_bar": {
|
||||
"description": "Render or update a progress bar.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"stage": {
|
||||
"type": "string",
|
||||
"description": "The current stage or phase name.",
|
||||
},
|
||||
"phase": {
|
||||
"type": "string",
|
||||
"description": "Alternate name for stage.",
|
||||
},
|
||||
"message": {
|
||||
"type": "string",
|
||||
"description": "A label or message for the progress bar.",
|
||||
},
|
||||
"label": {
|
||||
"type": "string",
|
||||
"description": "Alternate name for message.",
|
||||
},
|
||||
"total": {
|
||||
"type": "integer",
|
||||
"description": "Total number of steps.",
|
||||
},
|
||||
"steps": {
|
||||
"type": "integer",
|
||||
"description": "Alternate name for total.",
|
||||
},
|
||||
"count": {
|
||||
"type": "integer",
|
||||
"description": "Alternate name for total.",
|
||||
},
|
||||
"current": {
|
||||
"type": "integer",
|
||||
"description": "Current step number completed.",
|
||||
},
|
||||
"done": {
|
||||
"type": "integer",
|
||||
"description": "Alternate name for current.",
|
||||
},
|
||||
"remaining": {
|
||||
"type": "integer",
|
||||
"description": "Number of steps remaining.",
|
||||
},
|
||||
"sections_total": {
|
||||
"type": "integer",
|
||||
"description": "Total number of sections.",
|
||||
},
|
||||
"sections_completed": {
|
||||
"type": "integer",
|
||||
"description": "Number of sections completed.",
|
||||
},
|
||||
"completed_sections": {
|
||||
"type": "integer",
|
||||
"description": "Alternate name for sections_completed.",
|
||||
},
|
||||
},
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
"shell": {
|
||||
"description": "Execute a shell command and return stdout. Requires allow_shell. Use for: find, grep, cat. Prefer file_read for reading files (has max_chars to prevent context overflow). When creating files in new directories, always use 'mkdir -p <dir> && touch <path>'.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"command": {
|
||||
"type": "string",
|
||||
"description": "Shell command. Chain with && for multi-step: 'mkdir -p dir && touch dir/__init__.py' or 'find . -name \"*.cs\" | sort'.",
|
||||
},
|
||||
"args": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Additional command-line arguments.",
|
||||
},
|
||||
},
|
||||
"required": ["command"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
"python_exec": {
|
||||
"description": "Execute Python code in a sandbox. Requires exec_python. Available builtins: print, len, str, int, float, bool, list, dict, set, tuple, range, enumerate, zip, min, max, sum, abs, round, sorted, reversed, any, all, isinstance, type. Use 'import json' for JSON operations. For file operations use the file_read and file_write tools.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string",
|
||||
"description": "The Python code to execute.",
|
||||
},
|
||||
"args": {
|
||||
"type": "object",
|
||||
"description": "Optional variable bindings passed to the code sandbox.",
|
||||
},
|
||||
},
|
||||
"required": ["code"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def normalize_tool_entry(item: object) -> dict[str, Any]:
|
||||
"""Convert a raw tool config entry into OpenAI function-calling format.
|
||||
|
||||
Accepts three input shapes:
|
||||
|
||||
* **String** — a built-in tool name, e.g. ``"file_read"``.
|
||||
* **Dict with ``"name"``** — a simple tool reference, e.g.
|
||||
``{"name": "echo"}``. If the ``"name"`` matches a built-in tool,
|
||||
the tool's parameters schema is included automatically; otherwise a
|
||||
minimal schema with an empty ``properties`` dict is generated.
|
||||
* **Already-formatted OpenAI tool** — a dict with ``"type"`` and
|
||||
``"function"`` keys is returned unchanged.
|
||||
|
||||
Args:
|
||||
item: The raw config entry — a string, a dict, or an
|
||||
already-formatted OpenAI tool dict.
|
||||
|
||||
Returns:
|
||||
A dict in ``{"type": "function", "function": {...}}`` format.
|
||||
|
||||
Raises:
|
||||
ConfigurationError: If ``item`` is not a recognised format.
|
||||
"""
|
||||
if isinstance(item, dict):
|
||||
if "type" in item and "function" in item:
|
||||
return dict(item)
|
||||
tool_name = item.get("name")
|
||||
if not isinstance(tool_name, str) or not tool_name:
|
||||
raise ConfigurationError(
|
||||
f"Tool dict must contain a non-empty string 'name': {item!r}"
|
||||
)
|
||||
schema = _BUILTIN_TOOL_SCHEMAS.get(tool_name, {})
|
||||
return {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": tool_name,
|
||||
"description": schema.get(
|
||||
"description", f"Execute the '{tool_name}' tool."
|
||||
),
|
||||
"parameters": schema.get(
|
||||
"parameters",
|
||||
{"type": "object", "properties": {}},
|
||||
),
|
||||
},
|
||||
}
|
||||
if isinstance(item, str):
|
||||
if not item:
|
||||
raise ConfigurationError(
|
||||
f"Tool name must be a non-empty string, got {item!r}"
|
||||
)
|
||||
schema = _BUILTIN_TOOL_SCHEMAS.get(item, {})
|
||||
return {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": item,
|
||||
"description": schema.get("description", f"Execute the '{item}' tool."),
|
||||
"parameters": schema.get(
|
||||
"parameters",
|
||||
{"type": "object", "properties": {}},
|
||||
),
|
||||
},
|
||||
}
|
||||
raise ConfigurationError(
|
||||
f"Invalid tool config entry: {item!r}. "
|
||||
"Expected a string tool name, a dict with 'name' key, "
|
||||
"or an OpenAI-formatted tool dict."
|
||||
)
|
||||
+175
-26
@@ -77,6 +77,10 @@ class ToolAgent(Agent):
|
||||
"file_write": self._file_write_tool,
|
||||
"progress_bar": self._progress_bar_tool,
|
||||
}
|
||||
if self.allow_shell:
|
||||
self.builtin_tools["shell"] = self._shell_tool
|
||||
if self.config.get("exec_python", False):
|
||||
self.builtin_tools["python_exec"] = self._python_exec_tool
|
||||
|
||||
# Validate tools
|
||||
self._validate_tools()
|
||||
@@ -286,15 +290,19 @@ class ToolAgent(Agent):
|
||||
f"Dangerous command '{command}' blocked in safe mode"
|
||||
)
|
||||
|
||||
# Build command with arguments
|
||||
cmd_parts = [command]
|
||||
if "args" in args:
|
||||
cmd_parts.extend(str(arg) for arg in args["args"])
|
||||
# Build the full shell command string. When the caller provides
|
||||
# positional arguments via ``args["args"]`` they are appended;
|
||||
# otherwise the entire ``command`` string is used as-is so that
|
||||
# piped / chained / redirected shell constructs work correctly.
|
||||
cmd_str = command
|
||||
if "args" in args and args["args"]:
|
||||
cmd_str = command + " " + " ".join(str(a) for a in args["args"])
|
||||
|
||||
try:
|
||||
# Execute with timeout
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
*cmd_parts, stdout=subprocess.PIPE, stderr=subprocess.PIPE
|
||||
# Execute with timeout — use subprocess_shell so pipes, &&,
|
||||
# redirections, etc. are handled by the system shell.
|
||||
process = await asyncio.create_subprocess_shell(
|
||||
cmd_str, stdout=subprocess.PIPE, stderr=subprocess.PIPE
|
||||
)
|
||||
|
||||
stdout, stderr = await asyncio.wait_for(
|
||||
@@ -314,6 +322,42 @@ class ToolAgent(Agent):
|
||||
f"Command '{command}' timed out after {self.timeout} seconds"
|
||||
) from timeout_err
|
||||
|
||||
async def _shell_tool(
|
||||
self,
|
||||
args: dict[str, Any],
|
||||
context: Optional[dict[str, Any]], # pylint: disable=unused-argument
|
||||
) -> str:
|
||||
"""Execute an arbitrary shell command.
|
||||
|
||||
Requires ``allow_shell`` to be enabled on the agent.
|
||||
Accepts ``command`` as the executable and optional ``args``
|
||||
as a list of string arguments.
|
||||
"""
|
||||
if not self.allow_shell:
|
||||
raise ExecutionError("Shell execution is disabled for this agent")
|
||||
command = args.get("command", "")
|
||||
if not command:
|
||||
raise ExecutionError("Shell tool requires a 'command' argument")
|
||||
return await self._execute_shell_command(command, args)
|
||||
|
||||
async def _python_exec_tool(
|
||||
self,
|
||||
args: dict[str, Any],
|
||||
context: Optional[dict[str, Any]],
|
||||
) -> str:
|
||||
"""Execute arbitrary Python code in a sandboxed environment.
|
||||
|
||||
Requires ``exec_python`` to be enabled on the agent.
|
||||
Accepts ``code`` (the Python code string) and ``args``
|
||||
(a dict of variable bindings passed to the code sandbox).
|
||||
"""
|
||||
if not self.config.get("exec_python", False):
|
||||
raise ExecutionError("Python execution is disabled for this agent")
|
||||
code = args.get("code", "")
|
||||
if not code:
|
||||
raise ExecutionError("python_exec tool requires a 'code' argument")
|
||||
return await self._execute_python_code(code, args, context)
|
||||
|
||||
async def _execute_python_code(
|
||||
self, code: str, args: dict[str, Any], context: Optional[dict[str, Any]]
|
||||
) -> str:
|
||||
@@ -395,12 +439,15 @@ class ToolAgent(Agent):
|
||||
# Redirect stderr to suppress print(..., file=sys.stderr) statements
|
||||
old_stderr = sys.stderr
|
||||
sys.stderr = io.StringIO()
|
||||
try:
|
||||
exec(code, exec_env, exec_env) # nosec B102: sandboxed inline code execution # nosemgrep
|
||||
finally:
|
||||
sys.stderr = old_stderr
|
||||
else:
|
||||
old_stdout = sys.stdout
|
||||
sys.stdout = io.StringIO()
|
||||
try:
|
||||
exec(code, exec_env, exec_env) # nosec B102: sandboxed inline code execution # nosemgrep
|
||||
finally:
|
||||
stdout_output = sys.stdout.getvalue()
|
||||
sys.stdout = old_stdout
|
||||
if should_suppress:
|
||||
sys.stderr = old_stderr
|
||||
|
||||
logger.debug(f"After exec - context: {exec_env.get('context')}")
|
||||
|
||||
@@ -423,8 +470,23 @@ class ToolAgent(Agent):
|
||||
result = exec_env[key]
|
||||
break
|
||||
|
||||
captured = stdout_output.strip()
|
||||
if captured:
|
||||
if result is None:
|
||||
result = captured
|
||||
else:
|
||||
result = f"{result}\n\n{captured}"
|
||||
|
||||
return str(result) if result is not None else ""
|
||||
|
||||
except NameError as e:
|
||||
_name = str(e).split("'")[1] if "'" in str(e) else str(e)
|
||||
logger.error("Python code execution failed in %s: %s", self.name, e)
|
||||
raise ExecutionError(
|
||||
f"'{_name}' is not available in the Python sandbox. "
|
||||
f"Use the file_read and file_write tools for file operations, "
|
||||
f"and the shell tool for system commands."
|
||||
) from e
|
||||
except Exception as e:
|
||||
logger.error("Python code execution failed in %s: %s", self.name, e)
|
||||
raise ExecutionError(f"Python code execution failed: {str(e)}") from e
|
||||
@@ -515,7 +577,7 @@ class ToolAgent(Agent):
|
||||
async def _file_read_tool(
|
||||
self, args: dict[str, Any], context: Optional[dict[str, Any]]
|
||||
) -> str:
|
||||
"""File reading tool."""
|
||||
"""File reading tool with directory listing support and optional truncation."""
|
||||
filepath = args.get("file", "")
|
||||
if "args" in args and args["args"]:
|
||||
filepath = args["args"][0]
|
||||
@@ -523,33 +585,120 @@ class ToolAgent(Agent):
|
||||
if not filepath:
|
||||
raise ExecutionError("File read tool requires a file path")
|
||||
|
||||
# Detect shell commands mistakenly passed as file paths
|
||||
_shell_indicators = ["|", ";", "&&", "||", ">", "<"]
|
||||
_shell_commands = {
|
||||
"ls",
|
||||
"cat",
|
||||
"grep",
|
||||
"head",
|
||||
"tail",
|
||||
"find",
|
||||
"sort",
|
||||
"wc",
|
||||
"touch",
|
||||
"mkdir",
|
||||
"rm",
|
||||
"cp",
|
||||
"mv",
|
||||
"echo",
|
||||
"python",
|
||||
"python3",
|
||||
"read",
|
||||
"awk",
|
||||
"sed",
|
||||
"xargs",
|
||||
"cut",
|
||||
"tr",
|
||||
"tee",
|
||||
"env",
|
||||
"export",
|
||||
"set",
|
||||
"unset",
|
||||
"source",
|
||||
"chmod",
|
||||
"chown",
|
||||
}
|
||||
_first_word = filepath.strip().split()[0].lower() if filepath.strip() else ""
|
||||
if (
|
||||
any(ind in filepath for ind in _shell_indicators)
|
||||
or _first_word in _shell_commands
|
||||
):
|
||||
raise ExecutionError(
|
||||
f"'{filepath}' looks like a shell command, not a file path. "
|
||||
f"Use the shell tool instead of file_read for commands like "
|
||||
f"grep, find, ls, cat. Use file_read only for file/directory paths."
|
||||
)
|
||||
|
||||
if self.safe_mode:
|
||||
# Always block directory traversal attempts
|
||||
if ".." in filepath:
|
||||
raise ExecutionError("Unsafe file path blocked in safe mode")
|
||||
# Block absolute paths unless in unsafe mode
|
||||
if filepath.startswith("/") and not (
|
||||
context and context.get("_unsafe_mode", False)
|
||||
):
|
||||
raise ExecutionError("Unsafe file path blocked in safe mode")
|
||||
|
||||
max_chars = args.get("max_chars")
|
||||
if max_chars is not None:
|
||||
try:
|
||||
max_chars = int(max_chars)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ExecutionError(
|
||||
f"max_chars must be an integer, got {max_chars}"
|
||||
) from exc
|
||||
|
||||
try:
|
||||
if os.path.isdir(filepath):
|
||||
entries = sorted(os.listdir(filepath))
|
||||
result_parts = [f"[FILE_READ_SUCCESS]📁 Directory: {filepath}"]
|
||||
result_parts.append(f"Entries: {len(entries)}")
|
||||
for entry in entries:
|
||||
entry_path = os.path.join(filepath, entry)
|
||||
marker = "📁" if os.path.isdir(entry_path) else "📄"
|
||||
try:
|
||||
size = os.path.getsize(entry_path)
|
||||
size_str = f" ({size:,} bytes)"
|
||||
except OSError:
|
||||
size_str = ""
|
||||
result_parts.append(f" {marker} {entry}{size_str}")
|
||||
return "\n".join(result_parts)
|
||||
|
||||
with open(filepath, "r", encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
# Return content with clean terminal indicator
|
||||
# The content is still in the response for context/memory
|
||||
# but we add a prefix for clean terminal display
|
||||
line_count = content.count("\n") + 1
|
||||
|
||||
char_count = len(content)
|
||||
# Format: Special marker + metadata + full content
|
||||
# The special marker helps identify this for clean display
|
||||
return (
|
||||
line_count = content.count("\n") + 1
|
||||
truncated = False
|
||||
|
||||
if max_chars is not None and char_count > max_chars:
|
||||
content = content[:max_chars]
|
||||
truncated = True
|
||||
|
||||
header = (
|
||||
f"[FILE_READ_SUCCESS]📄 File: {filepath} | "
|
||||
f"Lines: {line_count} | Size: {char_count} chars\n"
|
||||
f"[FILE_CONTENT_START]\n{content}\n[FILE_CONTENT_END]"
|
||||
f"Lines: {line_count} | Size: {char_count} chars"
|
||||
)
|
||||
except Exception as e:
|
||||
raise ExecutionError(f"File read failed: {e}") from e
|
||||
if truncated:
|
||||
header += f" | TRUNCATED to {max_chars} chars"
|
||||
return f"{header}\n[FILE_CONTENT_START]\n{content}\n[FILE_CONTENT_END]"
|
||||
except FileNotFoundError:
|
||||
parent_dir = os.path.dirname(filepath) or "."
|
||||
raise ExecutionError(
|
||||
f"File not found: '{filepath}'. "
|
||||
f"Try listing the parent directory with file_read '{parent_dir}' "
|
||||
f"to see available files."
|
||||
) from None
|
||||
except OSError as e:
|
||||
raise ExecutionError(
|
||||
f"File read failed: {e}. "
|
||||
f"Check that the path is accessible and you have the required permissions."
|
||||
) from e
|
||||
except UnicodeDecodeError as e:
|
||||
raise ExecutionError(
|
||||
f"File read failed: cannot decode '{filepath}' as UTF-8. "
|
||||
f"The file may contain binary content. Use the shell tool "
|
||||
f"with appropriate encoding commands if binary extraction is needed."
|
||||
) from e
|
||||
|
||||
def _validate_file_write_args(self, filepath: str, content: str) -> None:
|
||||
"""Validate file write arguments."""
|
||||
|
||||
@@ -172,6 +172,8 @@ class Node: # pylint: disable=too-many-instance-attributes
|
||||
|
||||
for msg in reversed(messages):
|
||||
content = str(msg.get("content", ""))
|
||||
if not content.strip():
|
||||
continue
|
||||
total_chars += len(content)
|
||||
collected.append(deepcopy(msg))
|
||||
|
||||
|
||||
Reference in New Issue
Block a user