feat(agents): surface timeout budget in tool schemas and errors
CI / lint (pull_request) Successful in 1m5s
CI / quality (pull_request) Successful in 1m49s
CI / typecheck (pull_request) Successful in 2m2s
CI / build (pull_request) Successful in 1m39s
CI / unit_tests (pull_request) Successful in 5m12s
CI / integration_tests (pull_request) Has been cancelled
CI / coverage (pull_request) Has been cancelled
CI / status-check (pull_request) Has been cancelled
CI / security (pull_request) Has been cancelled
CI / benchmark (pull_request) Has been cancelled
CI / lint (pull_request) Successful in 1m5s
CI / quality (pull_request) Successful in 1m49s
CI / typecheck (pull_request) Successful in 2m2s
CI / build (pull_request) Successful in 1m39s
CI / unit_tests (pull_request) Successful in 5m12s
CI / integration_tests (pull_request) Has been cancelled
CI / coverage (pull_request) Has been cancelled
CI / status-check (pull_request) Has been cancelled
CI / security (pull_request) Has been cancelled
CI / benchmark (pull_request) Has been cancelled
The `shell` and `http_request` built-in tools are the only two whose execution actually honors the tool agent's `timeout` config field (ToolAgent._execute_shell_command, ToolAgent._http_request_tool), but their LLM-facing schemas (_BUILTIN_TOOL_SCHEMAS in llm_tools.py) never told the model a timeout applied, and neither tool's timeout error named the config field that controls it. - _BUILTIN_TOOL_SCHEMAS["shell"] and ["http_request"] descriptions now state that execution is bounded by the agent's configured `timeout` (seconds), matching the existing precedent of documenting `file_read`'s `max_chars` behavior in its schema. - _execute_shell_command's timeout ExecutionError now names `timeout` as the adjustable config field, in addition to the elapsed seconds it already reported. - _http_request_tool now catches asyncio.TimeoutError distinctly from the generic `except Exception`, so a request timeout produces a message naming the config field instead of the previous generic "HTTP request failed" text (which gave no indication a timeout was the cause). Non-timeout HTTP failures are unaffected — same exception handling, same message shape as before. No change to enforcement mechanics: `self.timeout` remains the sole config field (default 1s, Actor Configuration Standard §4.5). This is schema/message content only, within the extension precedent already established by ADR-2030 (D-1 schema registry, D-4/D-6 error-quality improvements) — no new ADR was needed for this change. Tests: extended features/tool_agent.feature (updated the two existing shell/http_request timeout scenarios to also assert the new config-field hint) and features/llm_tools_coverage.feature (new scenarios asserting both schema descriptions mention the timeout budget). Full suite: 141 features / 2782 scenarios / 13172 steps passed; coverage 96.8% (>=96.5% threshold). Robot integration_tests: 27/27 suites passed. ASV benchmark_regression: no significant change, as expected for a messaging-only change. ISSUES CLOSED: #82
This commit is contained in:
@@ -168,6 +168,7 @@ and this project adheres to [Clever Semantic Versioning](https://www.w3.org/subm
|
||||
|
||||
### Changed
|
||||
|
||||
- **Timeout Awareness Surfaced to the LLM for `shell` and `http_request` Tools (issue #82)** (`agents/llm_tools.py`, `agents/tool.py`): The LLM-facing tool-calling schemas for `shell` and `http_request` now state that execution is bounded by the agent's configured `timeout` (seconds) — previously only `file_read`'s `max_chars` guidance was documented in the schema, leaving the model unaware that these two tools could be killed/aborted mid-call. When a timeout is actually triggered, `ToolAgent._execute_shell_command` and `ToolAgent._http_request_tool` now name `timeout` as the adjustable agent config field in the raised `ExecutionError`, in addition to the elapsed seconds already reported. `_http_request_tool` also now catches the timeout condition distinctly from other request failures (connection errors, invalid URLs, etc.), which previously all collapsed into one generic "HTTP request failed" message with no indication a timeout occurred. No change to the underlying `timeout` enforcement mechanics (§4.5) — this is schema/message content only. BDD: scenarios in `features/tool_agent.feature` and `features/llm_tools_coverage.feature`.
|
||||
- **Real LangChain `usage_metadata` token extraction**: `LLMAgent.process_message()` now reads token counts directly from LangChain response metadata instead of estimating via tiktoken or a 4-chars/token heuristic. `ActorResult.prompt_tokens`/`completion_tokens` now reflect actual LLM billing data.
|
||||
- **`ActorResult`/`NodeUsage` moved to canonical `cleveractors.result` module**: Both types are now defined in `src/cleveractors/result.py` (ADR-2027 §3.1) and re-exported from `cleveractors.runtime` for backward compatibility. `cleveractors.__init__` updated to import from `result` instead of `runtime`.
|
||||
- **RxPy Architecture**: Converted routing system from synchronous to reactive (RxPy) streams using Observable/Subject patterns for asynchronous message flow.
|
||||
|
||||
@@ -31,4 +31,16 @@ Feature: LLM Tools Coverage
|
||||
Scenario: normalize_tool_entry rejects non-str non-dict type
|
||||
Given an llm_tools coverage test environment
|
||||
When I call normalize_tool_entry with integer 42
|
||||
Then a ConfigurationError mentioning Invalid tool config entry is raised
|
||||
Then a ConfigurationError mentioning Invalid tool config entry is raised
|
||||
|
||||
Scenario: shell tool schema description mentions the configurable timeout
|
||||
Given an llm_tools coverage test environment
|
||||
When I call normalize_tool_entry with string "shell"
|
||||
Then the result description should mention "timeout"
|
||||
And the result description should mention "config"
|
||||
|
||||
Scenario: http_request tool schema description mentions the configurable timeout
|
||||
Given an llm_tools coverage test environment
|
||||
When I call normalize_tool_entry with string "http_request"
|
||||
Then the result description should mention "timeout"
|
||||
And the result description should mention "config"
|
||||
@@ -73,3 +73,10 @@ def step_lt_same_dict(context):
|
||||
def step_lt_config_error(context, text):
|
||||
assert context.error is not None
|
||||
assert text in context.error
|
||||
|
||||
|
||||
@then('the result description should mention "{text}"')
|
||||
def step_lt_description_mentions(context, text):
|
||||
assert context.result is not None
|
||||
description = context.result["function"]["description"]
|
||||
assert text in description, f"Expected '{text}' in description: {description!r}"
|
||||
|
||||
@@ -242,6 +242,7 @@ Feature: ToolAgent Functionality
|
||||
When I create the ToolAgent
|
||||
And I process a message with the ToolAgent: "sleep 1"
|
||||
Then tool execution should fail with message containing "timed out after 0.1 seconds"
|
||||
And tool execution should fail with message containing "'timeout' config value"
|
||||
|
||||
Scenario: Tool agent shell command with non-zero exit code
|
||||
Given I am running in unsafe mode
|
||||
@@ -536,7 +537,8 @@ Feature: ToolAgent Functionality
|
||||
"""
|
||||
{"tool": "http_request", "args": {"url": "https://httpbin.org/delay/1", "method": "GET"}}
|
||||
"""
|
||||
Then tool execution should fail with message containing "HTTP request failed"
|
||||
Then tool execution should fail with message containing "timed out after 0.001 seconds"
|
||||
And tool execution should fail with message containing "'timeout' config value"
|
||||
|
||||
Scenario: Tool agent http_request tool with invalid URL
|
||||
Given a ToolAgent is configured with name "http_invalid_agent" and config
|
||||
|
||||
@@ -70,7 +70,7 @@ _BUILTIN_TOOL_SCHEMAS: dict[str, dict[str, Any]] = {
|
||||
},
|
||||
},
|
||||
"http_request": {
|
||||
"description": "Make an HTTP request and return the response.",
|
||||
"description": "Make an HTTP request and return the response. Execution is bounded by the agent's configured `timeout` (seconds, default 1); requests that exceed it are aborted and return a timeout error — increase `timeout` in the tool agent config for slow endpoints.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
@@ -208,7 +208,7 @@ _BUILTIN_TOOL_SCHEMAS: dict[str, dict[str, Any]] = {
|
||||
},
|
||||
},
|
||||
"shell": {
|
||||
"description": "Execute a shell command and return stdout. Requires allow_shell. Use for: find, grep, cat. Prefer file_read for reading files (has max_chars to prevent context overflow). When creating files in new directories, always use 'mkdir -p <dir> && touch <path>'.",
|
||||
"description": "Execute a shell command and return stdout. Requires allow_shell. Use for: find, grep, cat. Prefer file_read for reading files (has max_chars to prevent context overflow). When creating files in new directories, always use 'mkdir -p <dir> && touch <path>'. Execution is bounded by the agent's configured `timeout` (seconds, default 1); commands that exceed it are killed and return a timeout error — increase `timeout` in the tool agent config for long-running commands.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
||||
@@ -351,7 +351,9 @@ class ToolAgent(Agent):
|
||||
|
||||
except asyncio.TimeoutError as timeout_err:
|
||||
raise ExecutionError(
|
||||
f"Command '{command}' timed out after {self.timeout} seconds"
|
||||
f"Command '{command}' timed out after {self.timeout} seconds. "
|
||||
"Increase the tool agent's 'timeout' config value (in seconds) "
|
||||
"if this command legitimately needs more time."
|
||||
) from timeout_err
|
||||
|
||||
async def _shell_tool(
|
||||
@@ -603,6 +605,12 @@ class ToolAgent(Agent):
|
||||
) as response:
|
||||
content = await response.text()
|
||||
return f"Status: {response.status}\n\n{content}"
|
||||
except asyncio.TimeoutError as timeout_err:
|
||||
raise ExecutionError(
|
||||
f"HTTP request to '{url}' timed out after {self.timeout} seconds. "
|
||||
"Increase the tool agent's 'timeout' config value (in seconds) "
|
||||
"if this endpoint legitimately needs more time."
|
||||
) from timeout_err
|
||||
except Exception as e:
|
||||
raise ExecutionError(f"HTTP request failed: {e}") from e
|
||||
|
||||
|
||||
Reference in New Issue
Block a user