Allow the shell tool timeout to be overridden per invocation via a `timeout` argument #112

New issue

Closed

opened 2026-08-04 21:31:20 +00:00 by CoreRasurae · 1 comment

CoreRasurae commented

2026-08-04 21:31:20 +00:00

Member

Metadata

Commit Message: feat(agents): allow per-invocation timeout override on the shell tool
Branch: feature/m1-shell-timeout-override

Background and context

The tool agent timeout config field (Actor Configuration Standard §4.5, default 1 second) bounds "the maximum execution time in seconds for any single tool invocation." Today it is a static, per-agent value read once at construction (cleveractors.agents.tool.ToolAgent.__init__ sets self.timeout = cfg.get("timeout", 1)) and applied to every shell invocation via asyncio.wait_for(process.communicate(), timeout=self.timeout) in cleveractors.agents.tool.ToolAgent._execute_shell_command.

Issue #82 (feat(agents): surface timeout budget in tool schemas and errors, closed) made this constraint visible to the LLM: it added timeout language to the shell/http_request schema descriptions and made the timeout ExecutionError name the timeout config field. It deliberately stopped there — its own scope note states "No change to enforcement mechanics … self.timeout remains the sole config field."

That leaves a gap that actively harms autonomous runs. The guidance #82 surfaces tells the model to "increase the tool agent's timeout config value", but an LLM driving the shell tool has no runtime affordance to edit its own agent's YAML config — the only channel it has is the tool-call arguments. So when a command legitimately needs more than the default 1 second, the model is told to do something it cannot do, and it flails trying to satisfy the instruction through the only lever it has.

A real failure trace (the motivating case):

Command 'cd ./output_project && source .venv/bin/activate && pip show pytest-bdd 2>/dev/null || pip install pytest-bdd' timed out after 1 seconds. Increase the tool agent's 'timeout' config value (in seconds) if this command legitimately needs more time.
...
head: cannot open 'timeout' for reading: No such file or directory
head: cannot open '120' for reading: No such file or directory

The second error is the model attempting to "raise the timeout" the only way it can — by threading timeout 120 into the call — which _execute_shell_command concatenates onto the command line (cmd_str = command + " " + " ".join(...)), producing a nonsense command. The model has the right intent and no correct mechanism.

This issue closes that gap by making the shell timeout overridable per invocation through a first-class timeout tool argument, turning #82's advice into something the agent can actually act on.

Current behavior

cleveractors.agents.tool.ToolAgent._execute_shell_command always bounds execution by self.timeout (the single static per-agent value); there is no per-call override.
cleveractors.agents.llm_tools._BUILTIN_TOOL_SCHEMAS["shell"] exposes only command and args (a list of "Additional command-line arguments"). There is no timeout parameter the model can set, yet the schema description tells the model to "increase timeout in the tool agent config."
Because the only writable channel is args, a model trying to follow that advice ends up injecting timeout-related tokens into args, which are appended verbatim to the shell command string — corrupting the command (see trace above).
The timeout ExecutionError raised by _execute_shell_command directs the reader to edit agent config — an action available to a human config author but not to the LLM at runtime.

Expected behavior

_BUILTIN_TOOL_SCHEMAS["shell"] exposes an optional timeout parameter (a number, seconds). It is NOT added to the schema's required list. Its description states it overrides the agent's default timeout for this single invocation and is bounded by a maximum.
ToolAgent._execute_shell_command uses a per-invocation timeout taken from the call arguments when present and valid, in place of self.timeout, for the asyncio.wait_for bound.
When no per-invocation timeout is supplied, behavior is byte-for-byte unchanged: the call is bounded by self.timeout (default 1s per §4.5).
The per-invocation timeout is a control argument: it is read distinctly from the positional args list and MUST NOT be concatenated into the executed command string.
Argument validation runs first (per the project error-handling rules): a supplied timeout that is non-numeric, <= 0, or above the allowed ceiling raises ExecutionError with an explanatory message, before any subprocess is spawned.
The shell timeout ExecutionError message is rewritten to point the model at the actionable lever — retrying with a larger timeout argument (up to the ceiling) — rather than instructing it to edit config it cannot reach at runtime.

Design points to settle in the ADR / review (not yet fixed by this issue):

The ceiling for a per-call timeout. Proposal: a bounded maximum so a model cannot pin a worker indefinitely — either a fixed cap or a new max_timeout agent config field. Absent consensus, default to a fixed, documented cap.
Whether the per-call value may only raise the bound or may also lower it (proposal: either direction, within (0, ceiling]).

Out of scope (mirroring #82's scoping, noted for awareness): cleveractors.agents.tool.ToolAgent._http_request_tool shares the same self.timeout and has the same limitation. Extending the per-invocation override to http_request is a parallel follow-up issue, not addressed here, to keep this change atomic.

Acceptance criteria

_BUILTIN_TOOL_SCHEMAS["shell"]["parameters"]["properties"] contains an optional numeric timeout property whose description states it overrides the agent default for this invocation (bounded by the ceiling); timeout is absent from that schema's required array.
With a valid per-invocation timeout supplied, ToolAgent._execute_shell_command bounds asyncio.wait_for by that value rather than self.timeout (verifiable: a call given timeout larger than self.timeout completes a command that runs longer than self.timeout).
With no timeout supplied, a command exceeding self.timeout still raises the timeout ExecutionError — default behavior is unchanged.
A per-invocation timeout value never appears in the executed command string (verifiable: a shell call with a timeout argument runs the intended command, not a corrupted one).
A supplied timeout that is non-numeric, <= 0, or above the ceiling raises ExecutionError before a subprocess is created, with a message naming the accepted range.
The shell timeout ExecutionError message instructs retrying with a larger timeout argument and no longer instructs editing the agent config.
nox (all default sessions) is green and nox -s coverage_report stays ≥ 97%.

Supporting information

Actor Configuration Standard §4.5 (docs/index.md) — defines timeout as "Maximum execution time in seconds for any single tool invocation." A per-invocation override is an extension of this field's semantics and therefore requires the ADR-before-code process (see Subtasks): a decision record extending docs/adr/ADR-2030-tool-calling-spec-extensions.md, then a §4.5 spec update via the standard spec-revision procedure (bump the document Version and add a §21.1 Revision History row attributing the change to the ADR).
docs/adr/ADR-2030-tool-calling-spec-extensions.md — D-1 establishes _BUILTIN_TOOL_SCHEMAS as LLM-facing hints; D-3 (file_read max_chars) establishes the precedent that an optional tool argument may be added because §4.5.1 "allows additional arguments to be handled." A per-invocation timeout follows the same pattern.
Predecessor: #82 surfaced the timeout budget in schemas/messages but explicitly kept enforcement static. This issue makes that surfaced guidance actionable.
Relevant symbols at commit 67972f0:
- cleveractors.agents.tool.ToolAgent.__init__ (sets self.timeout = cfg.get("timeout", 1))
- cleveractors.agents.tool.ToolAgent._execute_shell_command (applies self.timeout; builds cmd_str; raises the timeout ExecutionError)
- cleveractors.agents.tool.ToolAgent._execute_tool (passes the full tool-call args dict into _execute_shell_command)
- cleveractors.agents.llm_tools._BUILTIN_TOOL_SCHEMAS (the shell schema)

Subtasks

ADR: add a decision record extending ADR-2030 for a per-invocation shell timeout override (including the ceiling decision); submit for review and get it accepted before writing code
Spec: update Actor Configuration Standard §4.5 per the spec-revision procedure — bump the document Version and add a §21.1 Revision History row attributing the change to the ADR (do not edit §4.5 prose inline without the version bump + attribution)
Add the optional timeout property to _BUILTIN_TOOL_SCHEMAS["shell"] and update its description to describe the override and its bound
In ToolAgent._execute_shell_command, validate and read a per-invocation timeout from the tool-call args, use it for the asyncio.wait_for bound with fallback to self.timeout, and ensure it is never concatenated into cmd_str
Rewrite the shell timeout ExecutionError message to direct the model to retry with a larger timeout argument (up to the ceiling)
Tests (Behave): per-call timeout honored (a long command succeeds when overridden, fails at the default); absent → default behavior; invalid values rejected before spawn; the timeout argument is not injected into the command; updated error-message wording
Tests (Robot): integration scenario invoking the shell tool with a per-invocation timeout against a real subprocess
Benchmarks (ASV): assess whether any perf-sensitive path changed (expected N/A — timeout only bounds wall-clock); document the assessment
Verify coverage >= 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
The ADR extending ADR-2030 is accepted and §4.5 of the standard is updated via the spec-revision procedure before implementation code is merged.
A Git commit is created where the first line matches the Commit Message in Metadata exactly.
The commit is pushed to the branch matching the Branch in Metadata exactly.
The commit is submitted as a PR to master, reviewed, and merged.

## Metadata Commit Message: `feat(agents): allow per-invocation timeout override on the shell tool` Branch: `feature/m1-shell-timeout-override` ## Background and context The tool agent `timeout` config field (Actor Configuration Standard §4.5, default `1` second) bounds "the maximum execution time in seconds for any single tool invocation." Today it is a **static, per-agent** value read once at construction (`cleveractors.agents.tool.ToolAgent.__init__` sets `self.timeout = cfg.get("timeout", 1)`) and applied to every `shell` invocation via `asyncio.wait_for(process.communicate(), timeout=self.timeout)` in `cleveractors.agents.tool.ToolAgent._execute_shell_command`. Issue #82 (`feat(agents): surface timeout budget in tool schemas and errors`, closed) made this constraint *visible* to the LLM: it added timeout language to the `shell`/`http_request` schema descriptions and made the timeout `ExecutionError` name the `timeout` config field. It deliberately stopped there — its own scope note states "No change to enforcement mechanics … `self.timeout` remains the sole config field." That leaves a gap that actively harms autonomous runs. The guidance #82 surfaces tells the model to *"increase the tool agent's `timeout` config value"*, but an LLM driving the `shell` tool has **no runtime affordance to edit its own agent's YAML config** — the only channel it has is the tool-call arguments. So when a command legitimately needs more than the default 1 second, the model is told to do something it cannot do, and it flails trying to satisfy the instruction through the only lever it has. A real failure trace (the motivating case): ``` Command 'cd ./output_project && source .venv/bin/activate && pip show pytest-bdd 2>/dev/null || pip install pytest-bdd' timed out after 1 seconds. Increase the tool agent's 'timeout' config value (in seconds) if this command legitimately needs more time. ... head: cannot open 'timeout' for reading: No such file or directory head: cannot open '120' for reading: No such file or directory ``` The second error is the model attempting to "raise the timeout" the only way it can — by threading `timeout 120` into the call — which `_execute_shell_command` concatenates onto the command line (`cmd_str = command + " " + " ".join(...)`), producing a nonsense command. The model has the right intent and no correct mechanism. This issue closes that gap by making the shell timeout **overridable per invocation** through a first-class `timeout` tool argument, turning #82's advice into something the agent can actually act on. ## Current behavior 1. `cleveractors.agents.tool.ToolAgent._execute_shell_command` always bounds execution by `self.timeout` (the single static per-agent value); there is no per-call override. 2. `cleveractors.agents.llm_tools._BUILTIN_TOOL_SCHEMAS["shell"]` exposes only `command` and `args` (a list of "Additional command-line arguments"). There is **no** `timeout` parameter the model can set, yet the schema description tells the model to "increase `timeout` in the tool agent config." 3. Because the only writable channel is `args`, a model trying to follow that advice ends up injecting timeout-related tokens into `args`, which are appended verbatim to the shell command string — corrupting the command (see trace above). 4. The timeout `ExecutionError` raised by `_execute_shell_command` directs the reader to edit agent config — an action available to a human config author but not to the LLM at runtime. ## Expected behavior 1. `_BUILTIN_TOOL_SCHEMAS["shell"]` exposes an **optional** `timeout` parameter (a number, seconds). It is NOT added to the schema's `required` list. Its description states it overrides the agent's default timeout for this single invocation and is bounded by a maximum. 2. `ToolAgent._execute_shell_command` uses a per-invocation `timeout` taken from the call arguments when present and valid, in place of `self.timeout`, for the `asyncio.wait_for` bound. 3. When no per-invocation `timeout` is supplied, behavior is byte-for-byte unchanged: the call is bounded by `self.timeout` (default `1`s per §4.5). 4. The per-invocation `timeout` is a **control argument**: it is read distinctly from the positional `args` list and MUST NOT be concatenated into the executed command string. 5. Argument validation runs first (per the project error-handling rules): a supplied `timeout` that is non-numeric, `<= 0`, or above the allowed ceiling raises `ExecutionError` with an explanatory message, before any subprocess is spawned. 6. The shell timeout `ExecutionError` message is rewritten to point the model at the actionable lever — retrying with a larger `timeout` argument (up to the ceiling) — rather than instructing it to edit config it cannot reach at runtime. **Design points to settle in the ADR / review (not yet fixed by this issue):** - The ceiling for a per-call `timeout`. Proposal: a bounded maximum so a model cannot pin a worker indefinitely — either a fixed cap or a new `max_timeout` agent config field. Absent consensus, default to a fixed, documented cap. - Whether the per-call value may only *raise* the bound or may also *lower* it (proposal: either direction, within `(0, ceiling]`). **Out of scope** (mirroring #82's scoping, noted for awareness): `cleveractors.agents.tool.ToolAgent._http_request_tool` shares the same `self.timeout` and has the same limitation. Extending the per-invocation override to `http_request` is a parallel follow-up issue, not addressed here, to keep this change atomic. ## Acceptance criteria - `_BUILTIN_TOOL_SCHEMAS["shell"]["parameters"]["properties"]` contains an optional numeric `timeout` property whose description states it overrides the agent default for this invocation (bounded by the ceiling); `timeout` is absent from that schema's `required` array. - With a valid per-invocation `timeout` supplied, `ToolAgent._execute_shell_command` bounds `asyncio.wait_for` by that value rather than `self.timeout` (verifiable: a call given `timeout` larger than `self.timeout` completes a command that runs longer than `self.timeout`). - With no `timeout` supplied, a command exceeding `self.timeout` still raises the timeout `ExecutionError` — default behavior is unchanged. - A per-invocation `timeout` value never appears in the executed command string (verifiable: a `shell` call with a `timeout` argument runs the intended command, not a corrupted one). - A supplied `timeout` that is non-numeric, `<= 0`, or above the ceiling raises `ExecutionError` before a subprocess is created, with a message naming the accepted range. - The shell timeout `ExecutionError` message instructs retrying with a larger `timeout` argument and no longer instructs editing the agent config. - `nox` (all default sessions) is green and `nox -s coverage_report` stays ≥ 97%. ## Supporting information - Actor Configuration Standard §4.5 (`docs/index.md`) — defines `timeout` as "Maximum execution time in seconds for any single tool invocation." A per-invocation override is an extension of this field's semantics and therefore requires the ADR-before-code process (see Subtasks): a decision record extending `docs/adr/ADR-2030-tool-calling-spec-extensions.md`, then a §4.5 spec update via the standard spec-revision procedure (bump the document Version and add a §21.1 Revision History row attributing the change to the ADR). - `docs/adr/ADR-2030-tool-calling-spec-extensions.md` — D-1 establishes `_BUILTIN_TOOL_SCHEMAS` as LLM-facing hints; D-3 (`file_read` `max_chars`) establishes the precedent that an **optional** tool argument may be added because §4.5.1 "allows additional arguments to be handled." A per-invocation `timeout` follows the same pattern. - Predecessor: #82 surfaced the timeout budget in schemas/messages but explicitly kept enforcement static. This issue makes that surfaced guidance actionable. - Relevant symbols at commit `67972f0`: - `cleveractors.agents.tool.ToolAgent.__init__` (sets `self.timeout = cfg.get("timeout", 1)`) - `cleveractors.agents.tool.ToolAgent._execute_shell_command` (applies `self.timeout`; builds `cmd_str`; raises the timeout `ExecutionError`) - `cleveractors.agents.tool.ToolAgent._execute_tool` (passes the full tool-call `args` dict into `_execute_shell_command`) - `cleveractors.agents.llm_tools._BUILTIN_TOOL_SCHEMAS` (the `shell` schema) ## Subtasks - [ ] ADR: add a decision record extending `ADR-2030` for a per-invocation `shell` `timeout` override (including the ceiling decision); submit for review and get it accepted before writing code - [ ] Spec: update Actor Configuration Standard §4.5 per the spec-revision procedure — bump the document Version and add a §21.1 Revision History row attributing the change to the ADR (do not edit §4.5 prose inline without the version bump + attribution) - [ ] Add the optional `timeout` property to `_BUILTIN_TOOL_SCHEMAS["shell"]` and update its description to describe the override and its bound - [ ] In `ToolAgent._execute_shell_command`, validate and read a per-invocation `timeout` from the tool-call args, use it for the `asyncio.wait_for` bound with fallback to `self.timeout`, and ensure it is never concatenated into `cmd_str` - [ ] Rewrite the shell timeout `ExecutionError` message to direct the model to retry with a larger `timeout` argument (up to the ceiling) - [ ] Tests (Behave): per-call `timeout` honored (a long command succeeds when overridden, fails at the default); absent → default behavior; invalid values rejected before spawn; the `timeout` argument is not injected into the command; updated error-message wording - [ ] Tests (Robot): integration scenario invoking the `shell` tool with a per-invocation `timeout` against a real subprocess - [ ] Benchmarks (ASV): assess whether any perf-sensitive path changed (expected N/A — timeout only bounds wall-clock); document the assessment - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - The ADR extending ADR-2030 is accepted and §4.5 of the standard is updated via the spec-revision procedure before implementation code is merged. - A Git commit is created where the first line matches the Commit Message in Metadata exactly. - The commit is pushed to the branch matching the Branch in Metadata exactly. - The commit is submitted as a PR to master, reviewed, and merged.

CoreRasurae added this to the v2.1.0 milestone

2026-08-04 21:31:20 +00:00

CoreRasurae added the

labels

2026-08-04 21:31:21 +00:00

CoreRasurae commented