LLM Agent Communication Retry Mechanisms with Exponential Backoff #69

New issue

Closed

opened 2026-07-02 17:36:36 +00:00 by CoreRasurae · 0 comments

CoreRasurae commented

2026-07-02 17:36:36 +00:00

Member

Copy link

Description

Introduce configurable retry mechanisms for all LLM agent communications (main agent and pruning agent) to handle transient LLM server failures gracefully.

The retry loop employs exponential backoff (initial 0.5s, doubled each retry) and terminates when either max_retries or max_retry_time is reached, whichever comes first. Setting either value to -1 disables that guard (infinite retries or infinite retry time). If both max_retries and max_retry_time are -1, retries continue indefinitely (strongly discouraged). Counter and accumulated wait time are reset on success. On eventual failure, the library exits with a timeout ExecutionError that includes the target URL and actor graph name.

Applies uniformly to both the main agent and the pruning agent (when allow_tool_output_pruning is active).

Acceptance Criteria

max_retries (default 7) and max_retry_time (default 60s) are configurable per LLM agent.
Setting max_retries = -1 allows infinite retries (bounded only by max_retry_time if finite).
Setting max_retry_time = -1 allows infinite retry time (bounded only by max_retries if finite).
Setting both to -1 creates infinite retries (documentation warns against this).
Exponential backoff starts at 0.5s and doubles each retry.
Retry loop terminates when either max_retries is reached or accumulated wait time >= max_retry_time (whichever comes first), considering -1 as disabled.
Retry counter and accumulated wait time are reset on successful communication.
The same client/connection is reused across retries (no re-establishment).
Timeout raises ExecutionError(kind="timeout") with URL, actor graph name, retry count, and accumulated wait time in the message.
Pruning agent uses the same retry configuration as the main agent.

Metadata

Commit Message: feat(agents): add LLM agent retry mechanisms with exponential backoff
Branch: feature/m2-llm-agent-retry-mechanisms

Subtasks

Implement max_retries and max_retry_time config fields in agent schema, with -1 sentinel for unbounded guards
Implement retry loop with exponential backoff in LLMAgent communication path, handling -1 guards
Implement retry loop with exponential backoff in pruning pass
Implement retry counter and accumulated wait time tracking with reset on success
Implement timeout error reporting (ExecutionError with URL, actor graph name, retry count, accumulated time, and disabled-guard indication)
Tests (Behave): Add BDD scenarios for retry behaviour including -1 sentinel cases
Tests (Robot): Add integration test with mock server simulating transient failures
ADR: docs/adr/ADR-2032-llm-agent-retry-mechanisms.md (included in this branch)
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.

## Description Introduce configurable retry mechanisms for all LLM agent communications (main agent and pruning agent) to handle transient LLM server failures gracefully. The retry loop employs exponential backoff (initial 0.5s, doubled each retry) and terminates when either `max_retries` or `max_retry_time` is reached, whichever comes first. Setting either value to `-1` disables that guard (infinite retries or infinite retry time). If both `max_retries` and `max_retry_time` are `-1`, retries continue indefinitely (strongly discouraged). Counter and accumulated wait time are reset on success. On eventual failure, the library exits with a timeout `ExecutionError` that includes the target URL and actor graph name. Applies uniformly to both the main agent and the pruning agent (when `allow_tool_output_pruning` is active). ## Acceptance Criteria - [ ] `max_retries` (default 7) and `max_retry_time` (default 60s) are configurable per LLM agent. - [ ] Setting `max_retries = -1` allows infinite retries (bounded only by `max_retry_time` if finite). - [ ] Setting `max_retry_time = -1` allows infinite retry time (bounded only by `max_retries` if finite). - [ ] Setting both to `-1` creates infinite retries (documentation warns against this). - [ ] Exponential backoff starts at 0.5s and doubles each retry. - [ ] Retry loop terminates when either `max_retries` is reached or accumulated wait time >= `max_retry_time` (whichever comes first), considering `-1` as disabled. - [ ] Retry counter and accumulated wait time are reset on successful communication. - [ ] The same client/connection is reused across retries (no re-establishment). - [ ] Timeout raises `ExecutionError(kind="timeout")` with URL, actor graph name, retry count, and accumulated wait time in the message. - [ ] Pruning agent uses the same retry configuration as the main agent. ## Metadata - **Commit Message**: `feat(agents): add LLM agent retry mechanisms with exponential backoff` - **Branch**: `feature/m2-llm-agent-retry-mechanisms` ## Subtasks - [ ] Implement `max_retries` and `max_retry_time` config fields in agent schema, with `-1` sentinel for unbounded guards - [ ] Implement retry loop with exponential backoff in `LLMAgent` communication path, handling `-1` guards - [ ] Implement retry loop with exponential backoff in pruning pass - [ ] Implement retry counter and accumulated wait time tracking with reset on success - [ ] Implement timeout error reporting (`ExecutionError` with URL, actor graph name, retry count, accumulated time, and disabled-guard indication) - [ ] Tests (Behave): Add BDD scenarios for retry behaviour including `-1` sentinel cases - [ ] Tests (Robot): Add integration test with mock server simulating transient failures - [ ] ADR: `docs/adr/ADR-2032-llm-agent-retry-mechanisms.md` (included in this branch) - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.

CoreRasurae referenced this issue from a commit

2026-07-02 17:36:54 +00:00

feat(agents): add LLM agent retry mechanisms with exponential backoff

CoreRasurae referenced this issue from a commit

2026-07-02 17:40:41 +00:00

feat(agents): add LLM agent retry mechanisms with exponential backoff

CoreRasurae referenced this issue from a commit

2026-07-02 17:44:08 +00:00

feat(agents): add LLM agent retry mechanisms with exponential backoff

CoreRasurae referenced this issue from a commit

2026-07-02 19:38:53 +00:00

feat(agents): add LLM agent retry mechanisms with exponential backoff