cleveragents/cleveractors-core

Fork 0

ConnectionError retry exhaustion silently swallowed, producing empty response instead of propagating ExecutionError #71

New issue

Closed

opened 2026-07-06 17:21:13 +00:00 by CoreRasurae · 1 comment

CoreRasurae commented

2026-07-06 17:21:13 +00:00

Member

Description

When all LLM communication retries are exhausted (e.g. a server is unreachable and httpx.ConnectError fires on every attempt), the ExecutionError(kind="timeout") raised by call_with_retry() is systematically swallowed instead of propagating to the caller. The graph continues executing subsequent nodes with an error string as message content, and eventually returns either an error string or — in the streaming path — an empty response.

Proposed Branch

fix/nodes-executionerror-propagation

Proposed Commit Title

fix(nodes): propagate ExecutionError from retry mechanism instead of swallowing it

Root Cause

The retry mechanism in call_with_retry() (ADR-2032, src/cleveractors/agents/retry.py) works correctly: it detects httpx.ConnectError, performs exponential backoff (up to 7 retries, 60s max), and raises ExecutionError(kind="timeout") with a detailed error message including provider URL, graph name, and retry count.

However, three layers of overly-broad exception handling systematically catch and neutralize this exception:

Layer 1: `Node._execute_agent()` — `src/cleveractors/langgraph/nodes.py:409-414`

except Exception as e:
    self.logger.error("Agent %s execution failed: %s", agent.name, e)
    agent_response = f"Error processing message: {str(e)}"

Catches ALL exceptions including ExecutionError(kind="timeout"). Converts the error to a string that is injected into graph state as if it were a valid agent response. The retry mechanism's entire work (backoff, budget tracking, error reporting) is wasted.

Layer 2: `Node.execute()` — `src/cleveractors/langgraph/nodes.py:231-257`

except Exception as e:
    return {"error": str(e), "failed_node": self.name}

Secondary catch-all for non-AGENT node types. For TOOL/FUNCTION nodes, returns an error dict that _execute_from_node() silently processes by falling back to the original input message — making the error invisible.

Layer 3: `PureLangGraph._execute_from_node()` — `src/cleveractors/langgraph/pure_graph.py:983-990`

except ExecutionError:
    raise       # intended to propagate — NEVER REACHED

The ExecutionError re-raise at line 983-987 is never reached because Layers 1 and 2 catch all exceptions first.

How the "jump to next node" happens

When _execute_from_node() receives the error dict {"error": ..., "failed_node": ...} from a TOOL/FUNCTION node, at line 814 it checks for the "messages" key (not found) and falls to line 823:

output_message = result.get("content", result.get("output", message))

No "content" or "output" keys in the error dict → falls back to the original message. The graph continues execution with the original input as if nothing happened.

How the "empty response" happens

In the streaming path (_stream_agent at nodes.py:597-610), exceptions ARE properly re-raised, so ExecutionError propagates to _execute_graph_stream() (runtime_dispatch.py:1288-1333):

except ExecutionError:
    executor.last_result = ActorResult(
        response="".join(response_parts),  # EMPTY if error before any tokens
        ...
    )
    raise

When the ConnectionError occurs before any tokens are yielded, response_parts is [] → response="".

Fix Required

A minimal two-change fix:

Change 1: `src/cleveractors/langgraph/nodes.py` — `_execute_agent()` (line 409)

Replace the broad except Exception with a handler that propagates ExecutionError and only catches non-fatal exceptions:

except ExecutionError:
    raise
except Exception as e:
    self.logger.error("Agent %s execution failed: %s", agent.name, e)
    agent_response = f"Error processing message: {str(e)}"

Change 2: `src/cleveractors/langgraph/nodes.py` — `execute()` (line 231)

Same pattern — let ExecutionError propagate to _execute_from_node()'s existing handler at pure_graph.py:983-987:

except ExecutionError:
    raise
except Exception as e:
    self.last_error = e
    self.logger.error("Node %s execution failed: %s", self.name, e)
    ...

Why this is sufficient

The ExecutionError propagation handler at _execute_from_node() line 983-987 already exists and correctly re-raises.
_execute_graph() at runtime_dispatch.py:403-404 already catches ExecutionError and re-raises after billing state capture.
_execute_graph_stream() at runtime_dispatch.py:1288 already handles ExecutionError properly.
Non-ExecutionError agent failures (programming errors, validation errors) are still caught and converted to error strings in graph state, preserving existing behavior.

Affected Files

File	Lines	Role
`src/cleveractors/langgraph/nodes.py`	409-414	`_execute_agent()` — primary swallow point
`src/cleveractors/langgraph/nodes.py`	231-257	`execute()` — secondary swallow point
`src/cleveractors/langgraph/pure_graph.py`	983-990	`_execute_from_node()` — existing (unreachable) `ExecutionError` propagator
`src/cleveractors/agents/retry.py`	172-174	`call_with_retry()` — correctly raises `ExecutionError` after exhausting retries

ADR-2032: LLM Agent Communication Retry Mechanisms (defines the retry contract that is being violated)
ADR-2029: Defines ExecutionError with kind="timeout"

## Description When all LLM communication retries are exhausted (e.g. a server is unreachable and `httpx.ConnectError` fires on every attempt), the `ExecutionError(kind="timeout")` raised by `call_with_retry()` is systematically swallowed instead of propagating to the caller. The graph continues executing subsequent nodes with an error string as message content, and eventually returns either an error string or — in the streaming path — an empty response. ## Proposed Branch `fix/nodes-executionerror-propagation` ## Proposed Commit Title ``` fix(nodes): propagate ExecutionError from retry mechanism instead of swallowing it ``` ## Root Cause The retry mechanism in `call_with_retry()` (ADR-2032, `src/cleveractors/agents/retry.py`) works correctly: it detects `httpx.ConnectError`, performs exponential backoff (up to 7 retries, 60s max), and raises `ExecutionError(kind="timeout")` with a detailed error message including provider URL, graph name, and retry count. However, three layers of overly-broad exception handling systematically catch and neutralize this exception: ### Layer 1: `Node._execute_agent()` — `src/cleveractors/langgraph/nodes.py:409-414` ```python except Exception as e: self.logger.error("Agent %s execution failed: %s", agent.name, e) agent_response = f"Error processing message: {str(e)}" ``` Catches **ALL** exceptions including `ExecutionError(kind="timeout")`. Converts the error to a string that is injected into graph state as if it were a valid agent response. The retry mechanism's entire work (backoff, budget tracking, error reporting) is wasted. ### Layer 2: `Node.execute()` — `src/cleveractors/langgraph/nodes.py:231-257` ```python except Exception as e: return {"error": str(e), "failed_node": self.name} ``` Secondary catch-all for non-AGENT node types. For TOOL/FUNCTION nodes, returns an error dict that `_execute_from_node()` silently processes by falling back to the original input message — making the error invisible. ### Layer 3: `PureLangGraph._execute_from_node()` — `src/cleveractors/langgraph/pure_graph.py:983-990` ```python except ExecutionError: raise # intended to propagate — NEVER REACHED ``` The `ExecutionError` re-raise at line 983-987 is **never reached** because Layers 1 and 2 catch all exceptions first. ## How the "jump to next node" happens When `_execute_from_node()` receives the error dict `{"error": ..., "failed_node": ...}` from a TOOL/FUNCTION node, at line 814 it checks for the `"messages"` key (not found) and falls to line 823: ```python output_message = result.get("content", result.get("output", message)) ``` No `"content"` or `"output"` keys in the error dict → falls back to the original `message`. The graph continues execution with the original input as if nothing happened. ## How the "empty response" happens In the **streaming path** (`_stream_agent` at `nodes.py:597-610`), exceptions ARE properly re-raised, so `ExecutionError` propagates to `_execute_graph_stream()` (`runtime_dispatch.py:1288-1333`): ```python except ExecutionError: executor.last_result = ActorResult( response="".join(response_parts), # EMPTY if error before any tokens ... ) raise ``` When the ConnectionError occurs before any tokens are yielded, `response_parts` is `[]` → `response=""`. ## Fix Required A minimal two-change fix: ### Change 1: `src/cleveractors/langgraph/nodes.py` — `_execute_agent()` (line 409) Replace the broad `except Exception` with a handler that propagates `ExecutionError` and only catches non-fatal exceptions: ```python except ExecutionError: raise except Exception as e: self.logger.error("Agent %s execution failed: %s", agent.name, e) agent_response = f"Error processing message: {str(e)}" ``` ### Change 2: `src/cleveractors/langgraph/nodes.py` — `execute()` (line 231) Same pattern — let `ExecutionError` propagate to `_execute_from_node()`'s existing handler at `pure_graph.py:983-987`: ```python except ExecutionError: raise except Exception as e: self.last_error = e self.logger.error("Node %s execution failed: %s", self.name, e) ... ``` ### Why this is sufficient - The `ExecutionError` propagation handler at `_execute_from_node()` line 983-987 already exists and correctly re-raises. - `_execute_graph()` at `runtime_dispatch.py:403-404` already catches `ExecutionError` and re-raises after billing state capture. - `_execute_graph_stream()` at `runtime_dispatch.py:1288` already handles `ExecutionError` properly. - Non-ExecutionError agent failures (programming errors, validation errors) are still caught and converted to error strings in graph state, preserving existing behavior. ## Affected Files | File | Lines | Role | |------|-------|------| | `src/cleveractors/langgraph/nodes.py` | 409-414 | `_execute_agent()` — primary swallow point | | `src/cleveractors/langgraph/nodes.py` | 231-257 | `execute()` — secondary swallow point | | `src/cleveractors/langgraph/pure_graph.py` | 983-990 | `_execute_from_node()` — existing (unreachable) `ExecutionError` propagator | | `src/cleveractors/agents/retry.py` | 172-174 | `call_with_retry()` — correctly raises `ExecutionError` after exhausting retries | ## Related ADRs - ADR-2032: LLM Agent Communication Retry Mechanisms (defines the retry contract that is being violated) - ADR-2029: Defines `ExecutionError` with `kind="timeout"`

CoreRasurae added the

labels

2026-07-06 17:30:17 +00:00

CoreRasurae referenced this issue from a commit

2026-07-06 17:54:54 +00:00

fix(nodes): propagate ExecutionError from retry mechanism instead of swallowing it

CoreRasurae referenced this issue from a pull request that will close it,

2026-07-06 17:55:00 +00:00

fix(nodes): propagate ExecutionError from retry mechanism instead of swallowing it #72

CoreRasurae added a new dependency

2026-07-06 17:55:51 +00:00

#72 fix(nodes): propagate ExecutionError from retry mechanism instead of swallowing it

CoreRasurae referenced this issue from a commit

2026-07-07 09:55:04 +00:00

fix(nodes): propagate ExecutionError from retry mechanism instead of swallowing it

Graa referenced this issue

2026-07-07 10:20:52 +00:00

Epic: LLM Agent Runtime Stabilization — reliability, resource enforcement & correctness hardening #77

Graa added a new dependency

2026-07-07 10:21:22 +00:00

#77 Epic: LLM Agent Runtime Stabilization — reliability, resource enforcement & correctness hardening

Graa referenced this issue

2026-07-07 11:45:02 +00:00

Invert default exception handling: stop swallowing by default; catch only recoverable exceptions and propagate the rest #78

CoreRasurae commented

2026-07-07 12:02:29 +00:00

Author

Member

Update: APIConnectionError not handled by retry mechanism

The initial fix for this issue addressed ExecutionError(kind="timeout") propagation (from httpx retry exhaustion). However, provider SDK connection errors (e.g. openai.APIConnectionError, anthropic.APIConnectionError) are NOT caught by _is_http_comms_error() because they inherit from Exception, not httpx.HTTPError. This means they are never retried — they propagate directly to process_message() which wraps them as ExecutionError(kind=""), and the node layer silently swallows them.

Root cause

_is_http_comms_error() in retry.py only checks isinstance(e, httpx.xxx) — provider SDKs wrap the underlying httpx transport error into their own exception classes (openai.APIConnectionError, anthropic.APIConnectionError) which are not in the httpx hierarchy.

Fix applied

retry.py: _is_http_comms_error() now checks the exception class name for APIConnectionError and APITimeoutError — these are provider-agnostic names used by all major SDKs (openai, anthropic). They are treated as transient connection failures and retried with exponential backoff. On exhaustion, ExecutionError(kind="timeout") is raised correctly.
llm.py: Safety net in process_message() — if a provider connection error slips through without being retried (e.g. max_retries=0), it is wrapped as ExecutionError(kind="timeout") instead of kind="", ensuring propagation through the node layer.

Files changed

src/cleveractors/agents/retry.py — _is_http_comms_error() extended
src/cleveractors/agents/llm.py — safety net in process_message()
src/cleveractors/langgraph/nodes.py — guard changed from e.kind == "timeout" to if e.kind (per non-blocking review suggestion)
features/nodes_coverage_gaps.feature — 3 new scenarios
features/steps/nodes_coverage_gaps_steps.py — step definitions for new scenarios
CHANGELOG.md — entry updated

## Update: APIConnectionError not handled by retry mechanism The initial fix for this issue addressed `ExecutionError(kind="timeout")` propagation (from httpx retry exhaustion). However, provider SDK connection errors (e.g. `openai.APIConnectionError`, `anthropic.APIConnectionError`) are NOT caught by `_is_http_comms_error()` because they inherit from `Exception`, not `httpx.HTTPError`. This means they are never retried — they propagate directly to `process_message()` which wraps them as `ExecutionError(kind="")`, and the node layer silently swallows them. ### Root cause `_is_http_comms_error()` in `retry.py` only checks `isinstance(e, httpx.xxx)` — provider SDKs wrap the underlying httpx transport error into their own exception classes (`openai.APIConnectionError`, `anthropic.APIConnectionError`) which are not in the httpx hierarchy. ### Fix applied 1. **`retry.py`**: `_is_http_comms_error()` now checks the exception class name for `APIConnectionError` and `APITimeoutError` — these are provider-agnostic names used by all major SDKs (openai, anthropic). They are treated as transient connection failures and retried with exponential backoff. On exhaustion, `ExecutionError(kind="timeout")` is raised correctly. 2. **`llm.py`**: Safety net in `process_message()` — if a provider connection error slips through without being retried (e.g. max_retries=0), it is wrapped as `ExecutionError(kind="timeout")` instead of `kind=""`, ensuring propagation through the node layer. ### Files changed - `src/cleveractors/agents/retry.py` — `_is_http_comms_error()` extended - `src/cleveractors/agents/llm.py` — safety net in `process_message()` - `src/cleveractors/langgraph/nodes.py` — guard changed from `e.kind == "timeout"` to `if e.kind` (per non-blocking review suggestion) - `features/nodes_coverage_gaps.feature` — 3 new scenarios - `features/steps/nodes_coverage_gaps_steps.py` — step definitions for new scenarios - `CHANGELOG.md` — entry updated

CoreRasurae referenced this issue from a commit

2026-07-07 12:02:53 +00:00

fix(retry,llm): handle provider APIConnectionError as retryable transient failure

CoreRasurae referenced this issue from a commit

2026-07-07 12:48:24 +00:00

fix(nodes): propagate ExecutionError from retry mechanism instead of swallowing it

CoreRasurae closed this issue

2026-07-07 16:06:09 +00:00