feat(execution-limits): add structured ExecutionError kind/reason fields; enforce all 5 execution limits in PureLangGraph #44

Merged
hurui200320 merged 1 commits from feat/execution-limits into master 2026-06-11 11:05:30 +00:00
9 changed files with 2077 additions and 54 deletions
+1
View File
@@ -15,6 +15,7 @@ and this project adheres to [Clever Semantic Versioning](https://www.w3.org/subm
### Added
- **Structured `ExecutionError` fields + 5-limit enforcement in `PureLangGraph`** (`cleveractors.core.exceptions.ExecutionError`, `cleveractors.langgraph.pure_graph.PureLangGraph`): `ExecutionError` gains `kind` (categorical: `depth`, `model_calls`, `tool_calls`, `timeout`, `cost`) and `reason` (sub-code: `budget_exhausted` or `missing_pricing_entry`) fields, both defaulting to `""` for backward compatibility with existing `raise ExecutionError(msg)` call sites. `PureLangGraph` now accepts `limits` and `pricing` constructor arguments. When `limits["max_depth"]` is supplied, a depth breach raises `ExecutionError(kind="depth")` instead of silently returning the current message. `max_model_calls` is checked before each AGENT-type node; `max_tool_calls` before each TOOL-type node. `execute()` wraps `_execute_from_node()` in `asyncio.wait_for()` when `limits["timeout_ms"]` is set, mapping `asyncio.TimeoutError` to `ExecutionError(kind="timeout")`. After each LLM node, cost is computed from the supplied `pricing` table (rates are USD per million tokens per ADR-2029); breach raises `ExecutionError(kind="cost", reason="budget_exhausted")`; a missing provider or model entry raises `ExecutionError(kind="cost", reason="missing_pricing_entry")`. `runtime_dispatch._execute_graph()` now passes `executor.limits` and `executor.pricing` to `PureLangGraph`. `ExecutionError` is exported from `cleveractors.__init__` and `__all__`. (ADR-2029, issue #15)
- **`create_executor()` router-facing API** (`cleveractors.create_executor`): New module-level factory function that constructs an `Executor` wrapping `PureLangGraph` and `AgentFactory`. Accepts `config_dict` (validated actor configuration), `credentials` (per-provider credential dict for per-request injection), `limits` (execution budget), and `pricing` (per-model cost table). `executor.execute(message)` runs the actor graph and returns an `ActorResult(response, prompt_tokens, completion_tokens, nodes)`. All four execution paths are supported — `llm`, `graph`, `tool`, and `multi_actor`. Credentials are passed to `AgentFactory` and never injected into the stored `config_dict` (ADR-2026 AC8). Exported from `cleveractors.__init__` and `__all__` (ADR-2024, ADR-2026, ADR-2029).
- **`ActorResult` and `NodeUsage` types** (`cleveractors.result`): Canonical dataclasses for the router-facing result API, now defined in `cleveractors.result` (ADR-2027). `ActorResult` carries the response string, aggregated `prompt_tokens`/`completion_tokens`, a non-empty `nodes: list[NodeUsage]` breakdown for per-model billing, and an optional opaque `state` blob for stateless graph resumption (ADR-2026). Re-exported from `cleveractors.runtime` for backward compatibility.
- **Real LangChain token extraction** (`LLMAgent`): `process_message()` now reads token counts from `response.usage_metadata` (primary path) with fallback to `response.response_metadata["token_usage"]`. Both paths are guarded with `isinstance(dict)` checks to prevent `AttributeError` on truthy non-dict provider values. Non-numeric token values are coerced via `_safe_int()` with fallback to 0 and a warning log. A warning distinguishing the failure cause (empty `usage_metadata`, missing `response_metadata`, or missing `token_usage` key) is emitted when no usage data is available.
+226
View File
@@ -0,0 +1,226 @@
Feature: Execution Limits Enforcement in PureLangGraph (ADR-2029)
As the CleverThis router
I want PureLangGraph to enforce per-request execution limits
So that budget exhaustion, depth breaches, and cost violations surface as
structured ExecutionError exceptions with the correct kind/reason fields
Background:
Given the execution limits test context is initialised (elim)
# ── ExecutionError structured fields ──────────────────────────────────────
Scenario: ExecutionError defaults kind and reason to empty strings
When an ExecutionError is raised with only a message (elim)
Then the kind field should be empty string (elim)
And the reason field should be empty string (elim)
Scenario: ExecutionError accepts kind with empty reason
When an ExecutionError is raised with kind "depth" and empty reason (elim)
Then the kind field should be "depth" (elim)
And the reason field should be empty string (elim)
Scenario: ExecutionError accepts kind cost with reason budget_exhausted
When an ExecutionError is raised with kind "cost" and reason "budget_exhausted" (elim)
Then the kind field should be "cost" (elim)
And the reason field should be "budget_exhausted" (elim)
Scenario: ExecutionError accepts kind cost with reason missing_pricing_entry
When an ExecutionError is raised with kind "cost" and reason "missing_pricing_entry" (elim)
Then the kind field should be "cost" (elim)
And the reason field should be "missing_pricing_entry" (elim)
Scenario: ExecutionError is exported from cleveractors top-level package
When ExecutionError is imported from cleveractors (elim)
Then the import should succeed and expose kind and reason attributes (elim)
# ── Depth limit ────────────────────────────────────────────────────────────
Scenario: Graph exceeding max_depth raises ExecutionError with kind "depth"
Given a linear 3-node graph with max_depth 1 (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "depth" (elim)
Scenario: Graph within max_depth completes successfully
Given a linear 2-node graph with max_depth 10 (elim)
When the graph is executed with a test message (elim)
Then no ExecutionError should be raised (elim)
# ── Model-call limit ───────────────────────────────────────────────────────
Scenario: Two AGENT nodes with max_model_calls 1 raises on second call
Given a graph with 2 AGENT nodes and max_model_calls 1 (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "model_calls" (elim)
Scenario: Two AGENT nodes with max_model_calls 2 completes successfully
Given a graph with 2 AGENT nodes and max_model_calls 2 (elim)
When the graph is executed with a test message (elim)
Then no ExecutionError should be raised (elim)
# ── Tool-call limit ────────────────────────────────────────────────────────
Scenario: Two TOOL nodes with max_tool_calls 1 raises on second call
Given a graph with 2 TOOL nodes and max_tool_calls 1 (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "tool_calls" (elim)
Scenario: Two TOOL nodes with max_tool_calls 2 completes successfully
Given a graph with 2 TOOL nodes and max_tool_calls 2 (elim)
When the graph is executed with a test message (elim)
Then no ExecutionError should be raised (elim)
# ── Timeout ───────────────────────────────────────────────────────────────
Scenario: Graph with slow node times out and raises ExecutionError kind "timeout"
Given a graph with a slow node and timeout_ms 50 (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "timeout" (elim)
Scenario: Graph completing before timeout succeeds
Given a graph with a fast node and timeout_ms 2000 (elim)
When the graph is executed with a test message (elim)
Then no ExecutionError should be raised (elim)
# ── Cost limit — budget_exhausted ─────────────────────────────────────────
Scenario: LLM node with high token count exceeding cost limit raises budget_exhausted
Given a graph with an LLM node using 2000000 prompt tokens and max_cost_usd 0.10 (elim)
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
Then an ExecutionError should be raised with kind "cost" (elim)
And the reason should be "budget_exhausted" (elim)
Scenario: LLM node with low token count within cost limit completes successfully
Given a graph with an LLM node using 100 prompt tokens and max_cost_usd 1.00 (elim)
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
Then no ExecutionError should be raised (elim)
# ── Cost limit — missing_pricing_entry ────────────────────────────────────
Scenario: LLM node with unknown provider raises missing_pricing_entry
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
When the graph is executed with pricing only for "anthropic" provider (elim)
Then an ExecutionError should be raised with kind "cost" (elim)
And the reason should be "missing_pricing_entry" (elim)
Scenario: LLM node with unknown model raises missing_pricing_entry
Given a graph with an LLM node for provider "openai" model "unknown-model" (elim)
When the graph is executed with pricing for "openai" but not "unknown-model" (elim)
Then an ExecutionError should be raised with kind "cost" (elim)
And the reason should be "missing_pricing_entry" (elim)
Scenario: Empty pricing table skips cost calculation entirely
Given a graph with an LLM node using 2000000 prompt tokens and max_cost_usd 0.001 (elim)
When the graph is executed with an empty pricing table (elim)
Then no ExecutionError should be raised (elim)
Scenario: LLM node with incomplete model pricing entry missing prompt rate raises missing_pricing_entry
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
When the graph is executed with model pricing entry missing the prompt rate (elim)
Then an ExecutionError should be raised with kind "cost" (elim)
And the reason should be "missing_pricing_entry" (elim)
Scenario: LLM node with non-numeric pricing rate raises missing_pricing_entry
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
When the graph is executed with a non-numeric prompt rate (elim)
Then an ExecutionError should be raised with kind "cost" (elim)
And the reason should be "missing_pricing_entry" (elim)
# ── Cost accumulation across multiple LLM nodes ───────────────────────────
Scenario: Cost accumulates across two LLM nodes and triggers budget_exhausted on second
Given a graph with 2 LLM nodes each using 1000000 prompt tokens and max_cost_usd 0.25 (elim)
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
Then an ExecutionError should be raised with kind "cost" (elim)
And the reason should be "budget_exhausted" (elim)
# ── Completion tokens cost contribution ───────────────────────────────────
Scenario: Completion tokens contribute to cost and can trigger budget_exhausted
Given a graph with an LLM node using 0 prompt tokens 1000000 completion tokens and max_cost_usd 0.50 (elim)
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt 0.60 completion rate (elim)
Then an ExecutionError should be raised with kind "cost" (elim)
And the reason should be "budget_exhausted" (elim)
# ── Malformed limit value validation ─────────────────────────────────────
Scenario: Non-numeric max_cost_usd raises ExecutionError with kind "cost"
Given a graph with an LLM node for non-numeric max_cost_usd validation (elim)
When the graph is executed with pricing and non-numeric max_cost_usd (elim)
Then an ExecutionError should be raised with kind "cost" (elim)
Scenario: Non-numeric max_model_calls raises ExecutionError with kind "model_calls"
Given a graph with 1 AGENT node and non-numeric max_model_calls (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "model_calls" (elim)
Scenario: Bool max_model_calls raises ExecutionError with kind "model_calls"
Given a graph with 1 AGENT node and bool False max_model_calls (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "model_calls" (elim)
Scenario: Non-numeric max_tool_calls raises ExecutionError with kind "tool_calls"
Given a graph with 1 TOOL node and non-numeric max_tool_calls (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "tool_calls" (elim)
Scenario: Bool max_tool_calls raises ExecutionError with kind "tool_calls"
Given a graph with 1 TOOL node and bool False max_tool_calls (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "tool_calls" (elim)
Scenario: Non-positive timeout_ms raises ExecutionError with kind "timeout"
Given a graph with a fast node and timeout_ms 0 (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "timeout" (elim)
Scenario: Bool timeout_ms raises ExecutionError with kind "timeout"
Given a graph with a fast node and bool True timeout_ms (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "timeout" (elim)
Scenario: Non-numeric max_depth raises ExecutionError with kind "depth"
Given a graph with a fast node and non-numeric max_depth (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "depth" (elim)
Scenario: Bool max_depth raises ExecutionError with kind "depth"
Given a graph with a fast node and bool False max_depth (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "depth" (elim)
Scenario: Bool max_cost_usd raises ExecutionError with kind "cost"
Given a graph with an LLM node for bool max_cost_usd validation (elim)
When the graph is executed with pricing and bool False max_cost_usd (elim)
Then an ExecutionError should be raised with kind "cost" (elim)
Scenario: Parallel graph cancels sibling tasks when one branch raises ExecutionError
Given a parallel graph where one branch raises an ExecutionError (elim)
When the graph is executed with a test message (elim)
Then an ExecutionError should be raised with kind "model_calls" (elim)
And the sibling branch should have been cancelled (elim)
Scenario: Non-dict _node_token_usage logs warning and skips token accounting
Given a graph with an LLM node that returns non-dict token usage (elim)
When the graph is executed with a test message (elim)
Then no ExecutionError should be raised (elim)
And a warning should have been logged for non-dict token usage (elim)
# ── Happy-path result correctness ─────────────────────────────────────────
Scenario: Graph within max_depth returns non-None result
Given a linear 2-node graph with max_depth 10 (elim)
When the graph is executed with a test message (elim)
Then no ExecutionError should be raised (elim)
And the result should be non-None (elim)
# ── PureLangGraph limits/pricing wired from Executor ──────────────────────
Scenario: limits and pricing are stored on Executor and propagated to PureLangGraph
Given an Executor with graph config and limits max_depth 100 pricing for openai (elim)
When the executor is checked for limit/pricing propagation (elim)
Then the PureLangGraph should have the correct limits and pricing (elim)
Scenario: Executor.execute() enforces max_depth through the full dispatch path (elim)
Given an Executor with a 2-node graph and max_depth 0 (elim)
When the executor execute method is called with a test message (elim)
Then an ExecutionError should be raised with kind "depth" (elim)
File diff suppressed because it is too large Load Diff
@@ -186,7 +186,12 @@ def step_pgg_nested_auto_finish_cycle(context):
Edge(source="writer", target="writer"),
],
)
graph = PureLangGraph(config)
# M-2 fix: pass an explicit max_depth so the graph terminates when the
# depth limit is reached rather than recursing to Python's stack limit.
# The old legacy heuristic (max(2000, len(nodes)*50)) previously acted as
# a safety net; now that the default is 2**31-1 per ADR-2029, tests that
# rely on cycle-via-auto_finish_active must supply their own limit.
graph = PureLangGraph(config, limits={"max_depth": 50})
# node that keeps returning same message (simulate cycle)
mock_node = MagicMock()
@@ -211,9 +216,21 @@ def step_pgg_nested_auto_finish_cycle(context):
state = graph.state_manager.get_state()
state.metadata["context"] = {}
result = _run(graph.execute("write section"))
# M-2 fix: the graph now raises ExecutionError(kind="depth") when the
# depth limit is reached (instead of silently returning the message via
# the old heuristic). The test only cares that auto_finish_active allowed
# the cycle to continue past the first visit — the depth-limit termination
# is the expected way the graph stops in this scenario.
from cleveractors.core.exceptions import ExecutionError
try:
result = _run(graph.execute("write section"))
context.pgg_results["graph_completed"] = result is not None
except ExecutionError as exc:
# Depth limit reached — graph terminated via ExecutionError, which is
# the correct ADR-2029 behaviour when max_depth is set.
context.pgg_results["graph_completed"] = exc.kind == "depth"
context.pgg_results["nested_af_continued"] = visit_count[0] >= 2
context.pgg_results["graph_completed"] = result is not None
# ---- Ping-pong detection with auto_finish_active in nested context ----
+86 -7
View File
@@ -2,6 +2,7 @@
import asyncio
from pathlib import Path
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
from behave import given, then, when
@@ -521,6 +522,7 @@ def step_cycle_detection(context):
@when("I execute a graph with auto_finish_active and a self-loop (pg_cov)")
def step_auto_finish_bypass(context):
from cleveractors.core.exceptions import ExecutionError
from cleveractors.langgraph.nodes import Edge, NodeConfig, NodeType
from cleveractors.langgraph.pure_graph import PureGraphConfig, PureLangGraph
@@ -535,13 +537,41 @@ def step_auto_finish_bypass(context):
Edge(source="looper", target="end"),
],
)
graph = PureLangGraph(config)
# M-2 fix: pass an explicit max_depth so the graph terminates via
# ExecutionError(kind="depth") rather than recursing to Python's stack
# limit. The test only cares that auto_finish_active bypassed cycle
# detection (allowing the loop to continue past the first visit).
graph = PureLangGraph(config, limits={"max_depth": 50})
state = graph.state_manager.get_state()
state.metadata["auto_finish_active"] = True
# Issue #6 fix: track visit count to verify that auto_finish_active
# actually allowed the loop to continue past the first visit. A
# regression that disabled the bypass would stop at visit 1, leaving
# visit_count[0] == 1 and failing the assertion in the Then step.
visit_count = [0]
looper_mock = MagicMock()
looper_mock.config = MagicMock()
looper_mock.config.type = NodeType.FUNCTION
looper_mock.config.name = "looper"
async def _looper_execute(state: Any) -> dict:
visit_count[0] += 1
return {"content": "continue please"}
looper_mock.execute = AsyncMock(side_effect=_looper_execute)
graph.nodes["looper"] = looper_mock
context.results["auto_finish_visit_count"] = visit_count
async def _run():
result = await graph.execute("continue please")
context.results["auto_finish_result"] = result
try:
result = await graph.execute("continue please")
context.results["auto_finish_result"] = result
except ExecutionError as exc:
# Depth limit reached — the loop ran until max_depth, which is
# the expected termination path when auto_finish_active bypasses
# cycle detection.
context.results["auto_finish_result"] = f"depth_limit_reached: {exc.kind}"
_run_async(_run(), context=context)
@@ -586,6 +616,7 @@ def step_ping_pong_detection(context):
@when("I execute a graph with auto_finish_active causing ping-pong pattern (pg_cov)")
def step_ping_pong_bypass(context):
from cleveractors.core.exceptions import ExecutionError
from cleveractors.langgraph.nodes import Edge, NodeConfig, NodeType
from cleveractors.langgraph.pure_graph import PureGraphConfig, PureLangGraph
@@ -610,16 +641,42 @@ def step_ping_pong_bypass(context):
Edge(source="router_node", target="end"),
],
)
# Issue #6 fix: track agent visit count to verify that auto_finish_active
# actually allowed the ping-pong to continue past the first cycle.
# The agent returns "GOTO_agent:continue" so the "no routing command" early
# exit does not fire — the loop continues until the depth limit is reached.
visit_count = [0]
mock_agent = MagicMock()
mock_agent.name = "mock_agent"
mock_agent.process_message = AsyncMock(return_value="hello from agent")
graph = PureLangGraph(config, agents={"mock_agent": mock_agent})
async def _agent_process(message: Any, context: Any = None) -> str:
visit_count[0] += 1
# Return a routing command so the graph does not stop at the
# "no routing command" early-exit check (lines 982-1008 in pure_graph.py).
# This ensures the ping-pong detection path is actually exercised.
return "GOTO_agent:continue"
mock_agent.process_message = AsyncMock(side_effect=_agent_process)
# M-2 fix: pass an explicit max_depth so the graph terminates via
# ExecutionError(kind="depth") rather than recursing to Python's stack
# limit. The test only cares that auto_finish_active bypassed ping-pong
# detection (allowing the loop to continue).
graph = PureLangGraph(
config, agents={"mock_agent": mock_agent}, limits={"max_depth": 50}
)
state = graph.state_manager.get_state()
state.metadata["auto_finish_active"] = True
context.results["bypass_visit_count"] = visit_count
async def _run():
result = await graph.execute("hello")
context.results["bypass_result"] = result
try:
result = await graph.execute("hello")
context.results["bypass_result"] = result
except ExecutionError as exc:
# Depth limit reached — the ping-pong ran until max_depth, which
# is the expected termination path when auto_finish_active bypasses
# ping-pong detection.
context.results["bypass_result"] = f"depth_limit_reached: {exc.kind}"
_run_async(_run(), context=context)
@@ -1272,6 +1329,17 @@ def step_verify_cycle_bypassed(context):
assert context.results.get("auto_finish_result") is not None, (
"auto_finish execution completed"
)
# Issue #6 fix: verify that the looper node was visited at least twice,
# confirming that auto_finish_active bypassed cycle detection and allowed
# the loop to continue past the first visit. A regression that disabled
# the bypass would stop at visit 1.
visit_count = context.results.get("auto_finish_visit_count")
if visit_count is not None:
assert visit_count[0] >= 2, (
f"Expected looper to be visited >= 2 times (auto_finish_active bypass), "
f"but was only visited {visit_count[0]} time(s). "
"The auto_finish_active cycle-detection bypass may have been removed."
)
@then("the ping-pong should be detected and current message returned (pg_cov)")
@@ -1284,6 +1352,17 @@ def step_verify_ping_pong_stopped(context):
@then("execution should continue past ping-pong detection (pg_cov)")
def step_verify_ping_pong_bypassed(context):
assert context.results.get("bypass_result") is not None, "ping-pong bypassed"
# Issue #6 fix: verify that the agent was visited at least twice,
# confirming that auto_finish_active bypassed ping-pong detection and
# allowed the loop to continue past the first cycle. A regression that
# disabled the bypass would stop at visit 1.
visit_count = context.results.get("bypass_visit_count")
if visit_count is not None:
assert visit_count[0] >= 2, (
f"Expected agent to be visited >= 2 times (auto_finish_active bypass), "
f"but was only visited {visit_count[0]} time(s). "
"The auto_finish_active ping-pong bypass may have been removed."
)
@then("it should route to the next node after start (pg_cov)")
+6 -1
View File
@@ -13,7 +13,11 @@ from cleveractors.agent import Agent
from cleveractors.config_utils import merge_configs
from cleveractors.context_manager import ContextManager
from cleveractors.core.application import ReactiveCleverAgentsApp
from cleveractors.core.exceptions import CleverAgentsException, ConfigurationError
from cleveractors.core.exceptions import (
CleverAgentsException,
ConfigurationError,
ExecutionError,
)
from cleveractors.result import ActorResult, NodeUsage
from cleveractors.runtime import (
Executor,
@@ -29,6 +33,7 @@ __all__ = [
"ConfigurationError",
"ContextManager",
"create_executor",
"ExecutionError",
"Executor",
"merge_configs",
"NodeUsage",
+24 -1
View File
@@ -35,7 +35,30 @@ class TemplateError(CleverAgentsException):
class ExecutionError(CleverAgentsException):
"""Exception raised when agent execution fails."""
"""Exception raised when agent execution fails.
Attributes:
kind: Categorical limit type that was breached. One of
``'depth'``, ``'model_calls'``, ``'tool_calls'``, ``'timeout'``,
``'cost'``, or ``''`` (empty string for non-limit errors).
reason: Disambiguating sub-code for ``kind='cost'``. One of
``'budget_exhausted'`` (accumulated cost exceeded
``max_cost_usd``), ``'missing_pricing_entry'`` (the pricing
table supplied to ``create_executor()`` does not contain an
entry for the provider/model used by this execution), or ``''``
(empty string for other error kinds and non-limit errors).
All existing ``raise ExecutionError(msg)`` call sites continue to work
because both ``kind`` and ``reason`` default to ``""``.
**ADR reference:** ADR-2029 Actor Execution Limits and Budget
Enforcement via cleveractors-core.
"""
def __init__(self, message: str, kind: str = "", reason: str = "") -> None:
super().__init__(message)
self.kind = kind
self.reason = reason
class ApplicationError(CleverAgentsException):
+369 -41
View File
@@ -8,6 +8,7 @@ allowing it to work properly in run mode without timeouts.
from __future__ import annotations
import asyncio
import copy
import logging
from collections import defaultdict, deque
from dataclasses import dataclass, field
@@ -16,6 +17,7 @@ from typing import Any, Deque, List, Optional, Set
from cleveractors.agents.base import Agent
from cleveractors.context_manager import ContextManager
from cleveractors.core.exceptions import ExecutionError
from cleveractors.langgraph.dynamic_router import DynamicRouterNode
from cleveractors.langgraph.nodes import Edge, Node, NodeConfig, NodeType
from cleveractors.langgraph.state import GraphState, StateManager
@@ -102,8 +104,29 @@ class PureLangGraph:
config: PureGraphConfig,
agents: Optional[dict[str, Agent]] = None,
context_manager: Optional[ContextManager] = None,
limits: dict[str, Any] | None = None,
pricing: dict[str, Any] | None = None,
):
"""Initialize PureLangGraph."""
"""Initialize PureLangGraph.
Args:
config: Graph configuration.
agents: Pre-created agent instances keyed by name.
context_manager: Optional context manager for global state.
limits: Execution budget dict from the router. Recognised keys:
``max_depth``, ``max_model_calls``, ``max_tool_calls``,
``timeout_ms``, ``max_cost_usd``. Any absent key means
"no enforcement for that dimension". When ``max_depth`` is
absent the library uses ``2**31 - 1`` as the internal ceiling
(ADR-2029 default; no real graph reaches this depth).
pricing: Pricing table supplied by the router. Structure:
``{provider: {model: {"prompt": <rate>, "completion": <rate>}}}``
where rates are USD per million tokens. An empty dict (or
``None``) disables cost calculation. When non-empty, every
LLM node invocation MUST have a matching entry a missing
entry raises ``ExecutionError(kind='cost',
reason='missing_pricing_entry')``. (ADR-2029)
"""
logger = logging.getLogger(__name__)
logger.debug(f"PureLangGraph.__init__ called with config type: {type(config)}")
@@ -120,6 +143,18 @@ class PureLangGraph:
self.context_manager = context_manager
self.logger = logging.getLogger(__name__)
# Execution limits and pricing table (ADR-2029, issue #15).
# M-3 fix: store defensive copies so that caller mutation after
# construction (e.g. updating pricing mid-flight or sharing an Executor
# across concurrent requests) cannot silently change enforcement
# behaviour. Both dicts are now security-sensitive (they control
# billing enforcement) and warrant the same treatment as
# runtime_dispatch.py already applies to executor.config.
self._limits: dict[str, Any] = dict(limits) if limits is not None else {}
self._pricing: dict[str, Any] = (
copy.deepcopy(pricing) if pricing is not None else {}
)
# Initialize nodes
self.nodes: dict[str, Node] = {}
self._initialize_nodes()
@@ -151,6 +186,16 @@ class PureLangGraph:
# (node_id, provider, model, prompt_tokens, completion_tokens).
self._node_usages: list[NodeUsageTuple] = []
# Per-execution counters for limit enforcement (issue #15).
# n-5 fix: initialise here only to satisfy the type-checker; the
# authoritative reset happens at the top of execute() before each
# invocation. The __init__ values are never read before execute()
# resets them (matching the pattern used for _node_usages,
# _execution_path, and _node_message_visits).
self._model_call_count: int = 0
self._tool_call_count: int = 0
self._accumulated_cost: float = 0.0
def _initialize_nodes(self) -> None:
"""Initialize all nodes in the graph."""
# Add start and end nodes if not present
@@ -327,6 +372,11 @@ class PureLangGraph:
self._execution_path = []
self._node_usages = []
# Reset per-execution limit counters (issue #15).
self._model_call_count = 0
self._tool_call_count = 0
self._accumulated_cost = 0.0
try:
# Determine the starting context for this execution
if global_context is not None:
@@ -373,10 +423,56 @@ class PureLangGraph:
}
self.state_manager.update_state(init_state_payload, node_id="input")
# Start execution from entry point
output = await self._execute_from_node(
self.config.entry_point, input_message
)
# Start execution from entry point.
# Wrap in asyncio.wait_for() when timeout_ms is specified (ADR-2029,
# issue #15). asyncio.TimeoutError is caught and re-raised as a
# structured ExecutionError so the router can map it to HTTP 429.
timeout_ms = self._limits.get("timeout_ms")
if timeout_ms is not None:
# m-7 fix: validate timeout_ms > 0 before passing to
# asyncio.wait_for(). A value of 0 causes immediate timeout
# on every request; a negative value has undefined behaviour.
# Raise ExecutionError(kind="timeout") with a clear diagnostic
# rather than letting asyncio produce a confusing error.
#
# Bool guard: float(True)==1.0 would silently set a 1ms timeout
# (effectively making every request time out); float(False)==0.0
# would be caught by the <= 0 check below but with a confusing
# message. Reject bools explicitly, matching the pattern used
# for max_depth/max_model_calls/max_tool_calls.
if isinstance(timeout_ms, bool):
raise ExecutionError(
f"Invalid timeout_ms value {timeout_ms!r}: "
"bool is not a valid timeout",
kind="timeout",
)
try:
_timeout_float = float(timeout_ms) / 1000.0
except (TypeError, ValueError) as _to_err:
raise ExecutionError(
f"Invalid timeout_ms value {timeout_ms!r}: {_to_err}",
kind="timeout",
) from _to_err
if _timeout_float <= 0:
raise ExecutionError(
f"Invalid timeout_ms value {timeout_ms!r}: "
"must be a positive number of milliseconds",
kind="timeout",
)
try:
output = await asyncio.wait_for(
self._execute_from_node(self.config.entry_point, input_message),
timeout=_timeout_float,
)
except asyncio.TimeoutError:
raise ExecutionError(
f"Execution timed out after {timeout_ms} ms",
kind="timeout",
) from None
else:
output = await self._execute_from_node(
self.config.entry_point, input_message
)
# Log output info for debugging truncation issues
if output:
@@ -440,24 +536,58 @@ class PureLangGraph:
f"_execute_from_node called with node_name='{node_name}', message type={type(message)}, depth={depth}"
)
# Prevent infinite recursion - but allow deeper traversal for complex workflows
# For workflows with many sections (like scientific paper writer), we need high limits
# Each section requires ~20 node visits, so 50 sections = 1000+ visits
# Use a generous limit that allows complex workflows to complete
max_depth = max(2000, len(self.nodes) * 50)
if depth > max_depth:
self.logger.warning(
f"Maximum recursion depth {max_depth} exceeded. Returning current output."
)
import sys
print(
f"MAX_DEPTH_EXCEEDED node={node_name} depth={depth} max={max_depth}",
file=sys.stderr,
)
# C-2 fix (ADR-2029, issue #15): The terminal "end"/"END" node has no
# execution logic and must never be subject to the depth check. Moving
# this short-circuit to the very top of _execute_from_node — before the
# depth check — ensures that any graph with max_depth ≤ N (number of
# real nodes) can still complete: the "end" node is always reached
# regardless of the accumulated depth counter.
if node_name == "end" or node_name == "END":
self.logger.debug(f"Reached terminal node: {node_name}")
return message
# Depth limit enforcement (ADR-2029, issue #15).
#
# When limits["max_depth"] is supplied by the router, enforce it strictly
# and raise ExecutionError(kind="depth") on breach so the router can map
# the error to HTTP 429 (ADR-2029).
#
# M-2 fix: When limits["max_depth"] is NOT supplied, use 2**31-1 as the
# internal ceiling per ADR-2029 ("the library uses a very large internal
# ceiling of 2**31 - 1 for max_depth"). This replaces the old legacy
# heuristic (max(2000, len(self.nodes) * 50)) which deviated from the spec.
# Existing callers that do not pass limits are unaffected in practice
# because no real graph reaches depth 2**31-1; the only tests that relied
# on the heuristic used depth=10000 which is still well below 2**31-1.
_max_depth_raw = self._limits.get("max_depth")
if _max_depth_raw is not None:
# M-4 fix: validate with a narrow try/except so a non-numeric
# max_depth value produces a structured ExecutionError(kind="depth")
# rather than a bare ValueError that the router cannot map.
# Also guard against bool (int(True)==1, int(False)==0) which would
# silently set a nonsensical limit.
if isinstance(_max_depth_raw, bool):
raise ExecutionError(
f"Invalid max_depth value {_max_depth_raw!r}: "
"bool is not a valid depth limit",
kind="depth",
)
try:
_max_depth_enforced: int = int(_max_depth_raw)
except (TypeError, ValueError) as _depth_err:
raise ExecutionError(
f"Invalid max_depth value {_max_depth_raw!r}: {_depth_err}",
kind="depth",
) from _depth_err
else:
_max_depth_enforced = 2**31 - 1
if depth > _max_depth_enforced:
raise ExecutionError(
f"Graph depth limit exceeded: "
f"depth {depth} > max_depth {_max_depth_enforced}",
kind="depth",
)
# Track node visits with message fingerprints to detect actual loops
# A loop is when the same node is visited with the same message twice
if not hasattr(self, "_node_message_visits"):
@@ -500,11 +630,10 @@ class PureLangGraph:
f"Node '{node_name}' visited {self._node_message_visits[visit_key]} times "
f"with the same message. Stopping execution to return output to user."
)
import sys
print(
f"LOOP_STOP node={node_name} visits={self._node_message_visits[visit_key]}",
file=sys.stderr,
self.logger.warning(
"LOOP_STOP node=%s visits=%d",
node_name,
self._node_message_visits[visit_key],
)
return message
@@ -550,11 +679,10 @@ class PureLangGraph:
f"Detected router-agent ping-pong loop with same agent: {recent}. "
f"Stopping execution to return output."
)
import sys
print(
f"PING_PONG_DETECTED recent={recent} auto_finish={auto_finish_active}",
file=sys.stderr,
self.logger.warning(
"PING_PONG_DETECTED recent=%s auto_finish=%s",
recent,
auto_finish_active,
)
if self._execution_path:
self._execution_path.pop()
@@ -569,11 +697,6 @@ class PureLangGraph:
return await self._execute_from_node(next_nodes[0], message, depth + 1)
return message
if node_name == "end" or node_name == "END":
# Terminal node - return the message
self.logger.debug(f"Reached terminal node: {node_name}")
return message
# Get the node
if node_name not in self.nodes:
self.logger.error(
@@ -587,6 +710,59 @@ class PureLangGraph:
self.logger.debug(f"Executing node: {node_name} (type: {node.config.type})")
self.execution_history.append(node_name)
# Execution limit enforcement: model_calls and tool_calls (ADR-2029, issue #15).
# Check BEFORE running the node so we never run the (limit + 1)-th invocation.
# AGENT-type nodes map to LLM model calls; TOOL-type nodes map to tool calls.
if node.config.type == NodeType.AGENT:
max_model = self._limits.get("max_model_calls")
if max_model is not None:
# M-4 fix: validate with a narrow try/except so a non-numeric
# or bool value produces a structured ExecutionError.
if isinstance(max_model, bool):
raise ExecutionError(
f"Invalid max_model_calls value {max_model!r}: "
"bool is not a valid call limit",
kind="model_calls",
)
try:
_max_model_int = int(max_model)
except (TypeError, ValueError) as _mc_err:
raise ExecutionError(
f"Invalid max_model_calls value {max_model!r}: {_mc_err}",
kind="model_calls",
) from _mc_err
if self._model_call_count >= _max_model_int:
raise ExecutionError(
f"model_calls limit exceeded: "
f"{self._model_call_count} >= {max_model}",
kind="model_calls",
)
self._model_call_count += 1
elif node.config.type == NodeType.TOOL:
max_tool = self._limits.get("max_tool_calls")
if max_tool is not None:
# M-4 fix: same narrow validation for max_tool_calls.
if isinstance(max_tool, bool):
raise ExecutionError(
f"Invalid max_tool_calls value {max_tool!r}: "
"bool is not a valid call limit",
kind="tool_calls",
)
try:
_max_tool_int = int(max_tool)
except (TypeError, ValueError) as _tc_err:
raise ExecutionError(
f"Invalid max_tool_calls value {max_tool!r}: {_tc_err}",
kind="tool_calls",
) from _tc_err
if self._tool_call_count >= _max_tool_int:
raise ExecutionError(
f"tool_calls limit exceeded: "
f"{self._tool_call_count} >= {max_tool}",
kind="tool_calls",
)
self._tool_call_count += 1
# Get current state
state = self.state_manager.get_state()
@@ -631,16 +807,138 @@ class PureLangGraph:
# Collect per-node token usage if this agent node reported it
# (AC4, issue #14). Node._execute_agent() sets
# _node_token_usage in the result dict for LLM agent nodes.
# m-1 fix: log a warning when _node_token_usage is present but
# not a dict so that malformed values surface in production logs
# rather than silently disabling token accounting and cost
# enforcement.
if _tok_info is not None and not isinstance(_tok_info, dict):
self.logger.warning(
"Node %s returned non-dict _node_token_usage (%r); "
"token accounting skipped",
node_name,
type(_tok_info).__name__,
)
if isinstance(_tok_info, dict):
# m-2 fix: compute _pt and _ct once at the top of the
# isinstance block and reuse them in both _node_usages and
# the cost block, eliminating the double _safe_token_int()
# call that doubled log noise for malformed inputs.
_pt = _safe_token_int(_tok_info.get("prompt_tokens", 0))
_ct = _safe_token_int(_tok_info.get("completion_tokens", 0))
self._node_usages.append(
(
str(_tok_info.get("node_id", node_name)),
str(_tok_info.get("provider", "unknown")),
str(_tok_info.get("model", "unknown")),
_safe_token_int(_tok_info.get("prompt_tokens", 0)),
_safe_token_int(_tok_info.get("completion_tokens", 0)),
_pt,
_ct,
)
)
# Cost accumulation and enforcement (ADR-2029, issue #15).
# Only when the router supplied a non-empty pricing table; an
# empty dict means cost tracking was not requested and is skipped
# entirely. When the table is non-empty, a missing provider or
# model entry is a hard error — proceeding with an assumed zero
# price is forbidden because it would enable unbounded spend.
if self._pricing:
_provider = str(_tok_info.get("provider", "unknown"))
_model = str(_tok_info.get("model", "unknown"))
# _pt and _ct already computed above (m-2 fix)
_provider_pricing = self._pricing.get(_provider)
if _provider_pricing is None or not isinstance(
_provider_pricing, dict
):
raise ExecutionError(
f"Missing pricing entry for provider '{_provider}'",
kind="cost",
reason="missing_pricing_entry",
)
_model_pricing = _provider_pricing.get(_model)
if _model_pricing is None or not isinstance(
_model_pricing, dict
):
raise ExecutionError(
f"Missing pricing entry for model '{_model}' "
f"under provider '{_provider}'",
kind="cost",
reason="missing_pricing_entry",
)
# Validate that both rate keys exist and are numeric
# (ADR-2029: "proceeding with a missing or zero price is
# forbidden"). A missing key or a non-numeric value is
# treated the same as a missing pricing entry — the
# library must not silently fall back to $0/token.
_prompt_rate_raw = _model_pricing.get("prompt")
_completion_rate_raw = _model_pricing.get("completion")
if _prompt_rate_raw is None or _completion_rate_raw is None:
raise ExecutionError(
f"Incomplete pricing entry for model '{_model}' "
f"under provider '{_provider}': missing 'prompt' "
f"or 'completion' rate key",
kind="cost",
reason="missing_pricing_entry",
)
try:
# Rates are USD per million tokens (ADR-2029 example:
# gpt-4.1-mini prompt=$0.15/1M, completion=$0.60/1M).
_prompt_rate = float(_prompt_rate_raw)
_completion_rate = float(_completion_rate_raw)
except (TypeError, ValueError) as _rate_err:
raise ExecutionError(
f"Invalid pricing rate for model '{_model}' "
f"under provider '{_provider}': {_rate_err}",
kind="cost",
reason="missing_pricing_entry",
) from _rate_err
_node_cost = (
_pt / 1_000_000.0 * _prompt_rate
+ _ct / 1_000_000.0 * _completion_rate
)
self._accumulated_cost += _node_cost
# C-1 fix (ADR-2029, issue #15): coerce max_cost_usd
# with a narrow try/except BEFORE the broad
# "except Exception" handler so that a malformed value
# (e.g. a non-numeric string) raises ExecutionError
# immediately rather than being silently swallowed by
# the broad handler and causing the graph to return the
# input string as if execution succeeded.
_max_cost = self._limits.get("max_cost_usd")
if _max_cost is not None:
# Bool guard: float(True)==1.0 / float(False)==0.0
# would silently set a nonsensical budget. Reject
# bools before the float() conversion, matching the
# pattern used for max_depth/max_model_calls/max_tool_calls.
if isinstance(_max_cost, bool):
raise ExecutionError(
f"Invalid max_cost_usd value {_max_cost!r}: "
"bool is not a valid cost limit",
kind="cost",
)
try:
_max_cost_float = float(_max_cost)
except (TypeError, ValueError) as _cost_err:
# A malformed limit value (e.g. a non-numeric string)
# is a router configuration error, not a missing pricing
# entry. Use reason="" (the documented default for
# non-cost-specific errors per ADR-2029) so router logs
# and alerts correctly identify the root cause.
raise ExecutionError(
f"Invalid max_cost_usd value "
f"{_max_cost!r}: {_cost_err}",
kind="cost",
) from _cost_err
if self._accumulated_cost > _max_cost_float:
raise ExecutionError(
f"Cost limit exceeded: "
f"{self._accumulated_cost:.6f} USD "
f"> {_max_cost} USD",
kind="cost",
reason="budget_exhausted",
)
else:
output_message = result
@@ -662,6 +960,11 @@ class PureLangGraph:
)
break
except ExecutionError:
# Limit-enforcement errors (depth, model_calls, tool_calls, timeout,
# cost) must propagate without suppression so the router can map
# them to the correct HTTP status code (ADR-2029, issue #15).
raise
except Exception as e:
self.logger.error(f"Error executing node {node_name}: {e}", exc_info=True)
output_message = message
@@ -716,13 +1019,38 @@ class PureLangGraph:
# Execute next nodes
if self.config.parallel_execution and len(next_nodes) > 1:
# Execute nodes in parallel
# Execute nodes in parallel.
# M-1 fix (ADR-2029, issue #15): when one branch raises an
# exception (e.g. ExecutionError for a limit breach), cancel all
# sibling tasks immediately so they do not continue spending budget
# after the limit has already been exceeded. asyncio.gather()
# with the default return_exceptions=False propagates the first
# exception to the awaiter but does NOT cancel the other in-flight
# tasks — we must do that explicitly.
self.logger.debug(f"Executing {len(next_nodes)} nodes in parallel")
tasks = [
self._execute_from_node(next_node, output_message, depth + 1)
asyncio.create_task(
self._execute_from_node(next_node, output_message, depth + 1)
)
for next_node in next_nodes
]
results = await asyncio.gather(*tasks)
try:
results = await asyncio.gather(*tasks)
except BaseException:
# Catch BaseException (not just Exception) because
# asyncio.CancelledError is a BaseException in Python 3.8+.
# When the outer asyncio.wait_for() timeout fires it raises
# CancelledError, which would not be caught by `except
# Exception` — leaving sibling tasks running indefinitely.
# The `raise` at the end re-propagates the original exception
# (including KeyboardInterrupt/SystemExit) so nothing is
# swallowed.
for t in tasks:
if not t.done():
t.cancel()
# Await cancelled tasks to suppress CancelledError noise
await asyncio.gather(*tasks, return_exceptions=True)
raise
# Clean up execution path for parallel execution
if hasattr(self, "_execution_path") and self._execution_path:
self._execution_path.pop()
+6 -1
View File
@@ -382,7 +382,12 @@ async def _execute_graph(
if conversation_history:
global_context["conversation_history"] = conversation_history
graph = PureLangGraph(config=pg_config, agents=agents)
graph = PureLangGraph(
config=pg_config,
agents=agents,
limits=executor.limits,
pricing=executor.pricing,
)
# PureLangGraph.execute() returns a 3-tuple:
# (response_str, final_state_dict, node_usages_list) where
# node_usages_list is a list of (node_id, provider, model,