feat(execution-limits): add structured ExecutionError kind/reason fields; enforce all 5 execution limits in PureLangGraph #44
@@ -15,6 +15,7 @@ and this project adheres to [Clever Semantic Versioning](https://www.w3.org/subm
|
||||
|
||||
### Added
|
||||
|
||||
- **Structured `ExecutionError` fields + 5-limit enforcement in `PureLangGraph`** (`cleveractors.core.exceptions.ExecutionError`, `cleveractors.langgraph.pure_graph.PureLangGraph`): `ExecutionError` gains `kind` (categorical: `depth`, `model_calls`, `tool_calls`, `timeout`, `cost`) and `reason` (sub-code: `budget_exhausted` or `missing_pricing_entry`) fields, both defaulting to `""` for backward compatibility with existing `raise ExecutionError(msg)` call sites. `PureLangGraph` now accepts `limits` and `pricing` constructor arguments. When `limits["max_depth"]` is supplied, a depth breach raises `ExecutionError(kind="depth")` instead of silently returning the current message. `max_model_calls` is checked before each AGENT-type node; `max_tool_calls` before each TOOL-type node. `execute()` wraps `_execute_from_node()` in `asyncio.wait_for()` when `limits["timeout_ms"]` is set, mapping `asyncio.TimeoutError` to `ExecutionError(kind="timeout")`. After each LLM node, cost is computed from the supplied `pricing` table (rates are USD per million tokens per ADR-2029); breach raises `ExecutionError(kind="cost", reason="budget_exhausted")`; a missing provider or model entry raises `ExecutionError(kind="cost", reason="missing_pricing_entry")`. `runtime_dispatch._execute_graph()` now passes `executor.limits` and `executor.pricing` to `PureLangGraph`. `ExecutionError` is exported from `cleveractors.__init__` and `__all__`. (ADR-2029, issue #15)
|
||||
- **`create_executor()` router-facing API** (`cleveractors.create_executor`): New module-level factory function that constructs an `Executor` wrapping `PureLangGraph` and `AgentFactory`. Accepts `config_dict` (validated actor configuration), `credentials` (per-provider credential dict for per-request injection), `limits` (execution budget), and `pricing` (per-model cost table). `executor.execute(message)` runs the actor graph and returns an `ActorResult(response, prompt_tokens, completion_tokens, nodes)`. All four execution paths are supported — `llm`, `graph`, `tool`, and `multi_actor`. Credentials are passed to `AgentFactory` and never injected into the stored `config_dict` (ADR-2026 AC8). Exported from `cleveractors.__init__` and `__all__` (ADR-2024, ADR-2026, ADR-2029).
|
||||
- **`ActorResult` and `NodeUsage` types** (`cleveractors.result`): Canonical dataclasses for the router-facing result API, now defined in `cleveractors.result` (ADR-2027). `ActorResult` carries the response string, aggregated `prompt_tokens`/`completion_tokens`, a non-empty `nodes: list[NodeUsage]` breakdown for per-model billing, and an optional opaque `state` blob for stateless graph resumption (ADR-2026). Re-exported from `cleveractors.runtime` for backward compatibility.
|
||||
- **Real LangChain token extraction** (`LLMAgent`): `process_message()` now reads token counts from `response.usage_metadata` (primary path) with fallback to `response.response_metadata["token_usage"]`. Both paths are guarded with `isinstance(dict)` checks to prevent `AttributeError` on truthy non-dict provider values. Non-numeric token values are coerced via `_safe_int()` with fallback to 0 and a warning log. A warning distinguishing the failure cause (empty `usage_metadata`, missing `response_metadata`, or missing `token_usage` key) is emitted when no usage data is available.
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
Feature: Execution Limits Enforcement in PureLangGraph (ADR-2029)
|
||||
As the CleverThis router
|
||||
I want PureLangGraph to enforce per-request execution limits
|
||||
So that budget exhaustion, depth breaches, and cost violations surface as
|
||||
structured ExecutionError exceptions with the correct kind/reason fields
|
||||
|
||||
Background:
|
||||
Given the execution limits test context is initialised (elim)
|
||||
|
||||
# ── ExecutionError structured fields ──────────────────────────────────────
|
||||
|
||||
Scenario: ExecutionError defaults kind and reason to empty strings
|
||||
When an ExecutionError is raised with only a message (elim)
|
||||
Then the kind field should be empty string (elim)
|
||||
And the reason field should be empty string (elim)
|
||||
|
||||
Scenario: ExecutionError accepts kind with empty reason
|
||||
When an ExecutionError is raised with kind "depth" and empty reason (elim)
|
||||
Then the kind field should be "depth" (elim)
|
||||
And the reason field should be empty string (elim)
|
||||
|
||||
Scenario: ExecutionError accepts kind cost with reason budget_exhausted
|
||||
When an ExecutionError is raised with kind "cost" and reason "budget_exhausted" (elim)
|
||||
Then the kind field should be "cost" (elim)
|
||||
And the reason field should be "budget_exhausted" (elim)
|
||||
|
||||
Scenario: ExecutionError accepts kind cost with reason missing_pricing_entry
|
||||
When an ExecutionError is raised with kind "cost" and reason "missing_pricing_entry" (elim)
|
||||
Then the kind field should be "cost" (elim)
|
||||
And the reason field should be "missing_pricing_entry" (elim)
|
||||
|
||||
Scenario: ExecutionError is exported from cleveractors top-level package
|
||||
When ExecutionError is imported from cleveractors (elim)
|
||||
Then the import should succeed and expose kind and reason attributes (elim)
|
||||
|
||||
# ── Depth limit ────────────────────────────────────────────────────────────
|
||||
|
||||
Scenario: Graph exceeding max_depth raises ExecutionError with kind "depth"
|
||||
Given a linear 3-node graph with max_depth 1 (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "depth" (elim)
|
||||
|
||||
Scenario: Graph within max_depth completes successfully
|
||||
Given a linear 2-node graph with max_depth 10 (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then no ExecutionError should be raised (elim)
|
||||
|
||||
# ── Model-call limit ───────────────────────────────────────────────────────
|
||||
|
||||
Scenario: Two AGENT nodes with max_model_calls 1 raises on second call
|
||||
Given a graph with 2 AGENT nodes and max_model_calls 1 (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
||||
|
||||
Scenario: Two AGENT nodes with max_model_calls 2 completes successfully
|
||||
Given a graph with 2 AGENT nodes and max_model_calls 2 (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then no ExecutionError should be raised (elim)
|
||||
|
||||
# ── Tool-call limit ────────────────────────────────────────────────────────
|
||||
|
||||
Scenario: Two TOOL nodes with max_tool_calls 1 raises on second call
|
||||
Given a graph with 2 TOOL nodes and max_tool_calls 1 (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "tool_calls" (elim)
|
||||
|
||||
Scenario: Two TOOL nodes with max_tool_calls 2 completes successfully
|
||||
Given a graph with 2 TOOL nodes and max_tool_calls 2 (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then no ExecutionError should be raised (elim)
|
||||
|
||||
# ── Timeout ───────────────────────────────────────────────────────────────
|
||||
|
||||
Scenario: Graph with slow node times out and raises ExecutionError kind "timeout"
|
||||
Given a graph with a slow node and timeout_ms 50 (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "timeout" (elim)
|
||||
|
||||
Scenario: Graph completing before timeout succeeds
|
||||
Given a graph with a fast node and timeout_ms 2000 (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then no ExecutionError should be raised (elim)
|
||||
|
||||
# ── Cost limit — budget_exhausted ─────────────────────────────────────────
|
||||
|
||||
Scenario: LLM node with high token count exceeding cost limit raises budget_exhausted
|
||||
Given a graph with an LLM node using 2000000 prompt tokens and max_cost_usd 0.10 (elim)
|
||||
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
||||
Then an ExecutionError should be raised with kind "cost" (elim)
|
||||
And the reason should be "budget_exhausted" (elim)
|
||||
|
||||
Scenario: LLM node with low token count within cost limit completes successfully
|
||||
Given a graph with an LLM node using 100 prompt tokens and max_cost_usd 1.00 (elim)
|
||||
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
||||
Then no ExecutionError should be raised (elim)
|
||||
|
||||
# ── Cost limit — missing_pricing_entry ────────────────────────────────────
|
||||
|
||||
Scenario: LLM node with unknown provider raises missing_pricing_entry
|
||||
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
|
||||
When the graph is executed with pricing only for "anthropic" provider (elim)
|
||||
Then an ExecutionError should be raised with kind "cost" (elim)
|
||||
And the reason should be "missing_pricing_entry" (elim)
|
||||
|
||||
Scenario: LLM node with unknown model raises missing_pricing_entry
|
||||
Given a graph with an LLM node for provider "openai" model "unknown-model" (elim)
|
||||
When the graph is executed with pricing for "openai" but not "unknown-model" (elim)
|
||||
Then an ExecutionError should be raised with kind "cost" (elim)
|
||||
And the reason should be "missing_pricing_entry" (elim)
|
||||
|
||||
Scenario: Empty pricing table skips cost calculation entirely
|
||||
Given a graph with an LLM node using 2000000 prompt tokens and max_cost_usd 0.001 (elim)
|
||||
When the graph is executed with an empty pricing table (elim)
|
||||
Then no ExecutionError should be raised (elim)
|
||||
|
||||
Scenario: LLM node with incomplete model pricing entry missing prompt rate raises missing_pricing_entry
|
||||
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
|
||||
When the graph is executed with model pricing entry missing the prompt rate (elim)
|
||||
Then an ExecutionError should be raised with kind "cost" (elim)
|
||||
And the reason should be "missing_pricing_entry" (elim)
|
||||
|
||||
Scenario: LLM node with non-numeric pricing rate raises missing_pricing_entry
|
||||
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
|
||||
When the graph is executed with a non-numeric prompt rate (elim)
|
||||
Then an ExecutionError should be raised with kind "cost" (elim)
|
||||
And the reason should be "missing_pricing_entry" (elim)
|
||||
|
||||
# ── Cost accumulation across multiple LLM nodes ───────────────────────────
|
||||
|
||||
Scenario: Cost accumulates across two LLM nodes and triggers budget_exhausted on second
|
||||
Given a graph with 2 LLM nodes each using 1000000 prompt tokens and max_cost_usd 0.25 (elim)
|
||||
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
||||
Then an ExecutionError should be raised with kind "cost" (elim)
|
||||
And the reason should be "budget_exhausted" (elim)
|
||||
|
||||
# ── Completion tokens cost contribution ───────────────────────────────────
|
||||
|
||||
Scenario: Completion tokens contribute to cost and can trigger budget_exhausted
|
||||
Given a graph with an LLM node using 0 prompt tokens 1000000 completion tokens and max_cost_usd 0.50 (elim)
|
||||
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt 0.60 completion rate (elim)
|
||||
Then an ExecutionError should be raised with kind "cost" (elim)
|
||||
And the reason should be "budget_exhausted" (elim)
|
||||
|
||||
# ── Malformed limit value validation ─────────────────────────────────────
|
||||
|
||||
Scenario: Non-numeric max_cost_usd raises ExecutionError with kind "cost"
|
||||
Given a graph with an LLM node for non-numeric max_cost_usd validation (elim)
|
||||
When the graph is executed with pricing and non-numeric max_cost_usd (elim)
|
||||
Then an ExecutionError should be raised with kind "cost" (elim)
|
||||
|
||||
Scenario: Non-numeric max_model_calls raises ExecutionError with kind "model_calls"
|
||||
Given a graph with 1 AGENT node and non-numeric max_model_calls (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
||||
|
||||
Scenario: Bool max_model_calls raises ExecutionError with kind "model_calls"
|
||||
Given a graph with 1 AGENT node and bool False max_model_calls (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
||||
|
||||
Scenario: Non-numeric max_tool_calls raises ExecutionError with kind "tool_calls"
|
||||
Given a graph with 1 TOOL node and non-numeric max_tool_calls (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "tool_calls" (elim)
|
||||
|
||||
Scenario: Bool max_tool_calls raises ExecutionError with kind "tool_calls"
|
||||
Given a graph with 1 TOOL node and bool False max_tool_calls (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "tool_calls" (elim)
|
||||
|
||||
Scenario: Non-positive timeout_ms raises ExecutionError with kind "timeout"
|
||||
Given a graph with a fast node and timeout_ms 0 (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "timeout" (elim)
|
||||
|
||||
Scenario: Bool timeout_ms raises ExecutionError with kind "timeout"
|
||||
Given a graph with a fast node and bool True timeout_ms (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "timeout" (elim)
|
||||
|
||||
Scenario: Non-numeric max_depth raises ExecutionError with kind "depth"
|
||||
Given a graph with a fast node and non-numeric max_depth (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "depth" (elim)
|
||||
|
||||
Scenario: Bool max_depth raises ExecutionError with kind "depth"
|
||||
Given a graph with a fast node and bool False max_depth (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "depth" (elim)
|
||||
|
||||
Scenario: Bool max_cost_usd raises ExecutionError with kind "cost"
|
||||
Given a graph with an LLM node for bool max_cost_usd validation (elim)
|
||||
When the graph is executed with pricing and bool False max_cost_usd (elim)
|
||||
Then an ExecutionError should be raised with kind "cost" (elim)
|
||||
|
||||
Scenario: Parallel graph cancels sibling tasks when one branch raises ExecutionError
|
||||
Given a parallel graph where one branch raises an ExecutionError (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
||||
And the sibling branch should have been cancelled (elim)
|
||||
|
||||
Scenario: Non-dict _node_token_usage logs warning and skips token accounting
|
||||
Given a graph with an LLM node that returns non-dict token usage (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then no ExecutionError should be raised (elim)
|
||||
And a warning should have been logged for non-dict token usage (elim)
|
||||
|
||||
# ── Happy-path result correctness ─────────────────────────────────────────
|
||||
|
||||
Scenario: Graph within max_depth returns non-None result
|
||||
Given a linear 2-node graph with max_depth 10 (elim)
|
||||
When the graph is executed with a test message (elim)
|
||||
Then no ExecutionError should be raised (elim)
|
||||
And the result should be non-None (elim)
|
||||
|
||||
# ── PureLangGraph limits/pricing wired from Executor ──────────────────────
|
||||
|
||||
Scenario: limits and pricing are stored on Executor and propagated to PureLangGraph
|
||||
Given an Executor with graph config and limits max_depth 100 pricing for openai (elim)
|
||||
When the executor is checked for limit/pricing propagation (elim)
|
||||
Then the PureLangGraph should have the correct limits and pricing (elim)
|
||||
|
||||
Scenario: Executor.execute() enforces max_depth through the full dispatch path (elim)
|
||||
Given an Executor with a 2-node graph and max_depth 0 (elim)
|
||||
When the executor execute method is called with a test message (elim)
|
||||
Then an ExecutionError should be raised with kind "depth" (elim)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -186,7 +186,12 @@ def step_pgg_nested_auto_finish_cycle(context):
|
||||
Edge(source="writer", target="writer"),
|
||||
],
|
||||
)
|
||||
graph = PureLangGraph(config)
|
||||
# M-2 fix: pass an explicit max_depth so the graph terminates when the
|
||||
# depth limit is reached rather than recursing to Python's stack limit.
|
||||
# The old legacy heuristic (max(2000, len(nodes)*50)) previously acted as
|
||||
# a safety net; now that the default is 2**31-1 per ADR-2029, tests that
|
||||
# rely on cycle-via-auto_finish_active must supply their own limit.
|
||||
graph = PureLangGraph(config, limits={"max_depth": 50})
|
||||
|
||||
# node that keeps returning same message (simulate cycle)
|
||||
mock_node = MagicMock()
|
||||
@@ -211,9 +216,21 @@ def step_pgg_nested_auto_finish_cycle(context):
|
||||
state = graph.state_manager.get_state()
|
||||
state.metadata["context"] = {}
|
||||
|
||||
result = _run(graph.execute("write section"))
|
||||
# M-2 fix: the graph now raises ExecutionError(kind="depth") when the
|
||||
# depth limit is reached (instead of silently returning the message via
|
||||
# the old heuristic). The test only cares that auto_finish_active allowed
|
||||
# the cycle to continue past the first visit — the depth-limit termination
|
||||
# is the expected way the graph stops in this scenario.
|
||||
from cleveractors.core.exceptions import ExecutionError
|
||||
|
||||
try:
|
||||
result = _run(graph.execute("write section"))
|
||||
context.pgg_results["graph_completed"] = result is not None
|
||||
except ExecutionError as exc:
|
||||
# Depth limit reached — graph terminated via ExecutionError, which is
|
||||
# the correct ADR-2029 behaviour when max_depth is set.
|
||||
context.pgg_results["graph_completed"] = exc.kind == "depth"
|
||||
context.pgg_results["nested_af_continued"] = visit_count[0] >= 2
|
||||
context.pgg_results["graph_completed"] = result is not None
|
||||
|
||||
|
||||
# ---- Ping-pong detection with auto_finish_active in nested context ----
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
from behave import given, then, when
|
||||
@@ -521,6 +522,7 @@ def step_cycle_detection(context):
|
||||
|
||||
@when("I execute a graph with auto_finish_active and a self-loop (pg_cov)")
|
||||
def step_auto_finish_bypass(context):
|
||||
from cleveractors.core.exceptions import ExecutionError
|
||||
from cleveractors.langgraph.nodes import Edge, NodeConfig, NodeType
|
||||
from cleveractors.langgraph.pure_graph import PureGraphConfig, PureLangGraph
|
||||
|
||||
@@ -535,13 +537,41 @@ def step_auto_finish_bypass(context):
|
||||
Edge(source="looper", target="end"),
|
||||
],
|
||||
)
|
||||
graph = PureLangGraph(config)
|
||||
# M-2 fix: pass an explicit max_depth so the graph terminates via
|
||||
# ExecutionError(kind="depth") rather than recursing to Python's stack
|
||||
# limit. The test only cares that auto_finish_active bypassed cycle
|
||||
# detection (allowing the loop to continue past the first visit).
|
||||
graph = PureLangGraph(config, limits={"max_depth": 50})
|
||||
state = graph.state_manager.get_state()
|
||||
state.metadata["auto_finish_active"] = True
|
||||
|
||||
# Issue #6 fix: track visit count to verify that auto_finish_active
|
||||
# actually allowed the loop to continue past the first visit. A
|
||||
# regression that disabled the bypass would stop at visit 1, leaving
|
||||
# visit_count[0] == 1 and failing the assertion in the Then step.
|
||||
visit_count = [0]
|
||||
looper_mock = MagicMock()
|
||||
looper_mock.config = MagicMock()
|
||||
looper_mock.config.type = NodeType.FUNCTION
|
||||
looper_mock.config.name = "looper"
|
||||
|
||||
async def _looper_execute(state: Any) -> dict:
|
||||
visit_count[0] += 1
|
||||
return {"content": "continue please"}
|
||||
|
||||
looper_mock.execute = AsyncMock(side_effect=_looper_execute)
|
||||
graph.nodes["looper"] = looper_mock
|
||||
context.results["auto_finish_visit_count"] = visit_count
|
||||
|
||||
async def _run():
|
||||
result = await graph.execute("continue please")
|
||||
context.results["auto_finish_result"] = result
|
||||
try:
|
||||
result = await graph.execute("continue please")
|
||||
context.results["auto_finish_result"] = result
|
||||
except ExecutionError as exc:
|
||||
# Depth limit reached — the loop ran until max_depth, which is
|
||||
# the expected termination path when auto_finish_active bypasses
|
||||
# cycle detection.
|
||||
context.results["auto_finish_result"] = f"depth_limit_reached: {exc.kind}"
|
||||
|
||||
_run_async(_run(), context=context)
|
||||
|
||||
@@ -586,6 +616,7 @@ def step_ping_pong_detection(context):
|
||||
|
||||
@when("I execute a graph with auto_finish_active causing ping-pong pattern (pg_cov)")
|
||||
def step_ping_pong_bypass(context):
|
||||
from cleveractors.core.exceptions import ExecutionError
|
||||
from cleveractors.langgraph.nodes import Edge, NodeConfig, NodeType
|
||||
from cleveractors.langgraph.pure_graph import PureGraphConfig, PureLangGraph
|
||||
|
||||
@@ -610,16 +641,42 @@ def step_ping_pong_bypass(context):
|
||||
Edge(source="router_node", target="end"),
|
||||
],
|
||||
)
|
||||
# Issue #6 fix: track agent visit count to verify that auto_finish_active
|
||||
# actually allowed the ping-pong to continue past the first cycle.
|
||||
# The agent returns "GOTO_agent:continue" so the "no routing command" early
|
||||
# exit does not fire — the loop continues until the depth limit is reached.
|
||||
visit_count = [0]
|
||||
mock_agent = MagicMock()
|
||||
mock_agent.name = "mock_agent"
|
||||
mock_agent.process_message = AsyncMock(return_value="hello from agent")
|
||||
graph = PureLangGraph(config, agents={"mock_agent": mock_agent})
|
||||
|
||||
async def _agent_process(message: Any, context: Any = None) -> str:
|
||||
visit_count[0] += 1
|
||||
# Return a routing command so the graph does not stop at the
|
||||
# "no routing command" early-exit check (lines 982-1008 in pure_graph.py).
|
||||
# This ensures the ping-pong detection path is actually exercised.
|
||||
return "GOTO_agent:continue"
|
||||
|
||||
mock_agent.process_message = AsyncMock(side_effect=_agent_process)
|
||||
# M-2 fix: pass an explicit max_depth so the graph terminates via
|
||||
# ExecutionError(kind="depth") rather than recursing to Python's stack
|
||||
# limit. The test only cares that auto_finish_active bypassed ping-pong
|
||||
# detection (allowing the loop to continue).
|
||||
graph = PureLangGraph(
|
||||
config, agents={"mock_agent": mock_agent}, limits={"max_depth": 50}
|
||||
)
|
||||
state = graph.state_manager.get_state()
|
||||
state.metadata["auto_finish_active"] = True
|
||||
context.results["bypass_visit_count"] = visit_count
|
||||
|
||||
async def _run():
|
||||
result = await graph.execute("hello")
|
||||
context.results["bypass_result"] = result
|
||||
try:
|
||||
result = await graph.execute("hello")
|
||||
context.results["bypass_result"] = result
|
||||
except ExecutionError as exc:
|
||||
# Depth limit reached — the ping-pong ran until max_depth, which
|
||||
# is the expected termination path when auto_finish_active bypasses
|
||||
# ping-pong detection.
|
||||
context.results["bypass_result"] = f"depth_limit_reached: {exc.kind}"
|
||||
|
||||
_run_async(_run(), context=context)
|
||||
|
||||
@@ -1272,6 +1329,17 @@ def step_verify_cycle_bypassed(context):
|
||||
assert context.results.get("auto_finish_result") is not None, (
|
||||
"auto_finish execution completed"
|
||||
)
|
||||
# Issue #6 fix: verify that the looper node was visited at least twice,
|
||||
# confirming that auto_finish_active bypassed cycle detection and allowed
|
||||
# the loop to continue past the first visit. A regression that disabled
|
||||
# the bypass would stop at visit 1.
|
||||
visit_count = context.results.get("auto_finish_visit_count")
|
||||
if visit_count is not None:
|
||||
assert visit_count[0] >= 2, (
|
||||
f"Expected looper to be visited >= 2 times (auto_finish_active bypass), "
|
||||
f"but was only visited {visit_count[0]} time(s). "
|
||||
"The auto_finish_active cycle-detection bypass may have been removed."
|
||||
)
|
||||
|
||||
|
||||
@then("the ping-pong should be detected and current message returned (pg_cov)")
|
||||
@@ -1284,6 +1352,17 @@ def step_verify_ping_pong_stopped(context):
|
||||
@then("execution should continue past ping-pong detection (pg_cov)")
|
||||
def step_verify_ping_pong_bypassed(context):
|
||||
assert context.results.get("bypass_result") is not None, "ping-pong bypassed"
|
||||
# Issue #6 fix: verify that the agent was visited at least twice,
|
||||
# confirming that auto_finish_active bypassed ping-pong detection and
|
||||
# allowed the loop to continue past the first cycle. A regression that
|
||||
# disabled the bypass would stop at visit 1.
|
||||
visit_count = context.results.get("bypass_visit_count")
|
||||
if visit_count is not None:
|
||||
assert visit_count[0] >= 2, (
|
||||
f"Expected agent to be visited >= 2 times (auto_finish_active bypass), "
|
||||
f"but was only visited {visit_count[0]} time(s). "
|
||||
"The auto_finish_active ping-pong bypass may have been removed."
|
||||
)
|
||||
|
||||
|
||||
@then("it should route to the next node after start (pg_cov)")
|
||||
|
||||
@@ -13,7 +13,11 @@ from cleveractors.agent import Agent
|
||||
from cleveractors.config_utils import merge_configs
|
||||
from cleveractors.context_manager import ContextManager
|
||||
from cleveractors.core.application import ReactiveCleverAgentsApp
|
||||
from cleveractors.core.exceptions import CleverAgentsException, ConfigurationError
|
||||
from cleveractors.core.exceptions import (
|
||||
CleverAgentsException,
|
||||
ConfigurationError,
|
||||
ExecutionError,
|
||||
)
|
||||
from cleveractors.result import ActorResult, NodeUsage
|
||||
from cleveractors.runtime import (
|
||||
Executor,
|
||||
@@ -29,6 +33,7 @@ __all__ = [
|
||||
"ConfigurationError",
|
||||
"ContextManager",
|
||||
"create_executor",
|
||||
"ExecutionError",
|
||||
"Executor",
|
||||
"merge_configs",
|
||||
"NodeUsage",
|
||||
|
||||
@@ -35,7 +35,30 @@ class TemplateError(CleverAgentsException):
|
||||
|
||||
|
||||
class ExecutionError(CleverAgentsException):
|
||||
"""Exception raised when agent execution fails."""
|
||||
"""Exception raised when agent execution fails.
|
||||
|
||||
Attributes:
|
||||
kind: Categorical limit type that was breached. One of
|
||||
``'depth'``, ``'model_calls'``, ``'tool_calls'``, ``'timeout'``,
|
||||
``'cost'``, or ``''`` (empty string for non-limit errors).
|
||||
reason: Disambiguating sub-code for ``kind='cost'``. One of
|
||||
``'budget_exhausted'`` (accumulated cost exceeded
|
||||
``max_cost_usd``), ``'missing_pricing_entry'`` (the pricing
|
||||
table supplied to ``create_executor()`` does not contain an
|
||||
entry for the provider/model used by this execution), or ``''``
|
||||
(empty string for other error kinds and non-limit errors).
|
||||
|
||||
All existing ``raise ExecutionError(msg)`` call sites continue to work
|
||||
because both ``kind`` and ``reason`` default to ``""``.
|
||||
|
||||
**ADR reference:** ADR-2029 — Actor Execution Limits and Budget
|
||||
Enforcement via cleveractors-core.
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, kind: str = "", reason: str = "") -> None:
|
||||
super().__init__(message)
|
||||
self.kind = kind
|
||||
self.reason = reason
|
||||
|
||||
|
||||
class ApplicationError(CleverAgentsException):
|
||||
|
||||
@@ -8,6 +8,7 @@ allowing it to work properly in run mode without timeouts.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import copy
|
||||
import logging
|
||||
from collections import defaultdict, deque
|
||||
from dataclasses import dataclass, field
|
||||
@@ -16,6 +17,7 @@ from typing import Any, Deque, List, Optional, Set
|
||||
|
||||
from cleveractors.agents.base import Agent
|
||||
from cleveractors.context_manager import ContextManager
|
||||
from cleveractors.core.exceptions import ExecutionError
|
||||
from cleveractors.langgraph.dynamic_router import DynamicRouterNode
|
||||
from cleveractors.langgraph.nodes import Edge, Node, NodeConfig, NodeType
|
||||
from cleveractors.langgraph.state import GraphState, StateManager
|
||||
@@ -102,8 +104,29 @@ class PureLangGraph:
|
||||
config: PureGraphConfig,
|
||||
agents: Optional[dict[str, Agent]] = None,
|
||||
context_manager: Optional[ContextManager] = None,
|
||||
limits: dict[str, Any] | None = None,
|
||||
pricing: dict[str, Any] | None = None,
|
||||
):
|
||||
"""Initialize PureLangGraph."""
|
||||
"""Initialize PureLangGraph.
|
||||
|
||||
Args:
|
||||
config: Graph configuration.
|
||||
agents: Pre-created agent instances keyed by name.
|
||||
context_manager: Optional context manager for global state.
|
||||
limits: Execution budget dict from the router. Recognised keys:
|
||||
``max_depth``, ``max_model_calls``, ``max_tool_calls``,
|
||||
``timeout_ms``, ``max_cost_usd``. Any absent key means
|
||||
"no enforcement for that dimension". When ``max_depth`` is
|
||||
absent the library uses ``2**31 - 1`` as the internal ceiling
|
||||
(ADR-2029 default; no real graph reaches this depth).
|
||||
pricing: Pricing table supplied by the router. Structure:
|
||||
``{provider: {model: {"prompt": <rate>, "completion": <rate>}}}``
|
||||
where rates are USD per million tokens. An empty dict (or
|
||||
``None``) disables cost calculation. When non-empty, every
|
||||
LLM node invocation MUST have a matching entry — a missing
|
||||
entry raises ``ExecutionError(kind='cost',
|
||||
reason='missing_pricing_entry')``. (ADR-2029)
|
||||
"""
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.debug(f"PureLangGraph.__init__ called with config type: {type(config)}")
|
||||
@@ -120,6 +143,18 @@ class PureLangGraph:
|
||||
self.context_manager = context_manager
|
||||
self.logger = logging.getLogger(__name__)
|
||||
|
||||
# Execution limits and pricing table (ADR-2029, issue #15).
|
||||
# M-3 fix: store defensive copies so that caller mutation after
|
||||
# construction (e.g. updating pricing mid-flight or sharing an Executor
|
||||
# across concurrent requests) cannot silently change enforcement
|
||||
# behaviour. Both dicts are now security-sensitive (they control
|
||||
# billing enforcement) and warrant the same treatment as
|
||||
# runtime_dispatch.py already applies to executor.config.
|
||||
self._limits: dict[str, Any] = dict(limits) if limits is not None else {}
|
||||
self._pricing: dict[str, Any] = (
|
||||
copy.deepcopy(pricing) if pricing is not None else {}
|
||||
)
|
||||
|
||||
# Initialize nodes
|
||||
self.nodes: dict[str, Node] = {}
|
||||
self._initialize_nodes()
|
||||
@@ -151,6 +186,16 @@ class PureLangGraph:
|
||||
# (node_id, provider, model, prompt_tokens, completion_tokens).
|
||||
self._node_usages: list[NodeUsageTuple] = []
|
||||
|
||||
# Per-execution counters for limit enforcement (issue #15).
|
||||
# n-5 fix: initialise here only to satisfy the type-checker; the
|
||||
# authoritative reset happens at the top of execute() before each
|
||||
# invocation. The __init__ values are never read before execute()
|
||||
# resets them (matching the pattern used for _node_usages,
|
||||
# _execution_path, and _node_message_visits).
|
||||
self._model_call_count: int = 0
|
||||
self._tool_call_count: int = 0
|
||||
self._accumulated_cost: float = 0.0
|
||||
|
||||
def _initialize_nodes(self) -> None:
|
||||
"""Initialize all nodes in the graph."""
|
||||
# Add start and end nodes if not present
|
||||
@@ -327,6 +372,11 @@ class PureLangGraph:
|
||||
self._execution_path = []
|
||||
self._node_usages = []
|
||||
|
||||
# Reset per-execution limit counters (issue #15).
|
||||
self._model_call_count = 0
|
||||
self._tool_call_count = 0
|
||||
self._accumulated_cost = 0.0
|
||||
|
||||
try:
|
||||
# Determine the starting context for this execution
|
||||
if global_context is not None:
|
||||
@@ -373,10 +423,56 @@ class PureLangGraph:
|
||||
}
|
||||
self.state_manager.update_state(init_state_payload, node_id="input")
|
||||
|
||||
# Start execution from entry point
|
||||
output = await self._execute_from_node(
|
||||
self.config.entry_point, input_message
|
||||
)
|
||||
# Start execution from entry point.
|
||||
# Wrap in asyncio.wait_for() when timeout_ms is specified (ADR-2029,
|
||||
# issue #15). asyncio.TimeoutError is caught and re-raised as a
|
||||
# structured ExecutionError so the router can map it to HTTP 429.
|
||||
timeout_ms = self._limits.get("timeout_ms")
|
||||
if timeout_ms is not None:
|
||||
# m-7 fix: validate timeout_ms > 0 before passing to
|
||||
# asyncio.wait_for(). A value of 0 causes immediate timeout
|
||||
# on every request; a negative value has undefined behaviour.
|
||||
# Raise ExecutionError(kind="timeout") with a clear diagnostic
|
||||
# rather than letting asyncio produce a confusing error.
|
||||
#
|
||||
# Bool guard: float(True)==1.0 would silently set a 1ms timeout
|
||||
# (effectively making every request time out); float(False)==0.0
|
||||
# would be caught by the <= 0 check below but with a confusing
|
||||
# message. Reject bools explicitly, matching the pattern used
|
||||
# for max_depth/max_model_calls/max_tool_calls.
|
||||
if isinstance(timeout_ms, bool):
|
||||
raise ExecutionError(
|
||||
f"Invalid timeout_ms value {timeout_ms!r}: "
|
||||
"bool is not a valid timeout",
|
||||
kind="timeout",
|
||||
)
|
||||
try:
|
||||
_timeout_float = float(timeout_ms) / 1000.0
|
||||
except (TypeError, ValueError) as _to_err:
|
||||
raise ExecutionError(
|
||||
f"Invalid timeout_ms value {timeout_ms!r}: {_to_err}",
|
||||
kind="timeout",
|
||||
) from _to_err
|
||||
if _timeout_float <= 0:
|
||||
raise ExecutionError(
|
||||
f"Invalid timeout_ms value {timeout_ms!r}: "
|
||||
"must be a positive number of milliseconds",
|
||||
kind="timeout",
|
||||
)
|
||||
try:
|
||||
output = await asyncio.wait_for(
|
||||
self._execute_from_node(self.config.entry_point, input_message),
|
||||
timeout=_timeout_float,
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
raise ExecutionError(
|
||||
f"Execution timed out after {timeout_ms} ms",
|
||||
kind="timeout",
|
||||
) from None
|
||||
else:
|
||||
output = await self._execute_from_node(
|
||||
self.config.entry_point, input_message
|
||||
)
|
||||
|
||||
# Log output info for debugging truncation issues
|
||||
if output:
|
||||
@@ -440,24 +536,58 @@ class PureLangGraph:
|
||||
f"_execute_from_node called with node_name='{node_name}', message type={type(message)}, depth={depth}"
|
||||
)
|
||||
|
||||
# Prevent infinite recursion - but allow deeper traversal for complex workflows
|
||||
# For workflows with many sections (like scientific paper writer), we need high limits
|
||||
# Each section requires ~20 node visits, so 50 sections = 1000+ visits
|
||||
# Use a generous limit that allows complex workflows to complete
|
||||
max_depth = max(2000, len(self.nodes) * 50)
|
||||
|
||||
if depth > max_depth:
|
||||
self.logger.warning(
|
||||
f"Maximum recursion depth {max_depth} exceeded. Returning current output."
|
||||
)
|
||||
import sys
|
||||
|
||||
print(
|
||||
f"MAX_DEPTH_EXCEEDED node={node_name} depth={depth} max={max_depth}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
# C-2 fix (ADR-2029, issue #15): The terminal "end"/"END" node has no
|
||||
# execution logic and must never be subject to the depth check. Moving
|
||||
# this short-circuit to the very top of _execute_from_node — before the
|
||||
# depth check — ensures that any graph with max_depth ≤ N (number of
|
||||
# real nodes) can still complete: the "end" node is always reached
|
||||
# regardless of the accumulated depth counter.
|
||||
if node_name == "end" or node_name == "END":
|
||||
self.logger.debug(f"Reached terminal node: {node_name}")
|
||||
return message
|
||||
|
||||
# Depth limit enforcement (ADR-2029, issue #15).
|
||||
#
|
||||
# When limits["max_depth"] is supplied by the router, enforce it strictly
|
||||
# and raise ExecutionError(kind="depth") on breach so the router can map
|
||||
# the error to HTTP 429 (ADR-2029).
|
||||
#
|
||||
# M-2 fix: When limits["max_depth"] is NOT supplied, use 2**31-1 as the
|
||||
# internal ceiling per ADR-2029 ("the library uses a very large internal
|
||||
# ceiling of 2**31 - 1 for max_depth"). This replaces the old legacy
|
||||
# heuristic (max(2000, len(self.nodes) * 50)) which deviated from the spec.
|
||||
# Existing callers that do not pass limits are unaffected in practice
|
||||
# because no real graph reaches depth 2**31-1; the only tests that relied
|
||||
# on the heuristic used depth=10000 which is still well below 2**31-1.
|
||||
_max_depth_raw = self._limits.get("max_depth")
|
||||
if _max_depth_raw is not None:
|
||||
# M-4 fix: validate with a narrow try/except so a non-numeric
|
||||
# max_depth value produces a structured ExecutionError(kind="depth")
|
||||
# rather than a bare ValueError that the router cannot map.
|
||||
# Also guard against bool (int(True)==1, int(False)==0) which would
|
||||
# silently set a nonsensical limit.
|
||||
if isinstance(_max_depth_raw, bool):
|
||||
raise ExecutionError(
|
||||
f"Invalid max_depth value {_max_depth_raw!r}: "
|
||||
"bool is not a valid depth limit",
|
||||
kind="depth",
|
||||
)
|
||||
try:
|
||||
_max_depth_enforced: int = int(_max_depth_raw)
|
||||
except (TypeError, ValueError) as _depth_err:
|
||||
raise ExecutionError(
|
||||
f"Invalid max_depth value {_max_depth_raw!r}: {_depth_err}",
|
||||
kind="depth",
|
||||
) from _depth_err
|
||||
else:
|
||||
_max_depth_enforced = 2**31 - 1
|
||||
if depth > _max_depth_enforced:
|
||||
raise ExecutionError(
|
||||
f"Graph depth limit exceeded: "
|
||||
f"depth {depth} > max_depth {_max_depth_enforced}",
|
||||
kind="depth",
|
||||
)
|
||||
|
||||
# Track node visits with message fingerprints to detect actual loops
|
||||
# A loop is when the same node is visited with the same message twice
|
||||
if not hasattr(self, "_node_message_visits"):
|
||||
@@ -500,11 +630,10 @@ class PureLangGraph:
|
||||
f"Node '{node_name}' visited {self._node_message_visits[visit_key]} times "
|
||||
f"with the same message. Stopping execution to return output to user."
|
||||
)
|
||||
import sys
|
||||
|
||||
print(
|
||||
f"LOOP_STOP node={node_name} visits={self._node_message_visits[visit_key]}",
|
||||
file=sys.stderr,
|
||||
self.logger.warning(
|
||||
"LOOP_STOP node=%s visits=%d",
|
||||
node_name,
|
||||
self._node_message_visits[visit_key],
|
||||
)
|
||||
return message
|
||||
|
||||
@@ -550,11 +679,10 @@ class PureLangGraph:
|
||||
f"Detected router-agent ping-pong loop with same agent: {recent}. "
|
||||
f"Stopping execution to return output."
|
||||
)
|
||||
import sys
|
||||
|
||||
print(
|
||||
f"PING_PONG_DETECTED recent={recent} auto_finish={auto_finish_active}",
|
||||
file=sys.stderr,
|
||||
self.logger.warning(
|
||||
"PING_PONG_DETECTED recent=%s auto_finish=%s",
|
||||
recent,
|
||||
auto_finish_active,
|
||||
)
|
||||
if self._execution_path:
|
||||
self._execution_path.pop()
|
||||
@@ -569,11 +697,6 @@ class PureLangGraph:
|
||||
return await self._execute_from_node(next_nodes[0], message, depth + 1)
|
||||
return message
|
||||
|
||||
if node_name == "end" or node_name == "END":
|
||||
# Terminal node - return the message
|
||||
self.logger.debug(f"Reached terminal node: {node_name}")
|
||||
return message
|
||||
|
||||
# Get the node
|
||||
if node_name not in self.nodes:
|
||||
self.logger.error(
|
||||
@@ -587,6 +710,59 @@ class PureLangGraph:
|
||||
self.logger.debug(f"Executing node: {node_name} (type: {node.config.type})")
|
||||
self.execution_history.append(node_name)
|
||||
|
||||
# Execution limit enforcement: model_calls and tool_calls (ADR-2029, issue #15).
|
||||
# Check BEFORE running the node so we never run the (limit + 1)-th invocation.
|
||||
# AGENT-type nodes map to LLM model calls; TOOL-type nodes map to tool calls.
|
||||
if node.config.type == NodeType.AGENT:
|
||||
max_model = self._limits.get("max_model_calls")
|
||||
if max_model is not None:
|
||||
# M-4 fix: validate with a narrow try/except so a non-numeric
|
||||
# or bool value produces a structured ExecutionError.
|
||||
if isinstance(max_model, bool):
|
||||
raise ExecutionError(
|
||||
f"Invalid max_model_calls value {max_model!r}: "
|
||||
"bool is not a valid call limit",
|
||||
kind="model_calls",
|
||||
)
|
||||
try:
|
||||
_max_model_int = int(max_model)
|
||||
except (TypeError, ValueError) as _mc_err:
|
||||
raise ExecutionError(
|
||||
f"Invalid max_model_calls value {max_model!r}: {_mc_err}",
|
||||
kind="model_calls",
|
||||
) from _mc_err
|
||||
if self._model_call_count >= _max_model_int:
|
||||
raise ExecutionError(
|
||||
f"model_calls limit exceeded: "
|
||||
f"{self._model_call_count} >= {max_model}",
|
||||
kind="model_calls",
|
||||
)
|
||||
self._model_call_count += 1
|
||||
elif node.config.type == NodeType.TOOL:
|
||||
max_tool = self._limits.get("max_tool_calls")
|
||||
if max_tool is not None:
|
||||
# M-4 fix: same narrow validation for max_tool_calls.
|
||||
if isinstance(max_tool, bool):
|
||||
raise ExecutionError(
|
||||
f"Invalid max_tool_calls value {max_tool!r}: "
|
||||
"bool is not a valid call limit",
|
||||
kind="tool_calls",
|
||||
)
|
||||
try:
|
||||
_max_tool_int = int(max_tool)
|
||||
except (TypeError, ValueError) as _tc_err:
|
||||
raise ExecutionError(
|
||||
f"Invalid max_tool_calls value {max_tool!r}: {_tc_err}",
|
||||
kind="tool_calls",
|
||||
) from _tc_err
|
||||
if self._tool_call_count >= _max_tool_int:
|
||||
raise ExecutionError(
|
||||
f"tool_calls limit exceeded: "
|
||||
f"{self._tool_call_count} >= {max_tool}",
|
||||
kind="tool_calls",
|
||||
)
|
||||
self._tool_call_count += 1
|
||||
|
||||
# Get current state
|
||||
state = self.state_manager.get_state()
|
||||
|
||||
@@ -631,16 +807,138 @@ class PureLangGraph:
|
||||
# Collect per-node token usage if this agent node reported it
|
||||
# (AC4, issue #14). Node._execute_agent() sets
|
||||
# _node_token_usage in the result dict for LLM agent nodes.
|
||||
# m-1 fix: log a warning when _node_token_usage is present but
|
||||
# not a dict so that malformed values surface in production logs
|
||||
# rather than silently disabling token accounting and cost
|
||||
# enforcement.
|
||||
if _tok_info is not None and not isinstance(_tok_info, dict):
|
||||
self.logger.warning(
|
||||
"Node %s returned non-dict _node_token_usage (%r); "
|
||||
"token accounting skipped",
|
||||
node_name,
|
||||
type(_tok_info).__name__,
|
||||
)
|
||||
if isinstance(_tok_info, dict):
|
||||
# m-2 fix: compute _pt and _ct once at the top of the
|
||||
# isinstance block and reuse them in both _node_usages and
|
||||
# the cost block, eliminating the double _safe_token_int()
|
||||
# call that doubled log noise for malformed inputs.
|
||||
_pt = _safe_token_int(_tok_info.get("prompt_tokens", 0))
|
||||
_ct = _safe_token_int(_tok_info.get("completion_tokens", 0))
|
||||
self._node_usages.append(
|
||||
(
|
||||
str(_tok_info.get("node_id", node_name)),
|
||||
str(_tok_info.get("provider", "unknown")),
|
||||
str(_tok_info.get("model", "unknown")),
|
||||
_safe_token_int(_tok_info.get("prompt_tokens", 0)),
|
||||
_safe_token_int(_tok_info.get("completion_tokens", 0)),
|
||||
_pt,
|
||||
_ct,
|
||||
)
|
||||
)
|
||||
|
||||
# Cost accumulation and enforcement (ADR-2029, issue #15).
|
||||
# Only when the router supplied a non-empty pricing table; an
|
||||
# empty dict means cost tracking was not requested and is skipped
|
||||
# entirely. When the table is non-empty, a missing provider or
|
||||
# model entry is a hard error — proceeding with an assumed zero
|
||||
# price is forbidden because it would enable unbounded spend.
|
||||
if self._pricing:
|
||||
_provider = str(_tok_info.get("provider", "unknown"))
|
||||
_model = str(_tok_info.get("model", "unknown"))
|
||||
# _pt and _ct already computed above (m-2 fix)
|
||||
|
||||
_provider_pricing = self._pricing.get(_provider)
|
||||
if _provider_pricing is None or not isinstance(
|
||||
_provider_pricing, dict
|
||||
):
|
||||
raise ExecutionError(
|
||||
f"Missing pricing entry for provider '{_provider}'",
|
||||
kind="cost",
|
||||
reason="missing_pricing_entry",
|
||||
)
|
||||
_model_pricing = _provider_pricing.get(_model)
|
||||
if _model_pricing is None or not isinstance(
|
||||
_model_pricing, dict
|
||||
):
|
||||
raise ExecutionError(
|
||||
f"Missing pricing entry for model '{_model}' "
|
||||
f"under provider '{_provider}'",
|
||||
kind="cost",
|
||||
reason="missing_pricing_entry",
|
||||
)
|
||||
|
||||
# Validate that both rate keys exist and are numeric
|
||||
# (ADR-2029: "proceeding with a missing or zero price is
|
||||
# forbidden"). A missing key or a non-numeric value is
|
||||
# treated the same as a missing pricing entry — the
|
||||
# library must not silently fall back to $0/token.
|
||||
_prompt_rate_raw = _model_pricing.get("prompt")
|
||||
_completion_rate_raw = _model_pricing.get("completion")
|
||||
if _prompt_rate_raw is None or _completion_rate_raw is None:
|
||||
raise ExecutionError(
|
||||
f"Incomplete pricing entry for model '{_model}' "
|
||||
f"under provider '{_provider}': missing 'prompt' "
|
||||
f"or 'completion' rate key",
|
||||
kind="cost",
|
||||
reason="missing_pricing_entry",
|
||||
)
|
||||
try:
|
||||
# Rates are USD per million tokens (ADR-2029 example:
|
||||
# gpt-4.1-mini prompt=$0.15/1M, completion=$0.60/1M).
|
||||
_prompt_rate = float(_prompt_rate_raw)
|
||||
_completion_rate = float(_completion_rate_raw)
|
||||
except (TypeError, ValueError) as _rate_err:
|
||||
raise ExecutionError(
|
||||
f"Invalid pricing rate for model '{_model}' "
|
||||
f"under provider '{_provider}': {_rate_err}",
|
||||
kind="cost",
|
||||
reason="missing_pricing_entry",
|
||||
) from _rate_err
|
||||
_node_cost = (
|
||||
_pt / 1_000_000.0 * _prompt_rate
|
||||
+ _ct / 1_000_000.0 * _completion_rate
|
||||
)
|
||||
self._accumulated_cost += _node_cost
|
||||
|
||||
# C-1 fix (ADR-2029, issue #15): coerce max_cost_usd
|
||||
# with a narrow try/except BEFORE the broad
|
||||
# "except Exception" handler so that a malformed value
|
||||
# (e.g. a non-numeric string) raises ExecutionError
|
||||
# immediately rather than being silently swallowed by
|
||||
# the broad handler and causing the graph to return the
|
||||
# input string as if execution succeeded.
|
||||
_max_cost = self._limits.get("max_cost_usd")
|
||||
if _max_cost is not None:
|
||||
# Bool guard: float(True)==1.0 / float(False)==0.0
|
||||
# would silently set a nonsensical budget. Reject
|
||||
# bools before the float() conversion, matching the
|
||||
# pattern used for max_depth/max_model_calls/max_tool_calls.
|
||||
if isinstance(_max_cost, bool):
|
||||
raise ExecutionError(
|
||||
f"Invalid max_cost_usd value {_max_cost!r}: "
|
||||
"bool is not a valid cost limit",
|
||||
kind="cost",
|
||||
)
|
||||
try:
|
||||
_max_cost_float = float(_max_cost)
|
||||
except (TypeError, ValueError) as _cost_err:
|
||||
# A malformed limit value (e.g. a non-numeric string)
|
||||
# is a router configuration error, not a missing pricing
|
||||
# entry. Use reason="" (the documented default for
|
||||
# non-cost-specific errors per ADR-2029) so router logs
|
||||
# and alerts correctly identify the root cause.
|
||||
raise ExecutionError(
|
||||
f"Invalid max_cost_usd value "
|
||||
f"{_max_cost!r}: {_cost_err}",
|
||||
kind="cost",
|
||||
) from _cost_err
|
||||
if self._accumulated_cost > _max_cost_float:
|
||||
raise ExecutionError(
|
||||
f"Cost limit exceeded: "
|
||||
f"{self._accumulated_cost:.6f} USD "
|
||||
f"> {_max_cost} USD",
|
||||
kind="cost",
|
||||
reason="budget_exhausted",
|
||||
)
|
||||
else:
|
||||
output_message = result
|
||||
|
||||
@@ -662,6 +960,11 @@ class PureLangGraph:
|
||||
)
|
||||
break
|
||||
|
||||
except ExecutionError:
|
||||
# Limit-enforcement errors (depth, model_calls, tool_calls, timeout,
|
||||
# cost) must propagate without suppression so the router can map
|
||||
# them to the correct HTTP status code (ADR-2029, issue #15).
|
||||
raise
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error executing node {node_name}: {e}", exc_info=True)
|
||||
output_message = message
|
||||
@@ -716,13 +1019,38 @@ class PureLangGraph:
|
||||
|
||||
# Execute next nodes
|
||||
if self.config.parallel_execution and len(next_nodes) > 1:
|
||||
# Execute nodes in parallel
|
||||
# Execute nodes in parallel.
|
||||
# M-1 fix (ADR-2029, issue #15): when one branch raises an
|
||||
# exception (e.g. ExecutionError for a limit breach), cancel all
|
||||
# sibling tasks immediately so they do not continue spending budget
|
||||
# after the limit has already been exceeded. asyncio.gather()
|
||||
# with the default return_exceptions=False propagates the first
|
||||
# exception to the awaiter but does NOT cancel the other in-flight
|
||||
# tasks — we must do that explicitly.
|
||||
self.logger.debug(f"Executing {len(next_nodes)} nodes in parallel")
|
||||
tasks = [
|
||||
self._execute_from_node(next_node, output_message, depth + 1)
|
||||
asyncio.create_task(
|
||||
self._execute_from_node(next_node, output_message, depth + 1)
|
||||
)
|
||||
for next_node in next_nodes
|
||||
]
|
||||
results = await asyncio.gather(*tasks)
|
||||
try:
|
||||
results = await asyncio.gather(*tasks)
|
||||
except BaseException:
|
||||
# Catch BaseException (not just Exception) because
|
||||
# asyncio.CancelledError is a BaseException in Python 3.8+.
|
||||
# When the outer asyncio.wait_for() timeout fires it raises
|
||||
# CancelledError, which would not be caught by `except
|
||||
# Exception` — leaving sibling tasks running indefinitely.
|
||||
# The `raise` at the end re-propagates the original exception
|
||||
# (including KeyboardInterrupt/SystemExit) so nothing is
|
||||
# swallowed.
|
||||
for t in tasks:
|
||||
if not t.done():
|
||||
t.cancel()
|
||||
# Await cancelled tasks to suppress CancelledError noise
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
raise
|
||||
# Clean up execution path for parallel execution
|
||||
if hasattr(self, "_execution_path") and self._execution_path:
|
||||
self._execution_path.pop()
|
||||
|
||||
@@ -382,7 +382,12 @@ async def _execute_graph(
|
||||
if conversation_history:
|
||||
global_context["conversation_history"] = conversation_history
|
||||
|
||||
graph = PureLangGraph(config=pg_config, agents=agents)
|
||||
graph = PureLangGraph(
|
||||
config=pg_config,
|
||||
agents=agents,
|
||||
limits=executor.limits,
|
||||
pricing=executor.pricing,
|
||||
)
|
||||
# PureLangGraph.execute() returns a 3-tuple:
|
||||
# (response_str, final_state_dict, node_usages_list) where
|
||||
# node_usages_list is a list of (node_id, provider, model,
|
||||
|
||||
Reference in New Issue
Block a user