hurui200320
17d99abce7
CI / quality (pull_request) Successful in 36s
CI / lint (pull_request) Successful in 39s
CI / security (pull_request) Successful in 58s
CI / build (pull_request) Successful in 58s
CI / typecheck (pull_request) Successful in 1m2s
CI / integration_tests (pull_request) Successful in 1m14s
CI / unit_tests (pull_request) Successful in 3m35s
CI / coverage (pull_request) Successful in 3m34s
CI / status-check (pull_request) Successful in 3s
CI / lint (push) Successful in 47s
CI / typecheck (push) Successful in 46s
CI / build (push) Successful in 53s
CI / quality (push) Successful in 1m4s
CI / security (push) Successful in 1m7s
CI / integration_tests (push) Successful in 1m13s
CI / unit_tests (push) Successful in 3m43s
CI / coverage (push) Successful in 3m42s
CI / status-check (push) Successful in 3s
AC1: ExecutionError gains kind (categorical: depth, model_calls, tool_calls,
timeout, cost) and reason (sub-code: budget_exhausted or
missing_pricing_entry) fields, both defaulting to empty string. All existing
raise ExecutionError(msg) call sites are backward-compatible.
AC2: PureLangGraph.__init__ accepts limits: dict[str, Any] and
pricing: dict[str, Any] (both default to {}). When limits['max_depth'] is
supplied, a depth breach raises ExecutionError(kind='depth') instead of
silently returning the current message (the old behaviour was a silent
data-loss bug). When no max_depth limit is supplied, the legacy heuristic
(max(2000, len(nodes)*50)) is used with a silent cap for backward
compatibility with existing callers that do not pass limits.
AC3: max_model_calls is checked before each NodeType.AGENT execution;
counter increments after the check so the limit is enforced before the
(limit+1)-th invocation. Breach raises ExecutionError(kind='model_calls').
AC4: max_tool_calls is checked before each NodeType.TOOL execution; same
pattern as model_calls. Breach raises ExecutionError(kind='tool_calls').
AC5: execute() wraps _execute_from_node() in asyncio.wait_for() when
limits['timeout_ms'] is set. asyncio.TimeoutError is caught and re-raised
as ExecutionError(kind='timeout') so the router can map it to HTTP 429.
AC6+AC7: After each LLM node, cost is computed from the pricing table
(rates are USD per million tokens, matching ADR-2029 example values such
as gpt-4.1-mini prompt=/bin/zsh.15/1M). Cost breach raises
ExecutionError(kind='cost', reason='budget_exhausted'). A missing
provider or model entry raises ExecutionError(kind='cost',
reason='missing_pricing_entry') — proceeding with assumed zero cost is
forbidden per ADR-2029. An empty pricing dict ({}) disables all cost
checking (cost tracking was not requested).
AC8: ExecutionError re-exported from cleveractors/__init__.py and
added to __all__.
Wire: runtime_dispatch._execute_graph() now passes executor.limits and
executor.pricing to PureLangGraph so the limits reach the enforcement
layer.
ExecutionError is also re-raised (not swallowed) in _execute_from_node()'s
broad except block so limit errors always propagate to the caller.
New BDD tests (features/execution_limits.feature, 19 scenarios):
- ExecutionError defaults kind and reason to empty strings
- ExecutionError accepts kind and reason keyword arguments
- ExecutionError exported from cleveractors top-level package
- Graph exceeding max_depth raises ExecutionError(kind='depth')
- Two AGENT nodes with max_model_calls=1 raises on 2nd call
- Two TOOL nodes with max_tool_calls=1 raises on 2nd call
- Slow node with timeout_ms=50 raises ExecutionError(kind='timeout')
- LLM node exceeding max_cost_usd raises budget_exhausted
- Unknown provider/model in pricing raises missing_pricing_entry
- Empty pricing table skips cost calculation entirely
- Executor limits/pricing propagation verified
Quality gates: lint pass, typecheck 0 errors,
unit_tests 2317/2317 pass, integration_tests pass,
coverage 97.17% (threshold 96.5%).
ISSUES CLOSED: #15
227 lines
13 KiB
Gherkin
227 lines
13 KiB
Gherkin
Feature: Execution Limits Enforcement in PureLangGraph (ADR-2029)
|
|
As the CleverThis router
|
|
I want PureLangGraph to enforce per-request execution limits
|
|
So that budget exhaustion, depth breaches, and cost violations surface as
|
|
structured ExecutionError exceptions with the correct kind/reason fields
|
|
|
|
Background:
|
|
Given the execution limits test context is initialised (elim)
|
|
|
|
# ── ExecutionError structured fields ──────────────────────────────────────
|
|
|
|
Scenario: ExecutionError defaults kind and reason to empty strings
|
|
When an ExecutionError is raised with only a message (elim)
|
|
Then the kind field should be empty string (elim)
|
|
And the reason field should be empty string (elim)
|
|
|
|
Scenario: ExecutionError accepts kind with empty reason
|
|
When an ExecutionError is raised with kind "depth" and empty reason (elim)
|
|
Then the kind field should be "depth" (elim)
|
|
And the reason field should be empty string (elim)
|
|
|
|
Scenario: ExecutionError accepts kind cost with reason budget_exhausted
|
|
When an ExecutionError is raised with kind "cost" and reason "budget_exhausted" (elim)
|
|
Then the kind field should be "cost" (elim)
|
|
And the reason field should be "budget_exhausted" (elim)
|
|
|
|
Scenario: ExecutionError accepts kind cost with reason missing_pricing_entry
|
|
When an ExecutionError is raised with kind "cost" and reason "missing_pricing_entry" (elim)
|
|
Then the kind field should be "cost" (elim)
|
|
And the reason field should be "missing_pricing_entry" (elim)
|
|
|
|
Scenario: ExecutionError is exported from cleveractors top-level package
|
|
When ExecutionError is imported from cleveractors (elim)
|
|
Then the import should succeed and expose kind and reason attributes (elim)
|
|
|
|
# ── Depth limit ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Graph exceeding max_depth raises ExecutionError with kind "depth"
|
|
Given a linear 3-node graph with max_depth 1 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "depth" (elim)
|
|
|
|
Scenario: Graph within max_depth completes successfully
|
|
Given a linear 2-node graph with max_depth 10 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
|
|
# ── Model-call limit ───────────────────────────────────────────────────────
|
|
|
|
Scenario: Two AGENT nodes with max_model_calls 1 raises on second call
|
|
Given a graph with 2 AGENT nodes and max_model_calls 1 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
|
|
|
Scenario: Two AGENT nodes with max_model_calls 2 completes successfully
|
|
Given a graph with 2 AGENT nodes and max_model_calls 2 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
|
|
# ── Tool-call limit ────────────────────────────────────────────────────────
|
|
|
|
Scenario: Two TOOL nodes with max_tool_calls 1 raises on second call
|
|
Given a graph with 2 TOOL nodes and max_tool_calls 1 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "tool_calls" (elim)
|
|
|
|
Scenario: Two TOOL nodes with max_tool_calls 2 completes successfully
|
|
Given a graph with 2 TOOL nodes and max_tool_calls 2 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
|
|
# ── Timeout ───────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Graph with slow node times out and raises ExecutionError kind "timeout"
|
|
Given a graph with a slow node and timeout_ms 50 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "timeout" (elim)
|
|
|
|
Scenario: Graph completing before timeout succeeds
|
|
Given a graph with a fast node and timeout_ms 2000 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
|
|
# ── Cost limit — budget_exhausted ─────────────────────────────────────────
|
|
|
|
Scenario: LLM node with high token count exceeding cost limit raises budget_exhausted
|
|
Given a graph with an LLM node using 2000000 prompt tokens and max_cost_usd 0.10 (elim)
|
|
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "budget_exhausted" (elim)
|
|
|
|
Scenario: LLM node with low token count within cost limit completes successfully
|
|
Given a graph with an LLM node using 100 prompt tokens and max_cost_usd 1.00 (elim)
|
|
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
|
|
# ── Cost limit — missing_pricing_entry ────────────────────────────────────
|
|
|
|
Scenario: LLM node with unknown provider raises missing_pricing_entry
|
|
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
|
|
When the graph is executed with pricing only for "anthropic" provider (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "missing_pricing_entry" (elim)
|
|
|
|
Scenario: LLM node with unknown model raises missing_pricing_entry
|
|
Given a graph with an LLM node for provider "openai" model "unknown-model" (elim)
|
|
When the graph is executed with pricing for "openai" but not "unknown-model" (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "missing_pricing_entry" (elim)
|
|
|
|
Scenario: Empty pricing table skips cost calculation entirely
|
|
Given a graph with an LLM node using 2000000 prompt tokens and max_cost_usd 0.001 (elim)
|
|
When the graph is executed with an empty pricing table (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
|
|
Scenario: LLM node with incomplete model pricing entry missing prompt rate raises missing_pricing_entry
|
|
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
|
|
When the graph is executed with model pricing entry missing the prompt rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "missing_pricing_entry" (elim)
|
|
|
|
Scenario: LLM node with non-numeric pricing rate raises missing_pricing_entry
|
|
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
|
|
When the graph is executed with a non-numeric prompt rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "missing_pricing_entry" (elim)
|
|
|
|
# ── Cost accumulation across multiple LLM nodes ───────────────────────────
|
|
|
|
Scenario: Cost accumulates across two LLM nodes and triggers budget_exhausted on second
|
|
Given a graph with 2 LLM nodes each using 1000000 prompt tokens and max_cost_usd 0.25 (elim)
|
|
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "budget_exhausted" (elim)
|
|
|
|
# ── Completion tokens cost contribution ───────────────────────────────────
|
|
|
|
Scenario: Completion tokens contribute to cost and can trigger budget_exhausted
|
|
Given a graph with an LLM node using 0 prompt tokens 1000000 completion tokens and max_cost_usd 0.50 (elim)
|
|
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt 0.60 completion rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "budget_exhausted" (elim)
|
|
|
|
# ── Malformed limit value validation ─────────────────────────────────────
|
|
|
|
Scenario: Non-numeric max_cost_usd raises ExecutionError with kind "cost"
|
|
Given a graph with an LLM node for non-numeric max_cost_usd validation (elim)
|
|
When the graph is executed with pricing and non-numeric max_cost_usd (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
|
|
Scenario: Non-numeric max_model_calls raises ExecutionError with kind "model_calls"
|
|
Given a graph with 1 AGENT node and non-numeric max_model_calls (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
|
|
|
Scenario: Bool max_model_calls raises ExecutionError with kind "model_calls"
|
|
Given a graph with 1 AGENT node and bool False max_model_calls (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
|
|
|
Scenario: Non-numeric max_tool_calls raises ExecutionError with kind "tool_calls"
|
|
Given a graph with 1 TOOL node and non-numeric max_tool_calls (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "tool_calls" (elim)
|
|
|
|
Scenario: Bool max_tool_calls raises ExecutionError with kind "tool_calls"
|
|
Given a graph with 1 TOOL node and bool False max_tool_calls (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "tool_calls" (elim)
|
|
|
|
Scenario: Non-positive timeout_ms raises ExecutionError with kind "timeout"
|
|
Given a graph with a fast node and timeout_ms 0 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "timeout" (elim)
|
|
|
|
Scenario: Bool timeout_ms raises ExecutionError with kind "timeout"
|
|
Given a graph with a fast node and bool True timeout_ms (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "timeout" (elim)
|
|
|
|
Scenario: Non-numeric max_depth raises ExecutionError with kind "depth"
|
|
Given a graph with a fast node and non-numeric max_depth (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "depth" (elim)
|
|
|
|
Scenario: Bool max_depth raises ExecutionError with kind "depth"
|
|
Given a graph with a fast node and bool False max_depth (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "depth" (elim)
|
|
|
|
Scenario: Bool max_cost_usd raises ExecutionError with kind "cost"
|
|
Given a graph with an LLM node for bool max_cost_usd validation (elim)
|
|
When the graph is executed with pricing and bool False max_cost_usd (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
|
|
Scenario: Parallel graph cancels sibling tasks when one branch raises ExecutionError
|
|
Given a parallel graph where one branch raises an ExecutionError (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
|
And the sibling branch should have been cancelled (elim)
|
|
|
|
Scenario: Non-dict _node_token_usage logs warning and skips token accounting
|
|
Given a graph with an LLM node that returns non-dict token usage (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
And a warning should have been logged for non-dict token usage (elim)
|
|
|
|
# ── Happy-path result correctness ─────────────────────────────────────────
|
|
|
|
Scenario: Graph within max_depth returns non-None result
|
|
Given a linear 2-node graph with max_depth 10 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
And the result should be non-None (elim)
|
|
|
|
# ── PureLangGraph limits/pricing wired from Executor ──────────────────────
|
|
|
|
Scenario: limits and pricing are stored on Executor and propagated to PureLangGraph
|
|
Given an Executor with graph config and limits max_depth 100 pricing for openai (elim)
|
|
When the executor is checked for limit/pricing propagation (elim)
|
|
Then the PureLangGraph should have the correct limits and pricing (elim)
|
|
|
|
Scenario: Executor.execute() enforces max_depth through the full dispatch path (elim)
|
|
Given an Executor with a 2-node graph and max_depth 0 (elim)
|
|
When the executor execute method is called with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "depth" (elim)
|