CoreRasurae
a8f68b8262
CI / lint (pull_request) Successful in 50s
CI / typecheck (pull_request) Successful in 55s
CI / security (pull_request) Successful in 53s
CI / quality (pull_request) Successful in 38s
CI / unit_tests (pull_request) Successful in 3m15s
CI / integration_tests (pull_request) Successful in 1m14s
CI / build (pull_request) Successful in 39s
CI / coverage (pull_request) Successful in 3m16s
CI / status-check (pull_request) Successful in 3s
CI / lint (push) Successful in 37s
CI / typecheck (push) Successful in 56s
CI / security (push) Successful in 54s
CI / quality (push) Successful in 37s
CI / unit_tests (push) Successful in 3m15s
CI / integration_tests (push) Successful in 1m14s
CI / build (push) Successful in 39s
CI / coverage (push) Successful in 3m16s
CI / status-check (push) Successful in 3s
Enforces the USD budget (max_cost_usd) as a hard pre-flight gate before every node execution and at each main-agent ainvoke round inside the multi-turn tool loop, with pruning passes silently skipped when the budget is exhausted. Pre-flight gate (pure_graph.py): _check_budget_pre_flight() raises ExecutionError(kind='cost', reason='budget_exhausted') immediately before node execution in both _execute_from_node and all three branches of _stream_from_node. In-node budget gating (llm.py): _check_budget_before_invoke() gates every _retry_ainvoke() call inside _execute_tool_loop, process_message, and stream_message. _should_skip_pruning() skips pruning passes when budget exhausted. _accumulate_cost_after_invoke() computes USD cost from token counts using the pricing table and advances current_accumulated_cost so subsequent budget checks see the updated cost. ContextVar plumbing (retry.py, pure_graph.py): current_accumulated_cost, current_max_cost_usd, and current_pricing ContextVars carry per-node budget state from PureLangGraph into LLMAgent. Set before node.execute() via _set_budget_context_for_node, reset in finally blocks via _reset_budget_context. Reviewer fixes (rui.hu #80): - Move cost accumulation inside try block in terminal streaming branch to prevent ContextVar leak (was set after finally reset). - Standardize log precision to $%.6f across all budget messages. - Remove orphaned 'already exhausted' step (criterion unreachable). - Add dedicated streaming budget enforcement scenario. - Remove redundant = None default on _reset_budget_context pricing_token. ISSUES CLOSED: #76
248 lines
15 KiB
Gherkin
248 lines
15 KiB
Gherkin
Feature: Execution Limits Enforcement in PureLangGraph (ADR-2029)
|
|
As the CleverThis router
|
|
I want PureLangGraph to enforce per-request execution limits
|
|
So that budget exhaustion, depth breaches, and cost violations surface as
|
|
structured ExecutionError exceptions with the correct kind/reason fields
|
|
|
|
Background:
|
|
Given the execution limits test context is initialised (elim)
|
|
|
|
# ── ExecutionError structured fields ──────────────────────────────────────
|
|
|
|
Scenario: ExecutionError defaults kind and reason to empty strings
|
|
When an ExecutionError is raised with only a message (elim)
|
|
Then the kind field should be empty string (elim)
|
|
And the reason field should be empty string (elim)
|
|
|
|
Scenario: ExecutionError accepts kind with empty reason
|
|
When an ExecutionError is raised with kind "depth" and empty reason (elim)
|
|
Then the kind field should be "depth" (elim)
|
|
And the reason field should be empty string (elim)
|
|
|
|
Scenario: ExecutionError accepts kind cost with reason budget_exhausted
|
|
When an ExecutionError is raised with kind "cost" and reason "budget_exhausted" (elim)
|
|
Then the kind field should be "cost" (elim)
|
|
And the reason field should be "budget_exhausted" (elim)
|
|
|
|
Scenario: ExecutionError accepts kind cost with reason missing_pricing_entry
|
|
When an ExecutionError is raised with kind "cost" and reason "missing_pricing_entry" (elim)
|
|
Then the kind field should be "cost" (elim)
|
|
And the reason field should be "missing_pricing_entry" (elim)
|
|
|
|
Scenario: ExecutionError is exported from cleveractors top-level package
|
|
When ExecutionError is imported from cleveractors (elim)
|
|
Then the import should succeed and expose kind and reason attributes (elim)
|
|
|
|
# ── Depth limit ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Graph exceeding max_depth raises ExecutionError with kind "depth"
|
|
Given a linear 3-node graph with max_depth 1 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "depth" (elim)
|
|
|
|
Scenario: Graph within max_depth completes successfully
|
|
Given a linear 2-node graph with max_depth 10 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
|
|
# ── Model-call limit ───────────────────────────────────────────────────────
|
|
|
|
Scenario: Two AGENT nodes with max_model_calls 1 raises on second call
|
|
Given a graph with 2 AGENT nodes and max_model_calls 1 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
|
|
|
Scenario: Two AGENT nodes with max_model_calls 2 completes successfully
|
|
Given a graph with 2 AGENT nodes and max_model_calls 2 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
|
|
# ── Tool-call limit ────────────────────────────────────────────────────────
|
|
|
|
Scenario: Two TOOL nodes with max_tool_calls 1 raises on second call
|
|
Given a graph with 2 TOOL nodes and max_tool_calls 1 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "tool_calls" (elim)
|
|
|
|
Scenario: Two TOOL nodes with max_tool_calls 2 completes successfully
|
|
Given a graph with 2 TOOL nodes and max_tool_calls 2 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
|
|
# ── Timeout ───────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Graph with slow node times out and raises ExecutionError kind "timeout"
|
|
Given a graph with a slow node and timeout_ms 50 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "timeout" (elim)
|
|
|
|
Scenario: Graph completing before timeout succeeds
|
|
Given a graph with a fast node and timeout_ms 2000 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
|
|
# ── Cost limit — budget_exhausted ─────────────────────────────────────────
|
|
|
|
Scenario: LLM node with high token count exceeding cost limit raises budget_exhausted
|
|
Given a graph with an LLM node using 2000000 prompt tokens and max_cost_usd 0.10 (elim)
|
|
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "budget_exhausted" (elim)
|
|
|
|
Scenario: LLM node with low token count within cost limit completes successfully
|
|
Given a graph with an LLM node using 100 prompt tokens and max_cost_usd 1.00 (elim)
|
|
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
|
|
# ── Cost limit — missing_pricing_entry ────────────────────────────────────
|
|
|
|
Scenario: LLM node with unknown provider raises missing_pricing_entry
|
|
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
|
|
When the graph is executed with pricing only for "anthropic" provider (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "missing_pricing_entry" (elim)
|
|
|
|
Scenario: LLM node with unknown model raises missing_pricing_entry
|
|
Given a graph with an LLM node for provider "openai" model "unknown-model" (elim)
|
|
When the graph is executed with pricing for "openai" but not "unknown-model" (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "missing_pricing_entry" (elim)
|
|
|
|
Scenario: Empty pricing table raises missing_pricing_entry when limit is specified
|
|
Given a graph with an LLM node using 2000000 prompt tokens and max_cost_usd 0.001 (elim)
|
|
When the graph is executed with an empty pricing table (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "missing_pricing_entry" (elim)
|
|
|
|
Scenario: LLM node with incomplete model pricing entry missing prompt rate raises missing_pricing_entry
|
|
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
|
|
When the graph is executed with model pricing entry missing the prompt rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "missing_pricing_entry" (elim)
|
|
|
|
Scenario: LLM node with non-numeric pricing rate raises missing_pricing_entry
|
|
Given a graph with an LLM node for provider "openai" model "gpt-4.1-mini" (elim)
|
|
When the graph is executed with a non-numeric prompt rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "missing_pricing_entry" (elim)
|
|
|
|
# ── Cost accumulation across multiple LLM nodes ───────────────────────────
|
|
|
|
Scenario: Cost accumulates across two LLM nodes and triggers budget_exhausted on second
|
|
Given a graph with 2 LLM nodes each using 1000000 prompt tokens and max_cost_usd 0.25 (elim)
|
|
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "budget_exhausted" (elim)
|
|
|
|
# ── Completion tokens cost contribution ───────────────────────────────────
|
|
|
|
Scenario: Completion tokens contribute to cost and can trigger budget_exhausted
|
|
Given a graph with an LLM node using 0 prompt tokens 1000000 completion tokens and max_cost_usd 0.50 (elim)
|
|
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt 0.60 completion rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "budget_exhausted" (elim)
|
|
|
|
# ── Malformed limit value validation ─────────────────────────────────────
|
|
|
|
Scenario: Non-numeric max_cost_usd raises ExecutionError with kind "cost"
|
|
Given a graph with an LLM node for non-numeric max_cost_usd validation (elim)
|
|
When the graph is executed with pricing and non-numeric max_cost_usd (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
|
|
Scenario: Non-numeric max_model_calls raises ExecutionError with kind "model_calls"
|
|
Given a graph with 1 AGENT node and non-numeric max_model_calls (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
|
|
|
Scenario: Bool max_model_calls raises ExecutionError with kind "model_calls"
|
|
Given a graph with 1 AGENT node and bool False max_model_calls (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
|
|
|
Scenario: Non-numeric max_tool_calls raises ExecutionError with kind "tool_calls"
|
|
Given a graph with 1 TOOL node and non-numeric max_tool_calls (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "tool_calls" (elim)
|
|
|
|
Scenario: Bool max_tool_calls raises ExecutionError with kind "tool_calls"
|
|
Given a graph with 1 TOOL node and bool False max_tool_calls (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "tool_calls" (elim)
|
|
|
|
Scenario: Non-positive timeout_ms raises ExecutionError with kind "timeout"
|
|
Given a graph with a fast node and timeout_ms 0 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "timeout" (elim)
|
|
|
|
Scenario: Bool timeout_ms raises ExecutionError with kind "timeout"
|
|
Given a graph with a fast node and bool True timeout_ms (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "timeout" (elim)
|
|
|
|
Scenario: Non-numeric max_depth raises ExecutionError with kind "depth"
|
|
Given a graph with a fast node and non-numeric max_depth (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "depth" (elim)
|
|
|
|
Scenario: Bool max_depth raises ExecutionError with kind "depth"
|
|
Given a graph with a fast node and bool False max_depth (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "depth" (elim)
|
|
|
|
Scenario: Bool max_cost_usd raises ExecutionError with kind "cost"
|
|
Given a graph with an LLM node for bool max_cost_usd validation (elim)
|
|
When the graph is executed with pricing and bool False max_cost_usd (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
|
|
Scenario: Parallel graph cancels sibling tasks when one branch raises ExecutionError
|
|
Given a parallel graph where one branch raises an ExecutionError (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "model_calls" (elim)
|
|
And the sibling branch should have been cancelled (elim)
|
|
|
|
Scenario: Non-dict _node_token_usage logs warning and skips token accounting
|
|
Given a graph with an LLM node that returns non-dict token usage (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
And a warning should have been logged for non-dict token usage (elim)
|
|
|
|
# ── Budget pre-flight enforcement (issue #76) ──────────────────────────────
|
|
|
|
Scenario: Zero max_cost_usd triggers pre-flight failure before first node
|
|
Given a graph with an LLM node using 100 prompt tokens and max_cost_usd 0.00 (elim)
|
|
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "budget_exhausted" (elim)
|
|
|
|
Scenario: Two LLM nodes exhaust budget via post-node check on first node
|
|
Given a graph with 2 LLM nodes each using 3000000 prompt tokens and max_cost_usd 0.40 (elim)
|
|
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "budget_exhausted" (elim)
|
|
|
|
Scenario: Second LLM node blocked by pre-flight when first node exactly hits limit
|
|
Given a graph with 2 LLM nodes each using 2000000 prompt tokens and max_cost_usd 0.30 (elim)
|
|
When the graph is executed with pricing for "openai"/"gpt-4.1-mini" at 0.15 prompt rate (elim)
|
|
Then an ExecutionError should be raised with kind "cost" (elim)
|
|
And the reason should be "budget_exhausted" (elim)
|
|
|
|
# ── Happy-path result correctness ─────────────────────────────────────────
|
|
|
|
Scenario: Graph within max_depth returns non-None result
|
|
Given a linear 2-node graph with max_depth 10 (elim)
|
|
When the graph is executed with a test message (elim)
|
|
Then no ExecutionError should be raised (elim)
|
|
And the result should be non-None (elim)
|
|
|
|
# ── PureLangGraph limits/pricing wired from Executor ──────────────────────
|
|
|
|
Scenario: limits and pricing are stored on Executor and propagated to PureLangGraph
|
|
Given an Executor with graph config and limits max_depth 100 pricing for openai (elim)
|
|
When the executor is checked for limit/pricing propagation (elim)
|
|
Then the PureLangGraph should have the correct limits and pricing (elim)
|
|
|
|
Scenario: Executor.execute() enforces max_depth through the full dispatch path (elim)
|
|
Given an Executor with a 2-node graph and max_depth 0 (elim)
|
|
When the executor execute method is called with a test message (elim)
|
|
Then an ExecutionError should be raised with kind "depth" (elim)
|