master
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a8f68b8262
|
fix(graph): enforce USD budget at each agent invocation with pre-flight gate
CI / lint (pull_request) Successful in 50s
CI / typecheck (pull_request) Successful in 55s
CI / security (pull_request) Successful in 53s
CI / quality (pull_request) Successful in 38s
CI / unit_tests (pull_request) Successful in 3m15s
CI / integration_tests (pull_request) Successful in 1m14s
CI / build (pull_request) Successful in 39s
CI / coverage (pull_request) Successful in 3m16s
CI / status-check (pull_request) Successful in 3s
CI / lint (push) Successful in 37s
CI / typecheck (push) Successful in 56s
CI / security (push) Successful in 54s
CI / quality (push) Successful in 37s
CI / unit_tests (push) Successful in 3m15s
CI / integration_tests (push) Successful in 1m14s
CI / build (push) Successful in 39s
CI / coverage (push) Successful in 3m16s
CI / status-check (push) Successful in 3s
Enforces the USD budget (max_cost_usd) as a hard pre-flight gate before every node execution and at each main-agent ainvoke round inside the multi-turn tool loop, with pruning passes silently skipped when the budget is exhausted. Pre-flight gate (pure_graph.py): _check_budget_pre_flight() raises ExecutionError(kind='cost', reason='budget_exhausted') immediately before node execution in both _execute_from_node and all three branches of _stream_from_node. In-node budget gating (llm.py): _check_budget_before_invoke() gates every _retry_ainvoke() call inside _execute_tool_loop, process_message, and stream_message. _should_skip_pruning() skips pruning passes when budget exhausted. _accumulate_cost_after_invoke() computes USD cost from token counts using the pricing table and advances current_accumulated_cost so subsequent budget checks see the updated cost. ContextVar plumbing (retry.py, pure_graph.py): current_accumulated_cost, current_max_cost_usd, and current_pricing ContextVars carry per-node budget state from PureLangGraph into LLMAgent. Set before node.execute() via _set_budget_context_for_node, reset in finally blocks via _reset_budget_context. Reviewer fixes (rui.hu #80): - Move cost accumulation inside try block in terminal streaming branch to prevent ContextVar leak (was set after finally reset). - Standardize log precision to $%.6f across all budget messages. - Remove orphaned 'already exhausted' step (criterion unreachable). - Add dedicated streaming budget enforcement scenario. - Remove redundant = None default on _reset_budget_context pricing_token. ISSUES CLOSED: #76 |
||
|
|
17d99abce7 |
feat(execution-limits): add structured ExecutionError kind/reason fields; enforce all 5 execution limits in PureLangGraph
CI / quality (pull_request) Successful in 36s
CI / lint (pull_request) Successful in 39s
CI / security (pull_request) Successful in 58s
CI / build (pull_request) Successful in 58s
CI / typecheck (pull_request) Successful in 1m2s
CI / integration_tests (pull_request) Successful in 1m14s
CI / unit_tests (pull_request) Successful in 3m35s
CI / coverage (pull_request) Successful in 3m34s
CI / status-check (pull_request) Successful in 3s
CI / lint (push) Successful in 47s
CI / typecheck (push) Successful in 46s
CI / build (push) Successful in 53s
CI / quality (push) Successful in 1m4s
CI / security (push) Successful in 1m7s
CI / integration_tests (push) Successful in 1m13s
CI / unit_tests (push) Successful in 3m43s
CI / coverage (push) Successful in 3m42s
CI / status-check (push) Successful in 3s
AC1: ExecutionError gains kind (categorical: depth, model_calls, tool_calls,
timeout, cost) and reason (sub-code: budget_exhausted or
missing_pricing_entry) fields, both defaulting to empty string. All existing
raise ExecutionError(msg) call sites are backward-compatible.
AC2: PureLangGraph.__init__ accepts limits: dict[str, Any] and
pricing: dict[str, Any] (both default to {}). When limits['max_depth'] is
supplied, a depth breach raises ExecutionError(kind='depth') instead of
silently returning the current message (the old behaviour was a silent
data-loss bug). When no max_depth limit is supplied, the legacy heuristic
(max(2000, len(nodes)*50)) is used with a silent cap for backward
compatibility with existing callers that do not pass limits.
AC3: max_model_calls is checked before each NodeType.AGENT execution;
counter increments after the check so the limit is enforced before the
(limit+1)-th invocation. Breach raises ExecutionError(kind='model_calls').
AC4: max_tool_calls is checked before each NodeType.TOOL execution; same
pattern as model_calls. Breach raises ExecutionError(kind='tool_calls').
AC5: execute() wraps _execute_from_node() in asyncio.wait_for() when
limits['timeout_ms'] is set. asyncio.TimeoutError is caught and re-raised
as ExecutionError(kind='timeout') so the router can map it to HTTP 429.
AC6+AC7: After each LLM node, cost is computed from the pricing table
(rates are USD per million tokens, matching ADR-2029 example values such
as gpt-4.1-mini prompt=/bin/zsh.15/1M). Cost breach raises
ExecutionError(kind='cost', reason='budget_exhausted'). A missing
provider or model entry raises ExecutionError(kind='cost',
reason='missing_pricing_entry') — proceeding with assumed zero cost is
forbidden per ADR-2029. An empty pricing dict ({}) disables all cost
checking (cost tracking was not requested).
AC8: ExecutionError re-exported from cleveractors/__init__.py and
added to __all__.
Wire: runtime_dispatch._execute_graph() now passes executor.limits and
executor.pricing to PureLangGraph so the limits reach the enforcement
layer.
ExecutionError is also re-raised (not swallowed) in _execute_from_node()'s
broad except block so limit errors always propagate to the caller.
New BDD tests (features/execution_limits.feature, 19 scenarios):
- ExecutionError defaults kind and reason to empty strings
- ExecutionError accepts kind and reason keyword arguments
- ExecutionError exported from cleveractors top-level package
- Graph exceeding max_depth raises ExecutionError(kind='depth')
- Two AGENT nodes with max_model_calls=1 raises on 2nd call
- Two TOOL nodes with max_tool_calls=1 raises on 2nd call
- Slow node with timeout_ms=50 raises ExecutionError(kind='timeout')
- LLM node exceeding max_cost_usd raises budget_exhausted
- Unknown provider/model in pricing raises missing_pricing_entry
- Empty pricing table skips cost calculation entirely
- Executor limits/pricing propagation verified
Quality gates: lint pass, typecheck 0 errors,
unit_tests 2317/2317 pass, integration_tests pass,
coverage 97.17% (threshold 96.5%).
ISSUES CLOSED: #15
|