CoreRasurae
a8f68b8262
CI / lint (pull_request) Successful in 50s
CI / typecheck (pull_request) Successful in 55s
CI / security (pull_request) Successful in 53s
CI / quality (pull_request) Successful in 38s
CI / unit_tests (pull_request) Successful in 3m15s
CI / integration_tests (pull_request) Successful in 1m14s
CI / build (pull_request) Successful in 39s
CI / coverage (pull_request) Successful in 3m16s
CI / status-check (pull_request) Successful in 3s
CI / lint (push) Successful in 37s
CI / typecheck (push) Successful in 56s
CI / security (push) Successful in 54s
CI / quality (push) Successful in 37s
CI / unit_tests (push) Successful in 3m15s
CI / integration_tests (push) Successful in 1m14s
CI / build (push) Successful in 39s
CI / coverage (push) Successful in 3m16s
CI / status-check (push) Successful in 3s
Enforces the USD budget (max_cost_usd) as a hard pre-flight gate before every node execution and at each main-agent ainvoke round inside the multi-turn tool loop, with pruning passes silently skipped when the budget is exhausted. Pre-flight gate (pure_graph.py): _check_budget_pre_flight() raises ExecutionError(kind='cost', reason='budget_exhausted') immediately before node execution in both _execute_from_node and all three branches of _stream_from_node. In-node budget gating (llm.py): _check_budget_before_invoke() gates every _retry_ainvoke() call inside _execute_tool_loop, process_message, and stream_message. _should_skip_pruning() skips pruning passes when budget exhausted. _accumulate_cost_after_invoke() computes USD cost from token counts using the pricing table and advances current_accumulated_cost so subsequent budget checks see the updated cost. ContextVar plumbing (retry.py, pure_graph.py): current_accumulated_cost, current_max_cost_usd, and current_pricing ContextVars carry per-node budget state from PureLangGraph into LLMAgent. Set before node.execute() via _set_budget_context_for_node, reset in finally blocks via _reset_budget_context. Reviewer fixes (rui.hu #80): - Move cost accumulation inside try block in terminal streaming branch to prevent ContextVar leak (was set after finally reset). - Standardize log precision to $%.6f across all budget messages. - Remove orphaned 'already exhausted' step (criterion unreachable). - Add dedicated streaming budget enforcement scenario. - Remove redundant = None default on _reset_budget_context pricing_token. ISSUES CLOSED: #76
117 lines
7.8 KiB
Gherkin
117 lines
7.8 KiB
Gherkin
Feature: LLMAgent._execute_tool_loop() — shared tool-call orchestration helper
|
|
|
|
# ── Basic single tool round ───────────────────────────────────────────────
|
|
|
|
Scenario: _execute_tool_loop returns result for single tool round
|
|
Given I have an LLMAgent with tool loop test config for tool "echo"
|
|
And I set a mock that makes one tool call then returns a final answer "Final answer from loop"
|
|
When I run _execute_tool_loop with an initial messages list
|
|
Then the loop result final_response content should be "Final answer from loop"
|
|
And the loop result accumulated_prompt should be greater than 0
|
|
And the loop result accumulated_completion should be greater than 0
|
|
And the loop result budget_exhausted should be false
|
|
And the loop result synthesis_was_run should be false
|
|
|
|
# ── Multi-round (two consecutive tool-call rounds) ────────────────────────
|
|
|
|
Scenario: _execute_tool_loop accumulates tokens across two tool-call rounds
|
|
Given I have an LLMAgent with tool loop test config for tool "echo"
|
|
And I set a mock that makes two tool calls then returns a final answer "Final answer two rounds"
|
|
When I run _execute_tool_loop with an initial messages list
|
|
Then the loop result final_response content should be "Final answer two rounds"
|
|
And the loop result accumulated_prompt should be at least 30
|
|
And the loop result accumulated_completion should be at least 15
|
|
|
|
# ── tool_max_rounds exhaustion ────────────────────────────────────────────
|
|
|
|
Scenario: _execute_tool_loop handles tool_max_rounds exhaustion
|
|
Given I have an LLMAgent with tool loop test config for tool "echo" and tool_max_rounds 1
|
|
And I set a mock that always returns tool_calls never a final answer
|
|
When I run _execute_tool_loop with an initial messages list
|
|
Then the loop result should contain some content from the synthesis prompt
|
|
And the loop result budget_exhausted should be false
|
|
|
|
# ── Token accumulation spanning all rounds ────────────────────────────────
|
|
|
|
Scenario: _execute_tool_loop token counts span all ainvoke rounds
|
|
Given I have an LLMAgent with tool loop test config for tool "echo"
|
|
And I set a mock that makes two tool calls then returns a final answer "Accumulated answer"
|
|
When I run _execute_tool_loop with an initial messages list
|
|
Then the loop result accumulated_prompt should equal the sum of all rounds prompt tokens
|
|
And the loop result accumulated_completion should equal the sum of all rounds completion tokens
|
|
|
|
# ── Token-budget exhaustion ───────────────────────────────────────────────
|
|
|
|
Scenario: _execute_tool_loop triggers synthesis when token budget exhausted
|
|
Given I have an LLMAgent with tool loop test config for tool "echo" and tiny token_budget_percent
|
|
And I set a mock that triggers budget exhaustion then returns "Budget exhausted answer"
|
|
When I run _execute_tool_loop with an initial messages list
|
|
Then the loop result budget_exhausted should be true
|
|
And the loop result final_response content should be "Budget exhausted answer"
|
|
|
|
# ── Stuck-model synthesis ─────────────────────────────────────────────────
|
|
|
|
Scenario: _execute_tool_loop runs synthesis when model is stuck in tool-only mode
|
|
Given I have an LLMAgent with tool loop test config for tool "echo" and tool_max_rounds 2
|
|
And I set a mock that returns empty content tool calls then "Synthesis answer"
|
|
When I run _execute_tool_loop with an initial messages list
|
|
Then the loop result synthesis_was_run should be true
|
|
And the loop result final_response content should be "Synthesis answer"
|
|
|
|
# ── Tool-output pruning ───────────────────────────────────────────────────
|
|
|
|
Scenario: _execute_tool_loop runs pruning pass and includes its tokens in accumulated counts
|
|
Given I have an LLMAgent with tool loop test config for tool "file_read" and pruning enabled
|
|
And I set a mock that makes one file_read tool call then returns "Pruned answer"
|
|
And I set a mock pruning model that adds extra tokens
|
|
When I run _execute_tool_loop with an initial messages list
|
|
Then the loop result final_response content should be "Pruned answer"
|
|
And the loop result accumulated_prompt should include pruning tokens
|
|
|
|
# ── Tool dispatch error ───────────────────────────────────────────────────
|
|
|
|
Scenario: _execute_tool_loop captures tool dispatch ExecutionError as ToolMessage
|
|
Given I have an LLMAgent with tool loop test config for tool "echo"
|
|
And I set a mock that calls a nonexistent tool then returns "After error answer"
|
|
When I run _execute_tool_loop with an initial messages list
|
|
Then the loop result final_response content should be "After error answer"
|
|
And the messages list should contain a ToolMessage with not available content
|
|
|
|
# ── Exception path: partial token counts preserved ────────────────────────
|
|
|
|
Scenario: _execute_tool_loop wraps exception in _ToolLoopError with partial counts
|
|
Given I have an LLMAgent with tool loop test config for tool "echo"
|
|
And I set a mock that returns one tool round then raises a RuntimeError
|
|
When I attempt to run _execute_tool_loop with an initial messages list
|
|
Then a _ToolLoopError should be raised
|
|
And the _ToolLoopError any_invocation_made should be true
|
|
And the _ToolLoopError accumulated_prompt should be greater than 0
|
|
|
|
# ── Exception before any invocation ──────────────────────────────────────
|
|
|
|
Scenario: _ToolLoopError has any_invocation_made false when error before first ainvoke
|
|
Given I have an LLMAgent with tool loop test config for tool "echo" and bad tool_max_rounds
|
|
When I attempt to run _execute_tool_loop with an initial messages list
|
|
Then a _ToolLoopError should be raised
|
|
And the _ToolLoopError any_invocation_made should be false
|
|
And the _ToolLoopError cause should be a ConfigurationError
|
|
|
|
# ── USD cost budget exhaustion (in-node gating, issue #76) ────────────────
|
|
|
|
Scenario: _execute_tool_loop raises budget_exhausted when pricing context is set and cost exceeds limit
|
|
Given I have an LLMAgent with tool loop test config for tool "echo" and tool_max_rounds 5
|
|
And I set a mock that makes one tool call then returns a final answer "cost triggered"
|
|
And the USD budget ContextVars are set for in-node gating
|
|
When I run _execute_tool_loop with an initial messages list
|
|
Then an ExecutionError with kind "cost" and reason "budget_exhausted" should be raised
|
|
|
|
Scenario: _execute_tool_loop skips pruning pass when budget tripped mid-loop (issue #76)
|
|
Given I have an LLMAgent with tool loop test config for tool "file_read" and pruning enabled
|
|
And I set a mock that makes one file_read tool call then returns "Final text"
|
|
And I set a mock pruning model that adds extra tokens
|
|
And the USD budget ContextVars are set so first round exhausts limit
|
|
When I run _execute_tool_loop with an initial messages list
|
|
Then an ExecutionError with kind "cost" and reason "budget_exhausted" should be raised
|
|
And the accumulated prompt should NOT include pruning tokens
|
|
And the accumulated completion should NOT include pruning tokens
|