cleveractors-core/features/llm_agent_tool_calling.feature

Feature: LLMAgent Tool Calling

  Scenario: LLMAgent reads tools from config at init
    Given I have an LLM agent configuration with tools [{"name": "echo"}, {"name": "math"}]
    When I create the LLM agent
    Then the agent should store tool definitions in _lc_tools with count 2

  Scenario: LLMAgent without tools has empty _lc_tools
    Given I have a basic LLM agent configuration (no tools)
    And that config does NOT include a tools field
    When I create the LLM agent
    Then the agent _lc_tools should be None

  Scenario: LLMAgent get_capabilities includes tool-calling when tools configured
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    When I create the LLM agent
    And I request the agent capabilities (tool calling)
    Then the capabilities should include "tool-calling"
    And the capabilities should still include "text-generation"

  Scenario: LLMAgent get_capabilities without tool-calling when no tools
    Given I have a basic LLM agent configuration
    And that config does NOT include a tools field
    When I create the LLM agent
    And I request the agent capabilities (tool calling)
    Then the capabilities should include "text-generation"
    And the capabilities should NOT include "tool-calling"

  Scenario: LLMAgent process_message passes tools to model when configured
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    When I create the LLM agent
    And I set a mock chat model that returns a plain text response
    And I tool_calling process a message "Hello"
    Then the chat_model ainvoke should have been called with a tools keyword

  Scenario: LLMAgent process_message without tools does not pass tools to model
    Given I have a basic LLM agent configuration
    And that config does NOT include a tools field
    When I create the LLM agent
    And I set a mock chat model that returns a plain text response
    And I tool_calling process a message "Hello"
    Then the chat_model ainvoke should NOT have been called with a tools keyword

  Scenario: LLMAgent _lc_tools passed to model as invoke keyword
    Given I have an LLM agent configuration with tools [{"name": "file_read"}]
    When I create the LLM agent
    And I set a mock chat model that returns a plain text response
    And I tool_calling process a message "Read a file"
    Then the chat_model ainvoke should have been called with a tools keyword

  Scenario: LLMAgent metadata includes tool counts when tools configured
    Given I have an LLM agent configuration with tools [{"name": "echo"}, {"name": "math"}]
    When I create the LLM agent
    And I request the agent metadata (tool calling)
    Then the metadata should have "tools_configured" set to true
    And the metadata should have "tool_count" equal to 2

  Scenario: LLMAgent injects synthesis prompt when model gets stuck in tool-only mode
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    And I set tool_max_rounds to 2
    When I create the LLM agent
    And I set a mock chat model that returns tool_calls with empty content for 2 rounds then text
    And I tool_calling process a message "Do something"
    Then the synthesis HumanMessage should have been appended to messages
    And the final ainvoke should have been called
    And the tool_calling result should contain "Synthesized answer"

  Scenario: LLMAgent propagates tool execution errors to ToolMessage for LLM self-correction
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    And I set tool_max_rounds to 1
    When I create the LLM agent
    And I set a mock chat model that triggers a tool with ExecutionError "test tool failure"
    And I tool_calling process a message "Fail me"
    Then the messages list should contain a ToolMessage with content containing _nonexistent_tool_xyz

  Scenario: LLMAgent handles empty tool name in malformed tool_calls
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    And I set tool_max_rounds to 1
    When I create the LLM agent
    And I set a mock chat model that returns tool_calls with an empty tool name
    And I tool_calling process a message "Call me"
    Then the messages list should contain a ToolMessage with content "Tool name is empty"

  # ── Token Budget Awareness (§4.4.7) ──────────────────────────────────

  Scenario: LLMAgent validates token_budget_percent range
    Given I have an LLM agent configuration with token_budget_percent 1.5
    When I attempt to create the LLM agent
    Then ConfigurationError should be raised containing "token_budget_percent must be in"

  Scenario: LLMAgent stores token_budget_percent when valid
    Given I have an LLM agent configuration with token_budget_percent 0.50
    When I create the LLM agent
    Then the agent _token_budget_percent should be 0.5

  Scenario: LLMAgent _token_budget_percent is None by default
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    Then the agent _token_budget_percent should be None

  # ── Tool Output Pruning (§4.4.8) ────────────────────────────────────

  Scenario: LLMAgent stores allow_tool_output_pruning when true
    Given I have an LLM agent config with allow_tool_output_pruning true
    When I create the LLM agent
    Then the agent _allow_tool_output_pruning should be true

  Scenario: LLMAgent allow_tool_output_pruning defaults to false
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    Then the agent _allow_tool_output_pruning should be false

  Scenario: LLMAgent stores pruning_model when configured
    Given I have an LLM agent config with pruning_model "gpt-3.5-turbo"
    When I create the LLM agent
    Then the agent _pruning_model should be "gpt-3.5-turbo"

  Scenario: LLMAgent pruning_model defaults to None
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    Then the agent _pruning_model should be None

  # ── pruning_model matching enforcement (issue #65) ──────────────────

  Scenario: pruning_model mismatch raises ConfigurationError
    Given I have an LLM agent config with model "gpt-4" and pruning_model "gpt-3.5-turbo"
    When I attempt to create the LLM agent
    Then ConfigurationError should be raised containing "must match"

  Scenario: pruning_model same as model succeeds
    Given I have an LLM agent config with model "gpt-4" and pruning_model "gpt-4"
    When I create the LLM agent
    Then the agent _pruning_model should be "gpt-4"
    And the agent model should be "gpt-4"

  # ── pruning_threshold (§4.4.8 D-2) ────────────────────────────────────

  Scenario: LLMAgent stores pruning_threshold when configured
    Given I have an LLM agent config with pruning_threshold 1024
    When I create the LLM agent
    Then the agent _pruning_threshold should be 1024

  Scenario: LLMAgent pruning_threshold defaults to 512
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    Then the agent _pruning_threshold should be 512

  Scenario: LLMAgent validates pruning_threshold must be positive
    Given I have an LLM agent config with pruning_threshold -10
    When I attempt to create the LLM agent
    Then ConfigurationError should be raised containing "pruning_threshold must be a positive integer"

  Scenario: LLMAgent validates pruning_threshold must be integer
    Given I have an LLM agent config with pruning_threshold "not_an_int"
    When I attempt to create the LLM agent
    Then ConfigurationError should be raised containing "pruning_threshold must be a positive integer"

  # ── pruning_tool_filter (§4.4.8 D-2) ─────────────────────────────────

  Scenario: LLMAgent stores pruning_tool_filter when configured
    Given I have an LLM agent config with pruning_tool_filter ["file_read", "shell", "http_request"]
    When I create the LLM agent
    Then the agent _pruning_tool_filter should be ["file_read", "shell", "http_request"]

  Scenario: LLMAgent pruning_tool_filter defaults to file_read and shell
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    Then the agent _pruning_tool_filter should be ["file_read", "shell"]

  Scenario: LLMAgent validates pruning_tool_filter must be a list
    Given I have an LLM agent config with pruning_tool_filter "not_a_list"
    When I attempt to create the LLM agent
    Then ConfigurationError should be raised containing "pruning_tool_filter must be a list"

  Scenario: LLMAgent validates pruning_tool_filter items must be strings
    Given I have an LLM agent config with pruning_tool_filter ["file_read", 123]
    When I attempt to create the LLM agent
    Then ConfigurationError should be raised containing "pruning_tool_filter items must be strings"

  Scenario: LLMAgent pruning_tool_filter empty list is preserved (disables pruning)
    Given I have an LLM agent config with pruning_tool_filter []
    When I create the LLM agent
    Then the agent _pruning_tool_filter should be []

  # ── Schema augmentation with pruning_tool_filter (§4.4.8 D-3) ─────────

  Scenario: Tool schema augmented only for tools in pruning_tool_filter
    Given I have an LLM agent config with allow_tool_output_pruning true
    And I have pruning_tool_filter ["file_read"]
    And I have tools [{"type": "function", "function": {"name": "file_read", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}}}},{"type": "function", "function": {"name": "shell", "parameters": {"type": "object", "properties": {"cmd": {"type": "string"}}}}}]
    When I create the LLM agent
    And I augment the tool schemas for pruning
    Then the file_read schema should contain output_prune parameter
    And the shell schema should NOT contain output_prune parameter

  # ── Schema Augmentation (§4.4.8 step 1) ────────────────────────────

  Scenario: Tool schemas augmented with output_prune when pruning enabled
    Given I have an LLM agent config with allow_tool_output_pruning true
    And I have tools [{"type": "function", "function": {"name": "file_read", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}}}}]
    When I create the LLM agent
    And I augment the tool schemas for pruning
    Then the augmented schemas should contain output_prune parameter

  Scenario: Tool schema augmentation does not require pruning_tool_filter to be set
    Given I have a basic LLM agent configuration (no tools)
    And I have tools [{"type": "function", "function": {"name": "file_read", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}}}}]
    When I create the LLM agent
    And I augment the tool schemas for pruning
    Then the augmented schemas should contain output_prune parameter

  Scenario: process_message passes augmented tools to ainvoke when pruning enabled
    Given I have an LLM agent config with allow_tool_output_pruning true
    And I have tools [{"type": "function", "function": {"name": "file_read", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}}}}]
    When I create the LLM agent
    And I set a mock chat model that returns a plain text response
    And I tool_calling process a message "Hello"
    Then the ainvoke tools kwarg should contain the output_prune parameter

  # ── Pruning Marker Parsing (§4.4.9) ─────────────────────────────────

  Scenario: Parse pruning response with info and output blocks
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    And I parse a pruning response with both info and output blocks
    Then the parsed result should contain PRUNE_INFO_START
    And the parsed result should contain PRUNE_OUTPUT_START

  Scenario: Parse pruning response with only output block
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    And I parse a pruning response with only an output block
    Then the parsed result should contain PRUNE_OUTPUT_START
    And the parsed result should NOT contain PRUNE_INFO_START

  Scenario: Parse pruning response with no markers falls back to full text
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    And I parse a pruning response with no markers containing "some extracted content"
    Then the parsed result should be "some extracted content"

  # ── Model Context Window (§4.4.7) ───────────────────────────────────

  Scenario: _get_model_context_window uses max_tokens when set above 10K
    Given I have an LLM agent config with max_tokens 200000
    When I create the LLM agent
    Then _get_model_context_window should return 200000

  Scenario: _get_model_context_window defaults to 128000 when max_tokens is small
    Given I have an LLM agent config with max_tokens 1000
    When I create the LLM agent
    Then _get_model_context_window should return 128000

  Scenario: _get_model_context_window defaults to 128000 when max_tokens not set
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    Then _get_model_context_window should return 128000

  # ── _run_pruning_pass with mocked pruning model (§4.4.8) ────────────

  # ── _run_pruning_pass with mocked pruning model (§4.4.8) ────────────

  Scenario: _run_pruning_pass returns marked-up response with token counts
    Given I have an LLM agent config with allow_tool_output_pruning true
    When I create the LLM agent
    And I set a mock pruning model that returns a marked-up response
    And I run the pruning pass on tool "file_read" with output "50KB of logs"
    Then the pruned result should contain PRUNE_INFO_START
    And the pruned result should contain PRUNE_OUTPUT_START
    And the pruned result should contain "extracted content"
    And the pruning token counts should be greater than zero

  Scenario: _run_pruning_pass falls back to raw output on error
    Given I have an LLM agent config with allow_tool_output_pruning true
    When I create the LLM agent
    And I set a mock pruning model that raises an exception
    And I run the pruning pass on tool "echo" with output "raw data"
    Then the pruned result should equal "raw data"

  Scenario: _run_pruning_pass returns empty raw output unchanged
    Given I have an LLM agent config with allow_tool_output_pruning true
    When I create the LLM agent
    And I run the pruning pass on tool "echo" with empty output
    Then the pruned result should be empty

  # ── Token usage metadata enforcement (issue #65) ────────────────────

  Scenario: _run_pruning_pass fails when response lacks usage_metadata
    Given I have an LLM agent config with allow_tool_output_pruning true
    When I create the LLM agent
    And I set a mock pruning model that returns a response without usage_metadata
    And I attempt to run the pruning pass on tool "file_read" with output "50KB of logs"
    Then a MissingUsageMetadataError should be raised containing "token usage metadata"

  Scenario: MissingUsageMetadataError in tool loop preserves accumulated tokens
    Given I have an LLM agent configuration with tools [{"name": "file_read"}]
    And I set allow_tool_output_pruning to true
    And I set pruning_threshold to 1
    And I set tool_max_rounds to 3
    And I set unsafe_mode to true
    When I create the LLM agent
    And I set a mock chat model that returns one file_read tool call then text
    And I set a mock pruning model that returns a response without usage_metadata
    And I attempt to tool_calling process a message "Test billing preservation"
    Then an ExecutionError should be raised containing "token usage metadata"
    And the last token usage should have accumulated counts from prior rounds

  # ── Token budget exhaustion in tool loop (§4.4.7) ──────────────────

  Scenario: Token budget exhaustion triggers synthesis flow
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    And I set token_budget_percent to 0.000001
    And I set tool_max_rounds to 5
    When I create the LLM agent
    And I set a mock chat model that returns tool_calls then text for budget exhaustion
    And I tool_calling process a message "Test budget"
    Then the tool_calling result should contain "Budget exhausted"

  # ── Pruning pass in tool loop (§4.4.8) ─────────────────────────────

  Scenario: Pruning pass NOT called for non-file_read tools
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    And I set allow_tool_output_pruning to true
    And I set tool_max_rounds to 2
    When I create the LLM agent
    And I set a mock chat model that returns one tool call then text
    And I set a mock pruning model that returns a marked-up response
    And I tool_calling process a message "Test pruning scope"
    Then the tool_calling result should contain "Final answer"
    And the pruning pass should NOT have been called

  # ── output_prune stripping (§4.4.8 step 1) ──────────────────────────

  Scenario: output_prune false skips pruning in tool loop
    Given I have an LLM agent configuration with tools [{"name": "file_read"}]
    And I set allow_tool_output_pruning to true
    And I set tool_max_rounds to 2
    When I create the LLM agent
    And I set a mock chat model that returns tool call with output_prune false then text
    And I set a mock pruning model that returns a marked-up response
    And I tool_calling process a message "Test opt out"
    Then the tool_calling result should contain "Final answer after tools"
    And the pruning pass should NOT have been called

  # ── Pruning with prune_context (§4.4.8 step 3) ──────────────────────

  Scenario: _run_pruning_pass with prune_context uses search directive
    Given I have an LLM agent config with allow_tool_output_pruning true
    When I create the LLM agent
    And I set a mock pruning model that returns a marked-up response
    And I run the pruning pass on "file_read" with output "100 lines" and prune_context "find errors"
    Then the pruned result should contain PRUNE_OUTPUT_START

  # ── Budget exhaustion with pruning enabled ──────────────────────────

  Scenario: Budget exhaustion with pruning enabled triggers pruning in synthesis round
    Given I have an LLM agent configuration with tools [{"name": "file_read"}]
    And I set token_budget_percent to 0.000001
    And I set allow_tool_output_pruning to true
    And I set tool_max_rounds to 5
    When I create the LLM agent
    And I set a mock chat model that triggers budget exhaustion with a file_read tool call
    And I set a mock pruning model that returns a marked-up response
    And I tool_calling process a message "Test budget with prune"
    Then the tool_calling result should contain "Budget exhausted"
    And the pruning pass should have been called

  # ── Stuck-model synthesis with tool calls ────────────────────────────

  Scenario: Stuck-model synthesis executes tool calls in synthesis round
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    And I set tool_max_rounds to 2
    When I create the LLM agent
    And I set a mock chat model that gets stuck then returns tool_calls in synthesis
    And I tool_calling process a message "Get stuck"
    Then the tool_calling result should contain "Synthesized answer"

  # ── Synthesis-round tool name validation (fix for issue #63) ────────

  Scenario: Synthesis round rejects hallucinated tool name not in declared tools
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    And I set tool_max_rounds to 2
    When I create the LLM agent
    And I set a mock chat model that gets stuck then returns hallucinated tool name in synthesis
    And I tool_calling process a message "Get stuck"
    Then the synthesis messages should contain a ToolMessage with content "Tool 'nonexistent_tool' is not available."

  # ── Additional coverage: token_budget_percent type validation ──────

  Scenario: LLMAgent validates token_budget_percent must be numeric
    Given I have a basic LLM agent configuration (no tools)
    And I set token_budget_percent to string "not_a_number"
    When I attempt to create the LLM agent
    Then ConfigurationError should be raised containing "token_budget_percent must be a number"

  # ── Additional coverage: pruning response missing end tag ──────────

  Scenario: Parse pruning response with missing end tag falls back to full text
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    And I parse a pruning response that has start tag but no end tag
    Then the parsed result should contain "Partial info without end tag"

  # ── Additional coverage: tool_max_rounds parse error ──────────────

  Scenario: LLMAgent rejects invalid tool_max_rounds on process_message
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    And I set tool_max_rounds to string "abc"
    When I create the LLM agent
    And I set a mock chat model that returns a plain text response
    And I attempt to tool_calling process a message "Hello"
    Then ConfigurationError should be raised containing "tool_max_rounds must be an integer"

  # ── Additional coverage: direct-format tool calls ──────────────────

  Scenario: LLMAgent handles tool_calls in direct format without function key
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    And I set tool_max_rounds to 2
    When I create the LLM agent
    And I set a mock chat model that returns a direct-format tool call then text
    And I tool_calling process a message "Test direct format"
    Then the tool_calling result should contain "Direct format answer"

  # ── _extract_token_counts coverage: response_metadata fallback ─────

  Scenario: _extract_token_counts extracts tokens via response_metadata fallback
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    And I extract token counts from a response using response_metadata fallback
    Then the extracted token counts should be (30, 15, True)

  Scenario: _extract_token_counts returns zero for empty usage_metadata
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    And I extract token counts from a response with empty usage_metadata
    Then the extracted token counts should be (0, 0, False)

  Scenario: _extract_token_counts returns zero when response_metadata has no token_usage key
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    And I extract token counts from a response with response_metadata without token_usage key
    Then the extracted token counts should be (0, 0, False)

  Scenario: _extract_token_counts returns zero for non-dict response_metadata
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    And I extract token counts from a response with non-dict response_metadata
    Then the extracted token counts should be (0, 0, False)

  Scenario: _extract_token_counts returns zero for missing response_metadata
    Given I have a basic LLM agent configuration (no tools)
    When I create the LLM agent
    And I extract token counts from a response with missing response_metadata
    Then the extracted token counts should be (0, 0, False)

  # ── Budget synthesis with tool calls and pruning ──────────────────

  Scenario: Budget synthesis tool calls trigger pruning pass
    Given I have an LLM agent configuration with tools [{"name": "file_read"}]
    And I set token_budget_percent to 0.000001
    And I set allow_tool_output_pruning to true
    And I set pruning_threshold to 1
    And I set tool_max_rounds to 5
    And I set unsafe_mode to true
    When I create the LLM agent
    And I set a mock chat model that triggers budget exhaustion then returns tool calls in synthesis
    And I set a mock pruning model that returns a marked-up response
    And I tool_calling process a message "Test budget synth pruning"
    Then the tool_calling result should contain "Synthesis answer with pruning"
    And the pruning pass should have been called

  # ── MissingUsageMetadataError in budget synthesis ─────────────────

  Scenario: MissingUsageMetadataError in budget synthesis preserves accumulated tokens
    Given I have an LLM agent configuration with tools [{"name": "file_read"}]
    And I set token_budget_percent to 0.000001
    And I set allow_tool_output_pruning to true
    And I set pruning_threshold to 1
    And I set tool_max_rounds to 5
    And I set unsafe_mode to true
    When I create the LLM agent
    And I set a mock chat model that triggers budget exhaustion then returns tool calls in synthesis
    And I set a mock pruning model that returns a response without usage_metadata
    And I attempt to tool_calling process a message "Test budget synth error"
    Then an ExecutionError should be raised containing "token usage metadata"
    And the last token usage should have accumulated counts from prior rounds

  # ── Stuck-model synthesis tool execution error ─────────────────────

  Scenario: Stuck-model synthesis tool error preserves accumulated tokens
    Given I have an LLM agent configuration with tools [{"name": "echo"}]
    And I set tool_max_rounds to 2
    When I create the LLM agent
    And I set a mock chat model that gets stuck then returns a tool call that raises ExecutionError
    And I tool_calling process a message "Get stuck with error"
    Then the tool_calling result should contain "Final answer after error"
    And the accumulated prompt tokens should be greater than 0
    And the accumulated completion tokens should be greater than 0