CoreRasurae
227e2feece
CI / build (pull_request) Successful in 45s
CI / quality (pull_request) Successful in 58s
CI / lint (pull_request) Successful in 1m15s
CI / typecheck (pull_request) Successful in 1m15s
CI / security (pull_request) Successful in 1m15s
CI / integration_tests (pull_request) Successful in 1m24s
CI / unit_tests (pull_request) Successful in 3m25s
CI / coverage (pull_request) Successful in 3m10s
CI / status-check (pull_request) Successful in 6s
CI / lint (push) Successful in 42s
CI / quality (push) Successful in 48s
CI / typecheck (push) Successful in 51s
CI / security (push) Successful in 49s
CI / build (push) Successful in 47s
CI / integration_tests (push) Successful in 1m10s
CI / unit_tests (push) Successful in 3m9s
CI / coverage (push) Successful in 3m7s
CI / status-check (push) Successful in 3s
Implements the complete LLM agent tool-calling pipeline — the tool-calling path was broken in multiple ways: tools from agent config were not passed to the LLM, tool execution was single-pass (the model could not see tool results or make follow-up calls), tool errors were discarded, and shell/ python_exec tools were never registered. Key changes: - Multi-turn tool-call loop: passes tools to the LLM via ainvoke(tools=...) and re-invokes after each batch of ToolResults, with a configurable round limit (tool_max_rounds config / TOOL_MAX_ROUNDS env var, default 20). - Tool schema module (llm_tools.py): normalize_tool_entry() converts string tool names and config dicts to OpenAI function-calling format with full parameter schemas and LLM-facing guidance (max_chars for file_read, sandbox builtins for python_exec, etc.). - file_read enhancements: directory listing with file sizes and type indicators, max_chars truncation to prevent context overflow, shell command detection with redirect to shell tool, file-not-found recovery hints with parent directory listing. - python_exec sandbox: stdout capture so print() produces visible output, NameError guidance directing LLM to file_read/file_write instead of using open(). - shell/python_exec tool registration in builtin_tools when allow_shell / exec_python is enabled. - create_subprocess_shell for shell commands (was create_subprocess_exec, which blocks pipes/redirections/chains). - Tool error propagation: ExecutionError/ConfigurationError details included in ToolMessages for LLM self-correction. - Stuck-model recovery: injects synthesizing prompt when model exhausts tool rounds without producing content. - LLM provider error details preserved in ExecutionError messages instead of generic "LLM processing failed". - Empty message filtering in _prepare_conversation_history() prevents whitespace-only messages from polluting downstream conversation history in multi-agent pipelines. Closes #59
90 lines
5.4 KiB
Gherkin
90 lines
5.4 KiB
Gherkin
Feature: LLM Agent Temperature Override, Cleanup, and Context History
|
|
As a developer
|
|
I want temperature overrides from context, http client cleanup, and conversation history fallback to work correctly
|
|
So that LLM agents adapt to runtime parameters and release resources without leaking connections
|
|
|
|
# ---- Temperature Override (lines 251-260) ----
|
|
|
|
Scenario: Temperature override from context is applied when different
|
|
Given I setup a test LLM agent for temperature override tests (llm_gaps)
|
|
When I process a message with _temperature_override set to 0.3 (llm_gaps)
|
|
Then the mock chat model temperature should be set to 0.3 (llm_gaps)
|
|
|
|
Scenario: Temperature override is skipped when value matches current temperature
|
|
Given I setup a test LLM agent for temperature override tests (llm_gaps)
|
|
When I process a message with _temperature_override equal to current temp (llm_gaps)
|
|
Then the mock chat model temperature should remain at its original value (llm_gaps)
|
|
|
|
Scenario: Temperature override works with other context data
|
|
Given I setup a test LLM agent for temperature override tests (llm_gaps)
|
|
When I process a message with _temperature_override 0.9 and extra context keys (llm_gaps)
|
|
Then the mock chat model temperature should be set to 0.9 (llm_gaps)
|
|
And the process message result should be a non-empty string (llm_gaps)
|
|
|
|
# ---- Missing API Key (lines 209-217) ----
|
|
|
|
Scenario: Missing API key raises ConfigurationError without env vars
|
|
Given I clear all relevant API key environment variables for testing (llm_gaps)
|
|
And I have an API-key-missing config for provider "openai" (coverage gap) (llm_gaps)
|
|
When I try to construct an LLMAgent with that config and cleared env (llm_gaps)
|
|
Then the raised exception should be a ConfigurationError (llm_gaps)
|
|
And the error message should mention "Missing API key" or "Failed to initialize" (llm_gaps)
|
|
|
|
Scenario: Missing API key for Google provider raises error
|
|
Given I clear all relevant API key environment variables for testing (llm_gaps)
|
|
And I have an API-key-missing Google config with model "gemini-pro" (coverage gap) (llm_gaps)
|
|
When I try to construct an LLMAgent with that config and cleared env (llm_gaps)
|
|
Then the raised exception should be a ConfigurationError (llm_gaps)
|
|
|
|
# ---- Conversation History from Context (line 304) ----
|
|
|
|
Scenario: Conversation history from context is used over memory
|
|
Given I setup an LLM agent with memory enabled for history tests (llm_gaps)
|
|
And I populate the agent memory with previous conversation history (llm_gaps)
|
|
When I process a message with conversation_history provided in the call context (llm_gaps)
|
|
Then the chat model invocation should include context history messages (llm_gaps)
|
|
And the number of messages sent to chat model should exceed 2 (llm_gaps)
|
|
|
|
Scenario: No conversation history in context falls back to memory
|
|
Given I setup an LLM agent with memory enabled for history tests (llm_gaps)
|
|
And I populate the agent memory with previous conversation history (llm_gaps)
|
|
When I process a message without conversation_history in the call context (llm_gaps)
|
|
Then the agent memory should have been queried for history (llm_gaps)
|
|
|
|
Scenario: Empty conversation history list in context
|
|
Given I setup an LLM agent with memory enabled for history tests (llm_gaps)
|
|
And I populate the agent memory with previous conversation history (llm_gaps)
|
|
When I process a message with conversation_history set to empty list (llm_gaps)
|
|
Then only the system message and current user message should be sent to chat model (llm_gaps)
|
|
|
|
Scenario: Conversation history from context with mixed roles
|
|
Given I setup an LLM agent with memory enabled for history tests (llm_gaps)
|
|
When I process a message with multi-role conversation_history in context (llm_gaps)
|
|
Then the messages sent to chat model should include user and assistant messages in correct order (llm_gaps)
|
|
|
|
# ---- cleanup() Method ----
|
|
# cleanup() only releases the agent's own _chat_model reference; it does NOT
|
|
# close any provider SDK clients (those are managed by shared lru_cache
|
|
# instances — see issue #57 for details).
|
|
|
|
Scenario: Cleanup sets _chat_model to None and does not close root_async_client
|
|
Given I setup an LLM agent with a mock root_async_client (llm_gaps)
|
|
When I await the cleanup method (llm_gaps)
|
|
Then the mock root_async_client close should NOT have been called (llm_gaps)
|
|
And the chat model should be None after cleanup (llm_gaps)
|
|
|
|
Scenario: Cleanup sets _chat_model to None and does not close root_client
|
|
Given I setup an LLM agent with both mock root_async_client and mock root_client (llm_gaps)
|
|
When I await the cleanup method (llm_gaps)
|
|
Then neither mock client close should have been called (llm_gaps)
|
|
And the chat model should be None after cleanup (llm_gaps)
|
|
|
|
Scenario: Cleanup is a no-op when chat_model has no http clients
|
|
Given I setup an LLM agent with a chat model that lacks root_async_client and root_client (llm_gaps)
|
|
When I await the cleanup method (llm_gaps)
|
|
Then the cleanup should complete without raising errors (llm_gaps)
|
|
|
|
Scenario: Non-numeric _temperature_override raises ConfigurationError in process_message
|
|
Given I setup a test LLM agent for temperature override tests (llm_gaps)
|
|
When I call process_message with _temperature_override "bad_string" (llm_gaps)
|
|
Then a ConfigurationError mentioning must be a number is raised (llm_gaps) |