cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity
Files
785bad712ca3befaea56c19ffecc33dd9b8249d0
cleveragents-core/features/base_agent_coverage.feature
T

351 lines
16 KiB
Gherkin
Raw Blame History

Feature: Base Agent Coverage
As a developer
I want comprehensive test coverage for the BaseAgent class
So that I can ensure the base agent functionality works correctly
Background:
Given the base agent module is importable
And I have a mock LLM provider configured for base agent
Scenario: BaseAgent cannot be instantiated directly (abstract class)
When I try to instantiate BaseAgent directly
Then I should get a TypeError about abstract methods
Scenario: Concrete agent can be instantiated with default parameters
When I create a concrete agent with default parameters in base agent
Then the agent should be initialized successfully in base agent
And the agent should have provider "openai"
And the agent should have model "gpt-4"
And the agent should have temperature 0.7
And the agent should have an llm attribute in base agent
And the agent should have a memory attribute in base agent
And the agent should have a graph attribute in base agent
And the agent should have an app attribute in base agent
Scenario: Concrete agent can be instantiated with custom parameters
When I create a concrete agent with parameters: in base agent
| parameter | value |
| provider | anthropic |
| model | claude-3 |
| temperature | 0.3 |
Then the agent should be initialized successfully in base agent
And the agent provider should be "anthropic" in base agent
And the agent model should be "claude-3" in base agent
And the agent temperature should be 0.3
Scenario: Concrete agent accepts provider-specific kwargs
When I create a concrete agent with provider kwargs: in base agent
| key | value |
| max_tokens | 1000 |
| top_p | 0.9 |
Then the agent should store provider_kwargs correctly in base agent
And provider_kwargs should contain "max_tokens"
And provider_kwargs should contain "top_p"
Scenario: Create OpenAI LLM with correct configuration
Given I have a concrete agent instance in base agent
When the agent creates an OpenAI LLM with model "gpt-4" and temperature 0.7
Then the LLM should be a ChatOpenAI instance
And the LLM creation should be logged in base agent
Scenario: Create Anthropic LLM with correct configuration
Given I have a concrete agent instance in base agent
When I create a concrete agent with provider "anthropic" and model "claude-3"
Then the LLM should be a ChatAnthropic instance
And the LLM creation should be logged in base agent
Scenario: Create Google LLM with correct configuration
Given I have a concrete agent instance in base agent
When I create a concrete agent with provider "google" and model "gemini-pro"
Then the LLM should be a ChatGoogleGenerativeAI instance
And the LLM creation should be logged in base agent
Scenario: Create LLM with unsupported provider raises ValueError
When I try to create a concrete agent with provider "unsupported"
Then I should get a ValueError about unsupported provider
And the error message should list supported providers in base agent
Scenario: Create LLM merges common and provider-specific parameters
When I create a concrete agent with model "gpt-4" and provider kwargs:
| key | value |
| max_tokens | 2000 |
Then the LLM should be created with merged parameters in base agent
And the merged parameters should include model in base agent
And the merged parameters should include temperature in base agent
And the merged parameters should include max_tokens in base agent
Scenario: Switch provider updates all configuration
Given I have a concrete agent with provider "openai"
When I switch to provider "anthropic" with model "claude-3"
Then the agent provider should be "anthropic" in base agent
And the agent model should be "claude-3" in base agent
And the llm should be recreated in base agent
And the graph should be rebuilt in base agent
And the app should be recompiled in base agent
And the provider switch should be logged in base agent
Scenario: Switch provider with new kwargs updates configuration
Given I have a concrete agent with provider "openai"
When I switch to provider "google" with model "gemini-pro" and kwargs:
| key | value |
| max_tokens | 1500 |
Then the agent provider_kwargs should contain "max_tokens"
And the llm should be recreated in base agent with new configuration
Scenario: Invoke executes workflow with input data
Given I have a concrete agent instance in base agent
When I invoke the agent with input data in base agent:
"""
{
"messages": [{"role": "user", "content": "test"}],
"context": {}
}
"""
Then the workflow should execute in base agent
And the result should be returned in base agent
And the result should contain the expected structure in base agent
Scenario: Invoke uses default config when none provided
Given I have a concrete agent instance in base agent
When I invoke the agent without providing config in base agent
Then the default config should be used in base agent
And the config should have thread_id "default"
Scenario: Invoke uses provided custom config
Given I have a concrete agent instance in base agent
When I invoke the agent with custom config: in base agent
"""
{
"configurable": {"thread_id": "test-thread-123"}
}
"""
Then the custom config should be used in base agent
And the config thread_id should be "test-thread-123"
Scenario: Invoke handles exceptions gracefully
Given I have a concrete agent instance in base agent
When the workflow execution raises an exception in base agent
And I invoke the agent with input data in base agent
Then the error should be caught in base agent
And the result should contain an error field in base agent
And the result should contain a result field set to None in base agent
And the result should preserve input messages in base agent
And the error should be logged in base agent
Scenario: Async invoke executes workflow asynchronously
Given I have a concrete agent instance in base agent
When I ainvoke the agent with input data in base agent:
"""
{
"messages": [{"role": "user", "content": "async test"}],
"context": {}
}
"""
Then the async workflow should execute in base agent
And the async result should be returned in base agent
Scenario: Async invoke uses default config when none provided
Given I have a concrete agent instance in base agent
When I ainvoke the agent without providing config in base agent
Then the default config should be used in base agent for async
Scenario: Async invoke uses provided custom config
Given I have a concrete agent instance in base agent
When I ainvoke the agent with custom config: in base agent
"""
{
"configurable": {"thread_id": "async-thread-456"}
}
"""
Then the custom config should be used in base agent for async
Scenario: Async invoke handles exceptions gracefully
Given I have a concrete agent instance in base agent
When the async workflow execution raises an exception in base agent
And I ainvoke the agent with input data in base agent
Then the async error should be caught in base agent
And the async result should contain an error field in base agent
And the async result should preserve input messages in base agent
And the async error should be logged in base agent
Scenario: Stream yields workflow execution events
Given I have a concrete agent instance in base agent
When I stream the agent with input data in base agent:
"""
{
"messages": [{"role": "user", "content": "stream test"}],
"context": {}
}
"""
Then the workflow should stream events in base agent
And each event should be yielded in base agent
Scenario: Stream uses default config when none provided
Given I have a concrete agent instance in base agent
When I stream the agent without providing config in base agent
Then the default config should be used in base agent for streaming
Scenario: Stream uses provided custom config
Given I have a concrete agent instance in base agent
When I stream the agent with custom config: in base agent
"""
{
"configurable": {"thread_id": "stream-thread-789"}
}
"""
Then the custom config should be used in base agent for streaming
Scenario: Stream handles exceptions gracefully
Given I have a concrete agent instance in base agent
When the stream execution raises an exception in base agent
And I stream the agent with input data in base agent
Then the stream error should be caught in base agent
And an error event should be yielded in base agent
And the stream error should be logged in base agent
Scenario: Memory saver is initialized correctly
Given I have a concrete agent instance in base agent
Then the memory attribute should be a MemorySaver instance in base agent
Scenario: Graph is compiled with checkpointer
Given I have a concrete agent instance in base agent
Then the app should be compiled from the graph in base agent
And the app should use the memory checkpointer in base agent
Scenario: Build graph is called during initialization
When I create a concrete agent with default parameters in base agent
Then the build_graph method should be called in base agent
And the graph should be stored in base agent
Scenario: Multiple provider switches work correctly
Given I have a concrete agent with provider "openai"
When I switch to provider "anthropic" with model "claude-3"
And I switch to provider "google" with model "gemini-pro" a second time in base agent
Then the agent provider should be "google" in base agent
And the agent model should be "gemini-pro" in base agent
And the llm should reflect the latest provider in base agent
Scenario: Provider kwargs are preserved between operations
Given I have a concrete agent with provider kwargs: in base agent
| key | value |
| max_tokens | 1000 |
When I invoke the agent with input data in base agent
Then the provider_kwargs should still contain "max_tokens"
Scenario: Logging captures LLM creation details
Given logging is enabled at INFO level in base agent
When I create a concrete agent with provider "openai" and model "gpt-4"
Then the log should contain "Creating openai LLM" in base agent
And the log should contain "model=gpt-4" in base agent
And the log should contain "temperature=0.7" in base agent
Scenario: Logging captures provider switch details
Given logging is enabled at INFO level in base agent
And I have a concrete agent with provider "openai"
When I switch to provider "anthropic" with model "claude-3"
Then the log should contain "Switched to anthropic provider" in base agent
And the log should contain "model claude-3" in base agent
Scenario: Logging captures workflow errors
Given logging is enabled at INFO level in base agent
And I have a concrete agent instance in base agent
When the workflow execution raises an exception in base agent with message "Test error"
And I invoke the agent with input data in base agent
Then the log should contain "Error executing agent workflow" in base agent
And the log should contain "Test error" in base agent
Scenario: Logging captures async workflow errors
Given logging is enabled at INFO level in base agent
And I have a concrete agent instance in base agent
When the async workflow execution raises an exception in base agent with message "Async test error"
And I ainvoke the agent with input data in base agent
Then the log should contain "Error executing agent workflow" in base agent
And the log should contain "Async test error" in base agent
Scenario: Logging captures stream errors
Given logging is enabled at INFO level in base agent
And I have a concrete agent instance in base agent
When the stream execution raises an exception in base agent with message "Stream test error"
And I stream the agent with input data in base agent
Then the log should contain "Error streaming agent workflow" in base agent
And the log should contain "Stream test error" in base agent
Scenario: AgentState TypedDict has correct structure
Given I can access AgentState in base agent
When I create an AgentState with all fields: in base agent
| field | type |
| messages | list |
| context | dict |
| result | dict |
| error | str |
| metadata | dict |
Then the state should store all fields correctly in base agent
Scenario: AgentState allows None for optional fields
Given I can access AgentState in base agent
When I create an AgentState with result None and error None in base agent
Then the state should accept None values in base agent
Scenario: Concrete agent graph builds with state
Given I have a concrete agent instance in base agent
Then the graph should be built with AgentState in base agent
Scenario: Invoke preserves messages from input in error case
Given I have a concrete agent instance in base agent
And I have input data with messages: in base agent
"""
[
{"role": "user", "content": "message 1"},
{"role": "assistant", "content": "message 2"}
]
"""
When the workflow execution raises an exception in base agent
And I invoke the agent with this input data in base agent
Then the error result should contain the original messages in base agent
Scenario: Temperature parameter is properly stored
When I create a concrete agent with temperature 0.1
Then the agent temperature should be 0.1
And the llm should be created with temperature 0.1
Scenario: Temperature parameter has correct default
When I create a concrete agent with default parameters in base agent
Then the agent temperature should be 0.7
Scenario: Model parameter is properly stored
When I create a concrete agent with model "custom-model"
Then the agent model should be "custom-model" in base agent
And the llm should be created with model "custom-model"
Scenario: Real BaseAgent subclass exercises base workflow methods
Given I have a real BaseAgent subclass instance in base agent
When I invoke the agent without providing config in base agent
Then the result should be returned in base agent
And the default config should be used in base agent
And the real base agent should record invoke thread "default"
When I ainvoke the agent without providing config in base agent
Then the async result should be returned in base agent
And the default config should be used for async in base agent
And the real base agent should record async thread "default"
When I stream the agent without providing config in base agent
Then the default config should be used for streaming in base agent
And each event should be yielded in base agent
And the real base agent should record stream thread "default"
Scenario: Real BaseAgent subclass surfaces workflow errors from base methods
Given I have a real BaseAgent subclass instance in base agent
When the workflow execution raises an exception in base agent with message "real invoke boom"
And I invoke the agent with input data in base agent
Then the error should be caught in base agent
And the result should contain an error field in base agent
And the result should preserve input messages in base agent
When the async workflow execution raises an exception in base agent with message "real async boom"
And I ainvoke the agent with input data in base agent
Then the async error should be caught in base agent
And the async result should contain an error field in base agent
And the async result should preserve input messages in base agent
When the stream execution raises an exception in base agent with message "real stream boom"
And I stream the agent with input data in base agent
Then the stream error should be caught in base agent
And an error event should be yielded in base agent
Reference in New Issue View Git Blame Copy Permalink