forked from cleveragents/cleveragents-core
349 lines
14 KiB
Gherkin
349 lines
14 KiB
Gherkin
Feature: Auto Debug Agent Coverage
|
|
As a developer
|
|
I want comprehensive test coverage for the AutoDebugAgent
|
|
So that I can ensure the auto-debug workflow works correctly
|
|
|
|
Background:
|
|
Given the auto debug agent module is importable
|
|
And I have a mock LLM provider configured for auto debug
|
|
|
|
Scenario: AutoDebugAgent can be instantiated with default parameters
|
|
When I create an AutoDebugAgent with default parameters
|
|
Then the agent should be initialized successfully for auto debug
|
|
And the agent should have a max_fix_attempts attribute set to 3
|
|
And the agent should have an llm provider configured for auto debug
|
|
|
|
Scenario: AutoDebugAgent can be instantiated with custom parameters
|
|
When I create an AutoDebugAgent with parameters:
|
|
| parameter | value |
|
|
| provider | openai |
|
|
| model | gpt-4 |
|
|
| temperature | 0.3 |
|
|
| max_fix_attempts | 5 |
|
|
Then the agent should be initialized successfully for auto debug
|
|
And the agent max_fix_attempts should be 5
|
|
And the agent temperature should be 0.3 for auto debug
|
|
|
|
Scenario: AutoDebugAgent builds a valid workflow graph
|
|
Given I have an AutoDebugAgent instance
|
|
When I build the workflow graph for auto debug
|
|
Then the graph should contain node "analyze_error" for auto debug
|
|
And the graph should contain node "generate_fix" for auto debug
|
|
And the graph should contain node "validate_fix" for auto debug
|
|
And the graph should contain node "finalize" for auto debug
|
|
And the entry point should be "analyze_error" for auto debug
|
|
|
|
Scenario: Analyze error step processes error message and code context
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with error details:
|
|
"""
|
|
{
|
|
"error_message": "NameError: name 'x' is not defined",
|
|
"code_context": "def test():\n return x + 1"
|
|
}
|
|
"""
|
|
When I execute the analyze_error step
|
|
Then the state messages should contain an error_analysis message
|
|
And the error_analysis should mention "Error analysis completed"
|
|
|
|
Scenario: Generate fix step creates a fix suggestion
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with error analysis completed
|
|
When I execute the generate_fix step
|
|
Then the state should contain current_fix
|
|
And the current_fix should have a description field
|
|
And the current_fix should have a code field
|
|
|
|
Scenario: Validate fix step validates the proposed fix
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Define variable x before use",
|
|
"code": "def test():\n x = 0\n return x + 1"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the state should contain fix_validated
|
|
And the fix_validated should be true
|
|
|
|
Scenario: Should retry fix returns "retry" when validation fails and under max attempts
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with validation results for auto debug:
|
|
"""
|
|
{
|
|
"fix_validated": false,
|
|
"attempted_fixes": []
|
|
}
|
|
"""
|
|
When I check if retry is needed
|
|
Then the decision should be "retry" for auto debug
|
|
|
|
Scenario: Should retry fix returns "retry" on second attempt
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with validation results for auto debug:
|
|
"""
|
|
{
|
|
"fix_validated": false,
|
|
"attempted_fixes": [{"attempt": 1}]
|
|
}
|
|
"""
|
|
When I check if retry is needed
|
|
Then the decision should be "retry" for auto debug
|
|
|
|
Scenario: Should retry fix returns "done" when validation succeeds
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with validation results for auto debug:
|
|
"""
|
|
{
|
|
"fix_validated": true,
|
|
"attempted_fixes": []
|
|
}
|
|
"""
|
|
When I check if retry is needed
|
|
Then the decision should be "done" for auto debug
|
|
|
|
Scenario: Should retry fix returns "done" when max attempts reached
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with validation results for auto debug:
|
|
"""
|
|
{
|
|
"fix_validated": false,
|
|
"attempted_fixes": [{"attempt": 1}, {"attempt": 2}, {"attempt": 3}]
|
|
}
|
|
"""
|
|
When I check if retry is needed
|
|
Then the decision should be "done" for auto debug
|
|
|
|
Scenario: Finalize step creates final result structure with success
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with successful fix:
|
|
"""
|
|
{
|
|
"fix_validated": true,
|
|
"current_fix": {"description": "Fixed", "code": "fixed code"},
|
|
"attempted_fixes": []
|
|
}
|
|
"""
|
|
When I execute the finalize step for auto debug
|
|
Then the state should contain a result field for auto debug
|
|
And the result should have a success field set to true for auto debug
|
|
And the result should have a fix field
|
|
And the result should have an attempts field
|
|
|
|
Scenario: Finalize step marks failure when validation fails after max attempts
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with failed validation after max attempts
|
|
When I execute the finalize step for auto debug
|
|
Then the result success field should be false for auto debug
|
|
|
|
Scenario: AutoDebugState holds required workflow data
|
|
Given I can create an AutoDebugState
|
|
When I initialize it with all required fields for auto debug:
|
|
| field | type |
|
|
| error_message | str |
|
|
| code_context | str |
|
|
| attempted_fixes | list |
|
|
| current_fix | dict |
|
|
| fix_validated | bool |
|
|
Then the state should store all fields correctly for auto debug
|
|
|
|
Scenario: Workflow graph edges connect nodes correctly
|
|
Given I have an AutoDebugAgent instance
|
|
When I build the workflow graph for auto debug
|
|
Then "analyze_error" should connect to "generate_fix" for auto debug
|
|
And "generate_fix" should connect to "validate_fix" for auto debug
|
|
And "validate_fix" should have conditional edges to "generate_fix" and "finalize" for auto debug
|
|
And "finalize" should connect to END for auto debug
|
|
|
|
Scenario: Full workflow with successful first fix
|
|
Given I have an AutoDebugAgent instance
|
|
And I have initial state with for auto debug:
|
|
"""
|
|
{
|
|
"error_message": "TypeError: unsupported operand",
|
|
"code_context": "x = '5' + 5",
|
|
"messages": [],
|
|
"attempted_fixes": []
|
|
}
|
|
"""
|
|
And the mock workflow returns valid fix on first attempt
|
|
When I run the complete workflow for auto debug
|
|
Then the workflow should complete successfully for auto debug
|
|
And the final result should have success true for auto debug
|
|
And the attempts should be 0
|
|
|
|
Scenario: Full workflow with one retry cycle
|
|
Given I have an AutoDebugAgent instance
|
|
And I have initial state with error details
|
|
And the mock workflow returns invalid fix on first attempt
|
|
And the mock workflow returns valid fix on second attempt
|
|
When I run the complete workflow for auto debug
|
|
Then the workflow should complete successfully for auto debug
|
|
And the attempts should be 1
|
|
|
|
Scenario: Full workflow reaching max fix attempts
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 2
|
|
And I have initial state with error details
|
|
And the mock workflow always returns invalid fix
|
|
When I run the complete workflow for auto debug
|
|
Then the workflow should complete for auto debug
|
|
And the attempts should be 2
|
|
And the final result success should be false for auto debug
|
|
|
|
Scenario: Analyze error logs appropriate messages
|
|
Given I have an AutoDebugAgent instance
|
|
And logging is enabled at INFO level for auto debug
|
|
And I have a state with error message and code context
|
|
When I execute the analyze_error step
|
|
Then the log should contain "Analyzing error message"
|
|
|
|
Scenario: Generate fix logs fix generation
|
|
Given I have an AutoDebugAgent instance
|
|
And logging is enabled at INFO level for auto debug
|
|
And I have a state with error analysis
|
|
When I execute the generate_fix step
|
|
Then the log should contain "Generating fix suggestion"
|
|
|
|
Scenario: Validate fix logs validation activity
|
|
Given I have an AutoDebugAgent instance
|
|
And logging is enabled at INFO level for auto debug
|
|
And I have a state with current fix
|
|
When I execute the validate_fix step
|
|
Then the log should contain "Validating fix"
|
|
|
|
Scenario: Finalize logs result summary
|
|
Given I have an AutoDebugAgent instance
|
|
And logging is enabled at INFO level for auto debug
|
|
And I have a state with validated fix
|
|
When I execute the finalize step for auto debug
|
|
Then the log should contain "Finalizing auto-debug results"
|
|
|
|
Scenario: Analyze error handles missing error message gracefully
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with incomplete error details
|
|
When I execute the analyze_error step
|
|
Then the state messages should be updated
|
|
|
|
Scenario: Generate fix handles empty attempted fixes list
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with no attempted fixes
|
|
When I execute the generate_fix step
|
|
Then the state should contain current_fix
|
|
|
|
Scenario: Validate fix handles missing current fix field
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state without current fix
|
|
When I execute the validate_fix step
|
|
Then the state should contain fix_validated
|
|
|
|
Scenario: Should retry fix handles missing fix_validated field
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state without fix_validated field
|
|
When I check if retry is needed
|
|
Then the decision should be "retry" for auto debug
|
|
|
|
Scenario: Should retry fix handles missing attempted_fixes field
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state without attempted_fixes field
|
|
When I check if retry is needed
|
|
Then the decision should be determined correctly
|
|
|
|
Scenario: Finalize handles missing attempted_fixes gracefully
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with minimal fields
|
|
When I execute the finalize step for auto debug
|
|
Then the result should have an attempts field with value 0
|
|
|
|
Scenario: State preserves messages through workflow
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with existing messages
|
|
When I execute the analyze_error step
|
|
Then the state should preserve previous messages
|
|
|
|
Scenario: Current fix updates between generate and validate
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state after error analysis
|
|
When I execute the generate_fix step
|
|
And I execute the validate_fix step
|
|
Then the current_fix should still be present
|
|
|
|
Scenario: AutoDebugAgent inherits from BaseAgent
|
|
Given I have an AutoDebugAgent instance
|
|
Then the agent should have provider attribute
|
|
And the agent should have model attribute
|
|
And the agent should have temperature attribute
|
|
And the agent should have llm attribute
|
|
And the agent should have graph attribute
|
|
|
|
Scenario: Should retry handles exactly max attempts
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with exactly max attempted fixes
|
|
When I check if retry is needed
|
|
Then the decision should be "done" for auto debug
|
|
|
|
Scenario: Analyze error appends to existing messages
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with 2 existing messages for auto debug
|
|
When I execute the analyze_error step
|
|
Then the state should have 3 messages
|
|
|
|
Scenario: Finalize result reflects validation status accurately
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with fix_validated as true
|
|
When I execute the finalize step for auto debug
|
|
Then the result success should match fix_validated
|
|
|
|
Scenario: Finalize result includes current fix details
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with detailed current fix
|
|
When I execute the finalize step for auto debug
|
|
Then the result fix should contain description
|
|
And the result fix should contain code
|
|
|
|
Scenario: Generate fix can be called multiple times
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state after first fix attempt
|
|
When I execute the generate_fix step
|
|
And I execute the generate_fix step again
|
|
Then both fix generations should complete
|
|
|
|
Scenario: Workflow entry point is analyze_error
|
|
Given I have an AutoDebugAgent instance
|
|
When I inspect the workflow graph for auto debug
|
|
Then the entry point should be "analyze_error" for auto debug
|
|
|
|
Scenario: Validate fix always sets fix_validated field
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with any current fix
|
|
When I execute the validate_fix step
|
|
Then the fix_validated field should be present
|
|
|
|
Scenario: Should retry evaluates fix_validated correctly when false
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with fix_validated false and 1 attempt
|
|
When I check if retry is needed
|
|
Then the decision should be "retry" for auto debug
|
|
|
|
Scenario: Should retry evaluates fix_validated correctly when true
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with fix_validated true and 1 attempt
|
|
When I check if retry is needed
|
|
Then the decision should be "done" for auto debug
|
|
|
|
Scenario: Max fix attempts can be configured on initialization
|
|
When I create an AutoDebugAgent with max_fix_attempts of 10
|
|
Then the agent max_fix_attempts should be 10
|
|
|
|
Scenario: Temperature defaults to 0.3 for deterministic debugging
|
|
When I create an AutoDebugAgent with default parameters
|
|
Then the agent temperature should be 0.3 for auto debug
|
|
|
|
Scenario: AutoDebugAgent supports provider kwargs
|
|
When I create an AutoDebugAgent with provider_kwargs:
|
|
| kwarg | value |
|
|
| max_tokens | 2000 |
|
|
| top_p | 0.9 |
|
|
Then the agent should be initialized successfully for auto debug
|
|
And the agent should have provider_kwargs stored
|