forked from cleveragents/cleveragents-core
588 lines
23 KiB
Gherkin
588 lines
23 KiB
Gherkin
Feature: Auto Debug Agent Coverage
|
|
As a developer
|
|
I want comprehensive test coverage for the AutoDebugAgent
|
|
So that I can ensure the auto-debug workflow works correctly
|
|
|
|
Background:
|
|
Given the auto debug agent module is importable
|
|
And I have a mock LLM provider configured for auto debug
|
|
And logging is enabled at INFO level for auto debug
|
|
|
|
Scenario: AutoDebugAgent can be instantiated with default parameters
|
|
When I create an AutoDebugAgent with default parameters
|
|
Then the agent should be initialized successfully for auto debug
|
|
And the agent should have a max_fix_attempts attribute set to 3
|
|
And the agent should have an llm provider configured for auto debug
|
|
|
|
Scenario: AutoDebugAgent can be instantiated with custom parameters
|
|
When I create an AutoDebugAgent with parameters:
|
|
| parameter | value |
|
|
| provider | openai |
|
|
| model | gpt-4 |
|
|
| temperature | 0.3 |
|
|
| max_fix_attempts | 5 |
|
|
Then the agent should be initialized successfully for auto debug
|
|
And the agent max_fix_attempts should be 5
|
|
And the agent temperature should be 0.3 for auto debug
|
|
|
|
Scenario: AutoDebugAgent builds a valid workflow graph
|
|
Given I have an AutoDebugAgent instance
|
|
When I build the workflow graph for auto debug
|
|
Then the graph should contain node "analyze_error" for auto debug
|
|
And the graph should contain node "generate_fix" for auto debug
|
|
And the graph should contain node "validate_fix" for auto debug
|
|
And the graph should contain node "finalize" for auto debug
|
|
And the entry point should be "analyze_error" for auto debug
|
|
|
|
Scenario: Analyze error step processes error message and code context
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with error details:
|
|
"""
|
|
{
|
|
"error_message": "NameError: name 'x' is not defined",
|
|
"code_context": "def test():\n return x + 1"
|
|
}
|
|
"""
|
|
When I execute the analyze_error step
|
|
Then the state messages should contain an error_analysis message
|
|
And the error_analysis should mention "Error analysis completed"
|
|
|
|
Scenario: Generate fix step creates a fix suggestion
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with error analysis completed
|
|
When I execute the generate_fix step
|
|
Then the state should contain current_fix
|
|
And the current_fix should have a description field
|
|
And the current_fix should have a code field
|
|
|
|
Scenario: Validate fix step validates the proposed fix
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Define variable x before use",
|
|
"code": "def test():\n x = 0\n return x + 1"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the state should contain fix_validated
|
|
And the fix_validated should be true
|
|
|
|
Scenario: Should retry fix returns "retry" when validation fails and under max attempts
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with validation results for auto debug:
|
|
"""
|
|
{
|
|
"fix_validated": false,
|
|
"attempted_fixes": []
|
|
}
|
|
"""
|
|
When I check if retry is needed
|
|
Then the decision should be "retry" for auto debug
|
|
|
|
Scenario: Should retry fix returns "retry" on second attempt
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with validation results for auto debug:
|
|
"""
|
|
{
|
|
"fix_validated": false,
|
|
"attempted_fixes": [{"attempt": 1}]
|
|
}
|
|
"""
|
|
When I check if retry is needed
|
|
Then the decision should be "retry" for auto debug
|
|
|
|
Scenario: Should retry fix returns "done" when validation succeeds
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with validation results for auto debug:
|
|
"""
|
|
{
|
|
"fix_validated": true,
|
|
"attempted_fixes": []
|
|
}
|
|
"""
|
|
When I check if retry is needed
|
|
Then the decision should be "done" for auto debug
|
|
|
|
Scenario: Should retry fix returns "done" when max attempts reached
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with validation results for auto debug:
|
|
"""
|
|
{
|
|
"fix_validated": false,
|
|
"attempted_fixes": [{"attempt": 1}, {"attempt": 2}, {"attempt": 3}]
|
|
}
|
|
"""
|
|
When I check if retry is needed
|
|
Then the decision should be "done" for auto debug
|
|
|
|
Scenario: Finalize step creates final result structure with success
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with successful fix:
|
|
"""
|
|
{
|
|
"fix_validated": true,
|
|
"current_fix": {"description": "Fixed", "code": "fixed code"},
|
|
"attempted_fixes": []
|
|
}
|
|
"""
|
|
When I execute the finalize step for auto debug
|
|
Then the state should contain a result field for auto debug
|
|
And the result should have a success field set to true for auto debug
|
|
And the result should have a fix field
|
|
And the result should have an attempts field
|
|
|
|
Scenario: Finalize step marks failure when validation fails after max attempts
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with failed validation after max attempts
|
|
When I execute the finalize step for auto debug
|
|
Then the result success field should be false for auto debug
|
|
|
|
Scenario: AutoDebugState holds required workflow data
|
|
Given I can create an AutoDebugState
|
|
When I initialize it with all required fields for auto debug:
|
|
| field | type |
|
|
| error_message | str |
|
|
| code_context | str |
|
|
| attempted_fixes | list |
|
|
| current_fix | dict |
|
|
| fix_validated | bool |
|
|
Then the state should store all fields correctly for auto debug
|
|
|
|
Scenario: Workflow graph edges connect nodes correctly
|
|
Given I have an AutoDebugAgent instance
|
|
When I build the workflow graph for auto debug
|
|
Then "analyze_error" should connect to "generate_fix" for auto debug
|
|
And "generate_fix" should connect to "validate_fix" for auto debug
|
|
And "validate_fix" should have conditional edges to "generate_fix" and "finalize" for auto debug
|
|
And "finalize" should connect to END for auto debug
|
|
|
|
Scenario: Full workflow with successful first fix
|
|
Given I have an AutoDebugAgent instance
|
|
And I have initial state with for auto debug:
|
|
"""
|
|
{
|
|
"error_message": "TypeError: unsupported operand",
|
|
"code_context": "x = '5' + 5",
|
|
"messages": [],
|
|
"attempted_fixes": []
|
|
}
|
|
"""
|
|
And the mock workflow returns valid fix on first attempt
|
|
When I run the complete workflow for auto debug
|
|
Then the workflow should complete successfully for auto debug
|
|
And the final result should have success true for auto debug
|
|
And the attempts should be 0
|
|
|
|
Scenario: Full workflow with one retry cycle
|
|
Given I have an AutoDebugAgent instance
|
|
And I have initial state with error details
|
|
And the mock workflow returns invalid fix on first attempt
|
|
And the mock workflow returns valid fix on second attempt
|
|
When I run the complete workflow for auto debug
|
|
Then the workflow should complete successfully for auto debug
|
|
And the attempts should be 1
|
|
|
|
Scenario: Full workflow reaching max fix attempts
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 2
|
|
And I have initial state with error details
|
|
And the mock workflow always returns invalid fix
|
|
When I run the complete workflow for auto debug
|
|
Then the workflow should complete for auto debug
|
|
And the attempts should be 2
|
|
And the final result success should be false for auto debug
|
|
|
|
Scenario: Analyze error logs appropriate messages
|
|
Given I have an AutoDebugAgent instance
|
|
And logging is enabled at INFO level for auto debug
|
|
And I have a state with error message and code context
|
|
When I execute the analyze_error step
|
|
Then the log should contain "Analyzing error message"
|
|
|
|
Scenario: Generate fix logs fix generation
|
|
Given I have an AutoDebugAgent instance
|
|
And logging is enabled at INFO level for auto debug
|
|
And I have a state with error analysis
|
|
When I execute the generate_fix step
|
|
Then the log should contain "Generating fix suggestion"
|
|
|
|
Scenario: Validate fix logs validation activity
|
|
Given I have an AutoDebugAgent instance
|
|
And logging is enabled at INFO level for auto debug
|
|
And I have a state with current fix
|
|
When I execute the validate_fix step
|
|
Then the log should contain "Validating fix"
|
|
|
|
Scenario: Finalize logs result summary
|
|
Given I have an AutoDebugAgent instance
|
|
And logging is enabled at INFO level for auto debug
|
|
And I have a state with validated fix
|
|
When I execute the finalize step for auto debug
|
|
Then the log should contain "Finalizing auto-debug results"
|
|
|
|
Scenario: Analyze error handles missing error message gracefully
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with incomplete error details
|
|
When I execute the analyze_error step
|
|
Then the state messages should be updated
|
|
|
|
Scenario: Generate fix handles empty attempted fixes list
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with no attempted fixes
|
|
When I execute the generate_fix step
|
|
Then the state should contain current_fix
|
|
|
|
Scenario: Validate fix handles missing current fix field
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state without current fix
|
|
When I execute the validate_fix step
|
|
Then the state should contain fix_validated
|
|
|
|
Scenario: Should retry fix handles missing fix_validated field
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state without fix_validated field
|
|
When I check if retry is needed
|
|
Then the decision should be "retry" for auto debug
|
|
|
|
Scenario: Should retry fix handles missing attempted_fixes field
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state without attempted_fixes field
|
|
When I check if retry is needed
|
|
Then the decision should be determined correctly
|
|
|
|
Scenario: Finalize handles missing attempted_fixes gracefully
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with minimal fields
|
|
When I execute the finalize step for auto debug
|
|
Then the result should have an attempts field with value 0
|
|
|
|
Scenario: State preserves messages through workflow
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with existing messages
|
|
When I execute the analyze_error step
|
|
Then the state should preserve previous messages
|
|
|
|
Scenario: Current fix updates between generate and validate
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state after error analysis
|
|
When I execute the generate_fix step
|
|
And I execute the validate_fix step
|
|
Then the current_fix should still be present
|
|
|
|
Scenario: AutoDebugAgent inherits from BaseAgent
|
|
Given I have an AutoDebugAgent instance
|
|
Then the agent should have provider attribute
|
|
And the agent should have model attribute
|
|
And the agent should have temperature attribute
|
|
And the agent should have llm attribute
|
|
And the agent should have graph attribute
|
|
|
|
Scenario: Should retry handles exactly max attempts
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with exactly max attempted fixes
|
|
When I check if retry is needed
|
|
Then the decision should be "done" for auto debug
|
|
|
|
Scenario: Analyze error appends to existing messages
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with 2 existing messages for auto debug
|
|
When I execute the analyze_error step
|
|
Then the state should have 3 messages
|
|
|
|
Scenario: Finalize result reflects validation status accurately
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with fix_validated as true
|
|
When I execute the finalize step for auto debug
|
|
Then the result success should match fix_validated
|
|
|
|
Scenario: Finalize result includes current fix details
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with detailed current fix
|
|
When I execute the finalize step for auto debug
|
|
Then the result fix should contain description
|
|
And the result fix should contain code
|
|
|
|
Scenario: Generate fix can be called multiple times
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state after first fix attempt
|
|
When I execute the generate_fix step
|
|
And I execute the generate_fix step again
|
|
Then both fix generations should complete
|
|
|
|
Scenario: Workflow entry point is analyze_error
|
|
Given I have an AutoDebugAgent instance
|
|
When I inspect the workflow graph for auto debug
|
|
Then the entry point should be "analyze_error" for auto debug
|
|
|
|
Scenario: Validate fix always sets fix_validated field
|
|
Given I have an AutoDebugAgent instance
|
|
And I have a state with any current fix
|
|
When I execute the validate_fix step
|
|
Then the fix_validated field should be present
|
|
|
|
Scenario: Should retry evaluates fix_validated correctly when false
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with fix_validated false and 1 attempt
|
|
When I check if retry is needed
|
|
Then the decision should be "retry" for auto debug
|
|
|
|
Scenario: Should retry evaluates fix_validated correctly when true
|
|
Given I have an AutoDebugAgent instance with max_fix_attempts of 3
|
|
And I have a state with fix_validated true and 1 attempt
|
|
When I check if retry is needed
|
|
Then the decision should be "done" for auto debug
|
|
|
|
Scenario: Max fix attempts can be configured on initialization
|
|
When I create an AutoDebugAgent with max_fix_attempts of 10
|
|
Then the agent max_fix_attempts should be 10
|
|
|
|
Scenario: Temperature defaults to 0.3 for deterministic debugging
|
|
When I create an AutoDebugAgent with default parameters
|
|
Then the agent temperature should be 0.3 for auto debug
|
|
|
|
|
|
|
|
# Coverage for lines 133: analyze_error with real LLM response (not "Mock LLM response")
|
|
Scenario: Analyze error with non-mock LLM response returns actual content
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a non-mock analysis response "This is a type mismatch error"
|
|
And I have a state with error details:
|
|
"""
|
|
{
|
|
"error_message": "TypeError: expected int but got str",
|
|
"code_context": "x = '5' + 5"
|
|
}
|
|
"""
|
|
When I execute the analyze_error step
|
|
Then the error_analysis content should be "This is a type mismatch error"
|
|
|
|
# Coverage for lines 134-136: analyze_error exception handling
|
|
Scenario: Analyze error handles LLM exception gracefully
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM raises an exception during analysis
|
|
And I have a state with error details:
|
|
"""
|
|
{
|
|
"error_message": "SyntaxError: invalid syntax",
|
|
"code_context": "def foo(:"
|
|
}
|
|
"""
|
|
When I execute the analyze_error step
|
|
Then the error_analysis content should be "Error analysis completed"
|
|
And a warning should be logged containing "LLM analysis failed"
|
|
|
|
# Coverage for lines 223-225: generate_fix with valid JSON response
|
|
Scenario: Generate fix parses valid JSON from non-mock LLM response
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a valid JSON fix response
|
|
And I have a state with error analysis completed
|
|
When I execute the generate_fix step
|
|
Then the current_fix description should be "Add variable definition"
|
|
And the current_fix code should be "x = 0; return x + 1"
|
|
And the current_fix files_to_modify should contain "test.py"
|
|
|
|
# Coverage for lines 226-232: generate_fix with invalid JSON response (fallback)
|
|
Scenario: Generate fix falls back when LLM returns non-JSON response
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a non-JSON fix response "Just add x = 0 before the return"
|
|
And I have a state with error analysis completed
|
|
When I execute the generate_fix step
|
|
Then the current_fix description should contain "Fix attempt"
|
|
And the current_fix code should be "Just add x = 0 before the return"
|
|
And the current_fix files_to_modify should be empty
|
|
|
|
# Coverage for lines 233-239: generate_fix exception handling
|
|
Scenario: Generate fix handles LLM exception gracefully
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM raises an exception during fix generation
|
|
And I have a state with error analysis completed
|
|
When I execute the generate_fix step
|
|
Then the current_fix should have a description field
|
|
And the current_fix should have a code field
|
|
And a warning should be logged containing "LLM fix generation failed"
|
|
|
|
# Coverage for lines 297-299: validate_fix with valid JSON response (is_valid true)
|
|
Scenario: Validate fix parses valid JSON with is_valid true from non-mock response
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a valid JSON validation response with is_valid true
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Add variable x before use",
|
|
"code": "x = 0\nreturn x + 1"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be true
|
|
|
|
# Coverage for lines 297-299: validate_fix with valid JSON response (is_valid false)
|
|
Scenario: Validate fix parses valid JSON with is_valid false from non-mock response
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a valid JSON validation response with is_valid false
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Incorrect fix attempt",
|
|
"code": "return x"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be false
|
|
|
|
# Coverage for lines 300-305: validate_fix JSON decode error with positive fallback
|
|
Scenario: Validate fix falls back to text parsing with positive indicators
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a non-JSON validation response "The fix is valid and correct"
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Add x = 0",
|
|
"code": "x = 0; return x + 1"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be true
|
|
|
|
# Coverage for lines 300-305: validate_fix JSON decode error without positive indicators
|
|
Scenario: Validate fix falls back to text parsing without positive indicators
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a non-JSON validation response "The code still has issues"
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Broken fix",
|
|
"code": "return undefined_var"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be false
|
|
|
|
# Coverage for lines 306-309: validate_fix exception handling
|
|
Scenario: Validate fix handles LLM exception gracefully
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM raises an exception during validation
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Some fix",
|
|
"code": "some code"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be true
|
|
And a warning should be logged containing "LLM validation failed"
|
|
|
|
# Coverage for lines 314-317: tracking failed fix attempts
|
|
Scenario: Validate fix tracks failed attempts when validation is false
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a valid JSON validation response with is_valid false
|
|
And I have a state with current fix and empty attempted_fixes:
|
|
"""
|
|
{
|
|
"description": "Failed fix",
|
|
"code": "bad code"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be false
|
|
And the attempted_fixes should have 1 entry
|
|
And the attempted_fixes should contain the current fix
|
|
|
|
# Coverage for lines 314-317: multiple failed attempts are tracked
|
|
Scenario: Validate fix appends to existing attempted_fixes when validation fails
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a valid JSON validation response with is_valid false
|
|
And I have a state with current fix and one existing attempted fix
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be false
|
|
And the attempted_fixes should have 2 entries
|
|
|
|
# Coverage: validate_fix with "resolves" in response
|
|
Scenario: Validate fix detects "resolves" keyword in fallback parsing
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a non-JSON validation response "This change resolves the issue"
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Fix that resolves issue",
|
|
"code": "fixed code"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be true
|
|
|
|
# Coverage: validate_fix with "fixes" in response
|
|
Scenario: Validate fix detects "fixes" keyword in fallback parsing
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a non-JSON validation response "This fixes the problem"
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Fix that fixes problem",
|
|
"code": "fixed code"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be true
|
|
|
|
# Coverage: generate_fix includes previous attempts in prompt
|
|
Scenario: Generate fix includes previous attempt descriptions in prompt
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a valid JSON fix response
|
|
And I have a state with two previous fix attempts
|
|
When I execute the generate_fix step
|
|
Then the current_fix should have a description field
|
|
And the attempt number should be 3
|
|
|
|
# Edge case: Empty content from LLM in generate_fix
|
|
Scenario: Generate fix handles empty non-mock response
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns an empty non-mock response for fix generation
|
|
And I have a state with error analysis completed
|
|
When I execute the generate_fix step
|
|
Then the current_fix should have a description field
|
|
And the current_fix code should be empty string
|
|
|
|
# Edge case: Validation response with is_valid as string
|
|
Scenario: Validate fix handles is_valid as string "true" in JSON
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a JSON validation response with is_valid as string "true"
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Fix",
|
|
"code": "code"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be true
|
|
|
|
# Edge case: Validation response with is_valid missing
|
|
Scenario: Validate fix handles missing is_valid field in JSON response
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a JSON validation response without is_valid field
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Fix",
|
|
"code": "code"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be false
|
|
|
|
# Edge case: Case insensitive keyword detection in validation fallback
|
|
Scenario: Validate fix keyword detection is case insensitive
|
|
Given I have an AutoDebugAgent instance
|
|
And the LLM returns a non-JSON validation response "VALID solution found"
|
|
And I have a state with a current fix:
|
|
"""
|
|
{
|
|
"description": "Fix",
|
|
"code": "code"
|
|
}
|
|
"""
|
|
When I execute the validate_fix step
|
|
Then the fix_validated should be true
|
|
|