forked from cleveragents/cleveragents-core
146 lines
7.4 KiB
Gherkin
146 lines
7.4 KiB
Gherkin
Feature: AutoDebug graph node coverage boost
|
|
Additional scenarios that exercise previously uncovered code paths
|
|
in the auto_debug.py graph module, targeting individual node methods
|
|
and edge-case LLM response handling.
|
|
|
|
Background:
|
|
Given the auto debug graph module is imported
|
|
|
|
# -----------------------------------------------------------------------
|
|
# _analyze_error node
|
|
# -----------------------------------------------------------------------
|
|
Scenario: Analyze error with Mock LLM response yields default analysis
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
When I call _analyze_error with an error state
|
|
Then the analysis message should be "Error analysis completed"
|
|
|
|
Scenario: Analyze error with real content passes through LLM output
|
|
Given an auto debug agent with a mock LLM returning "Root cause is a null pointer"
|
|
When I call _analyze_error with an error state
|
|
Then the analysis message should be "Root cause is a null pointer"
|
|
|
|
Scenario: Analyze error handles LLM invocation failure gracefully
|
|
Given an auto debug agent with a failing LLM
|
|
When I call _analyze_error with an error state
|
|
Then the analysis message should be "Error analysis completed"
|
|
|
|
# -----------------------------------------------------------------------
|
|
# _generate_fix node
|
|
# -----------------------------------------------------------------------
|
|
Scenario: Generate fix with Mock LLM response yields default fix data
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
And a state with an existing error analysis message
|
|
When I call _generate_fix with the prepared state
|
|
Then the current fix description should be "Fix suggestion"
|
|
And the current fix code should be "# Fixed code"
|
|
|
|
Scenario: Generate fix parses valid JSON from LLM
|
|
Given an auto debug agent with a mock LLM returning valid fix JSON
|
|
And a state with an existing error analysis message
|
|
When I call _generate_fix with the prepared state
|
|
Then the current fix description should be "Parsed JSON fix"
|
|
And the current fix code should be "print('fixed')"
|
|
|
|
Scenario: Generate fix falls back on invalid JSON from LLM
|
|
Given an auto debug agent with a mock LLM returning "not valid json at all"
|
|
And a state with an existing error analysis message
|
|
When I call _generate_fix with the prepared state
|
|
Then the current fix description should be "Fix attempt 1"
|
|
And the current fix code should be "not valid json at all"
|
|
|
|
Scenario: Generate fix handles LLM invocation failure gracefully
|
|
Given an auto debug agent with a failing LLM
|
|
And a state with an existing error analysis message
|
|
When I call _generate_fix with the prepared state
|
|
Then the current fix description should be "Fix suggestion"
|
|
And the current fix code should be "# Fixed code"
|
|
|
|
Scenario: Generate fix includes previous attempt count in fallback description
|
|
Given an auto debug agent with a mock LLM returning "still broken"
|
|
And a state with two previous attempted fixes
|
|
When I call _generate_fix with the prepared state
|
|
Then the current fix description should be "Fix attempt 3"
|
|
|
|
# -----------------------------------------------------------------------
|
|
# _validate_fix node
|
|
# -----------------------------------------------------------------------
|
|
Scenario: Validate fix with Mock LLM response marks fix as valid
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should be marked as validated
|
|
|
|
Scenario: Validate fix parses valid JSON with is_valid true
|
|
Given an auto debug agent with a mock LLM returning valid validation JSON true
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should be marked as validated
|
|
|
|
Scenario: Validate fix parses valid JSON with is_valid false
|
|
Given an auto debug agent with a mock LLM returning valid validation JSON false
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should not be marked as validated
|
|
And the current fix should be appended to attempted fixes
|
|
|
|
Scenario: Validate fix handles invalid JSON containing positive keywords
|
|
Given an auto debug agent with a mock LLM returning "The fix is valid and correct"
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should be marked as validated
|
|
|
|
Scenario: Validate fix handles invalid JSON without positive keywords
|
|
Given an auto debug agent with a mock LLM returning "The fix does not work"
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should not be marked as validated
|
|
And the current fix should be appended to attempted fixes
|
|
|
|
Scenario: Validate fix handles LLM invocation failure gracefully
|
|
Given an auto debug agent with a failing LLM
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should be marked as validated
|
|
|
|
# -----------------------------------------------------------------------
|
|
# _should_retry_fix routing
|
|
# -----------------------------------------------------------------------
|
|
Scenario: Should retry fix returns retry when validation failed and under max attempts
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
When I check should_retry_fix with validated false and 1 attempt
|
|
Then the routing decision should be "retry"
|
|
|
|
Scenario: Should retry fix returns done when validation succeeded
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
When I check should_retry_fix with validated true and 0 attempts
|
|
Then the routing decision should be "done"
|
|
|
|
Scenario: Should retry fix returns done when max attempts reached
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response" and max 2 fix attempts
|
|
When I check should_retry_fix with validated false and 2 attempts
|
|
Then the routing decision should be "done"
|
|
|
|
# -----------------------------------------------------------------------
|
|
# _finalize node
|
|
# -----------------------------------------------------------------------
|
|
Scenario: Finalize produces result dict from successful state
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
And a finalize state with fix_validated true and 0 attempted fixes
|
|
When I call _finalize with the prepared state
|
|
Then the result should indicate success true
|
|
And the result attempts count should be 0
|
|
|
|
Scenario: Finalize produces result dict from failed state
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
And a finalize state with fix_validated false and 3 attempted fixes
|
|
When I call _finalize with the prepared state
|
|
Then the result should indicate success false
|
|
And the result attempts count should be 3
|
|
|
|
# -----------------------------------------------------------------------
|
|
# Constructor edge case
|
|
# -----------------------------------------------------------------------
|
|
Scenario: AutoDebugAgent constructor raises ValueError when no LLM is provided
|
|
When I try to create an AutoDebugAgent with no LLM
|
|
Then a ValueError should be raised with the missing provider message
|