Files
cleveragents-core/features/auto_debug_coverage_boost.feature

146 lines
7.4 KiB
Gherkin

Feature: AutoDebug graph node coverage boost
Additional scenarios that exercise previously uncovered code paths
in the auto_debug.py graph module, targeting individual node methods
and edge-case LLM response handling.
Background:
Given the auto debug graph module is imported
# -----------------------------------------------------------------------
# _analyze_error node
# -----------------------------------------------------------------------
Scenario: Analyze error with Mock LLM response yields default analysis
Given an auto debug agent with a mock LLM returning "Mock LLM response"
When I call _analyze_error with an error state
Then the analysis message should be "Error analysis completed"
Scenario: Analyze error with real content passes through LLM output
Given an auto debug agent with a mock LLM returning "Root cause is a null pointer"
When I call _analyze_error with an error state
Then the analysis message should be "Root cause is a null pointer"
Scenario: Analyze error handles LLM invocation failure gracefully
Given an auto debug agent with a failing LLM
When I call _analyze_error with an error state
Then the analysis message should be "Error analysis completed"
# -----------------------------------------------------------------------
# _generate_fix node
# -----------------------------------------------------------------------
Scenario: Generate fix with Mock LLM response yields default fix data
Given an auto debug agent with a mock LLM returning "Mock LLM response"
And a state with an existing error analysis message
When I call _generate_fix with the prepared state
Then the current fix description should be "Fix suggestion"
And the current fix code should be "# Fixed code"
Scenario: Generate fix parses valid JSON from LLM
Given an auto debug agent with a mock LLM returning valid fix JSON
And a state with an existing error analysis message
When I call _generate_fix with the prepared state
Then the current fix description should be "Parsed JSON fix"
And the current fix code should be "print('fixed')"
Scenario: Generate fix falls back on invalid JSON from LLM
Given an auto debug agent with a mock LLM returning "not valid json at all"
And a state with an existing error analysis message
When I call _generate_fix with the prepared state
Then the current fix description should be "Fix attempt 1"
And the current fix code should be "not valid json at all"
Scenario: Generate fix handles LLM invocation failure gracefully
Given an auto debug agent with a failing LLM
And a state with an existing error analysis message
When I call _generate_fix with the prepared state
Then the current fix description should be "Fix suggestion"
And the current fix code should be "# Fixed code"
Scenario: Generate fix includes previous attempt count in fallback description
Given an auto debug agent with a mock LLM returning "still broken"
And a state with two previous attempted fixes
When I call _generate_fix with the prepared state
Then the current fix description should be "Fix attempt 3"
# -----------------------------------------------------------------------
# _validate_fix node
# -----------------------------------------------------------------------
Scenario: Validate fix with Mock LLM response marks fix as valid
Given an auto debug agent with a mock LLM returning "Mock LLM response"
And a state with a current fix to validate
When I call _validate_fix with the prepared state
Then the fix should be marked as validated
Scenario: Validate fix parses valid JSON with is_valid true
Given an auto debug agent with a mock LLM returning valid validation JSON true
And a state with a current fix to validate
When I call _validate_fix with the prepared state
Then the fix should be marked as validated
Scenario: Validate fix parses valid JSON with is_valid false
Given an auto debug agent with a mock LLM returning valid validation JSON false
And a state with a current fix to validate
When I call _validate_fix with the prepared state
Then the fix should not be marked as validated
And the current fix should be appended to attempted fixes
Scenario: Validate fix handles invalid JSON containing positive keywords
Given an auto debug agent with a mock LLM returning "The fix is valid and correct"
And a state with a current fix to validate
When I call _validate_fix with the prepared state
Then the fix should be marked as validated
Scenario: Validate fix handles invalid JSON without positive keywords
Given an auto debug agent with a mock LLM returning "The fix does not work"
And a state with a current fix to validate
When I call _validate_fix with the prepared state
Then the fix should not be marked as validated
And the current fix should be appended to attempted fixes
Scenario: Validate fix handles LLM invocation failure gracefully
Given an auto debug agent with a failing LLM
And a state with a current fix to validate
When I call _validate_fix with the prepared state
Then the fix should be marked as validated
# -----------------------------------------------------------------------
# _should_retry_fix routing
# -----------------------------------------------------------------------
Scenario: Should retry fix returns retry when validation failed and under max attempts
Given an auto debug agent with a mock LLM returning "Mock LLM response"
When I check should_retry_fix with validated false and 1 attempt
Then the routing decision should be "retry"
Scenario: Should retry fix returns done when validation succeeded
Given an auto debug agent with a mock LLM returning "Mock LLM response"
When I check should_retry_fix with validated true and 0 attempts
Then the routing decision should be "done"
Scenario: Should retry fix returns done when max attempts reached
Given an auto debug agent with a mock LLM returning "Mock LLM response" and max 2 fix attempts
When I check should_retry_fix with validated false and 2 attempts
Then the routing decision should be "done"
# -----------------------------------------------------------------------
# _finalize node
# -----------------------------------------------------------------------
Scenario: Finalize produces result dict from successful state
Given an auto debug agent with a mock LLM returning "Mock LLM response"
And a finalize state with fix_validated true and 0 attempted fixes
When I call _finalize with the prepared state
Then the result should indicate success true
And the result attempts count should be 0
Scenario: Finalize produces result dict from failed state
Given an auto debug agent with a mock LLM returning "Mock LLM response"
And a finalize state with fix_validated false and 3 attempted fixes
When I call _finalize with the prepared state
Then the result should indicate success false
And the result attempts count should be 3
# -----------------------------------------------------------------------
# Constructor edge case
# -----------------------------------------------------------------------
Scenario: AutoDebugAgent constructor raises ValueError when no LLM is provided
When I try to create an AutoDebugAgent with no LLM
Then a ValueError should be raised with the missing provider message