ac74edd175
CI / lint (pull_request) Successful in 59s
CI / typecheck (pull_request) Successful in 1m16s
CI / quality (pull_request) Successful in 54s
CI / security (pull_request) Successful in 1m10s
CI / build (pull_request) Successful in 34s
CI / push-validation (pull_request) Successful in 35s
CI / helm (pull_request) Successful in 40s
CI / unit_tests (pull_request) Successful in 5m2s
CI / docker (pull_request) Successful in 1m30s
CI / integration_tests (pull_request) Successful in 9m8s
CI / coverage (pull_request) Successful in 9m18s
CI / status-check (pull_request) Successful in 5s
CI / lint (push) Successful in 52s
CI / build (push) Successful in 50s
CI / quality (push) Successful in 55s
CI / typecheck (push) Successful in 1m1s
CI / security (push) Successful in 1m4s
CI / push-validation (push) Successful in 24s
CI / helm (push) Successful in 56s
CI / unit_tests (push) Successful in 4m50s
CI / docker (push) Successful in 1m32s
CI / integration_tests (push) Successful in 8m29s
CI / coverage (push) Successful in 9m35s
CI / status-check (push) Successful in 3s
CI / benchmark-publish (push) Has been cancelled
CI / benchmark-regression (push) Has been cancelled
The _validate_fix BDD scenario "Validate fix handles LLM invocation failure gracefully" was asserting fix_validated=True (fail-open), but the code already sets is_valid=False on LLM exception (fail-safe). Update the scenario step to "the fix should not be marked as validated" and remove the pragma: no cover comment since this branch is now exercised by the test. ISSUES CLOSED: #10496
146 lines
7.4 KiB
Gherkin
146 lines
7.4 KiB
Gherkin
Feature: AutoDebug graph node coverage boost
|
|
Additional scenarios that exercise previously uncovered code paths
|
|
in the auto_debug.py graph module, targeting individual node methods
|
|
and edge-case LLM response handling.
|
|
|
|
Background:
|
|
Given the auto debug graph module is imported
|
|
|
|
# -----------------------------------------------------------------------
|
|
# _analyze_error node
|
|
# -----------------------------------------------------------------------
|
|
Scenario: Analyze error with Mock LLM response yields default analysis
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
When I call _analyze_error with an error state
|
|
Then the analysis message should be "Error analysis completed"
|
|
|
|
Scenario: Analyze error with real content passes through LLM output
|
|
Given an auto debug agent with a mock LLM returning "Root cause is a null pointer"
|
|
When I call _analyze_error with an error state
|
|
Then the analysis message should be "Root cause is a null pointer"
|
|
|
|
Scenario: Analyze error handles LLM invocation failure gracefully
|
|
Given an auto debug agent with a failing LLM
|
|
When I call _analyze_error with an error state
|
|
Then the analysis message should be "Error analysis completed"
|
|
|
|
# -----------------------------------------------------------------------
|
|
# _generate_fix node
|
|
# -----------------------------------------------------------------------
|
|
Scenario: Generate fix with Mock LLM response yields default fix data
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
And a state with an existing error analysis message
|
|
When I call _generate_fix with the prepared state
|
|
Then the current fix description should be "Fix suggestion"
|
|
And the current fix code should be "# Fixed code"
|
|
|
|
Scenario: Generate fix parses valid JSON from LLM
|
|
Given an auto debug agent with a mock LLM returning valid fix JSON
|
|
And a state with an existing error analysis message
|
|
When I call _generate_fix with the prepared state
|
|
Then the current fix description should be "Parsed JSON fix"
|
|
And the current fix code should be "print('fixed')"
|
|
|
|
Scenario: Generate fix falls back on invalid JSON from LLM
|
|
Given an auto debug agent with a mock LLM returning "not valid json at all"
|
|
And a state with an existing error analysis message
|
|
When I call _generate_fix with the prepared state
|
|
Then the current fix description should be "Fix attempt 1"
|
|
And the current fix code should be "not valid json at all"
|
|
|
|
Scenario: Generate fix handles LLM invocation failure gracefully
|
|
Given an auto debug agent with a failing LLM
|
|
And a state with an existing error analysis message
|
|
When I call _generate_fix with the prepared state
|
|
Then the current fix description should be "Fix suggestion"
|
|
And the current fix code should be "# Fixed code"
|
|
|
|
Scenario: Generate fix includes previous attempt count in fallback description
|
|
Given an auto debug agent with a mock LLM returning "still broken"
|
|
And a state with two previous attempted fixes
|
|
When I call _generate_fix with the prepared state
|
|
Then the current fix description should be "Fix attempt 3"
|
|
|
|
# -----------------------------------------------------------------------
|
|
# _validate_fix node
|
|
# -----------------------------------------------------------------------
|
|
Scenario: Validate fix with Mock LLM response marks fix as valid
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should be marked as validated
|
|
|
|
Scenario: Validate fix parses valid JSON with is_valid true
|
|
Given an auto debug agent with a mock LLM returning valid validation JSON true
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should be marked as validated
|
|
|
|
Scenario: Validate fix parses valid JSON with is_valid false
|
|
Given an auto debug agent with a mock LLM returning valid validation JSON false
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should not be marked as validated
|
|
And the current fix should be appended to attempted fixes
|
|
|
|
Scenario: Validate fix handles invalid JSON containing positive keywords
|
|
Given an auto debug agent with a mock LLM returning "The fix is valid and correct"
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should be marked as validated
|
|
|
|
Scenario: Validate fix handles invalid JSON without positive keywords
|
|
Given an auto debug agent with a mock LLM returning "The fix does not work"
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should not be marked as validated
|
|
And the current fix should be appended to attempted fixes
|
|
|
|
Scenario: Validate fix handles LLM invocation failure gracefully
|
|
Given an auto debug agent with a failing LLM
|
|
And a state with a current fix to validate
|
|
When I call _validate_fix with the prepared state
|
|
Then the fix should not be marked as validated
|
|
|
|
# -----------------------------------------------------------------------
|
|
# _should_retry_fix routing
|
|
# -----------------------------------------------------------------------
|
|
Scenario: Should retry fix returns retry when validation failed and under max attempts
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
When I check should_retry_fix with validated false and 1 attempt
|
|
Then the routing decision should be "retry"
|
|
|
|
Scenario: Should retry fix returns done when validation succeeded
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
When I check should_retry_fix with validated true and 0 attempts
|
|
Then the routing decision should be "done"
|
|
|
|
Scenario: Should retry fix returns done when max attempts reached
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response" and max 2 fix attempts
|
|
When I check should_retry_fix with validated false and 2 attempts
|
|
Then the routing decision should be "done"
|
|
|
|
# -----------------------------------------------------------------------
|
|
# _finalize node
|
|
# -----------------------------------------------------------------------
|
|
Scenario: Finalize produces result dict from successful state
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
And a finalize state with fix_validated true and 0 attempted fixes
|
|
When I call _finalize with the prepared state
|
|
Then the result should indicate success true
|
|
And the result attempts count should be 0
|
|
|
|
Scenario: Finalize produces result dict from failed state
|
|
Given an auto debug agent with a mock LLM returning "Mock LLM response"
|
|
And a finalize state with fix_validated false and 3 attempted fixes
|
|
When I call _finalize with the prepared state
|
|
Then the result should indicate success false
|
|
And the result attempts count should be 3
|
|
|
|
# -----------------------------------------------------------------------
|
|
# Constructor edge case
|
|
# -----------------------------------------------------------------------
|
|
Scenario: AutoDebugAgent constructor raises ValueError when no LLM is provided
|
|
When I try to create an AutoDebugAgent with no LLM
|
|
Then a ValueError should be raised with the missing provider message
|