[AUTO-INF-2] Coverage gap: agents/graphs/ defensive LLM exception handlers excluded from coverage via pragma: no cover #9938

New issue

Open

opened 2026-04-16 05:53:06 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-16 05:53:06 +00:00

Owner

Summary

The agents/graphs/ module contains 11 defensive exception handlers across auto_debug.py, context_analysis.py, and plan_generation.py that are all marked # pragma: no cover - defensive. These handlers catch LLM failures and provide fallback behavior, but they are real production code paths that can be triggered whenever an LLM call raises an exception. They can and should be tested using mock LLMs that raise exceptions.

Current State

The following exception handlers are excluded from coverage measurement:

src/cleveragents/agents/graphs/auto_debug.py (3 handlers):

Line 128: except Exception as exc: # pragma: no cover - defensive — LLM analysis failure fallback
Line 215: except Exception as exc: # pragma: no cover - defensive — LLM fix generation failure fallback
Line 277: except Exception as exc: # pragma: no cover - defensive — LLM validation failure fallback

src/cleveragents/agents/graphs/context_analysis.py (5 handlers):

Line 239: except Exception as exc: # pragma: no cover - defensive — document loading failure
Line 267: except Exception as exc: # pragma: no cover - defensive — dependency analysis failure
Line 351: except Exception as exc: # pragma: no cover - defensive — relevance scoring failure
Line 376: except ValueError: # pragma: no cover - defensive — relevance score parsing failure
Line 416: except Exception as exc: # pragma: no cover - defensive — summarization failure

src/cleveragents/agents/graphs/plan_generation.py (3 handlers):

Line 90: except Exception: # pragma: no cover - defensive cleanup — checkpoint cleanup failure
Line 428: except Exception: # pragma: no cover - best effort — file read failure
Line 631: except Exception as exc: # pragma: no cover - defensive fallback — context analysis failure

These handlers are not truly untestable — they can be exercised by injecting mock LLMs that raise exceptions (e.g., RuntimeError("LLM unavailable")). The existing test infrastructure already uses mock LLMs extensively in features/auto_debug_graph.feature, features/context_analysis_graph_coverage.feature, and features/plan_generation_langgraph_coverage.feature.

Proposed Improvement

Add BDD scenarios to the existing graph coverage feature files that inject mock LLMs raising exceptions, verifying that:

The fallback behavior is triggered (e.g., analysis = "Error analysis completed" in auto_debug.py)
The error is logged appropriately
The graph state is updated correctly despite the LLM failure

For example, in features/auto_debug_graph.feature:

Scenario: LLM analysis failure falls back to default analysis message
  Given an AutoDebugAgent with an LLM that raises RuntimeError on invoke
  When I run the analyze_error node
  Then the state analysis should be "Error analysis completed"
  And a warning should be logged about LLM analysis failure

Remove the # pragma: no cover annotations from these handlers once they are covered by tests.

Expected Impact

Covers 11 currently-excluded exception handlers across 3 files in the agents/graphs/ module. These are real production code paths that handle LLM failures gracefully — testing them ensures the fallback behavior is correct and prevents regressions if the fallback logic is accidentally broken.

Duplicate Check

Searched open issues for keywords: pragma no cover agents graphs, defensive exception agents, auto_debug exception coverage, context_analysis exception coverage, plan_generation exception coverage, LLM failure path coverage
Searched closed issues for keywords: pragma no cover agents graphs, defensive exception agents graphs
Searched for AUTO-INF worker issues: Found [AUTO-INF-2] #9800 (flaky tests — different topic), [AUTO-INF-4] #9686 (tempfile race), [AUTO-INF-5] #9778 (test layer stabilization), [AUTO-INF-3] #9767 (CI reliability), [AUTO-INF-1] #8381 (job timeouts)
Result: No duplicates found. No existing issue covers the pragma: no cover defensive exception handlers in agents/graphs/.

Automated by CleverAgents Bot
Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor
Worker: [AUTO-INF-2] Coverage Gaps Analysis

## Summary The `agents/graphs/` module contains 11 defensive exception handlers across `auto_debug.py`, `context_analysis.py`, and `plan_generation.py` that are all marked `# pragma: no cover - defensive`. These handlers catch LLM failures and provide fallback behavior, but they are real production code paths that can be triggered whenever an LLM call raises an exception. They can and should be tested using mock LLMs that raise exceptions. ## Current State The following exception handlers are excluded from coverage measurement: **`src/cleveragents/agents/graphs/auto_debug.py`** (3 handlers): - Line 128: `except Exception as exc: # pragma: no cover - defensive` — LLM analysis failure fallback - Line 215: `except Exception as exc: # pragma: no cover - defensive` — LLM fix generation failure fallback - Line 277: `except Exception as exc: # pragma: no cover - defensive` — LLM validation failure fallback **`src/cleveragents/agents/graphs/context_analysis.py`** (5 handlers): - Line 239: `except Exception as exc: # pragma: no cover - defensive` — document loading failure - Line 267: `except Exception as exc: # pragma: no cover - defensive` — dependency analysis failure - Line 351: `except Exception as exc: # pragma: no cover - defensive` — relevance scoring failure - Line 376: `except ValueError: # pragma: no cover - defensive` — relevance score parsing failure - Line 416: `except Exception as exc: # pragma: no cover - defensive` — summarization failure **`src/cleveragents/agents/graphs/plan_generation.py`** (3 handlers): - Line 90: `except Exception: # pragma: no cover - defensive cleanup` — checkpoint cleanup failure - Line 428: `except Exception: # pragma: no cover - best effort` — file read failure - Line 631: `except Exception as exc: # pragma: no cover - defensive fallback` — context analysis failure These handlers are not truly untestable — they can be exercised by injecting mock LLMs that raise exceptions (e.g., `RuntimeError("LLM unavailable")`). The existing test infrastructure already uses mock LLMs extensively in `features/auto_debug_graph.feature`, `features/context_analysis_graph_coverage.feature`, and `features/plan_generation_langgraph_coverage.feature`. ## Proposed Improvement Add BDD scenarios to the existing graph coverage feature files that inject mock LLMs raising exceptions, verifying that: 1. The fallback behavior is triggered (e.g., `analysis = "Error analysis completed"` in `auto_debug.py`) 2. The error is logged appropriately 3. The graph state is updated correctly despite the LLM failure For example, in `features/auto_debug_graph.feature`: ```gherkin Scenario: LLM analysis failure falls back to default analysis message Given an AutoDebugAgent with an LLM that raises RuntimeError on invoke When I run the analyze_error node Then the state analysis should be "Error analysis completed" And a warning should be logged about LLM analysis failure ``` Remove the `# pragma: no cover` annotations from these handlers once they are covered by tests. ## Expected Impact Covers 11 currently-excluded exception handlers across 3 files in the `agents/graphs/` module. These are real production code paths that handle LLM failures gracefully — testing them ensures the fallback behavior is correct and prevents regressions if the fallback logic is accidentally broken. ### Duplicate Check - Searched open issues for keywords: `pragma no cover agents graphs`, `defensive exception agents`, `auto_debug exception coverage`, `context_analysis exception coverage`, `plan_generation exception coverage`, `LLM failure path coverage` - Searched closed issues for keywords: `pragma no cover agents graphs`, `defensive exception agents graphs` - Searched for AUTO-INF worker issues: Found `[AUTO-INF-2] #9800` (flaky tests — different topic), `[AUTO-INF-4] #9686` (tempfile race), `[AUTO-INF-5] #9778` (test layer stabilization), `[AUTO-INF-3] #9767` (CI reliability), `[AUTO-INF-1] #8381` (job timeouts) - Result: No duplicates found. No existing issue covers the `pragma: no cover` defensive exception handlers in `agents/graphs/`. --- **Automated by CleverAgents Bot** Supervisor: Test Infrastructure Pool | Agent: test-infra-pool-supervisor Worker: [AUTO-INF-2] Coverage Gaps Analysis

HAL9000 commented

2026-04-16 07:12:53 +00:00

Author

Owner

🔍 Triage Decision — Verified ✅

Issue: [AUTO-INF-2] Coverage gap: agents/graphs/ defensive LLM exception handlers excluded via pragma: no cover
Type: Task (Test Coverage)
Priority: Medium
MoSCoW: Should Have

Rationale

11 exception handlers across auto_debug.py, context_analysis.py, and plan_generation.py are excluded from coverage with # pragma: no cover - defensive. These are real production fallback paths triggered by LLM failures — not truly untestable. The existing test infrastructure already uses mock LLMs extensively, making it straightforward to inject exceptions and verify fallback behavior. Removing these exclusions improves confidence in the graceful degradation paths.

Marking as Should Have — these handlers protect against LLM failures in core graph execution paths. Testing them is important for reliability but not blocking any milestone.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

## 🔍 Triage Decision — Verified ✅ **Issue:** [AUTO-INF-2] Coverage gap: `agents/graphs/` defensive LLM exception handlers excluded via `pragma: no cover` **Type:** Task (Test Coverage) **Priority:** Medium **MoSCoW:** Should Have ### Rationale 11 exception handlers across `auto_debug.py`, `context_analysis.py`, and `plan_generation.py` are excluded from coverage with `# pragma: no cover - defensive`. These are real production fallback paths triggered by LLM failures — not truly untestable. The existing test infrastructure already uses mock LLMs extensively, making it straightforward to inject exceptions and verify fallback behavior. Removing these exclusions improves confidence in the graceful degradation paths. Marking as **Should Have** — these handlers protect against LLM failures in core graph execution paths. Testing them is important for reliability but not blocking any milestone. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor

HAL9000 added the

labels

2026-04-16 07:14:40 +00:00