temp/docs/reference/semantic_validation_coverage.md

# Semantic Validation Coverage Expectations

## Overview

The semantic validation subsystem provides static analysis of Python source code
through a pluggable rule-based architecture.  This document describes the test
coverage expectations for the validation fixture suites.

## Rule Coverage Matrix

Each built-in rule is exercised by dedicated fixture suites that test both
passing (no violation detected) and failing (violation correctly flagged)
scenarios.

| Rule | Fixture Suite | Pass Scenarios | Fail Scenarios |
| :--- | :------------ | :------------- | :------------- |
| `SyntaxCheckRule` | Language-porting mismatches | Valid Python patterns (diamond inheritance, metaclasses, walrus operator, star unpacking) | — |
| `MissingImportRule` | Dependency graph violations | Valid stdlib private imports (`_thread`), mixed valid imports | Private module imports, from-private imports |
| `BrokenReferenceRule` | API surface changes, Cross-file symbols | Valid API usage, properly imported cross-module symbols | Renamed function references, incompatible type usage, undefined cross-module references, wildcard imports (known limitation) |
| `DuplicateImportRule` | Dependency graph violations, Duplicate relative imports | Mixed valid imports, clean relative imports, absolute imports (not flagged), syntax errors (skipped) | Duplicate relative imports, triple duplicates, direct duplicates, multiple self-references |
| `APIMisuseRule` | Language-porting mismatches, API surface changes | List comprehension side effects (not flagged), dynamic attribute access, aliased imports (known limitation) | eval-based dynamic calls, eval in porting context, exec for code generation |
| `MissingSymbolRule` | API surface changes, Cross-file symbols | Closure variables (enclosing scope tracking), deeply nested closures | Missing symbols in functions, class method missing symbols, undefined in class methods |

## Fixture Categories

### 1. Language-Porting Mismatches

Fixtures in `features/fixtures/validation/language_porting_mismatches.json`
cover Python-specific patterns that are difficult or impossible to port to
other languages: list comprehension side effects, dynamic attribute access,
diamond inheritance, metaclasses, star unpacking, and walrus operators.

**Coverage expectation:** Pass fixtures verify that valid Python porting
patterns are not falsely flagged.  Fail fixtures (`eval_in_porting_context`,
`exec_for_code_generation`) verify that genuinely dangerous API usage is
detected regardless of the porting context.

### 2. Dependency Graph Violations

Fixtures in `features/fixtures/validation/dependency_graph_violations.json`
cover import-related violations: duplicate relative imports (cycle indicators),
private module imports not in stdlib, and valid import patterns as negative
controls.

**Coverage expectation:** Failure fixtures trigger `duplicate_import` or
`missing_import` rules.  Pass fixtures confirm no false positives on valid
patterns.

### 3. API Surface Changes

Fixtures in `features/fixtures/validation/api_surface_changes.json` cover
renamed functions, missing symbols, incompatible types, dynamic API calls
via `eval`, and known limitation fixtures for import aliasing bypass.

**Coverage expectation:** Failure fixtures trigger `broken_reference`,
`missing_symbol`, or `api_misuse` rules.  Pass fixtures confirm valid API
usage is not flagged.  Aliasing limitation fixtures
(`aliased_pickle_not_caught`, `from_import_alias_not_caught`) document that
`APIMisuseRule` does not resolve import aliases.

### 4. Cross-File Symbol Resolution

Fixtures in `features/fixtures/validation/cross_file_symbols.json` cover
cross-module references, properly imported symbols, closure scoping (including
deeply nested closures), class method undefined references, and wildcard
imports.

**Coverage expectation:** Single-file rules cannot detect true cross-file
resolution failures when the symbol appears imported.  Closure fixtures
verify that `MissingSymbolRule` correctly tracks enclosing function scopes.
Fixtures test the boundaries of what single-file analysis can detect.

### 5. Duplicate Relative Import Detection

Fixtures in `features/fixtures/validation/circular_import_detection.json`
cover duplicate relative imports within a single file, multiple self-
references, clean imports, and edge cases (absolute imports, syntax errors).

> **Note:** `DuplicateImportRule` detects duplicate `from . import X`
> statements **within a single file** as a heuristic for structural problems.
> It does **not** perform cross-file circular dependency detection.  The
> fixture category name reflects duplicate relative import detection, not
> true circular import analysis.

**Coverage expectation:** `DuplicateImportRule` flags repeated relative
`from . import X` patterns as duplicate indicators.  Absolute imports are
not flagged.  Syntax errors cause the rule to skip gracefully.

## Known Limitations

The following known limitations are documented via fixture tests:

| Limitation | Rule | Fixture | Description |
| :--------- | :--- | :------ | :---------- |
| Import aliasing bypass | `APIMisuseRule` | `aliased_pickle_not_caught`, `from_import_alias_not_caught` | Aliased imports (`import X as Y`, `from X import func as alias`) bypass detection because the rule matches module/function names literally |
| Wildcard import resolution | `BrokenReferenceRule` | `wildcard_import_symbol` | Names imported via `from X import *` are not tracked, causing false positives on valid wildcard-imported names |
| No cross-file cycle detection | `DuplicateImportRule` | — | Only detects duplicates within a single file; cannot trace import cycles across modules |

## Test Infrastructure

- **Shared utilities:** `features/fixtures/validation/suite_helpers.py` — common
  fixture loading, lookup, rule-map, and verification logic used by all three
  test frameworks.
- **BDD (Behave):** `features/semantic_validation_suite.feature` — 38 scenarios
  covering all five fixture categories with individual fixture validation and
  schema-validated loading tests.
- **Integration (Robot):** `robot/semantic_validation_suite.robot` — 11 test
  cases that load fixtures and run them through rules via a helper script.
- **Benchmarks (ASV):** `benchmarks/semantic_validation_suite_bench.py` — 6
  benchmark suites measuring fixture load time, individual suite runtime, and
  combined full-suite runtime.

## Coverage Target

All semantic validation code must maintain **>=97%** line coverage as measured
by `nox -s coverage_report`.  The fixture suites contribute to this target by
exercising rule logic across diverse code patterns and edge cases.