cleveragents-core/docs/reference/semantic_validation_coverage.md

Semantic Validation Coverage Expectations

Overview

The semantic validation subsystem provides static analysis of Python source code through a pluggable rule-based architecture. This document describes the test coverage expectations for the validation fixture suites.

Rule Coverage Matrix

Each built-in rule is exercised by dedicated fixture suites that test both passing (no violation detected) and failing (violation correctly flagged) scenarios.

Rule	Fixture Suite	Pass Scenarios	Fail Scenarios
SyntaxCheckRule	Language-porting mismatches	Valid Python patterns (diamond inheritance, metaclasses, walrus operator, star unpacking)	—
MissingImportRule	Dependency graph violations	Valid stdlib private imports (_thread), mixed valid imports	Private module imports, from-private imports
BrokenReferenceRule	API surface changes, Cross-file symbols	Valid API usage, properly imported cross-module symbols	Renamed function references, incompatible type usage, undefined cross-module references, wildcard imports (known limitation)
DuplicateImportRule	Dependency graph violations, Duplicate relative imports	Mixed valid imports, clean relative imports, absolute imports (not flagged), syntax errors (skipped)	Duplicate relative imports, triple duplicates, direct duplicates, multiple self-references
APIMisuseRule	Language-porting mismatches, API surface changes	List comprehension side effects (not flagged), dynamic attribute access, aliased imports (known limitation)	eval-based dynamic calls, eval in porting context, exec for code generation
MissingSymbolRule	API surface changes, Cross-file symbols	Closure variables (enclosing scope tracking), deeply nested closures	Missing symbols in functions, class method missing symbols, undefined in class methods

Fixture Categories

1. Language-Porting Mismatches

Fixtures in features/fixtures/validation/language_porting_mismatches.json cover Python-specific patterns that are difficult or impossible to port to other languages: list comprehension side effects, dynamic attribute access, diamond inheritance, metaclasses, star unpacking, and walrus operators.

Coverage expectation: Pass fixtures verify that valid Python porting patterns are not falsely flagged. Fail fixtures (eval_in_porting_context, exec_for_code_generation) verify that genuinely dangerous API usage is detected regardless of the porting context.

2. Dependency Graph Violations

Fixtures in features/fixtures/validation/dependency_graph_violations.json cover import-related violations: duplicate relative imports (cycle indicators), private module imports not in stdlib, and valid import patterns as negative controls.

Coverage expectation: Failure fixtures trigger duplicate_import or missing_import rules. Pass fixtures confirm no false positives on valid patterns.

3. API Surface Changes

Fixtures in features/fixtures/validation/api_surface_changes.json cover renamed functions, missing symbols, incompatible types, dynamic API calls via eval, and known limitation fixtures for import aliasing bypass.

Coverage expectation: Failure fixtures trigger broken_reference, missing_symbol, or api_misuse rules. Pass fixtures confirm valid API usage is not flagged. Aliasing limitation fixtures (aliased_pickle_not_caught, from_import_alias_not_caught) document that APIMisuseRule does not resolve import aliases.

4. Cross-File Symbol Resolution

Fixtures in features/fixtures/validation/cross_file_symbols.json cover cross-module references, properly imported symbols, closure scoping (including deeply nested closures), class method undefined references, and wildcard imports.

Coverage expectation: Single-file rules cannot detect true cross-file resolution failures when the symbol appears imported. Closure fixtures verify that MissingSymbolRule correctly tracks enclosing function scopes. Fixtures test the boundaries of what single-file analysis can detect.

5. Duplicate Relative Import Detection

Fixtures in features/fixtures/validation/circular_import_detection.json cover duplicate relative imports within a single file, multiple self- references, clean imports, and edge cases (absolute imports, syntax errors).

Note: DuplicateImportRule detects duplicate from . import X statements within a single file as a heuristic for structural problems. It does not perform cross-file circular dependency detection. The fixture category name reflects duplicate relative import detection, not true circular import analysis.

Coverage expectation: DuplicateImportRule flags repeated relative from . import X patterns as duplicate indicators. Absolute imports are not flagged. Syntax errors cause the rule to skip gracefully.

Known Limitations

The following known limitations are documented via fixture tests:

Limitation	Rule	Fixture	Description
Import aliasing bypass	APIMisuseRule	aliased_pickle_not_caught, from_import_alias_not_caught	Aliased imports (import X as Y, from X import func as alias) bypass detection because the rule matches module/function names literally
Wildcard import resolution	BrokenReferenceRule	wildcard_import_symbol	Names imported via from X import * are not tracked, causing false positives on valid wildcard-imported names
No cross-file cycle detection	DuplicateImportRule	—	Only detects duplicates within a single file; cannot trace import cycles across modules

Test Infrastructure

• Shared utilities: features/fixtures/validation/suite_helpers.py — common fixture loading, lookup, rule-map, and verification logic used by all three test frameworks.
• BDD (Behave): features/semantic_validation_suite.feature — 38 scenarios covering all five fixture categories with individual fixture validation and schema-validated loading tests.
• Integration (Robot): robot/semantic_validation_suite.robot — 11 test cases that load fixtures and run them through rules via a helper script.
• Benchmarks (ASV): benchmarks/semantic_validation_suite_bench.py — 6 benchmark suites measuring fixture load time, individual suite runtime, and combined full-suite runtime.

Coverage Target

All semantic validation code must maintain >=97% line coverage as measured by nox -s coverage_report. The fixture suites contribute to this target by exercising rule logic across diverse code patterns and edge cases.