cleveragents-core/docs/reference/semantic_validation.md

Semantic Validation

The Semantic Validation Service provides lightweight, AST-based semantic analysis for Python projects during the strategize/execute phases. It detects syntax errors, missing imports, broken references, duplicate imports, API misuse, and missing symbols before code is applied. Non-Python files are automatically skipped.

Overview

Semantic checks are exposed as Validation tools attachable per resource via the Tool Registry. Results integrate into the ValidationPipeline as informational by default, meaning failures are reported but do not block the plan lifecycle unless configured otherwise.

Source Code ──► SemanticValidationService.check_file() ──► [SemanticCheckResult, ...]
                        │
                        ├── SyntaxCheckRule
                        ├── MissingImportRule
                        ├── BrokenReferenceRule
                        ├── DuplicateImportRule
                        ├── APIMisuseRule
                        └── MissingSymbolRule

Note: check_file() returns [] immediately for non-Python files (files without .py or .pyi extension).

Built-in Rules

Rule	Description	Default Severity
syntax_error	Parses source via ast.parse	error
missing_import	Flags imports not in sys.stdlib_module_names	warn
broken_reference	Detects names not defined in any scope (scope-aware)	warn
duplicate_import	Finds duplicate relative imports	error
api_misuse	AST-based detection of dangerous calls (see below)	warn
missing_symbol	Finds undefined names in functions and class methods	info

Rename: The rule was previously registered as dependency_cycle and is now registered as duplicate_import to better reflect its purpose. The class alias DependencyCycleRule remains available for backward compatibility.

APIMisuseRule Dangerous Calls

The api_misuse rule uses AST analysis (not regex) to detect calls to:

• eval(), exec(), compile()
• __import__()
• os.system(), os.popen()
• subprocess.call(), subprocess.run(), subprocess.Popen()
• pickle.load(), pickle.loads()
• marshal.loads()

Because the rule inspects ast.Call nodes, it does not false-positive on string literals that happen to contain these function names.

Severity Mapping

Each rule produces findings with a severity level. The severity determines how the finding integrates with the ValidationPipeline:

Severity	Pipeline Mode	Behaviour
error	required	Failure blocks the operation
warn	informational	Failure is reported, does not block
info	informational	Informational only

The mapping is configurable via validation.semantic.severity_mapping.

Required vs Informational Attachment Modes

Semantic validations attach to the ValidationPipeline in one of two modes:

• Required (error severity): If a required semantic validation fails, ValidationSummary.all_required_passed becomes False, blocking the plan from proceeding to the Apply phase.

• Informational (warn/info severity): If an informational validation fails, the finding is logged at INFO level for review but does not affect the all_required_passed property.

This allows teams to start with informational semantic checks and progressively promote them to required as confidence grows.

Configuration

Three configuration keys control semantic validation:

Key	Type	Default	Description
validation.semantic.enabled	bool	true	Global on/off for semantic validation
validation.semantic.python.enabled	bool	true	Python-specific checks on/off
validation.semantic.severity_mapping	dict	(below)	Per-rule severity overrides

Default severity mapping:

validation.semantic.severity_mapping:
  syntax_error: error
  missing_import: warn
  broken_reference: warn
  duplicate_import: error
  api_misuse: warn
  missing_symbol: info

Per-project Configuration

Set validation.semantic.enabled: false in the project configuration to disable semantic validation for a specific project:

# .cleveragents/config.yaml
validation:
  semantic:
    enabled: false

Per-plan Configuration

Override in plan metadata to enable/disable per plan execution:

validation:
  semantic:
    enabled: true
    python:
      enabled: true

Rule Registry

The SemanticRuleRegistry manages pluggable validation rules:

from cleveragents.application.services import (
    SemanticRuleRegistry,
    create_default_registry,
)

# Create with built-in rules
registry = create_default_registry()

# List registered rules
print(registry.list_rules())
# ['api_misuse', 'broken_reference', 'duplicate_import',
#  'missing_import', 'missing_symbol', 'syntax_error']
#
# Note: 'duplicate_import' is provided by DuplicateImportRule

# Look up a rule
rule = registry.get("syntax_error")

# Register a custom rule
registry.register(my_custom_rule)

Custom Rule Protocol

Any object implementing the SemanticValidationRule protocol can be registered:

class MyCustomRule:
    @property
    def name(self) -> str:
        return "my_custom_check"

    def check(self, source: str, filename: str) -> SemanticCheckResult:
        # ... run analysis ...
        return SemanticCheckResult(
            passed=True,
            message="All good",
        )

Caching

The SemanticValidationCache avoids re-running checks on unchanged files by caching results keyed by (rule_name, file_sha256). The cache uses an LRU (Least Recently Used) eviction policy with a configurable max_size (default: 4096 entries). All operations are thread-safe.

from cleveragents.application.services import SemanticValidationCache

cache = SemanticValidationCache(max_size=256)
file_hash = cache.compute_hash(source_code)

# Check cache before running expensive rule
cached = cache.get("syntax_error", file_hash)
if cached is None:
    result = rule.check(source_code, filename)
    cache.put("syntax_error", file_hash, result)

The service automatically uses caching when check_file() is called. Calling check_file() with the same source content will return cached results without re-executing rules.

When the cache reaches max_size, the least-recently-used entry is automatically evicted to make room for the new one.

Output Schema Normalisation

All semantic check results follow a consistent normalised schema:

{
  "passed": true,
  "message": "No syntax errors in test.py",
  "data": null
}

The data field contains structured finding details when present:

{
  "passed": false,
  "message": "Syntax error at test.py:3: invalid syntax",
  "data": {
    "filename": "test.py",
    "line": 3,
    "detail": "invalid syntax"
  }
}

The normalise_output() method on SemanticValidationService produces this schema from any SemanticCheckResult.

Pipeline Integration

Use as_pipeline_results() to get results compatible with the ValidationPipeline:

service = SemanticValidationService()
results = service.as_pipeline_results(source, "module.py")

# Each result dict has:
# - passed: bool
# - message: str
# - data: dict | None
# - severity: str ("info" | "warn" | "error")
# - mode: str ("required" | "informational")

API Reference

SemanticValidationService

Method	Description
check_file()	Run all (or selected) rules on source code
as_pipeline_results()	Same as check_file but returns pipeline dicts
normalise_output()	Normalise a single result to passed/message/data
enabled	Whether semantic validation is globally enabled
python_enabled	Whether Python-specific checks are enabled
registry	The rule registry instance
cache	The result cache instance