cleveragents-core/docs/reference/semantic_validation.md

# Semantic Validation

The **Semantic Validation Service** provides lightweight, AST-based semantic
analysis for Python projects during the strategize/execute phases. It detects
syntax errors, missing imports, broken references, duplicate imports, API
misuse, and missing symbols before code is applied. Non-Python files are
automatically skipped.

## Overview

Semantic checks are exposed as Validation tools attachable per resource via the
Tool Registry. Results integrate into the `ValidationPipeline` as informational
by default, meaning failures are reported but do not block the plan lifecycle
unless configured otherwise.

```
Source Code ──► SemanticValidationService.check_file() ──► [SemanticCheckResult, ...]
                        │
                        ├── SyntaxCheckRule
                        ├── MissingImportRule
                        ├── BrokenReferenceRule
                        ├── DuplicateImportRule
                        ├── APIMisuseRule
                        └── MissingSymbolRule
```

> **Note:** `check_file()` returns `[]` immediately for non-Python files
> (files without `.py` or `.pyi` extension).

## Built-in Rules

| Rule               | Description                                           | Default Severity |
|--------------------|-------------------------------------------------------|-----------------|
| `syntax_error`     | Parses source via `ast.parse`                          | `error`         |
| `missing_import`   | Flags imports not in `sys.stdlib_module_names`          | `warn`          |
| `broken_reference` | Detects names not defined in any scope (scope-aware)    | `warn`          |
| `duplicate_import` | Finds duplicate relative imports                        | `error`         |
| `api_misuse`       | AST-based detection of dangerous calls (see below)      | `warn`          |
| `missing_symbol`   | Finds undefined names in functions and class methods    | `info`          |

> **Rename:** The rule was previously registered as `dependency_cycle` and is
> now registered as `duplicate_import` to better reflect its purpose.
> The class alias `DependencyCycleRule` remains available for backward
> compatibility.

### APIMisuseRule Dangerous Calls

The `api_misuse` rule uses AST analysis (not regex) to detect calls to:

- `eval()`, `exec()`, `compile()`
- `__import__()`
- `os.system()`, `os.popen()`
- `subprocess.call()`, `subprocess.run()`, `subprocess.Popen()`
- `pickle.load()`, `pickle.loads()`
- `marshal.loads()`

Because the rule inspects `ast.Call` nodes, it does **not** false-positive on
string literals that happen to contain these function names.

## Severity Mapping

Each rule produces findings with a severity level. The severity determines
how the finding integrates with the `ValidationPipeline`:

| Severity | Pipeline Mode    | Behaviour                          |
|----------|------------------|------------------------------------|
| `error`  | `required`       | Failure **blocks** the operation   |
| `warn`   | `informational`  | Failure is reported, does not block |
| `info`   | `informational`  | Informational only                  |

The mapping is configurable via `validation.semantic.severity_mapping`.

## Required vs Informational Attachment Modes

Semantic validations attach to the `ValidationPipeline` in one of two modes:

- **Required** (`error` severity): If a required semantic validation fails,
  `ValidationSummary.all_required_passed` becomes `False`, blocking the plan
  from proceeding to the Apply phase.

- **Informational** (`warn`/`info` severity): If an informational validation
  fails, the finding is logged at INFO level for review but does **not**
  affect the `all_required_passed` property.

This allows teams to start with informational semantic checks and
progressively promote them to required as confidence grows.

## Configuration

Three configuration keys control semantic validation:

| Key                                      | Type   | Default | Description                            |
|------------------------------------------|--------|---------|----------------------------------------|
| `validation.semantic.enabled`            | `bool` | `true`  | Global on/off for semantic validation  |
| `validation.semantic.python.enabled`     | `bool` | `true`  | Python-specific checks on/off          |
| `validation.semantic.severity_mapping`   | `dict` | (below) | Per-rule severity overrides            |

Default severity mapping:

```yaml
validation.semantic.severity_mapping:
  syntax_error: error
  missing_import: warn
  broken_reference: warn
  duplicate_import: error
  api_misuse: warn
  missing_symbol: info
```

### Per-project Configuration

Set `validation.semantic.enabled: false` in the project configuration to
disable semantic validation for a specific project:

```yaml
# .cleveragents/config.yaml
validation:
  semantic:
    enabled: false
```

### Per-plan Configuration

Override in plan metadata to enable/disable per plan execution:

```yaml
validation:
  semantic:
    enabled: true
    python:
      enabled: true
```

## Rule Registry

The `SemanticRuleRegistry` manages pluggable validation rules:

```python
from cleveragents.application.services import (
    SemanticRuleRegistry,
    create_default_registry,
)

# Create with built-in rules
registry = create_default_registry()

# List registered rules
print(registry.list_rules())
# ['api_misuse', 'broken_reference', 'duplicate_import',
#  'missing_import', 'missing_symbol', 'syntax_error']
#
# Note: 'duplicate_import' is provided by DuplicateImportRule

# Look up a rule
rule = registry.get("syntax_error")

# Register a custom rule
registry.register(my_custom_rule)
```

### Custom Rule Protocol

Any object implementing the `SemanticValidationRule` protocol can be
registered:

```python
class MyCustomRule:
    @property
    def name(self) -> str:
        return "my_custom_check"

    def check(self, source: str, filename: str) -> SemanticCheckResult:
        # ... run analysis ...
        return SemanticCheckResult(
            passed=True,
            message="All good",
        )
```

## Caching

The `SemanticValidationCache` avoids re-running checks on unchanged files
by caching results keyed by `(rule_name, file_sha256)`. The cache uses an
**LRU (Least Recently Used)** eviction policy with a configurable
`max_size` (default: 4096 entries). All operations are **thread-safe**.

```python
from cleveragents.application.services import SemanticValidationCache

cache = SemanticValidationCache(max_size=256)
file_hash = cache.compute_hash(source_code)

# Check cache before running expensive rule
cached = cache.get("syntax_error", file_hash)
if cached is None:
    result = rule.check(source_code, filename)
    cache.put("syntax_error", file_hash, result)
```

The service automatically uses caching when `check_file()` is called.
Calling `check_file()` with the same source content will return cached
results without re-executing rules.

When the cache reaches `max_size`, the least-recently-used entry is
automatically evicted to make room for the new one.

## Output Schema Normalisation

All semantic check results follow a consistent normalised schema:

```json
{
  "passed": true,
  "message": "No syntax errors in test.py",
  "data": null
}
```

The `data` field contains structured finding details when present:

```json
{
  "passed": false,
  "message": "Syntax error at test.py:3: invalid syntax",
  "data": {
    "filename": "test.py",
    "line": 3,
    "detail": "invalid syntax"
  }
}
```

The `normalise_output()` method on `SemanticValidationService` produces
this schema from any `SemanticCheckResult`.

## Pipeline Integration

Use `as_pipeline_results()` to get results compatible with the
`ValidationPipeline`:

```python
service = SemanticValidationService()
results = service.as_pipeline_results(source, "module.py")

# Each result dict has:
# - passed: bool
# - message: str
# - data: dict | None
# - severity: str ("info" | "warn" | "error")
# - mode: str ("required" | "informational")
```

## API Reference

### `SemanticValidationService`

| Method                | Description                                    |
|-----------------------|------------------------------------------------|
| `check_file()`       | Run all (or selected) rules on source code     |
| `as_pipeline_results()` | Same as `check_file` but returns pipeline dicts |
| `normalise_output()`  | Normalise a single result to passed/message/data |
| `enabled`            | Whether semantic validation is globally enabled |
| `python_enabled`     | Whether Python-specific checks are enabled     |
| `registry`           | The rule registry instance                     |
| `cache`              | The result cache instance                      |