cleveragents-core/docs/reference/validation_pipeline.md

# Validation Pipeline

The **Validation Pipeline** orchestrates deterministic execution of validations
against plan resources, enforcing required vs informational semantics and
producing aggregated summaries for the plan lifecycle.

## Overview

When a plan reaches the end of its Execute phase, the validation pipeline
runs all attached validations.  Each validation is modelled as a
`ValidationCommand` that specifies the target resource, execution mode,
arguments, and timeout.  Results are collected into `ValidationResult`
objects and aggregated into a `ValidationSummary`.

```
ValidationCommand(s) ──► ValidationPipeline.run() ──► ValidationSummary
```

## Ordering Rules

Validations are sorted deterministically before execution:

1. **Resource name** (alphabetical)
2. **Mode** (`informational` before `required` by enum value)
3. **Validation name** (alphabetical)

This ensures repeatable execution order regardless of the order in which
validations are registered or attached.

## Required vs Informational Behaviour

| Mode            | On Pass | On Fail                                      |
|-----------------|---------|----------------------------------------------|
| `required`      | OK      | Blocks: `all_required_passed` becomes `False` |
| `informational` | OK      | Logged but does **not** block                 |

- A `ValidationSummary` with `all_required_passed == False` signals that
  the plan should not proceed to the Apply phase.
- Informational failures are logged at INFO level for review but do not
  affect the `all_required_passed` property.

## Timeout Handling

Each `ValidationCommand` carries a `timeout_seconds` field (default 30 s).
Timeouts are enforced using daemon threads with `thread.join(timeout=N)`.
If a validation exceeds its timeout:

- The daemon thread is abandoned (not blocked on).
- The result is marked `passed=False` and `timed_out=True`.
- The error field contains `TimeoutError: exceeded <N>s`.
- The duration reflects the wall-clock time up to the timeout.

## Concurrency and Capture

The pipeline uses a `concurrent.futures.ThreadPoolExecutor` with a
configurable `max_workers` (default 4).  Validations are submitted
concurrently but results are re-sorted to maintain deterministic ordering.

Before execution, `sys.stdout` and `sys.stderr` are replaced with
thread-local stream wrappers (`_ThreadLocalStream`).  Each worker thread
captures its own output independently, avoiding cross-thread pollution.
The original streams are restored in a `finally` block after all
validations complete.

Setting `max_workers=1` forces purely sequential execution, which is
useful for debugging or when validations have interdependencies.

## Output Normalisation

Validation executors may return arbitrary output.  The pipeline normalises
all results to a consistent schema:

| Executor Returns       | Normalised To                               |
|------------------------|---------------------------------------------|
| `{"passed": True, ...}`| Used directly                               |
| Non-dict value         | `passed=False`, message describes the error |
| Missing `passed` key   | Defaults to `False`                         |
| Missing `message` key  | Defaults to `"No message provided"`         |
| Non-dict `data` value  | Wrapped in `{"raw": <value>}`               |

When an executor raises an exception, the result is recorded as
`passed=False` with a message describing the exception type and details
(e.g., `Validation raised RuntimeError: ...`).  The exception
information is preserved in the `error` field.

Captured stdout/stderr during validation execution is stored in the
result's `data` dict under `_captured_stdout` and `_captured_stderr` keys.

## Read-Only Resource Guard

If a validation targets a resource marked as read-only (via the
`read_only_resources` parameter), the pipeline **skips** the validation
and returns a passing result with a descriptive message.  A warning is
logged when this occurs.

## Integration with Plan Lifecycle

The `run_for_plan()` method extends `run()` by persisting the validation
summary into the plan's metadata dict:

```python
pipeline = ValidationPipeline(commands, executor)
summary = pipeline.run_for_plan(plan_metadata=plan.metadata)
# plan.metadata["validation_summary"] now contains the full summary
```

This allows downstream phases (Apply, CLI display) to inspect validation
results without re-running the pipeline.

## Models Reference

### ValidationCommand

| Field              | Type             | Default | Description                |
|--------------------|------------------|---------|----------------------------|
| `validation_name`  | `str`            | —       | Validation identifier      |
| `resource_id`      | `str`            | —       | Target resource ULID       |
| `resource_name`    | `str`            | —       | Human-readable name        |
| `mode`             | `ValidationMode` | —       | `required`/`informational` |
| `arguments`        | `dict`           | `{}`    | Executor arguments         |
| `timeout_seconds`  | `float`          | `30.0`  | Timeout in seconds         |

### ValidationResult

| Field              | Type             | Default | Description                |
|--------------------|------------------|---------|----------------------------|
| `validation_name`  | `str`            | —       | Validation identifier      |
| `resource_id`      | `str`            | —       | Resource ULID              |
| `resource_name`    | `str`            | —       | Human-readable name        |
| `mode`             | `ValidationMode` | —       | `required`/`informational` |
| `passed`           | `bool`           | —       | Pass/fail status           |
| `message`          | `str`            | —       | Result description         |
| `data`             | `dict \| None`   | `None`  | Structured data            |
| `duration_ms`      | `float`          | —       | Execution time (ms)        |
| `error`            | `str \| None`    | `None`  | Error message              |
| `timed_out`        | `bool`           | `False` | Timeout flag               |

### ValidationSummary

| Field                   | Type                    | Description              |
|-------------------------|-------------------------|--------------------------|
| `total`                 | `int`                   | Total validations        |
| `required_passed`       | `int`                   | Required passes          |
| `required_failed`       | `int`                   | Required failures        |
| `informational_passed`  | `int`                   | Informational passes     |
| `informational_failed`  | `int`                   | Informational failures   |
| `results`               | `list[ValidationResult]`| All results              |
| `all_required_passed`   | `bool` (property)       | True if no req. failures |

---

# Validation Pipeline — Apply Gating

## Overview

The validation apply gate enforces that attached validations pass before
the apply phase proceeds. Validations are attached to resources and run
automatically during the Execute-to-Apply transition. Required validation
failures block apply; informational failures are recorded but do not block.

## Attachment Model

### `ValidationAttachment`

Represents a validation bound to a resource with scope and mode.

| Field             | Type              | Description                          |
|-------------------|-------------------|--------------------------------------|
| `attachment_id`   | `str`             | Unique ULID for this attachment      |
| `validation_name` | `str`             | Name of the validation tool          |
| `resource_id`     | `str`             | Target resource ID                   |
| `resource_name`   | `str`             | Human-readable resource name         |
| `mode`            | `ValidationMode`  | `required` or `informational`        |
| `scope`           | `AttachmentScope` | `direct`, `project`, or `plan`       |
| `scope_target`    | `str`             | Project name or plan ID for scoping  |
| `arguments`       | `dict[str, Any]`  | Arguments passed to the validation   |

### `AttachmentScope`

| Value     | Description                                        |
|-----------|----------------------------------------------------|
| `direct`  | Always active for the resource                     |
| `project` | Active when resource is accessed via a project     |
| `plan`    | Active only for a specific plan                    |

When multiple scopes apply, the **union** of all applicable validations
is collected and run.

## Validation Modes

| Mode            | Behavior on Failure                              |
|-----------------|--------------------------------------------------|
| `required`      | Blocks apply, plan stays in Execute phase        |
| `informational` | Recorded in summary but does **not** block apply |

## Result Models

### `ApplyValidationResult`

Per-validation execution result.

| Field             | Type             | Description                         |
|-------------------|------------------|-------------------------------------|
| `attachment_id`   | `str`            | Which attachment was evaluated      |
| `validation_name` | `str`            | Validation tool name                |
| `resource_id`     | `str`            | Target resource                     |
| `mode`            | `ValidationMode` | Required or informational           |
| `passed`          | `bool`           | Whether the validation passed       |
| `message`         | `str`            | Result message                      |
| `data`            | `dict`           | Additional structured data          |
| `duration_ms`     | `int`            | Execution time in milliseconds      |
| `error`           | `str | None`     | Error message if execution failed   |
| `timed_out`       | `bool`           | Whether execution timed out         |

### `ApplyValidationSummary`

Aggregated results with gating decision.

**Properties:** `total`, `required_passed`, `required_failed`,
`informational_passed`, `informational_failed`, `all_required_passed`,
`is_empty`

**Methods:**
- `to_plan_metadata()` — Dict for plan `validation_summary` field
- `format_cli_output()` — Human-readable summary for CLI display

## Runner Interface

### `ValidationRunner` (ABC)

Abstract interface for executing a single validation:

```python
def run_validation(
    self,
    attachment: ValidationAttachment,
    context: dict[str, Any],
) -> ApplyValidationResult: ...
```

### `DefaultValidationRunner`

Default stub using text matching. Checks if the validation name or
argument values appear in the context. Production implementations
should invoke the actual validation tool via the tool execution pipeline.

## Apply Validation Gate

### `ApplyValidationGate`

Orchestrates the full validation-before-apply flow:

```python
gate = ApplyValidationGate(runner=DefaultValidationRunner())
summary = gate.run(plan_id, attachments, context)

if gate.should_block_apply(summary):
    reasons = gate.get_failure_reasons(summary)
    # Block apply, report reasons
else:
    # Proceed to apply
```

### Example Output

When apply is **blocked**:

```
Validation Gate: BLOCKED
  Required: 1 passed, 1 failed
  Informational: 0 passed, 0 failed
  Failures:
    - lint-check on res-001: 3 lint errors found
```

When apply is **allowed**:

```
Validation Gate: PASSED
  Required: 2 passed, 0 failed
  Informational: 1 passed, 0 failed
```

## Integration with Plan Metadata

The validation summary is stored in the plan's `validation_summary` field
via `ApplyValidationSummary.to_plan_metadata()`:

```python
{
    "total": 3,
    "required_passed": 2,
    "required_failed": 1,
    "informational_passed": 0,
    "informational_failed": 0,
    "all_required_passed": false,
    "evaluated_at": "2026-02-22T10:30:00+00:00",
    "results": [...]
}
```