CoreRasurae
301 lines
12 KiB
Markdown
301 lines
12 KiB
Markdown
# Validation Pipeline
|
|
|
|
The **Validation Pipeline** orchestrates deterministic execution of validations
|
|
against plan resources, enforcing required vs informational semantics and
|
|
producing aggregated summaries for the plan lifecycle.
|
|
|
|
## Overview
|
|
|
|
When a plan reaches the end of its Execute phase, the validation pipeline
|
|
runs all attached validations. Each validation is modelled as a
|
|
`ValidationCommand` that specifies the target resource, execution mode,
|
|
arguments, and timeout. Results are collected into `ValidationResult`
|
|
objects and aggregated into a `ValidationSummary`.
|
|
|
|
```
|
|
ValidationCommand(s) ──► ValidationPipeline.run() ──► ValidationSummary
|
|
```
|
|
|
|
## Ordering Rules
|
|
|
|
Validations are sorted deterministically before execution:
|
|
|
|
1. **Resource name** (alphabetical)
|
|
2. **Mode** (`informational` before `required` by enum value)
|
|
3. **Validation name** (alphabetical)
|
|
|
|
This ensures repeatable execution order regardless of the order in which
|
|
validations are registered or attached.
|
|
|
|
## Required vs Informational Behaviour
|
|
|
|
| Mode | On Pass | On Fail |
|
|
|-----------------|---------|----------------------------------------------|
|
|
| `required` | OK | Blocks: `all_required_passed` becomes `False` |
|
|
| `informational` | OK | Logged but does **not** block |
|
|
|
|
- A `ValidationSummary` with `all_required_passed == False` signals that
|
|
the plan should not proceed to the Apply phase.
|
|
- Informational failures are logged at INFO level for review but do not
|
|
affect the `all_required_passed` property.
|
|
|
|
## Timeout Handling
|
|
|
|
Each `ValidationCommand` carries a `timeout_seconds` field (default 30 s).
|
|
Timeouts are enforced using daemon threads with `thread.join(timeout=N)`.
|
|
If a validation exceeds its timeout:
|
|
|
|
- The daemon thread is abandoned (not blocked on).
|
|
- The result is marked `passed=False` and `timed_out=True`.
|
|
- The error field contains `TimeoutError: exceeded <N>s`.
|
|
- The duration reflects the wall-clock time up to the timeout.
|
|
|
|
## Concurrency and Capture
|
|
|
|
The pipeline uses a `concurrent.futures.ThreadPoolExecutor` with a
|
|
configurable `max_workers` (default 4). Validations are submitted
|
|
concurrently but results are re-sorted to maintain deterministic ordering.
|
|
|
|
Before execution, `sys.stdout` and `sys.stderr` are replaced with
|
|
thread-local stream wrappers (`_ThreadLocalStream`). Each worker thread
|
|
captures its own output independently, avoiding cross-thread pollution.
|
|
The original streams are restored in a `finally` block after all
|
|
validations complete.
|
|
|
|
Setting `max_workers=1` forces purely sequential execution, which is
|
|
useful for debugging or when validations have interdependencies.
|
|
|
|
## Output Normalisation
|
|
|
|
Validation executors may return arbitrary output. The pipeline normalises
|
|
all results to a consistent schema:
|
|
|
|
| Executor Returns | Normalised To |
|
|
|------------------------|---------------------------------------------|
|
|
| `{"passed": True, ...}`| Used directly |
|
|
| Non-dict value | `passed=False`, message describes the error |
|
|
| Missing `passed` key | Defaults to `False` |
|
|
| Missing `message` key | Defaults to `"No message provided"` |
|
|
| Non-dict `data` value | Wrapped in `{"raw": <value>}` |
|
|
|
|
When an executor raises an exception, the result is recorded as
|
|
`passed=False` with a message describing the exception type and details
|
|
(e.g., `Validation raised RuntimeError: ...`). The exception
|
|
information is preserved in the `error` field.
|
|
|
|
Captured stdout/stderr during validation execution is stored in the
|
|
result's `data` dict under `_captured_stdout` and `_captured_stderr` keys.
|
|
|
|
## Read-Only Resource Guard
|
|
|
|
If a validation targets a resource marked as read-only (via the
|
|
`read_only_resources` parameter), the pipeline **skips** the validation
|
|
and returns a passing result with a descriptive message. A warning is
|
|
logged when this occurs.
|
|
|
|
## Integration with Plan Lifecycle
|
|
|
|
The `run_for_plan()` method extends `run()` by persisting the validation
|
|
summary into the plan's metadata dict:
|
|
|
|
```python
|
|
pipeline = ValidationPipeline(commands, executor)
|
|
summary = pipeline.run_for_plan(plan_metadata=plan.metadata)
|
|
# plan.metadata["validation_summary"] now contains the full summary
|
|
```
|
|
|
|
This allows downstream phases (Apply, CLI display) to inspect validation
|
|
results without re-running the pipeline.
|
|
|
|
## Models Reference
|
|
|
|
### ValidationCommand
|
|
|
|
| Field | Type | Default | Description |
|
|
|--------------------|------------------|---------|----------------------------|
|
|
| `validation_name` | `str` | — | Validation identifier |
|
|
| `resource_id` | `str` | — | Target resource ULID |
|
|
| `resource_name` | `str` | — | Human-readable name |
|
|
| `mode` | `ValidationMode` | — | `required`/`informational` |
|
|
| `arguments` | `dict` | `{}` | Executor arguments |
|
|
| `timeout_seconds` | `float` | `30.0` | Timeout in seconds |
|
|
|
|
### ValidationResult
|
|
|
|
| Field | Type | Default | Description |
|
|
|--------------------|------------------|---------|----------------------------|
|
|
| `validation_name` | `str` | — | Validation identifier |
|
|
| `resource_id` | `str` | — | Resource ULID |
|
|
| `resource_name` | `str` | — | Human-readable name |
|
|
| `mode` | `ValidationMode` | — | `required`/`informational` |
|
|
| `passed` | `bool` | — | Pass/fail status |
|
|
| `message` | `str` | — | Result description |
|
|
| `data` | `dict \| None` | `None` | Structured data |
|
|
| `duration_ms` | `float` | — | Execution time (ms) |
|
|
| `error` | `str \| None` | `None` | Error message |
|
|
| `timed_out` | `bool` | `False` | Timeout flag |
|
|
|
|
### ValidationSummary
|
|
|
|
| Field | Type | Description |
|
|
|-------------------------|-------------------------|--------------------------|
|
|
| `total` | `int` | Total validations |
|
|
| `required_passed` | `int` | Required passes |
|
|
| `required_failed` | `int` | Required failures |
|
|
| `informational_passed` | `int` | Informational passes |
|
|
| `informational_failed` | `int` | Informational failures |
|
|
| `results` | `list[ValidationResult]`| All results |
|
|
| `all_required_passed` | `bool` (property) | True if no req. failures |
|
|
|
|
---
|
|
|
|
# Validation Pipeline — Apply Gating
|
|
|
|
## Overview
|
|
|
|
The validation apply gate enforces that attached validations pass before
|
|
the apply phase proceeds. Validations are attached to resources and run
|
|
automatically during the Execute-to-Apply transition. Required validation
|
|
failures block apply; informational failures are recorded but do not block.
|
|
|
|
## Attachment Model
|
|
|
|
### `ValidationAttachment`
|
|
|
|
Represents a validation bound to a resource with scope and mode.
|
|
|
|
| Field | Type | Description |
|
|
|-------------------|-------------------|--------------------------------------|
|
|
| `attachment_id` | `str` | Unique ULID for this attachment |
|
|
| `validation_name` | `str` | Name of the validation tool |
|
|
| `resource_id` | `str` | Target resource ID |
|
|
| `resource_name` | `str` | Human-readable resource name |
|
|
| `mode` | `ValidationMode` | `required` or `informational` |
|
|
| `scope` | `AttachmentScope` | `direct`, `project`, or `plan` |
|
|
| `scope_target` | `str` | Project name or plan ID for scoping |
|
|
| `arguments` | `dict[str, Any]` | Arguments passed to the validation |
|
|
|
|
### `AttachmentScope`
|
|
|
|
| Value | Description |
|
|
|-----------|----------------------------------------------------|
|
|
| `direct` | Always active for the resource |
|
|
| `project` | Active when resource is accessed via a project |
|
|
| `plan` | Active only for a specific plan |
|
|
|
|
When multiple scopes apply, the **union** of all applicable validations
|
|
is collected and run.
|
|
|
|
## Validation Modes
|
|
|
|
| Mode | Behavior on Failure |
|
|
|-----------------|--------------------------------------------------|
|
|
| `required` | Blocks apply, plan stays in Execute phase |
|
|
| `informational` | Recorded in summary but does **not** block apply |
|
|
|
|
## Result Models
|
|
|
|
### `ApplyValidationResult`
|
|
|
|
Per-validation execution result.
|
|
|
|
| Field | Type | Description |
|
|
|-------------------|------------------|-------------------------------------|
|
|
| `attachment_id` | `str` | Which attachment was evaluated |
|
|
| `validation_name` | `str` | Validation tool name |
|
|
| `resource_id` | `str` | Target resource |
|
|
| `mode` | `ValidationMode` | Required or informational |
|
|
| `passed` | `bool` | Whether the validation passed |
|
|
| `message` | `str` | Result message |
|
|
| `data` | `dict` | Additional structured data |
|
|
| `duration_ms` | `int` | Execution time in milliseconds |
|
|
| `error` | `str | None` | Error message if execution failed |
|
|
| `timed_out` | `bool` | Whether execution timed out |
|
|
|
|
### `ApplyValidationSummary`
|
|
|
|
Aggregated results with gating decision.
|
|
|
|
**Properties:** `total`, `required_passed`, `required_failed`,
|
|
`informational_passed`, `informational_failed`, `all_required_passed`,
|
|
`is_empty`
|
|
|
|
**Methods:**
|
|
- `to_plan_metadata()` — Dict for plan `validation_summary` field
|
|
- `format_cli_output()` — Human-readable summary for CLI display
|
|
|
|
## Runner Interface
|
|
|
|
### `ValidationRunner` (ABC)
|
|
|
|
Abstract interface for executing a single validation:
|
|
|
|
```python
|
|
def run_validation(
|
|
self,
|
|
attachment: ValidationAttachment,
|
|
context: dict[str, Any],
|
|
) -> ApplyValidationResult: ...
|
|
```
|
|
|
|
### `DefaultValidationRunner`
|
|
|
|
Default stub using text matching. Checks if the validation name or
|
|
argument values appear in the context. Production implementations
|
|
should invoke the actual validation tool via the tool execution pipeline.
|
|
|
|
## Apply Validation Gate
|
|
|
|
### `ApplyValidationGate`
|
|
|
|
Orchestrates the full validation-before-apply flow:
|
|
|
|
```python
|
|
gate = ApplyValidationGate(runner=DefaultValidationRunner())
|
|
summary = gate.run(plan_id, attachments, context)
|
|
|
|
if gate.should_block_apply(summary):
|
|
reasons = gate.get_failure_reasons(summary)
|
|
# Block apply, report reasons
|
|
else:
|
|
# Proceed to apply
|
|
```
|
|
|
|
### Example Output
|
|
|
|
When apply is **blocked**:
|
|
|
|
```
|
|
Validation Gate: BLOCKED
|
|
Required: 1 passed, 1 failed
|
|
Informational: 0 passed, 0 failed
|
|
Failures:
|
|
- lint-check on res-001: 3 lint errors found
|
|
```
|
|
|
|
When apply is **allowed**:
|
|
|
|
```
|
|
Validation Gate: PASSED
|
|
Required: 2 passed, 0 failed
|
|
Informational: 1 passed, 0 failed
|
|
```
|
|
|
|
## Integration with Plan Metadata
|
|
|
|
The validation summary is stored in the plan's `validation_summary` field
|
|
via `ApplyValidationSummary.to_plan_metadata()`:
|
|
|
|
```python
|
|
{
|
|
"total": 3,
|
|
"required_passed": 2,
|
|
"required_failed": 1,
|
|
"informational_passed": 0,
|
|
"informational_failed": 0,
|
|
"all_required_passed": false,
|
|
"evaluated_at": "2026-02-22T10:30:00+00:00",
|
|
"results": [...]
|
|
}
|
|
```
|