cleveragents-core/docs/reference/validation_pipeline.md

Validation Pipeline

The Validation Pipeline orchestrates deterministic execution of validations against plan resources, enforcing required vs informational semantics and producing aggregated summaries for the plan lifecycle.

Overview

When a plan reaches the end of its Execute phase, the validation pipeline runs all attached validations. Each validation is modelled as a ValidationCommand that specifies the target resource, execution mode, arguments, and timeout. Results are collected into ValidationResult objects and aggregated into a ValidationSummary.

ValidationCommand(s) ──► ValidationPipeline.run() ──► ValidationSummary

Ordering Rules

Validations are sorted deterministically before execution:

1. Resource name (alphabetical)
2. Mode (informational before required by enum value)
3. Validation name (alphabetical)

This ensures repeatable execution order regardless of the order in which validations are registered or attached.

Required vs Informational Behaviour

Mode	On Pass	On Fail
required	OK	Blocks: all_required_passed becomes False
informational	OK	Logged but does not block

• A ValidationSummary with all_required_passed == False signals that the plan should not proceed to the Apply phase.
• Informational failures are logged at INFO level for review but do not affect the all_required_passed property.

Timeout Handling

Each ValidationCommand carries a timeout_seconds field (default 30 s). Timeouts are enforced using daemon threads with thread.join(timeout=N). If a validation exceeds its timeout:

• The daemon thread is abandoned (not blocked on).
• The result is marked passed=False and timed_out=True.
• The error field contains TimeoutError: exceeded <N>s.
• The duration reflects the wall-clock time up to the timeout.

Concurrency and Capture

The pipeline uses a concurrent.futures.ThreadPoolExecutor with a configurable max_workers (default 4). Validations are submitted concurrently but results are re-sorted to maintain deterministic ordering.

Before execution, sys.stdout and sys.stderr are replaced with thread-local stream wrappers (_ThreadLocalStream). Each worker thread captures its own output independently, avoiding cross-thread pollution. The original streams are restored in a finally block after all validations complete.

Setting max_workers=1 forces purely sequential execution, which is useful for debugging or when validations have interdependencies.

Output Normalisation

Validation executors may return arbitrary output. The pipeline normalises all results to a consistent schema:

Executor Returns	Normalised To
{"passed": True, ...}	Used directly
Non-dict value	passed=False, message describes the error
Missing passed key	Defaults to False
Missing message key	Defaults to "No message provided"
Non-dict data value	Wrapped in {"raw": <value>}

When an executor raises an exception, the result is recorded as passed=False with a message describing the exception type and details (e.g., Validation raised RuntimeError: ...). The exception information is preserved in the error field.

Captured stdout/stderr during validation execution is stored in the result's data dict under _captured_stdout and _captured_stderr keys.

Read-Only Resource Guard

If a validation targets a resource marked as read-only (via the read_only_resources parameter), the pipeline skips the validation and returns a passing result with a descriptive message. A warning is logged when this occurs.

Integration with Plan Lifecycle

The run_for_plan() method extends run() by persisting the validation summary into the plan's metadata dict:

pipeline = ValidationPipeline(commands, executor)
summary = pipeline.run_for_plan(plan_metadata=plan.metadata)
# plan.metadata["validation_summary"] now contains the full summary

This allows downstream phases (Apply, CLI display) to inspect validation results without re-running the pipeline.

Models Reference

ValidationCommand

Field	Type	Default	Description
validation_name	str	—	Validation identifier
resource_id	str	—	Target resource ULID
resource_name	str	—	Human-readable name
mode	ValidationMode	—	required/informational
arguments	dict	{}	Executor arguments
timeout_seconds	float	30.0	Timeout in seconds

ValidationResult

Field	Type	Default	Description
validation_name	str	—	Validation identifier
resource_id	str	—	Resource ULID
resource_name	str	—	Human-readable name
mode	ValidationMode	—	required/informational
passed	bool	—	Pass/fail status
message	str	—	Result description
data	dict | None	None	Structured data
duration_ms	float	—	Execution time (ms)
error	str | None	None	Error message
timed_out	bool	False	Timeout flag

ValidationSummary

Field	Type	Description
total	int	Total validations
required_passed	int	Required passes
required_failed	int	Required failures
informational_passed	int	Informational passes
informational_failed	int	Informational failures
results	list[ValidationResult]	All results
all_required_passed	bool (property)	True if no req. failures

---

Validation Pipeline — Apply Gating

Overview

The validation apply gate enforces that attached validations pass before the apply phase proceeds. Validations are attached to resources and run automatically during the Execute-to-Apply transition. Required validation failures block apply; informational failures are recorded but do not block.

Attachment Model

ValidationAttachment

Represents a validation bound to a resource with scope and mode.

Field	Type	Description
attachment_id	str	Unique ULID for this attachment
validation_name	str	Name of the validation tool
resource_id	str	Target resource ID
resource_name	str	Human-readable resource name
mode	ValidationMode	required or informational
scope	AttachmentScope	direct, project, or plan
scope_target	str	Project name or plan ID for scoping
arguments	dict[str, Any]	Arguments passed to the validation

AttachmentScope

Value	Description
direct	Always active for the resource
project	Active when resource is accessed via a project
plan	Active only for a specific plan

When multiple scopes apply, the union of all applicable validations is collected and run.

Validation Modes

Mode	Behavior on Failure
required	Blocks apply, plan stays in Execute phase
informational	Recorded in summary but does not block apply

Result Models

ApplyValidationResult

Per-validation execution result.

Field	Type	Description
attachment_id	str	Which attachment was evaluated
validation_name	str	Validation tool name
resource_id	str	Target resource
mode	ValidationMode	Required or informational
passed	bool	Whether the validation passed
message	str	Result message
data	dict	Additional structured data
duration_ms	int	Execution time in milliseconds
error	`str	None`
timed_out	bool	Whether execution timed out

ApplyValidationSummary

Aggregated results with gating decision.

Properties: total, required_passed, required_failed, informational_passed, informational_failed, all_required_passed, is_empty

Methods:

• to_plan_metadata() — Dict for plan validation_summary field
• format_cli_output() — Human-readable summary for CLI display

Runner Interface

ValidationRunner (ABC)

Abstract interface for executing a single validation:

def run_validation(
    self,
    attachment: ValidationAttachment,
    context: dict[str, Any],
) -> ApplyValidationResult: ...

DefaultValidationRunner

Default stub using text matching. Checks if the validation name or argument values appear in the context. Production implementations should invoke the actual validation tool via the tool execution pipeline.

Apply Validation Gate

ApplyValidationGate

Orchestrates the full validation-before-apply flow:

gate = ApplyValidationGate(runner=DefaultValidationRunner())
summary = gate.run(plan_id, attachments, context)

if gate.should_block_apply(summary):
    reasons = gate.get_failure_reasons(summary)
    # Block apply, report reasons
else:
    # Proceed to apply

Example Output

When apply is blocked:

Validation Gate: BLOCKED
  Required: 1 passed, 1 failed
  Informational: 0 passed, 0 failed
  Failures:
    - lint-check on res-001: 3 lint errors found

When apply is allowed:

Validation Gate: PASSED
  Required: 2 passed, 0 failed
  Informational: 1 passed, 0 failed

Integration with Plan Metadata

The validation summary is stored in the plan's validation_summary field via ApplyValidationSummary.to_plan_metadata():

{
    "total": 3,
    "required_passed": 2,
    "required_failed": 1,
    "informational_passed": 0,
    "informational_failed": 0,
    "all_required_passed": false,
    "evaluated_at": "2026-02-22T10:30:00+00:00",
    "results": [...]
}