Feature: Validation pipeline As a plan executor I want a validation pipeline that runs validations deterministically So that required failures block and informational failures are reported Background: Given a validation pipeline test environment # ── Empty pipeline ───────────────────────────────────────────── Scenario: Empty validation list returns passing summary Given an empty list of validation commands When the validation pipeline runs Then the validation summary total is 0 And all required validations passed # ── Single required validation ───────────────────────────────── Scenario: Single required validation passes Given a required vp command "check-syntax" on resource "repo-main" And the vp executor returns passed for "check-syntax" When the validation pipeline runs Then the validation summary total is 1 And the validation summary required_passed is 1 And all required validations passed Scenario: Single required validation fails and blocks Given a required vp command "check-syntax" on resource "repo-main" And the vp executor returns failed for "check-syntax" with message "Syntax error on line 42" When the validation pipeline runs Then the validation summary total is 1 And the validation summary required_failed is 1 And not all required validations passed # ── Single informational validation ──────────────────────────── Scenario: Single informational validation passes Given an informational vp command "lint-check" on resource "repo-main" And the vp executor returns passed for "lint-check" When the validation pipeline runs Then the validation summary total is 1 And the validation summary informational_passed is 1 And all required validations passed Scenario: Single informational validation fails but does not block Given an informational vp command "lint-check" on resource "repo-main" And the vp executor returns failed for "lint-check" with message "Linting warnings found" When the validation pipeline runs Then the validation summary total is 1 And the validation summary informational_failed is 1 And all required validations passed # ── Multiple validations sorted by resource/mode/name ────────── Scenario: Multiple validations are sorted by resource name then mode then name Given a required vp command "z-check" on resource "beta-repo" And a required vp command "a-check" on resource "alpha-repo" And an informational vp command "b-lint" on resource "alpha-repo" And the vp executor returns passed for all validations When the validation pipeline runs Then the validation results are sorted by resource then mode then name And the validation summary total is 3 # ── Timeout handling ─────────────────────────────────────────── Scenario: Validation that exceeds timeout is marked as timed out Given a required vp command "slow-check" on resource "repo-main" with timeout 0.2 And the vp executor will timeout for "slow-check" When the validation pipeline runs Then the vp result for "slow-check" has timed_out true And the vp result for "slow-check" has passed false And the vp result for "slow-check" has error containing "TimeoutError" # ── Error normalisation ──────────────────────────────────────── Scenario: Executor returning non-dict output is normalised to failure Given a required vp command "broken-check" on resource "repo-main" And the vp executor returns non-dict output for "broken-check" When the validation pipeline runs Then the vp result for "broken-check" has passed false And the vp result for "broken-check" has message containing "non-dict" Scenario: Executor missing passed key defaults to false Given a required vp command "partial-check" on resource "repo-main" And the vp executor returns dict without passed key for "partial-check" When the validation pipeline runs Then the vp result for "partial-check" has passed false Scenario: Executor raising exception is captured as error Given a required vp command "error-check" on resource "repo-main" And the vp executor raises an exception for "error-check" When the validation pipeline runs Then the vp result for "error-check" has passed false And the vp result for "error-check" has error containing "RuntimeError" # ── Required + informational mixed results ───────────────────── Scenario: Mixed required and informational with all required passing Given a required vp command "req-check" on resource "repo-main" And an informational vp command "info-lint" on resource "repo-main" And the vp executor returns passed for "req-check" And the vp executor returns failed for "info-lint" with message "Info warning" When the validation pipeline runs Then the validation summary required_passed is 1 And the validation summary informational_failed is 1 And all required validations passed Scenario: Mixed validations where required fails blocks Given a required vp command "req-check" on resource "repo-main" And an informational vp command "info-lint" on resource "repo-main" And the vp executor returns failed for "req-check" with message "Required failed" And the vp executor returns passed for "info-lint" When the validation pipeline runs Then the validation summary required_failed is 1 And the validation summary informational_passed is 1 And not all required validations passed # ── Summary counts verification ──────────────────────────────── Scenario: Summary counts are accurate for multiple validations Given a required vp command "req-a" on resource "repo-1" And a required vp command "req-b" on resource "repo-1" And an informational vp command "info-a" on resource "repo-1" And an informational vp command "info-b" on resource "repo-1" And the vp executor returns passed for "req-a" And the vp executor returns failed for "req-b" with message "Failed" And the vp executor returns passed for "info-a" And the vp executor returns failed for "info-b" with message "Info fail" When the validation pipeline runs Then the validation summary total is 4 And the validation summary required_passed is 1 And the validation summary required_failed is 1 And the validation summary informational_passed is 1 And the validation summary informational_failed is 1 # ── Grouping by resource ─────────────────────────────────────── Scenario: Validation results can be grouped by resource Given a required vp command "check-a" on resource "repo-alpha" And a required vp command "check-b" on resource "repo-beta" And the vp executor returns passed for all validations When the validation pipeline runs Then the results grouped by resource have 2 groups And the vp resource group "repo-alpha" has 1 result And the vp resource group "repo-beta" has 1 result # ── Read-only resource guard ─────────────────────────────────── Scenario: Skip validation on read-only resource Given a required vp command "write-check" on resource "readonly-repo" having resource_id "RO001" And the vp resource "RO001" is marked as read-only And the vp executor returns passed for all validations When the validation pipeline runs Then the vp result for "write-check" has passed true And the vp result for "write-check" has message containing "read-only" # ── ValidationSummary all_required_passed property ───────────── Scenario: ValidationSummary all_required_passed is true when no required failures Given a validation summary with 0 required failures Then the summary all_required_passed property is true Scenario: ValidationSummary all_required_passed is false when required failures exist Given a validation summary with 2 required failures Then the summary all_required_passed property is false # ── Pipeline with max_workers=1 ──────────────────────────────── Scenario: Pipeline with max_workers=1 executes sequentially Given a required vp command "seq-a" on resource "repo-1" And a required vp command "seq-b" on resource "repo-1" And the vp executor returns passed for all validations And the pipeline max_workers is set to 1 When the validation pipeline runs Then the validation summary total is 2 And all required validations passed # ── Validation with data payload ─────────────────────────────── Scenario: Validation result includes data payload Given a required vp command "data-check" on resource "repo-main" And the vp executor returns passed with data for "data-check" When the validation pipeline runs Then the vp result for "data-check" has data key "detail" # ── run_for_plan integration ─────────────────────────────────── Scenario: run_for_plan stores summary in plan metadata Given a required vp command "plan-check" on resource "repo-main" And the vp executor returns passed for "plan-check" When the validation pipeline runs for plan with metadata Then the plan metadata contains validation_summary And the plan metadata validation_summary total is 1 # ── Normalisation edge cases ─────────────────────────────────── Scenario: Executor returns non-bool passed value is normalised to false Given a required vp command "nonbool-check" on resource "repo-main" And the vp executor returns non-bool passed for "nonbool-check" When the validation pipeline runs Then the vp result for "nonbool-check" has passed false Scenario: Executor returns non-string message is coerced to string Given a required vp command "nonstr-msg" on resource "repo-main" And the vp executor returns non-string message for "nonstr-msg" When the validation pipeline runs Then the vp result for "nonstr-msg" has passed true Scenario: Executor returns non-dict data is wrapped in raw key Given a required vp command "nondict-data" on resource "repo-main" And the vp executor returns non-dict data for "nondict-data" When the validation pipeline runs Then the vp result for "nondict-data" has data key "raw" # ── Stdout/stderr capture ────────────────────────────────────── Scenario: Validation that prints to stdout has output captured Given a required vp command "stdout-check" on resource "repo-main" And the vp executor prints to stdout for "stdout-check" When the validation pipeline runs Then the vp result for "stdout-check" has data key "_captured_stdout" Scenario: Validation that prints to stderr has output captured Given a required vp command "stderr-check" on resource "repo-main" And the vp executor prints to stderr for "stderr-check" When the validation pipeline runs Then the vp result for "stderr-check" has data key "_captured_stderr" # ── Duration tracking ──────────────────────────────────────────── Scenario: Validation result has positive duration Given a required vp command "dur-check" on resource "repo-main" And the vp executor returns passed for "dur-check" When the validation pipeline runs Then the vp result for "dur-check" has positive duration_ms