CoreRasurae
236 lines
12 KiB
Gherkin
236 lines
12 KiB
Gherkin
Feature: Validation pipeline
|
|
As a plan executor
|
|
I want a validation pipeline that runs validations deterministically
|
|
So that required failures block and informational failures are reported
|
|
|
|
Background:
|
|
Given a validation pipeline test environment
|
|
|
|
# ── Empty pipeline ─────────────────────────────────────────────
|
|
|
|
Scenario: Empty validation list returns passing summary
|
|
Given an empty list of validation commands
|
|
When the validation pipeline runs
|
|
Then the validation summary total is 0
|
|
And all required validations passed
|
|
|
|
# ── Single required validation ─────────────────────────────────
|
|
|
|
Scenario: Single required validation passes
|
|
Given a required vp command "check-syntax" on resource "repo-main"
|
|
And the vp executor returns passed for "check-syntax"
|
|
When the validation pipeline runs
|
|
Then the validation summary total is 1
|
|
And the validation summary required_passed is 1
|
|
And all required validations passed
|
|
|
|
Scenario: Single required validation fails and blocks
|
|
Given a required vp command "check-syntax" on resource "repo-main"
|
|
And the vp executor returns failed for "check-syntax" with message "Syntax error on line 42"
|
|
When the validation pipeline runs
|
|
Then the validation summary total is 1
|
|
And the validation summary required_failed is 1
|
|
And not all required validations passed
|
|
|
|
# ── Single informational validation ────────────────────────────
|
|
|
|
Scenario: Single informational validation passes
|
|
Given an informational vp command "lint-check" on resource "repo-main"
|
|
And the vp executor returns passed for "lint-check"
|
|
When the validation pipeline runs
|
|
Then the validation summary total is 1
|
|
And the validation summary informational_passed is 1
|
|
And all required validations passed
|
|
|
|
Scenario: Single informational validation fails but does not block
|
|
Given an informational vp command "lint-check" on resource "repo-main"
|
|
And the vp executor returns failed for "lint-check" with message "Linting warnings found"
|
|
When the validation pipeline runs
|
|
Then the validation summary total is 1
|
|
And the validation summary informational_failed is 1
|
|
And all required validations passed
|
|
|
|
# ── Multiple validations sorted by resource/mode/name ──────────
|
|
|
|
Scenario: Multiple validations are sorted by resource name then mode then name
|
|
Given a required vp command "z-check" on resource "beta-repo"
|
|
And a required vp command "a-check" on resource "alpha-repo"
|
|
And an informational vp command "b-lint" on resource "alpha-repo"
|
|
And the vp executor returns passed for all validations
|
|
When the validation pipeline runs
|
|
Then the validation results are sorted by resource then mode then name
|
|
And the validation summary total is 3
|
|
|
|
# ── Timeout handling ───────────────────────────────────────────
|
|
|
|
Scenario: Validation that exceeds timeout is marked as timed out
|
|
Given a required vp command "slow-check" on resource "repo-main" with timeout 0.2
|
|
And the vp executor will timeout for "slow-check"
|
|
When the validation pipeline runs
|
|
Then the vp result for "slow-check" has timed_out true
|
|
And the vp result for "slow-check" has passed false
|
|
And the vp result for "slow-check" has error containing "TimeoutError"
|
|
|
|
# ── Error normalisation ────────────────────────────────────────
|
|
|
|
Scenario: Executor returning non-dict output is normalised to failure
|
|
Given a required vp command "broken-check" on resource "repo-main"
|
|
And the vp executor returns non-dict output for "broken-check"
|
|
When the validation pipeline runs
|
|
Then the vp result for "broken-check" has passed false
|
|
And the vp result for "broken-check" has message containing "non-dict"
|
|
|
|
Scenario: Executor missing passed key defaults to false
|
|
Given a required vp command "partial-check" on resource "repo-main"
|
|
And the vp executor returns dict without passed key for "partial-check"
|
|
When the validation pipeline runs
|
|
Then the vp result for "partial-check" has passed false
|
|
|
|
Scenario: Executor raising exception is captured as error
|
|
Given a required vp command "error-check" on resource "repo-main"
|
|
And the vp executor raises an exception for "error-check"
|
|
When the validation pipeline runs
|
|
Then the vp result for "error-check" has passed false
|
|
And the vp result for "error-check" has error containing "RuntimeError"
|
|
|
|
# ── Required + informational mixed results ─────────────────────
|
|
|
|
Scenario: Mixed required and informational with all required passing
|
|
Given a required vp command "req-check" on resource "repo-main"
|
|
And an informational vp command "info-lint" on resource "repo-main"
|
|
And the vp executor returns passed for "req-check"
|
|
And the vp executor returns failed for "info-lint" with message "Info warning"
|
|
When the validation pipeline runs
|
|
Then the validation summary required_passed is 1
|
|
And the validation summary informational_failed is 1
|
|
And all required validations passed
|
|
|
|
Scenario: Mixed validations where required fails blocks
|
|
Given a required vp command "req-check" on resource "repo-main"
|
|
And an informational vp command "info-lint" on resource "repo-main"
|
|
And the vp executor returns failed for "req-check" with message "Required failed"
|
|
And the vp executor returns passed for "info-lint"
|
|
When the validation pipeline runs
|
|
Then the validation summary required_failed is 1
|
|
And the validation summary informational_passed is 1
|
|
And not all required validations passed
|
|
|
|
# ── Summary counts verification ────────────────────────────────
|
|
|
|
Scenario: Summary counts are accurate for multiple validations
|
|
Given a required vp command "req-a" on resource "repo-1"
|
|
And a required vp command "req-b" on resource "repo-1"
|
|
And an informational vp command "info-a" on resource "repo-1"
|
|
And an informational vp command "info-b" on resource "repo-1"
|
|
And the vp executor returns passed for "req-a"
|
|
And the vp executor returns failed for "req-b" with message "Failed"
|
|
And the vp executor returns passed for "info-a"
|
|
And the vp executor returns failed for "info-b" with message "Info fail"
|
|
When the validation pipeline runs
|
|
Then the validation summary total is 4
|
|
And the validation summary required_passed is 1
|
|
And the validation summary required_failed is 1
|
|
And the validation summary informational_passed is 1
|
|
And the validation summary informational_failed is 1
|
|
|
|
# ── Grouping by resource ───────────────────────────────────────
|
|
|
|
Scenario: Validation results can be grouped by resource
|
|
Given a required vp command "check-a" on resource "repo-alpha"
|
|
And a required vp command "check-b" on resource "repo-beta"
|
|
And the vp executor returns passed for all validations
|
|
When the validation pipeline runs
|
|
Then the results grouped by resource have 2 groups
|
|
And the vp resource group "repo-alpha" has 1 result
|
|
And the vp resource group "repo-beta" has 1 result
|
|
|
|
# ── Read-only resource guard ───────────────────────────────────
|
|
|
|
Scenario: Skip validation on read-only resource
|
|
Given a required vp command "write-check" on resource "readonly-repo" having resource_id "RO001"
|
|
And the vp resource "RO001" is marked as read-only
|
|
And the vp executor returns passed for all validations
|
|
When the validation pipeline runs
|
|
Then the vp result for "write-check" has passed true
|
|
And the vp result for "write-check" has message containing "read-only"
|
|
|
|
# ── ValidationSummary all_required_passed property ─────────────
|
|
|
|
Scenario: ValidationSummary all_required_passed is true when no required failures
|
|
Given a validation summary with 0 required failures
|
|
Then the summary all_required_passed property is true
|
|
|
|
Scenario: ValidationSummary all_required_passed is false when required failures exist
|
|
Given a validation summary with 2 required failures
|
|
Then the summary all_required_passed property is false
|
|
|
|
# ── Pipeline with max_workers=1 ────────────────────────────────
|
|
|
|
Scenario: Pipeline with max_workers=1 executes sequentially
|
|
Given a required vp command "seq-a" on resource "repo-1"
|
|
And a required vp command "seq-b" on resource "repo-1"
|
|
And the vp executor returns passed for all validations
|
|
And the pipeline max_workers is set to 1
|
|
When the validation pipeline runs
|
|
Then the validation summary total is 2
|
|
And all required validations passed
|
|
|
|
# ── Validation with data payload ───────────────────────────────
|
|
|
|
Scenario: Validation result includes data payload
|
|
Given a required vp command "data-check" on resource "repo-main"
|
|
And the vp executor returns passed with data for "data-check"
|
|
When the validation pipeline runs
|
|
Then the vp result for "data-check" has data key "detail"
|
|
|
|
# ── run_for_plan integration ───────────────────────────────────
|
|
|
|
Scenario: run_for_plan stores summary in plan metadata
|
|
Given a required vp command "plan-check" on resource "repo-main"
|
|
And the vp executor returns passed for "plan-check"
|
|
When the validation pipeline runs for plan with metadata
|
|
Then the plan metadata contains validation_summary
|
|
And the plan metadata validation_summary total is 1
|
|
|
|
# ── Normalisation edge cases ───────────────────────────────────
|
|
|
|
Scenario: Executor returns non-bool passed value is normalised to false
|
|
Given a required vp command "nonbool-check" on resource "repo-main"
|
|
And the vp executor returns non-bool passed for "nonbool-check"
|
|
When the validation pipeline runs
|
|
Then the vp result for "nonbool-check" has passed false
|
|
|
|
Scenario: Executor returns non-string message is coerced to string
|
|
Given a required vp command "nonstr-msg" on resource "repo-main"
|
|
And the vp executor returns non-string message for "nonstr-msg"
|
|
When the validation pipeline runs
|
|
Then the vp result for "nonstr-msg" has passed true
|
|
|
|
Scenario: Executor returns non-dict data is wrapped in raw key
|
|
Given a required vp command "nondict-data" on resource "repo-main"
|
|
And the vp executor returns non-dict data for "nondict-data"
|
|
When the validation pipeline runs
|
|
Then the vp result for "nondict-data" has data key "raw"
|
|
|
|
# ── Stdout/stderr capture ──────────────────────────────────────
|
|
|
|
Scenario: Validation that prints to stdout has output captured
|
|
Given a required vp command "stdout-check" on resource "repo-main"
|
|
And the vp executor prints to stdout for "stdout-check"
|
|
When the validation pipeline runs
|
|
Then the vp result for "stdout-check" has data key "_captured_stdout"
|
|
|
|
Scenario: Validation that prints to stderr has output captured
|
|
Given a required vp command "stderr-check" on resource "repo-main"
|
|
And the vp executor prints to stderr for "stderr-check"
|
|
When the validation pipeline runs
|
|
Then the vp result for "stderr-check" has data key "_captured_stderr"
|
|
|
|
# ── Duration tracking ────────────────────────────────────────────
|
|
|
|
Scenario: Validation result has positive duration
|
|
Given a required vp command "dur-check" on resource "repo-main"
|
|
And the vp executor returns passed for "dur-check"
|
|
When the validation pipeline runs
|
|
Then the vp result for "dur-check" has positive duration_ms
|