cleveragents-core/features/consolidated_validation.feature

Feature: Consolidated Validation
  Combined scenarios from: validation_apply, validation_edge_cases, validation_pipeline_stream_coverage

  # ============================================================
  # Originally from: validation_apply.feature
  # Feature: Validation apply gate
  # ============================================================

  Scenario: Create a required validation attachment
    Given a validation apply test environment
    Given a validation attachment "lint-check" for resource "res-001" with mode "required"
    Then the attachment mode should be "required"
    And the attachment scope should be "direct"


  Scenario: Create an informational validation attachment
    Given a validation apply test environment
    Given a validation attachment "style-check" for resource "res-002" with mode "informational"
    Then the attachment mode should be "informational"


  Scenario: Create a project-scoped attachment
    Given a validation apply test environment
    Given a project-scoped attachment "lint-check" for resource "res-001" targeting "my-project"
    Then the attachment scope should be "project"
    And the attachment scope target should be "my-project"


  Scenario: Create a plan-scoped attachment
    Given a validation apply test environment
    Given a plan-scoped attachment "lint-check" for resource "res-001" targeting "01PLAN0001"
    Then the attachment scope should be "plan"
    And the attachment scope target should be "01PLAN0001"

    # -- DefaultValidationRunner ---------------------------------------------


  Scenario: DefaultValidationRunner passes when context matches
    Given a validation apply test environment
    Given a validation attachment "lint-check" for resource "res-001" with mode "required"
    And a validation run context with key "lint-check" and value "passed"
    When I execute the validation with DefaultValidationRunner
    Then the validation result should be passed


  Scenario: DefaultValidationRunner fails when context has no match
    Given a validation apply test environment
    Given a validation attachment "lint-check" for resource "res-001" with mode "required"
    And a validation run context with key "status" and value "no matches"
    When I execute the validation with DefaultValidationRunner
    Then the validation result should be failed


  Scenario: DefaultValidationRunner matches via arguments
    Given a validation apply test environment
    Given a validation attachment "coverage-check" for resource "res-001" with mode "required"
    And the attachment has argument "threshold" with value "90"
    And a validation run context with key "coverage" and value "90 percent"
    When I execute the validation with DefaultValidationRunner
    Then the validation result should be passed

    # -- ApplyValidationSummary ----------------------------------------------


  Scenario: Summary all required passed with no failures
    Given a validation apply test environment
    Given an apply validation summary with 2 required passed and 0 required failed
    Then the apply summary should report all required passed
    And the apply summary total should be 2


  Scenario: Summary blocked when required failure exists
    Given a validation apply test environment
    Given an apply validation summary with 1 required passed and 1 required failed
    Then the apply summary should not report all required passed
    And the apply summary required failed count should be 1


  Scenario: Summary with informational results does not block
    Given a validation apply test environment
    Given an apply validation summary with 0 required passed and 0 required failed
    And the summary has 1 informational passed and 1 informational failed
    Then the apply summary should report all required passed
    And the apply summary informational failed count should be 1


  Scenario: Empty summary blocks apply
    Given a validation apply test environment
    Given an empty apply validation summary
    Then the apply summary should be empty
    And the apply summary should not report all required passed


  Scenario: Summary to_plan_metadata has expected fields
    Given a validation apply test environment
    Given an apply validation summary with 1 required passed and 1 required failed
    Then the plan metadata should have key "total"
    And the plan metadata should have key "all_required_passed"
    And the plan metadata should have key "results"


  Scenario: Summary format_cli_output shows BLOCKED on failures
    Given a validation apply test environment
    Given an apply validation summary with 0 required passed and 1 required failed
    Then the validation gate CLI output should contain "BLOCKED"


  Scenario: Summary format_cli_output shows no details when neither message nor error
    Given a validation apply test environment
    Given an apply validation summary with a bare failure result
    Then the validation gate CLI output should contain "no details"


  Scenario: Summary format_cli_output shows error in failure detail
    Given a validation apply test environment
    Given an apply validation summary with an error result
    Then the validation gate CLI output should contain "execution timed out"


  Scenario: Summary format_cli_output shows PASSED when all pass
    Given a validation apply test environment
    Given an apply validation summary with 2 required passed and 0 required failed
    Then the validation gate CLI output should contain "PASSED"


  Scenario: DefaultValidationRunner populates duration_ms
    Given a validation apply test environment
    Given a validation attachment "lint-check" for resource "res-001" with mode "required"
    And a validation run context with key "lint-check" and value "passed"
    When I execute the validation with DefaultValidationRunner
    Then the validation result duration_ms should be >= 0


  Scenario: DefaultValidationRunner handles exception during execution
    Given a validation apply test environment
    Given a validation attachment "err-val" for resource "res-err" with mode "required"
    And a validation run context that raises an error during iteration
    When I execute the validation with DefaultValidationRunner
    Then the validation result should be failed
    And the validation result should have an error message

    # -- ApplyValidationGate -------------------------------------------------


  Scenario: Gate allows apply when all required pass
    Given a validation apply test environment
    Given a validation attachment "lint-check" for resource "res-001" with mode "required"
    And a validation run context with key "lint-check" and value "ok"
    When I run the apply validation gate for plan "01PLAN0001"
    Then the gate should allow apply


  Scenario: Gate blocks apply when required validation fails
    Given a validation apply test environment
    Given a validation attachment "lint-check" for resource "res-001" with mode "required"
    And a validation run context with key "other" and value "irrelevant"
    When I run the apply validation gate for plan "01PLAN0001"
    Then the gate should block apply
    And the gate failure reasons should mention "lint-check"


  Scenario: Gate allows apply despite informational failures
    Given a validation apply test environment
    Given a validation attachment "style-check" for resource "res-001" with mode "informational"
    And a validation run context with key "other" and value "irrelevant"
    When I run the apply validation gate for plan "01PLAN0001"
    Then the gate should allow apply


  Scenario: Gate with multiple attachments mixed results
    Given a validation apply test environment
    Given a validation attachment "lint-check" for resource "res-001" with mode "required"
    And a validation attachment "style-check" for resource "res-001" with mode "informational"
    And a validation run context with key "lint-check" and value "passed"
    When I run the apply validation gate for plan "01PLAN0001"
    Then the gate should allow apply
    And the apply summary total should be 2


  Scenario: Gate failure reasons include error text when result has error
    Given a validation apply test environment
    Given an apply validation summary with an error result
    Then the gate failure reasons should mention "error:"


  Scenario: Gate with no attachments blocks apply
    Given a validation apply test environment
    When I run the apply validation gate for plan "01PLAN0001" with no attachments
    Then the gate should block apply
    And the apply summary should be empty


  # ============================================================
  # Originally from: validation_edge_cases.feature
  # Feature: Validation edge cases
  # ============================================================

  Scenario: Detect tool output missing required status field
    Given a validation edge case test environment
    Given validation edge fixture "malformed_tool_output" is loaded
    When validation edge processes fixture "missing_required_status_field"
    Then validation edge should report error containing "missing required field: status"


  Scenario: Detect tool output with wrong type for status
    Given a validation edge case test environment
    Given validation edge fixture "malformed_tool_output" is loaded
    When validation edge processes fixture "wrong_type_for_status"
    Then validation edge should report error containing "invalid type for field 'status'"


  Scenario: Detect tool output with null result payload
    Given a validation edge case test environment
    Given validation edge fixture "malformed_tool_output" is loaded
    When validation edge processes fixture "null_result_payload"
    Then validation edge should report error containing "result payload must not be null"


  Scenario: Detect tool output with extra null fields
    Given a validation edge case test environment
    Given validation edge fixture "malformed_tool_output" is loaded
    When validation edge processes fixture "extra_null_fields"
    Then validation edge should report error containing "unexpected null fields"


  Scenario: Detect empty tool output object
    Given a validation edge case test environment
    Given validation edge fixture "malformed_tool_output" is loaded
    When validation edge processes fixture "empty_output_object"
    Then validation edge should report error containing "must contain at least"


  Scenario: Detect string tool output instead of object
    Given a validation edge case test environment
    Given validation edge fixture "malformed_tool_output" is loaded
    When validation edge processes fixture "output_is_string_not_object"
    Then validation edge should report error containing "must be an object"

    # ── Missing resource references ────────────────────────────────


  Scenario: Detect dangling action reference in plan
    Given a validation edge case test environment
    Given validation edge fixture "missing_resources" is loaded
    When validation edge checks resource fixture "dangling_action_reference"
    Then validation edge should report error containing "action not found"


  Scenario: Detect unresolvable project path
    Given a validation edge case test environment
    Given validation edge fixture "missing_resources" is loaded
    When validation edge checks resource fixture "unresolvable_project_path"
    Then validation edge should report error containing "unresolvable project path"


  Scenario: Detect missing actor for execution phase
    Given a validation edge case test environment
    Given validation edge fixture "missing_resources" is loaded
    When validation edge checks resource fixture "missing_actor_for_phase"
    Then validation edge should report error containing "missing actor for phase"


  Scenario: Detect dangling parent plan reference
    Given a validation edge case test environment
    Given validation edge fixture "missing_resources" is loaded
    When validation edge checks resource fixture "dangling_parent_plan_id"
    Then validation edge should report error containing "parent plan not found"


  Scenario: Detect circular resource dependency
    Given a validation edge case test environment
    Given validation edge fixture "missing_resources" is loaded
    When validation edge checks resource fixture "circular_resource_dependency"
    Then validation edge should report error containing "circular dependency detected"

    # ── Validation timeout behavior ────────────────────────────────


  Scenario: Detect slow schema validator exceeding timeout
    Given a validation edge case test environment
    Given validation edge fixture "validation_timeouts" is loaded
    When validation edge simulates timeout for fixture "slow_schema_validator"
    Then validation edge should report error containing "validation timeout"


  Scenario: Detect zero timeout causes immediate failure
    Given a validation edge case test environment
    Given validation edge fixture "validation_timeouts" is loaded
    When validation edge simulates timeout for fixture "zero_timeout_immediate_fail"
    Then validation edge should report error containing "exceeded 0ms deadline"


  Scenario: Detect negative timeout is rejected
    Given a validation edge case test environment
    Given validation edge fixture "validation_timeouts" is loaded
    When validation edge simulates timeout for fixture "negative_timeout_rejected"
    Then validation edge should report error containing "invalid timeout"


  Scenario: Detect timeout with partial results reported
    Given a validation edge case test environment
    Given validation edge fixture "validation_timeouts" is loaded
    When validation edge simulates timeout for fixture "timeout_with_partial_results"
    Then validation edge should report error containing "completed 5/20 items"

    # ── Invalid schema transform results ───────────────────────────


  Scenario: Detect transform returning non-dict schema
    Given a validation edge case test environment
    Given validation edge fixture "invalid_schema_transforms" is loaded
    When validation edge applies transform fixture "transform_returns_non_dict"
    Then validation edge should report error containing "expected dict, got str"


  Scenario: Detect transform returning schema without type field
    Given a validation edge case test environment
    Given validation edge fixture "invalid_schema_transforms" is loaded
    When validation edge applies transform fixture "transform_returns_missing_type"
    Then validation edge should report error containing "without 'type' field"


  Scenario: Detect transform returning null schema
    Given a validation edge case test environment
    Given validation edge fixture "invalid_schema_transforms" is loaded
    When validation edge applies transform fixture "transform_returns_null_schema"
    Then validation edge should report error containing "returned null schema"

    # ── Mixed ordering and duplicate attachment IDs ────────────────


  Scenario: Required validations ordered before informational
    Given a validation edge case test environment
    Given validation edge fixture "mixed_ordering" is loaded
    When validation edge sorts ordering fixture "required_before_informational"
    Then validation edge ordering should match expected order


  Scenario: Duplicate attachment IDs across required validations rejected
    Given a validation edge case test environment
    Given validation edge fixture "mixed_ordering" is loaded
    When validation edge sorts ordering fixture "duplicate_attachment_ids_rejected"
    Then validation edge should report error containing "duplicate attachment ID"


  Scenario: Duplicate attachment IDs across levels rejected
    Given a validation edge case test environment
    Given validation edge fixture "mixed_ordering" is loaded
    When validation edge sorts ordering fixture "duplicate_attachment_ids_across_levels"
    Then validation edge should report error containing "duplicate attachment ID"


  Scenario: Unknown validation level is rejected
    Given a validation edge case test environment
    Given validation edge fixture "mixed_ordering" is loaded
    When validation edge sorts ordering fixture "unknown_validation_level"
    Then validation edge should report error containing "unknown validation level"

    # ── Concurrent validation conflicts ────────────────────────────


  Scenario: Concurrent validators on same resource detect conflict
    Given a validation edge case test environment
    Given validation edge has two concurrent validators for resource "shared-config"
    When validation edge runs both validators concurrently
    Then validation edge should detect a concurrent modification conflict

    # ── Rollback on validation failure ─────────────────────────────


  Scenario: Validation failure triggers state rollback
    Given a validation edge case test environment
    Given validation edge has a staged change set with 3 pending changes
    When validation edge encounters a required validation failure
    Then validation edge should rollback all 3 pending changes
    And validation edge rollback result should contain "rolled back"


  # ============================================================
  # Originally from: validation_pipeline_stream_coverage.feature
  # Feature: Validation pipeline _ThreadLocalStream coverage
  # ============================================================

  Scenario: encoding property returns the original stream encoding
    Given a thread-local stream wrapping a mock original stream
    Given the original stream has encoding "utf-8"
    When I read the thread-local stream encoding
    Then the encoding should be "utf-8"


  Scenario: encoding property falls back to utf-8 when original has no encoding
    Given a thread-local stream wrapping a mock original stream
    Given the original stream has no encoding attribute
    When I read the thread-local stream encoding
    Then the encoding should be "utf-8"

    # --- writable method -----------------------------------------------------


  Scenario: writable returns True
    Given a thread-local stream wrapping a mock original stream
    When I check if the thread-local stream is writable
    Then writable should return True

    # --- readable method -----------------------------------------------------


  Scenario: readable returns False
    Given a thread-local stream wrapping a mock original stream
    When I check if the thread-local stream is readable
    Then readable should return False

    # --- flush method with active buffer -------------------------------------


  Scenario: flush flushes the capture buffer when capturing is active
    Given a thread-local stream wrapping a mock original stream
    Given capture is started on the thread-local stream
    When I write "hello" to the thread-local stream
    And I flush the thread-local stream
    Then the flush should succeed without error

    # --- flush method without active buffer ----------------------------------


  Scenario: flush flushes the original stream when no capture is active
    Given a thread-local stream wrapping a mock original stream
    When I flush the thread-local stream
    Then the original stream flush should be called

    # --- isatty method -------------------------------------------------------


  Scenario: isatty delegates to the original stream
    Given a thread-local stream wrapping a mock original stream
    Given the original stream isatty returns True
    When I check if the thread-local stream isatty
    Then isatty should return True


  Scenario: isatty returns False when original returns False
    Given a thread-local stream wrapping a mock original stream
    Given the original stream isatty returns False
    When I check if the thread-local stream isatty
    Then isatty should return False

    # --- write with and without capture ------------------------------------


  Scenario: write without capture goes to original stream
    Given a thread-local stream wrapping a mock original stream
    When I write "direct output" to the thread-local stream
    Then the original stream should have received "direct output"


  Scenario: write with capture goes to capture buffer
    Given a thread-local stream wrapping a mock original stream
    Given capture is started on the thread-local stream
    When I write "captured output" to the thread-local stream
    And capture is stopped on the thread-local stream
    Then the captured text should be "captured output"

    # --- stop_capture when no capture was started ----------------------------


  Scenario: stop_capture returns empty string when no capture was started
    Given a thread-local stream wrapping a mock original stream
    When capture is stopped on the thread-local stream
    Then the captured text should be empty