feat: implement structural component output validation
Replace exact character matching with structural component checking for output validation. Implements three validators covering plan tree output, decision CLI dicts, and structured session snapshots. - validate_plan_tree: validates node dicts for required keys (decision_id, type, sequence, question, children), ULID format, correct types, and sibling ordering - validate_decision_dict: validates decision CLI output against Decision.as_cli_dict() schema with field presence, type, ULID, confidence range [0..1], bool fields - validate_structured_output: validates StructuredOutput envelope for command, session_id (ULID), status membership, exit_code, elements integrity - validate_structured_component_output: unified dispatcher by target_type BDD tests in features/structural_validation.feature. ISSUES CLOSED: #11147
This commit is contained in:
Forgejo
@@ -0,0 +1,170 @@
|
||||
Feature: Structural component output validation
|
||||
|
||||
Output validation checks structural components (not exact characters).
|
||||
Required fields are validated for presence and correct type.
|
||||
Structural relationships are validated (parent-child, ordering).
|
||||
Minor formatting differences do not cause validation failures.
|
||||
Validation failures produce actionable error messages.
|
||||
|
||||
Based on Epic #8137 - structural output validation overhaul.
|
||||
|
||||
Scenario Outline: validate_plan_tree accepts valid nodes
|
||||
Given the plan tree contains {node_count:d} node(s)
|
||||
When I validate the plan tree
|
||||
Then it should be structurally valid
|
||||
And it should report no errors
|
||||
Examples:
|
||||
| node_count |
|
||||
| 1 |
|
||||
| 5 |
|
||||
| 20 |
|
||||
|
||||
Scenario: validate_plan_tree rejects nodes missing required keys
|
||||
Given the plan tree contains a node missing "decision_id"
|
||||
When I validate the plan tree
|
||||
Then it should be structurally invalid
|
||||
And it should report errors including "missing required key(s)"
|
||||
|
||||
Scenario: validate_plan_tree rejects invalid ULID in decision_id
|
||||
Given the plan tree contains a node with invalid ULID "not-a-ulid-12345"
|
||||
When I validate the plan tree
|
||||
Then it should be structurally invalid
|
||||
And it should report errors including "must be a valid ULID"
|
||||
|
||||
Scenario: validate_plan_tree accepts empty children list
|
||||
Given the plan tree contains a node with empty "children" list
|
||||
When I validate the plan tree
|
||||
Then it should be structurally valid
|
||||
|
||||
Scenario: validate_plan_tree rejects duplicate sequences for siblings
|
||||
Given two sibling nodes share the same sequence number
|
||||
When I validate the plan tree
|
||||
Then it should be structurally invalid
|
||||
And it should report errors including "duplicate sequence"
|
||||
|
||||
Scenario Outline: validate_plan_tree accepts non-negative sequences
|
||||
Given a plan tree node with sequence {seq}
|
||||
When I validate the plan tree
|
||||
Then it should be structurally valid
|
||||
Examples:
|
||||
| seq |
|
||||
| 0 |
|
||||
| 1 |
|
||||
| 100 |
|
||||
|
||||
Scenario: validate_plan_tree rejects negative sequence
|
||||
Given a plan tree node with sequence "-1"
|
||||
When I validate the plan tree
|
||||
Then it should be structurally invalid
|
||||
And error message must include "must be a non-negative integer"
|
||||
|
||||
Scenario Outline: validate_decision_dict accepts valid CLI dict for field {field}
|
||||
Given I have a decision dict with "{field}" set to {value}
|
||||
When I validate the decision dict
|
||||
Then it should be structurally valid
|
||||
Examples:
|
||||
| field | value |
|
||||
| decision_id | "01ARZ3NDEKTSV4XXFFJFRC889A"|
|
||||
| plan_id | "01ARZ3NDEKTSV4XXFFJFRC889B"|
|
||||
| type | "strategy_choice" |
|
||||
| sequence | 5 |
|
||||
| question | "Which approach?" |
|
||||
| chosen | "A" |
|
||||
| confidence | 0.75 |
|
||||
| parent | "(root)" |
|
||||
| is_correction | false |
|
||||
| superseded | false |
|
||||
|
||||
Scenario: validate_decision_dict rejects confidence outside [0,1]
|
||||
Given I have a decision dict with "confidence" set to 1.5
|
||||
When I validate the decision dict
|
||||
Then it should be structurally invalid
|
||||
And error message must include "must be in range [0.0, 1.0]"
|
||||
|
||||
Scenario: validate_decision_dict accepts None confidence
|
||||
Given I have a decision dict with "confidence" set to null
|
||||
When I validate the decision dict
|
||||
Then it should be structurally valid
|
||||
|
||||
Scenario: validate_decision_dict rejects non-string question
|
||||
Given I have a decision dict with "question" set to 42
|
||||
When I validate the decision dict
|
||||
Then it should be structurally invalid
|
||||
And error message must include "must be a non-empty string"
|
||||
|
||||
Scenario: validate_decision_dict rejects parent that is neither ULID nor "(root)"
|
||||
Given I have a decision dict with "parent" set to "random-string"
|
||||
When I validate the decision dict
|
||||
Then it should be structurally invalid
|
||||
And error message must include "must be a ULID or '(root)'"
|
||||
|
||||
Scenario: validate_decision_dict rejects is_correction that is not boolean
|
||||
Given I have a decision dict with "is_correction" set to "yes"
|
||||
When I validate the decision dict
|
||||
Then it should be structurally invalid
|
||||
And error message must include "must be a boolean"
|
||||
|
||||
Scenario Outline: validate_structured_output accepts valid envelope for field {field}
|
||||
Given I have a structured output with "{field}" set to {value} (and all other required fields present)
|
||||
When I validate the structured output
|
||||
Then it should be structurally valid
|
||||
Examples:
|
||||
| field | value |
|
||||
| command | "agents plan list" |
|
||||
| session_id| "01ARZ3NDEKTSV4XXFFJFRC889A"|
|
||||
| status | "ok" |
|
||||
| exit_code | 0 |
|
||||
|
||||
Scenario: validate_structured_output rejects invalid status value
|
||||
Given I have a structured output with "status" set to "running" (and all other fields valid)
|
||||
When I validate the structured output
|
||||
Then it should be structurally invalid
|
||||
And error must include "must be one of"
|
||||
|
||||
Scenario: validate_structured_output rejects negative exit_code
|
||||
Given I have a structured output with "exit_code" set to "-1" (and all other fields valid)
|
||||
When I validate the structured output
|
||||
Then it should be structurally invalid
|
||||
And error must include "must be >= 0"
|
||||
|
||||
Scenario: validate_structured_output accepts zero elements list
|
||||
Given I have a structured output with empty "elements" list (and all other fields valid)
|
||||
When I validate the structured output
|
||||
Then it should be structurally valid
|
||||
|
||||
Scenario: validate_structured_output rejects invalid element format
|
||||
Given I have a structured output with an element missing "kind" field
|
||||
When I validate the structured output
|
||||
Then it should be structurally invalid
|
||||
And error must include "missing 'kind' field"
|
||||
|
||||
Scenario Outline: validate_structured_component_output routes to correct validator for type {target}
|
||||
Given I target validation for "{target}" with valid data
|
||||
When I call validate_structured_component_output
|
||||
Then it should dispatch to the matching validator
|
||||
And return a valid result
|
||||
Examples:
|
||||
| target |
|
||||
| plan_tree |
|
||||
| decision |
|
||||
| structured_output|
|
||||
|
||||
Scenario: validate_structured_component_output rejects unknown target type
|
||||
Given I target validation for "unknown_type" with valid data
|
||||
When I call validate_structured_component_output
|
||||
Then it should raise a ValidationError with message containing "Unknown target_type"
|
||||
|
||||
Scenario: Structural output is robust to minor formatting differences
|
||||
Given I have a decision dict where whitespace may vary between fields
|
||||
When I validate the decision dict
|
||||
Then validation should pass (structure matched, not exact characters)
|
||||
|
||||
Scenario: Plan tree nodes preserve parent-child structural relationships
|
||||
Given a plan tree with parent and child node references connected by ULID
|
||||
When I validate the plan tree structure
|
||||
Then parent-child relationships should be validly connected
|
||||
|
||||
Scenario: Validation error messages are actionable
|
||||
Given I have invalid plan tree data
|
||||
When I validate the plan tree
|
||||
Then the first error message should identify "node[0]" and the specific field issue
|
||||
Reference in New Issue
Block a user