HAL9000
ac81ea2ceb
The Behevare-parallel runner in this project does not support inline
parameter substitution for '{param}' or '<param>' markers within
Scenario Outline Examples. All 4 original Scenario Outlines were failing
because parameters were not being substituted, causing UndefinedStep and
ValueError failures.
Fix: Convert all Scenario Outline scenarios to regular Scenarios with
explicit literal step definitions. Each unique Gherkin line gets its own
@Given/@When/@Then step definition matching the exact string.
Also fix pre-existing bugs identified in PR review #8719:
- Fix undefined step by quoting {{seq}} in feature (line 46)
- Fix ctx.decision_result → ctx.validation_result context variable (line 184)
- Fix ctx.struct_result → ctx.validation_result context variable (line 210)
- Replace # type: ignore[arg-type] with proper Callable[[Any], dict] type
- Add missing literal step definitions for structured_output tests
ISSUES CLOSED: #11161
232 lines
11 KiB
Gherkin
232 lines
11 KiB
Gherkin
Feature: Structural component output validation
|
|
|
|
Output validation checks structural components (not exact characters).
|
|
Required fields are validated for presence and correct type.
|
|
Structural relationships are validated (parent-child, ordering).
|
|
Minor formatting differences do not cause validation failures.
|
|
Validation failures produce actionable error messages.
|
|
|
|
Based on Epic #8137 - structural output validation overhaul.
|
|
|
|
Scenario: validate_plan_tree accepts valid node count of 1
|
|
Given the plan tree contains 1 node(s)
|
|
When I validate the plan tree
|
|
Then it should be structurally valid
|
|
And it should report no errors
|
|
|
|
Scenario: validate_plan_tree accepts valid node count of 5
|
|
Given the plan tree contains 5 node(s)
|
|
When I validate the plan tree
|
|
Then it should be structurally valid
|
|
And it should report no errors
|
|
|
|
Scenario: validate_plan_tree accepts valid node count of 20
|
|
Given the plan tree contains 20 node(s)
|
|
When I validate the plan tree
|
|
Then it should be structurally valid
|
|
And it should report no errors
|
|
|
|
Scenario: validate_plan_tree rejects nodes missing required keys
|
|
Given the plan tree contains a node missing "decision_id"
|
|
When I validate the plan tree
|
|
Then it should be structurally invalid
|
|
And it should report errors including "missing required key(s)"
|
|
|
|
Scenario: validate_plan_tree rejects invalid ULID in decision_id
|
|
Given the plan tree contains a node with invalid ULID "not-a-ulid-12345"
|
|
When I validate the plan tree
|
|
Then it should be structurally invalid
|
|
And it should report errors including "must be a valid ULID"
|
|
|
|
Scenario: validate_plan_tree accepts empty children list
|
|
Given the plan tree contains a node with empty "children" list
|
|
When I validate the plan tree
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_plan_tree rejects duplicate sequences for siblings
|
|
Given two sibling nodes share the same sequence number
|
|
When I validate the plan tree
|
|
Then it should be structurally invalid
|
|
And it should report errors including "duplicate sequence"
|
|
|
|
Scenario: validate_plan_tree accepts non-negative sequence 0
|
|
Given a plan tree node with sequence "0"
|
|
When I validate the plan tree
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_plan_tree accepts non-negative sequence 1
|
|
Given a plan tree node with sequence "1"
|
|
When I validate the plan tree
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_plan_tree accepts non-negative sequence 100
|
|
Given a plan tree node with sequence "100"
|
|
When I validate the plan tree
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_plan_tree rejects negative sequence
|
|
Given a plan tree node with sequence "-1"
|
|
When I validate the plan tree
|
|
Then it should be structurally invalid
|
|
And error message must include "must be a non-negative integer"
|
|
|
|
Scenario: validate_decision_dict accepts valid CLI dict for decision_id field
|
|
Given I have a decision dict with "decision_id" set to 01ARZ3NDEKTSV4XXFFJFRC889A
|
|
When I validate the decision dict
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_decision_dict accepts valid CLI dict for plan_id field
|
|
Given I have a decision dict with "plan_id" set to 01ARZ3NDEKTSV4XXFFJFRC889B
|
|
When I validate the decision dict
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_decision_dict accepts valid CLI dict for type field
|
|
Given I have a decision dict with "type" set to strategy_choice
|
|
When I validate the decision dict
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_decision_dict accepts valid CLI dict for sequence field
|
|
Given I have a decision dict with "sequence" set to 5
|
|
When I validate the decision dict
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_decision_dict accepts valid CLI dict for question field
|
|
Given I have a decision dict with "question" set to Which approach?
|
|
When I validate the decision dict
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_decision_dict accepts valid CLI dict for chosen field
|
|
Given I have a decision dict with "chosen" set to A
|
|
When I validate the decision dict
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_decision_dict accepts valid CLI dict for confidence field
|
|
Given I have a decision dict with "confidence" set to 0.75
|
|
When I validate the decision dict
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_decision_dict accepts valid CLI dict for parent field
|
|
Given I have a decision dict with "parent" set to (root)
|
|
When I validate the decision dict
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_decision_dict accepts valid CLI dict for is_correction field
|
|
Given I have a decision dict with "is_correction" set to false
|
|
When I validate the decision dict
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_decision_dict accepts valid CLI dict for superseded field
|
|
Given I have a decision dict with "superseded" set to false
|
|
When I validate the decision dict
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_decision_dict rejects confidence outside [0,1]
|
|
Given I have a decision dict with "confidence" set to 1.5
|
|
When I validate the decision dict
|
|
Then it should be structurally invalid
|
|
And error message must include "must be in range [0.0, 1.0]"
|
|
|
|
Scenario: validate_decision_dict accepts None confidence
|
|
Given I have a decision dict with "confidence" set to null
|
|
When I validate the decision dict
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_decision_dict rejects non-string question
|
|
Given I have a decision dict with "question" set to 42
|
|
When I validate the decision dict
|
|
Then it should be structurally invalid
|
|
And error message must include "must be a non-empty string"
|
|
|
|
Scenario: validate_decision_dict rejects parent that is neither ULID nor "(root)"
|
|
Given I have a decision dict with "parent" set to "random-string"
|
|
When I validate the decision dict
|
|
Then it should be structurally invalid
|
|
And error message must include "must be a ULID or '(root)'"
|
|
|
|
Scenario: validate_decision_dict rejects is_correction that is not boolean
|
|
Given I have a decision dict with "is_correction" set to "yes"
|
|
When I validate the decision dict
|
|
Then it should be structurally invalid
|
|
And error message must include "must be a boolean"
|
|
|
|
Scenario: validate_structured_output accepts valid envelope for command field
|
|
Given I have a structured output with "command" set to agents plan list (and all other required fields present)
|
|
When I validate the structured output
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_structured_output accepts valid envelope for session_id field
|
|
Given I have a structured output with "session_id" set to 01ARZ3NDEKTSV4XXFFJFRC889A (and all other required fields present)
|
|
When I validate the structured output
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_structured_output accepts valid envelope for status field
|
|
Given I have a structured output with "status" set to ok (and all other required fields present)
|
|
When I validate the structured output
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_structured_output accepts valid envelope for exit_code field
|
|
Given I have a structured output with "exit_code" set to 0 (and all other required fields present)
|
|
When I validate the structured output
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_structured_output rejects invalid status value
|
|
Given I have a structured output with "status" set to "running" (and all other required fields present)
|
|
When I validate the structured output
|
|
Then it should be structurally invalid
|
|
And error must include "must be one of"
|
|
|
|
Scenario: validate_structured_output rejects negative exit_code
|
|
Given I have a structured output with "exit_code" set to "-1" (and all other required fields present)
|
|
When I validate the structured output
|
|
Then it should be structurally invalid
|
|
And error must include "must be >= 0"
|
|
|
|
Scenario: validate_structured_output accepts zero elements list
|
|
Given I have a structured output with empty "elements" list (and all other fields valid)
|
|
When I validate the structured output
|
|
Then it should be structurally valid
|
|
|
|
Scenario: validate_structured_output rejects invalid element format
|
|
Given I have a structured output with an element missing "kind" field
|
|
When I validate the structured output
|
|
Then it should be structurally invalid
|
|
And error must include "missing 'kind' field"
|
|
|
|
Scenario: validate_structured_component_output routes plan_tree to correct validator
|
|
Given I target validation for "plan_tree" with valid data
|
|
When I call validate_structured_component_output
|
|
Then it should dispatch to the matching validator
|
|
And return a valid result
|
|
|
|
Scenario: validate_structured_component_output routes decision to correct validator
|
|
Given I target validation for "decision" with valid data
|
|
When I call validate_structured_component_output
|
|
Then it should dispatch to the matching validator
|
|
And return a valid result
|
|
|
|
Scenario: validate_structured_component_output routes structured_output to correct validator
|
|
Given I target validation for "structured_output" with valid data
|
|
When I call validate_structured_component_output
|
|
Then it should dispatch to the matching validator
|
|
And return a valid result
|
|
|
|
Scenario: validate_structured_component_output rejects unknown target type
|
|
Given I target validation for "unknown_type" with valid data
|
|
When I call validate_structured_component_output
|
|
Then it should raise a ValidationError with message containing "Unknown target_type"
|
|
|
|
Scenario: Structural output is robust to minor formatting differences
|
|
Given I have a decision dict where whitespace may vary between fields
|
|
When I validate the decision dict
|
|
Then validation should pass (structure matched, not exact characters)
|
|
|
|
Scenario: Plan tree nodes preserve parent-child structural relationships
|
|
Given a plan tree with parent and child node references connected by ULID
|
|
When I validate the plan tree structure
|
|
Then parent-child relationships should be validly connected
|
|
|
|
Scenario: Validation error messages are actionable
|
|
Given I have invalid plan tree data
|
|
When I validate the plan tree
|
|
Then the first error message should identify "node[0]" and the specific field issue
|