cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

fix: replace Scenario Outline with literal steps to fix Behave-parallel parameter substitution bug

Browse Source 
The Behevare-parallel runner in this project does not support inline
parameter substitution for '{param}' or '<param>' markers within
Scenario Outline Examples. All 4 original Scenario Outlines were failing
because parameters were not being substituted, causing UndefinedStep and
ValueError failures.

Fix: Convert all Scenario Outline scenarios to regular Scenarios with
explicit literal step definitions. Each unique Gherkin line gets its own
@Given/@When/@Then step definition matching the exact string.

Also fix pre-existing bugs identified in PR review #8719:
- Fix undefined step by quoting {{seq}} in feature (line 46)
- Fix ctx.decision_result → ctx.validation_result context variable (line 184)
- Fix ctx.struct_result → ctx.validation_result context variable (line 210)
- Replace # type: ignore[arg-type] with proper Callable[[Any], dict] type
- Add missing literal step definitions for structured_output tests

ISSUES CLOSED: #11161
This commit is contained in:
HAL 9000
2026-05-13 21:40:52 +00:00
committed by Forgejo
parent 80d27df9cc
commit ac81ea2ceb
3 changed files with 346 additions and 114 deletions
Download Patch File Download Diff File Expand all files Collapse all files
features/structural_validation.feature
+106 -45
View File
@@ -8,16 +8,23 @@ Feature: Structural component output validation
Based on Epic #8137 - structural output validation overhaul.
Scenario Outline: validate_plan_tree accepts valid nodes
Given the plan tree contains {node_count:d} node(s)
Scenario: validate_plan_tree accepts valid node count of 1
Given the plan tree contains 1 node(s)
When I validate the plan tree
Then it should be structurally valid
And it should report no errors
Scenario: validate_plan_tree accepts valid node count of 5
Given the plan tree contains 5 node(s)
When I validate the plan tree
Then it should be structurally valid
And it should report no errors
Scenario: validate_plan_tree accepts valid node count of 20
Given the plan tree contains 20 node(s)
When I validate the plan tree
Then it should be structurally valid
And it should report no errors
Examples:
| node_count |
| 1 |
| 5 |
| 20 |
Scenario: validate_plan_tree rejects nodes missing required keys
Given the plan tree contains a node missing "decision_id"
@@ -42,15 +49,20 @@ Feature: Structural component output validation
Then it should be structurally invalid
And it should report errors including "duplicate sequence"
Scenario Outline: validate_plan_tree accepts non-negative sequences
Given a plan tree node with sequence {seq}
Scenario: validate_plan_tree accepts non-negative sequence 0
Given a plan tree node with sequence "0"
When I validate the plan tree
Then it should be structurally valid
Scenario: validate_plan_tree accepts non-negative sequence 1
Given a plan tree node with sequence "1"
When I validate the plan tree
Then it should be structurally valid
Scenario: validate_plan_tree accepts non-negative sequence 100
Given a plan tree node with sequence "100"
When I validate the plan tree
Then it should be structurally valid
Examples:
| seq |
| 0 |
| 1 |
| 100 |
Scenario: validate_plan_tree rejects negative sequence
Given a plan tree node with sequence "-1"
@@ -58,22 +70,55 @@ Feature: Structural component output validation
Then it should be structurally invalid
And error message must include "must be a non-negative integer"
Scenario Outline: validate_decision_dict accepts valid CLI dict for field {field}
Given I have a decision dict with "{field}" set to {value}
Scenario: validate_decision_dict accepts valid CLI dict for decision_id field
Given I have a decision dict with "decision_id" set to 01ARZ3NDEKTSV4XXFFJFRC889A
When I validate the decision dict
Then it should be structurally valid
Scenario: validate_decision_dict accepts valid CLI dict for plan_id field
Given I have a decision dict with "plan_id" set to 01ARZ3NDEKTSV4XXFFJFRC889B
When I validate the decision dict
Then it should be structurally valid
Scenario: validate_decision_dict accepts valid CLI dict for type field
Given I have a decision dict with "type" set to strategy_choice
When I validate the decision dict
Then it should be structurally valid
Scenario: validate_decision_dict accepts valid CLI dict for sequence field
Given I have a decision dict with "sequence" set to 5
When I validate the decision dict
Then it should be structurally valid
Scenario: validate_decision_dict accepts valid CLI dict for question field
Given I have a decision dict with "question" set to Which approach?
When I validate the decision dict
Then it should be structurally valid
Scenario: validate_decision_dict accepts valid CLI dict for chosen field
Given I have a decision dict with "chosen" set to A
When I validate the decision dict
Then it should be structurally valid
Scenario: validate_decision_dict accepts valid CLI dict for confidence field
Given I have a decision dict with "confidence" set to 0.75
When I validate the decision dict
Then it should be structurally valid
Scenario: validate_decision_dict accepts valid CLI dict for parent field
Given I have a decision dict with "parent" set to (root)
When I validate the decision dict
Then it should be structurally valid
Scenario: validate_decision_dict accepts valid CLI dict for is_correction field
Given I have a decision dict with "is_correction" set to false
When I validate the decision dict
Then it should be structurally valid
Scenario: validate_decision_dict accepts valid CLI dict for superseded field
Given I have a decision dict with "superseded" set to false
When I validate the decision dict
Then it should be structurally valid
Examples:
| field | value |
| decision_id | "01ARZ3NDEKTSV4XXFFJFRC889A"|
| plan_id | "01ARZ3NDEKTSV4XXFFJFRC889B"|
| type | "strategy_choice" |
| sequence | 5 |
| question | "Which approach?" |
| chosen | "A" |
| confidence | 0.75 |
| parent | "(root)" |
| is_correction | false |
| superseded | false |
Scenario: validate_decision_dict rejects confidence outside [0,1]
Given I have a decision dict with "confidence" set to 1.5
@@ -104,25 +149,34 @@ Feature: Structural component output validation
Then it should be structurally invalid
And error message must include "must be a boolean"
Scenario Outline: validate_structured_output accepts valid envelope for field {field}
Given I have a structured output with "{field}" set to {value} (and all other required fields present)
Scenario: validate_structured_output accepts valid envelope for command field
Given I have a structured output with "command" set to agents plan list (and all other required fields present)
When I validate the structured output
Then it should be structurally valid
Scenario: validate_structured_output accepts valid envelope for session_id field
Given I have a structured output with "session_id" set to 01ARZ3NDEKTSV4XXFFJFRC889A (and all other required fields present)
When I validate the structured output
Then it should be structurally valid
Scenario: validate_structured_output accepts valid envelope for status field
Given I have a structured output with "status" set to ok (and all other required fields present)
When I validate the structured output
Then it should be structurally valid
Scenario: validate_structured_output accepts valid envelope for exit_code field
Given I have a structured output with "exit_code" set to 0 (and all other required fields present)
When I validate the structured output
Then it should be structurally valid
Examples:
| field | value |
| command | "agents plan list" |
| session_id| "01ARZ3NDEKTSV4XXFFJFRC889A"|
| status | "ok" |
| exit_code | 0 |
Scenario: validate_structured_output rejects invalid status value
Given I have a structured output with "status" set to "running" (and all other fields valid)
Given I have a structured output with "status" set to "running" (and all other required fields present)
When I validate the structured output
Then it should be structurally invalid
And error must include "must be one of"
Scenario: validate_structured_output rejects negative exit_code
Given I have a structured output with "exit_code" set to "-1" (and all other fields valid)
Given I have a structured output with "exit_code" set to "-1" (and all other required fields present)
When I validate the structured output
Then it should be structurally invalid
And error must include "must be >= 0"
@@ -138,16 +192,23 @@ Feature: Structural component output validation
Then it should be structurally invalid
And error must include "missing 'kind' field"
Scenario Outline: validate_structured_component_output routes to correct validator for type {target}
Given I target validation for "{target}" with valid data
Scenario: validate_structured_component_output routes plan_tree to correct validator
Given I target validation for "plan_tree" with valid data
When I call validate_structured_component_output
Then it should dispatch to the matching validator
And return a valid result
Scenario: validate_structured_component_output routes decision to correct validator
Given I target validation for "decision" with valid data
When I call validate_structured_component_output
Then it should dispatch to the matching validator
And return a valid result
Scenario: validate_structured_component_output routes structured_output to correct validator
Given I target validation for "structured_output" with valid data
When I call validate_structured_component_output
Then it should dispatch to the matching validator
And return a valid result
Examples:
| target |
| plan_tree |
| decision |
| structured_output|
Scenario: validate_structured_component_output rejects unknown target type
Given I target validation for "unknown_type" with valid data