aditya
2764fcef5c
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 16s
CI / build (pull_request) Successful in 25s
CI / quality (pull_request) Successful in 34s
CI / typecheck (pull_request) Successful in 39s
CI / security (pull_request) Successful in 52s
CI / unit_tests (pull_request) Successful in 2m51s
CI / integration_tests (pull_request) Successful in 3m41s
CI / docker (pull_request) Successful in 56s
CI / e2e_tests (pull_request) Successful in 3m56s
CI / coverage (pull_request) Successful in 6m28s
CI / benchmark-regression (pull_request) Successful in 38m39s
Address review-driven fixes across actor schema, preflight guardrails, docs/examples, and Behave/Robot coverage: unify preflight warning behavior with shared role-warning logic, resolve actor-name to config payloads in production preflight flow, harden response_format validation/coercion edge cases, extract duplicated helper logic, and expand negative-path test coverage. Also fix cross-scenario patcher leakage in step modules to eliminate full-run-only coverage failures.
138 lines
6.0 KiB
Gherkin
138 lines
6.0 KiB
Gherkin
Feature: Plan Generation Pre-flight Guardrails — 7 checks before execution
|
|
As a plan executor
|
|
I want pre-flight checks to validate plan readiness
|
|
So that ill-formed plans are rejected before entering the Strategize phase
|
|
|
|
# Check 1: Action Schema Validation
|
|
Scenario: pfg-check1 Valid action passes schema validation
|
|
Given a plan preflight guardrail
|
|
And an action registry containing "refactor-action"
|
|
When I check action schema for "refactor-action"
|
|
Then the check should pass with message containing "found and valid"
|
|
|
|
Scenario: pfg-check1-fail Missing action fails schema validation
|
|
Given a plan preflight guardrail
|
|
And an empty action registry
|
|
When I check action schema for "nonexistent-action"
|
|
Then the check should fail with message containing "not found"
|
|
|
|
# Check 2: Actor Availability
|
|
Scenario: pfg-check2 All actor roles registered passes availability check
|
|
Given a plan preflight guardrail
|
|
And an actor registry with all 4 roles registered
|
|
When I check actor availability
|
|
Then the check should pass
|
|
|
|
Scenario: pfg-check2-fail Missing actor roles fails availability check
|
|
Given a plan preflight guardrail
|
|
And an actor registry missing "estimation" role
|
|
When I check actor availability
|
|
Then the check should fail with message containing "estimation"
|
|
|
|
Scenario: pfg-check2-warn Estimation actor missing response_format emits warning
|
|
Given a plan preflight guardrail with all registries populated
|
|
And an estimation actor registry entry without response_format
|
|
When I run all preflight checks
|
|
Then no PreflightRejection should be raised
|
|
And the preflight warnings should contain "response_format"
|
|
|
|
Scenario: pfg-check2-warn Estimation actor wrong context_view emits warning
|
|
Given a plan preflight guardrail with all registries populated
|
|
And an estimation actor registry entry with non-strategist context_view
|
|
When I run all preflight checks
|
|
Then no PreflightRejection should be raised
|
|
And the preflight warnings should contain "context_view"
|
|
|
|
Scenario: pfg-check2-warn Estimation actor as ActorConfigSchema emits warning
|
|
Given a plan preflight guardrail with all registries populated
|
|
And an estimation actor registry entry as ActorConfigSchema without response_format
|
|
When I run all preflight checks
|
|
Then no PreflightRejection should be raised
|
|
And the preflight warnings should contain "response_format"
|
|
|
|
# Check 3: Skill/Tool Existence
|
|
Scenario: pfg-check3 All tools and skills exist passes existence check
|
|
Given a plan preflight guardrail
|
|
And a tool registry containing "git-diff" and "file-read"
|
|
When I check skill/tool existence for tools "git-diff,file-read"
|
|
Then the check should pass
|
|
|
|
Scenario: pfg-check3-fail Missing tool fails existence check
|
|
Given a plan preflight guardrail
|
|
And a tool registry containing "git-diff"
|
|
When I check skill/tool existence for tools "git-diff,missing-tool"
|
|
Then the check should fail with message containing "missing-tool"
|
|
|
|
# Check 4: Automation Policy
|
|
Scenario: pfg-check4 Valid automation profile passes policy check
|
|
Given a plan preflight guardrail
|
|
And a valid automation profile
|
|
When I check automation policy
|
|
Then the check should pass
|
|
|
|
Scenario: pfg-check4-fail No automation profile fails policy check
|
|
Given a plan preflight guardrail
|
|
And no automation profile
|
|
When I check automation policy
|
|
Then the check should fail with message containing "No automation profile"
|
|
|
|
# Check 5: Rollback Feasibility
|
|
Scenario: pfg-check5 All tools checkpointable passes rollback check
|
|
Given a plan preflight guardrail
|
|
And require_checkpoints is true
|
|
And tool capabilities showing all tools checkpointable
|
|
When I check rollback feasibility
|
|
Then the check should pass
|
|
|
|
Scenario: pfg-check5-fail Non-checkpointable tool fails rollback check
|
|
Given a plan preflight guardrail
|
|
And require_checkpoints is true
|
|
And tool capabilities with "unsafe-tool" not checkpointable
|
|
When I check rollback feasibility
|
|
Then the check should fail with message containing "unsafe-tool"
|
|
|
|
Scenario: pfg-check5-skip Checkpoints not required skips rollback check
|
|
Given a plan preflight guardrail
|
|
And require_checkpoints is false
|
|
When I check rollback feasibility
|
|
Then the check should pass with message containing "skipped"
|
|
|
|
# Check 6: Resource Accessibility
|
|
Scenario: pfg-check6 Existing resource paths pass accessibility check
|
|
Given a plan preflight guardrail
|
|
And resource paths that exist on filesystem
|
|
When I check resource accessibility
|
|
Then the check should pass
|
|
|
|
Scenario: pfg-check6-fail Missing resource path fails accessibility check
|
|
Given a plan preflight guardrail
|
|
And resource path "/nonexistent/path/12345"
|
|
When I check resource accessibility
|
|
Then the check should fail with message containing "Inaccessible"
|
|
|
|
# Check 7: Validation Resolution
|
|
Scenario: pfg-check7 All validations resolvable passes resolution check
|
|
Given a plan preflight guardrail
|
|
And a validation registry containing "lint-check" and "type-check"
|
|
When I check validation resolution for "lint-check,type-check"
|
|
Then the check should pass
|
|
|
|
Scenario: pfg-check7-fail Unresolvable validation fails resolution check
|
|
Given a plan preflight guardrail
|
|
And a validation registry containing "lint-check"
|
|
When I check validation resolution for "lint-check,missing-validation"
|
|
Then the check should fail with message containing "missing-validation"
|
|
|
|
# Integration: run_all_checks
|
|
Scenario: pfg-all All checks pass produces clean report
|
|
Given a plan preflight guardrail with all registries populated
|
|
When I run all preflight checks
|
|
Then the report should show all checks passed
|
|
And no PreflightRejection should be raised
|
|
|
|
Scenario: pfg-all-fail First failure causes PreflightRejection
|
|
Given a plan preflight guardrail with missing action
|
|
When I run all preflight checks expecting rejection
|
|
Then a PreflightRejection should be raised
|
|
And the rejection should identify "action_schema_validation"
|