286 lines
13 KiB
Gherkin
286 lines
13 KiB
Gherkin
Feature: Safety Profile Domain Model
|
|
As a developer
|
|
I want safety profile domain models with constraint validation
|
|
So that plan execution safety can be configured and enforced
|
|
|
|
# ---- Default construction ----
|
|
|
|
Scenario: Default safety profile has correct defaults
|
|
When I create a default safety profile
|
|
Then the safety profile should be created
|
|
And the safety require_sandbox should be true
|
|
And the safety require_checkpoints should be true
|
|
And the safety require_human_approval should be false
|
|
And the safety allow_unsafe_tools should be false
|
|
And the safety max_retries_per_step should be 3
|
|
And the safety max_cost_per_plan should be none
|
|
And the safety max_total_cost should be none
|
|
And the safety allowed_skill_categories should be empty
|
|
|
|
# ---- Field parsing ----
|
|
|
|
Scenario: Safety profile accepts allowed_skill_categories
|
|
When I create a safety profile with categories "code,test,deploy"
|
|
Then the safety profile should be created
|
|
And the safety allowed_skill_categories count should be 3
|
|
|
|
Scenario: Safety profile deduplicates categories
|
|
When I create a safety profile with categories "code,test,code"
|
|
Then the safety profile should be created
|
|
And the safety allowed_skill_categories count should be 2
|
|
|
|
Scenario: Safety profile strips blank categories
|
|
When I create a safety profile with categories "code, ,test, "
|
|
Then the safety profile should be created
|
|
And the safety allowed_skill_categories count should be 2
|
|
|
|
# ---- Constraint validation ----
|
|
|
|
Scenario: Safety profile rejects negative max_cost_per_plan
|
|
When I try to create a safety profile with max_cost_per_plan -1.0
|
|
Then a safety validation error should be raised
|
|
|
|
Scenario: Safety profile rejects negative max_total_cost
|
|
When I try to create a safety profile with max_total_cost -5.0
|
|
Then a safety validation error should be raised
|
|
|
|
Scenario: Safety profile rejects negative max_retries_per_step
|
|
When I try to create a safety profile with max_retries_per_step -1
|
|
Then a safety validation error should be raised
|
|
|
|
Scenario: Safety profile rejects max_retries_per_step above 100
|
|
When I try to create a safety profile with max_retries_per_step 101
|
|
Then a safety validation error should be raised
|
|
|
|
Scenario: Safety profile accepts zero max_cost_per_plan
|
|
When I create a safety profile with max_cost_per_plan 0.0
|
|
Then the safety profile should be created
|
|
And the safety max_cost_per_plan should be 0.0
|
|
|
|
Scenario: Safety profile accepts zero max_retries_per_step
|
|
When I create a safety profile with max_retries_per_step 0
|
|
Then the safety profile should be created
|
|
And the safety max_retries_per_step should be 0
|
|
|
|
# ---- Cross-field validation ----
|
|
|
|
Scenario: Safety profile rejects max_cost_per_plan exceeding max_total_cost
|
|
When I try to create a safety profile with cost 1000.0 and total 10.0
|
|
Then a safety validation error should be raised
|
|
|
|
Scenario: Safety profile accepts max_cost_per_plan equal to max_total_cost
|
|
When I create a safety profile with cost 100.0 and total 100.0
|
|
Then the safety profile should be created
|
|
And the safety max_cost_per_plan should be 100.0
|
|
And the safety max_total_cost should be 100.0
|
|
|
|
# ---- Category type guard ----
|
|
|
|
Scenario: Safety profile rejects non-string categories
|
|
When I try to create a safety profile with non-string categories
|
|
Then a safety validation error should be raised
|
|
|
|
Scenario: Safety profile rejects non-list categories
|
|
When I try to create a safety profile with non-list categories
|
|
Then a safety validation error should be raised
|
|
|
|
# ---- Frozen immutability ----
|
|
|
|
Scenario: Safety profile is immutable (frozen)
|
|
When I create a default safety profile
|
|
And I try to assign max_retries_per_step -1 on the safety profile
|
|
Then a safety validation error should be raised
|
|
|
|
# ---- from_config factory ----
|
|
|
|
Scenario: Safety profile from config loads correctly
|
|
When I load a safety profile from config with require_sandbox false
|
|
Then the safety profile should be created
|
|
And the safety require_sandbox should be false
|
|
|
|
Scenario: Safety profile from config with all fields
|
|
When I load a safety profile from full config
|
|
Then the safety profile should be created
|
|
And the safety require_sandbox should be false
|
|
And the safety require_checkpoints should be false
|
|
And the safety require_human_approval should be true
|
|
And the safety allow_unsafe_tools should be true
|
|
And the safety max_cost_per_plan should be 50.0
|
|
And the safety max_retries_per_step should be 5
|
|
And the safety max_total_cost should be 200.0
|
|
|
|
# ---- DEFAULT_SAFETY_PROFILE constant ----
|
|
|
|
Scenario: DEFAULT_SAFETY_PROFILE has sensible defaults
|
|
When I load the default safety profile constant
|
|
Then the safety profile should be created
|
|
And the safety require_sandbox should be true
|
|
And the safety require_checkpoints should be true
|
|
And the safety require_human_approval should be false
|
|
And the safety allow_unsafe_tools should be false
|
|
And the safety max_retries_per_step should be 3
|
|
|
|
# ---- SafetyProfileRef ----
|
|
|
|
Scenario: SafetyProfileRef stores name and provenance
|
|
When I create a safety profile ref with name "strict" and provenance "plan"
|
|
Then the safety profile ref should have name "strict"
|
|
And the safety profile ref should have provenance "plan"
|
|
|
|
Scenario: SafetyProfileRef rejects empty name
|
|
When I try to create a safety profile ref with empty name
|
|
Then a safety validation error should be raised
|
|
|
|
# ---- resolve_safety_profile precedence ----
|
|
|
|
Scenario: resolve_safety_profile returns plan-level when all levels set
|
|
Given a plan safety profile with allow_unsafe_tools true
|
|
And an action safety profile with allow_unsafe_tools false
|
|
And a project safety profile with allow_unsafe_tools false
|
|
And a global safety profile with allow_unsafe_tools false
|
|
When I resolve the safety profile
|
|
Then the resolved provenance should be "plan"
|
|
And the resolved profile allow_unsafe_tools should be true
|
|
|
|
Scenario: resolve_safety_profile returns action-level when no plan-level
|
|
Given an action safety profile with allow_unsafe_tools true
|
|
And a project safety profile with allow_unsafe_tools false
|
|
And a global safety profile with allow_unsafe_tools false
|
|
When I resolve the safety profile
|
|
Then the resolved provenance should be "action"
|
|
And the resolved profile allow_unsafe_tools should be true
|
|
|
|
Scenario: resolve_safety_profile returns project-level when no plan or action
|
|
Given a project safety profile with allow_unsafe_tools true
|
|
And a global safety profile with allow_unsafe_tools false
|
|
When I resolve the safety profile
|
|
Then the resolved provenance should be "project"
|
|
And the resolved profile allow_unsafe_tools should be true
|
|
|
|
Scenario: resolve_safety_profile returns global-level when only global set
|
|
Given a global safety profile with allow_unsafe_tools true
|
|
When I resolve the safety profile
|
|
Then the resolved provenance should be "global"
|
|
And the resolved profile allow_unsafe_tools should be true
|
|
|
|
Scenario: resolve_safety_profile returns default when all levels are None
|
|
When I resolve the safety profile with no levels
|
|
Then the resolved provenance should be "global"
|
|
And the resolved profile allow_unsafe_tools should be false
|
|
And the resolved profile require_sandbox should be true
|
|
|
|
Scenario: resolve_safety_profile preserves all 8 fields from plan-level
|
|
Given a full plan safety profile with all fields customized
|
|
And a full action safety profile with defaults
|
|
When I resolve the safety profile
|
|
Then the resolved provenance should be "plan"
|
|
And the resolved profile require_sandbox should be false
|
|
And the resolved profile require_checkpoints should be false
|
|
And the resolved profile allow_unsafe_tools should be true
|
|
And the resolved profile require_human_approval should be true
|
|
And the resolved profile max_cost_per_plan should be 50.0
|
|
And the resolved profile max_retries_per_step should be 7
|
|
And the resolved profile max_total_cost should be 200.0
|
|
And the resolved profile allowed_skill_categories should be "code,test"
|
|
|
|
Scenario: resolve_safety_profile preserves all 8 fields from action-level
|
|
Given a full action safety profile with all fields customized
|
|
When I resolve the safety profile
|
|
Then the resolved provenance should be "action"
|
|
And the resolved profile require_sandbox should be false
|
|
And the resolved profile require_checkpoints should be false
|
|
And the resolved profile allow_unsafe_tools should be true
|
|
And the resolved profile require_human_approval should be true
|
|
And the resolved profile max_cost_per_plan should be 50.0
|
|
And the resolved profile max_retries_per_step should be 7
|
|
And the resolved profile max_total_cost should be 200.0
|
|
And the resolved profile allowed_skill_categories should be "code,test"
|
|
|
|
# ---- model_dump round-trip ----
|
|
|
|
Scenario: Safety profile model_dump produces valid dict
|
|
When I create a safety profile and dump it
|
|
Then the safety dump should have key "require_sandbox"
|
|
Then the safety dump should have key "require_checkpoints"
|
|
And the safety dump should have key "max_retries_per_step"
|
|
And the safety dump should have key "allow_unsafe_tools"
|
|
And the safety dump require_sandbox should be true
|
|
|
|
# ---- Action with safety profile ----
|
|
|
|
Scenario: Action accepts safety profile in from_config
|
|
When I create an action with safety profile from config
|
|
Then the action safety profile should not be none
|
|
And the action safety profile require_sandbox should be false
|
|
|
|
Scenario: Action as_cli_dict includes safety profile
|
|
When I create an action with safety profile from config
|
|
Then the action cli dict should contain safety_profile key
|
|
|
|
# ---- from_yaml edge cases ----
|
|
|
|
Scenario: Safety profile from_yaml rejects non-existent file
|
|
When I try to load a safety profile from a non-existent yaml file
|
|
Then a safety FileNotFoundError should be raised
|
|
|
|
Scenario: Safety profile from_yaml rejects non-dict content
|
|
When I try to load a safety profile from yaml with non-dict content
|
|
Then a safety ValueError should be raised
|
|
|
|
# ---- allowed_tools / denied_tools fields ----
|
|
|
|
Scenario: Safety profile accepts allowed_tools list
|
|
When I create a safety profile with allowed_tools "local/reader,local/writer"
|
|
Then the safety profile should be created
|
|
And the safety allowed_tools count should be 2
|
|
|
|
Scenario: Safety profile accepts denied_tools list
|
|
When I create a safety profile with denied_tools "net/http,net/ftp"
|
|
Then the safety profile should be created
|
|
And the safety denied_tools count should be 2
|
|
|
|
Scenario: Safety profile deduplicates allowed_tools
|
|
When I create a safety profile with allowed_tools "local/reader,local/reader"
|
|
Then the safety profile should be created
|
|
And the safety allowed_tools count should be 1
|
|
|
|
Scenario: Safety profile deduplicates denied_tools
|
|
When I create a safety profile with denied_tools "net/http,net/http"
|
|
Then the safety profile should be created
|
|
And the safety denied_tools count should be 1
|
|
|
|
Scenario: Safety profile default has empty allowed_tools and denied_tools
|
|
When I create a default safety profile
|
|
Then the safety profile should be created
|
|
And the safety allowed_tools should be empty
|
|
And the safety denied_tools should be empty
|
|
|
|
# ---- Built-in safety profiles ----
|
|
|
|
Scenario: Unrestricted built-in profile has allow_unsafe_tools true
|
|
When I load the unrestricted built-in safety profile
|
|
Then the safety profile should be created
|
|
And the safety allow_unsafe_tools should be true
|
|
And the safety require_sandbox should be false
|
|
And the safety require_checkpoints should be false
|
|
|
|
Scenario: Read-only built-in profile has allow_unsafe_tools false
|
|
When I load the read-only built-in safety profile
|
|
Then the safety profile should be created
|
|
And the safety allow_unsafe_tools should be false
|
|
|
|
Scenario: No-network built-in profile requires sandbox
|
|
When I load the no-network built-in safety profile
|
|
Then the safety profile should be created
|
|
And the safety require_sandbox should be true
|
|
|
|
Scenario: Sandboxed built-in profile requires sandbox and checkpoints
|
|
When I load the sandboxed built-in safety profile
|
|
Then the safety profile should be created
|
|
And the safety require_sandbox should be true
|
|
And the safety require_checkpoints should be true
|
|
|
|
Scenario: get_builtin_safety_profile raises KeyError for unknown name
|
|
When I try to load an unknown built-in safety profile "nonexistent"
|
|
Then a safety KeyError should be raised
|