freemo
cbb985627d
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 16s
CI / quality (pull_request) Successful in 21s
CI / lint (pull_request) Successful in 22s
CI / security (pull_request) Successful in 36s
CI / typecheck (pull_request) Successful in 37s
CI / integration_tests (pull_request) Successful in 4m49s
CI / unit_tests (pull_request) Successful in 5m54s
CI / docker (pull_request) Successful in 38s
CI / benchmark-regression (pull_request) Successful in 15m25s
CI / coverage (pull_request) Successful in 41m50s
215 lines
8.2 KiB
Gherkin
215 lines
8.2 KiB
Gherkin
Feature: Automation Profile Guards
|
|
As a developer
|
|
I want automation profile guards to enforce runtime constraints
|
|
So that tool invocations are gated by budget, call limits, and policies
|
|
|
|
# ---- Guard allows tool within limits ----
|
|
|
|
Scenario: Guard allows tool within call limits
|
|
Given a profile with max_tool_calls_per_step 5
|
|
When I check guard for tool "read_file" with 3 calls so far
|
|
Then the guard result should be allowed
|
|
|
|
Scenario: Guard allows tool with no guards configured
|
|
Given a profile with no guards
|
|
When I check guard for tool "any_tool"
|
|
Then the guard result should be allowed
|
|
|
|
# ---- Guard blocks tool exceeding max_tool_calls ----
|
|
|
|
Scenario: Guard blocks tool exceeding max_tool_calls
|
|
Given a profile with max_tool_calls_per_step 3
|
|
When I check guard for tool "read_file" with 3 calls so far
|
|
Then the guard result should not be allowed
|
|
And the guard reason should mention "limit"
|
|
And the guard should require approval
|
|
|
|
Scenario: Guard blocks tool at exact max_tool_calls boundary
|
|
Given a profile with max_tool_calls_per_step 0
|
|
When I check guard for tool "read_file" with 0 calls so far
|
|
Then the guard result should not be allowed
|
|
|
|
# ---- Guard blocks tool on denylist ----
|
|
|
|
Scenario: Guard blocks tool on denylist
|
|
Given a profile with denylist "shell_exec,file_delete"
|
|
When I check guard for tool "shell_exec"
|
|
Then the guard result should not be allowed
|
|
And the guard reason should mention "denylist"
|
|
And the guard should require approval
|
|
|
|
Scenario: Guard allows tool not on denylist
|
|
Given a profile with denylist "shell_exec,file_delete"
|
|
When I check guard for tool "read_file"
|
|
Then the guard result should be allowed
|
|
|
|
# ---- Guard allows tool on allowlist ----
|
|
|
|
Scenario: Guard allows tool on allowlist
|
|
Given a profile with allowlist "read_file,list_dir"
|
|
When I check guard for tool "read_file"
|
|
Then the guard result should be allowed
|
|
|
|
Scenario: Guard blocks tool not on allowlist
|
|
Given a profile with allowlist "read_file,list_dir"
|
|
When I check guard for tool "write_file"
|
|
Then the guard result should not be allowed
|
|
And the guard reason should mention "allowlist"
|
|
|
|
# ---- Guard requires approval for writes ----
|
|
|
|
Scenario: Guard requires approval for write operations
|
|
Given a profile with require_approval_for_writes true
|
|
When I check guard for tool "write_file" with is_write true
|
|
Then the guard result should not be allowed
|
|
And the guard reason should mention "Write"
|
|
And the guard should require approval
|
|
|
|
Scenario: Guard allows non-write operations when writes require approval
|
|
Given a profile with require_approval_for_writes true
|
|
When I check guard for tool "read_file" with is_write false
|
|
Then the guard result should be allowed
|
|
|
|
# ---- Guard requires approval for apply ----
|
|
|
|
Scenario: Guard requires approval for apply phase
|
|
Given a profile with require_approval_for_apply true
|
|
When I check guard for tool "__apply__"
|
|
Then the guard result should not be allowed
|
|
And the guard reason should mention "Apply"
|
|
And the guard should require approval
|
|
|
|
Scenario: Guard allows non-apply tools when apply requires approval
|
|
Given a profile with require_approval_for_apply true
|
|
When I check guard for tool "read_file"
|
|
Then the guard result should be allowed
|
|
|
|
# ---- Cost budget enforcement ----
|
|
|
|
Scenario: Guard blocks tool when cost budget exceeded
|
|
Given a profile with max_total_cost 10.0
|
|
When I check guard for tool "expensive_tool" with cost 10.0
|
|
Then the guard result should not be allowed
|
|
And the guard reason should mention "Cost"
|
|
And the guard should require approval
|
|
|
|
Scenario: Guard allows tool within cost budget
|
|
Given a profile with max_total_cost 10.0
|
|
When I check guard for tool "cheap_tool" with cost 5.0
|
|
Then the guard result should be allowed
|
|
|
|
# ---- Effective profile resolution ----
|
|
|
|
Scenario: Effective profile resolves plan-level first
|
|
Given a profile service with profiles "cautious" and "manual"
|
|
When I resolve effective profile with plan "cautious"
|
|
Then the resolved guard profile name should be "cautious"
|
|
|
|
Scenario: Effective profile falls back to action level
|
|
Given a profile service with profiles "cautious" and "manual"
|
|
When I resolve effective profile with action "cautious"
|
|
Then the resolved guard profile name should be "cautious"
|
|
|
|
Scenario: Effective profile falls back to default
|
|
Given a profile service with profiles "cautious" and "manual"
|
|
When I resolve effective profile with default "cautious"
|
|
Then the resolved guard profile name should be "cautious"
|
|
|
|
Scenario: Effective profile uses global when no overrides
|
|
Given a profile service with global default "manual"
|
|
When I resolve effective profile with no overrides
|
|
Then the resolved guard profile name should be "manual"
|
|
|
|
# ---- Custom profile YAML loading ----
|
|
|
|
Scenario: Load custom profile from YAML file
|
|
Given a YAML profile file with guards
|
|
When I load the profile from YAML
|
|
Then the loaded profile should have guards
|
|
And the loaded profile guards max_tool_calls_per_step should be 10
|
|
|
|
Scenario: Load custom profile from YAML via service
|
|
Given a YAML profile file with guards
|
|
When I load the profile via service from_yaml
|
|
Then the loaded profile should have guards
|
|
|
|
# ---- Built-in profile immutability ----
|
|
|
|
Scenario: Built-in profiles cannot be updated
|
|
Given a profile service
|
|
When I try to update built-in profile "manual"
|
|
Then a profile service validation error should be raised
|
|
|
|
Scenario: Built-in profiles cannot be deleted
|
|
Given a profile service
|
|
When I try to delete built-in profile "manual"
|
|
Then a profile service validation error should be raised
|
|
|
|
# ---- Guard model validation ----
|
|
|
|
Scenario: AutomationGuard rejects negative max_tool_calls
|
|
When I try to create a guard with max_tool_calls -1
|
|
Then a guard model validation error should be raised
|
|
|
|
Scenario: AutomationGuard rejects negative max_total_cost
|
|
When I try to create a guard with max_total_cost -1.0
|
|
Then a guard model validation error should be raised
|
|
|
|
Scenario: AutomationGuard accepts zero max_tool_calls
|
|
When I create a guard with max_tool_calls 0
|
|
Then the guard should be created
|
|
|
|
Scenario: AutomationGuard accepts None values
|
|
When I create a guard with defaults
|
|
Then the guard max_tool_calls_per_step should be None
|
|
And the guard max_total_cost should be None
|
|
|
|
# ---- GuardResult model ----
|
|
|
|
Scenario: GuardResult allowed with no reason
|
|
When I create a guard result allowed true
|
|
Then the guard result allowed should be true
|
|
And the guard result reason should be None
|
|
|
|
Scenario: GuardResult denied with reason
|
|
When I create a guard result denied with reason "blocked"
|
|
Then the guard result allowed should be false
|
|
And the guard result reason should be "blocked"
|
|
And the guard result requires_approval should be true
|
|
|
|
# ---- Service evaluate_guard ----
|
|
|
|
Scenario: Service evaluate_guard delegates to profile
|
|
Given a profile service
|
|
When I evaluate guard for profile "full-auto" tool "any_tool"
|
|
Then the guard result should be allowed
|
|
|
|
Scenario: Service evaluate_guard rejects empty profile name
|
|
Given a profile service
|
|
When I try to evaluate guard with empty profile name
|
|
Then a profile service validation error should be raised
|
|
|
|
Scenario: Service evaluate_guard rejects empty tool name
|
|
Given a profile service
|
|
When I try to evaluate guard with empty tool name
|
|
Then a profile service validation error should be raised
|
|
|
|
# ---- Profile with guards from config dict ----
|
|
|
|
Scenario: Profile from config dict with guards
|
|
When I create a profile from config with guards
|
|
Then the profile should have guards configured
|
|
And the profile guards should have denylist
|
|
|
|
# ---- check_guard argument validation ----
|
|
|
|
Scenario: check_guard rejects negative cost_so_far
|
|
Given a profile with max_total_cost 10.0
|
|
When I try to check guard with negative cost
|
|
Then a guard argument error should be raised
|
|
|
|
Scenario: check_guard rejects negative calls_so_far
|
|
Given a profile with max_tool_calls_per_step 5
|
|
When I try to check guard with negative calls
|
|
Then a guard argument error should be raised
|