173 lines
8.4 KiB
Plaintext
173 lines
8.4 KiB
Plaintext
*** Settings ***
|
|
Documentation Safety Profile Enforcement integration smoke tests
|
|
... Verifies resolve_safety_profile precedence and
|
|
... ToolRuntime safety enforcement (unsafe tool gating,
|
|
... skill category allow-lists, checkpoint requirements).
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_safety_profile_enforcement.py
|
|
|
|
*** Test Cases ***
|
|
Resolve Safety Profile Precedence Plan Over Action
|
|
[Documentation] Plan-level profile takes precedence over action-level
|
|
${result}= Run Process ${PYTHON} ${HELPER} resolve-precedence cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} resolve-precedence-ok
|
|
|
|
Resolve Safety Profile Default When All None
|
|
[Documentation] All-None resolution returns DEFAULT_SAFETY_PROFILE with GLOBAL provenance
|
|
${result}= Run Process ${PYTHON} ${HELPER} resolve-default cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} resolve-default-ok
|
|
|
|
Unsafe Tool Blocked By Safety Profile
|
|
[Documentation] Unsafe tool raises ToolSafetyViolationError when profile forbids
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-unsafe-blocked cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-unsafe-blocked-ok
|
|
|
|
Unsafe Tool Allowed By Safety Profile
|
|
[Documentation] Unsafe tool executes when profile permits allow_unsafe_tools
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-unsafe-allowed cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-unsafe-allowed-ok
|
|
|
|
Skill Category Blocked By Safety Profile
|
|
[Documentation] Tool in wrong skill category is blocked by allow-list
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-category-blocked cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-category-blocked-ok
|
|
|
|
Skill Category Allowed By Safety Profile
|
|
[Documentation] Tool in correct skill category runs normally
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-category-allowed cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-category-allowed-ok
|
|
|
|
Checkpoint Required By Safety Profile
|
|
[Documentation] Non-checkpointable tool blocked when profile requires checkpoints
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-checkpoint cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-checkpoint-ok
|
|
|
|
Backward Compatibility Without Safety Profile
|
|
[Documentation] Unsafe tool runs when no safety profile on context
|
|
${result}= Run Process ${PYTHON} ${HELPER} backward-compat cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} backward-compat-ok
|
|
|
|
Sandbox Required Blocks Writing Tool Without Sandbox Id
|
|
[Documentation] Writing tool raises ToolSandboxRequiredError when sandbox required but missing
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-sandbox-blocked cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-sandbox-blocked-ok
|
|
|
|
Sandbox Required Allows Writing Tool With Sandbox Id
|
|
[Documentation] Writing tool executes when sandbox required and sandbox_id set
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-sandbox-allowed cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-sandbox-allowed-ok
|
|
|
|
Human Approval Required Blocks Tool Without Approval
|
|
[Documentation] Tool blocked when require_human_approval=True and no approval
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-human-approval cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-human-approval-ok
|
|
|
|
Cost Limit Blocks Tool When Exceeded
|
|
[Documentation] Tool blocked when accumulated cost exceeds max_cost_per_plan
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-cost-limit cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-cost-limit-ok
|
|
|
|
Retry Limit Blocks Tool When Exceeded
|
|
[Documentation] Tool blocked when step retry count exceeds max_retries_per_step
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-retry-limit cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-retry-limit-ok
|
|
|
|
Context Summary Includes Safety Profile Flag
|
|
[Documentation] ToolExecutionContext.as_summary includes has_safety_profile
|
|
${result}= Run Process ${PYTHON} ${HELPER} context-summary cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} context-summary-ok
|
|
|
|
Allow-list Blocks Tool Not On List
|
|
[Documentation] Tool not on allow-list raises ToolAccessDeniedError
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-allowlist-blocked cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-allowlist-blocked-ok
|
|
|
|
Allow-list Permits Tool On List
|
|
[Documentation] Tool on allow-list executes normally
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-allowlist-allowed cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-allowlist-allowed-ok
|
|
|
|
Deny-list Blocks Tool On List
|
|
[Documentation] Tool on deny-list raises ToolAccessDeniedError
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-denylist-blocked cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-denylist-blocked-ok
|
|
|
|
Deny-list Takes Precedence Over Allow-list
|
|
[Documentation] Tool in both lists is denied (deny-list wins)
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-denylist-precedence cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-denylist-precedence-ok
|
|
|
|
Builtin Unrestricted Profile Allows Unsafe Tools
|
|
[Documentation] unrestricted built-in profile permits unsafe tool execution
|
|
${result}= Run Process ${PYTHON} ${HELPER} builtin-unrestricted cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} builtin-unrestricted-ok
|
|
|
|
Builtin Read-Only Profile Blocks Unsafe Tools
|
|
[Documentation] read-only built-in profile blocks unsafe tool execution
|
|
${result}= Run Process ${PYTHON} ${HELPER} builtin-read-only-blocks-unsafe cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} builtin-read-only-blocks-unsafe-ok
|