Luis Mendes
b4b96d213c
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 15s
CI / build (pull_request) Successful in 15s
CI / quality (pull_request) Successful in 29s
CI / security (pull_request) Successful in 34s
CI / typecheck (pull_request) Successful in 39s
CI / unit_tests (pull_request) Successful in 2m19s
CI / docker (pull_request) Successful in 40s
CI / integration_tests (pull_request) Successful in 3m1s
CI / coverage (pull_request) Successful in 7m20s
CI / lint (push) Successful in 13s
CI / quality (push) Successful in 16s
CI / build (push) Successful in 18s
CI / security (push) Successful in 31s
CI / typecheck (push) Successful in 35s
CI / benchmark-regression (push) Has been skipped
CI / unit_tests (push) Successful in 2m10s
CI / docker (push) Successful in 39s
CI / integration_tests (push) Successful in 3m4s
CI / coverage (push) Successful in 4m45s
CI / benchmark-publish (push) Successful in 14m51s
CI / benchmark-regression (pull_request) Successful in 26m49s
Implement safety profile resolution and enforcement in the tool
execution pipeline, replacing the NotImplementedError stub with
working precedence logic and runtime safety checks.
Core changes:
- resolve_safety_profile() now resolves plan > action > project >
global precedence, returning the highest-priority non-None profile
(or DEFAULT_SAFETY_PROFILE with GLOBAL provenance when all None)
- ToolExecutionContext gains an optional safety_profile field
- ToolRuntime._enforce_capabilities() extended with three new checks:
* Unsafe tool gating: blocks tools with unsafe=True when profile
has allow_unsafe_tools=False (ToolSafetyViolationError)
* Skill category allow-list: blocks tools whose skill category
is not in allowed_skill_categories (ToolSafetyViolationError)
* Checkpoint requirement: OR-combines ctx.require_checkpoints
with safety_profile.require_checkpoints
- New ToolSafetyViolationError in tool error hierarchy
Test coverage:
- 30 updated BDD scenarios in safety_profile.feature (resolve
precedence replaces NotImplementedError stub test)
- 24 new BDD scenarios in safety_profile_enforcement.feature
- 9 Robot Framework integration smoke tests
- 4 ASV benchmark suites (construction, serialization, resolution,
provenance enum)
All nox sessions pass (typecheck 0 errors, unit_tests 7735 scenarios
0 failures, coverage 97%, integration_tests 9/9 passed, benchmarks
complete).
ISSUES CLOSED: #345
125 lines
6.0 KiB
Plaintext
125 lines
6.0 KiB
Plaintext
*** Settings ***
|
|
Documentation Safety Profile Enforcement integration smoke tests
|
|
... Verifies resolve_safety_profile precedence and
|
|
... ToolRuntime safety enforcement (unsafe tool gating,
|
|
... skill category allow-lists, checkpoint requirements).
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_safety_profile_enforcement.py
|
|
|
|
*** Test Cases ***
|
|
Resolve Safety Profile Precedence Plan Over Action
|
|
[Documentation] Plan-level profile takes precedence over action-level
|
|
${result}= Run Process ${PYTHON} ${HELPER} resolve-precedence cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} resolve-precedence-ok
|
|
|
|
Resolve Safety Profile Default When All None
|
|
[Documentation] All-None resolution returns DEFAULT_SAFETY_PROFILE with GLOBAL provenance
|
|
${result}= Run Process ${PYTHON} ${HELPER} resolve-default cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} resolve-default-ok
|
|
|
|
Unsafe Tool Blocked By Safety Profile
|
|
[Documentation] Unsafe tool raises ToolSafetyViolationError when profile forbids
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-unsafe-blocked cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-unsafe-blocked-ok
|
|
|
|
Unsafe Tool Allowed By Safety Profile
|
|
[Documentation] Unsafe tool executes when profile permits allow_unsafe_tools
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-unsafe-allowed cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-unsafe-allowed-ok
|
|
|
|
Skill Category Blocked By Safety Profile
|
|
[Documentation] Tool in wrong skill category is blocked by allow-list
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-category-blocked cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-category-blocked-ok
|
|
|
|
Skill Category Allowed By Safety Profile
|
|
[Documentation] Tool in correct skill category runs normally
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-category-allowed cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-category-allowed-ok
|
|
|
|
Checkpoint Required By Safety Profile
|
|
[Documentation] Non-checkpointable tool blocked when profile requires checkpoints
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-checkpoint cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-checkpoint-ok
|
|
|
|
Backward Compatibility Without Safety Profile
|
|
[Documentation] Unsafe tool runs when no safety profile on context
|
|
${result}= Run Process ${PYTHON} ${HELPER} backward-compat cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} backward-compat-ok
|
|
|
|
Sandbox Required Blocks Writing Tool Without Sandbox Id
|
|
[Documentation] Writing tool raises ToolSandboxRequiredError when sandbox required but missing
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-sandbox-blocked cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-sandbox-blocked-ok
|
|
|
|
Sandbox Required Allows Writing Tool With Sandbox Id
|
|
[Documentation] Writing tool executes when sandbox required and sandbox_id set
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-sandbox-allowed cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-sandbox-allowed-ok
|
|
|
|
Human Approval Required Blocks Tool Without Approval
|
|
[Documentation] Tool blocked when require_human_approval=True and no approval
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-human-approval cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-human-approval-ok
|
|
|
|
Cost Limit Blocks Tool When Exceeded
|
|
[Documentation] Tool blocked when accumulated cost exceeds max_cost_per_plan
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-cost-limit cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-cost-limit-ok
|
|
|
|
Retry Limit Blocks Tool When Exceeded
|
|
[Documentation] Tool blocked when step retry count exceeds max_retries_per_step
|
|
${result}= Run Process ${PYTHON} ${HELPER} enforce-retry-limit cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} enforce-retry-limit-ok
|
|
|
|
Context Summary Includes Safety Profile Flag
|
|
[Documentation] ToolExecutionContext.as_summary includes has_safety_profile
|
|
${result}= Run Process ${PYTHON} ${HELPER} context-summary cwd=${WORKSPACE}
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} context-summary-ok
|