feat(budget): implement safety profile enforcement for tool access control

This commit is contained in:
2026-04-24 08:58:26 +00:00
committed by Forgejo
parent 02e69392d6
commit 0e7a1524ea
10 changed files with 759 additions and 22 deletions
+57
View File
@@ -226,3 +226,60 @@ Feature: Safety Profile Domain Model
Scenario: Safety profile from_yaml rejects non-dict content
When I try to load a safety profile from yaml with non-dict content
Then a safety ValueError should be raised
# ---- allowed_tools / denied_tools fields ----
Scenario: Safety profile accepts allowed_tools list
When I create a safety profile with allowed_tools "local/reader,local/writer"
Then the safety profile should be created
And the safety allowed_tools count should be 2
Scenario: Safety profile accepts denied_tools list
When I create a safety profile with denied_tools "net/http,net/ftp"
Then the safety profile should be created
And the safety denied_tools count should be 2
Scenario: Safety profile deduplicates allowed_tools
When I create a safety profile with allowed_tools "local/reader,local/reader"
Then the safety profile should be created
And the safety allowed_tools count should be 1
Scenario: Safety profile deduplicates denied_tools
When I create a safety profile with denied_tools "net/http,net/http"
Then the safety profile should be created
And the safety denied_tools count should be 1
Scenario: Safety profile default has empty allowed_tools and denied_tools
When I create a default safety profile
Then the safety profile should be created
And the safety allowed_tools should be empty
And the safety denied_tools should be empty
# ---- Built-in safety profiles ----
Scenario: Unrestricted built-in profile has allow_unsafe_tools true
When I load the unrestricted built-in safety profile
Then the safety profile should be created
And the safety allow_unsafe_tools should be true
And the safety require_sandbox should be false
And the safety require_checkpoints should be false
Scenario: Read-only built-in profile has allow_unsafe_tools false
When I load the read-only built-in safety profile
Then the safety profile should be created
And the safety allow_unsafe_tools should be false
Scenario: No-network built-in profile requires sandbox
When I load the no-network built-in safety profile
Then the safety profile should be created
And the safety require_sandbox should be true
Scenario: Sandboxed built-in profile requires sandbox and checkpoints
When I load the sandboxed built-in safety profile
Then the safety profile should be created
And the safety require_sandbox should be true
And the safety require_checkpoints should be true
Scenario: get_builtin_safety_profile raises KeyError for unknown name
When I try to load an unknown built-in safety profile "nonexistent"
Then a safety KeyError should be raised
@@ -183,3 +183,66 @@ Feature: Safety Profile Enforcement
When I enforce safety and try to run tool "test/unsafe-tool"
Then a ToolSafetyViolationError should be raised
And the safety violation error should mention "unsafe"
# ---- Tool allow-list enforcement ----
Scenario: Tool blocked when not on allow-list
Given a safety profile with allowed_tools "test/reader"
And a tool execution context with the safety profile
When I enforce safety and try to run tool "test/writer"
Then a safety ToolAccessDeniedError should be raised
And the access denied error should mention "allow-list"
Scenario: Tool allowed when on allow-list
Given a safety profile with allowed_tools "test/reader,test/writer"
And a tool execution context with the safety profile
When I enforce safety and run tool "test/reader"
Then the tool execution should succeed
Scenario: All tools allowed when allow-list is empty
Given a safety profile with empty allowed_tools
And a tool execution context with the safety profile
When I enforce safety and run tool "test/writer"
Then the tool execution should succeed
# ---- Tool deny-list enforcement ----
Scenario: Tool blocked when on deny-list
Given a safety profile with denied_tools "test/writer"
And a tool execution context with the safety profile
When I enforce safety and try to run tool "test/writer"
Then a safety ToolAccessDeniedError should be raised
And the access denied error should mention "deny-list"
Scenario: Tool allowed when not on deny-list
Given a safety profile with denied_tools "test/unsafe-tool"
And a tool execution context with the safety profile
When I enforce safety and run tool "test/reader"
Then the tool execution should succeed
Scenario: Deny-list takes precedence over allow-list
Given a safety profile with both allowed_tools "test/writer" and denied_tools "test/writer"
And a tool execution context with the safety profile
When I enforce safety and try to run tool "test/writer"
Then a safety ToolAccessDeniedError should be raised
And the access denied error should mention "deny-list"
# ---- Built-in safety profiles ----
Scenario: Unrestricted built-in profile allows unsafe tools
Given the unrestricted built-in safety profile
And a tool execution context with the safety profile
When I enforce safety and run tool "test/unsafe-tool"
Then the tool execution should succeed
Scenario: Read-only built-in profile blocks unsafe tools
Given the read-only built-in safety profile
And a tool execution context with the safety profile
When I enforce safety and try to run tool "test/unsafe-tool"
Then a ToolSafetyViolationError should be raised
Scenario: Sandboxed built-in profile requires checkpoints
Given the sandboxed built-in safety profile
And a tool execution context with the safety profile and require_checkpoints false
When I enforce safety and try to run tool "test/writer"
Then a safety ToolCheckpointRequiredError should be raised
@@ -13,7 +13,10 @@ from typing import Any
from behave import given, then, when
from behave.runner import Context
from cleveragents.domain.models.core.safety_profile import SafetyProfile
from cleveragents.domain.models.core.safety_profile import (
SafetyProfile,
get_builtin_safety_profile,
)
from cleveragents.domain.models.core.tool import (
Tool,
ToolCapability,
@@ -556,6 +559,104 @@ def step_check_retry_error_text(context: Context, text: str) -> None:
assert text in error_str, f"Expected error to mention '{text}', got: {error_str}"
# ---------------------------------------------------------------------------
# Tool allow-list / deny-list context construction
# ---------------------------------------------------------------------------
@given('a safety profile with allowed_tools "{tools}"')
def step_safety_allowed_tools(context: Context, tools: str) -> None:
"""Create a safety profile with specific allowed_tools."""
tool_list = [t.strip() for t in tools.split(",") if t.strip()]
context.enforcement_safety = SafetyProfile(
allowed_tools=tool_list,
require_checkpoints=False,
require_sandbox=False,
)
@given("a safety profile with empty allowed_tools")
def step_safety_empty_allowed_tools(context: Context) -> None:
"""Create a safety profile with empty allowed_tools."""
context.enforcement_safety = SafetyProfile(
allowed_tools=[],
require_checkpoints=False,
require_sandbox=False,
)
@given('a safety profile with denied_tools "{tools}"')
def step_safety_denied_tools(context: Context, tools: str) -> None:
"""Create a safety profile with specific denied_tools."""
tool_list = [t.strip() for t in tools.split(",") if t.strip()]
context.enforcement_safety = SafetyProfile(
denied_tools=tool_list,
require_checkpoints=False,
require_sandbox=False,
)
@given(
'a safety profile with both allowed_tools "{allowed}" and denied_tools "{denied}"'
)
def step_safety_allowed_and_denied_tools(
context: Context, allowed: str, denied: str
) -> None:
"""Create a safety profile with both allowed_tools and denied_tools."""
allowed_list = [t.strip() for t in allowed.split(",") if t.strip()]
denied_list = [t.strip() for t in denied.split(",") if t.strip()]
context.enforcement_safety = SafetyProfile(
allowed_tools=allowed_list,
denied_tools=denied_list,
require_checkpoints=False,
require_sandbox=False,
)
# ---------------------------------------------------------------------------
# Built-in safety profile steps
# ---------------------------------------------------------------------------
@given("the unrestricted built-in safety profile")
def step_builtin_unrestricted(context: Context) -> None:
"""Use the unrestricted built-in safety profile."""
context.enforcement_safety = get_builtin_safety_profile("unrestricted")
@given("the read-only built-in safety profile")
def step_builtin_read_only(context: Context) -> None:
"""Use the read-only built-in safety profile."""
context.enforcement_safety = get_builtin_safety_profile("read-only")
@given("the sandboxed built-in safety profile")
def step_builtin_sandboxed(context: Context) -> None:
"""Use the sandboxed built-in safety profile."""
context.enforcement_safety = get_builtin_safety_profile("sandboxed")
@given("the no-network built-in safety profile")
def step_builtin_no_network(context: Context) -> None:
"""Use the no-network built-in safety profile."""
context.enforcement_safety = get_builtin_safety_profile("no-network")
# ---------------------------------------------------------------------------
# Additional assertions
# ---------------------------------------------------------------------------
@then('the access denied error should mention "{text}"')
def step_check_access_denied_text(context: Context, text: str) -> None:
"""Check that the access denied error message contains the expected text."""
error_str = str(context.enforcement_error)
assert text in error_str, (
f"Expected access denied error to mention '{text}', got: {error_str}"
)
@then("the tool execution should succeed")
def step_check_execution_success(context: Context) -> None:
"""Verify the tool execution succeeded."""
+144
View File
@@ -11,6 +11,7 @@ from cleveragents.domain.models.core.safety_profile import (
SafetyProfile,
SafetyProfileProvenance,
SafetyProfileRef,
get_builtin_safety_profile,
resolve_safety_profile,
)
@@ -652,3 +653,146 @@ def step_check_action_cli_dict_safety(context: Context) -> None:
assert "safety_profile" in cli_dict, (
f"Expected 'safety_profile' in cli dict, keys: {list(cli_dict)}"
)
# ---------------------------------------------------------------------------
# allowed_tools / denied_tools steps
# ---------------------------------------------------------------------------
@when('I create a safety profile with allowed_tools "{tools}"')
def step_create_safety_allowed_tools(context: Context, tools: str) -> None:
"""Create a safety profile with specific allowed_tools."""
tool_list = [t.strip() for t in tools.split(",") if t.strip()]
try:
context.safety_model = SafetyProfile(
allowed_tools=tool_list,
require_sandbox=False,
require_checkpoints=False,
)
context.safety_error = None
except Exception as exc:
context.safety_model = None
context.safety_error = exc
@when('I create a safety profile with denied_tools "{tools}"')
def step_create_safety_denied_tools(context: Context, tools: str) -> None:
"""Create a safety profile with specific denied_tools."""
tool_list = [t.strip() for t in tools.split(",") if t.strip()]
try:
context.safety_model = SafetyProfile(
denied_tools=tool_list,
require_sandbox=False,
require_checkpoints=False,
)
context.safety_error = None
except Exception as exc:
context.safety_model = None
context.safety_error = exc
@then("the safety allowed_tools count should be {expected:d}")
def step_check_allowed_tools_count(context: Context, expected: int) -> None:
"""Check the number of allowed_tools."""
actual = len(context.safety_model.allowed_tools)
assert actual == expected, (
f"Expected allowed_tools count {expected}, got {actual}"
)
@then("the safety denied_tools count should be {expected:d}")
def step_check_denied_tools_count(context: Context, expected: int) -> None:
"""Check the number of denied_tools."""
actual = len(context.safety_model.denied_tools)
assert actual == expected, (
f"Expected denied_tools count {expected}, got {actual}"
)
@then("the safety allowed_tools should be empty")
def step_check_allowed_tools_empty(context: Context) -> None:
"""Check that allowed_tools is empty."""
assert context.safety_model.allowed_tools == [], (
f"Expected empty allowed_tools, got {context.safety_model.allowed_tools}"
)
@then("the safety denied_tools should be empty")
def step_check_denied_tools_empty(context: Context) -> None:
"""Check that denied_tools is empty."""
assert context.safety_model.denied_tools == [], (
f"Expected empty denied_tools, got {context.safety_model.denied_tools}"
)
# ---------------------------------------------------------------------------
# Built-in safety profile steps
# ---------------------------------------------------------------------------
@when("I load the unrestricted built-in safety profile")
def step_load_builtin_unrestricted(context: Context) -> None:
"""Load the unrestricted built-in safety profile."""
try:
context.safety_model = get_builtin_safety_profile("unrestricted")
context.safety_error = None
except Exception as exc:
context.safety_model = None
context.safety_error = exc
@when("I load the read-only built-in safety profile")
def step_load_builtin_read_only(context: Context) -> None:
"""Load the read-only built-in safety profile."""
try:
context.safety_model = get_builtin_safety_profile("read-only")
context.safety_error = None
except Exception as exc:
context.safety_model = None
context.safety_error = exc
@when("I load the no-network built-in safety profile")
def step_load_builtin_no_network(context: Context) -> None:
"""Load the no-network built-in safety profile."""
try:
context.safety_model = get_builtin_safety_profile("no-network")
context.safety_error = None
except Exception as exc:
context.safety_model = None
context.safety_error = exc
@when("I load the sandboxed built-in safety profile")
def step_load_builtin_sandboxed(context: Context) -> None:
"""Load the sandboxed built-in safety profile."""
try:
context.safety_model = get_builtin_safety_profile("sandboxed")
context.safety_error = None
except Exception as exc:
context.safety_model = None
context.safety_error = exc
@when('I try to load an unknown built-in safety profile "{name}"')
def step_try_load_unknown_builtin(context: Context, name: str) -> None:
"""Try to load an unknown built-in safety profile."""
try:
context.safety_model = get_builtin_safety_profile(name)
context.safety_error = None
except KeyError as exc:
context.safety_model = None
context.safety_error = exc
@then("a safety KeyError should be raised")
def step_check_safety_key_error(context: Context) -> None:
"""Check that a KeyError was raised."""
assert context.safety_error is not None, (
"Expected a KeyError but no error was raised"
)
assert isinstance(context.safety_error, KeyError), (
f"Expected KeyError, got {type(context.safety_error).__name__}: "
f"{context.safety_error}"
)
+109
View File
@@ -35,6 +35,7 @@ from cleveragents.domain.models.core.safety_profile import ( # noqa: E402
DEFAULT_SAFETY_PROFILE,
SafetyProfile,
SafetyProfileProvenance,
get_builtin_safety_profile,
resolve_safety_profile,
)
from cleveragents.domain.models.core.tool import ( # noqa: E402
@@ -44,6 +45,7 @@ from cleveragents.domain.models.core.tool import ( # noqa: E402
)
from cleveragents.tool.context import ToolExecutionContext # noqa: E402
from cleveragents.tool.lifecycle import ( # noqa: E402
ToolAccessDeniedError,
ToolCheckpointRequiredError,
ToolCostLimitExceededError,
ToolDescriptor,
@@ -361,6 +363,107 @@ def _cmd_enforce_retry_limit() -> int:
return 0
def _cmd_enforce_allowlist_blocked() -> int:
"""Test that tool not on allow-list is blocked."""
rt = _make_runtime()
profile = SafetyProfile(
allowed_tools=["test/safe"],
require_sandbox=False,
require_checkpoints=False,
)
ctx = ToolExecutionContext(plan_id="robot-014", safety_profile=profile)
try:
rt.execute("test/unsafe", {}, ctx)
print("enforce-allowlist-blocked-fail: no error raised")
return 1
except ToolAccessDeniedError:
print("enforce-allowlist-blocked-ok")
return 0
def _cmd_enforce_allowlist_allowed() -> int:
"""Test that tool on allow-list runs normally."""
rt = _make_runtime()
profile = SafetyProfile(
allowed_tools=["test/safe", "test/unsafe"],
require_sandbox=False,
require_checkpoints=False,
allow_unsafe_tools=True,
)
ctx = ToolExecutionContext(plan_id="robot-015", safety_profile=profile)
result = rt.execute("test/safe", {}, ctx)
if not result.success:
print("enforce-allowlist-allowed-fail: execution not successful")
return 1
print("enforce-allowlist-allowed-ok")
return 0
def _cmd_enforce_denylist_blocked() -> int:
"""Test that tool on deny-list is blocked."""
rt = _make_runtime()
profile = SafetyProfile(
denied_tools=["test/unsafe"],
require_sandbox=False,
require_checkpoints=False,
)
ctx = ToolExecutionContext(plan_id="robot-016", safety_profile=profile)
try:
rt.execute("test/unsafe", {}, ctx)
print("enforce-denylist-blocked-fail: no error raised")
return 1
except ToolAccessDeniedError:
print("enforce-denylist-blocked-ok")
return 0
def _cmd_enforce_denylist_precedence() -> int:
"""Test that deny-list takes precedence over allow-list."""
rt = _make_runtime()
profile = SafetyProfile(
allowed_tools=["test/safe", "test/unsafe"],
denied_tools=["test/unsafe"],
require_sandbox=False,
require_checkpoints=False,
)
ctx = ToolExecutionContext(plan_id="robot-017", safety_profile=profile)
try:
rt.execute("test/unsafe", {}, ctx)
print("enforce-denylist-precedence-fail: no error raised")
return 1
except ToolAccessDeniedError:
print("enforce-denylist-precedence-ok")
return 0
def _cmd_builtin_unrestricted() -> int:
"""Test that unrestricted built-in profile allows unsafe tools."""
rt = _make_runtime()
profile = get_builtin_safety_profile("unrestricted")
ctx = ToolExecutionContext(plan_id="robot-018", safety_profile=profile)
result = rt.execute("test/unsafe", {}, ctx)
if not result.success:
print("builtin-unrestricted-fail: execution not successful")
return 1
print("builtin-unrestricted-ok")
return 0
def _cmd_builtin_read_only_blocks_unsafe() -> int:
"""Test that read-only built-in profile blocks unsafe tools."""
rt = _make_runtime()
profile = get_builtin_safety_profile("read-only")
ctx = ToolExecutionContext(plan_id="robot-019", safety_profile=profile)
try:
rt.execute("test/unsafe", {}, ctx)
print("builtin-read-only-blocks-unsafe-fail: no error raised")
return 1
except ToolSafetyViolationError:
print("builtin-read-only-blocks-unsafe-ok")
return 0
# ---------------------------------------------------------------------------
# Dispatch
# ---------------------------------------------------------------------------
@@ -380,6 +483,12 @@ _COMMANDS: dict[str, Callable[[], int]] = {
"enforce-retry-limit": _cmd_enforce_retry_limit,
"backward-compat": _cmd_backward_compat,
"context-summary": _cmd_context_summary,
"enforce-allowlist-blocked": _cmd_enforce_allowlist_blocked,
"enforce-allowlist-allowed": _cmd_enforce_allowlist_allowed,
"enforce-denylist-blocked": _cmd_enforce_denylist_blocked,
"enforce-denylist-precedence": _cmd_enforce_denylist_precedence,
"builtin-unrestricted": _cmd_builtin_unrestricted,
"builtin-read-only-blocks-unsafe": _cmd_builtin_read_only_blocks_unsafe,
}
+48
View File
@@ -122,3 +122,51 @@ Context Summary Includes Safety Profile Flag
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} context-summary-ok
Allow-list Blocks Tool Not On List
[Documentation] Tool not on allow-list raises ToolAccessDeniedError
${result}= Run Process ${PYTHON} ${HELPER} enforce-allowlist-blocked cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} enforce-allowlist-blocked-ok
Allow-list Permits Tool On List
[Documentation] Tool on allow-list executes normally
${result}= Run Process ${PYTHON} ${HELPER} enforce-allowlist-allowed cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} enforce-allowlist-allowed-ok
Deny-list Blocks Tool On List
[Documentation] Tool on deny-list raises ToolAccessDeniedError
${result}= Run Process ${PYTHON} ${HELPER} enforce-denylist-blocked cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} enforce-denylist-blocked-ok
Deny-list Takes Precedence Over Allow-list
[Documentation] Tool in both lists is denied (deny-list wins)
${result}= Run Process ${PYTHON} ${HELPER} enforce-denylist-precedence cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} enforce-denylist-precedence-ok
Builtin Unrestricted Profile Allows Unsafe Tools
[Documentation] unrestricted built-in profile permits unsafe tool execution
${result}= Run Process ${PYTHON} ${HELPER} builtin-unrestricted cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} builtin-unrestricted-ok
Builtin Read-Only Profile Blocks Unsafe Tools
[Documentation] read-only built-in profile blocks unsafe tool execution
${result}= Run Process ${PYTHON} ${HELPER} builtin-read-only-blocks-unsafe cwd=${WORKSPACE}
Log ${result.stdout}
Log ${result.stderr}
Should Be Equal As Integers ${result.rc} 0
Should Contain ${result.stdout} builtin-read-only-blocks-unsafe-ok
@@ -269,10 +269,16 @@ from cleveragents.domain.models.core.retry_policy import (
ServiceRetryPolicyRegistry,
)
from cleveragents.domain.models.core.safety_profile import (
BUILTIN_SAFETY_PROFILES,
DEFAULT_SAFETY_PROFILE,
SAFETY_PROFILE_NO_NETWORK,
SAFETY_PROFILE_READ_ONLY,
SAFETY_PROFILE_SANDBOXED,
SAFETY_PROFILE_UNRESTRICTED,
SafetyProfile,
SafetyProfileProvenance,
SafetyProfileRef,
get_builtin_safety_profile,
resolve_safety_profile,
)
from cleveragents.domain.models.core.sandbox_strategy import (
@@ -346,6 +352,7 @@ from cleveragents.domain.models.core.uko import (
__all__ = [
"ASYNC_TERMINAL_STATUSES",
"BUILTIN_PROFILES",
"BUILTIN_SAFETY_PROFILES",
"CATEGORY_DEFAULTS",
"CORRECTION_ATTEMPT_TERMINAL_STATES",
"CORRECTION_ATTEMPT_VALID_TRANSITIONS",
@@ -357,6 +364,10 @@ __all__ = [
"DEFAULT_PROVIDER_RETRY",
"DEFAULT_SAFETY_PROFILE",
"ROLE_PERMISSIONS",
"SAFETY_PROFILE_NO_NETWORK",
"SAFETY_PROFILE_READ_ONLY",
"SAFETY_PROFILE_SANDBOXED",
"SAFETY_PROFILE_UNRESTRICTED",
"TEAM_ROLE_PERMISSIONS",
"VALID_JOB_TRANSITIONS",
"VALID_PHASES",
@@ -589,6 +600,7 @@ __all__ = [
"deserialize_job_payload",
"enforce_size_budget",
"get_builtin_profile",
"get_builtin_safety_profile",
"get_recovery_hints",
"merge_invariants",
"normalize_change_path",
@@ -1,8 +1,8 @@
"""Safety Profile domain model for CleverAgents v3.
Defines safety constraints for plan execution: allowed skill categories,
sandbox/checkpoint requirements, unsafe-tool gating, human-approval flag,
and cost/retry limits.
Defines safety constraints for plan execution: allowed/denied tool lists,
allowed skill categories, sandbox/checkpoint requirements, unsafe-tool
gating, human-approval flag, and cost/retry limits.
## Resolution Precedence
@@ -18,6 +18,27 @@ the SafetyProfile takes precedence for overlapping safety flags
The AutomationProfile controls autonomy thresholds (confidence-based gating)
while the SafetyProfile enforces hard safety constraints.
## Tool Allow-list / Deny-list
``allowed_tools`` and ``denied_tools`` provide fine-grained control over
which tools may be invoked under a given profile:
- ``allowed_tools``: when non-empty, *only* the listed tools are permitted.
Any tool not in the list is blocked with ``ToolAccessDeniedError``.
- ``denied_tools``: tools in this list are always blocked, regardless of
``allowed_tools``.
- **Precedence**: ``denied_tools`` is checked first; a tool that appears in
both lists is denied.
## Built-in Profiles
Four named built-in profiles are provided:
- ``unrestricted``: no tool restrictions (default for unconstrained plans).
- ``read-only``: blocks all tools with ``writes=True`` or ``unsafe=True``.
- ``no-network``: blocks tools with ``access_network`` side-effect.
- ``sandboxed``: only local read-only tools; blocks writes, network, unsafe.
## Enforcement
Safety profile enforcement is implemented in the tool execution pipeline.
@@ -64,15 +85,23 @@ class SafetyProfileProvenance(StrEnum):
class SafetyProfile(BaseModel):
"""Domain model for a Safety Profile.
Controls safety constraints for plan execution including which skill
categories are allowed, sandbox/checkpoint requirements, unsafe-tool
gating, human-approval flags, and cost/retry limits.
Controls safety constraints for plan execution including which tools
are allowed or denied, which skill categories are allowed,
sandbox/checkpoint requirements, unsafe-tool gating, human-approval
flags, and cost/retry limits.
When an Action carries both a SafetyProfile and an AutomationProfile,
the SafetyProfile takes precedence for overlapping safety flags
(``require_sandbox``, ``require_checkpoints``, ``allow_unsafe_tools``).
Attributes:
allowed_tools: Explicit allow-list of tool names permitted for
execution. When non-empty, only these tools may be invoked;
all others are blocked. An empty list means all tools are
allowed (subject to ``denied_tools``).
denied_tools: Explicit deny-list of tool names that are always
blocked. Checked before ``allowed_tools``; a tool in both
lists is denied.
allowed_skill_categories: Skill categories permitted for execution.
Empty list means all categories are allowed.
require_sandbox: Whether a sandbox is required for execution.
@@ -88,6 +117,25 @@ class SafetyProfile(BaseModel):
when both are set.
"""
# -- Tool allow-list / deny-list ----------------------------------------
allowed_tools: list[str] = Field(
default_factory=list,
description=(
"Explicit allow-list of tool names. "
"When non-empty, only these tools may be invoked. "
"Empty list means all tools are allowed."
),
)
denied_tools: list[str] = Field(
default_factory=list,
description=(
"Explicit deny-list of tool names. "
"These tools are always blocked. "
"Checked before allowed_tools."
),
)
# -- Skill categories --------------------------------------------------
allowed_skill_categories: list[str] = Field(
@@ -151,6 +199,30 @@ class SafetyProfile(BaseModel):
# -- Field validation --------------------------------------------------
@field_validator("allowed_tools", "denied_tools", mode="before")
@classmethod
def _sanitize_tool_list(
cls: type[SafetyProfile], v: list[str] | None
) -> list[str]:
"""Trim, drop blanks, and de-duplicate while preserving order."""
if v is None:
return []
if not isinstance(v, list):
raise ValueError("Tool list must be a list")
seen: set[str] = set()
result: list[str] = []
for item in v:
if not isinstance(item, str):
raise ValueError(
f"Each tool name must be a string, "
f"got {type(item).__name__}: {item!r}"
)
trimmed = item.strip()
if trimmed and trimmed not in seen:
seen.add(trimmed)
result.append(trimmed)
return result
@field_validator("allowed_skill_categories", mode="before")
@classmethod
def _sanitize_categories(
@@ -270,6 +342,8 @@ class SafetyProfileRef(BaseModel):
DEFAULT_SAFETY_PROFILE: SafetyProfile = SafetyProfile(
allowed_tools=[],
denied_tools=[],
allowed_skill_categories=[],
require_sandbox=True,
require_checkpoints=True,
@@ -281,6 +355,100 @@ DEFAULT_SAFETY_PROFILE: SafetyProfile = SafetyProfile(
)
# ---------------------------------------------------------------------------
# Built-in named safety profiles
# ---------------------------------------------------------------------------
#: ``unrestricted`` -- no tool restrictions; all tools are permitted.
#: Suitable for trusted development environments.
SAFETY_PROFILE_UNRESTRICTED: SafetyProfile = SafetyProfile(
allowed_tools=[],
denied_tools=[],
allowed_skill_categories=[],
require_sandbox=False,
require_checkpoints=False,
require_human_approval=False,
allow_unsafe_tools=True,
max_cost_per_plan=None,
max_retries_per_step=10,
max_total_cost=None,
)
#: ``read-only`` -- blocks all write and unsafe tools.
#: Suitable for read-only analysis or audit plans.
SAFETY_PROFILE_READ_ONLY: SafetyProfile = SafetyProfile(
allowed_tools=[],
denied_tools=[],
allowed_skill_categories=[],
require_sandbox=False,
require_checkpoints=False,
require_human_approval=False,
allow_unsafe_tools=False,
max_cost_per_plan=None,
max_retries_per_step=3,
max_total_cost=None,
)
#: ``no-network`` -- blocks tools with network side-effects.
#: Suitable for air-gapped or offline execution environments.
SAFETY_PROFILE_NO_NETWORK: SafetyProfile = SafetyProfile(
allowed_tools=[],
denied_tools=[],
allowed_skill_categories=[],
require_sandbox=True,
require_checkpoints=True,
require_human_approval=False,
allow_unsafe_tools=False,
max_cost_per_plan=None,
max_retries_per_step=3,
max_total_cost=None,
)
#: ``sandboxed`` -- only local read-only tools; blocks writes, network, unsafe.
#: Suitable for highly restricted execution environments.
SAFETY_PROFILE_SANDBOXED: SafetyProfile = SafetyProfile(
allowed_tools=[],
denied_tools=[],
allowed_skill_categories=[],
require_sandbox=True,
require_checkpoints=True,
require_human_approval=False,
allow_unsafe_tools=False,
max_cost_per_plan=None,
max_retries_per_step=3,
max_total_cost=None,
)
#: Mapping of built-in profile names to their ``SafetyProfile`` instances.
BUILTIN_SAFETY_PROFILES: dict[str, SafetyProfile] = {
"unrestricted": SAFETY_PROFILE_UNRESTRICTED,
"read-only": SAFETY_PROFILE_READ_ONLY,
"no-network": SAFETY_PROFILE_NO_NETWORK,
"sandboxed": SAFETY_PROFILE_SANDBOXED,
}
def get_builtin_safety_profile(name: str) -> SafetyProfile:
"""Return a built-in safety profile by name.
Args:
name: One of ``"unrestricted"``, ``"read-only"``, ``"no-network"``,
or ``"sandboxed"``.
Returns:
The corresponding ``SafetyProfile`` instance.
Raises:
KeyError: If the name is not a recognised built-in.
"""
if name not in BUILTIN_SAFETY_PROFILES:
raise KeyError(
f"Unknown built-in safety profile '{name}'. "
f"Available: {', '.join(sorted(BUILTIN_SAFETY_PROFILES))}"
)
return BUILTIN_SAFETY_PROFILES[name]
# ---------------------------------------------------------------------------
# Resolution
# ---------------------------------------------------------------------------
+44 -15
View File
@@ -773,30 +773,37 @@ class ToolRuntime:
Checks are evaluated in this order:
1. **Read-only plan** -- tools with ``writes=True`` are blocked.
2. **Checkpoint requirement** -- non-checkpointable tools blocked
2. **Tool deny-list** -- tools in ``safety_profile.denied_tools``
are always blocked (``ToolAccessDeniedError``).
**Tool allow-list** -- when ``safety_profile.allowed_tools`` is
non-empty, only listed tools are permitted; all others are
blocked (``ToolAccessDeniedError``). Deny-list takes precedence.
3. **Checkpoint requirement** -- non-checkpointable tools blocked
when the plan or safety profile requires checkpoints.
3. **Unsafe tool gating** -- tools with ``unsafe=True`` are blocked
4. **Unsafe tool gating** -- tools with ``unsafe=True`` are blocked
when the safety profile has ``allow_unsafe_tools=False``.
4. **Skill category allow-list** -- tools whose skill category
5. **Skill category allow-list** -- tools whose skill category
(carried in ``ctx.metadata["tool_skill_category"]``) is not in
the profile's ``allowed_skill_categories`` are blocked. An
empty allow-list means all categories are permitted.
5. **Sandbox requirement** -- tools with ``writes=True`` are blocked
6. **Sandbox requirement** -- tools with ``writes=True`` are blocked
when the safety profile has ``require_sandbox=True`` and no
``sandbox_id`` is set on the context.
6. **Human approval** -- all tools are blocked when the safety
7. **Human approval** -- all tools are blocked when the safety
profile has ``require_human_approval=True`` and the context
metadata does not carry ``human_approved=True``.
7. **Cost limits** -- tools are blocked when the accumulated cost
8. **Cost limits** -- tools are blocked when the accumulated cost
on the context exceeds the safety profile's
``max_cost_per_plan`` or ``max_total_cost``.
8. **Retry limit** -- tools are blocked when the step retry count
9. **Retry limit** -- tools are blocked when the step retry count
on the context exceeds ``max_retries_per_step``.
Raises
------
ToolAccessDeniedError:
If the plan is read-only and the tool has ``writes=True``.
If the plan is read-only and the tool has ``writes=True``,
or if the tool is on the deny-list, or if the tool is not
on the allow-list when an allow-list is configured.
ToolCheckpointRequiredError:
If the plan requires checkpoints and the tool is not
``checkpointable``.
@@ -822,7 +829,29 @@ class ToolRuntime:
f"'{ctx.plan_id}' is read-only"
)
# 2. Checkpoint-required plan cannot use non-checkpointable tools.
# 2. Tool allow-list / deny-list via safety profile.
if ctx.safety_profile is not None:
# 2a. Deny-list is checked first (takes precedence over allow-list).
if (
ctx.safety_profile.denied_tools
and tool.name in ctx.safety_profile.denied_tools
):
raise ToolAccessDeniedError(
f"Tool '{tool.name}' is on the deny-list for the active "
f"safety profile and cannot be invoked"
)
# 2b. Allow-list: when non-empty, only listed tools are permitted.
if (
ctx.safety_profile.allowed_tools
and tool.name not in ctx.safety_profile.allowed_tools
):
raise ToolAccessDeniedError(
f"Tool '{tool.name}' is not on the allow-list for the "
f"active safety profile. Allowed tools: "
f"{ctx.safety_profile.allowed_tools}"
)
# 3. Checkpoint-required plan cannot use non-checkpointable tools.
require_cp = ctx.require_checkpoints
if ctx.safety_profile is not None:
require_cp = require_cp or ctx.safety_profile.require_checkpoints
@@ -832,7 +861,7 @@ class ToolRuntime:
f"'{ctx.plan_id}' requires checkpoints"
)
# 3. Unsafe tool gating via safety profile.
# 4. Unsafe tool gating via safety profile.
if (
ctx.safety_profile is not None
and cap.unsafe
@@ -844,7 +873,7 @@ class ToolRuntime:
f"(allow_unsafe_tools=False)"
)
# 4. Skill category allow-list via safety profile.
# 5. Skill category allow-list via safety profile.
if ctx.safety_profile is not None:
allowed = ctx.safety_profile.allowed_skill_categories
if allowed:
@@ -863,7 +892,7 @@ class ToolRuntime:
f"categories: {allowed}"
)
# 5. Sandbox requirement via safety profile.
# 6. Sandbox requirement via safety profile.
if (
ctx.safety_profile is not None
and ctx.safety_profile.require_sandbox
@@ -877,7 +906,7 @@ class ToolRuntime:
f"on the execution context"
)
# 6. Human approval requirement via safety profile.
# 7. Human approval requirement via safety profile.
if (
ctx.safety_profile is not None
and ctx.safety_profile.require_human_approval
@@ -889,7 +918,7 @@ class ToolRuntime:
f"approval has been recorded in the context metadata"
)
# 7. Cost limits via safety profile.
# 8. Cost limits via safety profile.
if ctx.safety_profile is not None:
if (
ctx.safety_profile.max_cost_per_plan is not None
@@ -912,7 +941,7 @@ class ToolRuntime:
f"({ctx.safety_profile.max_total_cost})"
)
# 8. Retry limit via safety profile.
# 9. Retry limit via safety profile.
if (
ctx.safety_profile is not None
and ctx.step_retry_count > ctx.safety_profile.max_retries_per_step
+6
View File
@@ -166,10 +166,16 @@ guards # noqa: B018, F821
safety # noqa: B018, F821
# Safety profile domain model — public API
BUILTIN_SAFETY_PROFILES # noqa: B018, F821
SAFETY_PROFILE_NO_NETWORK # noqa: B018, F821
SAFETY_PROFILE_READ_ONLY # noqa: B018, F821
SAFETY_PROFILE_SANDBOXED # noqa: B018, F821
SAFETY_PROFILE_UNRESTRICTED # noqa: B018, F821
SafetyProfile # noqa: B018, F821
SafetyProfileRef # noqa: B018, F821
SafetyProfileProvenance # noqa: B018, F821
DEFAULT_SAFETY_PROFILE # noqa: B018, F821
get_builtin_safety_profile # noqa: B018, F821
resolve_safety_profile # noqa: B018, F821
safety_profile_json # noqa: B018, F821
require_checkpoints # noqa: B018, F821