cleveragents-core/docs/reference/safety_profiles.md

Safety Profiles

Safety profiles control hard safety constraints for plan execution. Unlike autonomy thresholds (which are confidence-dependent), safety constraints are binary invariants enforced regardless of the system's confidence level.

Overview

A SafetyProfile is a composed sub-model of AutomationProfile that groups all hard safety constraints. It can also be attached directly to an Action when only safety constraints are needed without full autonomy thresholds.

Fields

Field	Type	Default	Description
require_sandbox	bool	true	Require sandbox isolation for execution
require_checkpoints	bool	true	Require checkpointing before writes
allow_unsafe_tools	bool	false	Allow tools marked unsafe
require_human_approval	bool	false	Require human approval per action step
allowed_skill_categories	list[str]	[]	Skill categories allowed (empty = all)
max_cost_per_plan	float | None	None	Max cost in USD per plan
max_retries_per_step	int	3	Max retries per action step (0-100)
max_total_cost	float | None	None	Max total cost across all plans

Resolution Precedence

Safety profiles are resolved at plan use time using this precedence (highest to lowest):

1. Plan-level -- set via --automation-profile on agents plan use
2. Action-level -- set on the action via --automation-profile
3. Project-level -- set via agents config set core.automation-profile
4. Global-level -- set via agents config set core.automation-profile

Once resolved, the profile is locked to the plan -- subsequent changes to project or global profiles do not affect running plans.

When all levels are None, the DEFAULT_SAFETY_PROFILE is returned with GLOBAL provenance.

from cleveragents.domain.models.core.safety_profile import (
    SafetyProfile,
    resolve_safety_profile,
)

plan_safety = SafetyProfile(allow_unsafe_tools=True, require_sandbox=False)
resolved, provenance = resolve_safety_profile(plan_profile=plan_safety)
# provenance == SafetyProfileProvenance.PLAN

Enforcement

Safety profile constraints are enforced at tool activation and execution time in ToolRuntime._enforce_capabilities(). The checks are evaluated in order; the first failing check raises the corresponding error.

1. Read-Only Plans

Read-only plan enforcement (plan_read_only) is independent of the safety profile and always applies: tools with writes=True are blocked with ToolAccessDeniedError.

2. Checkpoint Requirements

When require_checkpoints=True (from either the context flag or the safety profile), tools without capability.checkpointable=True are blocked with a ToolCheckpointRequiredError.

3. Unsafe Tool Gating

Tools with capability.unsafe=True are blocked unless the safety profile has allow_unsafe_tools=True. When blocked, a ToolSafetyViolationError is raised.

4. Skill Category Allow-List

When allowed_skill_categories is non-empty, only tools belonging to a listed category may execute. The tool's skill category is carried in ToolExecutionContext.metadata["tool_skill_category"]. An empty list means all categories are allowed. If the metadata key is missing when the allow-list is non-empty, a ToolSafetyViolationError is raised with a clear message indicating the missing metadata.

5. Sandbox Requirement

When require_sandbox=True, tools with capability.writes=True require a sandbox_id to be set on the ToolExecutionContext. If no sandbox is available, a ToolSandboxRequiredError is raised. Read-only tools are not affected by this check since they do not modify resources.

6. Human Approval

When require_human_approval=True, all tool executions require that ToolExecutionContext.metadata["human_approved"] is set to True. If approval has not been recorded, a ToolHumanApprovalRequiredError is raised. The higher-level orchestrator is responsible for obtaining approval and setting this metadata before tool execution.

7. Cost Limits

When max_cost_per_plan is set, tools are blocked with a ToolCostLimitExceededError if ToolExecutionContext.accumulated_cost meets or exceeds the limit. Similarly, max_total_cost is checked against ToolExecutionContext.total_accumulated_cost. The caller is responsible for updating these cost fields after each tool execution.

8. Retry Limit

When max_retries_per_step is set, tools are blocked with a ToolRetryLimitExceededError if ToolExecutionContext.step_retry_count exceeds the limit. The caller is responsible for incrementing the retry count for each retry attempt.

Error Hierarchy

ToolRuntimeError
  |-- ToolAccessDeniedError            (read-only plan violation)
  |-- ToolCheckpointRequiredError      (checkpoint requirement violation)
  |-- ToolSafetyViolationError         (unsafe tool or category violation)
  |-- ToolSandboxRequiredError         (sandbox requirement violation)
  |-- ToolHumanApprovalRequiredError   (human approval not granted)
  |-- ToolCostLimitExceededError       (cost limit exceeded)
  |-- ToolRetryLimitExceededError      (retry limit exceeded)
  |-- ToolNotActivatedError
  |-- ToolActivationError
  |-- ToolExecutionError
  +-- ToolDeactivationError

Backward Compatibility

When no SafetyProfile is attached to the ToolExecutionContext (i.e., safety_profile=None), the safety-profile-specific checks (unsafe tool gating, skill category, sandbox requirement, human approval, cost limits, and retry limits) are skipped. Only the pre-existing plan_read_only and require_checkpoints context flags apply.

Default Profile

The DEFAULT_SAFETY_PROFILE constant provides sensible defaults matching the specification:

DEFAULT_SAFETY_PROFILE = SafetyProfile(
    allowed_skill_categories=[],
    require_sandbox=True,
    require_checkpoints=True,
    require_human_approval=False,
    allow_unsafe_tools=False,
    max_cost_per_plan=None,
    max_retries_per_step=3,
    max_total_cost=None,
)

Related

• ADR-041: Safety Profile Extraction
• Specification: Automation Profiles -- Profile Precedence section
• Specification: Tool Capability Metadata -- unsafe field