cleveragents-core/docs/reference/safety_profiles.md

# Safety Profiles

Safety profiles control hard safety constraints for plan execution. Unlike
autonomy thresholds (which are confidence-dependent), safety constraints are
binary invariants enforced regardless of the system's confidence level.

## Overview

A `SafetyProfile` is a composed sub-model of `AutomationProfile` that groups
all hard safety constraints. It can also be attached directly to an `Action`
when only safety constraints are needed without full autonomy thresholds.

## Fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `require_sandbox` | `bool` | `true` | Require sandbox isolation for execution |
| `require_checkpoints` | `bool` | `true` | Require checkpointing before writes |
| `allow_unsafe_tools` | `bool` | `false` | Allow tools marked `unsafe` |
| `require_human_approval` | `bool` | `false` | Require human approval per action step |
| `allowed_skill_categories` | `list[str]` | `[]` | Skill categories allowed (empty = all) |
| `max_cost_per_plan` | `float \| None` | `None` | Max cost in USD per plan |
| `max_retries_per_step` | `int` | `3` | Max retries per action step (0-100) |
| `max_total_cost` | `float \| None` | `None` | Max total cost across all plans |

## Resolution Precedence

Safety profiles are resolved at `plan use` time using this precedence
(highest to lowest):

1. **Plan-level** -- set via `--automation-profile` on `agents plan use`
2. **Action-level** -- set on the action via `--automation-profile`
3. **Project-level** -- set via `agents config set core.automation-profile`
4. **Global-level** -- set via `agents config set core.automation-profile`

Once resolved, the profile is **locked to the plan** -- subsequent changes
to project or global profiles do not affect running plans.

When all levels are `None`, the `DEFAULT_SAFETY_PROFILE` is returned with
`GLOBAL` provenance.

```python
from cleveragents.domain.models.core.safety_profile import (
    SafetyProfile,
    resolve_safety_profile,
)

plan_safety = SafetyProfile(allow_unsafe_tools=True, require_sandbox=False)
resolved, provenance = resolve_safety_profile(plan_profile=plan_safety)
# provenance == SafetyProfileProvenance.PLAN
```

## Enforcement

Safety profile constraints are enforced at tool activation and execution
time in `ToolRuntime._enforce_capabilities()`. The checks are evaluated
in order; the first failing check raises the corresponding error.

### 1. Read-Only Plans

Read-only plan enforcement (`plan_read_only`) is independent of the safety
profile and always applies: tools with `writes=True` are blocked with
`ToolAccessDeniedError`.

### 2. Checkpoint Requirements

When `require_checkpoints=True` (from either the context flag or the safety
profile), tools without `capability.checkpointable=True` are blocked with
a `ToolCheckpointRequiredError`.

### 3. Unsafe Tool Gating

Tools with `capability.unsafe=True` are blocked unless the safety profile
has `allow_unsafe_tools=True`. When blocked, a `ToolSafetyViolationError`
is raised.

### 4. Skill Category Allow-List

When `allowed_skill_categories` is non-empty, only tools belonging to a
listed category may execute. The tool's skill category is carried in
`ToolExecutionContext.metadata["tool_skill_category"]`. An empty list means
all categories are allowed. If the metadata key is missing when the
allow-list is non-empty, a `ToolSafetyViolationError` is raised with a
clear message indicating the missing metadata.

### 5. Sandbox Requirement

When `require_sandbox=True`, tools with `capability.writes=True` require a
`sandbox_id` to be set on the `ToolExecutionContext`. If no sandbox is
available, a `ToolSandboxRequiredError` is raised. Read-only tools are not
affected by this check since they do not modify resources.

### 6. Human Approval

When `require_human_approval=True`, all tool executions require that
`ToolExecutionContext.metadata["human_approved"]` is set to `True`. If
approval has not been recorded, a `ToolHumanApprovalRequiredError` is
raised. The higher-level orchestrator is responsible for obtaining approval
and setting this metadata before tool execution.

### 7. Cost Limits

When `max_cost_per_plan` is set, tools are blocked with a
`ToolCostLimitExceededError` if `ToolExecutionContext.accumulated_cost`
meets or exceeds the limit. Similarly, `max_total_cost` is checked against
`ToolExecutionContext.total_accumulated_cost`. The caller is responsible
for updating these cost fields after each tool execution.

### 8. Retry Limit

When `max_retries_per_step` is set, tools are blocked with a
`ToolRetryLimitExceededError` if `ToolExecutionContext.step_retry_count`
exceeds the limit. The caller is responsible for incrementing the retry
count for each retry attempt.

## Error Hierarchy

```
ToolRuntimeError
  |-- ToolAccessDeniedError            (read-only plan violation)
  |-- ToolCheckpointRequiredError      (checkpoint requirement violation)
  |-- ToolSafetyViolationError         (unsafe tool or category violation)
  |-- ToolSandboxRequiredError         (sandbox requirement violation)
  |-- ToolHumanApprovalRequiredError   (human approval not granted)
  |-- ToolCostLimitExceededError       (cost limit exceeded)
  |-- ToolRetryLimitExceededError      (retry limit exceeded)
  |-- ToolNotActivatedError
  |-- ToolActivationError
  |-- ToolExecutionError
  +-- ToolDeactivationError
```

## Backward Compatibility

When no `SafetyProfile` is attached to the `ToolExecutionContext` (i.e.,
`safety_profile=None`), the safety-profile-specific checks (unsafe tool
gating, skill category, sandbox requirement, human approval, cost limits,
and retry limits) are skipped. Only the pre-existing `plan_read_only` and
`require_checkpoints` context flags apply.

## Default Profile

The `DEFAULT_SAFETY_PROFILE` constant provides sensible defaults matching
the specification:

```python
DEFAULT_SAFETY_PROFILE = SafetyProfile(
    allowed_skill_categories=[],
    require_sandbox=True,
    require_checkpoints=True,
    require_human_approval=False,
    allow_unsafe_tools=False,
    max_cost_per_plan=None,
    max_retries_per_step=3,
    max_total_cost=None,
)
```

## Related

- [ADR-041: Safety Profile Extraction](../adr/ADR-041-safety-profile-extraction.md)
- [Specification: Automation Profiles](../specification.md) -- Profile Precedence section
- [Specification: Tool Capability Metadata](../specification.md) -- `unsafe` field