cleveragents-core/docs/reference/safety_profile.md

# Safety Profile

## Overview

A **Safety Profile** defines constraints on plan execution to enforce security
and cost boundaries. It controls which skill categories are permitted, whether
sandboxes and checkpoints are required, whether unsafe tools may be invoked,
whether human approval is needed, and sets cost and retry limits.

When an Action carries both a `safety_profile` and an `automation_profile`,
the **SafetyProfile takes precedence** for overlapping safety flags
(`require_sandbox`, `require_checkpoints`, `allow_unsafe_tools`). The
AutomationProfile controls autonomy thresholds (confidence-based gating)
while the SafetyProfile enforces hard safety constraints.

## Schema

| Field                       | Type            | Default | Description                                              |
|-----------------------------|-----------------|---------|----------------------------------------------------------|
| `allowed_skill_categories`  | `list[str]`     | `[]`    | Permitted skill categories (empty = all)                 |
| `require_sandbox`           | `bool`          | `True`  | Require sandbox for execution                            |
| `require_checkpoints`       | `bool`          | `True`  | Require checkpoints during execution                     |
| `require_human_approval`    | `bool`          | `False` | Require human approval per action step                   |
| `allow_unsafe_tools`        | `bool`          | `False` | Allow tools marked as unsafe to be invoked               |
| `max_cost_per_plan`         | `float \| None` | `None`  | Max cost in USD per plan (None = unlimited)              |
| `max_retries_per_step`      | `int`           | `3`     | Max retry attempts per step (0–100)                      |
| `max_total_cost`            | `float \| None` | `None`  | Max total cost in USD (None = unlimited)                 |

## Default Profile

The `DEFAULT_SAFETY_PROFILE` constant provides sensible defaults:

```python
DEFAULT_SAFETY_PROFILE = SafetyProfile(
    allowed_skill_categories=[],    # all categories allowed
    require_sandbox=True,
    require_checkpoints=True,
    require_human_approval=False,
    allow_unsafe_tools=False,
    max_cost_per_plan=None,         # no limit
    max_retries_per_step=3,
    max_total_cost=None,            # no limit
)
```

The default profile is **immutable** (`frozen=True`). Attempting to modify
fields after construction raises a `ValidationError`.

## Cross-Field Validation

The following cross-field constraints are enforced:

- **`max_cost_per_plan <= max_total_cost`**: When both cost limits are set,
  the per-plan cost must not exceed the total cost. Violating this raises a
  `ValidationError`.

## Input Validation

- `allowed_skill_categories`: duplicates removed, blanks stripped. Non-list
  input raises `ValueError`. Non-string items in the list raise `ValueError`.
- `max_cost_per_plan`: must be >= 0.0 if set
- `max_total_cost`: must be >= 0.0 if set
- `max_retries_per_step`: must be 0–100

## Precedence over AutomationProfile

When an Action carries both a `SafetyProfile` and an `AutomationProfile`,
the overlapping safety flags are resolved as follows:

| Flag                  | Wins          | Rationale                                  |
|-----------------------|---------------|--------------------------------------------|
| `require_sandbox`     | SafetyProfile | Hard safety constraint                     |
| `require_checkpoints` | SafetyProfile | Hard safety constraint                     |
| `allow_unsafe_tools`  | SafetyProfile | Hard safety constraint                     |

Non-overlapping fields from each profile apply independently. The
AutomationProfile governs autonomy thresholds (e.g., confidence-based
gating for auto-execution), while the SafetyProfile enforces hard safety
boundaries.

## YAML Configuration

Safety profiles can be loaded from YAML:

```yaml
require_sandbox: false
require_checkpoints: true
require_human_approval: true
allow_unsafe_tools: false
max_cost_per_plan: 50.0
max_retries_per_step: 5
max_total_cost: 200.0
allowed_skill_categories:
  - code
  - test
  - deploy
```

Load with:

```python
from cleveragents.domain.models.core.safety_profile import SafetyProfile

profile = SafetyProfile.from_yaml("path/to/safety.yaml")
# or from a dict:
profile = SafetyProfile.from_config(config_dict)
```

### YAML Error Handling

- **File not found**: `from_yaml()` raises `FileNotFoundError`
- **Non-dict content**: `from_yaml()` raises `ValueError` if the YAML root
  is not a mapping

## Action Integration

Actions can include an optional `safety_profile` field:

```yaml
name: local/secure-deploy
description: Deploy with safety constraints
definition_of_done: Deployment verified
strategy_actor: local/default
execution_actor: local/default
safety_profile:
  require_sandbox: true
  require_checkpoints: true
  require_human_approval: true
  allow_unsafe_tools: false
  max_cost_per_plan: 100.0
  allowed_skill_categories:
    - deploy
    - test
```

## Resolution Precedence

Safety profiles are resolved at plan-use time following:

```
plan > action > project > global
```

The resolved profile is captured in a `SafetyProfileRef` with provenance:

| Provenance | Description                           |
|------------|---------------------------------------|
| `PLAN`     | Profile set directly on the plan      |
| `ACTION`   | Inherited from the action template    |
| `PROJECT`  | Inherited from the project            |
| `GLOBAL`   | Global default profile                |

## Stub Enforcement (Local Mode)

In the current local-mode implementation, `resolve_safety_profile()` raises
`NotImplementedError`. This signals that real enforcement is deferred to a
future server-mode release. The resolution logic is:

```python
from cleveragents.domain.models.core.safety_profile import resolve_safety_profile

try:
    profile, provenance = resolve_safety_profile(
        plan_profile=plan_sp,
        action_profile=action_sp,
        project_profile=project_sp,
        global_profile=global_sp,
    )
except NotImplementedError:
    # Expected in local mode — enforcement not yet available
    pass
```

## Persistence

Safety profiles are stored as JSON in the `safety_profile_json` column on the
`actions` table. Serialization uses `model_dump_json()` and deserialization
uses `SafetyProfile.model_validate()`.

The column is added by migration `c4_001_safety_profile_column`.

## Specification Reference

Based on `docs/specification.md` sections "Automation Profiles" and
"Guardrails".