cleveragents/cleveragents-core

Fork 3

UAT: Automation profile YAML schema mismatch — spec defines `require_sandbox`, `require_checkpoints`, `allow_unsafe_tools` as top-level fields but implementation nests them under `safety:` #4901

New issue

Open

opened 2026-04-08 20:17:37 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-08 20:17:37 +00:00

Owner

Bug Report

Feature Area: Configuration System — Automation Profile YAML Schema
Severity: Medium — custom automation profile YAML files written per the spec fail validation
Found by: UAT tester, code analysis

What Was Tested

The automation profile YAML configuration schema was compared between the spec's JSON Schema definition and the AutomationProfile Pydantic model.

Expected Behavior (from spec)

Per docs/specification.md §Automation Profile Configuration Files, JSON Schema (lines 35741–35856), the safety flags are top-level fields in the YAML:

# Per spec — safety flags at top level
name: local/careful-auto
description: "Careful automation profile"
decompose_task: 0.0
create_tool: 0.0
select_tool: 1.0
edit_code: 0.0
execute_command: 0.0
create_file: 0.0
delete_content: 1.0
access_network: 1.0
install_dependency: 0.0
modify_config: 0.0
approve_plan: 0.0
require_sandbox: true        # ← top-level field per spec
require_checkpoints: true    # ← top-level field per spec
allow_unsafe_tools: false    # ← top-level field per spec

The spec's JSON Schema (line 35824) explicitly defines these as top-level properties:

"require_sandbox": { "type": "boolean", ... },
"require_checkpoints": { "type": "boolean", ... },
"allow_unsafe_tools": { "type": "boolean", ... }

And the spec's required array (line 35837) includes all three as required top-level fields.

Actual Behavior

The AutomationProfile Pydantic model in src/cleveragents/domain/models/core/automation_profile.py (lines 207–215) nests these under a safety: SafetyProfile sub-model:

safety: SafetyProfile = Field(
    default=DEFAULT_SAFETY_PROFILE,
    description="Composed safety profile controlling sandbox, checkpoint, and tool constraints.",
)

So the implementation expects:

name: local/careful-auto
# ... threshold fields ...
safety:
  require_sandbox: true        # ← nested under safety:
  require_checkpoints: true    # ← nested under safety:
  allow_unsafe_tools: false    # ← nested under safety:

There is no model_validator or flattening logic to accept the spec's top-level format. A user writing a YAML file per the spec's documented schema will get a validation error because require_sandbox is not a recognized top-level field (the model has extra="forbid").

Code Location

src/cleveragents/domain/models/core/automation_profile.py, lines 207–215 (safety field)
src/cleveragents/domain/models/core/automation_profile.py, line 414 (extra="forbid")

Steps to Reproduce

from cleveragents.domain.models.core.automation_profile import AutomationProfile

# This is what the spec says to write — fails with ValidationError
config = {
    "name": "local/test",
    "description": "Test",
    "decompose_task": 0.0,
    "create_tool": 0.0,
    "select_tool": 1.0,
    "edit_code": 0.0,
    "execute_command": 0.0,
    "create_file": 0.0,
    "delete_content": 1.0,
    "access_network": 1.0,
    "install_dependency": 0.0,
    "modify_config": 0.0,
    "approve_plan": 0.0,
    "require_sandbox": True,       # ← top-level per spec
    "require_checkpoints": True,   # ← top-level per spec
    "allow_unsafe_tools": False,   # ← top-level per spec
}
profile = AutomationProfile.from_config(config)
# Raises: pydantic.ValidationError: Extra inputs are not permitted [require_sandbox, require_checkpoints, allow_unsafe_tools]

Impact

Users cannot create custom automation profiles using the spec's documented YAML format
agents automation-profile add --config careful-auto.yaml fails for spec-compliant YAML files
The spec's example YAML files (lines 35967–35992) would fail validation

Definition of Done

AutomationProfile accepts require_sandbox, require_checkpoints, allow_unsafe_tools as top-level fields (matching the spec's JSON Schema)
A model_validator(mode="before") flattens these top-level safety flags into the safety sub-model, OR the model is restructured to have them at the top level
The spec's example YAML (lines 35967–35992) validates successfully
Existing code using safety.require_sandbox continues to work

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area:** Configuration System — Automation Profile YAML Schema **Severity:** Medium — custom automation profile YAML files written per the spec fail validation **Found by:** UAT tester, code analysis --- ### What Was Tested The automation profile YAML configuration schema was compared between the spec's JSON Schema definition and the `AutomationProfile` Pydantic model. ### Expected Behavior (from spec) Per `docs/specification.md` §Automation Profile Configuration Files, JSON Schema (lines 35741–35856), the safety flags are **top-level fields** in the YAML: ```yaml # Per spec — safety flags at top level name: local/careful-auto description: "Careful automation profile" decompose_task: 0.0 create_tool: 0.0 select_tool: 1.0 edit_code: 0.0 execute_command: 0.0 create_file: 0.0 delete_content: 1.0 access_network: 1.0 install_dependency: 0.0 modify_config: 0.0 approve_plan: 0.0 require_sandbox: true # ← top-level field per spec require_checkpoints: true # ← top-level field per spec allow_unsafe_tools: false # ← top-level field per spec ``` The spec's JSON Schema (line 35824) explicitly defines these as top-level properties: ```json "require_sandbox": { "type": "boolean", ... }, "require_checkpoints": { "type": "boolean", ... }, "allow_unsafe_tools": { "type": "boolean", ... } ``` And the spec's `required` array (line 35837) includes all three as required top-level fields. ### Actual Behavior The `AutomationProfile` Pydantic model in `src/cleveragents/domain/models/core/automation_profile.py` (lines 207–215) nests these under a `safety: SafetyProfile` sub-model: ```python safety: SafetyProfile = Field( default=DEFAULT_SAFETY_PROFILE, description="Composed safety profile controlling sandbox, checkpoint, and tool constraints.", ) ``` So the implementation expects: ```yaml name: local/careful-auto # ... threshold fields ... safety: require_sandbox: true # ← nested under safety: require_checkpoints: true # ← nested under safety: allow_unsafe_tools: false # ← nested under safety: ``` There is no `model_validator` or flattening logic to accept the spec's top-level format. A user writing a YAML file per the spec's documented schema will get a validation error because `require_sandbox` is not a recognized top-level field (the model has `extra="forbid"`). ### Code Location - `src/cleveragents/domain/models/core/automation_profile.py`, lines 207–215 (`safety` field) - `src/cleveragents/domain/models/core/automation_profile.py`, line 414 (`extra="forbid"`) ### Steps to Reproduce ```python from cleveragents.domain.models.core.automation_profile import AutomationProfile # This is what the spec says to write — fails with ValidationError config = { "name": "local/test", "description": "Test", "decompose_task": 0.0, "create_tool": 0.0, "select_tool": 1.0, "edit_code": 0.0, "execute_command": 0.0, "create_file": 0.0, "delete_content": 1.0, "access_network": 1.0, "install_dependency": 0.0, "modify_config": 0.0, "approve_plan": 0.0, "require_sandbox": True, # ← top-level per spec "require_checkpoints": True, # ← top-level per spec "allow_unsafe_tools": False, # ← top-level per spec } profile = AutomationProfile.from_config(config) # Raises: pydantic.ValidationError: Extra inputs are not permitted [require_sandbox, require_checkpoints, allow_unsafe_tools] ``` ### Impact - Users cannot create custom automation profiles using the spec's documented YAML format - `agents automation-profile add --config careful-auto.yaml` fails for spec-compliant YAML files - The spec's example YAML files (lines 35967–35992) would fail validation ### Definition of Done - [ ] `AutomationProfile` accepts `require_sandbox`, `require_checkpoints`, `allow_unsafe_tools` as top-level fields (matching the spec's JSON Schema) - [ ] A `model_validator(mode="before")` flattens these top-level safety flags into the `safety` sub-model, OR the model is restructured to have them at the top level - [ ] The spec's example YAML (lines 35967–35992) validates successfully - [ ] Existing code using `safety.require_sandbox` continues to work --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 referenced this issue

2026-04-08 20:19:44 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 referenced this issue

2026-04-08 20:22:25 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4799

HAL9000 added the

labels

2026-04-08 22:59:17 +00:00

freemo added

and removed

labels

2026-04-08 23:39:41 +00:00

freemo added this to the v3.5.0 milestone

2026-04-08 23:39:55 +00:00

freemo commented

2026-04-08 23:40:06 +00:00

Owner

Issue triaged by project owner:

State: Verified
Priority: Medium — Automation profile YAML schema mismatch with spec
Milestone: v3.5.0 — Automation profiles are M6 scope
Story Points: 5 — L — Aligning automation profile schema with spec (multiple fields)
MoSCoW: Should Have — Schema alignment is important for correctness but not blocking core execution
Parent Epic: #360 (Autonomy Hardening + Stubs M6)

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium — Automation profile YAML schema mismatch with spec - **Milestone**: v3.5.0 — Automation profiles are M6 scope - **Story Points**: 5 — L — Aligning automation profile schema with spec (multiple fields) - **MoSCoW**: Should Have — Schema alignment is important for correctness but not blocking core execution - **Parent Epic**: #360 (Autonomy Hardening + Stubs M6) --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner

HAL9000 referenced this issue

2026-04-09 01:52:32 +00:00

[AUTO-SPEC] Specification Update Report (Cycle 5) #5145