UAT: `SafetyProfile.max_cost_per_plan` field is defined but never enforced at runtime — per-plan cost limit has no effect #4394

New issue

Open

opened 2026-04-08 12:04:43 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-08 12:04:43 +00:00

Owner

Summary

The SafetyProfile model defines a max_cost_per_plan field (maximum cost in USD allowed per plan execution), but this field is never checked or enforced anywhere in the runtime execution path. Plans can exceed the per-plan cost limit without any enforcement.

Evidence

Field Definition (exists but unused)

src/cleveragents/domain/models/core/safety_profile.py:

max_cost_per_plan: float | None = Field(
    default=None,
    ge=0.0,
    description="Maximum cost in USD allowed per plan. None means no limit.",
)

Enforcement Gap

The AutomationGuard.check_guard() method in src/cleveragents/domain/models/core/automation_profile.py checks AutomationGuard.max_total_cost (a different field on a different model), but does NOT check SafetyProfile.max_cost_per_plan.

The SafetyProfile is composed into AutomationProfile via the safety field, but the check_guard() method only uses self.guards (an AutomationGuard instance), not self.safety (the SafetyProfile).

No Enforcement in Plan Execution

Searching the codebase for max_cost_per_plan usage:

Defined in safety_profile.py ✓
Cross-field validation (must be ≤ max_total_cost) ✓
No enforcement in plan executor, lifecycle service, or tool runtime ✗

The plan_preflight_guardrail.py check_automation_policy() method only checks if the profile is None — it does not validate cost limits.

Impact

Users who configure max_cost_per_plan in their safety profile get no enforcement
Plans can run indefinitely regardless of per-plan cost limits
The SafetyProfile documentation promises cost enforcement that doesn't exist
This is a security/cost-control gap for users relying on budget limits

Spec Reference

From docs/specification.md:

Safety Profile: A composed sub-model of an Automation Profile that groups all hard safety constraints: require_sandbox, require_checkpoints, allow_unsafe_tools, require_human_approval, allowed_skill_categories, max_cost_per_plan, max_retries_per_step, and max_total_cost.

The spec explicitly lists max_cost_per_plan as a hard safety constraint that should be enforced.

#4328 — Manual automation profile disrespected (broader enforcement gap)
UAT: CostTracker and FallbackSelector are dead code — never wired into provider execution path, budget enforcement is completely non-functional

Code Locations

src/cleveragents/domain/models/core/safety_profile.py — field definition
src/cleveragents/domain/models/core/automation_profile.py — check_guard() method (doesn't use self.safety.max_cost_per_plan)
src/cleveragents/application/services/plan_preflight_guardrail.py — check_automation_policy() (stub)
src/cleveragents/application/services/plan_lifecycle_service.py — plan execution (no cost enforcement)

Expected Behavior

When safety.max_cost_per_plan is set on an automation profile:

The plan executor should track cumulative cost per plan
When cost exceeds max_cost_per_plan, execution should halt and require human approval
The preflight guardrail should validate that the profile's cost limit is reasonable

Commit Message

fix(safety): enforce SafetyProfile.max_cost_per_plan in plan execution pipeline

Branch

fix/safety-profile-cost-enforcement

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Summary The `SafetyProfile` model defines a `max_cost_per_plan` field (maximum cost in USD allowed per plan execution), but this field is never checked or enforced anywhere in the runtime execution path. Plans can exceed the per-plan cost limit without any enforcement. ## Evidence ### Field Definition (exists but unused) `src/cleveragents/domain/models/core/safety_profile.py`: ```python max_cost_per_plan: float | None = Field( default=None, ge=0.0, description="Maximum cost in USD allowed per plan. None means no limit.", ) ``` ### Enforcement Gap The `AutomationGuard.check_guard()` method in `src/cleveragents/domain/models/core/automation_profile.py` checks `AutomationGuard.max_total_cost` (a different field on a different model), but does NOT check `SafetyProfile.max_cost_per_plan`. The `SafetyProfile` is composed into `AutomationProfile` via the `safety` field, but the `check_guard()` method only uses `self.guards` (an `AutomationGuard` instance), not `self.safety` (the `SafetyProfile`). ### No Enforcement in Plan Execution Searching the codebase for `max_cost_per_plan` usage: - Defined in `safety_profile.py` ✓ - Cross-field validation (must be ≤ `max_total_cost`) ✓ - **No enforcement in plan executor, lifecycle service, or tool runtime** ✗ The `plan_preflight_guardrail.py` `check_automation_policy()` method only checks if the profile is `None` — it does not validate cost limits. ## Impact - Users who configure `max_cost_per_plan` in their safety profile get no enforcement - Plans can run indefinitely regardless of per-plan cost limits - The `SafetyProfile` documentation promises cost enforcement that doesn't exist - This is a security/cost-control gap for users relying on budget limits ## Spec Reference From `docs/specification.md`: > Safety Profile: A composed sub-model of an Automation Profile that groups all hard safety constraints: `require_sandbox`, `require_checkpoints`, `allow_unsafe_tools`, `require_human_approval`, `allowed_skill_categories`, **`max_cost_per_plan`**, `max_retries_per_step`, and `max_total_cost`. The spec explicitly lists `max_cost_per_plan` as a hard safety constraint that should be enforced. ## Related Issues - #4328 — Manual automation profile disrespected (broader enforcement gap) - UAT: `CostTracker` and `FallbackSelector` are dead code — never wired into provider execution path, budget enforcement is completely non-functional ## Code Locations - `src/cleveragents/domain/models/core/safety_profile.py` — field definition - `src/cleveragents/domain/models/core/automation_profile.py` — `check_guard()` method (doesn't use `self.safety.max_cost_per_plan`) - `src/cleveragents/application/services/plan_preflight_guardrail.py` — `check_automation_policy()` (stub) - `src/cleveragents/application/services/plan_lifecycle_service.py` — plan execution (no cost enforcement) ## Expected Behavior When `safety.max_cost_per_plan` is set on an automation profile: 1. The plan executor should track cumulative cost per plan 2. When cost exceeds `max_cost_per_plan`, execution should halt and require human approval 3. The preflight guardrail should validate that the profile's cost limit is reasonable ## Commit Message `fix(safety): enforce SafetyProfile.max_cost_per_plan in plan execution pipeline` ## Branch `fix/safety-profile-cost-enforcement` --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added this to the v3.5.0 milestone

2026-04-08 17:42:55 +00:00

HAL9000 added the