cleveragents/cleveragents-core

Fork 3

UAT: agents plan use --automation-profile rejects custom profiles — only accepts built-in profiles #6226

New issue

Open

opened 2026-04-09 17:53:07 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-09 17:53:07 +00:00

Owner

Bug Report

Feature Area: automation-profiles
Severity: Critical — blocks core spec-required functionality
Spec Reference: §28600 "Custom Automation Profiles", §19030 "automation_profile field", §28589 "Profile Precedence"

Summary

agents plan use --automation-profile <custom-profile> rejects any custom (namespaced) automation profile with an error, even when the profile was successfully created via agents automation-profile add. Only the 8 built-in profile names are accepted.

Steps to Reproduce

# Step 1: Create a custom profile
cat > /tmp/careful-auto.yaml << 'EOF'
name: local/careful-auto
description: "Autonomous execution with mandatory sandbox and manual apply"
decompose_task: 0.0
create_tool: 0.0
select_tool: 1.0
edit_code: 0.0
execute_command: 0.0
create_file: 0.0
delete_content: 1.0
access_network: 1.0
install_dependency: 0.0
modify_config: 0.0
approve_plan: 0.0
safety:
  require_sandbox: true
  require_checkpoints: true
  allow_unsafe_tools: false
EOF

agents automation-profile add --config /tmp/careful-auto.yaml
# ✓ Profile Added: local/careful-auto

# Step 2: Try to use the custom profile with a plan
agents plan use my-action --automation-profile local/careful-auto
# ✗ FAILS with error

Expected Behavior (per spec §28600, §19030)

The command should accept any registered custom profile name (including namespaced names like local/careful-auto, acme/strict) and use it for the plan. The spec explicitly shows custom profiles being used with plans:

"The resolved automation profile name for this plan (e.g., trusted, auto, local/careful-auto). Determined at plan use time using the profile precedence rules (plan > action > project > global)."

Actual Behavior

Invalid automation profile: local/careful-auto. Available: auto, cautious, ci, full-auto, manual, review, supervised, trusted

The CLI aborts with a typer.Abort().

Root Cause

File: src/cleveragents/cli/commands/plan.py, lines 2073–2083

if automation_profile:
    from cleveragents.domain.models.core.automation_profile import (
        BUILTIN_PROFILES,
    )

    if automation_profile not in BUILTIN_PROFILES:  # ← BUG: only checks built-ins
        console.print(
            f"[red]Invalid automation profile:[/red] {automation_profile}. "
            f"Available: {', '.join(sorted(BUILTIN_PROFILES))}"
        )
        raise typer.Abort()

The validation only checks BUILTIN_PROFILES dict and rejects any profile not in that dict. It does not query the AutomationProfileService or the profile repository to check if the profile exists as a custom profile.

Secondary root cause: plan_lifecycle_service.py, _resolve_plan_profile_ref(), lines 1197–1201:

if chosen_profile not in BUILTIN_PROFILES:
    raise ValidationError(
        f"Unknown automation profile '{chosen_profile}'. "
        f"Available built-ins: {', '.join(sorted(BUILTIN_PROFILES))}"
    )

Even if the CLI validation were fixed, the service layer also rejects custom profiles.

Fix Required

plan.py: Replace the BUILTIN_PROFILES check with a call to AutomationProfileService.get_profile(automation_profile) to validate the profile exists (built-in or custom). Catch NotFoundError and display an appropriate error.
plan_lifecycle_service.py _resolve_plan_profile_ref(): Replace the BUILTIN_PROFILES check with a call to AutomationProfileService.get_profile() to allow custom profiles.

Impact

Custom profiles created via agents automation-profile add cannot be used with plans
The spec example local/careful-auto profile cannot be applied to any plan
The entire custom profile feature is non-functional for plan execution
Spec §28600 "Custom Automation Profiles" is completely blocked

Verification

# Reproduces the CLI-level rejection
from cleveragents.domain.models.core.automation_profile import BUILTIN_PROFILES

automation_profile = 'local/careful-auto'
if automation_profile not in BUILTIN_PROFILES:
    print(f"BUG: Custom profile rejected: {automation_profile}")
# Output: BUG: Custom profile rejected: local/careful-auto

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: automation-profiles **Severity**: Critical — blocks core spec-required functionality **Spec Reference**: §28600 "Custom Automation Profiles", §19030 "automation_profile field", §28589 "Profile Precedence" --- ## Summary `agents plan use --automation-profile <custom-profile>` rejects any custom (namespaced) automation profile with an error, even when the profile was successfully created via `agents automation-profile add`. Only the 8 built-in profile names are accepted. --- ## Steps to Reproduce ```bash # Step 1: Create a custom profile cat > /tmp/careful-auto.yaml << 'EOF' name: local/careful-auto description: "Autonomous execution with mandatory sandbox and manual apply" decompose_task: 0.0 create_tool: 0.0 select_tool: 1.0 edit_code: 0.0 execute_command: 0.0 create_file: 0.0 delete_content: 1.0 access_network: 1.0 install_dependency: 0.0 modify_config: 0.0 approve_plan: 0.0 safety: require_sandbox: true require_checkpoints: true allow_unsafe_tools: false EOF agents automation-profile add --config /tmp/careful-auto.yaml # ✓ Profile Added: local/careful-auto # Step 2: Try to use the custom profile with a plan agents plan use my-action --automation-profile local/careful-auto # ✗ FAILS with error ``` ## Expected Behavior (per spec §28600, §19030) The command should accept any registered custom profile name (including namespaced names like `local/careful-auto`, `acme/strict`) and use it for the plan. The spec explicitly shows custom profiles being used with plans: > "The resolved automation profile name for this plan (e.g., `trusted`, `auto`, `local/careful-auto`). Determined at `plan use` time using the profile precedence rules (plan > action > project > global)." ## Actual Behavior ``` Invalid automation profile: local/careful-auto. Available: auto, cautious, ci, full-auto, manual, review, supervised, trusted ``` The CLI aborts with a `typer.Abort()`. --- ## Root Cause **File**: `src/cleveragents/cli/commands/plan.py`, lines 2073–2083 ```python if automation_profile: from cleveragents.domain.models.core.automation_profile import ( BUILTIN_PROFILES, ) if automation_profile not in BUILTIN_PROFILES: # ← BUG: only checks built-ins console.print( f"[red]Invalid automation profile:[/red] {automation_profile}. " f"Available: {', '.join(sorted(BUILTIN_PROFILES))}" ) raise typer.Abort() ``` The validation only checks `BUILTIN_PROFILES` dict and rejects any profile not in that dict. It does not query the `AutomationProfileService` or the profile repository to check if the profile exists as a custom profile. **Secondary root cause**: `plan_lifecycle_service.py`, `_resolve_plan_profile_ref()`, lines 1197–1201: ```python if chosen_profile not in BUILTIN_PROFILES: raise ValidationError( f"Unknown automation profile '{chosen_profile}'. " f"Available built-ins: {', '.join(sorted(BUILTIN_PROFILES))}" ) ``` Even if the CLI validation were fixed, the service layer also rejects custom profiles. --- ## Fix Required 1. **`plan.py`**: Replace the `BUILTIN_PROFILES` check with a call to `AutomationProfileService.get_profile(automation_profile)` to validate the profile exists (built-in or custom). Catch `NotFoundError` and display an appropriate error. 2. **`plan_lifecycle_service.py` `_resolve_plan_profile_ref()`**: Replace the `BUILTIN_PROFILES` check with a call to `AutomationProfileService.get_profile()` to allow custom profiles. --- ## Impact - Custom profiles created via `agents automation-profile add` cannot be used with plans - The spec example `local/careful-auto` profile cannot be applied to any plan - The entire custom profile feature is non-functional for plan execution - Spec §28600 "Custom Automation Profiles" is completely blocked --- ## Verification ```python # Reproduces the CLI-level rejection from cleveragents.domain.models.core.automation_profile import BUILTIN_PROFILES automation_profile = 'local/careful-auto' if automation_profile not in BUILTIN_PROFILES: print(f"BUG: Custom profile rejected: {automation_profile}") # Output: BUG: Custom profile rejected: local/careful-auto ``` --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 17:53:24 +00:00

HAL9000 added this to the v3.3.0 milestone

2026-04-09 17:53:56 +00:00

HAL9000 referenced this issue

2026-04-09 18:09:58 +00:00

[AUTO-ARCH] Architecture Review (Cycle 7) #6242

HAL9000 referenced this issue

2026-04-09 18:12:46 +00:00

[AUTO-PROJ-OWN] Project Owner Report (Cycle 36) #6249

HAL9000 added a new dependency

2026-04-09 18:16:18 +00:00

#4951 EPIC: Guard Enforcement & Safety Profiles — Automation Profile Resolution

HAL9000 referenced this issue

2026-04-09 19:37:03 +00:00

[AUTO-UAT-POOL] UAT Testing Report (Cycle 2) #6263

HAL9000 referenced this issue

2026-04-13 01:32:01 +00:00

Epic: Plan Correction & Automation Profile Fixes (M4) #8050

HAL9000 referenced this issue

2026-04-13 05:05:56 +00:00

Epic: Automation Profile Resolution & Precedence (M6) #8231

HAL9000 referenced this issue

2026-04-14 20:11:00 +00:00

[AUTO-PROJ-OWN] Status: Project Owner Supervisor (Cycle 2) #9489

HAL9000 added

and removed

labels

2026-04-17 06:02:53 +00:00

HAL9000 commented

2026-04-17 06:05:42 +00:00

Author

Owner

✅ Verified — Critical: custom automation profiles are a core v3.3.0 feature and the CLI rejects them entirely. MoSCoW: Must Have — blocks the automation profile workflow.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner-pool-supervisor

✅ **Verified** — Critical: custom automation profiles are a core v3.3.0 feature and the CLI rejects them entirely. **MoSCoW: Must Have** — blocks the automation profile workflow. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner-pool-supervisor