UAT: `agents plan use --automation-profile` rejects custom (namespaced) profiles — only accepts built-in profile names #4008

New issue

Open

opened 2026-04-06 08:33:42 +00:00 by freemo · 0 comments

freemo commented

2026-04-06 08:33:42 +00:00

Owner

Metadata

Branch: fix/plan-use-custom-automation-profiles
Commit Message: fix(plan-lifecycle): support custom automation profiles in plan use and profile resolution
Milestone: (none — backlog)
Parent Epic: #3370

Bug Report

What Was Tested

The agents plan use --automation-profile <PROFILE> CLI command and the underlying PlanLifecycleService._resolve_plan_profile_ref() method were analyzed against the specification's requirement that custom (namespaced) automation profiles be supported.

Expected Behavior (from spec)

Per docs/specification.md lines 12498 and 28473:

--automation-profile PROFILE: Automation profile name (e.g., trusted, auto, local/careful-auto). Overrides the profile inherited from the action, project, or global config.
Custom profiles are created via YAML configuration files and registered with agents automation-profile add.
The spec explicitly shows local/careful-auto as a valid profile name for plan use.

Actual Behavior

CLI (src/cleveragents/cli/commands/plan.py, lines 1678–1683):

if automation_profile not in BUILTIN_PROFILES:
    console.print(
        f"[red]Invalid automation profile:[/red] {automation_profile}. "
        f"Available: {', '.join(sorted(BUILTIN_PROFILES))}"
    )
    raise typer.Abort()

The CLI only checks BUILTIN_PROFILES (the 8 built-in profiles) and immediately aborts if the profile name is not in that dict. Custom profiles registered via agents automation-profile add are rejected with an error.

Service layer (src/cleveragents/application/services/plan_lifecycle_service.py, lines 1197–1201):

if chosen_profile not in BUILTIN_PROFILES:
    raise ValidationError(
        f"Unknown automation profile '{chosen_profile}'. "
        f"Available built-ins: {', '.join(sorted(BUILTIN_PROFILES))}"
    )

The _resolve_plan_profile_ref() method also only checks BUILTIN_PROFILES and raises a ValidationError for any custom profile name.

Steps to Reproduce

Register a custom profile: agents automation-profile add --config ./my-profile.yaml (where the YAML has name: acme/strict)
Attempt to use it: agents plan use my-action --automation-profile acme/strict
Observe: CLI aborts with "Invalid automation profile: acme/strict. Available: auto, cautious, ci, full-auto, manual, review, supervised, trusted"

Code Locations

src/cleveragents/cli/commands/plan.py lines 1673–1683
src/cleveragents/application/services/plan_lifecycle_service.py lines 1197–1201

Fix Required

Both the CLI validation and the service layer need to check custom profiles from the AutomationProfileRepository (or AutomationProfileService) in addition to BUILTIN_PROFILES. The AutomationProfileService.get_profile() method already handles this correctly (checks built-ins first, then repository) — the fix should use that service instead of directly checking BUILTIN_PROFILES.

Subtasks

Update src/cleveragents/cli/commands/plan.py to validate custom profiles via AutomationProfileService instead of BUILTIN_PROFILES
Update src/cleveragents/application/services/plan_lifecycle_service.py _resolve_plan_profile_ref() to look up custom profiles from the repository
Add BDD scenario: agents plan use --automation-profile acme/strict succeeds when profile exists
Add BDD scenario: agents plan use --automation-profile acme/nonexistent fails with "not found" error

Definition of Done

This issue is complete when:

Custom profiles registered via agents automation-profile add can be used with agents plan use --automation-profile
The error message for unknown profiles mentions both built-ins and the ability to add custom profiles
All new BDD scenarios pass
All subtasks above are completed and checked off
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional details
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done
All nox stages pass
Coverage >= 97%

Backlog note: This issue was discovered during autonomous operation
on milestone . It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/plan-use-custom-automation-profiles` - **Commit Message**: `fix(plan-lifecycle): support custom automation profiles in plan use and profile resolution` - **Milestone**: _(none — backlog)_ - **Parent Epic**: #3370 ## Bug Report ### What Was Tested The `agents plan use --automation-profile <PROFILE>` CLI command and the underlying `PlanLifecycleService._resolve_plan_profile_ref()` method were analyzed against the specification's requirement that custom (namespaced) automation profiles be supported. ### Expected Behavior (from spec) Per `docs/specification.md` lines 12498 and 28473: - `--automation-profile PROFILE`: Automation profile name (e.g., `trusted`, `auto`, `local/careful-auto`). Overrides the profile inherited from the action, project, or global config. - Custom profiles are created via YAML configuration files and registered with `agents automation-profile add`. - The spec explicitly shows `local/careful-auto` as a valid profile name for `plan use`. ### Actual Behavior **CLI (`src/cleveragents/cli/commands/plan.py`, lines 1678–1683):** ```python if automation_profile not in BUILTIN_PROFILES: console.print( f"[red]Invalid automation profile:[/red] {automation_profile}. " f"Available: {', '.join(sorted(BUILTIN_PROFILES))}" ) raise typer.Abort() ``` The CLI only checks `BUILTIN_PROFILES` (the 8 built-in profiles) and immediately aborts if the profile name is not in that dict. Custom profiles registered via `agents automation-profile add` are rejected with an error. **Service layer (`src/cleveragents/application/services/plan_lifecycle_service.py`, lines 1197–1201):** ```python if chosen_profile not in BUILTIN_PROFILES: raise ValidationError( f"Unknown automation profile '{chosen_profile}'. " f"Available built-ins: {', '.join(sorted(BUILTIN_PROFILES))}" ) ``` The `_resolve_plan_profile_ref()` method also only checks `BUILTIN_PROFILES` and raises a `ValidationError` for any custom profile name. ### Steps to Reproduce 1. Register a custom profile: `agents automation-profile add --config ./my-profile.yaml` (where the YAML has `name: acme/strict`) 2. Attempt to use it: `agents plan use my-action --automation-profile acme/strict` 3. Observe: CLI aborts with "Invalid automation profile: acme/strict. Available: auto, cautious, ci, full-auto, manual, review, supervised, trusted" ### Code Locations - `src/cleveragents/cli/commands/plan.py` lines 1673–1683 - `src/cleveragents/application/services/plan_lifecycle_service.py` lines 1197–1201 ### Fix Required Both the CLI validation and the service layer need to check custom profiles from the `AutomationProfileRepository` (or `AutomationProfileService`) in addition to `BUILTIN_PROFILES`. The `AutomationProfileService.get_profile()` method already handles this correctly (checks built-ins first, then repository) — the fix should use that service instead of directly checking `BUILTIN_PROFILES`. ## Subtasks - [ ] Update `src/cleveragents/cli/commands/plan.py` to validate custom profiles via `AutomationProfileService` instead of `BUILTIN_PROFILES` - [ ] Update `src/cleveragents/application/services/plan_lifecycle_service.py` `_resolve_plan_profile_ref()` to look up custom profiles from the repository - [ ] Add BDD scenario: `agents plan use --automation-profile acme/strict` succeeds when profile exists - [ ] Add BDD scenario: `agents plan use --automation-profile acme/nonexistent` fails with "not found" error ## Definition of Done This issue is complete when: - [ ] Custom profiles registered via `agents automation-profile add` can be used with `agents plan use --automation-profile` - [ ] The error message for unknown profiles mentions both built-ins and the ability to add custom profiles - [ ] All new BDD scenarios pass - [ ] All subtasks above are completed and checked off - [ ] A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional details - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done - All nox stages pass - Coverage >= 97% > **Backlog note:** This issue was discovered during autonomous operation > on milestone <M>. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added a new dependency

2026-04-06 08:33:55 +00:00

#3370 Epic: Automation Profile & Safety Profile CLI — Cost Budgets, Guard Enforcement, and Profile Management

freemo added the

Points

label

2026-04-06 08:37:11 +00:00

freemo referenced this issue

2026-04-06 08:51:05 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo referenced this issue

2026-04-06 09:07:06 +00:00