UAT: `agents plan use --automation-profile` rejects all custom profiles — only built-in profiles accepted #3441

New issue

Open

opened 2026-04-05 16:51:36 +00:00 by freemo · 1 comment

freemo commented

2026-04-05 16:51:36 +00:00

Owner

Metadata

Branch: fix/plan-use-custom-automation-profile-resolution
Commit Message: fix(cli): allow custom automation profiles in plan use --automation-profile
Milestone: v3.5.0
Parent Epic: #397

Background

The agents plan use --automation-profile <NAME> command explicitly rejects any profile that is not in the BUILTIN_PROFILES dict, making it impossible to use custom profiles (e.g., local/careful-auto) with plan execution. This is a spec violation.

Per docs/specification.md §"Custom Automation Profiles":

"Custom profiles are created via YAML configuration files and registered with agents automation-profile add."
"agents plan use local/fix-auth-bug local/api-project --automation-profile trusted"
"agents plan use local/fix-auth-bug local/api-project --automation-profile local/db-cautious"

The spec explicitly shows custom profiles (e.g., local/db-cautious) being used with agents plan use --automation-profile. The spec also says the precedence chain resolves to any registered profile, not just built-ins.

Actual Behavior (from code analysis)

CLI layer — src/cleveragents/cli/commands/plan.py, lines 1539–1549:

if automation_profile:
    from cleveragents.domain.models.core.automation_profile import (
        BUILTIN_PROFILES,
    )

    if automation_profile not in BUILTIN_PROFILES:
        console.print(
            f"[red]Invalid automation profile:[/red] {automation_profile}. "
            f"Available: {', '.join(sorted(BUILTIN_PROFILES))}"
        )
        raise typer.Abort()

This hard-codes a check against BUILTIN_PROFILES only, rejecting any custom profile name.

Service layer — src/cleveragents/application/services/plan_lifecycle_service.py, lines 1197–1201:

if chosen_profile not in BUILTIN_PROFILES:
    raise ValidationError(
        f"Unknown automation profile '{chosen_profile}'. "
        f"Available built-ins: {', '.join(sorted(BUILTIN_PROFILES))}"
    )

Resolution layer — _resolve_profile_for_plan (lines 2119–2135) silently falls back to "manual" for any non-built-in name:

return BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES["manual"])

Steps to reproduce:

Create a custom automation profile: agents automation-profile add --config ./my-profile.yaml
Try to use it: agents plan use local/my-plan local/my-project --automation-profile local/my-profile
Observe: Error "Invalid automation profile: local/my-profile. Available: auto, cautious, ci, full-auto, manual, review, supervised, trusted"

Impact: Custom automation profiles are completely non-functional with plan execution. The agents automation-profile add command works, but the profiles it creates can never be used. This makes the entire custom profile feature useless.

Note: This is separate from bugs #3294, #3346, #3372 which cover guard enforcement not being called. This bug is about profile resolution being restricted to built-ins only.

Subtasks

Remove the BUILTIN_PROFILES-only guard in src/cleveragents/cli/commands/plan.py (lines 1539–1549) and replace with a lookup against both built-in and registered custom profiles
Update src/cleveragents/application/services/plan_lifecycle_service.py (lines 1197–1201) to resolve custom profiles from the profile registry in addition to BUILTIN_PROFILES
Update _resolve_profile_for_plan (lines 2119–2135) to look up custom profiles from the registry before falling back to BUILTIN_PROFILES["manual"]
Add Behave BDD scenarios covering: (a) valid built-in profile accepted, (b) valid custom profile accepted, (c) unknown profile name rejected with correct error message
Add Robot Framework integration test: register a custom profile via agents automation-profile add, then use it with agents plan use --automation-profile, verify plan executes under the custom profile's constraints
Update docstrings and inline comments in affected files to reflect that both built-in and custom profiles are valid

Definition of Done

agents plan use --automation-profile <custom-profile-name> successfully resolves and applies a registered custom profile
agents plan use --automation-profile <builtin-name> continues to work as before (no regression)
An unregistered/unknown profile name produces a clear error listing both built-ins and registered custom profiles
_resolve_profile_for_plan correctly resolves custom profiles from the registry without silently falling back to "manual"
All new Behave scenarios pass (nox -e unit_tests)
Robot Framework integration test passes (nox -e integration_tests)
All nox stages pass
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/plan-use-custom-automation-profile-resolution` - **Commit Message**: `fix(cli): allow custom automation profiles in plan use --automation-profile` - **Milestone**: v3.5.0 - **Parent Epic**: #397 ## Background The `agents plan use --automation-profile <NAME>` command explicitly rejects any profile that is not in the `BUILTIN_PROFILES` dict, making it impossible to use custom profiles (e.g., `local/careful-auto`) with plan execution. This is a spec violation. Per `docs/specification.md` §"Custom Automation Profiles": > "Custom profiles are created via YAML configuration files and registered with `agents automation-profile add`." > "agents plan use local/fix-auth-bug local/api-project --automation-profile trusted" > "agents plan use local/fix-auth-bug local/api-project --automation-profile local/db-cautious" The spec explicitly shows custom profiles (e.g., `local/db-cautious`) being used with `agents plan use --automation-profile`. The spec also says the precedence chain resolves to any registered profile, not just built-ins. ## Actual Behavior (from code analysis) **CLI layer** — `src/cleveragents/cli/commands/plan.py`, lines 1539–1549: ```python if automation_profile: from cleveragents.domain.models.core.automation_profile import ( BUILTIN_PROFILES, ) if automation_profile not in BUILTIN_PROFILES: console.print( f"[red]Invalid automation profile:[/red] {automation_profile}. " f"Available: {', '.join(sorted(BUILTIN_PROFILES))}" ) raise typer.Abort() ``` This hard-codes a check against `BUILTIN_PROFILES` only, rejecting any custom profile name. **Service layer** — `src/cleveragents/application/services/plan_lifecycle_service.py`, lines 1197–1201: ```python if chosen_profile not in BUILTIN_PROFILES: raise ValidationError( f"Unknown automation profile '{chosen_profile}'. " f"Available built-ins: {', '.join(sorted(BUILTIN_PROFILES))}" ) ``` **Resolution layer** — `_resolve_profile_for_plan` (lines 2119–2135) silently falls back to `"manual"` for any non-built-in name: ```python return BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES["manual"]) ``` **Steps to reproduce**: 1. Create a custom automation profile: `agents automation-profile add --config ./my-profile.yaml` 2. Try to use it: `agents plan use local/my-plan local/my-project --automation-profile local/my-profile` 3. Observe: Error "Invalid automation profile: local/my-profile. Available: auto, cautious, ci, full-auto, manual, review, supervised, trusted" **Impact**: Custom automation profiles are completely non-functional with plan execution. The `agents automation-profile add` command works, but the profiles it creates can never be used. This makes the entire custom profile feature useless. **Note**: This is separate from bugs #3294, #3346, #3372 which cover guard enforcement not being called. This bug is about profile resolution being restricted to built-ins only. ## Subtasks - [ ] Remove the `BUILTIN_PROFILES`-only guard in `src/cleveragents/cli/commands/plan.py` (lines 1539–1549) and replace with a lookup against both built-in and registered custom profiles - [ ] Update `src/cleveragents/application/services/plan_lifecycle_service.py` (lines 1197–1201) to resolve custom profiles from the profile registry in addition to `BUILTIN_PROFILES` - [ ] Update `_resolve_profile_for_plan` (lines 2119–2135) to look up custom profiles from the registry before falling back to `BUILTIN_PROFILES["manual"]` - [ ] Add Behave BDD scenarios covering: (a) valid built-in profile accepted, (b) valid custom profile accepted, (c) unknown profile name rejected with correct error message - [ ] Add Robot Framework integration test: register a custom profile via `agents automation-profile add`, then use it with `agents plan use --automation-profile`, verify plan executes under the custom profile's constraints - [ ] Update docstrings and inline comments in affected files to reflect that both built-in and custom profiles are valid ## Definition of Done - [ ] `agents plan use --automation-profile <custom-profile-name>` successfully resolves and applies a registered custom profile - [ ] `agents plan use --automation-profile <builtin-name>` continues to work as before (no regression) - [ ] An unregistered/unknown profile name produces a clear error listing both built-ins and registered custom profiles - [ ] `_resolve_profile_for_plan` correctly resolves custom profiles from the registry without silently falling back to `"manual"` - [ ] All new Behave scenarios pass (`nox -e unit_tests`) - [ ] Robot Framework integration test passes (`nox -e integration_tests`) - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added this to the v3.5.0 milestone

2026-04-05 16:51:41 +00:00

freemo added a new dependency

2026-04-05 16:51:49 +00:00

#397 Epic: Server & Autonomy Infrastructure

freemo commented

2026-04-05 17:04:47 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: Critical — Custom automation profiles are entirely non-functional. The automation-profile add command creates profiles that can never be used, making the entire custom profile feature dead code.
Milestone: v3.5.0 (already assigned)
Story Points: 3 (M) — Three code locations need updating (CLI guard, service validation, resolution fallback) plus tests. Well-scoped.
MoSCoW: Must Have — The spec explicitly requires custom profile support (§Custom Automation Profiles). v3.5.0 acceptance criterion "Automation profile resolution precedence correct (plan > action > global)" cannot pass without custom profile resolution.
Parent Epic: #397 (Server & Autonomy Infrastructure)

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Critical — Custom automation profiles are entirely non-functional. The `automation-profile add` command creates profiles that can never be used, making the entire custom profile feature dead code. - **Milestone**: v3.5.0 (already assigned) - **Story Points**: 3 (M) — Three code locations need updating (CLI guard, service validation, resolution fallback) plus tests. Well-scoped. - **MoSCoW**: Must Have — The spec explicitly requires custom profile support (§Custom Automation Profiles). v3.5.0 acceptance criterion "Automation profile resolution precedence correct (plan > action > global)" cannot pass without custom profile resolution. - **Parent Epic**: #397 (Server & Autonomy Infrastructure) --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo referenced this issue

2026-04-05 17:07:19 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo removed this from the v3.5.0 milestone

2026-04-06 21:05:29 +00:00