UAT: Custom automation profiles silently fall back to 'manual' during plan execution in _resolve_profile_for_plan #6231

New issue

Open

opened 2026-04-09 17:54:57 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-09 17:54:57 +00:00

Owner

Bug Report

Feature Area: automation-profiles
Severity: Critical — custom profiles are silently ignored during plan execution
Spec Reference: §28594 "Profile Locking", §19030 "automation_profile field", §28600 "Custom Automation Profiles"

Summary

When a plan has a custom (namespaced) automation profile set, the _resolve_profile_for_plan() method in PlanLifecycleService silently falls back to the manual profile instead of resolving the custom profile. This means plans using custom profiles execute with manual thresholds (all 1.0 — maximum human approval required) regardless of what the custom profile specifies.

Steps to Reproduce

This is a code-level bug that can be reproduced programmatically:

from cleveragents.domain.models.core.automation_profile import BUILTIN_PROFILES

# Simulate what _resolve_profile_for_plan does (line 2135 of plan_lifecycle_service.py)
def resolve_profile_for_plan(profile_name):
    return BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES['manual'])

# Custom profile silently falls back to manual
result = resolve_profile_for_plan('local/careful-auto')
print(f"Custom profile resolved to: {result.name}")
# Output: Custom profile resolved to: manual  ← BUG

Expected Behavior (per spec §28594)

"The effective profile for a plan is resolved at the moment of agents plan use. Once resolved, the profile is locked to that plan — subsequent changes to project or global profiles do not affect running plans."

When a plan has automation_profile.profile_name = 'local/careful-auto', the _resolve_profile_for_plan() method should look up the local/careful-auto profile from the AutomationProfileService (which checks both built-ins and the custom profile repository) and return it.

Actual Behavior

_resolve_profile_for_plan() only checks BUILTIN_PROFILES dict. If the profile name is not a built-in, it silently returns BUILTIN_PROFILES['manual'] (all thresholds = 1.0, maximum human approval). No error is raised, no warning is logged.

Root Cause

File: src/cleveragents/application/services/plan_lifecycle_service.py, lines 2119–2135

def _resolve_profile_for_plan(self, plan: Plan) -> AutomationProfile:
    """Resolve the effective AutomationProfile for a plan."""
    if plan.automation_profile is not None:
        profile_name = plan.automation_profile.profile_name
    else:
        profile_name = "manual"
    return BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES["manual"])  # ← BUG

The method uses BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES["manual"]) which:

Only looks up built-in profiles
Silently returns manual for any custom profile name
Does not use AutomationProfileService which has access to the custom profile repository

This method is called from:

should_auto_progress() (line 2162) — determines if plan phases auto-advance
_complete_apply_if_queued() (line 2303) — controls apply phase
plan_preflight_guardrail.py (line 1373) — pre-flight checks

Fix Required

plan_lifecycle_service.py: Inject AutomationProfileService into PlanLifecycleService and use it in _resolve_profile_for_plan():

def _resolve_profile_for_plan(self, plan: Plan) -> AutomationProfile:
    """Resolve the effective AutomationProfile for a plan."""
    if plan.automation_profile is not None:
        profile_name = plan.automation_profile.profile_name
    else:
        profile_name = "manual"
    
    # Use service to resolve (handles both built-in and custom profiles)
    if self._automation_profile_service is not None:
        try:
            return self._automation_profile_service.get_profile(profile_name)
        except NotFoundError:
            logger.warning(
                "Unknown automation profile '%s', falling back to manual",
                profile_name
            )
    
    # Fallback for when service is unavailable
    return BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES["manual"])

Impact

Plans using custom profiles execute with manual thresholds (all 1.0) instead of the configured thresholds
should_auto_progress() always returns False for plans with custom profiles (since manual has create_tool=1.0 and select_tool=1.0)
Plans with custom profiles never auto-advance between phases, requiring manual intervention for every phase transition
The custom profile feature is silently broken — no error is raised, making this very difficult to debug
Spec §28594 "profile locking" is violated: the locked profile is not actually used

Verification

# Verify the bug exists
from cleveragents.application.services.plan_lifecycle_service import PlanLifecycleService
import inspect

# Check _resolve_profile_for_plan source
src = inspect.getsource(PlanLifecycleService._resolve_profile_for_plan)
print(src)
# Shows: return BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES["manual"])
# No AutomationProfileService lookup present

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: automation-profiles **Severity**: Critical — custom profiles are silently ignored during plan execution **Spec Reference**: §28594 "Profile Locking", §19030 "automation_profile field", §28600 "Custom Automation Profiles" --- ## Summary When a plan has a custom (namespaced) automation profile set, the `_resolve_profile_for_plan()` method in `PlanLifecycleService` silently falls back to the `manual` profile instead of resolving the custom profile. This means plans using custom profiles execute with `manual` thresholds (all 1.0 — maximum human approval required) regardless of what the custom profile specifies. --- ## Steps to Reproduce This is a code-level bug that can be reproduced programmatically: ```python from cleveragents.domain.models.core.automation_profile import BUILTIN_PROFILES # Simulate what _resolve_profile_for_plan does (line 2135 of plan_lifecycle_service.py) def resolve_profile_for_plan(profile_name): return BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES['manual']) # Custom profile silently falls back to manual result = resolve_profile_for_plan('local/careful-auto') print(f"Custom profile resolved to: {result.name}") # Output: Custom profile resolved to: manual ← BUG ``` --- ## Expected Behavior (per spec §28594) > "The effective profile for a plan is resolved at the moment of `agents plan use`. Once resolved, the profile is **locked to that plan** — subsequent changes to project or global profiles do not affect running plans." When a plan has `automation_profile.profile_name = 'local/careful-auto'`, the `_resolve_profile_for_plan()` method should look up the `local/careful-auto` profile from the `AutomationProfileService` (which checks both built-ins and the custom profile repository) and return it. ## Actual Behavior `_resolve_profile_for_plan()` only checks `BUILTIN_PROFILES` dict. If the profile name is not a built-in, it silently returns `BUILTIN_PROFILES['manual']` (all thresholds = 1.0, maximum human approval). No error is raised, no warning is logged. --- ## Root Cause **File**: `src/cleveragents/application/services/plan_lifecycle_service.py`, lines 2119–2135 ```python def _resolve_profile_for_plan(self, plan: Plan) -> AutomationProfile: """Resolve the effective AutomationProfile for a plan.""" if plan.automation_profile is not None: profile_name = plan.automation_profile.profile_name else: profile_name = "manual" return BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES["manual"]) # ← BUG ``` The method uses `BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES["manual"])` which: 1. Only looks up built-in profiles 2. Silently returns `manual` for any custom profile name 3. Does not use `AutomationProfileService` which has access to the custom profile repository This method is called from: - `should_auto_progress()` (line 2162) — determines if plan phases auto-advance - `_complete_apply_if_queued()` (line 2303) — controls apply phase - `plan_preflight_guardrail.py` (line 1373) — pre-flight checks --- ## Fix Required **`plan_lifecycle_service.py`**: Inject `AutomationProfileService` into `PlanLifecycleService` and use it in `_resolve_profile_for_plan()`: ```python def _resolve_profile_for_plan(self, plan: Plan) -> AutomationProfile: """Resolve the effective AutomationProfile for a plan.""" if plan.automation_profile is not None: profile_name = plan.automation_profile.profile_name else: profile_name = "manual" # Use service to resolve (handles both built-in and custom profiles) if self._automation_profile_service is not None: try: return self._automation_profile_service.get_profile(profile_name) except NotFoundError: logger.warning( "Unknown automation profile '%s', falling back to manual", profile_name ) # Fallback for when service is unavailable return BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES["manual"]) ``` --- ## Impact - Plans using custom profiles execute with `manual` thresholds (all 1.0) instead of the configured thresholds - `should_auto_progress()` always returns `False` for plans with custom profiles (since `manual` has `create_tool=1.0` and `select_tool=1.0`) - Plans with custom profiles never auto-advance between phases, requiring manual intervention for every phase transition - The custom profile feature is silently broken — no error is raised, making this very difficult to debug - Spec §28594 "profile locking" is violated: the locked profile is not actually used --- ## Verification ```python # Verify the bug exists from cleveragents.application.services.plan_lifecycle_service import PlanLifecycleService import inspect # Check _resolve_profile_for_plan source src = inspect.getsource(PlanLifecycleService._resolve_profile_for_plan) print(src) # Shows: return BUILTIN_PROFILES.get(profile_name, BUILTIN_PROFILES["manual"]) # No AutomationProfileService lookup present ``` --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 17:55:08 +00:00

HAL9000 added this to the v3.3.0 milestone

2026-04-09 17:55:09 +00:00

HAL9000 referenced this issue