UAT: `PlanPreflightGuardrail.check_automation_policy()` is a stub — does not verify the automation profile permits execution #4017

New issue

Open

opened 2026-04-06 08:39:29 +00:00 by freemo · 0 comments

freemo commented

2026-04-06 08:39:29 +00:00

Owner

Metadata

Branch: fix/preflight-automation-policy-check
Commit Message: fix(preflight): implement automation policy check to verify profile permits execution
Milestone: (none — backlog)
Parent Epic: #3370

Bug Report

What Was Tested

The PlanPreflightGuardrail.check_automation_policy() method was analyzed against the specification's pre-flight check requirements.

Expected Behavior (from spec)

Per docs/specification.md line 28589:

Automation policy: Verify that the automation profile allows the requested level of autonomy for the target project, resources, and actions.

The pre-flight automation policy check should verify that the resolved automation profile is compatible with the plan's requirements — for example, checking that the profile doesn't block required operations, that the safety profile's constraints are satisfiable, or that the profile permits the requested autonomy level for the target resources.

Actual Behavior

src/cleveragents/application/services/plan_preflight_guardrail.py, lines 296–311:

def check_automation_policy(
    self,
    automation_profile: object | None,
) -> PreflightCheckResult:
    """Check 4: Verify the automation profile permits the requested autonomy."""
    if automation_profile is None:
        return PreflightCheckResult(
            PreflightCheckName.AUTOMATION_POLICY,
            False,
            "No automation profile provided",
        )
    return PreflightCheckResult(
        PreflightCheckName.AUTOMATION_POLICY,
        True,
        "Automation profile permits execution",
    )

The method only checks if the profile is None (FAIL) or not None (PASS). It does not:

Verify the profile name is valid (resolves to a known built-in or custom profile)
Check that the safety profile's constraints are satisfiable (e.g., require_sandbox=True but no sandbox available)
Verify the profile permits the requested autonomy level for the target resources
Check for conflicts between the profile's constraints and the plan's requirements

Any non-None value (including invalid profile names, stale profile references, or profiles with conflicting constraints) passes this check.

Steps to Reproduce

Create a plan with an invalid automation profile reference
Run the pre-flight check
Observe: check_automation_policy returns PASS even though the profile is invalid

Code Location

src/cleveragents/application/services/plan_preflight_guardrail.py lines 296–311

Fix Required

The check_automation_policy() method should:

Accept the resolved AutomationProfile object (not just object | None)
Verify the profile is a valid, resolvable profile
Check that the safety profile constraints are satisfiable given the plan context
Return FAIL with a descriptive message if any constraint cannot be satisfied

Subtasks

Update check_automation_policy() signature to accept AutomationProfile | None
Add validation that the profile is a valid, known profile
Add check that safety profile constraints are satisfiable
Add BDD scenarios for the automation policy pre-flight check
Update callers to pass the resolved AutomationProfile object

Definition of Done

Pre-flight automation policy check validates the profile is valid and its constraints are satisfiable
Invalid profiles are rejected before plan execution begins
All new BDD scenarios pass
All nox stages pass
Coverage >= 97%
PR merged

Backlog note: This issue was discovered during autonomous operation
on milestone v3.6.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/preflight-automation-policy-check` - **Commit Message**: `fix(preflight): implement automation policy check to verify profile permits execution` - **Milestone**: *(none — backlog)* - **Parent Epic**: #3370 ## Bug Report ### What Was Tested The `PlanPreflightGuardrail.check_automation_policy()` method was analyzed against the specification's pre-flight check requirements. ### Expected Behavior (from spec) Per `docs/specification.md` line 28589: > **Automation policy**: Verify that the automation profile allows the requested level of autonomy for the target project, resources, and actions. The pre-flight automation policy check should verify that the resolved automation profile is compatible with the plan's requirements — for example, checking that the profile doesn't block required operations, that the safety profile's constraints are satisfiable, or that the profile permits the requested autonomy level for the target resources. ### Actual Behavior **`src/cleveragents/application/services/plan_preflight_guardrail.py`, lines 296–311:** ```python def check_automation_policy( self, automation_profile: object | None, ) -> PreflightCheckResult: """Check 4: Verify the automation profile permits the requested autonomy.""" if automation_profile is None: return PreflightCheckResult( PreflightCheckName.AUTOMATION_POLICY, False, "No automation profile provided", ) return PreflightCheckResult( PreflightCheckName.AUTOMATION_POLICY, True, "Automation profile permits execution", ) ``` The method only checks if the profile is `None` (FAIL) or not `None` (PASS). It does not: 1. Verify the profile name is valid (resolves to a known built-in or custom profile) 2. Check that the safety profile's constraints are satisfiable (e.g., `require_sandbox=True` but no sandbox available) 3. Verify the profile permits the requested autonomy level for the target resources 4. Check for conflicts between the profile's constraints and the plan's requirements Any non-None value (including invalid profile names, stale profile references, or profiles with conflicting constraints) passes this check. ### Steps to Reproduce 1. Create a plan with an invalid automation profile reference 2. Run the pre-flight check 3. Observe: `check_automation_policy` returns PASS even though the profile is invalid ### Code Location - `src/cleveragents/application/services/plan_preflight_guardrail.py` lines 296–311 ### Fix Required The `check_automation_policy()` method should: 1. Accept the resolved `AutomationProfile` object (not just `object | None`) 2. Verify the profile is a valid, resolvable profile 3. Check that the safety profile constraints are satisfiable given the plan context 4. Return FAIL with a descriptive message if any constraint cannot be satisfied ## Subtasks - [ ] Update `check_automation_policy()` signature to accept `AutomationProfile | None` - [ ] Add validation that the profile is a valid, known profile - [ ] Add check that safety profile constraints are satisfiable - [ ] Add BDD scenarios for the automation policy pre-flight check - [ ] Update callers to pass the resolved `AutomationProfile` object ## Definition of Done - [ ] Pre-flight automation policy check validates the profile is valid and its constraints are satisfiable - [ ] Invalid profiles are rejected before plan execution begins - [ ] All new BDD scenarios pass - [ ] All nox stages pass - [ ] Coverage >= 97% - [ ] PR merged > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.6.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added a new dependency

2026-04-06 08:39:36 +00:00

#3370 Epic: Automation Profile & Safety Profile CLI — Cost Budgets, Guard Enforcement, and Profile Management

freemo referenced this issue

2026-04-06 08:42:28 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo added the

Points

label

2026-04-06 08:42:44 +00:00

freemo referenced this issue

2026-04-06 08:51:05 +00:00