cleveragents/cleveragents-core

Fork 3

UAT: `check_automation_policy()` preflight check is a stub — does not verify profile permits requested autonomy level #5629

New issue

Open

opened 2026-04-09 07:58:33 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-09 07:58:33 +00:00

Owner

Summary

PlanPreflightGuardrail.check_automation_policy() is a stub that only checks if automation_profile is not None. It does not verify that the automation profile actually permits the requested level of autonomy for the target project, resources, and actions, as required by the spec.

What Was Tested

Code-level analysis of:

src/cleveragents/application/services/plan_preflight_guardrail.py — check_automation_policy() method

Expected Behavior (from spec §"Plan Generation Guardrails")

Automation policy: Verify that the automation profile allows the requested level of autonomy for the target project, resources, and actions.

This check should validate that:

The profile's confidence thresholds are compatible with the plan's requirements
The profile's safety constraints (sandbox, checkpoints, unsafe tools) are compatible with the plan's tools
The profile is not blocked from operating on the target project or resources

Actual Behavior

The implementation is a stub that passes as long as any profile object is provided:

# src/cleveragents/application/services/plan_preflight_guardrail.py
def check_automation_policy(
    self,
    automation_profile: object | None,
) -> PreflightCheckResult:
    """Check 4: Verify the automation profile permits the requested autonomy."""
    if automation_profile is None:
        return PreflightCheckResult(
            PreflightCheckName.AUTOMATION_POLICY,
            False,
            "No automation profile provided",
        )
    return PreflightCheckResult(
        PreflightCheckName.AUTOMATION_POLICY,
        True,
        "Automation profile permits execution",  # Always passes!
    )

This means:

A plan using the manual profile (all thresholds at 1.0) passes the automation policy check even if the plan requires fully automatic execution
A plan with tools that require allow_unsafe_tools: true passes even if the profile has allow_unsafe_tools: false
The check provides no actual validation

Impact

Pre-flight guardrail check #4 is non-functional
Plans that should be rejected due to incompatible automation profiles proceed to Strategize
The spec's "7 pre-flight checks" guarantee is broken — only 6 are actually implemented

Code Location

Stub implementation: src/cleveragents/application/services/plan_preflight_guardrail.py:check_automation_policy() (~line 220)

Fix Required

check_automation_policy() should be updated to:

Accept the plan's required autonomy level (e.g., which phase transitions need to be automatic)
Accept the plan's tool list (to check against allow_unsafe_tools, allowed_skill_categories)
Verify the profile's thresholds and safety constraints are compatible with the plan's requirements
Return a failing result with a specific reason when incompatible

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Summary `PlanPreflightGuardrail.check_automation_policy()` is a stub that only checks if `automation_profile is not None`. It does not verify that the automation profile actually permits the requested level of autonomy for the target project, resources, and actions, as required by the spec. ## What Was Tested Code-level analysis of: - `src/cleveragents/application/services/plan_preflight_guardrail.py` — `check_automation_policy()` method ## Expected Behavior (from spec §"Plan Generation Guardrails") > **Automation policy**: Verify that the automation profile allows the requested level of autonomy for the target project, resources, and actions. This check should validate that: - The profile's confidence thresholds are compatible with the plan's requirements - The profile's safety constraints (sandbox, checkpoints, unsafe tools) are compatible with the plan's tools - The profile is not blocked from operating on the target project or resources ## Actual Behavior The implementation is a stub that passes as long as any profile object is provided: ```python # src/cleveragents/application/services/plan_preflight_guardrail.py def check_automation_policy( self, automation_profile: object | None, ) -> PreflightCheckResult: """Check 4: Verify the automation profile permits the requested autonomy.""" if automation_profile is None: return PreflightCheckResult( PreflightCheckName.AUTOMATION_POLICY, False, "No automation profile provided", ) return PreflightCheckResult( PreflightCheckName.AUTOMATION_POLICY, True, "Automation profile permits execution", # Always passes! ) ``` This means: - A plan using the `manual` profile (all thresholds at 1.0) passes the automation policy check even if the plan requires fully automatic execution - A plan with tools that require `allow_unsafe_tools: true` passes even if the profile has `allow_unsafe_tools: false` - The check provides no actual validation ## Impact - Pre-flight guardrail check #4 is non-functional - Plans that should be rejected due to incompatible automation profiles proceed to Strategize - The spec's "7 pre-flight checks" guarantee is broken — only 6 are actually implemented ## Code Location - Stub implementation: `src/cleveragents/application/services/plan_preflight_guardrail.py:check_automation_policy()` (~line 220) ## Fix Required `check_automation_policy()` should be updated to: 1. Accept the plan's required autonomy level (e.g., which phase transitions need to be automatic) 2. Accept the plan's tool list (to check against `allow_unsafe_tools`, `allowed_skill_categories`) 3. Verify the profile's thresholds and safety constraints are compatible with the plan's requirements 4. Return a failing result with a specific reason when incompatible --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 07:59:07 +00:00

HAL9000 referenced this issue

2026-04-09 08:01:58 +00:00

[AUTO-PROJ-OWN] Project Owner Report (Cycle 27) #5570

HAL9000 added this to the v3.5.0 milestone

2026-04-09 08:05:32 +00:00

HAL9000 commented

2026-04-09 08:07:21 +00:00

Author

Owner

Label compliance fix applied:

Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: backlog-groomer

Label compliance fix applied: - Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: backlog-groomer

HAL9000 added a new dependency

2026-04-09 08:16:55 +00:00

#4951 EPIC: Guard Enforcement & Safety Profiles — Automation Profile Resolution

HAL9000 referenced this issue

2026-04-09 08:20:03 +00:00

[AUTO-UAT-POOL] UAT Testing Report (Cycle 1) #5369

HAL9000 referenced this issue

2026-04-09 08:31:46 +00:00

[AUTO-EPIC] Epic Planning Health Report (Cycle 11) #5298

HAL9000 referenced this issue

2026-04-09 08:48:56 +00:00

[AUTO-ARCH] Architecture Review (Cycle 1) #5368