cleveragents/cleveragents-core

Fork 3

UAT: `Plan.effective_profile_snapshot` defaults to `'{}'` — spec requires a frozen snapshot of the resolved automation profile at plan creation time #4779

New issue

Open

opened 2026-04-08 18:56:41 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-08 18:56:41 +00:00

Owner

Bug Report

Feature Area: Plan lifecycle — Plan domain model / automation profile audit trail
Severity: Medium
Found by: UAT tester instance uat-worker-plan-lifecycle
Spec reference: docs/specification.md §Automation Profile (Glossary) — "frozen JSON snapshot of the automation profile at plan creation time"

What Was Tested

Code-level analysis of src/cleveragents/domain/models/core/plan.py — the Plan domain model's effective_profile_snapshot field.

Expected Behavior (from spec)

The spec defines the Automation Profile as:

A named set of confidence thresholds (each 0.0–1.0) gating which plan operations proceed automatically versus requiring human approval. [...] Each profile composes a Safety Profile that controls hard safety constraints.

The effective_profile_snapshot field is described in the code as:

"Frozen JSON snapshot of the automation profile at plan creation time. Used for audit / reproducibility."

This field should capture the actual resolved automation profile (with all threshold values, safety profile settings, etc.) at the moment the plan is created via agents plan use. This enables:

Audit: Knowing exactly what automation settings governed a plan's execution
Reproducibility: Being able to re-run a plan with the same automation settings
Debugging: Understanding why certain operations required human approval

Actual Behavior

The effective_profile_snapshot field defaults to '{}' (empty JSON object):

# src/cleveragents/domain/models/core/plan.py
effective_profile_snapshot: str = Field(
    default="{}",
    description=(
        "Frozen JSON snapshot of the automation profile at plan creation time"
    ),
)

The code comment explicitly acknowledges this is not satisfying the spec intent:

# NOTE: The default '{}' exists for backward compatibility with code
# paths that create Plan objects before the snapshot is populated.
# New plans SHOULD explicitly set this field to the resolved profile
# JSON at creation time; the empty default does not satisfy the spec
# intent of capturing a frozen profile for audit purposes.

Verification:

from cleveragents.domain.models.core.plan import Plan, PlanPhase, ProcessingState, PlanIdentity, NamespacedName
from ulid import ULID

plan = Plan(
    identity=PlanIdentity(plan_id=str(ULID())),
    namespaced_name=NamespacedName(namespace='local', name='test'),
    description='Test',
    action_name='local/action',
    phase=PlanPhase.STRATEGIZE,
    processing_state=ProcessingState.QUEUED,
)
print(plan.effective_profile_snapshot)  # Output: '{}'

Code Location

src/cleveragents/domain/models/core/plan.py — effective_profile_snapshot field (lines ~600-610)
src/cleveragents/application/services/plan_lifecycle_service.py — use_action() method — should populate this field when creating a plan

Root Cause

The PlanLifecycleService.use_action() method creates a Plan object but does not populate effective_profile_snapshot with the resolved automation profile's JSON. The field is left at its default '{}'.

Expected Fix

In PlanLifecycleService.use_action(), after resolving the automation profile, serialize it to JSON and set effective_profile_snapshot:

# In use_action():
resolved_profile = self._resolve_automation_profile(...)
profile_snapshot = resolved_profile.model_dump_json() if resolved_profile else "{}"

plan = Plan(
    ...
    effective_profile_snapshot=profile_snapshot,
    ...
)

Impact

Audit trail is broken: Plans created via agents plan use have effective_profile_snapshot = '{}' instead of the actual profile
Reproducibility is broken: Cannot determine what automation settings governed a plan's execution from the plan record alone
Debugging is impaired: Cannot tell from a plan's record whether it used manual, trusted, full-auto, or a custom profile
The code comment itself acknowledges this is a known spec violation

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area:** Plan lifecycle — Plan domain model / automation profile audit trail **Severity:** Medium **Found by:** UAT tester instance `uat-worker-plan-lifecycle` **Spec reference:** docs/specification.md §Automation Profile (Glossary) — "frozen JSON snapshot of the automation profile at plan creation time" --- ### What Was Tested Code-level analysis of `src/cleveragents/domain/models/core/plan.py` — the `Plan` domain model's `effective_profile_snapshot` field. ### Expected Behavior (from spec) The spec defines the Automation Profile as: > A named set of confidence thresholds (each `0.0`–`1.0`) gating which plan operations proceed automatically versus requiring human approval. [...] Each profile composes a **Safety Profile** that controls hard safety constraints. The `effective_profile_snapshot` field is described in the code as: > "Frozen JSON snapshot of the automation profile at plan creation time. Used for audit / reproducibility." This field should capture the **actual resolved automation profile** (with all threshold values, safety profile settings, etc.) at the moment the plan is created via `agents plan use`. This enables: 1. **Audit**: Knowing exactly what automation settings governed a plan's execution 2. **Reproducibility**: Being able to re-run a plan with the same automation settings 3. **Debugging**: Understanding why certain operations required human approval ### Actual Behavior The `effective_profile_snapshot` field defaults to `'{}'` (empty JSON object): ```python # src/cleveragents/domain/models/core/plan.py effective_profile_snapshot: str = Field( default="{}", description=( "Frozen JSON snapshot of the automation profile at plan creation time" ), ) ``` The code comment explicitly acknowledges this is not satisfying the spec intent: ```python # NOTE: The default '{}' exists for backward compatibility with code # paths that create Plan objects before the snapshot is populated. # New plans SHOULD explicitly set this field to the resolved profile # JSON at creation time; the empty default does not satisfy the spec # intent of capturing a frozen profile for audit purposes. ``` Verification: ```python from cleveragents.domain.models.core.plan import Plan, PlanPhase, ProcessingState, PlanIdentity, NamespacedName from ulid import ULID plan = Plan( identity=PlanIdentity(plan_id=str(ULID())), namespaced_name=NamespacedName(namespace='local', name='test'), description='Test', action_name='local/action', phase=PlanPhase.STRATEGIZE, processing_state=ProcessingState.QUEUED, ) print(plan.effective_profile_snapshot) # Output: '{}' ``` ### Code Location - `src/cleveragents/domain/models/core/plan.py` — `effective_profile_snapshot` field (lines ~600-610) - `src/cleveragents/application/services/plan_lifecycle_service.py` — `use_action()` method — should populate this field when creating a plan ### Root Cause The `PlanLifecycleService.use_action()` method creates a `Plan` object but does not populate `effective_profile_snapshot` with the resolved automation profile's JSON. The field is left at its default `'{}'`. ### Expected Fix In `PlanLifecycleService.use_action()`, after resolving the automation profile, serialize it to JSON and set `effective_profile_snapshot`: ```python # In use_action(): resolved_profile = self._resolve_automation_profile(...) profile_snapshot = resolved_profile.model_dump_json() if resolved_profile else "{}" plan = Plan( ... effective_profile_snapshot=profile_snapshot, ... ) ``` ### Impact - **Audit trail is broken**: Plans created via `agents plan use` have `effective_profile_snapshot = '{}'` instead of the actual profile - **Reproducibility is broken**: Cannot determine what automation settings governed a plan's execution from the plan record alone - **Debugging is impaired**: Cannot tell from a plan's record whether it used `manual`, `trusted`, `full-auto`, or a custom profile - The code comment itself acknowledges this is a known spec violation --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 referenced this issue

2026-04-08 18:58:06 +00:00

[Automated] CleverAgents Build Session - 2026-04-08 #4743

HAL9000 added the

labels

2026-04-08 23:00:12 +00:00

HAL9000 commented

2026-04-09 03:04:09 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: Medium — spec compliance bug identified by UAT testing
Story Points: 3 (M) — targeted fix to align implementation with spec
MoSCoW: Must Have — spec compliance is required for correct system behavior

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium — spec compliance bug identified by UAT testing - **Story Points**: 3 (M) — targeted fix to align implementation with spec - **MoSCoW**: Must Have — spec compliance is required for correct system behavior --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner

HAL9000 added

and removed

labels

2026-04-09 03:04:21 +00:00

HAL9000 added this to the v3.5.0 milestone