cleveragents/cleveragents-core

Fork 3

UAT: `effective_profile_snapshot` field defaults to `{}` instead of capturing the resolved automation profile — audit/reproducibility requirement violated #2378

New issue

Open

opened 2026-04-03 17:26:40 +00:00 by freemo · 1 comment

freemo commented

2026-04-03 17:26:40 +00:00

Owner

Metadata

Branch: fix/plan-effective-profile-snapshot-capture
Commit Message: fix(plan): populate effective_profile_snapshot with resolved profile JSON at plan creation
Milestone: v3.7.0
Parent Epic: #372

Background and Context

The Plan domain model in src/cleveragents/domain/models/core/plan.py defines an effective_profile_snapshot field intended to capture a frozen JSON snapshot of the resolved automation profile at plan creation time. Per docs/specification.md, this snapshot is required for audit and reproducibility — so that the exact profile configuration used when the plan was created can be reviewed later, even if the profile is subsequently modified or deleted.

The field currently defaults to "{}" (empty JSON object), and the code itself acknowledges this is a spec violation:

# NOTE: The default '{}' exists for backward compatibility with code
# paths that create Plan objects before the snapshot is populated.
# New plans SHOULD explicitly set this field to the resolved profile
# JSON at creation time; the empty default does not satisfy the spec
# intent of capturing a frozen profile for audit purposes.
effective_profile_snapshot: str = Field(
    default="{}",
    ...
)

The PlanLifecycleService.use_action() method creates new plans but never sets effective_profile_snapshot to the actual resolved profile JSON. The _resolve_plan_profile_ref() method resolves the profile correctly and returns an AutomationProfileRef, but its full JSON representation is never captured in effective_profile_snapshot.

Current Behavior

All newly created plans have effective_profile_snapshot = "{}" (empty JSON object), making the audit trail incomplete. The resolved profile is stored in plan.automation_profile (as an AutomationProfileRef with just the name and provenance), but the full profile configuration snapshot is never captured.

Steps to reproduce:

agents action create --config my-action.yaml
agents plan use local/my-action my-project
agents plan status <PLAN_ID> --format json
Observe effective_profile_snapshot is {}

Expected Behavior

Per spec, effective_profile_snapshot must contain a frozen JSON snapshot of the full resolved automation profile at plan creation time. This is required for:

Audit: Reviewers can inspect the exact profile configuration that was active when the plan was created.
Reproducibility: Plans can be re-executed with the same profile configuration even if the profile has since been modified.

Acceptance Criteria

PlanLifecycleService.use_action() sets effective_profile_snapshot to the full JSON serialization of the resolved AutomationProfile at plan creation time
effective_profile_snapshot is never "{}" for a newly created plan when a profile is resolvable
agents plan status <PLAN_ID> --format json shows a non-empty effective_profile_snapshot containing the full profile configuration
If no profile can be resolved (e.g., no global default), effective_profile_snapshot is set to "{}" and a warning is logged
Existing plans with effective_profile_snapshot = "{}" are not retroactively modified (backward compatibility preserved)

Subtasks

Investigate PlanLifecycleService.use_action() and _resolve_plan_profile_ref() to understand the full resolved profile object available at plan creation time
Serialize the resolved AutomationProfile to JSON and assign it to plan.effective_profile_snapshot in use_action()
Ensure the snapshot is taken after profile resolution and before the plan is persisted to the database
Add a guard: if profile resolution returns None, log a warning and leave effective_profile_snapshot as "{}"
Tests (Behave): Add scenario — newly created plan has non-empty effective_profile_snapshot matching the resolved profile
Tests (Behave): Add scenario — effective_profile_snapshot is stable (does not change if the profile is later modified)
Tests (Robot): Add integration test verifying agents plan status --format json shows populated effective_profile_snapshot
Verify coverage >= 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/plan-effective-profile-snapshot-capture` - **Commit Message**: `fix(plan): populate effective_profile_snapshot with resolved profile JSON at plan creation` - **Milestone**: v3.7.0 - **Parent Epic**: #372 ## Background and Context The `Plan` domain model in `src/cleveragents/domain/models/core/plan.py` defines an `effective_profile_snapshot` field intended to capture a **frozen JSON snapshot** of the resolved automation profile at plan creation time. Per `docs/specification.md`, this snapshot is required for audit and reproducibility — so that the exact profile configuration used when the plan was created can be reviewed later, even if the profile is subsequently modified or deleted. The field currently defaults to `"{}"` (empty JSON object), and the code itself acknowledges this is a spec violation: ```python # NOTE: The default '{}' exists for backward compatibility with code # paths that create Plan objects before the snapshot is populated. # New plans SHOULD explicitly set this field to the resolved profile # JSON at creation time; the empty default does not satisfy the spec # intent of capturing a frozen profile for audit purposes. effective_profile_snapshot: str = Field( default="{}", ... ) ``` The `PlanLifecycleService.use_action()` method creates new plans but **never sets `effective_profile_snapshot`** to the actual resolved profile JSON. The `_resolve_plan_profile_ref()` method resolves the profile correctly and returns an `AutomationProfileRef`, but its full JSON representation is never captured in `effective_profile_snapshot`. ## Current Behavior All newly created plans have `effective_profile_snapshot = "{}"` (empty JSON object), making the audit trail incomplete. The resolved profile is stored in `plan.automation_profile` (as an `AutomationProfileRef` with just the name and provenance), but the full profile configuration snapshot is never captured. **Steps to reproduce:** 1. `agents action create --config my-action.yaml` 2. `agents plan use local/my-action my-project` 3. `agents plan status <PLAN_ID> --format json` 4. Observe `effective_profile_snapshot` is `{}` ## Expected Behavior Per spec, `effective_profile_snapshot` must contain a frozen JSON snapshot of the **full resolved automation profile** at plan creation time. This is required for: - **Audit**: Reviewers can inspect the exact profile configuration that was active when the plan was created. - **Reproducibility**: Plans can be re-executed with the same profile configuration even if the profile has since been modified. ## Acceptance Criteria - [ ] `PlanLifecycleService.use_action()` sets `effective_profile_snapshot` to the full JSON serialization of the resolved `AutomationProfile` at plan creation time - [ ] `effective_profile_snapshot` is never `"{}"` for a newly created plan when a profile is resolvable - [ ] `agents plan status <PLAN_ID> --format json` shows a non-empty `effective_profile_snapshot` containing the full profile configuration - [ ] If no profile can be resolved (e.g., no global default), `effective_profile_snapshot` is set to `"{}"` and a warning is logged - [ ] Existing plans with `effective_profile_snapshot = "{}"` are not retroactively modified (backward compatibility preserved) ## Subtasks - [ ] Investigate `PlanLifecycleService.use_action()` and `_resolve_plan_profile_ref()` to understand the full resolved profile object available at plan creation time - [ ] Serialize the resolved `AutomationProfile` to JSON and assign it to `plan.effective_profile_snapshot` in `use_action()` - [ ] Ensure the snapshot is taken **after** profile resolution and **before** the plan is persisted to the database - [ ] Add a guard: if profile resolution returns `None`, log a warning and leave `effective_profile_snapshot` as `"{}"` - [ ] Tests (Behave): Add scenario — newly created plan has non-empty `effective_profile_snapshot` matching the resolved profile - [ ] Tests (Behave): Add scenario — `effective_profile_snapshot` is stable (does not change if the profile is later modified) - [ ] Tests (Robot): Add integration test verifying `agents plan status --format json` shows populated `effective_profile_snapshot` - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass - Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added this to the v3.7.0 milestone

2026-04-03 17:26:46 +00:00

freemo added a new dependency

2026-04-03 17:27:11 +00:00

#372 Legendary: Plan Lifecycle (Workstream A)

freemo referenced this issue

2026-04-03 17:32:31 +00:00

[UAT-SUPERVISOR] Progress Report — Cycle 1501 — v3.7.0 #2398

freemo referenced this issue

2026-04-03 17:32:32 +00:00

[CA-AUTO] UAT Pool Supervisor — v3.6.0 — Session Tracker #2360

freemo referenced this issue

2026-04-03 17:35:12 +00:00

[CA-AUTO] UAT Pool Supervisor — v3.6.0 — Session Tracker #2360

freemo referenced this issue

2026-04-03 17:42:38 +00:00

[UAT-SUPERVISOR] Progress Report — Cycle 1511 — v3.7.0 #2421

freemo referenced this issue

2026-04-03 17:52:45 +00:00

[UAT-SUPERVISOR] Progress Report — Cycle 1521 — v3.7.0 #2422

freemo commented

2026-04-03 18:25:20 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: Medium — effective_profile_snapshot defaulting to {} instead of capturing the resolved profile means audit/reproducibility is broken for plan execution.
Milestone: v3.7.0
MoSCoW: Should Have — Audit/reproducibility is important for production use but does not block plan execution.
Parent Epic: #372 (as specified in issue metadata — note: this Epic may be closed, will verify)

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Medium — `effective_profile_snapshot` defaulting to `{}` instead of capturing the resolved profile means audit/reproducibility is broken for plan execution. - **Milestone**: v3.7.0 - **MoSCoW**: Should Have — Audit/reproducibility is important for production use but does not block plan execution. - **Parent Epic**: #372 (as specified in issue metadata — note: this Epic may be closed, will verify) --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo removed this from the v3.7.0 milestone

2026-04-06 20:58:43 +00:00