cleveragents/cleveragents-core

Fork 3

UAT: validation_attempts, validation_fix_history, and final_validation_results not persisted to plan data model or database #5540

New issue

Open

opened 2026-04-09 07:17:58 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-09 07:17:58 +00:00

Owner

Bug Report

Feature Area: Validation Runner — Validation Result Recording / Plan Data Model

Severity: Critical — spec-required audit fields are computed but never stored, making them unavailable for plan inspection and auditing

What Was Tested

Code-level analysis of fix_then_revalidate.py, plan.py (domain model), and infrastructure/database/models.py against the validation data model requirements in docs/specification.md (Core Concepts > Validation > Validation Data Model section, lines 22929–22960).

Expected Behavior (from spec)

The specification (lines 22931–22938) defines four validation-related fields that must be stored in the plan data model:

Field	Location	Description
`validation_summary`	`plan.execution`	Array of validation results collected during Execute
`final_validation_results`	`plan.apply`	Snapshot of the final validation state at execution completion
`validation_attempts`	`plan.execution`	Total number of validation invocations across all fix-then-revalidate loops
`validation_fix_history`	`plan.execution`	Per-validation log of fix attempts

Actual Behavior (from code)

Only validation_summary is stored in the plan domain model and database:

validation_summary: ✓ Stored in Plan.validation_summary (domain model) and LifecyclePlanModel.validation_summary_json (DB)
final_validation_results: ✗ Computed in FixThenRevalidateResult.final_validation_results but never persisted to the plan model or DB
validation_attempts: ✗ Computed in FixThenRevalidateResult.validation_attempts but never persisted to the plan model or DB
validation_fix_history: ✗ Computed in FixThenRevalidateResult.validation_fix_history but never persisted to the plan model or DB

Evidence

fix_then_revalidate.py computes all four fields in FixThenRevalidateResult:

class FixThenRevalidateResult(BaseModel):
    validation_attempts: int
    validation_fix_history: list[FixAttemptRecord]
    validation_summary: list[ValidationResult]
    final_validation_results: list[ValidationResult]

domain/models/core/plan.py only has validation_summary:

validation_summary: dict[str, Any] | None = Field(...)
# validation_attempts, validation_fix_history, final_validation_results are MISSING

infrastructure/database/models.py only has validation_summary_json:

validation_summary_json = Column(Text, nullable=True)
# No columns for validation_attempts, validation_fix_history, final_validation_results

Impact

Auditing broken: agents plan show cannot display how many fix attempts were made or what fixes were tried
Debugging broken: Users cannot inspect the fix-then-revalidate history to understand why a plan failed
Spec compliance broken: The validation_attempts counter (spec line 22937) is unavailable for plan summaries
final_validation_results (spec line 22936) — the "quick reference" snapshot for the apply phase — is never stored

Fix Required

Add validation_attempts: int | None, validation_fix_history: list[dict] | None, and final_validation_results: list[dict] | None fields to Plan domain model in domain/models/core/plan.py
Add corresponding columns to LifecyclePlanModel in infrastructure/database/models.py (as JSON columns)
Add Alembic migration for the new columns
Wire FixThenRevalidateOrchestrator.run_fix_loop() result into plan persistence — after the fix loop completes, persist validation_attempts, validation_fix_history, and final_validation_results to the plan record
Update plan_executor.py or the calling service to persist these fields after the fix loop

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: Validation Runner — Validation Result Recording / Plan Data Model **Severity**: Critical — spec-required audit fields are computed but never stored, making them unavailable for plan inspection and auditing --- ## What Was Tested Code-level analysis of `fix_then_revalidate.py`, `plan.py` (domain model), and `infrastructure/database/models.py` against the validation data model requirements in `docs/specification.md` (Core Concepts > Validation > Validation Data Model section, lines 22929–22960). ## Expected Behavior (from spec) The specification (lines 22931–22938) defines four validation-related fields that must be stored in the plan data model: | Field | Location | Description | |-------|----------|-------------| | `validation_summary` | `plan.execution` | Array of validation results collected during Execute | | `final_validation_results` | `plan.apply` | Snapshot of the final validation state at execution completion | | `validation_attempts` | `plan.execution` | Total number of validation invocations across all fix-then-revalidate loops | | `validation_fix_history` | `plan.execution` | Per-validation log of fix attempts | ## Actual Behavior (from code) Only `validation_summary` is stored in the plan domain model and database: - **`validation_summary`**: ✓ Stored in `Plan.validation_summary` (domain model) and `LifecyclePlanModel.validation_summary_json` (DB) - **`final_validation_results`**: ✗ Computed in `FixThenRevalidateResult.final_validation_results` but **never persisted** to the plan model or DB - **`validation_attempts`**: ✗ Computed in `FixThenRevalidateResult.validation_attempts` but **never persisted** to the plan model or DB - **`validation_fix_history`**: ✗ Computed in `FixThenRevalidateResult.validation_fix_history` but **never persisted** to the plan model or DB ### Evidence **`fix_then_revalidate.py`** computes all four fields in `FixThenRevalidateResult`: ```python class FixThenRevalidateResult(BaseModel): validation_attempts: int validation_fix_history: list[FixAttemptRecord] validation_summary: list[ValidationResult] final_validation_results: list[ValidationResult] ``` **`domain/models/core/plan.py`** only has `validation_summary`: ```python validation_summary: dict[str, Any] | None = Field(...) # validation_attempts, validation_fix_history, final_validation_results are MISSING ``` **`infrastructure/database/models.py`** only has `validation_summary_json`: ```python validation_summary_json = Column(Text, nullable=True) # No columns for validation_attempts, validation_fix_history, final_validation_results ``` ## Impact 1. **Auditing broken**: `agents plan show` cannot display how many fix attempts were made or what fixes were tried 2. **Debugging broken**: Users cannot inspect the fix-then-revalidate history to understand why a plan failed 3. **Spec compliance broken**: The `validation_attempts` counter (spec line 22937) is unavailable for plan summaries 4. **`final_validation_results`** (spec line 22936) — the "quick reference" snapshot for the apply phase — is never stored ## Fix Required 1. Add `validation_attempts: int | None`, `validation_fix_history: list[dict] | None`, and `final_validation_results: list[dict] | None` fields to `Plan` domain model in `domain/models/core/plan.py` 2. Add corresponding columns to `LifecyclePlanModel` in `infrastructure/database/models.py` (as JSON columns) 3. Add Alembic migration for the new columns 4. Wire `FixThenRevalidateOrchestrator.run_fix_loop()` result into plan persistence — after the fix loop completes, persist `validation_attempts`, `validation_fix_history`, and `final_validation_results` to the plan record 5. Update `plan_executor.py` or the calling service to persist these fields after the fix loop --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 07:18:28 +00:00

HAL9000 added this to the v3.2.0 milestone

2026-04-09 07:24:09 +00:00

HAL9000 commented

2026-04-09 07:25:31 +00:00

Author

Owner

Label compliance fix applied:

Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: backlog-groomer

Label compliance fix applied: - Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: backlog-groomer

HAL9000 referenced this issue

2026-04-09 07:40:05 +00:00

Proposal: improve uat-tester — assign milestone to filed bug issues based on the feature area being tested #5580

HAL9000 referenced this issue

2026-04-09 07:44:12 +00:00

[AUTO-ARCH] Architecture Review (Cycle 1) #5368

HAL9000 referenced this issue

2026-04-09 08:20:03 +00:00

[AUTO-UAT-POOL] UAT Testing Report (Cycle 1) #5369