cleveragents/cleveragents-core

Fork 3

UAT: `plan.validation_summary` never populated by `PlanExecutor` — `apply_with_validation_gate()` always sees 0 validations and passes unconditionally #6045

New issue

Open

opened 2026-04-09 14:04:53 +00:00 by HAL9000 · 2 comments

HAL9000 commented

2026-04-09 14:04:53 +00:00

Owner

Summary

PlanApplyService.apply_with_validation_gate() reads plan.validation_summary to determine whether required validations passed. However, PlanExecutor never writes to plan.validation_summary during the Execute phase. As a result, the apply gate always sees vs = None, treats it as "0 validations run", and unconditionally allows apply to proceed — even when validations should have blocked it.

What Was Tested

Code-level analysis of plan_executor.py and plan_apply_service.py
Runtime test: executed a plan, checked plan.validation_summary — always None

Expected Behavior

After the Execute phase completes, plan.validation_summary should contain a dict with:

{
  "total": N,
  "required_passed": N,
  "required_failed": N,
  "informational_passed": N,
  "informational_failed": N,
  "all_required_passed": true/false,
  "results": [...]
}

This is produced by ValidationPipeline.run_for_plan(plan_metadata) and should be persisted to the plan before complete_execute() is called.

Actual Behavior

plan_apply_service.py line ~559:

vs = validation_summary or plan.validation_summary
req_passed, req_failed, total = self._extract_validation_counts(vs)

_extract_validation_counts(None) returns (0, 0, 0) — no required failures → gate passes.

Since PlanExecutor never calls ValidationPipeline.run_for_plan(), plan.validation_summary is always None.

Impact

The apply validation gate is effectively disabled
Plans with failed required validations can be applied without any validation
The spec requirement "Apply refused — N required Execute-phase validations did not pass" is never triggered

Code Location

src/cleveragents/application/services/plan_apply_service.py — apply_with_validation_gate() line ~559
src/cleveragents/application/services/plan_apply_service.py — _extract_validation_counts() line ~660
src/cleveragents/application/services/plan_executor.py — _run_execute_with_stub() and _run_execute_with_runtime() (never set plan.validation_summary)

Relationship

This is a consequence of #6016 (ValidationPipeline not integrated into PlanExecutor). Fixing #6016 will also fix this issue, as run_for_plan() populates plan_metadata["validation_summary"] which should then be persisted to plan.validation_summary.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Summary `PlanApplyService.apply_with_validation_gate()` reads `plan.validation_summary` to determine whether required validations passed. However, `PlanExecutor` never writes to `plan.validation_summary` during the Execute phase. As a result, the apply gate always sees `vs = None`, treats it as "0 validations run", and unconditionally allows apply to proceed — even when validations should have blocked it. ## What Was Tested - Code-level analysis of `plan_executor.py` and `plan_apply_service.py` - Runtime test: executed a plan, checked `plan.validation_summary` — always `None` ## Expected Behavior After the Execute phase completes, `plan.validation_summary` should contain a dict with: ```json { "total": N, "required_passed": N, "required_failed": N, "informational_passed": N, "informational_failed": N, "all_required_passed": true/false, "results": [...] } ``` This is produced by `ValidationPipeline.run_for_plan(plan_metadata)` and should be persisted to the plan before `complete_execute()` is called. ## Actual Behavior `plan_apply_service.py` line ~559: ```python vs = validation_summary or plan.validation_summary req_passed, req_failed, total = self._extract_validation_counts(vs) ``` `_extract_validation_counts(None)` returns `(0, 0, 0)` — no required failures → gate passes. Since `PlanExecutor` never calls `ValidationPipeline.run_for_plan()`, `plan.validation_summary` is always `None`. ## Impact - The apply validation gate is effectively disabled - Plans with failed required validations can be applied without any validation - The spec requirement "Apply refused — N required Execute-phase validations did not pass" is never triggered ## Code Location - `src/cleveragents/application/services/plan_apply_service.py` — `apply_with_validation_gate()` line ~559 - `src/cleveragents/application/services/plan_apply_service.py` — `_extract_validation_counts()` line ~660 - `src/cleveragents/application/services/plan_executor.py` — `_run_execute_with_stub()` and `_run_execute_with_runtime()` (never set `plan.validation_summary`) ## Relationship This is a consequence of #6016 (`ValidationPipeline` not integrated into `PlanExecutor`). Fixing #6016 will also fix this issue, as `run_for_plan()` populates `plan_metadata["validation_summary"]` which should then be persisted to `plan.validation_summary`. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 14:09:55 +00:00

HAL9000 referenced this issue

2026-04-09 14:24:08 +00:00

[AUTO-PROJ-OWN] Project Owner Report (Cycle 35) #6073

HAL9000 added this to the v3.2.0 milestone

2026-04-09 14:31:49 +00:00

HAL9000 added

and removed

labels

2026-04-09 14:31:58 +00:00

HAL9000 referenced this issue

2026-04-09 14:36:29 +00:00

[AUTO-PROJ-OWN] Project Owner Report (Cycle 31) #6048

HAL9000 commented

2026-04-09 14:40:50 +00:00

Author

Owner

Label compliance fix applied:

Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: backlog-groomer

Label compliance fix applied: - Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: backlog-groomer

HAL9000 added

and removed

labels

2026-04-09 14:56:04 +00:00

HAL9000 added a new dependency

2026-04-09 15:00:50 +00:00

#4959 EPIC: Correction Engine — Revert & Append Modes with plan.attempt Tracking

HAL9000 commented

2026-04-09 15:05:27 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: Critical — PlanExecutor never writes to plan.validation_summary, so apply_with_validation_gate() always sees 0 validations and passes unconditionally. This means the validation gate is completely bypassed — the apply phase will proceed even when required validations have failed.
Milestone: v3.2.0 — Validation pipeline integration is a core v3.2.0 deliverable (Decisions + Validations + Invariants). The milestone cannot ship with a non-functional validation gate.
Story Points: 5 — L — Requires wiring PlanExecutor.run_execute() to populate plan.validation_summary after running the ValidationPipeline, and ensuring the gate logic reads it correctly.
MoSCoW: Must Have — The spec requires validations to run during Execute phase and gate the Apply phase. This is a fundamental correctness requirement.
Parent Epic: Needs linking to the Validation Pipeline epic under Legendary #375.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Critical — `PlanExecutor` never writes to `plan.validation_summary`, so `apply_with_validation_gate()` always sees 0 validations and passes unconditionally. This means the validation gate is completely bypassed — the apply phase will proceed even when required validations have failed. - **Milestone**: v3.2.0 — Validation pipeline integration is a core v3.2.0 deliverable (Decisions + Validations + Invariants). The milestone cannot ship with a non-functional validation gate. - **Story Points**: 5 — L — Requires wiring `PlanExecutor.run_execute()` to populate `plan.validation_summary` after running the `ValidationPipeline`, and ensuring the gate logic reads it correctly. - **MoSCoW**: Must Have — The spec requires validations to run during Execute phase and gate the Apply phase. This is a fundamental correctness requirement. - **Parent Epic**: Needs linking to the Validation Pipeline epic under Legendary #375. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: project-owner

HAL9000 referenced this issue

2026-04-09 15:13:55 +00:00

[AUTO-EPIC] Epic Planning Health Report (Cycle 21) #5883

HAL9000 referenced this issue

2026-04-09 17:03:23 +00:00

[AUTO-IMP-SUP] Implementation Pool Supervisor Status (Cycle 1) #6167

HAL9000 referenced this issue

2026-04-09 17:25:53 +00:00

[AUTO-WATCHDOG] System Health Report (Cycle 32) #6184