cleveragents/cleveragents-core

Fork 3

UAT: `ValidationPipeline` not integrated into plan lifecycle — validations never execute during plan phases #3487

New issue

Open

opened 2026-04-05 18:34:34 +00:00 by freemo · 1 comment

freemo commented

2026-04-05 18:34:34 +00:00

Owner

Metadata

Branch: fix/m3-validation-pipeline-lifecycle-integration
Commit Message: fix(plan-lifecycle): integrate ValidationPipeline into plan lifecycle phases
Milestone: v3.2.0
Parent Epic: #357

Bug Report

Description

The ValidationPipeline service (src/cleveragents/application/services/validation_pipeline.py) is fully implemented but is never called during the plan lifecycle. The PlanLifecycleService (src/cleveragents/application/services/plan_lifecycle_service.py) has no reference to ValidationPipeline at all. This means that even when validations are attached to resources via agents validation attach, they are never executed during the Strategize, Execute, or Apply phases of the plan lifecycle.

Spec Reference

Line 28592: "Validation attachment resolution: Pre-resolve the validations that will apply to this plan (from resource-direct, project, and plan attachment scopes) and verify they are all registered and their tool definitions are valid. This ensures validation failures during Execute are due to the work product, not misconfigured validations."
Line 22808: "final_validation_results | plan.apply | Snapshot of the final validation state at the time execution completed (all required validations passing)."
Line 22478: "The strategy may note that certain validations are informational and thus non-blocking."
Line 22560: "If the validation is informational, the error is still recorded but does not block."

Expected Behavior

During the plan lifecycle (particularly before the Apply phase), the system should:

Resolve all applicable validation attachments for the plan's resources (direct, project-scoped, and plan-scoped)
Execute each validation via ValidationPipeline.run()
Block the Apply phase if any required validations fail (all_required_passed == False)
Record informational failures without blocking
Store the validation summary in plan metadata (validation_summary field)

Actual Behavior

ValidationPipeline.run() is never called during the plan lifecycle. Validations attached to resources via agents validation attach are silently ignored during plan execution. The plan.validation_summary field is never populated from actual validation runs.

Code Locations

src/cleveragents/application/services/plan_lifecycle_service.py — no import or call to ValidationPipeline
src/cleveragents/application/services/validation_pipeline.py — fully implemented but unused in lifecycle
src/cleveragents/application/container.py line 774: validation_pipeline=None — the FixThenRevalidateOrchestrator is wired with no pipeline

Evidence

# In plan_lifecycle_service.py - grep shows ZERO references to ValidationPipeline:
# grep -r "ValidationPipeline" src/cleveragents/application/services/plan_lifecycle_service.py
# (no output)

Impact

Critical — the entire validation subsystem is non-functional at runtime. Required validations that should block plan execution never run, meaning plans can be applied even when their resources fail validation checks. This is a core acceptance criterion for the v3.2.0 milestone.

Subtasks

Audit PlanLifecycleService to identify all phase transition points where validation must be invoked (pre-Strategize resolution, pre-Apply gate)
Inject ValidationPipeline into PlanLifecycleService via the DI container (container.py)
Fix container.py line 774: wire FixThenRevalidateOrchestrator with the real ValidationPipeline instance (remove validation_pipeline=None)
Implement pre-Strategize validation attachment resolution: resolve direct, project-scoped, and plan-scoped attachments; verify all are registered and tool definitions are valid
Implement pre-Apply validation gate: call ValidationPipeline.run() for all resolved validations; block Apply if all_required_passed == False
Record informational validation failures without blocking Apply phase
Populate plan.validation_summary / plan.metadata["validation_summary"] from actual ValidationPipeline.run() results
Write Behave unit tests covering: validations run during lifecycle, required failure blocks Apply, informational failure does not block, summary stored in metadata
Write Robot Framework integration test confirming end-to-end: attach validation → run plan → validation executes → result reflected in plan state
Ensure nox -e typecheck passes (no # type: ignore suppressions)
Ensure coverage >= 97%

Definition of Done

ValidationPipeline.run() is called at the correct lifecycle phase transition points in PlanLifecycleService
FixThenRevalidateOrchestrator in container.py is wired with a real ValidationPipeline instance
Validation attachment resolution (direct, project, plan scopes) occurs before the Strategize phase
Apply phase is blocked when any required validation fails (all_required_passed == False)
Informational validation failures are recorded but do not block Apply
plan.validation_summary is populated from actual ValidationPipeline.run() results
All Behave unit tests pass (nox -e unit_tests)
All Robot Framework integration tests pass (nox -e integration_tests)
All nox stages pass
Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Metadata - **Branch**: `fix/m3-validation-pipeline-lifecycle-integration` - **Commit Message**: `fix(plan-lifecycle): integrate ValidationPipeline into plan lifecycle phases` - **Milestone**: v3.2.0 - **Parent Epic**: #357 ## Bug Report ### Description The `ValidationPipeline` service (`src/cleveragents/application/services/validation_pipeline.py`) is fully implemented but is never called during the plan lifecycle. The `PlanLifecycleService` (`src/cleveragents/application/services/plan_lifecycle_service.py`) has no reference to `ValidationPipeline` at all. This means that even when validations are attached to resources via `agents validation attach`, they are never executed during the Strategize, Execute, or Apply phases of the plan lifecycle. ### Spec Reference - **Line 28592**: "Validation attachment resolution: Pre-resolve the validations that will apply to this plan (from resource-direct, project, and plan attachment scopes) and verify they are all registered and their tool definitions are valid. This ensures validation failures during Execute are due to the work product, not misconfigured validations." - **Line 22808**: "`final_validation_results` | `plan.apply` | Snapshot of the final validation state at the time execution completed (all required validations passing)." - **Line 22478**: "The strategy may note that certain validations are informational and thus non-blocking." - **Line 22560**: "If the validation is informational, the error is still recorded but does not block." ### Expected Behavior During the plan lifecycle (particularly before the Apply phase), the system should: 1. Resolve all applicable validation attachments for the plan's resources (direct, project-scoped, and plan-scoped) 2. Execute each validation via `ValidationPipeline.run()` 3. Block the Apply phase if any `required` validations fail (`all_required_passed == False`) 4. Record informational failures without blocking 5. Store the validation summary in plan metadata (`validation_summary` field) ### Actual Behavior `ValidationPipeline.run()` is never called during the plan lifecycle. Validations attached to resources via `agents validation attach` are silently ignored during plan execution. The `plan.validation_summary` field is never populated from actual validation runs. ### Code Locations - `src/cleveragents/application/services/plan_lifecycle_service.py` — no import or call to `ValidationPipeline` - `src/cleveragents/application/services/validation_pipeline.py` — fully implemented but unused in lifecycle - `src/cleveragents/application/container.py` line 774: `validation_pipeline=None` — the `FixThenRevalidateOrchestrator` is wired with no pipeline ### Evidence ```python # In plan_lifecycle_service.py - grep shows ZERO references to ValidationPipeline: # grep -r "ValidationPipeline" src/cleveragents/application/services/plan_lifecycle_service.py # (no output) ``` ### Impact **Critical** — the entire validation subsystem is non-functional at runtime. Required validations that should block plan execution never run, meaning plans can be applied even when their resources fail validation checks. This is a core acceptance criterion for the v3.2.0 milestone. ## Subtasks - [ ] Audit `PlanLifecycleService` to identify all phase transition points where validation must be invoked (pre-Strategize resolution, pre-Apply gate) - [ ] Inject `ValidationPipeline` into `PlanLifecycleService` via the DI container (`container.py`) - [ ] Fix `container.py` line 774: wire `FixThenRevalidateOrchestrator` with the real `ValidationPipeline` instance (remove `validation_pipeline=None`) - [ ] Implement pre-Strategize validation attachment resolution: resolve direct, project-scoped, and plan-scoped attachments; verify all are registered and tool definitions are valid - [ ] Implement pre-Apply validation gate: call `ValidationPipeline.run()` for all resolved validations; block Apply if `all_required_passed == False` - [ ] Record informational validation failures without blocking Apply phase - [ ] Populate `plan.validation_summary` / `plan.metadata["validation_summary"]` from actual `ValidationPipeline.run()` results - [ ] Write Behave unit tests covering: validations run during lifecycle, required failure blocks Apply, informational failure does not block, summary stored in metadata - [ ] Write Robot Framework integration test confirming end-to-end: attach validation → run plan → validation executes → result reflected in plan state - [ ] Ensure `nox -e typecheck` passes (no `# type: ignore` suppressions) - [ ] Ensure coverage >= 97% ## Definition of Done - [ ] `ValidationPipeline.run()` is called at the correct lifecycle phase transition points in `PlanLifecycleService` - [ ] `FixThenRevalidateOrchestrator` in `container.py` is wired with a real `ValidationPipeline` instance - [ ] Validation attachment resolution (direct, project, plan scopes) occurs before the Strategize phase - [ ] Apply phase is blocked when any `required` validation fails (`all_required_passed == False`) - [ ] Informational validation failures are recorded but do not block Apply - [ ] `plan.validation_summary` is populated from actual `ValidationPipeline.run()` results - [ ] All Behave unit tests pass (`nox -e unit_tests`) - [ ] All Robot Framework integration tests pass (`nox -e integration_tests`) - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo added this to the v3.2.0 milestone

2026-04-05 18:34:38 +00:00

freemo added a new dependency

2026-04-05 18:34:48 +00:00

#357 Epic: Decisions + Validations + Invariants (M3)

freemo referenced this issue

2026-04-05 18:39:51 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 18:45:07 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 18:52:57 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 19:05:02 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo commented

2026-04-05 19:33:25 +00:00

Author

Owner

Issue verified and triaged:

Priority: Critical — the entire validation subsystem is non-functional at runtime. ValidationPipeline is implemented but never called during plan lifecycle.
Milestone: v3.2.0 (already assigned)
Story Points: 5 (L) — requires DI wiring, lifecycle integration at multiple phase transition points, and informational vs required validation handling.
Parent Epic: #357 (already linked)
Dependency: Related to #3490 (ApplyValidationGate stub) and #3482 (tool_type check). All three form the validation subsystem fix cluster.
Next step: This issue is now ready for implementation.

Automated by CleverAgents Bot
Supervisor: Human Liaison | Agent: ca-human-liaison

Issue verified and triaged: - **Priority**: Critical — the entire validation subsystem is non-functional at runtime. ValidationPipeline is implemented but never called during plan lifecycle. - **Milestone**: v3.2.0 (already assigned) - **Story Points**: 5 (L) — requires DI wiring, lifecycle integration at multiple phase transition points, and informational vs required validation handling. - **Parent Epic**: #357 (already linked) - **Dependency**: Related to #3490 (ApplyValidationGate stub) and #3482 (tool_type check). All three form the validation subsystem fix cluster. - **Next step**: This issue is now ready for implementation. --- **Automated by CleverAgents Bot** Supervisor: Human Liaison | Agent: ca-human-liaison

freemo referenced this issue

2026-04-05 19:33:28 +00:00

UAT: ApplyValidationGate uses DefaultValidationRunner stub — apply-phase validation does not execute registered validation tools #3490

freemo referenced this issue

2026-04-05 19:33:30 +00:00

UAT: agents validation attach accepts plain tools (tool_type="tool") — spec requires rejection #3482

freemo referenced this issue

2026-04-05 19:49:58 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377