UAT: ValidationPipeline never invoked during Execute phase — fix_revalidate_orchestrator stored but not called #3280

Open
opened 2026-04-05 09:02:23 +00:00 by freemo · 3 comments
Owner

Background

The spec (docs/specification.md line 22595) states: "attached validations run automatically at the end of Execute (the system handles collection and invocation)". The spec also states (line 22892): "validations execute automatically at end of Execute phase".

PlanExecutor._run_execute_with_stub() and PlanExecutor._run_execute_with_runtime() (in src/cleveragents/application/services/plan_executor.py) never invoke the ValidationPipeline or the fix_revalidate_orchestrator. The fix_revalidate_orchestrator is accepted as a constructor parameter and stored as self._fix_revalidate_orchestrator, but is only exposed as a read-only property — it is never called during the execute flow.

Current Behavior

No validations are run at the end of Execute. The fix_revalidate_orchestrator is stored but never invoked. Plans complete Execute without any validation gate.

Code locations:

  • src/cleveragents/application/services/plan_executor.py_run_execute_with_stub() and _run_execute_with_runtime() (no validation invocation)
  • src/cleveragents/application/services/validation_pipeline.pyValidationPipeline class (exists but unused in execute flow)
  • src/cleveragents/application/services/fix_then_revalidate.pyFixThenRevalidateOrchestrator (exists but never called)

Expected Behavior

At the end of the Execute phase, the system should:

  1. Collect all applicable validations for each resource the plan accessed
  2. Run required validations via ValidationPipeline
  3. Run informational validations via ValidationPipeline
  4. If required validations fail, invoke FixThenRevalidateOrchestrator.run_fix_loop() to attempt self-correction
  5. Block plan completion if required validations cannot be resolved

Steps to Reproduce

  1. Inspect PlanExecutor._run_execute_with_stub() in src/cleveragents/application/services/plan_executor.py
  2. Note that self._fix_revalidate_orchestrator is never called
  3. Inspect PlanExecutor._run_execute_with_runtime() — same issue
  4. The ValidationPipeline class exists in src/cleveragents/application/services/validation_pipeline.py but is never imported or used in plan_executor.py

Impact

  • The v3.2.0 milestone acceptance criterion "Validation runner executes required and informational validations during Execute" is not met
  • Plans can complete Execute with broken code/invalid state — no quality gate
  • The fix-then-revalidate loop for self-correction is completely bypassed

Metadata

  • Branch: fix/m3-validation-pipeline-execute-invocation
  • Commit Message: fix(plan-executor): invoke ValidationPipeline and FixThenRevalidateOrchestrator at end of Execute phase
  • Milestone: v3.2.0
  • Parent Epic: #375

Subtasks

  • Write a failing BDD scenario (Gherkin) in features/ that proves ValidationPipeline is not invoked at end of Execute
  • Write a failing BDD scenario that proves FixThenRevalidateOrchestrator.run_fix_loop() is not called when required validations fail
  • Import ValidationPipeline into plan_executor.py
  • Add validation collection step at end of _run_execute_with_stub() — collect applicable validations for each accessed resource
  • Add ValidationPipeline invocation (required validations) at end of _run_execute_with_stub()
  • Add ValidationPipeline invocation (informational validations) at end of _run_execute_with_stub()
  • Add FixThenRevalidateOrchestrator.run_fix_loop() invocation when required validations fail in _run_execute_with_stub()
  • Mirror all the above changes in _run_execute_with_runtime()
  • Block plan completion if required validations cannot be resolved after fix loop
  • Ensure all nox stages pass
  • Verify coverage >= 97%

Definition of Done

  • A BDD scenario exists that proves ValidationPipeline is invoked at the end of Execute for both stub and runtime paths
  • A BDD scenario exists that proves FixThenRevalidateOrchestrator.run_fix_loop() is called when required validations fail
  • A BDD scenario exists that proves plan completion is blocked when required validations cannot be resolved
  • ValidationPipeline is imported and called in both _run_execute_with_stub() and _run_execute_with_runtime()
  • FixThenRevalidateOrchestrator.run_fix_loop() is called when required validations fail
  • Required validation failures block plan completion
  • Informational validation failures do not block plan completion
  • All nox stages pass
  • Coverage >= 97%

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Background The spec (`docs/specification.md` line 22595) states: *"attached validations run automatically at the end of Execute (the system handles collection and invocation)"*. The spec also states (line 22892): *"validations execute automatically at end of Execute phase"*. `PlanExecutor._run_execute_with_stub()` and `PlanExecutor._run_execute_with_runtime()` (in `src/cleveragents/application/services/plan_executor.py`) never invoke the `ValidationPipeline` or the `fix_revalidate_orchestrator`. The `fix_revalidate_orchestrator` is accepted as a constructor parameter and stored as `self._fix_revalidate_orchestrator`, but is only exposed as a read-only property — it is never called during the execute flow. ## Current Behavior No validations are run at the end of Execute. The `fix_revalidate_orchestrator` is stored but never invoked. Plans complete Execute without any validation gate. Code locations: - `src/cleveragents/application/services/plan_executor.py` — `_run_execute_with_stub()` and `_run_execute_with_runtime()` (no validation invocation) - `src/cleveragents/application/services/validation_pipeline.py` — `ValidationPipeline` class (exists but unused in execute flow) - `src/cleveragents/application/services/fix_then_revalidate.py` — `FixThenRevalidateOrchestrator` (exists but never called) ## Expected Behavior At the end of the Execute phase, the system should: 1. Collect all applicable validations for each resource the plan accessed 2. Run required validations via `ValidationPipeline` 3. Run informational validations via `ValidationPipeline` 4. If required validations fail, invoke `FixThenRevalidateOrchestrator.run_fix_loop()` to attempt self-correction 5. Block plan completion if required validations cannot be resolved ## Steps to Reproduce 1. Inspect `PlanExecutor._run_execute_with_stub()` in `src/cleveragents/application/services/plan_executor.py` 2. Note that `self._fix_revalidate_orchestrator` is never called 3. Inspect `PlanExecutor._run_execute_with_runtime()` — same issue 4. The `ValidationPipeline` class exists in `src/cleveragents/application/services/validation_pipeline.py` but is never imported or used in `plan_executor.py` ## Impact - The v3.2.0 milestone acceptance criterion "Validation runner executes required and informational validations during Execute" is not met - Plans can complete Execute with broken code/invalid state — no quality gate - The fix-then-revalidate loop for self-correction is completely bypassed ## Metadata - **Branch**: `fix/m3-validation-pipeline-execute-invocation` - **Commit Message**: `fix(plan-executor): invoke ValidationPipeline and FixThenRevalidateOrchestrator at end of Execute phase` - **Milestone**: v3.2.0 - **Parent Epic**: #375 ## Subtasks - [ ] Write a failing BDD scenario (Gherkin) in `features/` that proves `ValidationPipeline` is not invoked at end of Execute - [ ] Write a failing BDD scenario that proves `FixThenRevalidateOrchestrator.run_fix_loop()` is not called when required validations fail - [ ] Import `ValidationPipeline` into `plan_executor.py` - [ ] Add validation collection step at end of `_run_execute_with_stub()` — collect applicable validations for each accessed resource - [ ] Add `ValidationPipeline` invocation (required validations) at end of `_run_execute_with_stub()` - [ ] Add `ValidationPipeline` invocation (informational validations) at end of `_run_execute_with_stub()` - [ ] Add `FixThenRevalidateOrchestrator.run_fix_loop()` invocation when required validations fail in `_run_execute_with_stub()` - [ ] Mirror all the above changes in `_run_execute_with_runtime()` - [ ] Block plan completion if required validations cannot be resolved after fix loop - [ ] Ensure all nox stages pass - [ ] Verify coverage >= 97% ## Definition of Done - [ ] A BDD scenario exists that proves `ValidationPipeline` is invoked at the end of Execute for both stub and runtime paths - [ ] A BDD scenario exists that proves `FixThenRevalidateOrchestrator.run_fix_loop()` is called when required validations fail - [ ] A BDD scenario exists that proves plan completion is blocked when required validations cannot be resolved - [ ] `ValidationPipeline` is imported and called in both `_run_execute_with_stub()` and `_run_execute_with_runtime()` - [ ] `FixThenRevalidateOrchestrator.run_fix_loop()` is called when required validations fail - [ ] Required validation failures block plan completion - [ ] Informational validation failures do not block plan completion - [ ] All nox stages pass - [ ] Coverage >= 97% --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator
freemo added this to the v3.2.0 milestone 2026-04-05 09:04:21 +00:00
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: Critical (keeping existing — validation pipeline is a core v3.2.0 deliverable)
  • Milestone: v3.2.0 (already assigned, keeping)
  • MoSCoW: Must Have — v3.2.0 acceptance criteria requires validation during Execute. The ValidationPipeline and FixThenRevalidateOrchestrator exist but are never invoked, meaning plans complete without any quality gate.
  • Parent Epic: #375 (as noted in issue body)

This is another critical gap in the v3.2.0 subsystem — alongside the correction flow bugs, the validation pipeline is also non-functional.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Critical (keeping existing — validation pipeline is a core v3.2.0 deliverable) - **Milestone**: v3.2.0 (already assigned, keeping) - **MoSCoW**: Must Have — v3.2.0 acceptance criteria requires validation during Execute. The ValidationPipeline and FixThenRevalidateOrchestrator exist but are never invoked, meaning plans complete without any quality gate. - **Parent Epic**: #375 (as noted in issue body) This is another critical gap in the v3.2.0 subsystem — alongside the correction flow bugs, the validation pipeline is also non-functional. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Author
Owner

Issue triaged by project owner — confirming existing triage:

  • State: Verified (already set)
  • Priority: Critical — ValidationPipeline is completely disconnected from the Execute flow. Plans complete without any quality gate.
  • Milestone: v3.2.0 — validation during Execute is a core v3.2.0 acceptance criterion
  • MoSCoW: Must Have — the spec explicitly states "attached validations run automatically at the end of Execute." This is a blocking acceptance criterion for v3.2.0.
  • Parent Epic: #375 (referenced in issue body)

This is the second Critical/Must Have bug found in this triage session. Combined with #3270 (MergeConflictError swallowed in v3.3.0), these two issues represent the most significant blockers across the active milestones.


Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner — confirming existing triage: - **State**: Verified ✅ (already set) - **Priority**: Critical ✅ — ValidationPipeline is completely disconnected from the Execute flow. Plans complete without any quality gate. - **Milestone**: v3.2.0 ✅ — validation during Execute is a core v3.2.0 acceptance criterion - **MoSCoW**: Must Have ✅ — the spec explicitly states "attached validations run automatically at the end of Execute." This is a blocking acceptance criterion for v3.2.0. - **Parent Epic**: #375 (referenced in issue body) This is the **second Critical/Must Have bug** found in this triage session. Combined with #3270 (MergeConflictError swallowed in v3.3.0), these two issues represent the most significant blockers across the active milestones. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
Author
Owner

Issue triaged by project owner:

  • State: Verified
  • Priority: Critical
  • MoSCoW: Must Have — ValidationPipeline not being invoked during Execute phase means validation-gated apply is non-functional. This directly blocks the v3.5.0 acceptance criterion: "validation-gated apply"
  • Parent Epic: #397 (Server & Autonomy Infrastructure)

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Critical - **MoSCoW**: Must Have — ValidationPipeline not being invoked during Execute phase means validation-gated apply is non-functional. This directly blocks the v3.5.0 acceptance criterion: "validation-gated apply" - **Parent Epic**: #397 (Server & Autonomy Infrastructure) --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner
freemo removed this from the v3.2.0 milestone 2026-04-07 00:10:56 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
Reference
cleveragents/cleveragents-core#3280
No description provided.