UAT: DefaultValidationRunner in validation_apply.py is a stub/placeholder that does not invoke actual validation tools #3816

New issue

Open

opened 2026-04-06 06:42:33 +00:00 by freemo · 0 comments

freemo commented

2026-04-06 06:42:33 +00:00

Owner

Metadata

Branch: fix/backlog-default-validation-runner-stub
Commit Message: fix(validation): replace DefaultValidationRunner stub with real tool execution pipeline
Milestone: (backlog — no milestone)
Parent Epic: #946

Background and Context

The DefaultValidationRunner class in src/cleveragents/application/services/validation_apply.py is explicitly documented as a "stub implementation using text matching" and a "placeholder". This means the apply-phase validation gating (ApplyValidationGate) does not actually execute registered validation tools against resources — it only performs naive text matching against context strings.

Per docs/specification.md, validations are a core safety mechanism: required validations that fail must block plan execution. With only a stub runner in place, this guarantee is never enforced.

Backlog note: This issue was discovered during autonomous operation
on milestone v3.2.0. It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

Current Behavior

The DefaultValidationRunner.run_validation() method (lines ~260–310 in validation_apply.py) uses a simple heuristic: it checks if the validation name appears in the context text via string matching. This is explicitly documented as:

"Stub implementation using text matching"
"This is a placeholder — real implementations invoke the actual validation tool"

class DefaultValidationRunner(ValidationRunner):
    """Default validation runner using text matching.

    For each attachment, checks if the validation name appears in the
    context values. This is a simple default for development and testing.
    Production implementations should use the full tool execution pipeline.
    """

The ApplyValidationGate in validation_apply.py is used in the plan apply service (plan_apply_service.py) and is the primary mechanism for validation gating before apply. With only a stub runner, validations never actually execute their registered code.

Code Locations:

src/cleveragents/application/services/validation_apply.py — DefaultValidationRunner class (lines ~260–310)
src/cleveragents/application/services/validation_apply.py — ApplyValidationGate class (lines ~320–380)
src/cleveragents/application/services/plan_apply_service.py — uses ApplyValidationGate

Steps to Reproduce:

Register a validation: agents validation add --config ./my-validation.yaml
Attach it to a resource: agents validation attach my-resource local/my-validation
Run a plan that uses the resource
Observe that the validation's code is never executed; only text matching is performed

Expected Behavior

Per docs/specification.md (Validation section):

Validations are executed as standard tool calls during the Execute phase
Each applicable validation is invoked and returns a structured JSON result with a mandatory passed boolean
Required validations that fail must block plan execution (gating)
The ApplyValidationGate must invoke the actual registered validation tool implementation, not perform text matching

Impact

Validation gating during plan apply is non-functional
Required validations that should block apply do not actually execute their checks
The spec's core guarantee that "required validation failure blocks execution" is not enforced

Subtasks

Audit DefaultValidationRunner.run_validation() and document all call sites
Design a RealValidationRunner (or replace DefaultValidationRunner) that invokes the actual tool execution pipeline
Wire the real runner into ApplyValidationGate so it calls registered validation tool code
Ensure the runner returns a structured JSON result with a mandatory passed boolean per spec
Ensure required validation failures correctly block plan apply
Tests (Behave): Add unit test scenarios covering real tool invocation, passed=true, passed=false (required blocks, informational does not)
Tests (Robot): Add integration test for end-to-end validation gating during plan apply
Verify coverage ≥ 97% via nox -s coverage_report
Run nox (all default sessions) and fix any errors

Definition of Done

All subtasks above are completed and checked off
DefaultValidationRunner stub is replaced with a real implementation that invokes registered validation tool code
ApplyValidationGate correctly blocks apply when a required validation returns passed: false
A Git commit is created where the first line of the commit message matches the "Commit Message" in Metadata exactly, followed by a blank line, then additional details
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done
All nox stages pass
Coverage ≥ 97%

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/backlog-default-validation-runner-stub` - **Commit Message**: `fix(validation): replace DefaultValidationRunner stub with real tool execution pipeline` - **Milestone**: *(backlog — no milestone)* - **Parent Epic**: #946 ## Background and Context The `DefaultValidationRunner` class in `src/cleveragents/application/services/validation_apply.py` is explicitly documented as a "stub implementation using text matching" and a "placeholder". This means the apply-phase validation gating (`ApplyValidationGate`) does not actually execute registered validation tools against resources — it only performs naive text matching against context strings. Per `docs/specification.md`, validations are a core safety mechanism: required validations that fail must block plan execution. With only a stub runner in place, this guarantee is never enforced. > **Backlog note:** This issue was discovered during autonomous operation > on milestone v3.2.0. It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. ## Current Behavior The `DefaultValidationRunner.run_validation()` method (lines ~260–310 in `validation_apply.py`) uses a simple heuristic: it checks if the validation name appears in the context text via string matching. This is explicitly documented as: - "Stub implementation using text matching" - "This is a placeholder — real implementations invoke the actual validation tool" ```python class DefaultValidationRunner(ValidationRunner): """Default validation runner using text matching. For each attachment, checks if the validation name appears in the context values. This is a simple default for development and testing. Production implementations should use the full tool execution pipeline. """ ``` The `ApplyValidationGate` in `validation_apply.py` is used in the plan apply service (`plan_apply_service.py`) and is the primary mechanism for validation gating before apply. With only a stub runner, validations never actually execute their registered code. **Code Locations:** - `src/cleveragents/application/services/validation_apply.py` — `DefaultValidationRunner` class (lines ~260–310) - `src/cleveragents/application/services/validation_apply.py` — `ApplyValidationGate` class (lines ~320–380) - `src/cleveragents/application/services/plan_apply_service.py` — uses `ApplyValidationGate` **Steps to Reproduce:** 1. Register a validation: `agents validation add --config ./my-validation.yaml` 2. Attach it to a resource: `agents validation attach my-resource local/my-validation` 3. Run a plan that uses the resource 4. Observe that the validation's `code` is never executed; only text matching is performed ## Expected Behavior Per `docs/specification.md` (Validation section): - Validations are executed as standard tool calls during the Execute phase - Each applicable validation is invoked and returns a structured JSON result with a mandatory `passed` boolean - Required validations that fail must block plan execution (gating) - The `ApplyValidationGate` must invoke the actual registered validation tool implementation, not perform text matching ## Impact - Validation gating during plan apply is non-functional - Required validations that should block apply do not actually execute their checks - The spec's core guarantee that "required validation failure blocks execution" is not enforced ## Subtasks - [ ] Audit `DefaultValidationRunner.run_validation()` and document all call sites - [ ] Design a `RealValidationRunner` (or replace `DefaultValidationRunner`) that invokes the actual tool execution pipeline - [ ] Wire the real runner into `ApplyValidationGate` so it calls registered validation tool code - [ ] Ensure the runner returns a structured JSON result with a mandatory `passed` boolean per spec - [ ] Ensure `required` validation failures correctly block plan apply - [ ] Tests (Behave): Add unit test scenarios covering real tool invocation, `passed=true`, `passed=false` (required blocks, informational does not) - [ ] Tests (Robot): Add integration test for end-to-end validation gating during plan apply - [ ] Verify coverage ≥ 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions) and fix any errors ## Definition of Done - [ ] All subtasks above are completed and checked off - [ ] `DefaultValidationRunner` stub is replaced with a real implementation that invokes registered validation tool code - [ ] `ApplyValidationGate` correctly blocks apply when a `required` validation returns `passed: false` - [ ] A Git commit is created where the **first line** of the commit message matches the "Commit Message" in Metadata exactly, followed by a blank line, then additional details - [ ] The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly - [ ] The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done - [ ] All nox stages pass - [ ] Coverage ≥ 97% --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added a new dependency

2026-04-06 06:42:50 +00:00

#946 Epic: Stub Package Cleanup — Remove Ghost Packages and Empty Stub Modules

freemo referenced this issue

2026-04-06 06:46:45 +00:00

[Automated] CleverAgents Build Session - 2026-04-06 #3775

freemo added the

Points

label

2026-04-06 06:50:04 +00:00

freemo referenced this issue

2026-04-06 06:51:25 +00:00