cleveragents-core/docs/adr/ADR-013-validation-abstraction.md

adr_number, title, status_history, tier, authors, superseded_by, related_adrs, acceptance

adr_number	title	status_history	tier	authors	superseded_by	related_adrs	acceptance
13	Validation Abstraction	2026-02-16 Proposed Jeffrey Phillips Freeman 2026-02-16 Accepted Jeffrey Phillips Freeman	2	Jeffrey Phillips Freeman	null	number title relationship 6 Plan Lifecycle Validations gate phase transitions and can trigger phase reversion number title relationship 11 Tool System Validations are a specialized Tool subtype inheriting the tool lifecycle number title relationship 12 Skill System Validation tools can be composed into skills alongside regular tools number title relationship 16 Invariant System Invariant enforcement uses validation tools as its execution mechanism	votes_for votes_against abstentions voter comment Jeffrey Phillips Freeman <Jeffrey.Freeman@CleverThis.com> Modeling validation as a Tool subtype with structured pass/fail keeps the system uniform and composable

Context

Plans must verify that their work meets quality, correctness, and compliance criteria before changes are applied to real project resources. This verification must be structured (machine-parseable pass/fail results, not free-text), composable (multiple validations can run in parallel), and gating (required validations block Apply if they fail). The system needs a validation mechanism that integrates with the existing tool infrastructure rather than being a separate system.

Decision Drivers

• Verification results must be machine-parseable (structured pass/fail), not free-text LLM interpretation
• Validations must integrate with the existing tool infrastructure rather than creating a parallel system
• Required validations must gate the Apply phase — failed checks block changes from reaching real resources
• Read-only enforcement is essential for safe parallel execution and retry without side effects
• Validation attachment must support multiple scoping levels (direct, project, plan) for fine-grained control
• Existing tools should be reusable as validations via wrapping, avoiding reimplementation

Decision

A Validation is a specialized subtype of Tool designed to verify work. It extends Tool via standard class inheritance, adding mode (required/informational), a structured JSON return format ({ "passed": bool, "message": str, "data": object }), and strict read-only enforcement. Validations share the tool namespace, reuse the entire tool infrastructure, and are composable into skills and actor graphs.

Design

Relationship to Tool

Validations inherit all Tool fields: name, description, source, input_schema, output_schema, capability, resource_slots, lifecycle hooks, code, mcp_server/mcp_tool_name, timeout, idempotent.

Additional validation-specific fields: mode, structured return format enforcement, read-only enforcement, attachment scoping, attachment ULID, wraps (for tool wrapping), and transform.

Validations share the same namespace as tools. They are a superset (validation IS-A tool). Unified management via agents tool and agents validation commands.

Validation Mode

Required (mode: required): Must pass for execution to proceed to Apply. On failure: fix-then-revalidate loop with retry limit (default: 3), then strategy revision, then escalation to user.

Informational (mode: informational): Recorded but does not block execution. Useful for advisory checks and gradual adoption. Mode is set at registration time and cannot be overridden per-attachment.

Structured Return Format

{
  "passed": true,
  "message": "All 247 tests passed, 94% coverage",
  "data": {
    "tests_run": 247,
    "tests_passed": 247,
    "coverage_percent": 94.2
  }
}

passed (boolean) is required. message (string) and data (object) are optional. Malformed returns are treated as errors with passed: false.

Read-Only Enforcement

Validations are always read-only: writes: false, checkpointable: false, read_only: true. Enforced at registration time and at runtime. This guarantee enables safe parallelization, speculative validation, and retry without side-effect concerns.

Attachment Scoping

Validations are attached to resources with three scoping levels:

1. Direct attachment: Always active for that resource, regardless of plan or project.
2. Project-scoped attachment: Active only when the resource is accessed through the specified project.
3. Plan-scoped attachment: Active only for the specified plan.

Attachment resolution: collect all applicable attachments (direct → project → plan), union them, most-specific scope wins for identical arguments. Each attachment gets a system-assigned ULID, enabling the same validation to be attached multiple times with different arguments.

Validation Lifecycle in Plan Execution

Strategize: Not executed, but the strategy actor is aware of applicable validations and plans accordingly.

Execute: Final step of execution. Collect all applicable validations → execute in parallel (safe due to read-only) → process results. All required pass → proceed to Apply. Any required fail → fix-then-revalidate loop.

Apply: Not run. The sandbox model ensures all verification happens during Execute. Apply is the point of no return.

Required Validation Failure Handling

1. Diagnose (examine message and data)
2. Self-fix attempt by the execution actor
3. Re-validation
4. Retry limit reached (default: 3)
5. Request strategy revision (revert to Strategize)
6. Escalate to user
7. Terminal failure if still unresolved

Tool Wrapping

Validations can wrap existing tools using wraps and transform fields. The wraps field references an existing tool; the transform function interprets its output as pass/fail. Wrapped tools inherit input_schema, resource_slots, and timeout from the wrapped tool. The transform is a pure data transformation that receives tool_output and returns the structured validation result. Multiple validations can wrap the same tool with different transforms.

Validations in Actor Graphs and Skills

Validations can appear as tool nodes in actor graphs, with return values available to downstream nodes. Being in a skill makes a validation callable; attachment determines whether it acts as a mandatory gate.

Constraints

• Validations must always be read-only. Registration of a validation with writes: true is rejected.
• The structured return format is enforced. passed (boolean) is required in every return value.
• Validation mode (required / informational) is set at registration and cannot be overridden per-attachment.
• Validations run during Execute, never during Apply.
• Required validation failures must follow the fix-then-revalidate → strategy revision → escalation sequence. No bypassing.
• Wrapped tool transforms must be pure data transformations with no side effects.

Consequences

Positive

• Extending Tool rather than creating a separate system means validations benefit from the entire tool infrastructure — lifecycle, adapters, resource bindings, skills, actor graphs.
• Read-only enforcement enables safe parallel execution of all validations.
• Attachment scoping provides fine-grained control over which validations apply in which contexts.
• The structured return format enables machine-driven fix-then-revalidate loops.
• Tool wrapping enables reusing existing tools as validations without reimplementation.

Negative

• The IS-A relationship with Tool means validation-specific behavior (mode, gating) must be carefully distinguished from general tool behavior in the execution engine.
• Attachment scoping with three levels and system-assigned ULIDs adds management complexity.
• The fix-then-revalidate loop with strategy revision escalation is a complex state machine.

Risks

• Required validations that are flaky (intermittent failures unrelated to code quality) could block plans and erode trust in the system.
• The retry limit is a blunt instrument — some failures need more attempts while others should fail immediately.
• Informational validations may be ignored entirely if there is no mechanism to surface their results prominently.

Alternatives Considered

Separate validation system (not extending Tool) — Would duplicate the lifecycle, adapter, registry, and namespace infrastructure. The specification explicitly chose inheritance to avoid this duplication.

Validations only in Apply phase — Would require running validations on real (non-sandboxed) resources, risking failures after changes are partially committed. The sandbox model requires validation during Execute.

Compliance

• Read-only enforcement tests: Tests verify that validation registration rejects writes: true and that runtime execution blocks any write attempts.
• Structured return tests: Tests verify that well-formed, malformed, and missing return values are handled correctly.
• Attachment scoping tests: Tests verify correct resolution across direct, project-scoped, and plan-scoped attachments.
• Failure handling tests: BDD scenarios exercise the full failure sequence: fix → revalidate → retry limit → strategy revision → escalation → terminal failure.
• Wrapping tests: Tests verify that wrapped tool transforms correctly interpret tool output and that inheritance rules are applied.