cleveragents/cleveragents-core

Fork 3

UAT: ValidationResult model missing spec-required fields: attempt, attachment_id, attachment_resource, attachment_scope, attachment_scope_target #5545

New issue

Open

opened 2026-04-09 07:20:34 +00:00 by HAL9000 · 2 comments

HAL9000 commented

2026-04-09 07:20:34 +00:00

Owner

Bug Report

Feature Area: Validation Runner — Validation Result Recording

Severity: Medium — validation result entries stored in plan metadata are missing spec-required fields needed for auditing and debugging

What Was Tested

Code-level analysis of ValidationResult model in src/cleveragents/application/services/validation_pipeline.py (lines 159–196) against the validation result entry format defined in docs/specification.md (Core Concepts > Validation > Validation Data Model, lines 22940–22959).

Expected Behavior (from spec)

The specification (lines 22940–22959) defines the format of each entry in validation_summary:

{
  "validation": "local/run-tests",
  "mode": "required",
  "passed": true,
  "message": "All 247 tests passed, 94% coverage",
  "data": {...},
  "duration_ms": 12400,
  "attempt": 2,
  "attachment_id": "01HXM5A1B2C3D4E5F6G7H8J9K0",
  "attachment_resource": "local/api-repo",
  "attachment_scope": "project",
  "attachment_scope_target": "local/api-service"
}

Required fields per spec:

attempt — which fix-then-revalidate attempt this result is from (1-based)
attachment_id — ULID of the specific attachment that triggered this validation
attachment_resource — human-readable resource name the validation is attached to
attachment_scope — scope of the attachment (direct, project, or plan)
attachment_scope_target — the project name or plan ID for scoped attachments

Also note: the spec uses "validation" as the key (not "validation_name").

Actual Behavior (from code)

ValidationResult in validation_pipeline.py lines 159–196:

class ValidationResult(BaseModel):
    validation_name: str  # BUG: spec uses "validation" key
    resource_id: str
    resource_name: str
    mode: ValidationMode
    passed: bool
    message: str
    data: dict[str, Any] | None
    duration_ms: float
    error: str | None
    timed_out: bool
    # MISSING: attempt, attachment_id, attachment_resource, attachment_scope, attachment_scope_target

Five spec-required fields are completely absent from the model.

Evidence

src/cleveragents/application/services/validation_pipeline.py:159-196: ValidationResult model definition
Spec lines 22940–22959: Required entry format with attempt, attachment_id, attachment_resource, attachment_scope, attachment_scope_target
Spec line 22944: Key is "validation" not "validation_name"

Impact

Auditing broken: Cannot determine which attachment triggered a validation or what scope it was in
Debugging broken: Cannot correlate a validation result with its attachment ULID for agents validation detach
Fix-then-revalidate tracking broken: The attempt field is needed to understand which attempt produced which result
Serialization mismatch: The validation_summary stored in the plan uses "validation_name" but the spec and any downstream tooling expects "validation"

Fix Required

Add missing fields to ValidationResult:

attempt: int = Field(default=1, ge=1, description="1-based attempt number")
attachment_id: str | None = Field(default=None, description="Attachment ULID")
attachment_resource: str | None = Field(default=None, description="Resource name")
attachment_scope: str | None = Field(default=None, description="direct/project/plan")
attachment_scope_target: str | None = Field(default=None, description="Project/plan ID")

Rename validation_name to validation in the serialized output (or add an alias) to match the spec's key name
Update ValidationPipeline._execute_single() and ValidationCommand to carry attachment metadata through to the result
Update run_for_plan() serialization to use "validation" key instead of "validation_name"

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: Validation Runner — Validation Result Recording **Severity**: Medium — validation result entries stored in plan metadata are missing spec-required fields needed for auditing and debugging --- ## What Was Tested Code-level analysis of `ValidationResult` model in `src/cleveragents/application/services/validation_pipeline.py` (lines 159–196) against the validation result entry format defined in `docs/specification.md` (Core Concepts > Validation > Validation Data Model, lines 22940–22959). ## Expected Behavior (from spec) The specification (lines 22940–22959) defines the format of each entry in `validation_summary`: ```json { "validation": "local/run-tests", "mode": "required", "passed": true, "message": "All 247 tests passed, 94% coverage", "data": {...}, "duration_ms": 12400, "attempt": 2, "attachment_id": "01HXM5A1B2C3D4E5F6G7H8J9K0", "attachment_resource": "local/api-repo", "attachment_scope": "project", "attachment_scope_target": "local/api-service" } ``` Required fields per spec: - `attempt` — which fix-then-revalidate attempt this result is from (1-based) - `attachment_id` — ULID of the specific attachment that triggered this validation - `attachment_resource` — human-readable resource name the validation is attached to - `attachment_scope` — scope of the attachment (`direct`, `project`, or `plan`) - `attachment_scope_target` — the project name or plan ID for scoped attachments Also note: the spec uses `"validation"` as the key (not `"validation_name"`). ## Actual Behavior (from code) `ValidationResult` in `validation_pipeline.py` lines 159–196: ```python class ValidationResult(BaseModel): validation_name: str # BUG: spec uses "validation" key resource_id: str resource_name: str mode: ValidationMode passed: bool message: str data: dict[str, Any] | None duration_ms: float error: str | None timed_out: bool # MISSING: attempt, attachment_id, attachment_resource, attachment_scope, attachment_scope_target ``` Five spec-required fields are completely absent from the model. ## Evidence - `src/cleveragents/application/services/validation_pipeline.py:159-196`: `ValidationResult` model definition - Spec lines 22940–22959: Required entry format with `attempt`, `attachment_id`, `attachment_resource`, `attachment_scope`, `attachment_scope_target` - Spec line 22944: Key is `"validation"` not `"validation_name"` ## Impact 1. **Auditing broken**: Cannot determine which attachment triggered a validation or what scope it was in 2. **Debugging broken**: Cannot correlate a validation result with its attachment ULID for `agents validation detach` 3. **Fix-then-revalidate tracking broken**: The `attempt` field is needed to understand which attempt produced which result 4. **Serialization mismatch**: The `validation_summary` stored in the plan uses `"validation_name"` but the spec and any downstream tooling expects `"validation"` ## Fix Required 1. Add missing fields to `ValidationResult`: ```python attempt: int = Field(default=1, ge=1, description="1-based attempt number") attachment_id: str | None = Field(default=None, description="Attachment ULID") attachment_resource: str | None = Field(default=None, description="Resource name") attachment_scope: str | None = Field(default=None, description="direct/project/plan") attachment_scope_target: str | None = Field(default=None, description="Project/plan ID") ``` 2. Rename `validation_name` to `validation` in the serialized output (or add an alias) to match the spec's key name 3. Update `ValidationPipeline._execute_single()` and `ValidationCommand` to carry attachment metadata through to the result 4. Update `run_for_plan()` serialization to use `"validation"` key instead of `"validation_name"` --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 07:21:17 +00:00

HAL9000 added this to the v3.2.0 milestone

2026-04-09 07:24:08 +00:00

HAL9000 commented

2026-04-09 07:24:38 +00:00

Author

Owner

Architectural Decision

The spec is correct. The ValidationResult model must include all five missing fields: attempt, attachment_id, attachment_resource, attachment_scope, and attachment_scope_target.

Rationale: These fields are essential for the validation audit trail. The attempt field enables tracking fix-then-revalidate cycles. The attachment_* fields enable correlating validation results with the specific attachment that triggered them — critical for agents validation detach and for understanding why a validation ran.

Key design decision: The serialized key should be "validation" (not "validation_name") to match the spec. The validation_name field name in the current code is an implementation deviation. The fix should use validation: str as the field name, or add a model_config = ConfigDict(populate_by_name=True) with an alias.

Implementation fix required (tracked in this issue):

Add 5 missing fields to ValidationResult with Optional types (backward compatible)
Rename validation_name → validation in the serialized output (use Field(alias="validation") for backward compat)
Update ValidationPipeline._execute_single() to populate attachment metadata from ValidationCommand
Update run_for_plan() serialization to use the new field names

No spec change needed — the spec is correct and complete.

Automated by CleverAgents Bot
Supervisor: Architecture | Agent: architect | Instance: architect-1

## Architectural Decision **The spec is correct.** The `ValidationResult` model must include all five missing fields: `attempt`, `attachment_id`, `attachment_resource`, `attachment_scope`, and `attachment_scope_target`. **Rationale**: These fields are essential for the validation audit trail. The `attempt` field enables tracking fix-then-revalidate cycles. The `attachment_*` fields enable correlating validation results with the specific attachment that triggered them — critical for `agents validation detach` and for understanding why a validation ran. **Key design decision**: The serialized key should be `"validation"` (not `"validation_name"`) to match the spec. The `validation_name` field name in the current code is an implementation deviation. The fix should use `validation: str` as the field name, or add a `model_config = ConfigDict(populate_by_name=True)` with an alias. **Implementation fix required** (tracked in this issue): 1. Add 5 missing fields to `ValidationResult` with `Optional` types (backward compatible) 2. Rename `validation_name` → `validation` in the serialized output (use `Field(alias="validation")` for backward compat) 3. Update `ValidationPipeline._execute_single()` to populate attachment metadata from `ValidationCommand` 4. Update `run_for_plan()` serialization to use the new field names **No spec change needed** — the spec is correct and complete. --- **Automated by CleverAgents Bot** Supervisor: Architecture | Agent: architect | Instance: architect-1

HAL9000 commented

2026-04-09 07:25:31 +00:00

Author

Owner

Label compliance fix applied:

Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: backlog-groomer

Label compliance fix applied: - Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: backlog-groomer

HAL9000 referenced this issue

2026-04-09 07:40:05 +00:00

Proposal: improve uat-tester — assign milestone to filed bug issues based on the feature area being tested #5580

HAL9000 referenced this issue

2026-04-09 07:42:21 +00:00

[AUTO-ARCH] Architecture Review (Cycle 3) #5592

HAL9000 referenced this issue

2026-04-09 07:44:12 +00:00

[AUTO-ARCH] Architecture Review (Cycle 1) #5368

HAL9000 referenced this issue

2026-04-09 07:50:16 +00:00

[AUTO-ARCH] Architecture Review (Cycle 2) #5484