cleveragents/cleveragents-core

Fork 3

UAT: _normalise_executor_output() discards original output on non-dict return — spec requires data.raw_output preservation #5543

New issue

Open

opened 2026-04-09 07:19:07 +00:00 by HAL9000 · 1 comment

HAL9000 commented

2026-04-09 07:19:07 +00:00

Owner

Bug Report

Feature Area: Validation Runner — Validation Error Handling

Severity: Medium — debugging information is lost when validations return malformed output

What Was Tested

Code-level analysis of _normalise_executor_output() in src/cleveragents/application/services/validation_pipeline.py (lines 241–264) against the validation return format enforcement requirements in docs/specification.md (Core Concepts > Validation > Structured Return Format, line 22521).

Expected Behavior (from spec)

The specification (line 22521) states:

Return format enforcement: If a validation returns output that is not valid JSON, or valid JSON that lacks the passed boolean field, the system treats the invocation as an error (distinct from a validation failure). The validation result is recorded as "passed": false with a system-generated message indicating the malformed return, and the original output is preserved in data.raw_output for debugging.

Actual Behavior (from code)

_normalise_executor_output() in validation_pipeline.py lines 249–254:

def _normalise_executor_output(raw: Any) -> dict[str, Any]:
    if not isinstance(raw, dict):
        return {
            "passed": False,
            "message": f"Validation returned non-dict output: {raw!r}",
            "data": None,  # BUG: should be {"raw_output": raw}
        }

When a validation returns non-dict output, data is set to None — the original output is discarded. This violates the spec requirement to preserve the original output in data.raw_output.

Evidence

src/cleveragents/application/services/validation_pipeline.py:249-254: data: None when non-dict output
Spec line 22521: "the original output is preserved in data.raw_output for debugging"

Impact

When a validation tool is misconfigured and returns non-dict output (e.g., a string, integer, or list), the original output is lost. Developers debugging the issue cannot see what the validation actually returned, making it much harder to diagnose and fix the misconfiguration.

Fix Required

Change _normalise_executor_output() to preserve the original output in data.raw_output:

def _normalise_executor_output(raw: Any) -> dict[str, Any]:
    if not isinstance(raw, dict):
        return {
            "passed": False,
            "message": f"Validation returned non-dict output: {raw!r}",
            "data": {"raw_output": raw},  # Preserve original output per spec
        }
    passed = raw.get("passed", False)
    if not isinstance(passed, bool):
        passed = False
    message = raw.get("message", "No message provided")
    if not isinstance(message, str):
        message = str(message)
    data = raw.get("data")
    if data is not None and not isinstance(data, dict):
        data = {"raw": data}
    return {"passed": passed, "message": message, "data": data}

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: Validation Runner — Validation Error Handling **Severity**: Medium — debugging information is lost when validations return malformed output --- ## What Was Tested Code-level analysis of `_normalise_executor_output()` in `src/cleveragents/application/services/validation_pipeline.py` (lines 241–264) against the validation return format enforcement requirements in `docs/specification.md` (Core Concepts > Validation > Structured Return Format, line 22521). ## Expected Behavior (from spec) The specification (line 22521) states: > **Return format enforcement**: If a validation returns output that is not valid JSON, or valid JSON that lacks the `passed` boolean field, the system treats the invocation as an error (distinct from a validation failure). The validation result is recorded as `"passed": false` with a system-generated `message` indicating the malformed return, and **the original output is preserved in `data.raw_output` for debugging**. ## Actual Behavior (from code) `_normalise_executor_output()` in `validation_pipeline.py` lines 249–254: ```python def _normalise_executor_output(raw: Any) -> dict[str, Any]: if not isinstance(raw, dict): return { "passed": False, "message": f"Validation returned non-dict output: {raw!r}", "data": None, # BUG: should be {"raw_output": raw} } ``` When a validation returns non-dict output, `data` is set to `None` — the original output is **discarded**. This violates the spec requirement to preserve the original output in `data.raw_output`. ## Evidence - `src/cleveragents/application/services/validation_pipeline.py:249-254`: `data: None` when non-dict output - Spec line 22521: "the original output is preserved in `data.raw_output` for debugging" ## Impact When a validation tool is misconfigured and returns non-dict output (e.g., a string, integer, or list), the original output is lost. Developers debugging the issue cannot see what the validation actually returned, making it much harder to diagnose and fix the misconfiguration. ## Fix Required Change `_normalise_executor_output()` to preserve the original output in `data.raw_output`: ```python def _normalise_executor_output(raw: Any) -> dict[str, Any]: if not isinstance(raw, dict): return { "passed": False, "message": f"Validation returned non-dict output: {raw!r}", "data": {"raw_output": raw}, # Preserve original output per spec } passed = raw.get("passed", False) if not isinstance(passed, bool): passed = False message = raw.get("message", "No message provided") if not isinstance(message, str): message = str(message) data = raw.get("data") if data is not None and not isinstance(data, dict): data = {"raw": data} return {"passed": passed, "message": message, "data": data} ``` --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 07:19:38 +00:00

HAL9000 added this to the v3.2.0 milestone

2026-04-09 07:24:08 +00:00

HAL9000 commented

2026-04-09 07:25:31 +00:00

Author

Owner

Label compliance fix applied:

Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: backlog-groomer

Label compliance fix applied: - Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: backlog-groomer

HAL9000 referenced this issue

2026-04-09 07:40:05 +00:00

Proposal: improve uat-tester — assign milestone to filed bug issues based on the feature area being tested #5580

HAL9000 referenced this issue

2026-04-09 07:44:12 +00:00

[AUTO-ARCH] Architecture Review (Cycle 1) #5368

HAL9000 referenced this issue

2026-04-09 08:20:03 +00:00

[AUTO-UAT-POOL] UAT Testing Report (Cycle 1) #5369