cleveragents/cleveragents-core

Fork 3

UAT: `_normalise_executor_output` does not preserve original output in `data.raw_output` — spec requires malformed validation returns to be preserved for debugging #2965

New issue

Open

opened 2026-04-05 02:58:20 +00:00 by freemo · 1 comment

freemo commented

2026-04-05 02:58:20 +00:00

Owner

Metadata

Branch: fix/validation-pipeline-normalise-preserve-raw-output
Commit Message: fix(validation-pipeline): preserve original output in data.raw_output for malformed validation returns
Milestone: v3.7.0
Parent Epic: #394

What Was Tested

The _normalise_executor_output() function in src/cleveragents/application/services/validation_pipeline.py was analyzed against the specification's return format enforcement requirements.

Expected Behavior (from spec)

Per docs/specification.md § Structured Return Format:

"If a validation returns output that is not valid JSON, or valid JSON that lacks the passed boolean field, the system treats the invocation as an error (distinct from a validation failure). The validation result is recorded as "passed": false with a system-generated message indicating the malformed return, and the original output is preserved in data.raw_output for debugging."

So when a validation returns non-dict output (or a dict without passed), the result must be:

{
  "passed": false,
  "message": "<system-generated message>",
  "data": {"raw_output": <original output>}
}

Actual Behavior

In src/cleveragents/application/services/validation_pipeline.py, _normalise_executor_output() (lines 249–254):

if not isinstance(raw, dict):
    return {
        "passed": False,
        "message": f"Validation returned non-dict output: {raw!r}",
        "data": None,  # ← BUG: should be {"raw_output": raw}
    }

When a validation returns non-dict output, data is set to None instead of {"raw_output": raw}. The original output is lost, making debugging impossible.

Similarly, when passed is not a bool (lines 255–257), the original raw dict is not preserved in data.raw_output.

Steps to Reproduce

Create a validation whose run() function returns a non-dict value (e.g., a string "ok")
Execute the validation through the pipeline
Observe: ValidationResult.data is None instead of {"raw_output": "ok"}

Code Location

src/cleveragents/application/services/validation_pipeline.py, _normalise_executor_output(), lines 249–264
Fix: Change "data": None to "data": {"raw_output": raw} for non-dict case
Fix: When passed is not a bool, preserve the original dict in data.raw_output

Severity

Medium — debugging malformed validation returns is significantly impaired without the original output preserved in data.raw_output.

Subtasks

Write a TDD issue-capture Behave scenario (tagged @tdd_expected_fail) demonstrating that non-dict output does NOT currently populate data.raw_output
Write a TDD issue-capture Behave scenario (tagged @tdd_expected_fail) demonstrating that a dict missing passed bool does NOT currently populate data.raw_output
Fix _normalise_executor_output(): change "data": None to "data": {"raw_output": raw} for the non-dict branch
Fix _normalise_executor_output(): preserve the original raw dict in data.raw_output when passed is not a bool
Remove @tdd_expected_fail tags and confirm all Behave unit tests pass
Add/update Robot Framework integration test covering malformed validation return normalisation
Add/update ASV benchmark for _normalise_executor_output if applicable
Run nox (all sessions) and confirm all quality gates pass
Open PR referencing this issue

Definition of Done

All subtasks above are checked
_normalise_executor_output() sets data to {"raw_output": <original>} for all malformed-return branches, matching the spec
Behave unit tests cover both malformed-return branches (non-dict and missing passed bool)
Robot Framework integration test covers the normalisation behaviour end-to-end
All nox stages pass
Coverage >= 97%
PR is merged and this issue is closed

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/validation-pipeline-normalise-preserve-raw-output` - **Commit Message**: `fix(validation-pipeline): preserve original output in data.raw_output for malformed validation returns` - **Milestone**: v3.7.0 - **Parent Epic**: #394 ## What Was Tested The `_normalise_executor_output()` function in `src/cleveragents/application/services/validation_pipeline.py` was analyzed against the specification's return format enforcement requirements. ## Expected Behavior (from spec) Per `docs/specification.md` § Structured Return Format: > "If a validation returns output that is not valid JSON, or valid JSON that lacks the `passed` boolean field, the system treats the invocation as an error (distinct from a validation failure). The validation result is recorded as `"passed": false` with a system-generated `message` indicating the malformed return, and **the original output is preserved in `data.raw_output` for debugging**." So when a validation returns non-dict output (or a dict without `passed`), the result must be: ```json { "passed": false, "message": "<system-generated message>", "data": {"raw_output": <original output>} } ``` ## Actual Behavior In `src/cleveragents/application/services/validation_pipeline.py`, `_normalise_executor_output()` (lines 249–254): ```python if not isinstance(raw, dict): return { "passed": False, "message": f"Validation returned non-dict output: {raw!r}", "data": None, # ← BUG: should be {"raw_output": raw} } ``` When a validation returns non-dict output, `data` is set to `None` instead of `{"raw_output": raw}`. The original output is lost, making debugging impossible. Similarly, when `passed` is not a bool (lines 255–257), the original raw dict is not preserved in `data.raw_output`. ## Steps to Reproduce 1. Create a validation whose `run()` function returns a non-dict value (e.g., a string `"ok"`) 2. Execute the validation through the pipeline 3. Observe: `ValidationResult.data` is `None` instead of `{"raw_output": "ok"}` ## Code Location - `src/cleveragents/application/services/validation_pipeline.py`, `_normalise_executor_output()`, lines 249–264 - Fix: Change `"data": None` to `"data": {"raw_output": raw}` for non-dict case - Fix: When `passed` is not a bool, preserve the original dict in `data.raw_output` ## Severity Medium — debugging malformed validation returns is significantly impaired without the original output preserved in `data.raw_output`. ## Subtasks - [ ] Write a TDD issue-capture Behave scenario (tagged `@tdd_expected_fail`) demonstrating that non-dict output does NOT currently populate `data.raw_output` - [ ] Write a TDD issue-capture Behave scenario (tagged `@tdd_expected_fail`) demonstrating that a dict missing `passed` bool does NOT currently populate `data.raw_output` - [ ] Fix `_normalise_executor_output()`: change `"data": None` to `"data": {"raw_output": raw}` for the non-dict branch - [ ] Fix `_normalise_executor_output()`: preserve the original raw dict in `data.raw_output` when `passed` is not a bool - [ ] Remove `@tdd_expected_fail` tags and confirm all Behave unit tests pass - [ ] Add/update Robot Framework integration test covering malformed validation return normalisation - [ ] Add/update ASV benchmark for `_normalise_executor_output` if applicable - [ ] Run `nox` (all sessions) and confirm all quality gates pass - [ ] Open PR referencing this issue ## Definition of Done - [ ] All subtasks above are checked - [ ] `_normalise_executor_output()` sets `data` to `{"raw_output": <original>}` for all malformed-return branches, matching the spec - [ ] Behave unit tests cover both malformed-return branches (non-dict and missing `passed` bool) - [ ] Robot Framework integration test covers the normalisation behaviour end-to-end - [ ] All nox stages pass - [ ] Coverage >= 97% - [ ] PR is merged and this issue is closed --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added this to the v3.7.0 milestone

2026-04-05 02:58:42 +00:00

freemo added a new dependency

2026-04-05 02:58:55 +00:00

#394 Epic: Decision Framework

freemo commented

2026-04-05 03:00:23 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: Confirmed
MoSCoW: Should Have

Valid finding verified during batch triage.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Confirmed - **MoSCoW**: Should Have Valid finding verified during batch triage. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo referenced this issue

2026-04-05 06:15:22 +00:00

[CA-AUTO] UAT Pool Supervisor — v3.2.0–v3.7.0 — Session Tracker #3094

freemo referenced this issue

2026-04-05 07:21:47 +00:00

[CA-AUTO] UAT Pool Supervisor — v3.2.0–v3.7.0 — Session Tracker #3094

freemo referenced this issue

2026-04-05 07:37:16 +00:00

[CA-AUTO] Implementation Pool Supervisor — v3.2.0–v3.7.0 — Session Tracker — 2026-04-05 #3194

freemo referenced this issue

2026-04-05 17:35:29 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377