UAT: `agents diagnostics --format json` sets `status: "ok"` even when health checks report errors #6798

New issue

Open

opened 2026-04-10 02:07:06 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-10 02:07:06 +00:00

Owner

Background and Context

The specification requires the JSON/YAML output envelope status field to reflect the actual outcome of diagnostics. When health checks find errors, the status must be "error" and exit_code must be 1. When only warnings are present, status must be "warn".

Current Behavior

When agents diagnostics --format json runs and finds errors (e.g., database directory not writable), the envelope status is still "ok" even though data.has_errors is True:

{
  "command": "",
  "status": "ok",
  "exit_code": 0,
  "data": {
    ...
    "summary": {
      "errors": 1,
      "warnings": 7
    },
    "has_errors": true,
    "has_warnings": true
  },
  ...
}

This means a programmatic consumer checking status or exit_code would falsely conclude the system is healthy when it has critical errors.

Expected Behavior (from spec)

When diagnostics detects errors, the output must be:

{
  "command": "diagnostics",
  "status": "error",
  "exit_code": 1,
  ...
  "messages": [{ "level": "error", "text": "3 errors must be resolved before CleverAgents can operate" }]
}

When only warnings:

status: "warn", exit_code: 0

When all checks pass:

status: "ok", exit_code: 0

Steps to Reproduce

agents --format json diagnostics
# Observe status="ok" even with errors present
python3 -c "import subprocess, json; r=subprocess.run(['agents','--format','json','diagnostics'], capture_output=True, text=True); d=json.loads(r.stdout); print('status:', d['status'], '| exit_code:', r.returncode)"

Code Analysis

In src/cleveragents/cli/main.py lines 383–414:

@app.command()
def diagnostics(...):
    data = build_diagnostics_data()
    if fmt.lower() == OutputFormat.RICH.value:
        render_diagnostics_rich(data)
    else:
        typer.echo(format_output(data, fmt))  # Always uses default status="ok"
    
    if check and data["has_errors"]:
        raise typer.Exit(code=1)  # Only sets exit code if --check flag used

The format_output(data, fmt) call always uses the default status="ok". The code does check data["has_errors"] but only affects exit code when --check is explicitly passed. The status in the envelope is never set to "error" or "warn" based on actual check results.

Acceptance Criteria

format_output(data, fmt, command="diagnostics", status="error", exit_code=1) when data["has_errors"] is True
format_output(data, fmt, command="diagnostics", status="warn", exit_code=0) when only warnings present
format_output(data, fmt, command="diagnostics", status="ok", exit_code=0) when all checks pass
Appropriate messages text reflects actual error/warning count
Exit code reflects the actual status even without --check flag

Supporting Information

Spec reference: docs/specification.md → agents diagnostics → JSON examples (both success and error cases)
Code location: src/cleveragents/cli/main.py, lines 383–414
Runtime confirmed: system has errors but JSON shows "status": "ok"

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Background and Context The specification requires the JSON/YAML output envelope `status` field to reflect the actual outcome of diagnostics. When health checks find errors, the status must be `"error"` and `exit_code` must be `1`. When only warnings are present, status must be `"warn"`. ## Current Behavior When `agents diagnostics --format json` runs and finds errors (e.g., database directory not writable), the envelope `status` is still `"ok"` even though `data.has_errors` is `True`: ```json { "command": "", "status": "ok", "exit_code": 0, "data": { ... "summary": { "errors": 1, "warnings": 7 }, "has_errors": true, "has_warnings": true }, ... } ``` This means a programmatic consumer checking `status` or `exit_code` would falsely conclude the system is healthy when it has critical errors. ## Expected Behavior (from spec) When diagnostics detects errors, the output must be: ```json { "command": "diagnostics", "status": "error", "exit_code": 1, ... "messages": [{ "level": "error", "text": "3 errors must be resolved before CleverAgents can operate" }] } ``` When only warnings: - `status: "warn"`, `exit_code: 0` When all checks pass: - `status: "ok"`, `exit_code: 0` ## Steps to Reproduce ```bash agents --format json diagnostics # Observe status="ok" even with errors present python3 -c "import subprocess, json; r=subprocess.run(['agents','--format','json','diagnostics'], capture_output=True, text=True); d=json.loads(r.stdout); print('status:', d['status'], '| exit_code:', r.returncode)" ``` ## Code Analysis In `src/cleveragents/cli/main.py` lines 383–414: ```python @app.command() def diagnostics(...): data = build_diagnostics_data() if fmt.lower() == OutputFormat.RICH.value: render_diagnostics_rich(data) else: typer.echo(format_output(data, fmt)) # Always uses default status="ok" if check and data["has_errors"]: raise typer.Exit(code=1) # Only sets exit code if --check flag used ``` The `format_output(data, fmt)` call always uses the default `status="ok"`. The code does check `data["has_errors"]` but only affects exit code when `--check` is explicitly passed. The `status` in the envelope is never set to `"error"` or `"warn"` based on actual check results. ## Acceptance Criteria - [ ] `format_output(data, fmt, command="diagnostics", status="error", exit_code=1)` when `data["has_errors"]` is True - [ ] `format_output(data, fmt, command="diagnostics", status="warn", exit_code=0)` when only warnings present - [ ] `format_output(data, fmt, command="diagnostics", status="ok", exit_code=0)` when all checks pass - [ ] Appropriate `messages` text reflects actual error/warning count - [ ] Exit code reflects the actual status even without `--check` flag ## Supporting Information - Spec reference: `docs/specification.md` → `agents diagnostics` → JSON examples (both success and error cases) - Code location: `src/cleveragents/cli/main.py`, lines 383–414 - Runtime confirmed: system has errors but JSON shows `"status": "ok"` --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-10 02:07:10 +00:00

HAL9000 added this to the v3.4.0 milestone

2026-04-10 02:07:14 +00:00

HAL9000 referenced this issue