UAT: `agents diagnostics` missing spec-required `Errors (must fix)` panel; provider API key format not validated — check marks invalid keys as "configured" #6849

New issue

Open

opened 2026-04-10 03:23:04 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-10 03:23:04 +00:00

Owner

What Was Tested

Code analysis of src/cleveragents/cli/commands/system.py render_diagnostics_rich() and _check_providers() functions compared against the spec-defined diagnostics output for the error scenario.

Expected Behavior (from spec)

The spec (docs/specification.md, lines 1059-1066) shows that when diagnostics has errors, a Errors (must fix) panel must appear with detailed fix instructions:

╭─ Errors (must fix) ─────────────────────────────────────────────────╮
│ 1. Database is locked by PID 12847 — stop the other process or      │
│    delete the lock file at ~/.cleveragents/agents.db-lock            │
│ 2. Anthropic key starts with "pk-" — expected "sk-ant-" prefix       │
│    Run: agents config set provider.anthropic.api-key <correct-key>  │
│ 3. FAISS library not installed — vector search will not work        │
│    Run: pip install faiss-cpu                                       │
╰─────────────────────────────────────────────────────────────────────╯

Additionally, the spec shows that the Anthropic key check detects invalid key format (not just presence), reporting Anthropic key ERROR invalid key format for a key with wrong prefix.

Actual Behavior

Missing `Errors (must fix)` panel

render_diagnostics_rich() only renders:

A checks table
A Summary panel
A Recommendations panel
A final status line

There is no Errors (must fix) panel with detailed actionable fix instructions.

No API key format validation

_check_providers() only checks bool(api_key) — whether the key is truthy (non-empty). It does NOT validate the key format (e.g., OpenAI keys must start with sk-, Anthropic keys with sk-ant-). A key with an obviously wrong format (e.g., pk-wrongkey) would be marked OK configured.

Impact

Users with misconfigured (wrong format) API keys will see OK configured for a broken provider, and when errors do occur there are no actionable fix instructions surfaced in the Errors panel.

Steps to Reproduce

For the missing panel:

# With a locked database or another error condition:
agents diagnostics
# No "Errors (must fix)" panel appears — only "Recommendations"

For the key format issue:

ANTHROPIC_API_KEY=pk-wrongprefix-key agents diagnostics
# Shows "Anthropic key  OK  configured" instead of "ERROR  invalid key format"

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## What Was Tested Code analysis of `src/cleveragents/cli/commands/system.py` `render_diagnostics_rich()` and `_check_providers()` functions compared against the spec-defined diagnostics output for the error scenario. ## Expected Behavior (from spec) The spec (`docs/specification.md`, lines 1059-1066) shows that when diagnostics has errors, a `Errors (must fix)` panel must appear with detailed fix instructions: ``` ╭─ Errors (must fix) ─────────────────────────────────────────────────╮ │ 1. Database is locked by PID 12847 — stop the other process or │ │ delete the lock file at ~/.cleveragents/agents.db-lock │ │ 2. Anthropic key starts with "pk-" — expected "sk-ant-" prefix │ │ Run: agents config set provider.anthropic.api-key <correct-key> │ │ 3. FAISS library not installed — vector search will not work │ │ Run: pip install faiss-cpu │ ╰─────────────────────────────────────────────────────────────────────╯ ``` Additionally, the spec shows that the Anthropic key check detects **invalid key format** (not just presence), reporting `Anthropic key ERROR invalid key format` for a key with wrong prefix. ## Actual Behavior ### Missing `Errors (must fix)` panel `render_diagnostics_rich()` only renders: 1. A checks table 2. A Summary panel 3. A Recommendations panel 4. A final status line There is no `Errors (must fix)` panel with detailed actionable fix instructions. ### No API key format validation `_check_providers()` only checks `bool(api_key)` — whether the key is truthy (non-empty). It does NOT validate the key format (e.g., OpenAI keys must start with `sk-`, Anthropic keys with `sk-ant-`). A key with an obviously wrong format (e.g., `pk-wrongkey`) would be marked `OK configured`. ## Impact Users with misconfigured (wrong format) API keys will see `OK configured` for a broken provider, and when errors do occur there are no actionable fix instructions surfaced in the `Errors` panel. ## Steps to Reproduce For the missing panel: ```bash # With a locked database or another error condition: agents diagnostics # No "Errors (must fix)" panel appears — only "Recommendations" ``` For the key format issue: ```bash ANTHROPIC_API_KEY=pk-wrongprefix-key agents diagnostics # Shows "Anthropic key OK configured" instead of "ERROR invalid key format" ``` --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added this to the v3.4.0 milestone

2026-04-10 03:23:04 +00:00

HAL9000 added the

labels

2026-04-10 03:23:04 +00:00

HAL9000 referenced this issue