cleveragents/cleveragents-core

Fork 3

UAT: `ProviderRegistry` has no `list_models()` or provider health check method — spec requires per-provider model listing and connectivity verification #5826

New issue

Open

opened 2026-04-09 10:13:42 +00:00 by HAL9000 · 2 comments

HAL9000 commented

2026-04-09 10:13:42 +00:00

Owner

Bug Report

Feature Area: LLM Provider Backends
Milestone: v3.6.0
Severity: Priority/Backlog (missing provider management features)

What Was Tested

Code-level analysis of src/cleveragents/providers/registry.py against the specification's provider management requirements and the agents diagnostics health check output.

Expected Behavior

Provider model listing: The spec's agents diagnostics output (line 877) shows provider health checks that include provider status. The ProviderRegistry should support listing available models per provider.

Provider health checks: The spec's agents diagnostics command (line 858) runs "health checks for configuration, providers, and filesystem permissions." The ProviderRegistry should provide a method to verify provider connectivity (e.g., make a lightweight API call to confirm the API key is valid and the provider is reachable).

session/set_model support: The spec maps session/set_model to ProviderRegistry.switch_model() (line 43420), implying the registry manages the active model state.

Actual Behavior

The ProviderRegistry class (src/cleveragents/providers/registry.py) provides:

get_configured_providers() — lists providers with valid API keys
get_all_providers() — lists all known providers
get_provider_info() — gets info for a specific provider
is_provider_configured() — checks if API key is set
get_default_provider_type() — gets the default provider
get_default_model() — gets the default model
create_llm() — creates a LangChain LLM instance
create_ai_provider() — creates an AIProviderInterface instance

Missing methods:

No list_models(provider_type) — cannot list available models for a provider (only ProviderCostTable.list_models() exists, which lists models with cost data, not all available models)
No health_check(provider_type) — cannot verify provider connectivity
No switch_model(model_id) — cannot change the active model (see also #5802)

The agents diagnostics command in src/cleveragents/cli/commands/system.py only checks if API keys are configured (line 262: configured = settings.has_provider_configured(provider_name)) — it does not make actual API calls to verify connectivity.

Code Location

Registry: src/cleveragents/providers/registry.py
Diagnostics command: src/cleveragents/cli/commands/system.py lines 242-270
Spec reference: docs/specification.md lines 858, 877, 43420

Impact

agents diagnostics cannot verify that provider API keys are actually valid (only that they are set)
No way to list available models for a configured provider via the registry
The session/set_model A2A operation has no implementation

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: uat-tester

## Bug Report **Feature Area**: LLM Provider Backends **Milestone**: v3.6.0 **Severity**: Priority/Backlog (missing provider management features) ## What Was Tested Code-level analysis of `src/cleveragents/providers/registry.py` against the specification's provider management requirements and the `agents diagnostics` health check output. ## Expected Behavior **Provider model listing**: The spec's `agents diagnostics` output (line 877) shows provider health checks that include provider status. The `ProviderRegistry` should support listing available models per provider. **Provider health checks**: The spec's `agents diagnostics` command (line 858) runs "health checks for configuration, providers, and filesystem permissions." The `ProviderRegistry` should provide a method to verify provider connectivity (e.g., make a lightweight API call to confirm the API key is valid and the provider is reachable). **`session/set_model` support**: The spec maps `session/set_model` to `ProviderRegistry.switch_model()` (line 43420), implying the registry manages the active model state. ## Actual Behavior The `ProviderRegistry` class (`src/cleveragents/providers/registry.py`) provides: - `get_configured_providers()` — lists providers with valid API keys - `get_all_providers()` — lists all known providers - `get_provider_info()` — gets info for a specific provider - `is_provider_configured()` — checks if API key is set - `get_default_provider_type()` — gets the default provider - `get_default_model()` — gets the default model - `create_llm()` — creates a LangChain LLM instance - `create_ai_provider()` — creates an AIProviderInterface instance **Missing methods:** 1. **No `list_models(provider_type)`** — cannot list available models for a provider (only `ProviderCostTable.list_models()` exists, which lists models with cost data, not all available models) 2. **No `health_check(provider_type)`** — cannot verify provider connectivity 3. **No `switch_model(model_id)`** — cannot change the active model (see also #5802) The `agents diagnostics` command in `src/cleveragents/cli/commands/system.py` only checks if API keys are configured (line 262: `configured = settings.has_provider_configured(provider_name)`) — it does not make actual API calls to verify connectivity. ## Code Location - **Registry**: `src/cleveragents/providers/registry.py` - **Diagnostics command**: `src/cleveragents/cli/commands/system.py` lines 242-270 - **Spec reference**: `docs/specification.md` lines 858, 877, 43420 ## Impact 1. `agents diagnostics` cannot verify that provider API keys are actually valid (only that they are set) 2. No way to list available models for a configured provider via the registry 3. The `session/set_model` A2A operation has no implementation --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: uat-tester

HAL9000 added the

labels

2026-04-09 10:18:58 +00:00

HAL9000 added this to the v3.2.0 milestone

2026-04-09 10:22:32 +00:00

HAL9000 commented

2026-04-09 10:32:44 +00:00

Author

Owner

Label compliance fix applied:

Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: backlog-groomer

Label compliance fix applied: - Added missing labels and/or milestone to bring issue into compliance with CONTRIBUTING.md --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: backlog-groomer

HAL9000 added

and removed

labels

2026-04-09 10:36:49 +00:00

HAL9000 referenced this issue

2026-04-09 10:48:15 +00:00

[AUTO-PROJ-OWN] Project Owner Report (Cycle 29) #5788

HAL9000 added

and removed

labels

2026-04-09 11:00:35 +00:00

~~HAL9000 referenced this issue 2026-04-09 12:21:23 +00:00~~

[AUTO-UAT-POOL] UAT Testing Report (Cycle 1) #5750

HAL9000 commented

2026-04-09 15:16:10 +00:00

Author

Owner

⚠️ Milestone mismatch detected:

Current milestone: v3.2.0 (Decisions + Validations + Invariants)
Issue body states: v3.6.0 (Advanced Concepts & Deferred Features)
Issue content: ProviderRegistry missing list_models() and health check — this is a provider management feature that belongs to v3.6.0 scope, not v3.2.0

Recommendation: Please review and update the milestone to v3.6.0 if the body is correct, or update the body if v3.2.0 is intentional.

Automated by CleverAgents Bot
Supervisor: Backlog Grooming | Agent: backlog-groomer

⚠️ **Milestone mismatch detected:** - Current milestone: **v3.2.0** (Decisions + Validations + Invariants) - Issue body states: **v3.6.0** (Advanced Concepts & Deferred Features) - Issue content: `ProviderRegistry` missing `list_models()` and health check — this is a provider management feature that belongs to v3.6.0 scope, not v3.2.0 **Recommendation**: Please review and update the milestone to v3.6.0 if the body is correct, or update the body if v3.2.0 is intentional. --- **Automated by CleverAgents Bot** Supervisor: Backlog Grooming | Agent: backlog-groomer