cleveragents/cleveragents-core

Fork 3

UAT: `agents diagnostics` only checks 4 of 9 supported providers — Groq, Together AI, Cohere, Azure, Gemini missing #3422

New issue

Closed

opened 2026-04-05 16:38:23 +00:00 by freemo · 1 comment

freemo commented

2026-04-05 16:38:23 +00:00

Owner

Metadata

Branch: fix/diagnostics-provider-coverage
Commit Message: fix(cli): extend agents diagnostics to check all 9 supported providers
Milestone: (none — backlog)
Parent Epic: #3365

Backlog note: This issue was discovered during autonomous operation
on milestone (current active milestone). It does not block milestone completion and has been
placed in the backlog for human review and future milestone assignment.

Description

The _check_providers() function in src/cleveragents/cli/commands/system.py only checks API key configuration for 4 providers (OpenAI, Anthropic, Google, OpenRouter), but the ProviderRegistry supports 9 providers (OpenAI, Anthropic, Google, Azure, OpenRouter, Gemini, Cohere, Groq, Together AI).

The spec's agents diagnostics output example shows provider key checks, and users of Groq, Together AI, Cohere, Azure, or Gemini will not see any diagnostic feedback about their provider configuration.

What Was Tested

Code-level analysis of src/cleveragents/cli/commands/system.py and src/cleveragents/providers/registry.py.

Expected Behaviour (from spec)

The agents diagnostics command should check all configured/supported providers and report their status. The spec shows: "OPENAI_API_KEY WARN missing" and "Anthropic key OK configured" — implying all supported providers should be checked.

Actual Behaviour

_check_providers() at lines 249–254 only checks:

provider_checks = [
    ("openai", "OPENAI_API_KEY"),
    ("anthropic", "ANTHROPIC_API_KEY"),
    ("google", "GOOGLE_API_KEY"),
    ("openrouter", "OPENROUTER_API_KEY"),
]

Missing from diagnostics: Groq (GROQ_API_KEY), Together AI (TOGETHER_API_KEY), Cohere (COHERE_API_KEY), Azure (AZURE_OPENAI_API_KEY), Gemini (GEMINI_API_KEY).

Code Location

src/cleveragents/cli/commands/system.py, lines 249–254 (_check_providers() function starting at line 242).

Steps to Reproduce

Open src/cleveragents/cli/commands/system.py
Find _check_providers() function at line 242
Observe that only 4 providers are checked despite 9 being supported in the registry

Subtasks

Audit ProviderRegistry in src/cleveragents/providers/registry.py to confirm the full list of supported providers and their expected environment variable names
Extend provider_checks list in _check_providers() to include all 9 providers: Groq (GROQ_API_KEY), Together AI (TOGETHER_API_KEY), Cohere (COHERE_API_KEY), Azure (AZURE_OPENAI_API_KEY), Gemini (GEMINI_API_KEY)
Ensure the extended check list stays in sync with ProviderRegistry (consider deriving it dynamically from the registry)
Write/update Behave unit test scenarios in features/ covering all 9 provider checks in _check_providers()
Update Robot Framework integration tests in robot/ if any end-to-end diagnostics tests exist
Verify all nox quality gates pass

Definition of Done

_check_providers() reports diagnostic status for all 9 providers supported by ProviderRegistry
No provider supported by the registry is silently omitted from agents diagnostics output
Behave unit test scenarios cover each of the 9 provider checks (present key, missing key, invalid key where applicable)
All nox stages pass (nox -e lint, nox -e typecheck, nox -e unit_tests, nox -e integration_tests)
Coverage >= 97%
Fix committed on branch fix/diagnostics-provider-coverage with commit message fix(cli): extend agents diagnostics to check all 9 supported providers
Pull request merged and issue closed

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-uat-tester

## Metadata - **Branch**: `fix/diagnostics-provider-coverage` - **Commit Message**: `fix(cli): extend agents diagnostics to check all 9 supported providers` - **Milestone**: *(none — backlog)* - **Parent Epic**: #3365 > **Backlog note:** This issue was discovered during autonomous operation > on milestone (current active milestone). It does not block milestone completion and has been > placed in the backlog for human review and future milestone assignment. ## Description The `_check_providers()` function in `src/cleveragents/cli/commands/system.py` only checks API key configuration for **4 providers** (OpenAI, Anthropic, Google, OpenRouter), but the `ProviderRegistry` supports **9 providers** (OpenAI, Anthropic, Google, Azure, OpenRouter, Gemini, Cohere, Groq, Together AI). The spec's `agents diagnostics` output example shows provider key checks, and users of Groq, Together AI, Cohere, Azure, or Gemini will not see any diagnostic feedback about their provider configuration. ### What Was Tested Code-level analysis of `src/cleveragents/cli/commands/system.py` and `src/cleveragents/providers/registry.py`. ### Expected Behaviour (from spec) The `agents diagnostics` command should check all configured/supported providers and report their status. The spec shows: `"OPENAI_API_KEY WARN missing"` and `"Anthropic key OK configured"` — implying **all** supported providers should be checked. ### Actual Behaviour `_check_providers()` at lines 249–254 only checks: ```python provider_checks = [ ("openai", "OPENAI_API_KEY"), ("anthropic", "ANTHROPIC_API_KEY"), ("google", "GOOGLE_API_KEY"), ("openrouter", "OPENROUTER_API_KEY"), ] ``` **Missing from diagnostics:** Groq (`GROQ_API_KEY`), Together AI (`TOGETHER_API_KEY`), Cohere (`COHERE_API_KEY`), Azure (`AZURE_OPENAI_API_KEY`), Gemini (`GEMINI_API_KEY`). ### Code Location `src/cleveragents/cli/commands/system.py`, lines 249–254 (`_check_providers()` function starting at line 242). ### Steps to Reproduce 1. Open `src/cleveragents/cli/commands/system.py` 2. Find `_check_providers()` function at line 242 3. Observe that only 4 providers are checked despite 9 being supported in the registry ## Subtasks - [ ] Audit `ProviderRegistry` in `src/cleveragents/providers/registry.py` to confirm the full list of supported providers and their expected environment variable names - [ ] Extend `provider_checks` list in `_check_providers()` to include all 9 providers: Groq (`GROQ_API_KEY`), Together AI (`TOGETHER_API_KEY`), Cohere (`COHERE_API_KEY`), Azure (`AZURE_OPENAI_API_KEY`), Gemini (`GEMINI_API_KEY`) - [ ] Ensure the extended check list stays in sync with `ProviderRegistry` (consider deriving it dynamically from the registry) - [ ] Write/update Behave unit test scenarios in `features/` covering all 9 provider checks in `_check_providers()` - [ ] Update Robot Framework integration tests in `robot/` if any end-to-end diagnostics tests exist - [ ] Verify all nox quality gates pass ## Definition of Done - [ ] `_check_providers()` reports diagnostic status for all 9 providers supported by `ProviderRegistry` - [ ] No provider supported by the registry is silently omitted from `agents diagnostics` output - [ ] Behave unit test scenarios cover each of the 9 provider checks (present key, missing key, invalid key where applicable) - [ ] All nox stages pass (`nox -e lint`, `nox -e typecheck`, `nox -e unit_tests`, `nox -e integration_tests`) - [ ] Coverage >= 97% - [ ] Fix committed on branch `fix/diagnostics-provider-coverage` with commit message `fix(cli): extend agents diagnostics to check all 9 supported providers` - [ ] Pull request merged and issue closed --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-uat-tester

freemo added a new dependency

2026-04-05 16:38:40 +00:00

#3365 Epic: Additional LLM Provider Integrations — Cohere, Groq, Together AI, and Provider Abstraction

freemo referenced this issue

2026-04-05 16:41:36 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 16:43:07 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue

2026-04-05 16:44:25 +00:00

[Automated] CleverAgents Build Session - 2026-04-05 #3377

freemo referenced this issue from a commit

2026-04-05 18:07:40 +00:00

fix(cli): extend agents diagnostics to check all 9 supported providers

freemo referenced this issue from a pull request that will close it,

2026-04-05 18:07:50 +00:00

fix(cli): extend agents diagnostics to check all 9 supported providers #3469

freemo commented

2026-04-05 18:07:58 +00:00

Author

Owner

PR #3469 created on branch fix/diagnostics-provider-coverage. I will monitor and handle all review feedback until merged.

Implementation summary:

Extended _check_providers() in src/cleveragents/cli/commands/system.py to include all 9 providers: OpenAI, Anthropic, Google, Azure, OpenRouter, Gemini, Cohere, Groq, Together AI
Added features/diagnostics_provider_coverage.feature with 11 Behave scenarios
All quality gates pass: lint ✅, typecheck ✅, unit_tests ✅ (11/11 scenarios)

Automated by CleverAgents Bot
Supervisor: Implementation | Agent: ca-issue-worker

PR #3469 created on branch `fix/diagnostics-provider-coverage`. I will monitor and handle all review feedback until merged. **Implementation summary:** - Extended `_check_providers()` in `src/cleveragents/cli/commands/system.py` to include all 9 providers: OpenAI, Anthropic, Google, Azure, OpenRouter, Gemini, Cohere, Groq, Together AI - Added `features/diagnostics_provider_coverage.feature` with 11 Behave scenarios - All quality gates pass: lint ✅, typecheck ✅, unit_tests ✅ (11/11 scenarios) --- **Automated by CleverAgents Bot** Supervisor: Implementation | Agent: ca-issue-worker

freemo referenced this issue

2026-04-05 19:33:30 +00:00

fix(cli): extend agents diagnostics to check all 9 supported providers #3469

freemo referenced this issue from a commit

2026-04-05 20:58:54 +00:00

fix(cli): extend agents diagnostics to check all 9 supported providers

freemo referenced this issue from a commit

2026-04-05 21:06:09 +00:00

fix(cli): extend agents diagnostics to check all 9 supported providers

freemo closed this issue

2026-04-05 21:19:44 +00:00

freemo referenced this issue from a commit