fix(provider): remove FakeListLLM defaults #427

Closed
freemo wants to merge 1 commit from feature/m4-provider-fixes into master
Owner

Summary

Closes #323. Removes hard-coded FakeListLLM defaults from provider configuration to prevent test fixtures from leaking into production code paths. Replaces the silent None return when no providers are configured with a fail-fast ProviderNotConfiguredError, adds an explicit mock_providers settings flag with a guard that blocks accidental mock usage outside test mode, implements provider auto-detection with trace logging (chosen provider and selection reason), and introduces a resolve_provider_by_name() helper that emits clear errors when a requested provider is unknown or missing credentials.

Changes

  • Provider Registry (src/cleveragents/providers/registry.py): Added resolve_provider_by_name() for explicit provider lookup with descriptive errors. Added DEBUG-level trace logging to get_default_provider_type() so every selection path (env override, settings default, auto-detected, none found) emits the chosen provider and the reason.
  • Container (src/cleveragents/application/container.py): Replaced the silent return None fallback when no providers are configured with ProviderNotConfiguredError. Added debug logging for mock-mode activation and final provider selection.
  • Settings (src/cleveragents/config/settings.py): Added mock_providers boolean field (env: CLEVERAGENTS_MOCK_PROVIDERS) with a startup validation guard that warns when the flag is set but CLEVERAGENTS_TESTING_USE_MOCK_AI is not enabled.
  • Exceptions (src/cleveragents/core/exceptions.py): Added ProviderNotConfiguredError subclass of ProviderError for missing-provider scenarios.
  • Agent graphs (auto_debug.py, context_analysis.py, plan_generation.py): Updated provider wiring to use the validated registry path instead of direct FakeListLLM fallbacks.
  • Tests (Behave): Added features/provider_fixes.feature with 104 lines of BDD scenarios and features/steps/provider_fixes_steps.py (447 lines) covering provider auto-detection, mock flag validation, fail-fast behavior, and trace logging.
  • Tests (Robot): Added robot/provider_detection.robot and robot/helper_provider_detection.py for integration-level provider detection smoke tests.
  • Tests (ASV): Added benchmarks/provider_selection_bench.py for provider resolution baseline benchmarks.
  • CHANGELOG: Added entry describing the FakeListLLM defaults removal.

Dependencies

  • Blocks issue #323 (this PR must be merged before the issue can be closed)
## Summary Closes #323. Removes hard-coded FakeListLLM defaults from provider configuration to prevent test fixtures from leaking into production code paths. Replaces the silent `None` return when no providers are configured with a fail-fast `ProviderNotConfiguredError`, adds an explicit `mock_providers` settings flag with a guard that blocks accidental mock usage outside test mode, implements provider auto-detection with trace logging (chosen provider and selection reason), and introduces a `resolve_provider_by_name()` helper that emits clear errors when a requested provider is unknown or missing credentials. ### Changes - **Provider Registry** (`src/cleveragents/providers/registry.py`): Added `resolve_provider_by_name()` for explicit provider lookup with descriptive errors. Added `DEBUG`-level trace logging to `get_default_provider_type()` so every selection path (env override, settings default, auto-detected, none found) emits the chosen provider and the reason. - **Container** (`src/cleveragents/application/container.py`): Replaced the silent `return None` fallback when no providers are configured with `ProviderNotConfiguredError`. Added debug logging for mock-mode activation and final provider selection. - **Settings** (`src/cleveragents/config/settings.py`): Added `mock_providers` boolean field (env: `CLEVERAGENTS_MOCK_PROVIDERS`) with a startup validation guard that warns when the flag is set but `CLEVERAGENTS_TESTING_USE_MOCK_AI` is not enabled. - **Exceptions** (`src/cleveragents/core/exceptions.py`): Added `ProviderNotConfiguredError` subclass of `ProviderError` for missing-provider scenarios. - **Agent graphs** (`auto_debug.py`, `context_analysis.py`, `plan_generation.py`): Updated provider wiring to use the validated registry path instead of direct FakeListLLM fallbacks. - **Tests (Behave)**: Added `features/provider_fixes.feature` with 104 lines of BDD scenarios and `features/steps/provider_fixes_steps.py` (447 lines) covering provider auto-detection, mock flag validation, fail-fast behavior, and trace logging. - **Tests (Robot)**: Added `robot/provider_detection.robot` and `robot/helper_provider_detection.py` for integration-level provider detection smoke tests. - **Tests (ASV)**: Added `benchmarks/provider_selection_bench.py` for provider resolution baseline benchmarks. - **CHANGELOG**: Added entry describing the FakeListLLM defaults removal. ## Dependencies - Blocks issue #323 (this PR must be merged before the issue can be closed)
freemo added this to the v3.0.0 milestone 2026-02-25 03:30:00 +00:00
Author
Owner

CONTRIBUTING.md Compliance Review

Passing

  • Closing keyword in description: Closes #323 is present.
  • Forgejo dependency link: PR #427 correctly blocks issue #323; issue #323 correctly depends on PR #427. Direction is correct per CONTRIBUTING.md § Linking and Dependencies.
  • Type label: Type/Bug is present (exactly one Type/ label).
  • Milestone: v3.0.0 is assigned and matches issue #323's milestone.
  • Conventional Changelog format: Both commit subject lines (fix(provider): ... and docs: ...) follow the standard.
  • PR description: Updated to include a detailed summary of changes, motivation, and file-level breakdown.

Issues Requiring Fixes

  1. Commit message bodies are empty (CONTRIBUTING.md § Commit Message Format, § First Line vs. Commit Body):
    Both commits have only a subject line with no body. The issue's Definition of Done explicitly requires: "A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation." The main commit (fc115f73) changes 21 files with 1,150 insertions — this warrants a substantive body explaining what was done and why.

  2. Commit messages missing issue reference footer (CONTRIBUTING.md § Pull Request Process, requirement 4):
    "Every commit in the PR must reference the issue it addresses in its commit message footer (e.g., ISSUES CLOSED: #45 or Refs: #45)." Neither commit includes ISSUES CLOSED: #323 or Refs: #323 in the footer.

  3. CHANGELOG update in a separate commit (CONTRIBUTING.md § Commit Completeness):
    "Update ancillary files. Configuration, build scripts, changelogs, and other metadata affected by the change should be updated in the same commit." The CHANGELOG entry is in a separate docs: commit (a33ce4e6) rather than included in the main fix(provider): commit (fc115f73). These two commits should be squashed into one so the changelog update accompanies the code change.

Squash the two commits into a single commit with:

  • First line: fix(provider): remove FakeListLLM defaults (matching issue #323 Metadata exactly)
  • Body: A description of the implementation (provider auto-detection, mock_providers flag, ProviderNotConfiguredError, trace logging, test additions)
  • Footer: ISSUES CLOSED: #323
## CONTRIBUTING.md Compliance Review ### Passing - **Closing keyword in description**: `Closes #323` is present. - **Forgejo dependency link**: PR #427 correctly **blocks** issue #323; issue #323 correctly **depends on** PR #427. Direction is correct per CONTRIBUTING.md § Linking and Dependencies. - **Type label**: `Type/Bug` is present (exactly one `Type/` label). - **Milestone**: `v3.0.0` is assigned and matches issue #323's milestone. - **Conventional Changelog format**: Both commit subject lines (`fix(provider): ...` and `docs: ...`) follow the standard. - **PR description**: Updated to include a detailed summary of changes, motivation, and file-level breakdown. ### Issues Requiring Fixes 1. **Commit message bodies are empty** (CONTRIBUTING.md § Commit Message Format, § First Line vs. Commit Body): Both commits have only a subject line with no body. The issue's Definition of Done explicitly requires: *"A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation."* The main commit (`fc115f73`) changes 21 files with 1,150 insertions — this warrants a substantive body explaining what was done and why. 2. **Commit messages missing issue reference footer** (CONTRIBUTING.md § Pull Request Process, requirement 4): *"Every commit in the PR must reference the issue it addresses in its commit message footer (e.g., `ISSUES CLOSED: #45` or `Refs: #45`)."* Neither commit includes `ISSUES CLOSED: #323` or `Refs: #323` in the footer. 3. **CHANGELOG update in a separate commit** (CONTRIBUTING.md § Commit Completeness): *"Update ancillary files. Configuration, build scripts, changelogs, and other metadata affected by the change should be updated in the same commit."* The CHANGELOG entry is in a separate `docs:` commit (`a33ce4e6`) rather than included in the main `fix(provider):` commit (`fc115f73`). These two commits should be squashed into one so the changelog update accompanies the code change. ### Recommended Fix Squash the two commits into a single commit with: - **First line**: `fix(provider): remove FakeListLLM defaults` (matching issue #323 Metadata exactly) - **Body**: A description of the implementation (provider auto-detection, mock_providers flag, ProviderNotConfiguredError, trace logging, test additions) - **Footer**: `ISSUES CLOSED: #323`
freemo force-pushed feature/m4-provider-fixes from a33ce4e6cf
Some checks failed
CI / lint (pull_request) Successful in 21s
CI / quality (pull_request) Successful in 29s
CI / security (pull_request) Successful in 47s
CI / typecheck (pull_request) Successful in 54s
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 52s
CI / integration_tests (pull_request) Failing after 4m23s
CI / unit_tests (pull_request) Failing after 16m31s
CI / docker (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 21m36s
CI / coverage (pull_request) Failing after 54m6s
to a709e11092
Some checks failed
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 18s
CI / lint (pull_request) Successful in 21s
CI / quality (pull_request) Successful in 40s
CI / typecheck (pull_request) Successful in 41s
CI / security (pull_request) Successful in 54s
CI / integration_tests (pull_request) Failing after 3m23s
CI / unit_tests (pull_request) Failing after 9m44s
CI / docker (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 20m47s
CI / coverage (pull_request) Failing after 34m24s
2026-02-26 10:14:45 +00:00
Compare
freemo scheduled this pull request to auto merge when all checks succeed 2026-02-26 10:15:07 +00:00
freemo force-pushed feature/m4-provider-fixes from a709e11092
Some checks failed
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 18s
CI / lint (pull_request) Successful in 21s
CI / quality (pull_request) Successful in 40s
CI / typecheck (pull_request) Successful in 41s
CI / security (pull_request) Successful in 54s
CI / integration_tests (pull_request) Failing after 3m23s
CI / unit_tests (pull_request) Failing after 9m44s
CI / docker (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 20m47s
CI / coverage (pull_request) Failing after 34m24s
to 333576a823
Some checks failed
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 16s
CI / build (pull_request) Successful in 16s
CI / quality (pull_request) Successful in 18s
CI / security (pull_request) Successful in 31s
CI / typecheck (pull_request) Successful in 33s
CI / integration_tests (pull_request) Failing after 3m52s
CI / unit_tests (pull_request) Failing after 10m30s
CI / docker (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 21m15s
CI / coverage (pull_request) Failing after 33m59s
2026-02-26 11:49:25 +00:00
Compare
freemo force-pushed feature/m4-provider-fixes from 333576a823
Some checks failed
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Successful in 16s
CI / build (pull_request) Successful in 16s
CI / quality (pull_request) Successful in 18s
CI / security (pull_request) Successful in 31s
CI / typecheck (pull_request) Successful in 33s
CI / integration_tests (pull_request) Failing after 3m52s
CI / unit_tests (pull_request) Failing after 10m30s
CI / docker (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 21m15s
CI / coverage (pull_request) Failing after 33m59s
to 5baa4f2e28
Some checks failed
CI / lint (pull_request) Successful in 16s
CI / benchmark-publish (pull_request) Has been skipped
CI / quality (pull_request) Successful in 28s
CI / security (pull_request) Successful in 34s
CI / build (pull_request) Successful in 31s
CI / typecheck (pull_request) Successful in 1m2s
CI / unit_tests (pull_request) Has been cancelled
CI / integration_tests (pull_request) Has been cancelled
CI / docker (pull_request) Has been cancelled
CI / coverage (pull_request) Has been cancelled
CI / benchmark-regression (pull_request) Has been cancelled
2026-02-26 15:24:16 +00:00
Compare
freemo force-pushed feature/m4-provider-fixes from 5baa4f2e28
Some checks failed
CI / lint (pull_request) Successful in 16s
CI / benchmark-publish (pull_request) Has been skipped
CI / quality (pull_request) Successful in 28s
CI / security (pull_request) Successful in 34s
CI / build (pull_request) Successful in 31s
CI / typecheck (pull_request) Successful in 1m2s
CI / unit_tests (pull_request) Has been cancelled
CI / integration_tests (pull_request) Has been cancelled
CI / docker (pull_request) Has been cancelled
CI / coverage (pull_request) Has been cancelled
CI / benchmark-regression (pull_request) Has been cancelled
to 88eb4de677
Some checks failed
CI / lint (pull_request) Successful in 25s
CI / benchmark-publish (pull_request) Has been skipped
CI / quality (pull_request) Successful in 17s
CI / build (pull_request) Successful in 23s
CI / security (pull_request) Successful in 47s
CI / typecheck (pull_request) Successful in 1m11s
CI / integration_tests (pull_request) Failing after 4m46s
CI / unit_tests (pull_request) Failing after 21m22s
CI / docker (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 24m4s
CI / coverage (pull_request) Failing after 45m8s
2026-02-26 15:25:51 +00:00
Compare
freemo force-pushed feature/m4-provider-fixes from 88eb4de677
Some checks failed
CI / lint (pull_request) Successful in 25s
CI / benchmark-publish (pull_request) Has been skipped
CI / quality (pull_request) Successful in 17s
CI / build (pull_request) Successful in 23s
CI / security (pull_request) Successful in 47s
CI / typecheck (pull_request) Successful in 1m11s
CI / integration_tests (pull_request) Failing after 4m46s
CI / unit_tests (pull_request) Failing after 21m22s
CI / docker (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 24m4s
CI / coverage (pull_request) Failing after 45m8s
to 5e2d6b8b5d
Some checks failed
CI / lint (pull_request) Successful in 14s
CI / benchmark-publish (pull_request) Has been skipped
CI / quality (pull_request) Successful in 17s
CI / typecheck (pull_request) Successful in 32s
CI / security (pull_request) Successful in 34s
CI / build (pull_request) Successful in 31s
CI / integration_tests (pull_request) Failing after 5m21s
CI / benchmark-regression (pull_request) Successful in 20m4s
CI / unit_tests (pull_request) Failing after 28m1s
CI / docker (pull_request) Has been skipped
CI / coverage (pull_request) Failing after 1h22m44s
2026-02-26 16:53:15 +00:00
Compare
freemo force-pushed feature/m4-provider-fixes from 5e2d6b8b5d
Some checks failed
CI / lint (pull_request) Successful in 14s
CI / benchmark-publish (pull_request) Has been skipped
CI / quality (pull_request) Successful in 17s
CI / typecheck (pull_request) Successful in 32s
CI / security (pull_request) Successful in 34s
CI / build (pull_request) Successful in 31s
CI / integration_tests (pull_request) Failing after 5m21s
CI / benchmark-regression (pull_request) Successful in 20m4s
CI / unit_tests (pull_request) Failing after 28m1s
CI / docker (pull_request) Has been skipped
CI / coverage (pull_request) Failing after 1h22m44s
to cdd4244ae1
Some checks failed
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Failing after 13s
CI / build (pull_request) Successful in 15s
CI / quality (pull_request) Successful in 23s
CI / integration_tests (pull_request) Has been cancelled
CI / typecheck (pull_request) Has been cancelled
CI / security (pull_request) Has been cancelled
CI / unit_tests (pull_request) Has been cancelled
CI / benchmark-regression (pull_request) Has been cancelled
CI / docker (pull_request) Has been cancelled
CI / coverage (pull_request) Has been cancelled
2026-02-26 18:47:55 +00:00
Compare
freemo force-pushed feature/m4-provider-fixes from cdd4244ae1
Some checks failed
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Failing after 13s
CI / build (pull_request) Successful in 15s
CI / quality (pull_request) Successful in 23s
CI / integration_tests (pull_request) Has been cancelled
CI / typecheck (pull_request) Has been cancelled
CI / security (pull_request) Has been cancelled
CI / unit_tests (pull_request) Has been cancelled
CI / benchmark-regression (pull_request) Has been cancelled
CI / docker (pull_request) Has been cancelled
CI / coverage (pull_request) Has been cancelled
to 5c122a31e9
Some checks failed
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 17s
CI / quality (pull_request) Successful in 20s
CI / lint (pull_request) Failing after 22s
CI / security (pull_request) Successful in 33s
CI / typecheck (pull_request) Successful in 54s
CI / coverage (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Has been skipped
CI / integration_tests (pull_request) Failing after 2m32s
CI / unit_tests (pull_request) Failing after 9m58s
CI / docker (pull_request) Has been skipped
2026-02-26 18:48:20 +00:00
Compare
freemo scheduled this pull request to auto merge when all checks succeed 2026-02-26 18:49:57 +00:00
Author
Owner

Im going to close this and start over here...

Im going to close this and start over here...
freemo closed this pull request 2026-02-26 19:04:19 +00:00
Some checks failed
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 17s
Required
Details
CI / quality (pull_request) Successful in 20s
Required
Details
CI / lint (pull_request) Failing after 22s
Required
Details
CI / security (pull_request) Successful in 33s
Required
Details
CI / typecheck (pull_request) Successful in 54s
Required
Details
CI / coverage (pull_request) Has been skipped
Required
Details
CI / benchmark-regression (pull_request) Has been skipped
CI / integration_tests (pull_request) Failing after 2m32s
Required
Details
CI / unit_tests (pull_request) Failing after 9m58s
Required
Details
CI / docker (pull_request) Has been skipped
Required
Details

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
Reference
cleveragents/cleveragents-core!427
No description provided.