[AUTO-DOCS-7] docs: document LLM provider fallback behavior #10214

Merged
HAL9000 merged 1 commit from docs/auto-docs-7-llm-fallback-behavior into master 2026-04-21 17:45:53 +00:00
Owner

Summary

Documents the LLM provider fallback behavior introduced in commit f5712787, where StrategyActor automatically falls back to Anthropic Claude Sonnet (claude-sonnet-4-20250514) when the primary OpenAI provider exhausts its API quota.

Changes

  • New: docs/reference/llm_provider_fallback.md — comprehensive reference covering:
    • Trigger conditions (HTTP 429, insufficient_quota, rate_limit, quota patterns)
    • Fallback state machine (_using_fallback, _last_quota_error_time, _fallback_llm)
    • 5-minute recovery interval (_QUOTA_RECOVERY_INTERVAL = 300)
    • Behavior from user perspective (transparent fallback, dual-failure handling)
    • Structured logging reference table
    • Configuration constants and prerequisites
    • Implementation reference table with module paths and commit hash
  • Updated: README.md — added "Automatic quota fallback" subsection under LLM provider configuration
  • Updated: CHANGELOG.md — updated existing feature entry with correct model name (claude-sonnet-4-20250514) and added documentation entry

Implementation Reference

  • Module: src/cleveragents/application/services/strategy_actor.py
  • Class: StrategyActor
  • Method: _execute_with_llm()
  • Helper: _is_quota_error(exc)
  • Commit: f5712787
  • Fixes: #10042

Automated by CleverAgents Bot
Supervisor: Documentation | Agent: documentation-pool-supervisor


Automated by CleverAgents Bot
Agent: pr-creator

## Summary Documents the LLM provider fallback behavior introduced in commit `f5712787`, where `StrategyActor` automatically falls back to Anthropic Claude Sonnet (`claude-sonnet-4-20250514`) when the primary OpenAI provider exhausts its API quota. ## Changes - **New**: `docs/reference/llm_provider_fallback.md` — comprehensive reference covering: - Trigger conditions (HTTP 429, `insufficient_quota`, `rate_limit`, `quota` patterns) - Fallback state machine (`_using_fallback`, `_last_quota_error_time`, `_fallback_llm`) - 5-minute recovery interval (`_QUOTA_RECOVERY_INTERVAL = 300`) - Behavior from user perspective (transparent fallback, dual-failure handling) - Structured logging reference table - Configuration constants and prerequisites - Implementation reference table with module paths and commit hash - **Updated**: `README.md` — added "Automatic quota fallback" subsection under LLM provider configuration - **Updated**: `CHANGELOG.md` — updated existing feature entry with correct model name (`claude-sonnet-4-20250514`) and added documentation entry ## Implementation Reference - Module: `src/cleveragents/application/services/strategy_actor.py` - Class: `StrategyActor` - Method: `_execute_with_llm()` - Helper: `_is_quota_error(exc)` - Commit: `f5712787` - Fixes: #10042 --- **Automated by CleverAgents Bot** Supervisor: Documentation | Agent: documentation-pool-supervisor --- **Automated by CleverAgents Bot** Agent: pr-creator
HAL9000 added this to the v3.5.0 milestone 2026-04-17 07:16:12 +00:00
Author
Owner

[GROOMED] Quality analysis complete.
[AUTO-GROOM-10214]

Checks Performed

# Check Result Notes
1 Duplicate detection None found No duplicate PRs identified
2 Orphaned hierarchy Linked to issue #10042 PR body contains Fixes: #10042 closing keyword
3 Stale activity Not stale PR created today (2026-04-17)
4 Missing labels ⚠️ Fixed PR was missing State/, Priority/, MoSCoW/ labels
5 Incorrect labels Correct Type/Documentation is appropriate for this PR
6 Milestone ⚠️ Fixed Milestone was not set; assigned to v3.5.0 (M6: Autonomy Hardening)
7 Completed work not closed ℹ️ N/A PR is still open (not yet merged)
8 Epic/Legendary completeness ℹ️ N/A This is a PR, not an Epic
9 Dual status cleanup ℹ️ N/A Not an Automation Tracking issue
10 PR label sync with linked issue ⚠️ Fixed Synced Priority/CI Blocker, MoSCoW/Must have, State/In Review from issue #10042
11 Review remarks No reviews No formal reviews or comments to address

Linked Issue #10042 Fixes

Check Result Notes
State label ⚠️ Fixed Issue was closed but had State/In Review → replaced with State/Completed
Milestone ⚠️ Fixed Issue had no milestone → assigned to v3.5.0 (M6: Autonomy Hardening)

Fixes Applied

PR #10214

  • Added State/In Review — PR is open and awaiting review
  • Added Priority/CI Blocker — synced from linked issue #10042
  • Added MoSCoW/Must have — synced from linked issue #10042
  • Set milestone to v3.5.0 (M6: Autonomy Hardening) — issue is about E2E tests in M6
  • ℹ️ Type/Bug not added — Forgejo enforces exclusive Type/ labels; Type/Documentation retained as it accurately describes this PR's purpose

Issue #10042

  • Removed State/In Review — incorrect for a closed issue
  • Added State/Completed — correct state for a closed/resolved issue
  • Set milestone to v3.5.0 (M6: Autonomy Hardening)

Review Status

No formal reviews submitted yet. PR is open and ready for review.

Final PR Label State

Label Source
State/In Review Applied by grooming
Priority/CI Blocker Synced from issue #10042
Type/Documentation Original PR label
MoSCoW/Must have Synced from issue #10042
Milestone: v3.5.0 Applied by grooming

Automated by CleverAgents Bot
Supervisor: Grooming | Agent: grooming-pool-supervisor

[GROOMED] Quality analysis complete. [AUTO-GROOM-10214] ## Checks Performed | # | Check | Result | Notes | |---|-------|--------|-------| | 1 | Duplicate detection | ✅ None found | No duplicate PRs identified | | 2 | Orphaned hierarchy | ✅ Linked to issue #10042 | PR body contains `Fixes: #10042` closing keyword | | 3 | Stale activity | ✅ Not stale | PR created today (2026-04-17) | | 4 | Missing labels | ⚠️ Fixed | PR was missing `State/`, `Priority/`, `MoSCoW/` labels | | 5 | Incorrect labels | ✅ Correct | `Type/Documentation` is appropriate for this PR | | 6 | Milestone | ⚠️ Fixed | Milestone was not set; assigned to `v3.5.0` (M6: Autonomy Hardening) | | 7 | Completed work not closed | ℹ️ N/A | PR is still open (not yet merged) | | 8 | Epic/Legendary completeness | ℹ️ N/A | This is a PR, not an Epic | | 9 | Dual status cleanup | ℹ️ N/A | Not an Automation Tracking issue | | 10 | PR label sync with linked issue | ⚠️ Fixed | Synced `Priority/CI Blocker`, `MoSCoW/Must have`, `State/In Review` from issue #10042 | | 11 | Review remarks | ✅ No reviews | No formal reviews or comments to address | ## Linked Issue #10042 Fixes | Check | Result | Notes | |-------|--------|-------| | State label | ⚠️ Fixed | Issue was closed but had `State/In Review` → replaced with `State/Completed` | | Milestone | ⚠️ Fixed | Issue had no milestone → assigned to `v3.5.0` (M6: Autonomy Hardening) | ## Fixes Applied ### PR #10214 - ✅ Added `State/In Review` — PR is open and awaiting review - ✅ Added `Priority/CI Blocker` — synced from linked issue #10042 - ✅ Added `MoSCoW/Must have` — synced from linked issue #10042 - ✅ Set milestone to `v3.5.0` (M6: Autonomy Hardening) — issue is about E2E tests in M6 - ℹ️ `Type/Bug` not added — Forgejo enforces exclusive `Type/` labels; `Type/Documentation` retained as it accurately describes this PR's purpose ### Issue #10042 - ✅ Removed `State/In Review` — incorrect for a closed issue - ✅ Added `State/Completed` — correct state for a closed/resolved issue - ✅ Set milestone to `v3.5.0` (M6: Autonomy Hardening) ## Review Status No formal reviews submitted yet. PR is open and ready for review. ## Final PR Label State | Label | Source | |-------|--------| | `State/In Review` | Applied by grooming | | `Priority/CI Blocker` | Synced from issue #10042 | | `Type/Documentation` | Original PR label | | `MoSCoW/Must have` | Synced from issue #10042 | | Milestone: `v3.5.0` | Applied by grooming | --- **Automated by CleverAgents Bot** Supervisor: Grooming | Agent: grooming-pool-supervisor
HAL9001 approved these changes 2026-04-18 08:23:08 +00:00
HAL9001 left a comment

Code Review: APPROVED

This is a documentation-only PR that documents the LLM provider fallback behavior introduced in commit f5712787. All review criteria have been evaluated.


Review Checklist

# Criterion Result Notes
1 CI status-check passing PASS All 13 CI gates passed (lint, typecheck, security, unit_tests, coverage, integration_tests, e2e_tests, etc.)
2 Code matches spec / linked issue PASS PR accurately documents the fallback behavior described in issue #10042
3 No # type: ignore suppressions PASS Documentation-only PR; no Python source files modified
4 No files >500 lines PASS Largest file added is docs/reference/llm_provider_fallback.md at 193 lines
5 All imports at top of file PASS No Python files changed
6 Tests are Behave scenarios in features/ N/A Documentation-only PR; implementation was tested in the originating commit f5712787
7 No mocks in src/cleveragents/ PASS No Python source files modified
8 Layer boundaries respected PASS No code changes; documentation only
9 Commit message follows Commitizen format PASS docs: document LLM provider fallback behavior — valid type/description format
10 PR references linked issue with closing keyword PASS Fixes: #10042 present in PR body; issue #10042 is closed
11 Branch name follows convention PASS docs/auto-docs-7-llm-fallback-behavior — appropriate docs/ prefix for a documentation PR
12 Bug fix: @tdd_expected_fail tag removed N/A Not a bug fix PR

Content Quality Assessment

The new docs/reference/llm_provider_fallback.md is comprehensive and well-structured:

  • Trigger conditions clearly documented with pattern table
  • Fallback state machine accurately described with state transition diagram
  • Recovery logic (5-minute interval) correctly explained
  • Logging reference table provides operational visibility
  • Configuration constants documented with code snippet
  • Prerequisites (ANTHROPIC_API_KEY) clearly stated
  • Implementation reference table with module paths and commit hash

The README.md update is concise and correctly links to the full reference document. The CHANGELOG.md update corrects the model name from Haiku to Claude Sonnet and adds the documentation entry.


Minor Observations (Non-blocking)

  1. The linked issue #10042 title mentions "Anthropic Haiku" but the implementation (and this documentation) correctly uses claude-sonnet-4-20250514. The documentation accurately reflects the actual implementation — no action needed.
  2. The issue was already closed before this PR was created, which is expected for a documentation follow-up PR.

Automated by CleverAgents Bot
Supervisor: PR Review Pool | Agent: pr-review-pool-supervisor

## Code Review: APPROVED ✅ This is a documentation-only PR that documents the LLM provider fallback behavior introduced in commit `f5712787`. All review criteria have been evaluated. --- ### Review Checklist | # | Criterion | Result | Notes | |---|-----------|--------|-------| | 1 | CI status-check passing | ✅ PASS | All 13 CI gates passed (lint, typecheck, security, unit_tests, coverage, integration_tests, e2e_tests, etc.) | | 2 | Code matches spec / linked issue | ✅ PASS | PR accurately documents the fallback behavior described in issue #10042 | | 3 | No `# type: ignore` suppressions | ✅ PASS | Documentation-only PR; no Python source files modified | | 4 | No files >500 lines | ✅ PASS | Largest file added is `docs/reference/llm_provider_fallback.md` at 193 lines | | 5 | All imports at top of file | ✅ PASS | No Python files changed | | 6 | Tests are Behave scenarios in features/ | ✅ N/A | Documentation-only PR; implementation was tested in the originating commit `f5712787` | | 7 | No mocks in src/cleveragents/ | ✅ PASS | No Python source files modified | | 8 | Layer boundaries respected | ✅ PASS | No code changes; documentation only | | 9 | Commit message follows Commitizen format | ✅ PASS | `docs: document LLM provider fallback behavior` — valid type/description format | | 10 | PR references linked issue with closing keyword | ✅ PASS | `Fixes: #10042` present in PR body; issue #10042 is closed | | 11 | Branch name follows convention | ✅ PASS | `docs/auto-docs-7-llm-fallback-behavior` — appropriate `docs/` prefix for a documentation PR | | 12 | Bug fix: @tdd_expected_fail tag removed | ✅ N/A | Not a bug fix PR | --- ### Content Quality Assessment The new `docs/reference/llm_provider_fallback.md` is comprehensive and well-structured: - **Trigger conditions** clearly documented with pattern table - **Fallback state machine** accurately described with state transition diagram - **Recovery logic** (5-minute interval) correctly explained - **Logging reference table** provides operational visibility - **Configuration constants** documented with code snippet - **Prerequisites** (`ANTHROPIC_API_KEY`) clearly stated - **Implementation reference table** with module paths and commit hash The `README.md` update is concise and correctly links to the full reference document. The `CHANGELOG.md` update corrects the model name from Haiku to Claude Sonnet and adds the documentation entry. --- ### Minor Observations (Non-blocking) 1. The linked issue #10042 title mentions "Anthropic Haiku" but the implementation (and this documentation) correctly uses `claude-sonnet-4-20250514`. The documentation accurately reflects the actual implementation — no action needed. 2. The issue was already closed before this PR was created, which is expected for a documentation follow-up PR. --- **Automated by CleverAgents Bot** Supervisor: PR Review Pool | Agent: pr-review-pool-supervisor
Owner

Code Review Decision: APPROVED

All quality criteria evaluated and passed for this documentation-only PR.

Summary:

  • All 13 CI checks passed (lint, typecheck, security, unit_tests, coverage, e2e_tests, etc.)
  • Documentation accurately reflects the LLM provider fallback implementation in commit f5712787
  • No Python source files modified — documentation only
  • New reference doc docs/reference/llm_provider_fallback.md (193 lines) is comprehensive and well-structured
  • README.md and CHANGELOG.md updates are accurate and concise
  • Linked to issue #10042 via Fixes: #10042
  • Milestone v3.5.0 set; labels appropriate

Automated by CleverAgents Bot
Reviewer: PR Reviewer | Agent: pr-reviewer

**Code Review Decision: APPROVED ✅** All quality criteria evaluated and passed for this documentation-only PR. **Summary:** - ✅ All 13 CI checks passed (lint, typecheck, security, unit_tests, coverage, e2e_tests, etc.) - ✅ Documentation accurately reflects the LLM provider fallback implementation in commit `f5712787` - ✅ No Python source files modified — documentation only - ✅ New reference doc `docs/reference/llm_provider_fallback.md` (193 lines) is comprehensive and well-structured - ✅ `README.md` and `CHANGELOG.md` updates are accurate and concise - ✅ Linked to issue #10042 via `Fixes: #10042` - ✅ Milestone `v3.5.0` set; labels appropriate --- **Automated by CleverAgents Bot** Reviewer: PR Reviewer | Agent: pr-reviewer
Author
Owner

[GROOMED] Final quality analysis and approval confirmation.
[AUTO-GROOM-10214]

Summary

PR #10214 has been thoroughly groomed and reviewed. All quality criteria are satisfied.

Quality Checks

Check Status Details
Labels Complete State/In Review, Type/Documentation, Priority/CI Blocker, MoSCoW/Must have
Milestone Set v3.5.0 (M6: Autonomy Hardening)
Linked Issue Valid Closes #10042 via "Fixes: #10042" in PR body
Assignee ℹ️ None Not required for documentation PRs
Reviews APPROVED HAL9001 approved on 2026-04-18T08:23:08Z
REQUEST_CHANGES None No blocking reviews
CI Status Passing All 13 CI checks passed (lint, typecheck, security, unit_tests, coverage, e2e_tests, etc.)
Stale Activity Recent Last update 2026-04-18T08:39:51Z (today)
Duplicate Detection None No duplicate PRs identified
Previous Grooming Valid Prior grooming (2026-04-17) remains valid; all fixes applied

Content Quality

Documentation Accuracy: PR accurately documents the LLM provider fallback behavior introduced in commit f5712787

Completeness:

  • New reference doc docs/reference/llm_provider_fallback.md (193 lines) is comprehensive
  • Covers trigger conditions, state machine, recovery interval, logging, configuration
  • README.md and CHANGELOG.md updates are accurate and concise

No Source Code Changes: Documentation-only PR; no Python source files modified

Branch Naming: docs/auto-docs-7-llm-fallback-behavior follows convention

Final Status

PR State: Open, awaiting merge
Review State: APPROVED
Grooming State: COMPLETE
Ready for Merge: Yes (pending merge decision by maintainers)

Notes

  • PR is marked as mergeable: false in API response, but this appears to be a transient state or branch protection rule
  • All substantive quality criteria are satisfied
  • Reviewer (HAL9001) has approved with comprehensive checklist
  • No action items remain for grooming

Automated by CleverAgents Bot
Supervisor: Grooming | Agent: grooming-pool-supervisor

[GROOMED] Final quality analysis and approval confirmation. [AUTO-GROOM-10214] ## Summary PR #10214 has been thoroughly groomed and reviewed. All quality criteria are satisfied. ## Quality Checks | Check | Status | Details | |-------|--------|----------| | Labels | ✅ Complete | State/In Review, Type/Documentation, Priority/CI Blocker, MoSCoW/Must have | | Milestone | ✅ Set | v3.5.0 (M6: Autonomy Hardening) | | Linked Issue | ✅ Valid | Closes #10042 via "Fixes: #10042" in PR body | | Assignee | ℹ️ None | Not required for documentation PRs | | Reviews | ✅ APPROVED | HAL9001 approved on 2026-04-18T08:23:08Z | | REQUEST_CHANGES | ✅ None | No blocking reviews | | CI Status | ✅ Passing | All 13 CI checks passed (lint, typecheck, security, unit_tests, coverage, e2e_tests, etc.) | | Stale Activity | ✅ Recent | Last update 2026-04-18T08:39:51Z (today) | | Duplicate Detection | ✅ None | No duplicate PRs identified | | Previous Grooming | ✅ Valid | Prior grooming (2026-04-17) remains valid; all fixes applied | ## Content Quality ✅ **Documentation Accuracy**: PR accurately documents the LLM provider fallback behavior introduced in commit `f5712787` ✅ **Completeness**: - New reference doc `docs/reference/llm_provider_fallback.md` (193 lines) is comprehensive - Covers trigger conditions, state machine, recovery interval, logging, configuration - `README.md` and `CHANGELOG.md` updates are accurate and concise ✅ **No Source Code Changes**: Documentation-only PR; no Python source files modified ✅ **Branch Naming**: `docs/auto-docs-7-llm-fallback-behavior` follows convention ## Final Status **PR State**: Open, awaiting merge **Review State**: APPROVED ✅ **Grooming State**: COMPLETE ✅ **Ready for Merge**: Yes (pending merge decision by maintainers) ## Notes - PR is marked as `mergeable: false` in API response, but this appears to be a transient state or branch protection rule - All substantive quality criteria are satisfied - Reviewer (HAL9001) has approved with comprehensive checklist - No action items remain for grooming --- **Automated by CleverAgents Bot** Supervisor: Grooming | Agent: grooming-pool-supervisor
HAL9000 force-pushed docs/auto-docs-7-llm-fallback-behavior from 7fd004084f
All checks were successful
CI / lint (pull_request) Successful in 20s
CI / quality (pull_request) Successful in 19s
CI / build (pull_request) Successful in 36s
CI / push-validation (pull_request) Successful in 20s
CI / security (pull_request) Successful in 45s
CI / typecheck (pull_request) Successful in 46s
CI / helm (pull_request) Successful in 28s
CI / integration_tests (pull_request) Successful in 4m18s
CI / e2e_tests (pull_request) Successful in 4m37s
CI / unit_tests (pull_request) Successful in 5m36s
CI / docker (pull_request) Successful in 53s
CI / coverage (pull_request) Successful in 7m32s
CI / status-check (pull_request) Successful in 1s
to 4e86ffe596
Some checks failed
CI / push-validation (pull_request) Successful in 22s
CI / helm (pull_request) Successful in 32s
CI / build (pull_request) Successful in 3m47s
CI / lint (pull_request) Successful in 3m54s
CI / quality (pull_request) Successful in 4m20s
CI / typecheck (pull_request) Successful in 4m34s
CI / security (pull_request) Successful in 4m36s
CI / integration_tests (pull_request) Successful in 6m53s
CI / e2e_tests (pull_request) Successful in 6m58s
CI / unit_tests (pull_request) Successful in 7m20s
CI / docker (pull_request) Successful in 1m29s
CI / coverage (pull_request) Successful in 13m37s
CI / benchmark-regression (push) Waiting to run
CI / benchmark-publish (push) Waiting to run
CI / status-check (pull_request) Successful in 2s
CI / push-validation (push) Successful in 25s
CI / helm (push) Successful in 41s
CI / build (push) Successful in 3m50s
CI / lint (push) Successful in 3m57s
CI / quality (push) Successful in 4m23s
CI / typecheck (push) Successful in 4m30s
CI / security (push) Successful in 4m53s
CI / e2e_tests (push) Successful in 6m47s
CI / integration_tests (push) Successful in 7m49s
CI / unit_tests (push) Successful in 9m3s
CI / docker (push) Successful in 1m42s
CI / coverage (push) Successful in 15m19s
CI / status-check (push) Successful in 3s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Failing after 13m19s
2026-04-21 17:27:37 +00:00
Compare
HAL9000 scheduled this pull request to auto merge when all checks succeed 2026-04-21 17:31:27 +00:00
HAL9000 merged commit 4e86ffe596 into master 2026-04-21 17:45:53 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleveragents/cleveragents-core!10214
No description provided.