#10042: Add fallback to Anthropic Haiku when OpenAI quota is exhausted #10043

Merged

HAL9000 merged 2 commits from feature/10042-openai-quota-fallback into master 2026-04-17 04:11:42 +00:00

Conversation 3 Commits 2 Files Changed 6

+401 -64

2 Commits

Author	SHA1	Message	Date
HAL9000	51472c0b37	debug: upgrade logging levels for fallback diagnostics ... Change fallback LLM creation and invocation logs from DEBUG to WARNING level so they appear in Robot Framework test output. Also enhance error message to clearly show which provider failed and why. This change makes it possible to diagnose why the fallback is not working by seeing the actual logs in test output instead of having them filtered as DEBUG level messages. Logs now include: - 'Creating fallback LLM instance: anthropic/claude-sonnet-4-20250514' - 'Fallback LLM created, attempting invocation' - 'Using cached fallback LLM, attempting invocation' - 'FALLBACK PROVIDER FAILED: anthropic/claude-sonnet-4-20250514 returned error: [error details]' This will help diagnose why E2E tests fail with 'both providers exhausted' when Anthropic should have available credits.	2026-04-17 03:56:35 +00:00
CoreRasurae	f5712787e0	feat: add fallback to Anthropic Sonnet when OpenAI quota is exhausted ... Implements graceful degradation for E2E robot integration tests that hit OpenAI 429 quota limit errors. Changes: - Add _is_quota_error() helper to detect quota-specific API errors (429, insufficient_quota, rate_limit) - Modify _execute_with_llm() in StrategyActor to catch quota errors and attempt fallback to Anthropic Haiku - Configure fallback provider as 'anthropic/claude-sonnet-4-20250514' - Add comprehensive logging for quota error detection and provider fallback - Add E2E test scenarios for quota fallback verification When quota errors occur on both OpenAI and Anthropic fallback, tests now fail with a clear message explaining that the test outcome cannot be verified when no LLM provider is available. This ensures CI/CD pipelines properly track which tests could not be executed due to quota constraints, rather than silently skipping them and creating false confidence in test coverage. This ensures CI/CD pipelines can complete E2E tests even when the primary provider (OpenAI) hits quota limits, improving pipeline reliability and reducing false negatives caused by provider-specific issues. 1. **Cache fallback_llm instance** - Instead of recreating the fallback LLM every time a quota error occurs, cache it as an instance variable (self._fallback_llm). This avoids unnecessary re-initialization overhead. 2. **Implement quota recovery logic** - Add intelligent recovery behavior: - Track last quota error timestamp (self._last_quota_error_time) - Track fallback mode state (self._using_fallback) - Once quota error detected, switch to fallback provider - Only attempt to recover primary provider every 5 minutes (_QUOTA_RECOVERY_INTERVAL) - This avoids hammering primary provider with repeated quota errors 3. **Add detailed recovery logging** - Log quota fallback transitions and recovery attempts to improve observability and debugging. Benefits: - Reduced latency: No redundant primary provider calls after quota error - Reduced overhead: Cached fallback LLM instance, no per-call recreation - Better observability: Clear logging of fallback mode entry/exit - Intelligent recovery: Automatic recovery attempt after 5-minute interval Updated tests: - M6 E2E Event Queue Via Plan Lifecycle Transitions - M6 E2E Hierarchical Decomposition Via Plan Tree - M6 E2E Full Autonomy Acceptance Flow Fixes: #10042	2026-04-17 03:56:35 +00:00