Files
cleveragents-core/features/steps
CoreRasurae f5712787e0 feat: add fallback to Anthropic Sonnet when OpenAI quota is exhausted
Implements graceful degradation for E2E robot integration tests that hit OpenAI 429 quota limit errors.

Changes:
- Add _is_quota_error() helper to detect quota-specific API errors (429, insufficient_quota, rate_limit)
- Modify _execute_with_llm() in StrategyActor to catch quota errors and attempt fallback to Anthropic Haiku
- Configure fallback provider as 'anthropic/claude-sonnet-4-20250514'
- Add comprehensive logging for quota error detection and provider fallback
- Add E2E test scenarios for quota fallback verification

When quota errors occur on both OpenAI and Anthropic fallback, tests
now fail with a clear message explaining that the test outcome cannot
be verified when no LLM provider is available.
 This ensures CI/CD pipelines properly track which tests could not be
executed due to quota constraints, rather than silently skipping them
and creating false confidence in test coverage.

This ensures CI/CD pipelines can complete E2E tests even when the primary provider (OpenAI) hits quota limits,
improving pipeline reliability and reducing false negatives caused by provider-specific issues.

1. **Cache fallback_llm instance** - Instead of recreating the fallback LLM
   every time a quota error occurs, cache it as an instance variable
   (self._fallback_llm). This avoids unnecessary re-initialization overhead.

2. **Implement quota recovery logic** - Add intelligent recovery behavior:
   - Track last quota error timestamp (self._last_quota_error_time)
   - Track fallback mode state (self._using_fallback)
   - Once quota error detected, switch to fallback provider
   - Only attempt to recover primary provider every 5 minutes (_QUOTA_RECOVERY_INTERVAL)
   - This avoids hammering primary provider with repeated quota errors

3. **Add detailed recovery logging** - Log quota fallback transitions and
   recovery attempts to improve observability and debugging.

Benefits:
- Reduced latency: No redundant primary provider calls after quota error
- Reduced overhead: Cached fallback LLM instance, no per-call recreation
- Better observability: Clear logging of fallback mode entry/exit
- Intelligent recovery: Automatic recovery attempt after 5-minute interval

Updated tests:
- M6 E2E Event Queue Via Plan Lifecycle Transitions
- M6 E2E Hierarchical Decomposition Via Plan Tree
- M6 E2E Full Autonomy Acceptance Flow

Fixes: #10042
2026-04-17 03:56:35 +00:00
..