Implements graceful degradation for E2E robot integration tests that hit OpenAI 429 quota limit errors.
Changes:
- Add _is_quota_error() helper to detect quota-specific API errors (429, insufficient_quota, rate_limit)
- Modify _execute_with_llm() in StrategyActor to catch quota errors and attempt fallback to Anthropic Haiku
- Configure fallback provider as 'anthropic/claude-sonnet-4-20250514'
- Add comprehensive logging for quota error detection and provider fallback
- Add E2E test scenarios for quota fallback verification
When quota errors occur on both OpenAI and Anthropic fallback, tests
now fail with a clear message explaining that the test outcome cannot
be verified when no LLM provider is available.
This ensures CI/CD pipelines properly track which tests could not be
executed due to quota constraints, rather than silently skipping them
and creating false confidence in test coverage.
This ensures CI/CD pipelines can complete E2E tests even when the primary provider (OpenAI) hits quota limits,
improving pipeline reliability and reducing false negatives caused by provider-specific issues.
1. **Cache fallback_llm instance** - Instead of recreating the fallback LLM
every time a quota error occurs, cache it as an instance variable
(self._fallback_llm). This avoids unnecessary re-initialization overhead.
2. **Implement quota recovery logic** - Add intelligent recovery behavior:
- Track last quota error timestamp (self._last_quota_error_time)
- Track fallback mode state (self._using_fallback)
- Once quota error detected, switch to fallback provider
- Only attempt to recover primary provider every 5 minutes (_QUOTA_RECOVERY_INTERVAL)
- This avoids hammering primary provider with repeated quota errors
3. **Add detailed recovery logging** - Log quota fallback transitions and
recovery attempts to improve observability and debugging.
Benefits:
- Reduced latency: No redundant primary provider calls after quota error
- Reduced overhead: Cached fallback LLM instance, no per-call recreation
- Better observability: Clear logging of fallback mode entry/exit
- Intelligent recovery: Automatic recovery attempt after 5-minute interval
Updated tests:
- M6 E2E Event Queue Via Plan Lifecycle Transitions
- M6 E2E Hierarchical Decomposition Via Plan Tree
- M6 E2E Full Autonomy Acceptance Flow
Fixes: #10042
CoreRasurae
f5712787e0