diff --git a/CHANGELOG.md b/CHANGELOG.md index 252508d0f..0fbb93567 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,86 +5,8 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] -### Added - -- **Plan diff shows worktree branch changes** (#9231): `plan diff` now detects - the worktree branch `cleveragents/plan-` created during `plan execute` - and runs `git diff HEAD...` to display actual file changes. Falls - back to changeset-based diff when no worktree branch exists. Git operations - delegated to `GitWorktreeSandbox.diff_against_head()` in the Infrastructure - layer. - ### Fixed -- **Stale worktree branch cleanup on re-execute** (#7271): Re-executing a plan - after a failed attempt or diff review crashed with `fatal: a branch named - 'cleveragents/plan-' already exists`. Added `GitWorktreeSandbox.cleanup_stale()` - classmethod that idempotently removes stale worktree directories and branches - before creating a fresh sandbox. - -- **Worktree sandbox cleanup on plan cancel** (#9230): `plan cancel` now removes - the git worktree branch and directory created during execute, preventing resource - leaks from dangling worktrees. Delegates to `GitWorktreeSandbox.cleanup_stale()` - in the infrastructure layer. - -- **`plan apply` merge conflict cleanup** (#7250): When `git merge` fails due to - a conflict during `plan apply`, the apply command now reads the actual conflict - detail from `CalledProcessError.stdout` (git writes conflict info to stdout, not - stderr), runs `git merge --abort` to restore the repo to a clean state, transitions - the plan to `constrained` state per spec §18334-18336 (may revert to Strategize - for re-planning), and prints user-friendly guidance. Also handles - `subprocess.TimeoutExpired` on both merge and abort calls. Previously, merge - conflicts left conflict markers (`<<<<<<<`, `=======`, `>>>>>>>`) in project files - and the plan remained in `apply/queued` indefinitely. - -- **Actor v3 YAML Schema Validation in CLI** (#5869): The `agents actor add --config` - command now validates v3 YAML files using `ActorConfigSchema`, ensuring proper - schema compliance including cycle detection for GRAPH actors, required field - validation, and enum validation. v3 YAML is detected by the presence of ANY - `type` field (any value — invalid type values are then rejected by schema - validation) or a `version` field whose string value starts with `"3"` (e.g. - `"3"`, `"3.0"`, `"3.0.0"`). Configs with `type: null` are not treated as v3. - Invalid v3 actors are rejected with clear error messages before registration. - -- **Alembic Files Missing from Wheel Distribution** (#4180): Alembic configuration - (`alembic.ini`) and migration files are now part of the Python package structure - at `src/cleveragents/infrastructure/database/migrations/`. Previously, when - `agents init` was run in Docker containers or any wheel-based installation, - `FileNotFoundError` was raised because alembic files were stored at the - repository root and excluded from the wheel distribution. Now alembic files - follow standard Python packaging conventions and are automatically included. - `MigrationRunner._find_alembic_ini()` has been updated to search the new package - location as the primary anchor point. This fix enables `agents init` to work - correctly in all deployment modes: Docker containers, local pip installs - (wheel or editable), and development environments. -- **TDD Non-AssertionError Guard Visibility** (#8294): `apply_tdd_inversion` in -- **bug-hunt-pool-supervisor Non-Blocking Tracking** (#8835): The automation-tracking-manager - call in step 5 was blocking the main loop indefinitely, causing 3+ consecutive initialization - failures. Step 5 now explicitly marks tracking as best-effort -- if the call does not complete - within a reasonable time or fails, it is skipped and the supervisor continues to the next - cycle. A new Rule 9 reinforces that tracking must never block the main loop; core - functionality (module mapping, worker dispatch, monitoring) takes priority over status - reporting. - - `features/environment.py` now emits its non-assertion exception guard warning to - both the structured logger and `stderr` via a new `_warning_with_stderr` helper. - This makes the guard firing visible in standard Behave console output and CI log - snippets where the structured logging sink may not be displayed. BDD infrastructure - coverage added: a new scenario in `tdd_expected_fail_infrastructure.feature` - asserts that the warning is emitted to stderr when a non-AssertionError exception - is encountered in an `@tdd_expected_fail` scenario, and a second scenario asserts - the warning is NOT emitted when the exception is an `AssertionError`. The - `CONTRIBUTING.md` now documents that `@tdd_expected_fail` step definitions must - signal expected failures via `AssertionError`. - -- **Parallel Behave Runner Log Noise Reduction** (#8351): The parallel behave - runner now suppresses captured stdout/stderr for passing worker chunks and - only replays diagnostics for failed, errored, or crashed chunks. This makes - failure output significantly easier to spot in CI and local runs. A worker - crash (unhandled exception) is detected via an all-zero summary and the - captured traceback is always surfaced. -- **Directory Clustering Absolute Path Fix** (#9401): Fixed `DecompositionService._directory_key` to correctly handle absolute file paths by computing relative paths before extracting directory keys. Previously, the function used a fixed depth of 2 path components, causing all absolute paths to collapse into a single bucket (e.g., `/home` for every file on the system), making directory-based clustering completely ineffective. The fix adds an optional `root` parameter to `_directory_key()` and `ClusteringStrategy.cluster_by_directory()`, and updates `DecompositionService._build_hierarchy()` to compute the common root and pass it through, ensuring directory clustering groups paths by their actual directory hierarchy in production use. - - **Automation Profile Silent Fallback** (#8232): `_resolve_profile_for_plan` in `PlanLifecycleService` now raises a clear `ValidationError` when a plan's automation profile name is not a known built-in profile, instead of silently @@ -93,37 +15,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). message listing available built-in profiles. The resolved profile name is also logged at debug level for observability. -- **CheckpointManager rollback_to always returned False** (#7488): Fixed a data - integrity bug in `CheckpointManager.create_checkpoint()` where `sandbox_path` - was computed from `sandbox.context.sandbox_path` but never stored in the - checkpoint metadata. As a result, `rollback_to()` always found - `checkpoint.metadata.get("sandbox_path")` returning `None` and silently - skipped the rollback, returning `False`. The fix adds `sandbox_path` to the - metadata dict before constructing the `SandboxCheckpoint`, enabling - `rollback_to()` to correctly restore the sandbox filesystem state. - -- **Strategy Actor Code Review Follow-ups** (#10267): Implemented comprehensive - fixes for all 9 code review findings from PR #1175 strategy actor implementation: - - Module-level imports: Moved `json` import to module-level in - `plan_executor_coverage_steps.py` following Python conventions and ruff/isort - standards. - - Exception handling clarity: Removed redundant `json.JSONDecodeError` from - exception clause (Exception already subsumes it). - - Silent fallback logging: Added warning log when `actor.default.strategy` config - resolution fails, providing operator visibility into configuration issues. - - Structured content block handling: Enhanced `_extract_content()` in - `strategy_actor.py` to properly extract text from LangChain `MessageContentBlock` - dicts, preventing JSON parsing failures when structured message content is - encountered. - - Deduplicated constant: Removed local `_DEFAULT_ACTOR_NAME` definition from - `strategy_actor.py` and imported from canonical source in `strategy_resolution.py` - to prevent future drift. - - Test decoupling: Added `strategy_tree: StrategyTree | None` field to - `StrategizeResult` to allow tests to inspect the tree without calling private - `_execute_with_llm()` method, eliminating divergent ULID generation from - double-invocation pattern. Updated 6 step functions in `strategy_actor_llm_steps.py` - to use the public API instead of coupling to private implementation details. - ### Added - **TDD Issue-Capture Test Activation** (#7025): Replaced 234 bare `@skip` tags @@ -178,6 +69,27 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Priority-based filtering (Critical/High/Medium/Low) reduces noise. Backlog-groomer performs intelligent cleanup with age thresholds by priority. +- **OpenAI Quota Fallback to Anthropic Claude Sonnet** (#10042): Implemented graceful + degradation for E2E robot integration tests when OpenAI API hits quota limit errors + (429, insufficient_quota, rate_limit). The `StrategyActor` now detects quota-specific + errors via `_is_quota_error()` and automatically falls back to + `anthropic/claude-sonnet-4-20250514` for strategy decisions, ensuring CI/CD pipelines + complete E2E tests even when the primary provider hits capacity limits. Includes + intelligent recovery: the fallback LLM instance is cached (`_fallback_llm`), fallback + mode is tracked (`_using_fallback`), and the primary provider is re-attempted after a + 5-minute recovery interval (`_QUOTA_RECOVERY_INTERVAL = 300`). Added comprehensive + logging for quota error detection and provider fallback transitions, plus E2E test + scenarios for fallback verification. Implemented in + `StrategyActor._execute_with_llm()` in + `src/cleveragents/application/services/strategy_actor.py` (commit `f5712787`). + +- **Documentation: LLM Provider Fallback Behavior** [AUTO-DOCS-7]: Added + `docs/reference/llm_provider_fallback.md` documenting the automatic Anthropic Claude + Sonnet fallback when OpenAI quota is exhausted. Covers trigger conditions, fallback + state machine, recovery logic, logging reference, configuration constants, and + prerequisites. Updated `README.md` with an "Automatic quota fallback" subsection + under LLM provider configuration. + - **PR Agent Reorganization**: All PR-related agents renamed and reorganized to follow the `*-pool-supervisor` naming pattern. New agents added: `pr-editor` (safe PR editing with description preservation), `pr-manager` (unified PR interface), and @@ -260,15 +172,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). prefix check using OS path separators. Added regression test tagged `@tdd_issue_7558`. -- **File Edit Encoding Parameter** (#7559): `_handle_file_edit()` now correctly - respects the `encoding` parameter when reading and writing files. Previously the - function ignored the caller-supplied encoding and fell back to the platform default, - causing data corruption with non-UTF-8 files. The fix extracts `encoding` from the - tool inputs (defaulting to `"utf-8"`) and passes it to both `path.read_text()` and - `path.write_text()`. The `FILE_EDIT_SPEC` input schema was updated to declare the - `encoding` field. BDD scenarios were added to cover explicit encoding and the - UTF-8 default. - - **Validation Gate Empty-Run Guard** (#7508): Fixed `ApplyValidationSummary.all_required_passed` returning `True` when zero validations were run, silently bypassing the apply gate. The property now returns `False` when the validation result set is empty (`is_empty` is `True`), ensuring @@ -311,20 +214,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). `pr-merge-pool-supervisor` to the product-builder's supervisor launch list (18 total supervisors). Updated all numeric references, pre-flight checklists, and validation logic. -- `ActionRepository.update()` now uses explicit bulk `sa_delete()` + `session.flush()` - before re-inserting child rows for `action_arguments` and `action_invariants`, fixing - a `sqlite3.IntegrityError: UNIQUE constraint failed` crash when `agents plan use` was - called on an action that already had arguments registered via `action create`. (#4197) - -- Fixed CheckpointManager not wired into PlanExecutor — checkpoints were - never created during plan execution because `_get_plan_executor()` in - the CLI constructed PlanExecutor without a CheckpointManager (defaulted - to None, silently skipping all checkpoint hooks). `_get_plan_executor()` - now resolves the container singleton so CLI `plan execute` and `plan - rollback` share the same registry, and `_try_create_checkpoint()` raises - `PlanError` if checkpoint metadata cannot be persisted. Writable sandboxable - resources and write-capable tools now default to `checkpointable=True`, and - new Behave scenarios cover DI wiring, rollback, and capability defaults. (#1253) --- ## [3.8.0] — 2026-04-05 diff --git a/README.md b/README.md index 733986eda..64386fefc 100644 --- a/README.md +++ b/README.md @@ -170,3 +170,15 @@ Set `CLEVERAGENTS_DEFAULT_PROVIDER` to pin the global provider (for example `exp - Built-in actors (`/`) are immutable, custom actors must be named `local/`, and the default actor cannot be removed. Use `--unsafe` when adding/updating configs marked unsafe; runtime only warns when invoking unsafe actors. - `CLEVERAGENTS_TESTING_USE_MOCK_AI=true` forces the in-repo mock provider so Behave/Robot suites never hit external APIs. - The full capability matrix (streaming, tool calls, JSON mode, etc.) is documented in `docs/reference/providers.md`. + +### Automatic quota fallback + +When the primary LLM provider (e.g. OpenAI) returns a quota error (HTTP 429, +`insufficient_quota`, or `rate_limit`), `StrategyActor` automatically falls back to +**Anthropic Claude Sonnet (`claude-sonnet-4-20250514`)** for strategy decisions. +The fallback is transparent — plan operations continue without interruption. The +system re-attempts the primary provider after a 5-minute recovery interval. + +Requires `ANTHROPIC_API_KEY` to be set for the fallback to succeed. See +[`docs/reference/llm_provider_fallback.md`](docs/reference/llm_provider_fallback.md) +for full details including state machine, logging, and configuration constants. diff --git a/docs/reference/llm_provider_fallback.md b/docs/reference/llm_provider_fallback.md new file mode 100644 index 000000000..a193a89f9 --- /dev/null +++ b/docs/reference/llm_provider_fallback.md @@ -0,0 +1,193 @@ +# LLM Provider Fallback Behavior + +## Overview + +CleverAgents implements automatic LLM provider fallback to ensure resilience when the +primary provider (typically OpenAI) exhausts its API quota. When a quota error is +detected, the system transparently switches to **Anthropic Claude Sonnet +(`claude-sonnet-4-20250514`)** as the fallback provider, allowing plan strategize +operations to continue without interruption. + +This behavior was introduced in commit `f5712787` (feat: add fallback to Anthropic +Sonnet when OpenAI quota is exhausted) and is implemented in +`StrategyActor._execute_with_llm()` located in +`src/cleveragents/application/services/strategy_actor.py`. + +--- + +## Trigger Conditions + +The fallback activates when the primary LLM call raises an exception whose string +representation matches any of the following patterns (case-insensitive): + +| Pattern | Typical Source | +|---------|---------------| +| `insufficient_quota` | OpenAI quota exhaustion | +| `429` | HTTP 429 Too Many Requests | +| `quota` | Generic quota error from any provider | +| `rate_limit` | Rate-limit errors from any provider | + +Detection is performed by the module-level helper function `_is_quota_error(exc)` in +`src/cleveragents/application/services/strategy_actor.py`. + +--- + +## Fallback Provider + +| Constant | Value | Description | +|----------|-------|-------------| +| `_FALLBACK_PROVIDER` | `"anthropic"` | Provider identifier | +| `_FALLBACK_MODEL` | `"claude-sonnet-4-20250514"` | Model identifier | +| `_QUOTA_RECOVERY_INTERVAL` | `300` (seconds) | Time before re-attempting primary provider | + +The fallback LLM instance is created lazily on first use and then **cached** as +`StrategyActor._fallback_llm` to avoid per-call re-initialization overhead. + +--- + +## Fallback State Machine + +`StrategyActor` tracks fallback state using three instance variables: + +| Variable | Type | Description | +|----------|------|-------------| +| `_fallback_llm` | `Any \| None` | Cached fallback LLM instance | +| `_last_quota_error_time` | `float \| None` | Unix timestamp of the last quota error | +| `_using_fallback` | `bool` | Whether the actor is currently in fallback mode | + +### State Transitions + +``` +Normal mode (_using_fallback=False) + | + | Primary provider raises quota error + v +Fallback mode (_using_fallback=True, _last_quota_error_time=) + | + | _QUOTA_RECOVERY_INTERVAL (300 s) elapses + v +Recovery attempt (_using_fallback=False, primary re-tried) + | + +-- Primary succeeds --> Normal mode + +-- Primary raises quota error again --> Fallback mode +``` + +### Recovery Logic + +Once in fallback mode, `_execute_with_llm()` skips the primary provider call for +subsequent invocations until `_QUOTA_RECOVERY_INTERVAL` seconds have elapsed since +the last quota error. This prevents hammering the primary provider with repeated +quota errors. After the interval, the primary provider is re-attempted automatically. + +--- + +## Behavior from the User Perspective + +From the user's perspective, the fallback is **transparent**: + +- Plan strategize operations continue to produce valid strategy trees. +- Response quality and format are identical — both OpenAI and Anthropic Claude Sonnet + support the structured JSON output required by the strategy prompt. +- No user action is required to trigger or disable the fallback. +- The fallback is **temporary**: the system automatically attempts to recover to the + primary provider after 5 minutes. + +### When Both Providers Are Unavailable + +If the fallback provider (Anthropic Claude Sonnet) also fails, the original quota +exception from the primary provider is re-raised (chained from the fallback +exception). The `StrategyActor.execute()` method catches this and falls back to the +deterministic stub strategy (`_execute_stub()`), ensuring the plan lifecycle is never +blocked by LLM unavailability. + +In E2E test contexts, when both providers are unavailable, tests fail with a clear +message indicating that the test outcome cannot be verified — rather than silently +skipping and creating false confidence in test coverage. + +--- + +## Logging + +The fallback mechanism emits structured log events at the following levels: + +| Level | Event | Key Fields | +|-------|-------|-----------| +| `WARNING` | Quota error detected on primary provider | `plan_id`, `original_error` | +| `WARNING` | Creating fallback LLM instance | `plan_id` | +| `WARNING` | Reusing cached fallback LLM instance | `plan_id` | +| `WARNING` | Fallback recovery successful | `plan_id`, `fallback_provider`, `fallback_model` | +| `ERROR` | Fallback provider also failed | `plan_id`, `fallback_provider`, `fallback_model`, `fallback_error`, `fallback_error_type` | +| `DEBUG` | Still in quota fallback mode (with time remaining) | `plan_id` | +| `INFO` | Quota recovery interval elapsed, attempting primary | `plan_id`, `time_since_error` | + +All log events include `plan_id` for correlation. Enable `DEBUG` level logging to +observe fallback mode transitions in detail. + +--- + +## Configuration + +The fallback behavior is controlled by module-level constants in +`src/cleveragents/application/services/strategy_actor.py`. These are not currently +exposed as environment variables or configuration keys. To change the fallback +provider or recovery interval, modify the constants directly: + +```python +# src/cleveragents/application/services/strategy_actor.py +_FALLBACK_PROVIDER = "anthropic" # Provider identifier +_FALLBACK_MODEL = "claude-sonnet-4-20250514" # Model identifier +_QUOTA_RECOVERY_INTERVAL = 300 # Seconds before re-trying primary +``` + +> **Note:** The fallback provider requires a valid `ANTHROPIC_API_KEY` environment +> variable to be set. If the key is absent, the fallback LLM creation will fail and +> the original quota exception will be re-raised. + +--- + +## Prerequisites + +To enable the Anthropic fallback, ensure the following environment variable is set: + +```bash +export ANTHROPIC_API_KEY= +``` + +The primary provider (e.g. OpenAI) is configured as usual: + +```bash +export OPENAI_API_KEY= +``` + +See the [LLM provider configuration](../../README.md#llm-provider-configuration) +section of the README for the full list of supported providers and environment +variables. + +--- + +## Implementation Reference + +| Symbol | Location | Description | +|--------|----------|-------------| +| `_is_quota_error(exc)` | `src/cleveragents/application/services/strategy_actor.py` | Detects quota errors by string matching | +| `StrategyActor._execute_with_llm()` | `src/cleveragents/application/services/strategy_actor.py` | Orchestrates primary call, quota detection, and fallback | +| `StrategyActor._fallback_llm` | `src/cleveragents/application/services/strategy_actor.py` | Cached fallback LLM instance | +| `StrategyActor._last_quota_error_time` | `src/cleveragents/application/services/strategy_actor.py` | Timestamp of last quota error | +| `StrategyActor._using_fallback` | `src/cleveragents/application/services/strategy_actor.py` | Fallback mode flag | +| `_FALLBACK_PROVIDER` | `src/cleveragents/application/services/strategy_actor.py` | `"anthropic"` | +| `_FALLBACK_MODEL` | `src/cleveragents/application/services/strategy_actor.py` | `"claude-sonnet-4-20250514"` | +| `_QUOTA_RECOVERY_INTERVAL` | `src/cleveragents/application/services/strategy_actor.py` | `300` seconds | + +**Introduced in:** commit `f5712787` — *feat: add fallback to Anthropic Sonnet when +OpenAI quota is exhausted* + +**Fixes:** issue #10042 + +--- + +## Related Documentation + +- [Retry Policy](retry_policy.md) — service-level retry and circuit breaker configuration +- [Error Recovery](error_recovery.md) — plan-level error classification and recovery actions +- [Error Handling](error_handling.md) — error codes, classification, and secret redaction +- [Observability](observability.md) — structured logging and tracing configuration