diff --git a/CHANGELOG.md b/CHANGELOG.md index 0fbb93567..252508d0f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,8 +5,86 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Added + +- **Plan diff shows worktree branch changes** (#9231): `plan diff` now detects + the worktree branch `cleveragents/plan-` created during `plan execute` + and runs `git diff HEAD...` to display actual file changes. Falls + back to changeset-based diff when no worktree branch exists. Git operations + delegated to `GitWorktreeSandbox.diff_against_head()` in the Infrastructure + layer. + ### Fixed +- **Stale worktree branch cleanup on re-execute** (#7271): Re-executing a plan + after a failed attempt or diff review crashed with `fatal: a branch named + 'cleveragents/plan-' already exists`. Added `GitWorktreeSandbox.cleanup_stale()` + classmethod that idempotently removes stale worktree directories and branches + before creating a fresh sandbox. + +- **Worktree sandbox cleanup on plan cancel** (#9230): `plan cancel` now removes + the git worktree branch and directory created during execute, preventing resource + leaks from dangling worktrees. Delegates to `GitWorktreeSandbox.cleanup_stale()` + in the infrastructure layer. + +- **`plan apply` merge conflict cleanup** (#7250): When `git merge` fails due to + a conflict during `plan apply`, the apply command now reads the actual conflict + detail from `CalledProcessError.stdout` (git writes conflict info to stdout, not + stderr), runs `git merge --abort` to restore the repo to a clean state, transitions + the plan to `constrained` state per spec §18334-18336 (may revert to Strategize + for re-planning), and prints user-friendly guidance. Also handles + `subprocess.TimeoutExpired` on both merge and abort calls. Previously, merge + conflicts left conflict markers (`<<<<<<<`, `=======`, `>>>>>>>`) in project files + and the plan remained in `apply/queued` indefinitely. + +- **Actor v3 YAML Schema Validation in CLI** (#5869): The `agents actor add --config` + command now validates v3 YAML files using `ActorConfigSchema`, ensuring proper + schema compliance including cycle detection for GRAPH actors, required field + validation, and enum validation. v3 YAML is detected by the presence of ANY + `type` field (any value — invalid type values are then rejected by schema + validation) or a `version` field whose string value starts with `"3"` (e.g. + `"3"`, `"3.0"`, `"3.0.0"`). Configs with `type: null` are not treated as v3. + Invalid v3 actors are rejected with clear error messages before registration. + +- **Alembic Files Missing from Wheel Distribution** (#4180): Alembic configuration + (`alembic.ini`) and migration files are now part of the Python package structure + at `src/cleveragents/infrastructure/database/migrations/`. Previously, when + `agents init` was run in Docker containers or any wheel-based installation, + `FileNotFoundError` was raised because alembic files were stored at the + repository root and excluded from the wheel distribution. Now alembic files + follow standard Python packaging conventions and are automatically included. + `MigrationRunner._find_alembic_ini()` has been updated to search the new package + location as the primary anchor point. This fix enables `agents init` to work + correctly in all deployment modes: Docker containers, local pip installs + (wheel or editable), and development environments. +- **TDD Non-AssertionError Guard Visibility** (#8294): `apply_tdd_inversion` in +- **bug-hunt-pool-supervisor Non-Blocking Tracking** (#8835): The automation-tracking-manager + call in step 5 was blocking the main loop indefinitely, causing 3+ consecutive initialization + failures. Step 5 now explicitly marks tracking as best-effort -- if the call does not complete + within a reasonable time or fails, it is skipped and the supervisor continues to the next + cycle. A new Rule 9 reinforces that tracking must never block the main loop; core + functionality (module mapping, worker dispatch, monitoring) takes priority over status + reporting. + + `features/environment.py` now emits its non-assertion exception guard warning to + both the structured logger and `stderr` via a new `_warning_with_stderr` helper. + This makes the guard firing visible in standard Behave console output and CI log + snippets where the structured logging sink may not be displayed. BDD infrastructure + coverage added: a new scenario in `tdd_expected_fail_infrastructure.feature` + asserts that the warning is emitted to stderr when a non-AssertionError exception + is encountered in an `@tdd_expected_fail` scenario, and a second scenario asserts + the warning is NOT emitted when the exception is an `AssertionError`. The + `CONTRIBUTING.md` now documents that `@tdd_expected_fail` step definitions must + signal expected failures via `AssertionError`. + +- **Parallel Behave Runner Log Noise Reduction** (#8351): The parallel behave + runner now suppresses captured stdout/stderr for passing worker chunks and + only replays diagnostics for failed, errored, or crashed chunks. This makes + failure output significantly easier to spot in CI and local runs. A worker + crash (unhandled exception) is detected via an all-zero summary and the + captured traceback is always surfaced. +- **Directory Clustering Absolute Path Fix** (#9401): Fixed `DecompositionService._directory_key` to correctly handle absolute file paths by computing relative paths before extracting directory keys. Previously, the function used a fixed depth of 2 path components, causing all absolute paths to collapse into a single bucket (e.g., `/home` for every file on the system), making directory-based clustering completely ineffective. The fix adds an optional `root` parameter to `_directory_key()` and `ClusteringStrategy.cluster_by_directory()`, and updates `DecompositionService._build_hierarchy()` to compute the common root and pass it through, ensuring directory clustering groups paths by their actual directory hierarchy in production use. + - **Automation Profile Silent Fallback** (#8232): `_resolve_profile_for_plan` in `PlanLifecycleService` now raises a clear `ValidationError` when a plan's automation profile name is not a known built-in profile, instead of silently @@ -15,6 +93,37 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). message listing available built-in profiles. The resolved profile name is also logged at debug level for observability. +- **CheckpointManager rollback_to always returned False** (#7488): Fixed a data + integrity bug in `CheckpointManager.create_checkpoint()` where `sandbox_path` + was computed from `sandbox.context.sandbox_path` but never stored in the + checkpoint metadata. As a result, `rollback_to()` always found + `checkpoint.metadata.get("sandbox_path")` returning `None` and silently + skipped the rollback, returning `False`. The fix adds `sandbox_path` to the + metadata dict before constructing the `SandboxCheckpoint`, enabling + `rollback_to()` to correctly restore the sandbox filesystem state. + +- **Strategy Actor Code Review Follow-ups** (#10267): Implemented comprehensive + fixes for all 9 code review findings from PR #1175 strategy actor implementation: + - Module-level imports: Moved `json` import to module-level in + `plan_executor_coverage_steps.py` following Python conventions and ruff/isort + standards. + - Exception handling clarity: Removed redundant `json.JSONDecodeError` from + exception clause (Exception already subsumes it). + - Silent fallback logging: Added warning log when `actor.default.strategy` config + resolution fails, providing operator visibility into configuration issues. + - Structured content block handling: Enhanced `_extract_content()` in + `strategy_actor.py` to properly extract text from LangChain `MessageContentBlock` + dicts, preventing JSON parsing failures when structured message content is + encountered. + - Deduplicated constant: Removed local `_DEFAULT_ACTOR_NAME` definition from + `strategy_actor.py` and imported from canonical source in `strategy_resolution.py` + to prevent future drift. + - Test decoupling: Added `strategy_tree: StrategyTree | None` field to + `StrategizeResult` to allow tests to inspect the tree without calling private + `_execute_with_llm()` method, eliminating divergent ULID generation from + double-invocation pattern. Updated 6 step functions in `strategy_actor_llm_steps.py` + to use the public API instead of coupling to private implementation details. + ### Added - **TDD Issue-Capture Test Activation** (#7025): Replaced 234 bare `@skip` tags @@ -69,27 +178,6 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Priority-based filtering (Critical/High/Medium/Low) reduces noise. Backlog-groomer performs intelligent cleanup with age thresholds by priority. -- **OpenAI Quota Fallback to Anthropic Claude Sonnet** (#10042): Implemented graceful - degradation for E2E robot integration tests when OpenAI API hits quota limit errors - (429, insufficient_quota, rate_limit). The `StrategyActor` now detects quota-specific - errors via `_is_quota_error()` and automatically falls back to - `anthropic/claude-sonnet-4-20250514` for strategy decisions, ensuring CI/CD pipelines - complete E2E tests even when the primary provider hits capacity limits. Includes - intelligent recovery: the fallback LLM instance is cached (`_fallback_llm`), fallback - mode is tracked (`_using_fallback`), and the primary provider is re-attempted after a - 5-minute recovery interval (`_QUOTA_RECOVERY_INTERVAL = 300`). Added comprehensive - logging for quota error detection and provider fallback transitions, plus E2E test - scenarios for fallback verification. Implemented in - `StrategyActor._execute_with_llm()` in - `src/cleveragents/application/services/strategy_actor.py` (commit `f5712787`). - -- **Documentation: LLM Provider Fallback Behavior** [AUTO-DOCS-7]: Added - `docs/reference/llm_provider_fallback.md` documenting the automatic Anthropic Claude - Sonnet fallback when OpenAI quota is exhausted. Covers trigger conditions, fallback - state machine, recovery logic, logging reference, configuration constants, and - prerequisites. Updated `README.md` with an "Automatic quota fallback" subsection - under LLM provider configuration. - - **PR Agent Reorganization**: All PR-related agents renamed and reorganized to follow the `*-pool-supervisor` naming pattern. New agents added: `pr-editor` (safe PR editing with description preservation), `pr-manager` (unified PR interface), and @@ -172,6 +260,15 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). prefix check using OS path separators. Added regression test tagged `@tdd_issue_7558`. +- **File Edit Encoding Parameter** (#7559): `_handle_file_edit()` now correctly + respects the `encoding` parameter when reading and writing files. Previously the + function ignored the caller-supplied encoding and fell back to the platform default, + causing data corruption with non-UTF-8 files. The fix extracts `encoding` from the + tool inputs (defaulting to `"utf-8"`) and passes it to both `path.read_text()` and + `path.write_text()`. The `FILE_EDIT_SPEC` input schema was updated to declare the + `encoding` field. BDD scenarios were added to cover explicit encoding and the + UTF-8 default. + - **Validation Gate Empty-Run Guard** (#7508): Fixed `ApplyValidationSummary.all_required_passed` returning `True` when zero validations were run, silently bypassing the apply gate. The property now returns `False` when the validation result set is empty (`is_empty` is `True`), ensuring @@ -214,6 +311,20 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). `pr-merge-pool-supervisor` to the product-builder's supervisor launch list (18 total supervisors). Updated all numeric references, pre-flight checklists, and validation logic. +- `ActionRepository.update()` now uses explicit bulk `sa_delete()` + `session.flush()` + before re-inserting child rows for `action_arguments` and `action_invariants`, fixing + a `sqlite3.IntegrityError: UNIQUE constraint failed` crash when `agents plan use` was + called on an action that already had arguments registered via `action create`. (#4197) + +- Fixed CheckpointManager not wired into PlanExecutor — checkpoints were + never created during plan execution because `_get_plan_executor()` in + the CLI constructed PlanExecutor without a CheckpointManager (defaulted + to None, silently skipping all checkpoint hooks). `_get_plan_executor()` + now resolves the container singleton so CLI `plan execute` and `plan + rollback` share the same registry, and `_try_create_checkpoint()` raises + `PlanError` if checkpoint metadata cannot be persisted. Writable sandboxable + resources and write-capable tools now default to `checkpointable=True`, and + new Behave scenarios cover DI wiring, rollback, and capability defaults. (#1253) --- ## [3.8.0] — 2026-04-05 diff --git a/README.md b/README.md index 64386fefc..733986eda 100644 --- a/README.md +++ b/README.md @@ -170,15 +170,3 @@ Set `CLEVERAGENTS_DEFAULT_PROVIDER` to pin the global provider (for example `exp - Built-in actors (`/`) are immutable, custom actors must be named `local/`, and the default actor cannot be removed. Use `--unsafe` when adding/updating configs marked unsafe; runtime only warns when invoking unsafe actors. - `CLEVERAGENTS_TESTING_USE_MOCK_AI=true` forces the in-repo mock provider so Behave/Robot suites never hit external APIs. - The full capability matrix (streaming, tool calls, JSON mode, etc.) is documented in `docs/reference/providers.md`. - -### Automatic quota fallback - -When the primary LLM provider (e.g. OpenAI) returns a quota error (HTTP 429, -`insufficient_quota`, or `rate_limit`), `StrategyActor` automatically falls back to -**Anthropic Claude Sonnet (`claude-sonnet-4-20250514`)** for strategy decisions. -The fallback is transparent — plan operations continue without interruption. The -system re-attempts the primary provider after a 5-minute recovery interval. - -Requires `ANTHROPIC_API_KEY` to be set for the fallback to succeed. See -[`docs/reference/llm_provider_fallback.md`](docs/reference/llm_provider_fallback.md) -for full details including state machine, logging, and configuration constants. diff --git a/docs/reference/llm_provider_fallback.md b/docs/reference/llm_provider_fallback.md deleted file mode 100644 index a193a89f9..000000000 --- a/docs/reference/llm_provider_fallback.md +++ /dev/null @@ -1,193 +0,0 @@ -# LLM Provider Fallback Behavior - -## Overview - -CleverAgents implements automatic LLM provider fallback to ensure resilience when the -primary provider (typically OpenAI) exhausts its API quota. When a quota error is -detected, the system transparently switches to **Anthropic Claude Sonnet -(`claude-sonnet-4-20250514`)** as the fallback provider, allowing plan strategize -operations to continue without interruption. - -This behavior was introduced in commit `f5712787` (feat: add fallback to Anthropic -Sonnet when OpenAI quota is exhausted) and is implemented in -`StrategyActor._execute_with_llm()` located in -`src/cleveragents/application/services/strategy_actor.py`. - ---- - -## Trigger Conditions - -The fallback activates when the primary LLM call raises an exception whose string -representation matches any of the following patterns (case-insensitive): - -| Pattern | Typical Source | -|---------|---------------| -| `insufficient_quota` | OpenAI quota exhaustion | -| `429` | HTTP 429 Too Many Requests | -| `quota` | Generic quota error from any provider | -| `rate_limit` | Rate-limit errors from any provider | - -Detection is performed by the module-level helper function `_is_quota_error(exc)` in -`src/cleveragents/application/services/strategy_actor.py`. - ---- - -## Fallback Provider - -| Constant | Value | Description | -|----------|-------|-------------| -| `_FALLBACK_PROVIDER` | `"anthropic"` | Provider identifier | -| `_FALLBACK_MODEL` | `"claude-sonnet-4-20250514"` | Model identifier | -| `_QUOTA_RECOVERY_INTERVAL` | `300` (seconds) | Time before re-attempting primary provider | - -The fallback LLM instance is created lazily on first use and then **cached** as -`StrategyActor._fallback_llm` to avoid per-call re-initialization overhead. - ---- - -## Fallback State Machine - -`StrategyActor` tracks fallback state using three instance variables: - -| Variable | Type | Description | -|----------|------|-------------| -| `_fallback_llm` | `Any \| None` | Cached fallback LLM instance | -| `_last_quota_error_time` | `float \| None` | Unix timestamp of the last quota error | -| `_using_fallback` | `bool` | Whether the actor is currently in fallback mode | - -### State Transitions - -``` -Normal mode (_using_fallback=False) - | - | Primary provider raises quota error - v -Fallback mode (_using_fallback=True, _last_quota_error_time=) - | - | _QUOTA_RECOVERY_INTERVAL (300 s) elapses - v -Recovery attempt (_using_fallback=False, primary re-tried) - | - +-- Primary succeeds --> Normal mode - +-- Primary raises quota error again --> Fallback mode -``` - -### Recovery Logic - -Once in fallback mode, `_execute_with_llm()` skips the primary provider call for -subsequent invocations until `_QUOTA_RECOVERY_INTERVAL` seconds have elapsed since -the last quota error. This prevents hammering the primary provider with repeated -quota errors. After the interval, the primary provider is re-attempted automatically. - ---- - -## Behavior from the User Perspective - -From the user's perspective, the fallback is **transparent**: - -- Plan strategize operations continue to produce valid strategy trees. -- Response quality and format are identical — both OpenAI and Anthropic Claude Sonnet - support the structured JSON output required by the strategy prompt. -- No user action is required to trigger or disable the fallback. -- The fallback is **temporary**: the system automatically attempts to recover to the - primary provider after 5 minutes. - -### When Both Providers Are Unavailable - -If the fallback provider (Anthropic Claude Sonnet) also fails, the original quota -exception from the primary provider is re-raised (chained from the fallback -exception). The `StrategyActor.execute()` method catches this and falls back to the -deterministic stub strategy (`_execute_stub()`), ensuring the plan lifecycle is never -blocked by LLM unavailability. - -In E2E test contexts, when both providers are unavailable, tests fail with a clear -message indicating that the test outcome cannot be verified — rather than silently -skipping and creating false confidence in test coverage. - ---- - -## Logging - -The fallback mechanism emits structured log events at the following levels: - -| Level | Event | Key Fields | -|-------|-------|-----------| -| `WARNING` | Quota error detected on primary provider | `plan_id`, `original_error` | -| `WARNING` | Creating fallback LLM instance | `plan_id` | -| `WARNING` | Reusing cached fallback LLM instance | `plan_id` | -| `WARNING` | Fallback recovery successful | `plan_id`, `fallback_provider`, `fallback_model` | -| `ERROR` | Fallback provider also failed | `plan_id`, `fallback_provider`, `fallback_model`, `fallback_error`, `fallback_error_type` | -| `DEBUG` | Still in quota fallback mode (with time remaining) | `plan_id` | -| `INFO` | Quota recovery interval elapsed, attempting primary | `plan_id`, `time_since_error` | - -All log events include `plan_id` for correlation. Enable `DEBUG` level logging to -observe fallback mode transitions in detail. - ---- - -## Configuration - -The fallback behavior is controlled by module-level constants in -`src/cleveragents/application/services/strategy_actor.py`. These are not currently -exposed as environment variables or configuration keys. To change the fallback -provider or recovery interval, modify the constants directly: - -```python -# src/cleveragents/application/services/strategy_actor.py -_FALLBACK_PROVIDER = "anthropic" # Provider identifier -_FALLBACK_MODEL = "claude-sonnet-4-20250514" # Model identifier -_QUOTA_RECOVERY_INTERVAL = 300 # Seconds before re-trying primary -``` - -> **Note:** The fallback provider requires a valid `ANTHROPIC_API_KEY` environment -> variable to be set. If the key is absent, the fallback LLM creation will fail and -> the original quota exception will be re-raised. - ---- - -## Prerequisites - -To enable the Anthropic fallback, ensure the following environment variable is set: - -```bash -export ANTHROPIC_API_KEY= -``` - -The primary provider (e.g. OpenAI) is configured as usual: - -```bash -export OPENAI_API_KEY= -``` - -See the [LLM provider configuration](../../README.md#llm-provider-configuration) -section of the README for the full list of supported providers and environment -variables. - ---- - -## Implementation Reference - -| Symbol | Location | Description | -|--------|----------|-------------| -| `_is_quota_error(exc)` | `src/cleveragents/application/services/strategy_actor.py` | Detects quota errors by string matching | -| `StrategyActor._execute_with_llm()` | `src/cleveragents/application/services/strategy_actor.py` | Orchestrates primary call, quota detection, and fallback | -| `StrategyActor._fallback_llm` | `src/cleveragents/application/services/strategy_actor.py` | Cached fallback LLM instance | -| `StrategyActor._last_quota_error_time` | `src/cleveragents/application/services/strategy_actor.py` | Timestamp of last quota error | -| `StrategyActor._using_fallback` | `src/cleveragents/application/services/strategy_actor.py` | Fallback mode flag | -| `_FALLBACK_PROVIDER` | `src/cleveragents/application/services/strategy_actor.py` | `"anthropic"` | -| `_FALLBACK_MODEL` | `src/cleveragents/application/services/strategy_actor.py` | `"claude-sonnet-4-20250514"` | -| `_QUOTA_RECOVERY_INTERVAL` | `src/cleveragents/application/services/strategy_actor.py` | `300` seconds | - -**Introduced in:** commit `f5712787` — *feat: add fallback to Anthropic Sonnet when -OpenAI quota is exhausted* - -**Fixes:** issue #10042 - ---- - -## Related Documentation - -- [Retry Policy](retry_policy.md) — service-level retry and circuit breaker configuration -- [Error Recovery](error_recovery.md) — plan-level error classification and recovery actions -- [Error Handling](error_handling.md) — error codes, classification, and secret redaction -- [Observability](observability.md) — structured logging and tracing configuration