Add backends/strategy configuration and round-robin routing for virtual provider LLM agents #155

New issue

Closed

opened 2026-10-15 18:36:57 +00:00 by CoreRasurae · 0 comments

CoreRasurae commented

2026-10-15 18:36:57 +00:00

Member

Copy link

Metadata

Commit Message: feat(agents): add virtual provider backends config and round-robin routing

Branch: feature/m1-virtual-provider-round-robin

Background and context

Part of the virtual-provider Epic #154. Design is recorded in
docs/adr/ADR-2038-virtual-provider-load-balanced-backend-routing.md. This
issue delivers the full first increment: a type: llm agent can declare
backends (a hard-configured list of backend credential sets, each with its
own optional provider/model, defaulting to the agent's own) and the
default round_robin selection strategy, so that a single logical agent can
forward requests across multiple backend credential sets — e.g. multiple
OAuth subscriptions for the same provider — to increase effective capacity.

Current behavior

An agents.<name> entry of type: llm always constructs exactly one chat
model from its own provider/model/credentials
(cleveractors.agents.llm_client.build_chat_model). There is no field or
code path to configure more than one backend credential set for a single
logical agent, and the shared, provider-keyed credentials dict cannot hold
two distinct credential sets for the same provider name.

Expected behavior

agents.<name>.config for type: llm accepts an optional backends list
(minimum 2 entries) and an optional strategy field (default
"round_robin"; currently the only accepted value).
Each backends[i] entry accepts optional provider/model (defaulting to
the agent's own provider/model) and a REQUIRED credentials mapping,
validated independently per backend using the existing per-provider
validation logic (native-provider env-var resolution; _validate_base_url
SSRF checks for non-native base_url).
When backends is present, invoking the agent forwards the request to
exactly one backend — selected by round-robin order — and returns that
backend's response unmodified. No retry, failover, or health checking is
performed (see ADR-2038 Non-goals).
When backends is absent, agent construction and behavior are unchanged
from today.

Acceptance criteria

A backends list with >= 2 entries constructs successfully; invoking the
resulting agent N times (N = number of backends) selects each configured
backend exactly once, in list order, and the sequence wraps correctly
after the last backend.
A backends list with 0 or 1 entries raises a ConfigurationError
identifying the minimum-count requirement.
A backend entry missing credentials raises a ConfigurationError
identifying which backend index is missing credentials.
A backend entry that omits provider/model uses the agent's own
provider/model.
Two backend entries sharing the same provider but different
credentials are both constructed correctly and used independently (never
merged into, or looked up from, the shared provider-keyed credentials
dict).
strategy absent, or explicitly set to "round_robin", selects
round-robin order as described above.
strategy set to any other value raises a ConfigurationError naming the
unsupported value (until a follow-up issue adds "random").
A transient error (per ADR-2032 D-8) from the selected backend's
underlying chat-model call is retried against that same backend, using
the agent's existing max_retries/max_retry_time configuration
(ADR-2032) — exactly as retries already worked before the virtual provider
existed. The backend is never re-selected mid-retry-sequence. Once that
backend's retry budget is exhausted, the resulting ExecutionError
propagates to the caller unmodified; it is not retried against a different
backend, and the outer, unmodified LLMAgent._retry_ainvoke wrapper does
not re-trigger a further retry (ExecutionError is not a transient error).
A non-transient error (e.g. BadRequestError, AuthenticationError) from
the selected backend propagates immediately, unmodified, with no retry —
identical to today's single-provider behavior.
With memory_enabled: true, a multi-turn conversation that is round-robined
across different backends on successive turns still receives the complete
conversation history (system prompt + prior turns + new message) on every
turn, regardless of which backend is selected — no context is lost when the
selected backend changes between turns (see ADR-2038, "Conversation state
is unaffected by backend switching").
With reasoning_feedback: true and two reasoning-aware (non-native)
backends, a multi-turn tool-call loop that round-robins between them
still round-trips reasoning_content correctly on the turn that follows
a backend switch (see ADR-2038, "Compatibility with reasoning-aware
backends (ADR-2036)"). reasoning_feedback is read once, at the virtual
provider level, and applied identically to every backend — there is no
per-backend override.
Existing agents.<name> configurations that omit backends behave
byte-for-byte as before this change (non-regression).

Supporting information

ADR: docs/adr/ADR-2038-virtual-provider-load-balanced-backend-routing.md
Epic: #154
Parent Legendary: #9

Subtasks

Add backends/strategy fields to the LLM agent configuration schema
and validation (src/cleveractors/validation/_agents.py,
_vocabulary.py)
Implement per-backend chat-model construction, reusing the existing
native/non-native provider construction logic unchanged, with isolated
per-backend credentials
Implement a SelectionStrategy interface and RoundRobinStrategy
implementation
Implement VirtualChatModel as a genuine BaseChatModel subclass
(same precedent as ReasoningChatModel, ADR-2036) overriding only
_generate/_agenerate/_stream/_astream/_llm_type, that
selects a backend once per call and forwards unmodified, with a
concurrency-safe round-robin counter
Inside the wrapper, scope retries to the one backend selected for that
invocation by calling the existing, unmodified
cleveractors.agents.retry.call_with_retry internally (same
max_retries/max_retry_time the agent already reads from config) —
do not rely on LLMAgent._retry_ainvoke's outer wrapping to retry,
since that would re-select a backend on every retry attempt; no
changes to LLMAgent._retry_ainvoke, _run_pruning_pass, or
_start_stream are needed or permitted
Wire backends/strategy into
cleveractors.agents.llm_client.build_chat_model
Update docs/index.md §4.4 with the new fields and a new §4.4.7
describing virtual-provider routing semantics; bump the Actor
Configuration Standard version and add a §21.1 revision-history entry
Add an example actor configuration under examples/ demonstrating a
pooled, same-provider virtual agent
Tests (Behave): schema validation (minimum backend count, required
per-backend credentials, provider/model defaulting), round-robin
ordering across repeated invocations, non-regression for agents
without backends, a memory_enabled: true scenario proving full
conversation history is still sent on every turn when successive turns
are round-robined across different backends, and a retry-pinning
scenario proving a transient failure on the selected backend is
retried against that same backend (not a different one) and that an
ExecutionError from exhausted retries is not retried a second time
by the outer agent-level wrapper, and a reasoning_feedback: true
scenario proving reasoning_content round-trips correctly across a
backend switch between two reasoning-aware backends
Tests (Robot): integration test exercising a virtual provider with
>= 2 backends through the library API
Verify coverage >= 97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line matches the Commit Message
in Metadata exactly.
The commit is pushed to the branch matching the Branch in Metadata
exactly.
The commit is submitted as a PR to master, reviewed, and merged.

## Metadata **Commit Message:** feat(agents): add virtual provider backends config and round-robin routing **Branch:** feature/m1-virtual-provider-round-robin ## Background and context Part of the virtual-provider Epic #154. Design is recorded in `docs/adr/ADR-2038-virtual-provider-load-balanced-backend-routing.md`. This issue delivers the full first increment: a `type: llm` agent can declare `backends` (a hard-configured list of backend credential sets, each with its own optional `provider`/`model`, defaulting to the agent's own) and the default `round_robin` selection strategy, so that a single logical agent can forward requests across multiple backend credential sets — e.g. multiple OAuth subscriptions for the same provider — to increase effective capacity. ## Current behavior An `agents.<name>` entry of `type: llm` always constructs exactly one chat model from its own `provider`/`model`/credentials (`cleveractors.agents.llm_client.build_chat_model`). There is no field or code path to configure more than one backend credential set for a single logical agent, and the shared, provider-keyed `credentials` dict cannot hold two distinct credential sets for the same provider name. ## Expected behavior - `agents.<name>.config` for `type: llm` accepts an optional `backends` list (minimum 2 entries) and an optional `strategy` field (default `"round_robin"`; currently the only accepted value). - Each `backends[i]` entry accepts optional `provider`/`model` (defaulting to the agent's own `provider`/`model`) and a REQUIRED `credentials` mapping, validated independently per backend using the existing per-provider validation logic (native-provider env-var resolution; `_validate_base_url` SSRF checks for non-native `base_url`). - When `backends` is present, invoking the agent forwards the request to exactly one backend — selected by round-robin order — and returns that backend's response unmodified. No retry, failover, or health checking is performed (see ADR-2038 Non-goals). - When `backends` is absent, agent construction and behavior are unchanged from today. ## Acceptance criteria - A `backends` list with >= 2 entries constructs successfully; invoking the resulting agent N times (N = number of backends) selects each configured backend exactly once, in list order, and the sequence wraps correctly after the last backend. - A `backends` list with 0 or 1 entries raises a `ConfigurationError` identifying the minimum-count requirement. - A backend entry missing `credentials` raises a `ConfigurationError` identifying which backend index is missing credentials. - A backend entry that omits `provider`/`model` uses the agent's own `provider`/`model`. - Two backend entries sharing the same `provider` but different `credentials` are both constructed correctly and used independently (never merged into, or looked up from, the shared provider-keyed `credentials` dict). - `strategy` absent, or explicitly set to `"round_robin"`, selects round-robin order as described above. - `strategy` set to any other value raises a `ConfigurationError` naming the unsupported value (until a follow-up issue adds `"random"`). - A transient error (per ADR-2032 D-8) from the selected backend's underlying chat-model call is retried against **that same backend**, using the agent's existing `max_retries`/`max_retry_time` configuration (ADR-2032) — exactly as retries already worked before the virtual provider existed. The backend is never re-selected mid-retry-sequence. Once that backend's retry budget is exhausted, the resulting `ExecutionError` propagates to the caller unmodified; it is not retried against a different backend, and the outer, unmodified `LLMAgent._retry_ainvoke` wrapper does not re-trigger a further retry (`ExecutionError` is not a transient error). - A non-transient error (e.g. `BadRequestError`, `AuthenticationError`) from the selected backend propagates immediately, unmodified, with no retry — identical to today's single-provider behavior. - With `memory_enabled: true`, a multi-turn conversation that is round-robined across different backends on successive turns still receives the complete conversation history (system prompt + prior turns + new message) on every turn, regardless of which backend is selected — no context is lost when the selected backend changes between turns (see ADR-2038, "Conversation state is unaffected by backend switching"). - With `reasoning_feedback: true` and two reasoning-aware (non-native) backends, a multi-turn tool-call loop that round-robins between them still round-trips `reasoning_content` correctly on the turn that follows a backend switch (see ADR-2038, "Compatibility with reasoning-aware backends (ADR-2036)"). `reasoning_feedback` is read once, at the virtual provider level, and applied identically to every backend — there is no per-backend override. - Existing `agents.<name>` configurations that omit `backends` behave byte-for-byte as before this change (non-regression). ## Supporting information - ADR: `docs/adr/ADR-2038-virtual-provider-load-balanced-backend-routing.md` - Epic: #154 - Parent Legendary: #9 ## Subtasks - [ ] Add `backends`/`strategy` fields to the LLM agent configuration schema and validation (`src/cleveractors/validation/_agents.py`, `_vocabulary.py`) - [ ] Implement per-backend chat-model construction, reusing the existing native/non-native provider construction logic unchanged, with isolated per-backend credentials - [ ] Implement a `SelectionStrategy` interface and `RoundRobinStrategy` implementation - [ ] Implement `VirtualChatModel` as a genuine `BaseChatModel` subclass (same precedent as `ReasoningChatModel`, ADR-2036) overriding only `_generate`/`_agenerate`/`_stream`/`_astream`/`_llm_type`, that selects a backend once per call and forwards unmodified, with a concurrency-safe round-robin counter - [ ] Inside the wrapper, scope retries to the one backend selected for that invocation by calling the existing, unmodified `cleveractors.agents.retry.call_with_retry` internally (same `max_retries`/`max_retry_time` the agent already reads from config) — do not rely on `LLMAgent._retry_ainvoke`'s outer wrapping to retry, since that would re-select a backend on every retry attempt; no changes to `LLMAgent._retry_ainvoke`, `_run_pruning_pass`, or `_start_stream` are needed or permitted - [ ] Wire `backends`/`strategy` into `cleveractors.agents.llm_client.build_chat_model` - [ ] Update `docs/index.md` §4.4 with the new fields and a new §4.4.7 describing virtual-provider routing semantics; bump the Actor Configuration Standard version and add a §21.1 revision-history entry - [ ] Add an example actor configuration under `examples/` demonstrating a pooled, same-provider virtual agent - [ ] Tests (Behave): schema validation (minimum backend count, required per-backend credentials, provider/model defaulting), round-robin ordering across repeated invocations, non-regression for agents without `backends`, a `memory_enabled: true` scenario proving full conversation history is still sent on every turn when successive turns are round-robined across different backends, and a retry-pinning scenario proving a transient failure on the selected backend is retried against that same backend (not a different one) and that an `ExecutionError` from exhausted retries is not retried a second time by the outer agent-level wrapper, and a `reasoning_feedback: true` scenario proving `reasoning_content` round-trips correctly across a backend switch between two reasoning-aware backends - [ ] Tests (Robot): integration test exercising a virtual provider with >= 2 backends through the library API - [ ] Verify coverage >= 97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the first line matches the Commit Message in Metadata exactly. - The commit is pushed to the branch matching the Branch in Metadata exactly. - The commit is submitted as a PR to master, reviewed, and merged.

CoreRasurae added the

labels

2026-10-15 18:36:57 +00:00

CoreRasurae referenced this issue

2026-10-15 18:37:07 +00:00

Add random backend-selection strategy for virtual provider LLM agents #156