Epic: Virtual Provider — Load-Balanced LLM Backend Routing #154

New issue

Open

opened 2026-10-15 18:34:10 +00:00 by CoreRasurae · 0 comments

CoreRasurae commented

2026-10-15 18:34:10 +00:00

Member

Background and context

Operators holding multiple OAuth subscriptions for the same LLM provider
currently have no way to pool them behind a single logical agent to increase
effective capacity: an agents.<name> entry of type: llm binds to exactly
one provider + model + credential set (docs/index.md §4.4). This Epic
introduces a virtual provider — a logical LLM agent provider that
hard-configures several backend credential sets and forwards every request to
exactly one of them, selected by a load-balancing strategy (round-robin or
random), returning that backend's response unmodified. Nothing else is in
scope: no failover, no retry-across-backends, no health checking, no
weighted/least-connections selection.

Design is recorded in docs/adr/ADR-2038-virtual-provider-load-balanced-backend-routing.md.

Current behavior

An LLM agent always constructs a single chat model from its own provider /
model / credentials. There is no mechanism to spread a single logical
agent's traffic across multiple backend credential sets, even when those
sets are for the very same provider.

Expected behavior

An agents.<name> entry of type: llm can declare backends (a list of at
least two entries, each with its own optional provider/model — defaulting
to the agent's own — and a REQUIRED own credentials mapping) and strategy
("round_robin", the default, or "random"). Every invocation is forwarded
to exactly one backend chosen per the configured strategy; the selected
backend's response is returned unmodified.

Acceptance criteria

A virtual provider configured with >= 2 backends of the same provider/
model but distinct credentials (multiple OAuth subscriptions — the
primary motivating use case) can be constructed and successfully used to
send and receive messages, under both the round_robin and random
strategies.
A virtual provider configured with backends that override provider/
model per-backend (mixed providers/models) also works.
Agents that do not declare backends are completely unaffected
(non-regression) — single-provider construction and behavior are
byte-for-byte unchanged.

Supporting information

ADR: docs/adr/ADR-2038-virtual-provider-load-balanced-backend-routing.md
Parent Legendary: #9 ("Align cleveractors-core to The Actor Configuration Standard v1.0.0")

Subtasks

#155 — Add backends/strategy configuration and round-robin routing
#156 — Add random backend-selection strategy

Definition of Done

This Epic is complete when:

Both child issues are closed.
The Actor Configuration Standard (docs/index.md) documents backends/
strategy for type: llm agents, with an incremented standard version and
a §21.1 revision-history entry.
nox is green on master, including the new tests introduced by the
child issues.

## Background and context Operators holding multiple OAuth subscriptions for the same LLM provider currently have no way to pool them behind a single logical agent to increase effective capacity: an `agents.<name>` entry of `type: llm` binds to exactly one `provider` + `model` + credential set (`docs/index.md` §4.4). This Epic introduces a **virtual provider** — a logical LLM agent provider that hard-configures several backend credential sets and forwards every request to exactly one of them, selected by a load-balancing strategy (round-robin or random), returning that backend's response unmodified. Nothing else is in scope: no failover, no retry-across-backends, no health checking, no weighted/least-connections selection. Design is recorded in `docs/adr/ADR-2038-virtual-provider-load-balanced-backend-routing.md`. ## Current behavior An LLM agent always constructs a single chat model from its own `provider` / `model` / credentials. There is no mechanism to spread a single logical agent's traffic across multiple backend credential sets, even when those sets are for the very same provider. ## Expected behavior An `agents.<name>` entry of `type: llm` can declare `backends` (a list of at least two entries, each with its own optional `provider`/`model` — defaulting to the agent's own — and a REQUIRED own `credentials` mapping) and `strategy` (`"round_robin"`, the default, or `"random"`). Every invocation is forwarded to exactly one backend chosen per the configured strategy; the selected backend's response is returned unmodified. ## Acceptance criteria - A virtual provider configured with >= 2 backends of the same `provider`/ `model` but distinct `credentials` (multiple OAuth subscriptions — the primary motivating use case) can be constructed and successfully used to send and receive messages, under both the `round_robin` and `random` strategies. - A virtual provider configured with backends that override `provider`/ `model` per-backend (mixed providers/models) also works. - Agents that do not declare `backends` are completely unaffected (non-regression) — single-provider construction and behavior are byte-for-byte unchanged. ## Supporting information - ADR: `docs/adr/ADR-2038-virtual-provider-load-balanced-backend-routing.md` - Parent Legendary: #9 ("Align cleveractors-core to The Actor Configuration Standard v1.0.0") ## Subtasks - [ ] #155 — Add backends/strategy configuration and round-robin routing - [ ] #156 — Add random backend-selection strategy ## Definition of Done This Epic is complete when: - Both child issues are closed. - The Actor Configuration Standard (`docs/index.md`) documents `backends`/ `strategy` for `type: llm` agents, with an incremented standard version and a §21.1 revision-history entry. - `nox` is green on `master`, including the new tests introduced by the child issues.

Rows
Columns