feat(agents): add virtual provider backends config and round-robin routing #157

Merged
CoreRasurae merged 2 commits from feature/m1-virtual-provider-round-robin into develop 2026-10-19 21:15:07 +00:00
Member

Summary

Implements issue #155 (the first increment of Epic #154, ADR-2038): an optional backends list (>= 2 entries) and strategy field (currently only round_robin) on type: llm agents, so a single logical agent can pool several backend credential sets — e.g. multiple OAuth subscriptions for the same provider — and round-robin requests across them to multiply effective capacity.

Key design points (full rationale in ADR-2038, status: approved):

  • VirtualChatModel is a genuine BaseChatModel subclass (same precedent as ReasoningChatModel, ADR-2036) — Liskov-substitutable, no changes needed anywhere else in the agent/tool-loop/graph dispatch stack.
  • Each backend is built via the existing, unmodified per-provider construction/credential-validation logic; credentials are never merged into the shared provider-keyed credentials dict.
  • Backend selection happens exactly once per external invocation; retries (ADR-2032) and the first-chunk-only streaming retry are pinned to that same selected backend — LLMAgent._retry_ainvoke/_run_pruning_pass/stream_message required no changes.
  • reasoning_feedback (ADR-2036) is read once at the virtual-provider level and applied identically to every backend, so reasoning-aware backend switching still round-trips reasoning_content correctly.
  • Also fixes a pre-existing Behave test-isolation bug found while adding these scenarios: reasoning_provider_routing_steps.py's _reasoning_unavailable context flag wasn't reset between scenarios, silently leaking into later scenarios that reuse its shared "the chat model is constructed" step.

Test plan

  • nox -s lint — green
  • nox -s typecheck — green (Pyright strict, zero suppressions)
  • nox -s unit_tests — green, full suite (155 features / 3144 scenarios)
  • nox -s coverage_report — 96.9% (threshold 96.5%)
  • nox -s security_scan — green (bandit + semgrep)
  • nox -s dead_code — green (vulture)
  • nox -s integration_tests -- robot/virtual_provider_routing.robot — green (Executor → LLMAgent → VirtualChatModel end-to-end through a real tool-call loop, >= 2 backends)
  • ASV benchmark added (benchmarks/virtual_provider_benchmark.py) for the per-invocation backend-selection hot path
  • nox -s docs — mkdocs build green with the new guide page
  • docs/index.md §4.4/§4.4.7 updated, Version bumped 1.7.0 → 1.8.0, §21.1 revision-history row added
  • examples/packages/virtual-provider-pooled-agent.yaml added
  • docs/guides/virtual-provider-backend-routing.md added (practical guide, mirroring the reasoning-aware-llm-agents guide)
  • CHANGELOG.md updated

Closes #155

## Summary Implements issue #155 (the first increment of Epic #154, ADR-2038): an optional `backends` list (>= 2 entries) and `strategy` field (currently only `round_robin`) on `type: llm` agents, so a single logical agent can pool several backend credential sets — e.g. multiple OAuth subscriptions for the same provider — and round-robin requests across them to multiply effective capacity. Key design points (full rationale in ADR-2038, status: approved): - `VirtualChatModel` is a genuine `BaseChatModel` subclass (same precedent as `ReasoningChatModel`, ADR-2036) — Liskov-substitutable, no changes needed anywhere else in the agent/tool-loop/graph dispatch stack. - Each backend is built via the existing, unmodified per-provider construction/credential-validation logic; credentials are never merged into the shared provider-keyed credentials dict. - Backend selection happens exactly once per external invocation; retries (ADR-2032) and the first-chunk-only streaming retry are pinned to that same selected backend — `LLMAgent._retry_ainvoke`/`_run_pruning_pass`/`stream_message` required **no changes**. - `reasoning_feedback` (ADR-2036) is read once at the virtual-provider level and applied identically to every backend, so reasoning-aware backend switching still round-trips `reasoning_content` correctly. - Also fixes a pre-existing Behave test-isolation bug found while adding these scenarios: `reasoning_provider_routing_steps.py`'s `_reasoning_unavailable` context flag wasn't reset between scenarios, silently leaking into later scenarios that reuse its shared "the chat model is constructed" step. ## Test plan - [x] `nox -s lint` — green - [x] `nox -s typecheck` — green (Pyright strict, zero suppressions) - [x] `nox -s unit_tests` — green, full suite (155 features / 3144 scenarios) - [x] `nox -s coverage_report` — 96.9% (threshold 96.5%) - [x] `nox -s security_scan` — green (bandit + semgrep) - [x] `nox -s dead_code` — green (vulture) - [x] `nox -s integration_tests -- robot/virtual_provider_routing.robot` — green (Executor → LLMAgent → VirtualChatModel end-to-end through a real tool-call loop, >= 2 backends) - [x] ASV benchmark added (`benchmarks/virtual_provider_benchmark.py`) for the per-invocation backend-selection hot path - [x] `nox -s docs` — mkdocs build green with the new guide page - [x] `docs/index.md` §4.4/§4.4.7 updated, Version bumped 1.7.0 → 1.8.0, §21.1 revision-history row added - [x] `examples/packages/virtual-provider-pooled-agent.yaml` added - [x] `docs/guides/virtual-provider-backend-routing.md` added (practical guide, mirroring the reasoning-aware-llm-agents guide) - [x] CHANGELOG.md updated Closes #155
CoreRasurae added this to the v2.1.0 milestone 2026-10-16 16:51:11 +00:00
feat(agents): add virtual provider backends config and round-robin routing
Some checks failed
CI / lint (pull_request) Failing after 45s
CI / typecheck (pull_request) Successful in 1m33s
CI / integration_tests (pull_request) Has started running
CI / security (pull_request) Successful in 1m56s
CI / benchmark (pull_request) Has started running
CI / quality (pull_request) Successful in 2m5s
CI / build (pull_request) Successful in 2m28s
CI / unit_tests (pull_request) Successful in 6m20s
CI / coverage (pull_request) Has been skipped
CI / status-check (pull_request) Has been cancelled
9a16cc5955
Adds an optional `backends` list (>= 2 entries) and `strategy` field
(default and currently only supported value: `round_robin`) to `type: llm`
agent configuration, so a single logical agent can pool several backend
credential sets -- e.g. multiple OAuth subscriptions for the same provider
-- and forward each invocation to exactly one of them. This solves the
"single provider/model/credential-set per agent" limitation of the shared,
provider-keyed credentials dict, which cannot hold two distinct credential
sets for the same provider name.

Design is recorded in ADR-2038. Key decisions carried into this
implementation:
- `VirtualChatModel` is a genuine `BaseChatModel` subclass (same precedent
  as `ReasoningChatModel`, ADR-2036), overriding only
  `_generate`/`_agenerate`/`_stream`/`_astream`/`_llm_type` so it is
  Liskov-substitutable for any other chat model this project constructs.
- Each backend is built via the existing, unmodified per-provider
  construction/credential-validation logic
  (`llm_client._build_from_credentials`) -- no parallel validation path.
- Backend selection happens exactly once per external invocation; retries
  (ADR-2032) and the first-chunk-only streaming retry are performed
  internally against that same selected backend, never re-selecting
  mid-retry or mid-stream -- otherwise a retry would silently bounce
  between backends, which the issue explicitly calls out as must-not-happen.
  `LLMAgent._retry_ainvoke`/`_run_pruning_pass`/`stream_message` needed no
  changes; their own outer retry wrapping becomes a harmless no-op for a
  virtual provider since `VirtualChatModel` already exhausts retries
  internally before raising.
- `reasoning_feedback` is read once at the virtual-provider level and
  applied identically to every backend, so a reasoning-aware backend
  switch still round-trips `reasoning_content` correctly.

Also fixes a pre-existing Behave test-isolation bug surfaced while adding
these scenarios: `reasoning_provider_routing_steps.py`'s
`_reasoning_unavailable` context flag wasn't reset in `before_scenario`, so
it silently leaked into any later scenario reusing the shared "the chat
model is constructed" step, breaking unrelated native-provider construction.

Module: src/cleveractors/agents/llm_virtual_provider.py (new),
src/cleveractors/agents/llm_client.py, src/cleveractors/agents/llm.py,
src/cleveractors/validation/_agents.py, src/cleveractors/validation/_vocabulary.py.
BDD: features/virtual_provider.feature. Robot: robot/virtual_provider_routing.robot.
Benchmark: benchmarks/virtual_provider_benchmark.py. Docs: docs/index.md
§4.4/§4.4.7 (Version 1.8.0, §21.1), docs/guides/virtual-provider-backend-routing.md.
Example: examples/packages/virtual-provider-pooled-agent.yaml.

ISSUES CLOSED: #155

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CoreRasurae force-pushed feature/m1-virtual-provider-round-robin from 9a16cc5955
Some checks failed
CI / lint (pull_request) Failing after 45s
CI / typecheck (pull_request) Successful in 1m33s
CI / integration_tests (pull_request) Has started running
CI / security (pull_request) Successful in 1m56s
CI / benchmark (pull_request) Has started running
CI / quality (pull_request) Successful in 2m5s
CI / build (pull_request) Successful in 2m28s
CI / unit_tests (pull_request) Successful in 6m20s
CI / coverage (pull_request) Has been skipped
CI / status-check (pull_request) Has been cancelled
to fd8f9629ee
Some checks failed
CI / lint (pull_request) Successful in 1m16s
CI / quality (pull_request) Successful in 44s
CI / typecheck (pull_request) Successful in 2m23s
CI / build (pull_request) Successful in 2m6s
CI / security (pull_request) Successful in 2m58s
CI / integration_tests (pull_request) Successful in 4m40s
CI / unit_tests (pull_request) Successful in 5m25s
CI / coverage (pull_request) Successful in 4m38s
CI / status-check (pull_request) Successful in 2s
CI / benchmark (pull_request) Failing after 30m39s
2026-10-16 17:04:56 +00:00
Compare
hurui200320 left a comment

PR Review: !157 (Ticket #155)

Verdict: Approve

No critical or major issues found. The retry-pinning design is sound (verified against call_with_retry's control flow: an inner ExecutionError from exhausted retries is re-raised immediately, never re-retried or re-routed by the outer wrapper), round-robin wraps correctly, per-backend credential isolation is real, and the Behave/Robot suites assert actual behavior. The remaining findings are minor documentation/edge-case items — none block merge, but a few are trivial to fold in before it.

Scope note: no review comments, PR comments, or ticket comments exist anywhere, so there are no author decisions to honor. The random strategy is explicitly deferred to issue #156 and was not flagged. Red-tape items (commit footer format, labels) excluded per review instructions.

Critical Issues

None

Major Issues

None

Minor Issues

  1. docs/index.md lines 320–324 — duplicated table rows in the normative standard. The §4.4 field table contains the backends and strategy rows twice (four consecutive rows instead of two, exact duplicates). Recommendation: delete the second pair.

  2. examples/packages/virtual-provider-pooled-agent.yaml line 10 — example advises an unsupported value. The comment says "set strategy: random for uniform random selection instead", but random is not supported (VALID_VIRTUAL_PROVIDER_STRATEGIES = {"round_robin"}); following the shipped example produces a ConfigurationError. It also contradicts the guide ("round_robin is currently the only supported value"). Recommendation: reword to state that random is planned as a follow-up (issue #156).

  3. src/cleveractors/agents/llm_virtual_provider.py lines 365–366 — explicit provider: null/model: null in a backend entry is inconsistent between validation and construction. The validation layer treats None as absent (skips checks), but construction uses raw_backend.get("provider", provider), which returns None when the key exists with a null value — so None.lower() raises AttributeError, wrapped as the confusing Failed to initialize backends[i] (None/gpt-4) instead of inheriting the agent's default. Recommendation: normalize None to the agent default (or reject it explicitly in validation).

  4. src/cleveractors/agents/factory.py lines 533–544 — credential-injection-mode gap for virtual agents. In ADR-2026 injection mode, AgentFactory requires credentials[provider] to exist for any llm agent and raises missing credentials for provider — yet that looked-up slice is then entirely ignored by the virtual path (build_chat_model dispatches to build_virtual_chat_model before touching it). A backends-only agent (per-backend credentials, no top-level provider entry) cannot run in injection mode without an otherwise-unused dummy entry. Pre-existing gate, but the interaction is new, untested, and undocumented. Recommendation: skip the provider-credentials requirement when backends is present, or document it.

  5. Coverage 96.9% vs the 97% figure (ticket subtask / noxfile.py docstring lines 157–160; actual gate constant is the pre-existing 96.5, so CI is green). The gap is the new sync paths VirtualChatModel._generate/_stream (lines 184–204), which no Behave scenario exercises — all steps use ainvoke/astream. Recommendation: add two cheap scenarios using the sync .invoke()/.stream() (fakes already implement both), or explicitly reconcile the 96.5 vs 97.0 mismatch in the noxfile docstring.

Nits

  1. docs/index.md line 511 (§4.4.7) — the body lists random as a strategy without a "not yet supported" marker; only the revision-history row and the guide clarify that. A one-word qualifier would keep the normative body consistent with shipped behavior.
  2. src/cleveractors/agents/llm_virtual_provider.py line 43 — first cross-layer import of the private cleveractors.validation._vocabulary from the agents layer. Consider a neutral home for the two constants (or the public cleveractors.validation surface).
  3. llm_virtual_provider.py lines 251–267 — on a transient first-chunk failure, the abandoned async generator is never aclose()d before the retry creates a new one. This exactly mirrors the pre-existing LLMAgent.stream_message pattern (llm.py lines 2134–2148), so it is consistent, not a regression.
  4. llm_virtual_provider.py lines 384–390 — the non-ConfigurationError wrap omits the DEBUG-level log of the exception type that build_chat_model's sanitize path emits (llm_client.py lines 150–162); a small diagnostics loss.
  5. llm.py lines 823–834 — _run_pruning_pass constructs a fresh VirtualChatModel per pruning call, so pruning always lands on backend 0 (fresh round-robin counter each pass). Deterministic and harmless, but an undocumented emergent behavior; pruning timeout messages also lose the provider URL (_get_provider_url finds no URL attrs on the wrapper).
  6. features/virtual_provider.feature — no scenario for explicit strategy: round_robin (only the omitted-default case is tested end-to-end).
  7. The reasoning round-trip scenario (feature lines 176–180, steps 715–757) validates ReasoningChatModel payloads directly, bypassing VirtualChatModel. The mechanism it proves (reasoning travels in message kwargs + reasoning_feedback applied identically per backend) is what makes the virtual case work, so it is a legitimate proxy — an end-to-end variant would just be stronger.
  8. Branch hygiene: the required docs deliverables (§4.4/§4.4.7, version bump) live in a second commit (fd8f962) rather than the atomic feature commit, so neither commit alone fully implements the issue. Linear history with no merge commits, otherwise clean.

Summary

A well-executed, carefully-scoped increment. VirtualChatModel is a genuine BaseChatModel subclass that composes correctly with the existing stack: backend selection happens exactly once per invocation and is pinned through both the ADR-2032 retry loop and the first-chunk-only streaming retry; the outer _retry_ainvoke wrapping degenerates to a no-op exactly as the ticket requires; per-backend construction reuses the unmodified _build_from_credentials path (including SSRF checks and error sanitization); and the temperature-sync in _select() correctly makes the existing _temperature_override flow work through the virtual layer. All ticket acceptance criteria have corresponding, meaningful Behave scenarios, the Robot test drives the real Executor → LLMAgent → VirtualChatModel chain with only the network boundary stubbed, and the accompanying test-isolation fix in features/environment.py is a genuine pre-existing bug fix. The findings above are polish: fix the two documentation defects (trivial), and consider the null-handling and injection-mode items as fast-follows or fold-ins.

## PR Review: !157 (Ticket #155) ### Verdict: Approve No critical or major issues found. The retry-pinning design is sound (verified against `call_with_retry`'s control flow: an inner `ExecutionError` from exhausted retries is re-raised immediately, never re-retried or re-routed by the outer wrapper), round-robin wraps correctly, per-backend credential isolation is real, and the Behave/Robot suites assert actual behavior. The remaining findings are minor documentation/edge-case items — none block merge, but a few are trivial to fold in before it. *Scope note: no review comments, PR comments, or ticket comments exist anywhere, so there are no author decisions to honor. The `random` strategy is explicitly deferred to issue #156 and was **not** flagged. Red-tape items (commit footer format, labels) excluded per review instructions.* ### Critical Issues None ### Major Issues None ### Minor Issues 1. **`docs/index.md` lines 320–324 — duplicated table rows in the normative standard.** The §4.4 field table contains the `backends` and `strategy` rows twice (four consecutive rows instead of two, exact duplicates). Recommendation: delete the second pair. 2. **`examples/packages/virtual-provider-pooled-agent.yaml` line 10 — example advises an unsupported value.** The comment says *"set `strategy: random` for uniform random selection instead"*, but `random` is not supported (`VALID_VIRTUAL_PROVIDER_STRATEGIES = {"round_robin"}`); following the shipped example produces a `ConfigurationError`. It also contradicts the guide ("`round_robin` is currently the only supported value"). Recommendation: reword to state that `random` is planned as a follow-up (issue #156). 3. **`src/cleveractors/agents/llm_virtual_provider.py` lines 365–366 — explicit `provider: null`/`model: null` in a backend entry is inconsistent between validation and construction.** The validation layer treats `None` as absent (skips checks), but construction uses `raw_backend.get("provider", provider)`, which returns `None` when the key exists with a null value — so `None.lower()` raises `AttributeError`, wrapped as the confusing `Failed to initialize backends[i] (None/gpt-4)` instead of inheriting the agent's default. Recommendation: normalize `None` to the agent default (or reject it explicitly in validation). 4. **`src/cleveractors/agents/factory.py` lines 533–544 — credential-injection-mode gap for virtual agents.** In ADR-2026 injection mode, `AgentFactory` requires `credentials[provider]` to exist for any `llm` agent and raises `missing credentials for provider` — yet that looked-up slice is then entirely ignored by the virtual path (`build_chat_model` dispatches to `build_virtual_chat_model` before touching it). A `backends`-only agent (per-backend credentials, no top-level provider entry) cannot run in injection mode without an otherwise-unused dummy entry. Pre-existing gate, but the interaction is new, untested, and undocumented. Recommendation: skip the provider-credentials requirement when `backends` is present, or document it. 5. **Coverage 96.9% vs the 97% figure (ticket subtask / `noxfile.py` docstring lines 157–160; actual gate constant is the pre-existing 96.5, so CI is green).** The gap is the new sync paths `VirtualChatModel._generate`/`_stream` (lines 184–204), which no Behave scenario exercises — all steps use `ainvoke`/`astream`. Recommendation: add two cheap scenarios using the sync `.invoke()`/`.stream()` (fakes already implement both), or explicitly reconcile the 96.5 vs 97.0 mismatch in the noxfile docstring. ### Nits 1. `docs/index.md` line 511 (§4.4.7) — the body lists `random` as a strategy without a "not yet supported" marker; only the revision-history row and the guide clarify that. A one-word qualifier would keep the normative body consistent with shipped behavior. 2. `src/cleveractors/agents/llm_virtual_provider.py` line 43 — first cross-layer import of the private `cleveractors.validation._vocabulary` from the agents layer. Consider a neutral home for the two constants (or the public `cleveractors.validation` surface). 3. `llm_virtual_provider.py` lines 251–267 — on a transient first-chunk failure, the abandoned async generator is never `aclose()`d before the retry creates a new one. This exactly mirrors the pre-existing `LLMAgent.stream_message` pattern (llm.py lines 2134–2148), so it is consistent, not a regression. 4. `llm_virtual_provider.py` lines 384–390 — the non-`ConfigurationError` wrap omits the DEBUG-level log of the exception type that `build_chat_model`'s sanitize path emits (llm_client.py lines 150–162); a small diagnostics loss. 5. `llm.py` lines 823–834 — `_run_pruning_pass` constructs a fresh `VirtualChatModel` per pruning call, so pruning always lands on backend 0 (fresh round-robin counter each pass). Deterministic and harmless, but an undocumented emergent behavior; pruning timeout messages also lose the provider URL (`_get_provider_url` finds no URL attrs on the wrapper). 6. `features/virtual_provider.feature` — no scenario for *explicit* `strategy: round_robin` (only the omitted-default case is tested end-to-end). 7. The reasoning round-trip scenario (feature lines 176–180, steps 715–757) validates `ReasoningChatModel` payloads directly, bypassing `VirtualChatModel`. The mechanism it proves (reasoning travels in message kwargs + `reasoning_feedback` applied identically per backend) is what makes the virtual case work, so it is a legitimate proxy — an end-to-end variant would just be stronger. 8. Branch hygiene: the required docs deliverables (§4.4/§4.4.7, version bump) live in a second commit (`fd8f962`) rather than the atomic feature commit, so neither commit alone fully implements the issue. Linear history with no merge commits, otherwise clean. ### Summary A well-executed, carefully-scoped increment. `VirtualChatModel` is a genuine `BaseChatModel` subclass that composes correctly with the existing stack: backend selection happens exactly once per invocation and is pinned through both the ADR-2032 retry loop and the first-chunk-only streaming retry; the outer `_retry_ainvoke` wrapping degenerates to a no-op exactly as the ticket requires; per-backend construction reuses the unmodified `_build_from_credentials` path (including SSRF checks and error sanitization); and the temperature-sync in `_select()` correctly makes the existing `_temperature_override` flow work through the virtual layer. All ticket acceptance criteria have corresponding, meaningful Behave scenarios, the Robot test drives the real Executor → LLMAgent → VirtualChatModel chain with only the network boundary stubbed, and the accompanying test-isolation fix in `features/environment.py` is a genuine pre-existing bug fix. The findings above are polish: fix the two documentation defects (trivial), and consider the null-handling and injection-mode items as fast-follows or fold-ins.
CoreRasurae changed target branch from master to develop 2026-10-19 20:45:00 +00:00
CoreRasurae force-pushed feature/m1-virtual-provider-round-robin from fd8f9629ee
Some checks failed
CI / lint (pull_request) Successful in 1m16s
CI / quality (pull_request) Successful in 44s
CI / typecheck (pull_request) Successful in 2m23s
CI / build (pull_request) Successful in 2m6s
CI / security (pull_request) Successful in 2m58s
CI / integration_tests (pull_request) Successful in 4m40s
CI / unit_tests (pull_request) Successful in 5m25s
CI / coverage (pull_request) Successful in 4m38s
CI / status-check (pull_request) Successful in 2s
CI / benchmark (pull_request) Failing after 30m39s
to 9631ee670a
Some checks failed
CI / lint (pull_request) Successful in 35s
CI / quality (pull_request) Successful in 1m10s
CI / typecheck (pull_request) Successful in 2m10s
CI / security (pull_request) Successful in 2m23s
CI / build (pull_request) Successful in 1m49s
CI / integration_tests (pull_request) Successful in 3m45s
CI / unit_tests (pull_request) Successful in 4m59s
CI / benchmark (pull_request) Has been cancelled
CI / coverage (pull_request) Successful in 4m36s
CI / status-check (pull_request) Successful in 1s
CI / lint (push) Successful in 1m11s
CI / build (push) Successful in 1m11s
CI / quality (push) Successful in 1m44s
CI / security (push) Successful in 2m4s
CI / integration_tests (push) Successful in 4m42s
CI / unit_tests (push) Successful in 6m43s
CI / typecheck (push) Failing after 11m15s
CI / benchmark (push) Failing after 23m33s
CI / coverage (push) Has been skipped
CI / status-check (push) Failing after 4s
2026-10-19 20:56:35 +00:00
Compare
CoreRasurae deleted branch feature/m1-virtual-provider-round-robin 2026-10-19 21:15:08 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveractors-core!157
No description provided.