Executor never calls AgentFactory.validate_configuration(), so malformed agents.<name> entries fail silently #122

New issue

Open

opened 2026-08-08 21:16:37 +00:00 by CoreRasurae · 0 comments

CoreRasurae commented

2026-08-08 21:16:37 +00:00

Member

Metadata

Commit Message: fix(runtime): validate agent configuration in Executor before agent creation
Branch: fix/executor-validate-configuration

Background and context

AgentFactory.validate_configuration() (src/cleveractors/agents/factory.py line ~503)
already exists and does exactly what its name suggests: it walks config["agents"] and
raises a clear ConfigurationError when an agent entry is not a mapping, or is a mapping
missing the required "type" key. It is called from exactly one place:
core/application.py line 266, inside ReactiveCleverAgentsApp.load_configuration().

It is never called from Executor.__init__/create_executor (src/cleveractors/runtime.py
lines 77-131) — the documented (docs/guides/reasoning-aware-llm-agents.md) direct entry
point for constructing and running an executor. AgentFactory.create_agent /
_create_agent_instance (factory.py lines 176-300) have no equivalent guard of their
own: an agent entry with no "type" key silently defaults to type: llm, and one with
no "config" key silently defaults to config: {} — i.e. a bare, empty LLM agent with
every field (provider, model, system_prompt, tools, ...) at its hardcoded default.

Current behavior

Related to, but distinct from, #121 (which covers agent_template/package-reference
resolution not being implemented at all in this code path). Given a malformed agent
entry — for instance a typo'd key that isn't "type" — passed directly to
create_executor (per docs/guides/reasoning-aware-llm-agents.md's documented usage):

from cleveractors.runtime import create_executor

executor = create_executor(
    config_dict={
        "agents": {
            "worker": {"agents_template": "..."},  # typo: not a recognized key
        },
        "routes": {"main": {"type": "graph", "nodes": {...}, "edges": [...]}},
    },
    credentials={"openai_compatible": {"api_key": "...", "base_url": "..."}},
)
await executor.execute("hello")

produces:

ConfigurationError: missing credentials for provider: openai

This is confusing and misleading: nothing about the actual problem — that
agents.worker has no "type" key — is ever surfaced. Tracing it requires stepping
through AgentFactory.create_agent/_create_agent_instance/_instantiate by hand:

agent_config.get("type", "llm") → defaults to "llm" (no "type" key present).
agent_config.get("config", {}) → defaults to {} (no "config" key present).
The resulting LLMAgent's self.provider = config.get("provider", "openai")
(agents/llm.py line 306) → defaults to "openai".
AgentFactory._instantiate looks up self.credentials["openai"]; if the caller's
credentials dict only has an entry for the intended provider (e.g.
"openai_compatible"), this raises ConfigurationError("missing credentials for provider: openai").

The same silent-default behavior would equally mask a correctly-spelled but still
unresolved agent_template:/template: key (see #121), or any other malformed
agents.<name> entry — the credentials mismatch is only one of many confusing symptoms
this class of bug can produce.

Expected behavior

Executor.__init__ (or the earliest practical point in create_executor) should
validate the agent configuration — by calling AgentFactory.validate_configuration() (or
equivalent) — before any agent is created, so that a malformed agents.<name> entry fails
immediately with a clear, actionable ConfigurationError (e.g. "Agent 'worker' must specify a type"), instead of silently defaulting to an empty type: llm agent and
surfacing an unrelated error several layers downstream.

Acceptance criteria

Executor.__init__ (or the earliest practical point in create_executor) invokes
agent-configuration validation equivalent to AgentFactory.validate_configuration().
An agents.<name> entry that is not a mapping, or is missing "type", raises a
ConfigurationError naming the offending agent, before any LLM/tool/network call is
attempted.
ReactiveCleverAgentsApp's existing behavior is unchanged.
Reproduces and resolves the exact scenario in "Current behavior" above: constructing
an Executor with a malformed agents.<name> entry (e.g. a typo'd key) fails with a
validation error, not a downstream credentials error.
Tests (Behave): scenario asserting Executor raises ConfigurationError for a
malformed agents.<name> entry (missing type, non-mapping value) before any
credential/provider check runs.

Supporting information

Related but distinct from #121 (agent-type/registry-package reference resolution is not
implemented at all in this same code path) — this issue is about the silent-default
failure mode, independent of whether reference resolution ever gets implemented.
src/cleveractors/agents/factory.py lines 176-300 (create_agent/_create_agent_instance),
line 503 (validate_configuration, currently dead code from this path's perspective).
src/cleveractors/core/application.py line 266 (the only existing caller of
validate_configuration()).
src/cleveractors/runtime.py lines 77-131 (Executor.__init__ — no validation call).
src/cleveractors/agents/llm.py line 306 (LLMAgent provider default).

Subtasks

Call AgentFactory.validate_configuration() (or extract/reuse its logic) from
Executor.__init__, after the AgentFactory is constructed and before any agent is
created.
Confirm/adjust validate_configuration()'s error messages to be clear at this call
site (e.g. include the actor/graph name if available).
Tests (Behave): add scenarios for missing type, non-mapping agent entry, and a
typo'd-key case.
Verify coverage >=97% via nox -s coverage_report.
Run nox (all default sessions), fix any errors.

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
A Git commit is created where the first line of the commit message matches the
Commit Message in Metadata exactly, followed by a blank line, then additional lines
providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata
exactly.
The commit is submitted as a pull request to master, reviewed, and merged
before this issue is marked done.

## Metadata - **Commit Message**: `fix(runtime): validate agent configuration in Executor before agent creation` - **Branch**: `fix/executor-validate-configuration` ## Background and context `AgentFactory.validate_configuration()` (`src/cleveractors/agents/factory.py` line ~503) already exists and does exactly what its name suggests: it walks `config["agents"]` and raises a clear `ConfigurationError` when an agent entry is not a mapping, or is a mapping missing the required `"type"` key. It is called from exactly one place: `core/application.py` line 266, inside `ReactiveCleverAgentsApp.load_configuration()`. It is **never called** from `Executor.__init__`/`create_executor` (`src/cleveractors/runtime.py` lines 77-131) — the documented (`docs/guides/reasoning-aware-llm-agents.md`) direct entry point for constructing and running an executor. `AgentFactory.create_agent` / `_create_agent_instance` (`factory.py` lines 176-300) have no equivalent guard of their own: an agent entry with no `"type"` key silently defaults to `type: llm`, and one with no `"config"` key silently defaults to `config: {}` — i.e. a bare, empty LLM agent with every field (provider, model, system_prompt, tools, ...) at its hardcoded default. ## Current behavior Related to, but distinct from, #121 (which covers `agent_template`/package-reference resolution not being implemented at all in this code path). Given a malformed agent entry — for instance a typo'd key that isn't `"type"` — passed directly to `create_executor` (per `docs/guides/reasoning-aware-llm-agents.md`'s documented usage): ```python from cleveractors.runtime import create_executor executor = create_executor( config_dict={ "agents": { "worker": {"agents_template": "..."}, # typo: not a recognized key }, "routes": {"main": {"type": "graph", "nodes": {...}, "edges": [...]}}, }, credentials={"openai_compatible": {"api_key": "...", "base_url": "..."}}, ) await executor.execute("hello") ``` produces: ``` ConfigurationError: missing credentials for provider: openai ``` This is confusing and misleading: nothing about the actual problem — that `agents.worker` has no `"type"` key — is ever surfaced. Tracing it requires stepping through `AgentFactory.create_agent`/`_create_agent_instance`/`_instantiate` by hand: 1. `agent_config.get("type", "llm")` → defaults to `"llm"` (no `"type"` key present). 2. `agent_config.get("config", {})` → defaults to `{}` (no `"config"` key present). 3. The resulting `LLMAgent`'s `self.provider = config.get("provider", "openai")` (`agents/llm.py` line 306) → defaults to `"openai"`. 4. `AgentFactory._instantiate` looks up `self.credentials["openai"]`; if the caller's `credentials` dict only has an entry for the *intended* provider (e.g. `"openai_compatible"`), this raises `ConfigurationError("missing credentials for provider: openai")`. The same silent-default behavior would equally mask a correctly-spelled but still unresolved `agent_template:`/`template:` key (see #121), or any other malformed `agents.<name>` entry — the credentials mismatch is only one of many confusing symptoms this class of bug can produce. ## Expected behavior `Executor.__init__` (or the earliest practical point in `create_executor`) should validate the agent configuration — by calling `AgentFactory.validate_configuration()` (or equivalent) — before any agent is created, so that a malformed `agents.<name>` entry fails immediately with a clear, actionable `ConfigurationError` (e.g. `"Agent 'worker' must specify a type"`), instead of silently defaulting to an empty `type: llm` agent and surfacing an unrelated error several layers downstream. ## Acceptance criteria - [ ] `Executor.__init__` (or the earliest practical point in `create_executor`) invokes agent-configuration validation equivalent to `AgentFactory.validate_configuration()`. - [ ] An `agents.<name>` entry that is not a mapping, or is missing `"type"`, raises a `ConfigurationError` naming the offending agent, before any LLM/tool/network call is attempted. - [ ] `ReactiveCleverAgentsApp`'s existing behavior is unchanged. - [ ] Reproduces and resolves the exact scenario in "Current behavior" above: constructing an `Executor` with a malformed `agents.<name>` entry (e.g. a typo'd key) fails with a validation error, not a downstream credentials error. - [ ] Tests (Behave): scenario asserting `Executor` raises `ConfigurationError` for a malformed `agents.<name>` entry (missing `type`, non-mapping value) before any credential/provider check runs. ## Supporting information - Related but distinct from #121 (agent-type/registry-package reference resolution is not implemented at all in this same code path) — this issue is about the silent-default *failure mode*, independent of whether reference resolution ever gets implemented. - `src/cleveractors/agents/factory.py` lines 176-300 (`create_agent`/`_create_agent_instance`), line 503 (`validate_configuration`, currently dead code from this path's perspective). - `src/cleveractors/core/application.py` line 266 (the only existing caller of `validate_configuration()`). - `src/cleveractors/runtime.py` lines 77-131 (`Executor.__init__` — no validation call). - `src/cleveractors/agents/llm.py` line 306 (`LLMAgent` provider default). ## Subtasks - [ ] Call `AgentFactory.validate_configuration()` (or extract/reuse its logic) from `Executor.__init__`, after the `AgentFactory` is constructed and before any agent is created. - [ ] Confirm/adjust `validate_configuration()`'s error messages to be clear at this call site (e.g. include the actor/graph name if available). - [ ] Tests (Behave): add scenarios for missing `type`, non-mapping agent entry, and a typo'd-key case. - [ ] Verify coverage >=97% via `nox -s coverage_report`. - [ ] Run `nox` (all default sessions), fix any errors. ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done.

CoreRasurae added the

labels

2026-08-08 21:16:37 +00:00

CoreRasurae referenced this issue

2026-08-08 21:55:47 +00:00

ReactiveCleverAgentsApp has no credentials param, so non-native providers (openai_compatible) can never work through it #123