f281fa3b24
CI / lint (pull_request) Successful in 35s
CI / typecheck (pull_request) Successful in 50s
CI / security (pull_request) Successful in 50s
CI / quality (pull_request) Successful in 32s
CI / unit_tests (pull_request) Successful in 3m32s
CI / integration_tests (pull_request) Successful in 59s
CI / build (pull_request) Successful in 34s
CI / lint (push) Successful in 33s
CI / build (push) Successful in 37s
CI / quality (push) Successful in 48s
CI / typecheck (push) Successful in 48s
CI / security (push) Successful in 1m4s
CI / integration_tests (push) Successful in 1m18s
CI / coverage (pull_request) Successful in 3m34s
CI / status-check (pull_request) Successful in 3s
CI / unit_tests (push) Successful in 3m45s
CI / coverage (push) Successful in 3m34s
CI / status-check (push) Successful in 3s
Implements Wave 3 of the cleveractors-core integration (ADR-2026, ADR-2028). ## AgentFactory - Added optional credentials: dict[str, dict[str, str]] | None parameter to __init__. Stored as self.credentials. - When creating LLMAgent instances, the full credentials dict is forwarded via a new credentials constructor kwarg, threading per-request API keys without touching the stored actor config_dict. - Added explicit type annotation ReactiveAgentFactory: type[AgentFactory]. - _create_agent_instance raises ConfigurationError when credentials is not None but does not contain an entry for the requested provider (AC4 compliance). ## LLMAgent - Added optional credentials: dict[str, str] | None parameter to __init__. Stored as self._credentials; config_dict is never modified. - LangChain client construction deferred to first access (lazy init) via a chat_model property backed by _ensure_chat_model(). The property setter allows test-injection of mock models without going through the lazy-init path. - _create_chat_model() dispatches to credential-injection / standalone paths. - Extracted to llm_client.py and llm_providers.py for modularity. - Defined shared module-level constants DEFAULT_MODEL, DEFAULT_TEMPERATURE, DEFAULT_MAX_TOKENS to eliminate duplicated magic numbers across llm.py, runtime.py, and factory.py. - Added public credentials property exposing self._credentials. - Added type narrowing guard in chat_model property to satisfy static analysis. - Added temperature_override type validation in process_message. - Guarded cleanup() null-assignment with _chat_model_lock (TOCTOU fix). ## SSRF Prevention (ADR-2028) - _validate_base_url(): https-only, userinfo rejection, localhost rejection, raw IP literal rejection, numeric-hostname (dotted-hex/octal/compact) rejection, percent-decode loop with iteration cap (10), trailing-dot FQDN strip. - DNS resolution intentionally omitted (blocking call in async path). - Canonical URL reconstruction uses lowercased scheme. ## Credential Leak Prevention - All logger.exception(...) replaced with logger.debug(..., exc_info=True). - Error messages use type(e).__name__ instead of str(e). - process_message exception chains broken with from None to prevent LangChain authentication errors from leaking via __cause__. ## Runtime fixes - Executor._execute_llm: shallow copy for setdefault mutations. - Executor._execute_graph: restructured try/finally for resource cleanup. - Uses DEFAULT_MODEL, DEFAULT_TEMPERATURE, DEFAULT_MAX_TOKENS from llm.py. - Defensive copy of credentials dict in Executor.__init__ to prevent caller mutation after construction. ## Test improvements - Added BDD scenarios for llm_imports.py ImportError fallback branches using sys.modules manipulation with a custom import hook, exercised via subprocess. - Added BDD scenario for AgentFactory.create_agents_from_config() credential threading to all created agents. - Removed tautological dataclass test scenarios (NodeUsage, ActorResult) from runtime_coverage.feature. - Fixed misleading scenario title in runtime_coverage.feature (exception chain suppressed, not chained) and updated step to assert __cause__ is None. - Added CHANGELOG entry for ReactiveAgentFactory backward-compatibility alias. - Removed duplicate log emission in _check_known_provider_domain. ISSUES CLOSED: #12
455 lines
17 KiB
Python
455 lines
17 KiB
Python
"""Step definitions for per-request credential injection (core factory + lazy init).
|
|
|
|
Covers AgentFactory credentials parameter, LLMAgent lazy init, agent cache
|
|
behaviour, factory-to-agent credential threading, process_message end-to-end,
|
|
and cleanup.
|
|
|
|
Provider routing scenarios live in credential_native_provider_steps.py and
|
|
credential_ssrf_steps.py. Temperature override and __repr__ redaction
|
|
scenarios live in credential_agent_state_steps.py.
|
|
Executor-level tests live in credential_executor_steps.py.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Any, cast
|
|
from unittest.mock import MagicMock, patch
|
|
|
|
from behave import given, step, then, when
|
|
from behave.api.async_step import async_run_until_complete
|
|
from features.mocks.credential_helpers import (
|
|
make_template_renderer,
|
|
minimal_actor_config,
|
|
mock_chat_model,
|
|
)
|
|
|
|
from cleveractors.agents.factory import AgentFactory
|
|
from cleveractors.agents.llm import LLMAgent
|
|
from cleveractors.core.exceptions import (
|
|
AgentCreationError,
|
|
ConfigurationError,
|
|
ExecutionError,
|
|
)
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Shared Given step — used by factory, composite, and validation scenarios
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given("a minimal actor config with a single openai LLM agent")
|
|
def step_minimal_actor_config(context: Any) -> None:
|
|
"""Set up a minimal actor config."""
|
|
context.actor_config = minimal_actor_config()
|
|
context.template_renderer = make_template_renderer()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# LLMAgent lazy init scenarios
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given(
|
|
"I construct an LLMAgent for openai with api_key in config and no credentials dict"
|
|
)
|
|
def step_llm_agent_openai_standalone(context: Any) -> None:
|
|
"""Construct an LLMAgent in standalone mode (no credentials dict)."""
|
|
context.agent_config = {
|
|
"provider": "openai",
|
|
"api_key": "standalone-api-key",
|
|
}
|
|
context.renderer = make_template_renderer()
|
|
context.agent = LLMAgent(
|
|
name="test_agent",
|
|
config=context.agent_config,
|
|
template_renderer=context.renderer,
|
|
)
|
|
|
|
|
|
@then("the internal _chat_model attribute should be None")
|
|
def step_chat_model_none(context: Any) -> None:
|
|
"""Verify _chat_model is None (lazy init not yet triggered)."""
|
|
assert context.agent._chat_model is None, (
|
|
f"Expected _chat_model to be None immediately after construction, "
|
|
f"got {context.agent._chat_model!r}"
|
|
)
|
|
|
|
|
|
@when("I access the chat_model property")
|
|
def step_access_chat_model_property(context: Any) -> None:
|
|
"""Access the chat_model property (triggers lazy init).
|
|
|
|
For the google credential-injection test, we patch
|
|
ChatGoogleGenerativeAI at access time. For non-native provider
|
|
routing tests using test domains, we bypass _validate_base_url
|
|
DNS resolution since those tests are about routing, not SSRF
|
|
prevention (which has its own dedicated scenarios).
|
|
"""
|
|
patches: list[Any] = []
|
|
|
|
if getattr(context, "_need_google_patch", False):
|
|
mock_google_cls = MagicMock()
|
|
mock_google_cls.return_value = MagicMock()
|
|
patches.append(
|
|
patch("cleveractors.agents.llm.ChatGoogleGenerativeAI", mock_google_cls)
|
|
)
|
|
|
|
if getattr(context, "_skip_base_url_validation", False):
|
|
from urllib.parse import urlparse
|
|
|
|
from cleveractors.agents import llm_client as lc_module
|
|
|
|
patches.append(
|
|
patch.object(
|
|
lc_module,
|
|
"_validate_base_url",
|
|
side_effect=lambda url: (url, urlparse(url).hostname),
|
|
)
|
|
)
|
|
|
|
# Activate all patches, access property, then clean up
|
|
for p in patches:
|
|
p.start()
|
|
try:
|
|
context.accessed_chat_model = context.agent.chat_model
|
|
finally:
|
|
for p in reversed(patches):
|
|
p.stop()
|
|
|
|
|
|
@then("the internal _chat_model attribute should not be None")
|
|
def step_chat_model_not_none(context: Any) -> None:
|
|
"""Verify _chat_model is set after property access."""
|
|
assert context.agent._chat_model is not None, (
|
|
"Expected _chat_model to be set after property access, but it is still None"
|
|
)
|
|
|
|
|
|
@when("I assign a mock object to the chat_model property")
|
|
def step_assign_mock_to_chat_model(context: Any) -> None:
|
|
"""Assign a mock to the chat_model property (setter path)."""
|
|
context.injected_mock = MagicMock()
|
|
context.agent.chat_model = context.injected_mock
|
|
|
|
|
|
@then("the _chat_model attribute should be the mock object")
|
|
def step_chat_model_is_mock(context: Any) -> None:
|
|
"""Verify _chat_model is the injected mock."""
|
|
assert context.agent._chat_model is context.injected_mock, (
|
|
"Expected _chat_model to be the injected mock"
|
|
)
|
|
|
|
|
|
@then("accessing chat_model again returns the same mock")
|
|
def step_chat_model_getter_returns_mock(context: Any) -> None:
|
|
"""Verify getter returns injected mock without triggering lazy init."""
|
|
result = context.agent.chat_model
|
|
assert result is context.injected_mock, (
|
|
"Expected chat_model getter to return the injected mock"
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# AgentFactory threads credentials to LLMAgent
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@step("I call create_agent for the llm agent")
|
|
def step_factory_create_agent(context: Any) -> None:
|
|
"""Call factory.create_agent for the LLM agent name."""
|
|
context.created_agent = context.factory.create_agent("my_agent")
|
|
|
|
|
|
@then("the created LLMAgent should have the credentials dict set")
|
|
def step_created_agent_has_credentials(context: Any) -> None:
|
|
"""Verify the created LLMAgent has the provider-specific credentials slice."""
|
|
assert isinstance(context.created_agent, LLMAgent), (
|
|
f"Expected LLMAgent, got {type(context.created_agent).__name__}"
|
|
)
|
|
# The factory extracts the provider-specific slice from the credentials dict
|
|
# (e.g. credentials["openai"] → {"api_key": "injected-key"}).
|
|
expected = context.credentials.get("openai")
|
|
assert context.created_agent._credentials == expected, (
|
|
f"Expected _credentials {expected!r}, "
|
|
f"got {context.created_agent._credentials!r}"
|
|
)
|
|
|
|
|
|
@when("I call create_agent for the llm agent with api_key in config")
|
|
def step_factory_create_agent_with_api_key(context: Any) -> None:
|
|
"""Call factory.create_agent; agent has api_key in config (standalone mode)."""
|
|
context.actor_config["agents"]["my_agent"]["config"]["api_key"] = "test-key"
|
|
context.factory = AgentFactory(
|
|
config=context.actor_config,
|
|
template_renderer=context.template_renderer,
|
|
)
|
|
context.created_agent = context.factory.create_agent("my_agent")
|
|
|
|
|
|
@then("the created LLMAgent should have credentials equal to None")
|
|
def step_created_agent_credentials_none(context: Any) -> None:
|
|
"""Verify the created LLMAgent has credentials=None when factory had none."""
|
|
assert isinstance(context.created_agent, LLMAgent), (
|
|
f"Expected LLMAgent, got {type(context.created_agent).__name__}"
|
|
)
|
|
assert context.created_agent._credentials is None, (
|
|
f"Expected _credentials to be None, "
|
|
f"got {type(context.created_agent._credentials).__name__}"
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# process_message end-to-end with credentials
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given('I inject a mock chat model that returns "{response_text}"')
|
|
def step_inject_mock_chat_model(context: Any, response_text: str) -> None:
|
|
"""Inject a mock chat model that returns the specified response."""
|
|
context.expected_response = response_text
|
|
context.agent.chat_model = mock_chat_model(response_text)
|
|
|
|
|
|
@when('I call process_message with "{message}"')
|
|
@async_run_until_complete
|
|
async def step_call_process_message(context: Any, message: str) -> None:
|
|
"""Call process_message and store the result."""
|
|
context.process_result = await context.agent.process_message(message)
|
|
|
|
|
|
@when('I call process_message with "{message}" (expecting error)')
|
|
@async_run_until_complete
|
|
async def step_call_process_message_expecting_error(context: Any, message: str) -> None:
|
|
"""Call process_message and capture any exception raised."""
|
|
try:
|
|
context.process_result = await context.agent.process_message(message)
|
|
context.raised_exception = None
|
|
except ConfigurationError as exc:
|
|
context.raised_exception = exc
|
|
except Exception as exc:
|
|
context.raised_exception = exc
|
|
|
|
|
|
@then('the response should be "{expected_text}"')
|
|
def step_response_should_be(context: Any, expected_text: str) -> None:
|
|
"""Verify the process_message result matches the expected text."""
|
|
assert context.process_result == expected_text, (
|
|
f"Expected response {expected_text!r}, got {context.process_result!r}"
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# cleanup() when _chat_model is None
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@when("I call cleanup on the agent")
|
|
@async_run_until_complete
|
|
async def step_call_cleanup(context: Any) -> None:
|
|
"""Call cleanup on the agent."""
|
|
context.cleanup_error = None
|
|
try:
|
|
await context.agent.cleanup()
|
|
except Exception as e:
|
|
context.cleanup_error = e
|
|
|
|
|
|
@when("I call cleanup on the agent again")
|
|
@async_run_until_complete
|
|
async def step_call_cleanup_again(context: Any) -> None:
|
|
"""Call cleanup on the agent a second time (idempotency test)."""
|
|
try:
|
|
await context.agent.cleanup()
|
|
except Exception as e:
|
|
context.cleanup_error = e
|
|
|
|
|
|
@then("no error should occur and _chat_model should remain None")
|
|
def step_cleanup_noop(context: Any) -> None:
|
|
"""Verify cleanup did not raise and _chat_model is still None."""
|
|
assert context.cleanup_error is None, (
|
|
f"Expected no error from cleanup, got {context.cleanup_error!r}"
|
|
)
|
|
assert context.agent._chat_model is None, (
|
|
f"Expected _chat_model to remain None after cleanup, "
|
|
f"got {context.agent._chat_model!r}"
|
|
)
|
|
|
|
|
|
@given(
|
|
"I construct an LLMAgent for openai with credentials dict containing only "
|
|
"api_key and config containing temperature 0.0"
|
|
)
|
|
def step_openai_agent_with_creds_and_temperature_zero(context: Any) -> None:
|
|
"""Construct LLMAgent with temperature=0.0 — a valid, falsy value."""
|
|
context.credentials = {"api_key": "injected-openai-key"}
|
|
config: dict[str, Any] = {
|
|
"provider": "openai",
|
|
"model": "gpt-3.5-turbo",
|
|
"temperature": 0.0,
|
|
}
|
|
context.renderer = make_template_renderer()
|
|
context.agent = LLMAgent(
|
|
name="test_openai_temp_zero",
|
|
config=config,
|
|
template_renderer=context.renderer,
|
|
credentials=context.credentials,
|
|
)
|
|
|
|
|
|
@then("the agent temperature should be 0.0")
|
|
def step_agent_temperature_is_zero(context: Any) -> None:
|
|
"""Verify temperature is exactly 0.0 — not replaced by default 0.7."""
|
|
assert context.agent.temperature == 0.0, (
|
|
f"Expected temperature 0.0, got {context.agent.temperature}"
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# M2: validate_credentials_structure error paths (AgentFactory)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given(
|
|
'I pass a credentials dict with a non-dict value for "{provider}" to AgentFactory'
|
|
)
|
|
def step_factory_non_dict_cred_entry(context: Any, provider: str) -> None:
|
|
"""Prepare credentials where an entry is not a dict.
|
|
|
|
The actual AgentFactory construction (and error capture) happens in the
|
|
When step — this Given step only prepares data.
|
|
"""
|
|
context.actor_config = minimal_actor_config()
|
|
context.template_renderer = make_template_renderer()
|
|
context.credentials = {
|
|
provider: cast(dict[str, str], "not-a-dict")
|
|
} # intentionally invalid
|
|
|
|
|
|
@given(
|
|
"I pass a credentials dict with a non-str inner value for "
|
|
'"{provider}".api_key to AgentFactory'
|
|
)
|
|
def step_factory_non_str_inner_value(context: Any, provider: str) -> None:
|
|
"""Prepare credentials where inner api_key is not a str.
|
|
|
|
The actual AgentFactory construction (and error capture) happens in the
|
|
When step — this Given step only prepares data.
|
|
"""
|
|
context.actor_config = minimal_actor_config()
|
|
context.template_renderer = make_template_renderer()
|
|
context.credentials = {
|
|
provider: cast(dict[str, str], {"api_key": 12345})
|
|
} # intentionally invalid
|
|
|
|
|
|
@when("I call create_agent for the llm agent (expecting error)")
|
|
def step_factory_create_agent_expecting_error(context: Any) -> None:
|
|
"""Call factory.create_agent and capture any exception."""
|
|
try:
|
|
context.created_agent = context.factory.create_agent("my_agent")
|
|
context.raised_exception = None
|
|
except ConfigurationError as exc:
|
|
context.raised_exception = exc
|
|
except Exception as exc:
|
|
context.raised_exception = exc
|
|
|
|
|
|
@when("I create an AgentFactory with the prepared credentials (expecting error)")
|
|
def step_factory_create_with_bad_credentials(context: Any) -> None:
|
|
"""Create AgentFactory with prepared bad credentials; capture any error."""
|
|
try:
|
|
context.factory = AgentFactory(
|
|
config=context.actor_config,
|
|
template_renderer=context.template_renderer,
|
|
credentials=context.credentials,
|
|
)
|
|
context.raised_exception = None
|
|
except ConfigurationError as exc:
|
|
context.raised_exception = exc
|
|
context.factory = None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# m4: AgentCreationError wrapping in factory._create_agent_instance
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given("I patch LLMAgent.__init__ to raise RuntimeError")
|
|
def step_patch_llm_agent_init_runtime_error(context: Any) -> None:
|
|
"""Patch LLMAgent.__init__ to raise RuntimeError so we can test
|
|
that AgentFactory wraps unexpected exceptions in AgentCreationError.
|
|
"""
|
|
patcher = patch.object(
|
|
LLMAgent,
|
|
"__init__",
|
|
side_effect=RuntimeError("Unexpected constructor failure"),
|
|
)
|
|
patcher.start()
|
|
if not hasattr(context, "_active_patches"):
|
|
context._active_patches = []
|
|
context._active_patches.append(patcher)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Generic Then step for non-ConfigurationError exception assertions
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@then('the raised error should be an AgentCreationError containing "{expected_text}"')
|
|
def step_agent_creation_error_contains(context: Any, expected_text: str) -> None:
|
|
"""Verify the captured exception is AgentCreationError with expected message."""
|
|
assert context.raised_exception is not None, (
|
|
"Expected an error to be raised, but none was captured"
|
|
)
|
|
assert isinstance(context.raised_exception, AgentCreationError), (
|
|
f"Expected AgentCreationError, got {type(context.raised_exception).__name__}: "
|
|
f"{context.raised_exception!r}"
|
|
)
|
|
assert expected_text in str(context.raised_exception), (
|
|
f"Expected error message to contain {expected_text!r}, "
|
|
f"got {str(context.raised_exception)!r}"
|
|
)
|
|
|
|
|
|
@then('the raised error should be an ExecutionError containing "{expected_text}"')
|
|
def step_execution_error_contains(context: Any, expected_text: str) -> None:
|
|
"""Verify the captured exception is ExecutionError with expected message."""
|
|
assert context.raised_exception is not None, (
|
|
"Expected an error to be raised, but none was captured"
|
|
)
|
|
assert isinstance(context.raised_exception, ExecutionError), (
|
|
f"Expected ExecutionError, got {type(context.raised_exception).__name__}: "
|
|
f"{context.raised_exception!r}"
|
|
)
|
|
assert expected_text in str(context.raised_exception), (
|
|
f"Expected error message to contain {expected_text!r}, "
|
|
f"got {str(context.raised_exception)!r}"
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# M2: Inner-dict mutation protection
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@given('I mutate the original credentials dict\'s api_key to "{mutated_value}"')
|
|
def step_mutate_credentials_api_key(context: Any, mutated_value: str) -> None:
|
|
"""Mutate the original (caller-side) credentials dict api_key value.
|
|
|
|
The stored credentials should be unaffected because
|
|
LLMAgent.__init__ shallow-copies the credentials dict.
|
|
"""
|
|
context.credentials["api_key"] = mutated_value
|
|
|
|
|
|
@then("the agent's stored api_key should still be the original value")
|
|
def step_stored_api_key_unchanged(context: Any) -> None:
|
|
"""Verify the agent's stored api_key was not affected by caller mutation."""
|
|
creds = context.agent._credentials
|
|
assert creds is not None, "Expected credentials to be set on agent"
|
|
actual = creds.get("api_key")
|
|
assert actual == "injected-openai-key", (
|
|
f"Expected stored api_key to be 'injected-openai-key' (original value), "
|
|
f"got {actual!r}"
|
|
)
|