diff --git a/features/runtime_coverage.feature b/features/runtime_coverage.feature new file mode 100644 index 0000000..1d0661f --- /dev/null +++ b/features/runtime_coverage.feature @@ -0,0 +1,123 @@ +Feature: Runtime Executor API + As a developer + I want the router-facing Executor API to correctly create executors and dispatch to LLM, graph, tool, and multi-actor agents + So that the CleverThis router can invoke actors via a stable interface + + Background: + Given the runtime test environment is initialized + + Scenario: create_executor constructs an Executor with all parameters + Given a valid actor config dict with type "llm" + And credentials dict with openai provider + And limits dict with max_depth 5 + And pricing dict with per_token_cost 0.01 + When I call create_executor + Then an Executor instance should be returned + And the executor config should match the config dict + And the executor credentials should match the credentials dict + And the executor limits should match the limits dict + And the executor pricing should match the pricing dict + + Scenario: create_executor with None limits and pricing defaults to empty dicts + Given a valid actor config dict with type "llm" + And credentials dict with openai provider + When I call create_executor with None limits and pricing + Then an Executor instance should be returned + And the executor limits should be an empty dictionary + And the executor pricing should be an empty dictionary + + Scenario: execute dispatches to LLM agent when type is "llm" + Given a valid actor config dict with type "llm" and openai provider + And credentials dict with openai api key + When I execute the actor with message "Hello" + Then the execution should return an ActorResult + And the ActorResult response should be non-empty + And the ActorResult should have token usage tracked + + Scenario: execute dispatches to graph agent when type is "graph" + Given a valid actor config dict with type "graph" and route definition + And credentials dict with openai provider + When I execute the actor with message "Hello graph" + Then the execution should return an ActorResult + And the ActorResult should have at least one node usage entry + + Scenario: execute dispatches to tool agent when type is "tool" + Given a valid actor config dict with type "tool" and tools list + And credentials dict with openai provider + When I execute the actor with message "echo test" + Then the execution should return an ActorResult + And the ActorResult should have zero prompt tokens for tool agents + + Scenario: execute dispatches to multi_actor when type is "multi_actor" + Given a multi-actor config dict with multiple sub-actors + And credentials dict with openai provider + When I execute the actor with message "Hello multi" + Then the execution should return an ActorResult + And the node usage IDs should be prefixed with the default actor name + + Scenario: execute raises ConfigurationError for unknown actor type + Given a valid actor config dict with type "unknown_type" + And credentials dict with openai provider + When I execute the actor with message "test" + Then a ConfigurationError should be raised for runtime + And the error message should mention the unknown actor type + + Scenario: _execute_llm handles execution failure gracefully + Given a valid actor config dict with type "llm" and openai provider + And credentials dict with openai api key + And the LLM agent is configured to fail + When I execute the actor with message "Hello" + Then a ConfigurationError should be raised for runtime with the original exception chained + + Scenario: _execute_graph builds PureGraphConfig from route definition + Given a valid actor config dict with type "graph" and full route definition + And credentials dict with openai provider + When I execute the actor with message "Hello graph world" + Then the execution should return an ActorResult + + Scenario: _execute_multi_actor raises ConfigurationError with no actors + Given a multi-actor config dict with empty actors + And credentials dict with openai provider + When I execute the actor with message "test" + Then a ConfigurationError should be raised for runtime about no actors + + Scenario: _build_factory_config injects credentials into agent configs + Given a multi-actor config dict with llm agents block + And credentials dict with openai provider + When I call _build_factory_config on the executor + Then the factory config should have credentials injected into the agents block + + Scenario: _build_factory_config adds global context with credentials and limits + Given a valid actor config dict with type "llm" + And credentials dict with openai provider + And limits dict with max_depth 10 + When I call _build_factory_config on the executor + Then the factory config context global should contain credentials and limits + + Scenario: _estimate_tokens uses tiktoken when available + Given a prompt string of 100 characters + And a response string of 50 characters + When I estimate tokens for model "gpt-3.5-turbo" + Then the estimated prompt tokens should be greater than 0 + And the estimated completion tokens should be greater than 0 + + Scenario: _estimate_tokens falls back to heuristic when tiktoken unavailable + Given a prompt string of 400 characters + And a response string of 200 characters + When I estimate tokens with tiktoken unavailable + Then the estimated prompt tokens should be proportional to the prompt length + And the estimated completion tokens should be proportional to the response length + + Scenario: NodeUsage dataclass stores per-node token usage correctly + Given a NodeUsage with node_id "test_node" and provider "openai" and prompt_tokens 100 and completion_tokens 50 + Then the NodeUsage node_id should be "test_node" + And the NodeUsage provider should be "openai" + And the NodeUsage prompt_tokens should be 100 + And the NodeUsage completion_tokens should be 50 + + Scenario: ActorResult dataclass stores execution result correctly + Given an ActorResult with response "test response" and prompt_tokens 100 and completion_tokens 50 + Then the ActorResult response should be "test response" + And the ActorResult prompt_tokens should be 100 + And the ActorResult completion_tokens should be 50 + And the ActorResult nodes should be empty by default \ No newline at end of file diff --git a/features/steps/runtime_coverage_steps.py b/features/steps/runtime_coverage_steps.py new file mode 100644 index 0000000..772d9f9 --- /dev/null +++ b/features/steps/runtime_coverage_steps.py @@ -0,0 +1,666 @@ +"""Step definitions for Runtime Executor API BDD tests.""" + +import asyncio +from unittest.mock import AsyncMock, MagicMock, patch + +from behave import given, then, when +from behave.api.async_step import async_run_until_complete + +from cleveractors.core.exceptions import ConfigurationError +from cleveractors.runtime import ( + ActorResult, + Executor, + NodeUsage, + _estimate_tokens, + create_executor, +) + + +@given("the runtime test environment is initialized") +def step_runtime_init(context): + context.test_executor = None + context.test_result = None + context.test_error = None + + +def _build_base_config(type_name): + return { + "name": f"test_{type_name}", + "type": type_name, + } + + +@given('a valid actor config dict with type "unknown_type"') +def step_config_unknown(context): + context.config_dict = _build_base_config("unknown_type") + + +@given('a valid actor config dict with type "llm"') +def step_config_llm_basic(context): + context.config_dict = _build_base_config("llm") + context.config_dict.update( + { + "provider": "openai", + "model": "gpt-3.5-turbo", + "system_prompt": "You are a helpful assistant.", + "temperature": 0.7, + "max_tokens": 1000, + } + ) + + +@given('a valid actor config dict with type "graph" and route definition') +def step_config_graph_route(context): + context.config_dict = _build_base_config("graph") + context.config_dict["route"] = { + "nodes": [ + {"id": "start", "type": "start"}, + {"id": "end", "type": "end"}, + ], + "edges": [], + "entry_node": "start", + "exit_nodes": ["end"], + } + + +@given('a valid actor config dict with type "tool" and tools list') +def step_config_tool(context): + context.config_dict = _build_base_config("tool") + context.config_dict["tools"] = ["echo"] + context.config_dict["config"] = {"tools": ["echo"]} + + +@given('a valid actor config dict with type "llm" and openai provider') +def step_config_llm_openai(context): + context.config_dict = _build_base_config("llm") + context.config_dict.update( + { + "provider": "openai", + "model": "gpt-3.5-turbo", + "system_prompt": "You are a helpful assistant.", + "temperature": 0.7, + "max_tokens": 1000, + } + ) + + +@given('a valid actor config dict with type "graph" and openai provider') +def step_config_graph_openai(context): + context.config_dict = _build_base_config("graph") + context.config_dict.update( + { + "provider": "openai", + "model": "gpt-3.5-turbo", + "system_prompt": "You are a helpful assistant.", + "temperature": 0.7, + "max_tokens": 1000, + } + ) + context.config_dict["route"] = { + "nodes": [ + {"id": "start", "type": "start"}, + {"id": "end", "type": "end"}, + ], + "edges": [], + "entry_node": "start", + "exit_nodes": ["end"], + } + + +@given('a valid actor config dict with type "tool" and openai provider') +def step_config_tool_openai(context): + context.config_dict = _build_base_config("tool") + context.config_dict.update( + { + "provider": "openai", + "model": "gpt-3.5-turbo", + "tools": ["echo"], + "config": {"tools": ["echo"]}, + } + ) + + +@given('a valid actor config dict with type "graph" and full route definition') +def step_config_graph_full(context): + context.config_dict = _build_base_config("graph") + context.config_dict["route"] = { + "nodes": [ + {"id": "start", "type": "start"}, + {"id": "process", "type": "function", "function": "process_input"}, + {"id": "llm_node", "type": "llm", "agent": "test_agent"}, + {"id": "end", "type": "end"}, + ], + "edges": [ + {"from": "start", "to": "process"}, + {"from": "process", "to": "llm_node"}, + {"from": "llm_node", "to": "end"}, + ], + "entry_node": "start", + "exit_nodes": ["end"], + } + context.config_dict["agents"] = { + "test_agent": {"type": "llm", "config": {"provider": "openai"}}, + } + + +@given('a valid actor config dict with type "llm" and provider "custom"') +def step_config_llm_custom(context): + context.config_dict = _build_base_config("llm") + context.config_dict.update( + { + "provider": "custom", + "model": "custom-model", + "system_prompt": "You are helpful.", + "temperature": 0.5, + "max_tokens": 500, + } + ) + + +@given('a valid actor config dict with type "tool" and config block tools') +def step_config_tool_block(context): + context.config_dict = _build_base_config("tool") + context.config_dict["config"] = {"tools": ["echo", "math"]} + + +@given("credentials dict with openai provider") +def step_creds_openai(context): + context.credentials = { + "openai": { + "api_key": "sk-test-key-12345", + "base_url": "https://api.openai.com/v1", + } + } + + +@given("credentials dict with openai api key") +def step_creds_openai_key(context): + context.credentials = {"openai": {"api_key": "sk-test-executor-key"}} + + +@given('credentials dict with openai api key "sk-test-injected"') +def step_creds_openai_injected(context): + context.credentials = {"openai": {"api_key": "sk-test-injected"}} + + +@given("credentials dict with openai_compatible provider") +def step_creds_compat(context): + context.credentials = { + "openai_compatible": { + "api_key": "sk-compat-key", + "base_url": "https://custom.api/v1", + } + } + + +@given("limits dict with max_depth {depth}") +def step_limits(context, depth): + context.limits = {"max_depth": int(depth)} + + +@given("pricing dict with per_token_cost {cost}") +def step_pricing(context, cost): + context.pricing = {"per_token_cost": float(cost)} + + +@given("a multi-actor config dict with multiple sub-actors") +def step_multi_actor_config(context): + context.config_dict = { + "name": "test_multi_actor", + "type": "multi_actor", + "actors": { + "default": { + "type": "llm", + "provider": "openai", + "model": "gpt-3.5-turbo", + "system_prompt": "You are helpful.", + "temperature": 0.7, + "max_tokens": 1000, + }, + "secondary": { + "type": "tool", + "tools": ["echo"], + }, + }, + "cleveragents": {"default_actor": "default"}, + } + + +@given("a multi-actor config dict with actors but no default_actor") +def step_multi_actor_no_default(context): + context.config_dict = { + "name": "test_multi_no_default", + "type": "multi_actor", + "actors": { + "first": { + "type": "llm", + "provider": "openai", + "model": "gpt-3.5-turbo", + "system_prompt": "You are helpful.", + "temperature": 0.7, + "max_tokens": 1000, + }, + }, + "cleveragents": {"default_actor": "nonexistent"}, + } + + +@given("a multi-actor config dict with empty actors") +def step_multi_actor_empty(context): + context.config_dict = { + "name": "test_empty_actors", + "type": "multi_actor", + "actors": {}, + "cleveragents": {"default_actor": "default"}, + } + + +@given("a multi-actor config dict with llm agents block") +def step_multi_actor_llm_agents(context): + context.config_dict = { + "name": "test_multi_llm_agents", + "type": "multi_actor", + "actors": { + "agent1": { + "type": "llm", + "provider": "openai", + "model": "gpt-3.5-turbo", + "system_prompt": "You are helpful.", + "temperature": 0.7, + "max_tokens": 1000, + }, + }, + "agents": { + "agent1": { + "type": "llm", + "provider": "openai", + "config": {"provider": "openai"}, + }, + }, + "cleveragents": {"default_actor": "agent1"}, + } + + +@given("a config dict with actors key but no type field") +def step_config_actors_no_type(context): + context.config_dict = { + "name": "test_auto_multi", + "actors": { + "default": { + "type": "llm", + "provider": "openai", + "model": "gpt-3.5-turbo", + "system_prompt": "You are helpful.", + "temperature": 0.7, + "max_tokens": 1000, + }, + }, + "cleveragents": {"default_actor": "default"}, + } + + +@given("a config dict with only llm config and no type or actors key") +def step_config_llm_only(context): + context.config_dict = { + "name": "test_auto_llm", + "provider": "openai", + "model": "gpt-3.5-turbo", + "system_prompt": "You are helpful.", + "temperature": 0.7, + "max_tokens": 1000, + } + + +@given("the LLM agent is configured to fail") +def step_llm_fail(context): + context._llm_should_fail = True + + +@given("a prompt string of {length} characters") +def step_prompt_length(context, length): + context.prompt = "x" * int(length) + + +@given("a response string of {length} characters") +def step_response_length(context, length): + context.response = "y" * int(length) + + +@given( + "a NodeUsage with node_id {node_id} and provider {provider} and prompt_tokens {pt} and completion_tokens {ct}" +) +def step_node_usage(context, node_id, provider, pt, ct): + node_id = node_id.strip('"') + provider = provider.strip('"') + context.node_usage = NodeUsage( + node_id=node_id, + provider=provider, + model="test-model", + prompt_tokens=int(pt), + completion_tokens=int(ct), + ) + + +@given( + "an ActorResult with response {resp} and prompt_tokens {pt} and completion_tokens {ct}" +) +def step_actor_result(context, resp, pt, ct): + resp = resp.strip('"') + context.actor_result = ActorResult( + response=resp, prompt_tokens=int(pt), completion_tokens=int(ct) + ) + + +@when("I call create_executor") +def step_call_create_executor(context): + context.test_executor = create_executor( + config_dict=context.config_dict, + credentials=context.credentials, + limits=context.limits if hasattr(context, "limits") else None, + pricing=context.pricing if hasattr(context, "pricing") else None, + ) + + +@when("I call create_executor with None limits and pricing") +def step_call_create_executor_none(context): + context.test_executor = create_executor( + config_dict=context.config_dict, + credentials=context.credentials, + limits=None, + pricing=None, + ) + + +@when("I execute the actor with message {msg}") +@async_run_until_complete +async def step_execute_actor(context, msg): + msg = msg.strip('"') + executor = create_executor( + config_dict=context.config_dict, + credentials=context.credentials, + limits=context.limits if hasattr(context, "limits") and context.limits else {}, + pricing=context.pricing + if hasattr(context, "pricing") and context.pricing + else {}, + ) + context.test_executor = executor + + with ( + patch("cleveractors.agents.llm.LLMAgent") as mock_llm, + patch("cleveractors.templates.renderer.TemplateRenderer") as mock_renderer, + patch("cleveractors.agents.tool.ToolAgent") as mock_tool, + patch("cleveractors.agents.factory.AgentFactory") as mock_factory, + patch("cleveractors.langgraph.pure_graph.PureLangGraph") as mock_pure_graph, + patch("cleveractors.runtime._estimate_tokens", return_value=(100, 50)), + ): + mock_renderer_instance = MagicMock() + mock_renderer.return_value = mock_renderer_instance + + if getattr(context, "_llm_should_fail", False): + mock_llm_instance = MagicMock() + mock_llm_instance.process_message = AsyncMock( + side_effect=Exception("LLM API error") + ) + mock_llm.return_value = mock_llm_instance + else: + mock_llm_instance = MagicMock() + mock_llm_instance.process_message = AsyncMock( + return_value="Mock LLM response" + ) + mock_llm.return_value = mock_llm_instance + + mock_tool_instance = MagicMock() + mock_tool_instance.process_message = AsyncMock( + return_value="Mock tool response" + ) + mock_tool.return_value = mock_tool_instance + + mock_graph_instance = MagicMock() + mock_graph_instance.execute = AsyncMock(return_value="Mock graph response") + mock_pure_graph.return_value = mock_graph_instance + + mock_factory_instance = MagicMock() + mock_factory_instance.create_agent = MagicMock(return_value=mock_llm_instance) + mock_factory.return_value = mock_factory_instance + + try: + context.test_result = await executor.execute(msg) + context.test_error = None + except Exception as e: + context.test_error = e + context.test_result = None + + +@when("I call _build_factory_config on the executor") +def step_call_build_factory(context): + executor = create_executor( + config_dict=context.config_dict, + credentials=context.credentials, + limits=context.limits if hasattr(context, "limits") and context.limits else {}, + pricing=context.pricing + if hasattr(context, "pricing") and context.pricing + else {}, + ) + context.test_executor = executor + context.factory_config = executor._build_factory_config() + + +@when("I estimate tokens for model {model}") +def step_estimate_tokens(context, model): + model = model.strip('"') + context.prompt_tokens, context.completion_tokens = _estimate_tokens( + context.prompt, context.response, model, "openai" + ) + + +@when("I estimate tokens with tiktoken unavailable") +def step_estimate_tokens_fallback(context): + with ( + patch("cleveractors.runtime.tiktoken", None, create=True), + patch("builtins.__import__", side_effect=ImportError) as _mock_import, + ): + context.prompt_tokens, context.completion_tokens = _estimate_tokens( + context.prompt, context.response, "unknown-model", "unknown" + ) + + +@then("an Executor instance should be returned") +def step_assert_executor(context): + assert context.test_executor is not None + assert isinstance(context.test_executor, Executor) + + +@then("the executor config should match the config dict") +def step_assert_config(context): + assert context.test_executor.config == context.config_dict + + +@then("the executor credentials should match the credentials dict") +def step_assert_credentials(context): + assert context.test_executor.credentials == context.credentials + + +@then("the executor limits should match the limits dict") +def step_assert_limits(context): + assert context.test_executor.limits == context.limits + + +@then("the executor pricing should match the pricing dict") +def step_assert_pricing(context): + assert context.test_executor.pricing == context.pricing + + +@then("the executor limits should be an empty dictionary") +def step_assert_limits_empty(context): + assert context.test_executor.limits == {} + + +@then("the executor pricing should be an empty dictionary") +def step_assert_pricing_empty(context): + assert context.test_executor.pricing == {} + + +@then("the execution should return an ActorResult") +def step_assert_actor_result(context): + assert context.test_result is not None + assert isinstance(context.test_result, ActorResult) + + +@then("the ActorResult response should be non-empty") +def step_assert_response(context): + assert context.test_result.response is not None + assert len(context.test_result.response) > 0 + + +@then("the ActorResult should have token usage tracked") +def step_assert_tokens(context): + assert context.test_result.prompt_tokens >= 0 + assert context.test_result.completion_tokens >= 0 + + +@then("the ActorResult should have at least one node usage entry") +def step_assert_nodes(context): + assert len(context.test_result.nodes) >= 1 + + +@then("the ActorResult should have zero prompt tokens for tool agents") +def step_assert_zero_tokens(context): + assert context.test_result.prompt_tokens == 0 + assert context.test_result.completion_tokens == 0 + + +@then("the node usage IDs should be prefixed with the default actor name") +def step_assert_node_prefix(context): + for node in context.test_result.nodes: + assert node.node_id.startswith("default.") + + +@then("a ConfigurationError should be raised for runtime") +def step_assert_config_error(context): + assert context.test_error is not None + assert isinstance(context.test_error, ConfigurationError) + + +@then("the error message should mention the unknown actor type") +def step_assert_error_unknown_type(context): + assert "unknown_type" in str(context.test_error).lower() + + +@then( + "a ConfigurationError should be raised for runtime with the original exception chained" +) +def step_assert_chained_error(context): + assert context.test_error is not None + assert isinstance(context.test_error, ConfigurationError) + + +@then("the PureGraphConfig should be built with correct nodes and edges") +def step_assert_graph_config(context): + assert context.test_result is not None + assert isinstance(context.test_result, ActorResult) + + +@then("the LLM agent should be initialized with the injected api key") +def step_assert_injected_key(context): + assert context.test_result is not None + assert context.test_executor is not None + assert context.test_executor.credentials is not None + assert "openai" in context.test_executor.credentials + assert context.test_executor.credentials["openai"]["api_key"] == "sk-test-injected" + + +@then("a ConfigurationError should be raised for runtime about no actors") +def step_assert_no_actors_error(context): + assert context.test_error is not None + assert isinstance(context.test_error, ConfigurationError) + + +@then("the factory config should have credentials injected into the agents block") +def step_assert_factory_creds(context): + assert context.factory_config is not None + assert "agents" in context.factory_config + assert "context" in context.factory_config + + +@then("the factory config context global should contain credentials and limits") +def step_assert_factory_context(context): + assert context.factory_config is not None + assert "context" in context.factory_config + assert "global" in context.factory_config["context"] + assert "credentials" in context.factory_config["context"]["global"] + assert "limits" in context.factory_config["context"]["global"] + + +@then("the estimated prompt tokens should be greater than 0") +def step_assert_prompt_tokens(context): + assert context.prompt_tokens > 0 + + +@then("the estimated completion tokens should be greater than 0") +def step_assert_completion_tokens(context): + assert context.completion_tokens > 0 + + +@then("the estimated prompt tokens should be proportional to the prompt length") +def step_assert_prompt_proportional(context): + assert context.prompt_tokens > 0 + assert context.prompt_tokens >= len(context.prompt) // 4 - 1 + + +@then("the estimated completion tokens should be proportional to the response length") +def step_assert_completion_proportional(context): + assert context.completion_tokens > 0 + assert context.completion_tokens >= len(context.response) // 4 - 1 + + +@then("the estimated completion tokens should be 1 or more") +def step_assert_completion_tokens_min(context): + assert context.completion_tokens >= 1 + + +@then("the estimated prompt tokens should be 1 or more") +def step_assert_prompt_tokens_min(context): + assert context.prompt_tokens >= 1 + + +@then("the NodeUsage node_id should be {expected}") +def step_assert_node_id(context, expected): + expected = expected.strip('"') + assert context.node_usage.node_id == expected + + +@then("the NodeUsage provider should be {expected}") +def step_assert_node_provider(context, expected): + expected = expected.strip('"') + assert context.node_usage.provider == expected + + +@then("the NodeUsage prompt_tokens should be {expected}") +def step_assert_node_pt(context, expected): + assert context.node_usage.prompt_tokens == int(expected) + + +@then("the NodeUsage completion_tokens should be {expected}") +def step_assert_node_ct(context, expected): + assert context.node_usage.completion_tokens == int(expected) + + +@then("the ActorResult response should be {expected}") +def step_assert_ar_response(context, expected): + expected = expected.strip('"') + assert context.actor_result.response == expected + + +@then("the ActorResult prompt_tokens should be {expected}") +def step_assert_ar_pt(context, expected): + assert context.actor_result.prompt_tokens == int(expected) + + +@then("the ActorResult completion_tokens should be {expected}") +def step_assert_ar_ct(context, expected): + assert context.actor_result.completion_tokens == int(expected) + + +@then("the ActorResult nodes should be empty by default") +def step_assert_ar_nodes_empty(context): + assert context.actor_result.nodes == []