test(runtime): add BDD coverage tests for runtime Executor API #34
@@ -0,0 +1,123 @@
|
||||
Feature: Runtime Executor API
|
||||
As a developer
|
||||
I want the router-facing Executor API to correctly create executors and dispatch to LLM, graph, tool, and multi-actor agents
|
||||
So that the CleverThis router can invoke actors via a stable interface
|
||||
|
||||
Background:
|
||||
Given the runtime test environment is initialized
|
||||
|
||||
Scenario: create_executor constructs an Executor with all parameters
|
||||
Given a valid actor config dict with type "llm"
|
||||
And credentials dict with openai provider
|
||||
And limits dict with max_depth 5
|
||||
And pricing dict with per_token_cost 0.01
|
||||
When I call create_executor
|
||||
Then an Executor instance should be returned
|
||||
And the executor config should match the config dict
|
||||
And the executor credentials should match the credentials dict
|
||||
And the executor limits should match the limits dict
|
||||
And the executor pricing should match the pricing dict
|
||||
|
||||
Scenario: create_executor with None limits and pricing defaults to empty dicts
|
||||
Given a valid actor config dict with type "llm"
|
||||
And credentials dict with openai provider
|
||||
When I call create_executor with None limits and pricing
|
||||
Then an Executor instance should be returned
|
||||
And the executor limits should be an empty dictionary
|
||||
And the executor pricing should be an empty dictionary
|
||||
|
||||
Scenario: execute dispatches to LLM agent when type is "llm"
|
||||
Given a valid actor config dict with type "llm" and openai provider
|
||||
And credentials dict with openai api key
|
||||
When I execute the actor with message "Hello"
|
||||
Then the execution should return an ActorResult
|
||||
And the ActorResult response should be non-empty
|
||||
And the ActorResult should have token usage tracked
|
||||
|
||||
Scenario: execute dispatches to graph agent when type is "graph"
|
||||
Given a valid actor config dict with type "graph" and route definition
|
||||
And credentials dict with openai provider
|
||||
When I execute the actor with message "Hello graph"
|
||||
Then the execution should return an ActorResult
|
||||
And the ActorResult should have at least one node usage entry
|
||||
|
||||
Scenario: execute dispatches to tool agent when type is "tool"
|
||||
Given a valid actor config dict with type "tool" and tools list
|
||||
And credentials dict with openai provider
|
||||
When I execute the actor with message "echo test"
|
||||
Then the execution should return an ActorResult
|
||||
And the ActorResult should have zero prompt tokens for tool agents
|
||||
|
||||
Scenario: execute dispatches to multi_actor when type is "multi_actor"
|
||||
Given a multi-actor config dict with multiple sub-actors
|
||||
And credentials dict with openai provider
|
||||
When I execute the actor with message "Hello multi"
|
||||
Then the execution should return an ActorResult
|
||||
And the node usage IDs should be prefixed with the default actor name
|
||||
|
||||
Scenario: execute raises ConfigurationError for unknown actor type
|
||||
Given a valid actor config dict with type "unknown_type"
|
||||
And credentials dict with openai provider
|
||||
When I execute the actor with message "test"
|
||||
Then a ConfigurationError should be raised for runtime
|
||||
And the error message should mention the unknown actor type
|
||||
|
||||
Scenario: _execute_llm handles execution failure gracefully
|
||||
Given a valid actor config dict with type "llm" and openai provider
|
||||
And credentials dict with openai api key
|
||||
And the LLM agent is configured to fail
|
||||
When I execute the actor with message "Hello"
|
||||
Then a ConfigurationError should be raised for runtime with the original exception chained
|
||||
|
||||
Scenario: _execute_graph builds PureGraphConfig from route definition
|
||||
Given a valid actor config dict with type "graph" and full route definition
|
||||
And credentials dict with openai provider
|
||||
When I execute the actor with message "Hello graph world"
|
||||
Then the execution should return an ActorResult
|
||||
|
||||
Scenario: _execute_multi_actor raises ConfigurationError with no actors
|
||||
Given a multi-actor config dict with empty actors
|
||||
And credentials dict with openai provider
|
||||
When I execute the actor with message "test"
|
||||
Then a ConfigurationError should be raised for runtime about no actors
|
||||
|
||||
Scenario: _build_factory_config injects credentials into agent configs
|
||||
Given a multi-actor config dict with llm agents block
|
||||
And credentials dict with openai provider
|
||||
When I call _build_factory_config on the executor
|
||||
Then the factory config should have credentials injected into the agents block
|
||||
|
||||
Scenario: _build_factory_config adds global context with credentials and limits
|
||||
Given a valid actor config dict with type "llm"
|
||||
And credentials dict with openai provider
|
||||
And limits dict with max_depth 10
|
||||
When I call _build_factory_config on the executor
|
||||
Then the factory config context global should contain credentials and limits
|
||||
|
||||
Scenario: _estimate_tokens uses tiktoken when available
|
||||
Given a prompt string of 100 characters
|
||||
And a response string of 50 characters
|
||||
When I estimate tokens for model "gpt-3.5-turbo"
|
||||
Then the estimated prompt tokens should be greater than 0
|
||||
And the estimated completion tokens should be greater than 0
|
||||
|
||||
Scenario: _estimate_tokens falls back to heuristic when tiktoken unavailable
|
||||
Given a prompt string of 400 characters
|
||||
And a response string of 200 characters
|
||||
When I estimate tokens with tiktoken unavailable
|
||||
Then the estimated prompt tokens should be proportional to the prompt length
|
||||
And the estimated completion tokens should be proportional to the response length
|
||||
|
||||
Scenario: NodeUsage dataclass stores per-node token usage correctly
|
||||
Given a NodeUsage with node_id "test_node" and provider "openai" and prompt_tokens 100 and completion_tokens 50
|
||||
Then the NodeUsage node_id should be "test_node"
|
||||
And the NodeUsage provider should be "openai"
|
||||
And the NodeUsage prompt_tokens should be 100
|
||||
And the NodeUsage completion_tokens should be 50
|
||||
|
||||
Scenario: ActorResult dataclass stores execution result correctly
|
||||
Given an ActorResult with response "test response" and prompt_tokens 100 and completion_tokens 50
|
||||
Then the ActorResult response should be "test response"
|
||||
And the ActorResult prompt_tokens should be 100
|
||||
And the ActorResult completion_tokens should be 50
|
||||
And the ActorResult nodes should be empty by default
|
||||
@@ -0,0 +1,666 @@
|
||||
"""Step definitions for Runtime Executor API BDD tests."""
|
||||
|
||||
import asyncio
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
from behave import given, then, when
|
||||
from behave.api.async_step import async_run_until_complete
|
||||
|
||||
from cleveractors.core.exceptions import ConfigurationError
|
||||
from cleveractors.runtime import (
|
||||
ActorResult,
|
||||
Executor,
|
||||
NodeUsage,
|
||||
_estimate_tokens,
|
||||
create_executor,
|
||||
)
|
||||
|
||||
|
||||
@given("the runtime test environment is initialized")
|
||||
def step_runtime_init(context):
|
||||
context.test_executor = None
|
||||
context.test_result = None
|
||||
context.test_error = None
|
||||
|
||||
|
||||
def _build_base_config(type_name):
|
||||
return {
|
||||
"name": f"test_{type_name}",
|
||||
"type": type_name,
|
||||
}
|
||||
|
||||
|
||||
@given('a valid actor config dict with type "unknown_type"')
|
||||
def step_config_unknown(context):
|
||||
context.config_dict = _build_base_config("unknown_type")
|
||||
|
||||
|
||||
@given('a valid actor config dict with type "llm"')
|
||||
def step_config_llm_basic(context):
|
||||
context.config_dict = _build_base_config("llm")
|
||||
context.config_dict.update(
|
||||
{
|
||||
"provider": "openai",
|
||||
"model": "gpt-3.5-turbo",
|
||||
"system_prompt": "You are a helpful assistant.",
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 1000,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@given('a valid actor config dict with type "graph" and route definition')
|
||||
def step_config_graph_route(context):
|
||||
context.config_dict = _build_base_config("graph")
|
||||
context.config_dict["route"] = {
|
||||
"nodes": [
|
||||
{"id": "start", "type": "start"},
|
||||
{"id": "end", "type": "end"},
|
||||
],
|
||||
"edges": [],
|
||||
|
|
||||
"entry_node": "start",
|
||||
"exit_nodes": ["end"],
|
||||
}
|
||||
|
||||
|
||||
@given('a valid actor config dict with type "tool" and tools list')
|
||||
def step_config_tool(context):
|
||||
context.config_dict = _build_base_config("tool")
|
||||
context.config_dict["tools"] = ["echo"]
|
||||
context.config_dict["config"] = {"tools": ["echo"]}
|
||||
|
||||
|
||||
@given('a valid actor config dict with type "llm" and openai provider')
|
||||
def step_config_llm_openai(context):
|
||||
context.config_dict = _build_base_config("llm")
|
||||
context.config_dict.update(
|
||||
{
|
||||
"provider": "openai",
|
||||
"model": "gpt-3.5-turbo",
|
||||
"system_prompt": "You are a helpful assistant.",
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 1000,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@given('a valid actor config dict with type "graph" and openai provider')
|
||||
def step_config_graph_openai(context):
|
||||
context.config_dict = _build_base_config("graph")
|
||||
context.config_dict.update(
|
||||
{
|
||||
"provider": "openai",
|
||||
"model": "gpt-3.5-turbo",
|
||||
"system_prompt": "You are a helpful assistant.",
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 1000,
|
||||
}
|
||||
)
|
||||
context.config_dict["route"] = {
|
||||
"nodes": [
|
||||
{"id": "start", "type": "start"},
|
||||
brent.edwards
commented
See See `step_config_graph_route`.
|
||||
{"id": "end", "type": "end"},
|
||||
],
|
||||
"edges": [],
|
||||
"entry_node": "start",
|
||||
"exit_nodes": ["end"],
|
||||
}
|
||||
|
||||
|
||||
@given('a valid actor config dict with type "tool" and openai provider')
|
||||
def step_config_tool_openai(context):
|
||||
context.config_dict = _build_base_config("tool")
|
||||
context.config_dict.update(
|
||||
{
|
||||
"provider": "openai",
|
||||
"model": "gpt-3.5-turbo",
|
||||
"tools": ["echo"],
|
||||
"config": {"tools": ["echo"]},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@given('a valid actor config dict with type "graph" and full route definition')
|
||||
def step_config_graph_full(context):
|
||||
context.config_dict = _build_base_config("graph")
|
||||
context.config_dict["route"] = {
|
||||
"nodes": [
|
||||
{"id": "start", "type": "start"},
|
||||
{"id": "process", "type": "function", "function": "process_input"},
|
||||
{"id": "llm_node", "type": "llm", "agent": "test_agent"},
|
||||
{"id": "end", "type": "end"},
|
||||
],
|
||||
"edges": [
|
||||
{"from": "start", "to": "process"},
|
||||
{"from": "process", "to": "llm_node"},
|
||||
{"from": "llm_node", "to": "end"},
|
||||
],
|
||||
"entry_node": "start",
|
||||
"exit_nodes": ["end"],
|
||||
}
|
||||
context.config_dict["agents"] = {
|
||||
"test_agent": {"type": "llm", "config": {"provider": "openai"}},
|
||||
}
|
||||
|
||||
|
||||
@given('a valid actor config dict with type "llm" and provider "custom"')
|
||||
def step_config_llm_custom(context):
|
||||
context.config_dict = _build_base_config("llm")
|
||||
context.config_dict.update(
|
||||
{
|
||||
"provider": "custom",
|
||||
"model": "custom-model",
|
||||
"system_prompt": "You are helpful.",
|
||||
"temperature": 0.5,
|
||||
"max_tokens": 500,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@given('a valid actor config dict with type "tool" and config block tools')
|
||||
def step_config_tool_block(context):
|
||||
context.config_dict = _build_base_config("tool")
|
||||
context.config_dict["config"] = {"tools": ["echo", "math"]}
|
||||
|
||||
|
||||
@given("credentials dict with openai provider")
|
||||
def step_creds_openai(context):
|
||||
context.credentials = {
|
||||
"openai": {
|
||||
"api_key": "sk-test-key-12345",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@given("credentials dict with openai api key")
|
||||
def step_creds_openai_key(context):
|
||||
context.credentials = {"openai": {"api_key": "sk-test-executor-key"}}
|
||||
|
||||
|
||||
@given('credentials dict with openai api key "sk-test-injected"')
|
||||
def step_creds_openai_injected(context):
|
||||
context.credentials = {"openai": {"api_key": "sk-test-injected"}}
|
||||
|
||||
|
||||
@given("credentials dict with openai_compatible provider")
|
||||
def step_creds_compat(context):
|
||||
context.credentials = {
|
||||
"openai_compatible": {
|
||||
"api_key": "sk-compat-key",
|
||||
"base_url": "https://custom.api/v1",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@given("limits dict with max_depth {depth}")
|
||||
def step_limits(context, depth):
|
||||
context.limits = {"max_depth": int(depth)}
|
||||
|
||||
|
||||
@given("pricing dict with per_token_cost {cost}")
|
||||
def step_pricing(context, cost):
|
||||
context.pricing = {"per_token_cost": float(cost)}
|
||||
|
||||
|
||||
@given("a multi-actor config dict with multiple sub-actors")
|
||||
def step_multi_actor_config(context):
|
||||
context.config_dict = {
|
||||
"name": "test_multi_actor",
|
||||
"type": "multi_actor",
|
||||
"actors": {
|
||||
"default": {
|
||||
"type": "llm",
|
||||
"provider": "openai",
|
||||
"model": "gpt-3.5-turbo",
|
||||
"system_prompt": "You are helpful.",
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 1000,
|
||||
},
|
||||
"secondary": {
|
||||
"type": "tool",
|
||||
"tools": ["echo"],
|
||||
},
|
||||
},
|
||||
"cleveragents": {"default_actor": "default"},
|
||||
}
|
||||
|
||||
|
||||
@given("a multi-actor config dict with actors but no default_actor")
|
||||
def step_multi_actor_no_default(context):
|
||||
context.config_dict = {
|
||||
"name": "test_multi_no_default",
|
||||
"type": "multi_actor",
|
||||
"actors": {
|
||||
"first": {
|
||||
"type": "llm",
|
||||
"provider": "openai",
|
||||
"model": "gpt-3.5-turbo",
|
||||
"system_prompt": "You are helpful.",
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 1000,
|
||||
},
|
||||
},
|
||||
"cleveragents": {"default_actor": "nonexistent"},
|
||||
}
|
||||
|
||||
|
||||
@given("a multi-actor config dict with empty actors")
|
||||
def step_multi_actor_empty(context):
|
||||
context.config_dict = {
|
||||
"name": "test_empty_actors",
|
||||
"type": "multi_actor",
|
||||
"actors": {},
|
||||
"cleveragents": {"default_actor": "default"},
|
||||
}
|
||||
|
||||
|
||||
@given("a multi-actor config dict with llm agents block")
|
||||
def step_multi_actor_llm_agents(context):
|
||||
context.config_dict = {
|
||||
"name": "test_multi_llm_agents",
|
||||
"type": "multi_actor",
|
||||
"actors": {
|
||||
"agent1": {
|
||||
"type": "llm",
|
||||
"provider": "openai",
|
||||
"model": "gpt-3.5-turbo",
|
||||
"system_prompt": "You are helpful.",
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 1000,
|
||||
},
|
||||
},
|
||||
"agents": {
|
||||
"agent1": {
|
||||
"type": "llm",
|
||||
"provider": "openai",
|
||||
"config": {"provider": "openai"},
|
||||
},
|
||||
},
|
||||
"cleveragents": {"default_actor": "agent1"},
|
||||
}
|
||||
|
||||
|
||||
@given("a config dict with actors key but no type field")
|
||||
|
brent.edwards
commented
Here's the type field that's supposed to be missing. Here's the type field that's supposed to be missing.
|
||||
def step_config_actors_no_type(context):
|
||||
context.config_dict = {
|
||||
"name": "test_auto_multi",
|
||||
"actors": {
|
||||
"default": {
|
||||
"type": "llm",
|
||||
"provider": "openai",
|
||||
"model": "gpt-3.5-turbo",
|
||||
"system_prompt": "You are helpful.",
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 1000,
|
||||
},
|
||||
},
|
||||
"cleveragents": {"default_actor": "default"},
|
||||
}
|
||||
|
||||
|
||||
@given("a config dict with only llm config and no type or actors key")
|
||||
def step_config_llm_only(context):
|
||||
context.config_dict = {
|
||||
"name": "test_auto_llm",
|
||||
"provider": "openai",
|
||||
"model": "gpt-3.5-turbo",
|
||||
"system_prompt": "You are helpful.",
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 1000,
|
||||
}
|
||||
|
||||
|
||||
@given("the LLM agent is configured to fail")
|
||||
def step_llm_fail(context):
|
||||
context._llm_should_fail = True
|
||||
|
||||
|
||||
@given("a prompt string of {length} characters")
|
||||
def step_prompt_length(context, length):
|
||||
context.prompt = "x" * int(length)
|
||||
|
||||
|
||||
@given("a response string of {length} characters")
|
||||
def step_response_length(context, length):
|
||||
context.response = "y" * int(length)
|
||||
|
||||
|
||||
@given(
|
||||
"a NodeUsage with node_id {node_id} and provider {provider} and prompt_tokens {pt} and completion_tokens {ct}"
|
||||
)
|
||||
def step_node_usage(context, node_id, provider, pt, ct):
|
||||
node_id = node_id.strip('"')
|
||||
provider = provider.strip('"')
|
||||
context.node_usage = NodeUsage(
|
||||
node_id=node_id,
|
||||
provider=provider,
|
||||
model="test-model",
|
||||
prompt_tokens=int(pt),
|
||||
completion_tokens=int(ct),
|
||||
)
|
||||
|
||||
|
||||
@given(
|
||||
"an ActorResult with response {resp} and prompt_tokens {pt} and completion_tokens {ct}"
|
||||
)
|
||||
def step_actor_result(context, resp, pt, ct):
|
||||
resp = resp.strip('"')
|
||||
context.actor_result = ActorResult(
|
||||
response=resp, prompt_tokens=int(pt), completion_tokens=int(ct)
|
||||
)
|
||||
|
||||
|
||||
@when("I call create_executor")
|
||||
def step_call_create_executor(context):
|
||||
context.test_executor = create_executor(
|
||||
config_dict=context.config_dict,
|
||||
credentials=context.credentials,
|
||||
limits=context.limits if hasattr(context, "limits") else None,
|
||||
pricing=context.pricing if hasattr(context, "pricing") else None,
|
||||
)
|
||||
|
||||
|
||||
@when("I call create_executor with None limits and pricing")
|
||||
def step_call_create_executor_none(context):
|
||||
context.test_executor = create_executor(
|
||||
config_dict=context.config_dict,
|
||||
credentials=context.credentials,
|
||||
limits=None,
|
||||
pricing=None,
|
||||
)
|
||||
|
||||
|
||||
@when("I execute the actor with message {msg}")
|
||||
@async_run_until_complete
|
||||
async def step_execute_actor(context, msg):
|
||||
msg = msg.strip('"')
|
||||
executor = create_executor(
|
||||
config_dict=context.config_dict,
|
||||
credentials=context.credentials,
|
||||
limits=context.limits if hasattr(context, "limits") and context.limits else {},
|
||||
pricing=context.pricing
|
||||
if hasattr(context, "pricing") and context.pricing
|
||||
else {},
|
||||
)
|
||||
context.test_executor = executor
|
||||
|
||||
with (
|
||||
patch("cleveractors.agents.llm.LLMAgent") as mock_llm,
|
||||
patch("cleveractors.templates.renderer.TemplateRenderer") as mock_renderer,
|
||||
patch("cleveractors.agents.tool.ToolAgent") as mock_tool,
|
||||
patch("cleveractors.agents.factory.AgentFactory") as mock_factory,
|
||||
patch("cleveractors.langgraph.pure_graph.PureLangGraph") as mock_pure_graph,
|
||||
patch("cleveractors.runtime._estimate_tokens", return_value=(100, 50)),
|
||||
):
|
||||
mock_renderer_instance = MagicMock()
|
||||
mock_renderer.return_value = mock_renderer_instance
|
||||
|
||||
if getattr(context, "_llm_should_fail", False):
|
||||
mock_llm_instance = MagicMock()
|
||||
mock_llm_instance.process_message = AsyncMock(
|
||||
side_effect=Exception("LLM API error")
|
||||
)
|
||||
mock_llm.return_value = mock_llm_instance
|
||||
else:
|
||||
mock_llm_instance = MagicMock()
|
||||
mock_llm_instance.process_message = AsyncMock(
|
||||
return_value="Mock LLM response"
|
||||
)
|
||||
mock_llm.return_value = mock_llm_instance
|
||||
|
||||
mock_tool_instance = MagicMock()
|
||||
mock_tool_instance.process_message = AsyncMock(
|
||||
return_value="Mock tool response"
|
||||
)
|
||||
mock_tool.return_value = mock_tool_instance
|
||||
|
||||
mock_graph_instance = MagicMock()
|
||||
mock_graph_instance.execute = AsyncMock(return_value="Mock graph response")
|
||||
mock_pure_graph.return_value = mock_graph_instance
|
||||
|
||||
mock_factory_instance = MagicMock()
|
||||
mock_factory_instance.create_agent = MagicMock(return_value=mock_llm_instance)
|
||||
mock_factory.return_value = mock_factory_instance
|
||||
|
||||
try:
|
||||
context.test_result = await executor.execute(msg)
|
||||
context.test_error = None
|
||||
except Exception as e:
|
||||
context.test_error = e
|
||||
context.test_result = None
|
||||
|
||||
|
||||
@when("I call _build_factory_config on the executor")
|
||||
def step_call_build_factory(context):
|
||||
executor = create_executor(
|
||||
config_dict=context.config_dict,
|
||||
credentials=context.credentials,
|
||||
limits=context.limits if hasattr(context, "limits") and context.limits else {},
|
||||
pricing=context.pricing
|
||||
if hasattr(context, "pricing") and context.pricing
|
||||
else {},
|
||||
)
|
||||
context.test_executor = executor
|
||||
context.factory_config = executor._build_factory_config()
|
||||
|
||||
|
||||
@when("I estimate tokens for model {model}")
|
||||
def step_estimate_tokens(context, model):
|
||||
model = model.strip('"')
|
||||
context.prompt_tokens, context.completion_tokens = _estimate_tokens(
|
||||
context.prompt, context.response, model, "openai"
|
||||
)
|
||||
|
||||
|
||||
@when("I estimate tokens with tiktoken unavailable")
|
||||
def step_estimate_tokens_fallback(context):
|
||||
with (
|
||||
patch("cleveractors.runtime.tiktoken", None, create=True),
|
||||
patch("builtins.__import__", side_effect=ImportError) as _mock_import,
|
||||
):
|
||||
context.prompt_tokens, context.completion_tokens = _estimate_tokens(
|
||||
context.prompt, context.response, "unknown-model", "unknown"
|
||||
)
|
||||
|
||||
|
||||
@then("an Executor instance should be returned")
|
||||
def step_assert_executor(context):
|
||||
assert context.test_executor is not None
|
||||
assert isinstance(context.test_executor, Executor)
|
||||
|
||||
|
||||
@then("the executor config should match the config dict")
|
||||
def step_assert_config(context):
|
||||
assert context.test_executor.config == context.config_dict
|
||||
|
||||
|
||||
@then("the executor credentials should match the credentials dict")
|
||||
def step_assert_credentials(context):
|
||||
assert context.test_executor.credentials == context.credentials
|
||||
|
||||
|
||||
@then("the executor limits should match the limits dict")
|
||||
def step_assert_limits(context):
|
||||
assert context.test_executor.limits == context.limits
|
||||
|
||||
|
||||
@then("the executor pricing should match the pricing dict")
|
||||
def step_assert_pricing(context):
|
||||
assert context.test_executor.pricing == context.pricing
|
||||
|
||||
|
||||
@then("the executor limits should be an empty dictionary")
|
||||
def step_assert_limits_empty(context):
|
||||
assert context.test_executor.limits == {}
|
||||
|
||||
|
||||
@then("the executor pricing should be an empty dictionary")
|
||||
def step_assert_pricing_empty(context):
|
||||
assert context.test_executor.pricing == {}
|
||||
|
||||
|
||||
@then("the execution should return an ActorResult")
|
||||
|
brent.edwards
commented
This description doesn't match the code. "the expected reply" implies that there is only one reply. The code allows any reply that isn't empty. This description doesn't match the code. "the expected reply" implies that there is only one reply. The code allows any reply that isn't empty.
|
||||
def step_assert_actor_result(context):
|
||||
assert context.test_result is not None
|
||||
assert isinstance(context.test_result, ActorResult)
|
||||
|
||||
|
||||
@then("the ActorResult response should be non-empty")
|
||||
def step_assert_response(context):
|
||||
assert context.test_result.response is not None
|
||||
assert len(context.test_result.response) > 0
|
||||
|
||||
|
||||
@then("the ActorResult should have token usage tracked")
|
||||
def step_assert_tokens(context):
|
||||
assert context.test_result.prompt_tokens >= 0
|
||||
assert context.test_result.completion_tokens >= 0
|
||||
|
||||
|
||||
@then("the ActorResult should have at least one node usage entry")
|
||||
def step_assert_nodes(context):
|
||||
assert len(context.test_result.nodes) >= 1
|
||||
|
||||
|
||||
@then("the ActorResult should have zero prompt tokens for tool agents")
|
||||
def step_assert_zero_tokens(context):
|
||||
assert context.test_result.prompt_tokens == 0
|
||||
assert context.test_result.completion_tokens == 0
|
||||
|
brent.edwards
commented
This doesn't match the description. It should be
This doesn't match the description.
It should be
`assert node.node_id.startsWith("default.")`
|
||||
|
||||
|
||||
@then("the node usage IDs should be prefixed with the default actor name")
|
||||
def step_assert_node_prefix(context):
|
||||
for node in context.test_result.nodes:
|
||||
assert node.node_id.startswith("default.")
|
||||
|
||||
|
||||
@then("a ConfigurationError should be raised for runtime")
|
||||
def step_assert_config_error(context):
|
||||
assert context.test_error is not None
|
||||
assert isinstance(context.test_error, ConfigurationError)
|
||||
|
||||
|
||||
@then("the error message should mention the unknown actor type")
|
||||
def step_assert_error_unknown_type(context):
|
||||
assert "unknown_type" in str(context.test_error).lower()
|
||||
|
||||
|
||||
@then(
|
||||
"a ConfigurationError should be raised for runtime with the original exception chained"
|
||||
)
|
||||
def step_assert_chained_error(context):
|
||||
assert context.test_error is not None
|
||||
assert isinstance(context.test_error, ConfigurationError)
|
||||
|
||||
|
||||
@then("the PureGraphConfig should be built with correct nodes and edges")
|
||||
def step_assert_graph_config(context):
|
||||
assert context.test_result is not None
|
||||
|
brent.edwards
commented
Again -- the description and the assertion do not match. Check that the API key is in the LLM agent. Again -- the description and the assertion do not match. Check that the API key is in the LLM agent.
|
||||
assert isinstance(context.test_result, ActorResult)
|
||||
|
||||
|
||||
@then("the LLM agent should be initialized with the injected api key")
|
||||
def step_assert_injected_key(context):
|
||||
assert context.test_result is not None
|
||||
assert context.test_executor is not None
|
||||
assert context.test_executor.credentials is not None
|
||||
assert "openai" in context.test_executor.credentials
|
||||
assert context.test_executor.credentials["openai"]["api_key"] == "sk-test-injected"
|
||||
|
||||
|
||||
@then("a ConfigurationError should be raised for runtime about no actors")
|
||||
def step_assert_no_actors_error(context):
|
||||
assert context.test_error is not None
|
||||
assert isinstance(context.test_error, ConfigurationError)
|
||||
|
||||
|
||||
@then("the factory config should have credentials injected into the agents block")
|
||||
def step_assert_factory_creds(context):
|
||||
assert context.factory_config is not None
|
||||
assert "agents" in context.factory_config
|
||||
assert "context" in context.factory_config
|
||||
|
||||
|
||||
@then("the factory config context global should contain credentials and limits")
|
||||
def step_assert_factory_context(context):
|
||||
assert context.factory_config is not None
|
||||
assert "context" in context.factory_config
|
||||
assert "global" in context.factory_config["context"]
|
||||
assert "credentials" in context.factory_config["context"]["global"]
|
||||
assert "limits" in context.factory_config["context"]["global"]
|
||||
|
||||
|
||||
@then("the estimated prompt tokens should be greater than 0")
|
||||
def step_assert_prompt_tokens(context):
|
||||
assert context.prompt_tokens > 0
|
||||
|
||||
|
||||
@then("the estimated completion tokens should be greater than 0")
|
||||
def step_assert_completion_tokens(context):
|
||||
assert context.completion_tokens > 0
|
||||
|
||||
|
||||
@then("the estimated prompt tokens should be proportional to the prompt length")
|
||||
def step_assert_prompt_proportional(context):
|
||||
assert context.prompt_tokens > 0
|
||||
assert context.prompt_tokens >= len(context.prompt) // 4 - 1
|
||||
|
||||
|
||||
@then("the estimated completion tokens should be proportional to the response length")
|
||||
def step_assert_completion_proportional(context):
|
||||
assert context.completion_tokens > 0
|
||||
assert context.completion_tokens >= len(context.response) // 4 - 1
|
||||
|
||||
|
||||
@then("the estimated completion tokens should be 1 or more")
|
||||
def step_assert_completion_tokens_min(context):
|
||||
assert context.completion_tokens >= 1
|
||||
|
||||
|
||||
@then("the estimated prompt tokens should be 1 or more")
|
||||
def step_assert_prompt_tokens_min(context):
|
||||
assert context.prompt_tokens >= 1
|
||||
|
||||
|
||||
@then("the NodeUsage node_id should be {expected}")
|
||||
def step_assert_node_id(context, expected):
|
||||
expected = expected.strip('"')
|
||||
assert context.node_usage.node_id == expected
|
||||
|
||||
|
||||
@then("the NodeUsage provider should be {expected}")
|
||||
def step_assert_node_provider(context, expected):
|
||||
expected = expected.strip('"')
|
||||
assert context.node_usage.provider == expected
|
||||
|
||||
|
||||
@then("the NodeUsage prompt_tokens should be {expected}")
|
||||
def step_assert_node_pt(context, expected):
|
||||
assert context.node_usage.prompt_tokens == int(expected)
|
||||
|
||||
|
||||
@then("the NodeUsage completion_tokens should be {expected}")
|
||||
def step_assert_node_ct(context, expected):
|
||||
assert context.node_usage.completion_tokens == int(expected)
|
||||
|
||||
|
||||
@then("the ActorResult response should be {expected}")
|
||||
def step_assert_ar_response(context, expected):
|
||||
expected = expected.strip('"')
|
||||
assert context.actor_result.response == expected
|
||||
|
||||
|
||||
@then("the ActorResult prompt_tokens should be {expected}")
|
||||
def step_assert_ar_pt(context, expected):
|
||||
assert context.actor_result.prompt_tokens == int(expected)
|
||||
|
||||
|
||||
@then("the ActorResult completion_tokens should be {expected}")
|
||||
def step_assert_ar_ct(context, expected):
|
||||
assert context.actor_result.completion_tokens == int(expected)
|
||||
|
||||
|
||||
@then("the ActorResult nodes should be empty by default")
|
||||
def step_assert_ar_nodes_empty(context):
|
||||
assert context.actor_result.nodes == []
|
||||
Reference in New Issue
Block a user
The above config dict doesn't look valid for two reasons: