forked from cleveragents/cleveragents-core
325 lines
14 KiB
Gherkin
325 lines
14 KiB
Gherkin
Feature: Tool Call Router
|
|
As a developer integrating multiple LLM providers
|
|
I want a tool call router that normalizes provider-specific formats
|
|
So that tools execute uniformly regardless of the calling provider
|
|
|
|
# ---- Provider Format Detection ----
|
|
|
|
Scenario: Detect OpenAI format from arguments string
|
|
Given a tool call payload with name "test/echo" and arguments '{"key": "value"}'
|
|
When I detect the provider format
|
|
Then the detected format should be "openai"
|
|
|
|
Scenario: Detect Anthropic format from input dict
|
|
Given a tool call payload with name "test/echo" and input {"key": "value"}
|
|
When I detect the provider format
|
|
Then the detected format should be "anthropic"
|
|
|
|
Scenario: Detect LangChain format from type tool_call
|
|
Given a tool call payload with name "test/echo" and type "tool_call" and args {"key": "value"}
|
|
When I detect the provider format
|
|
Then the detected format should be "langchain"
|
|
|
|
Scenario: Detect LangChain format from args key
|
|
Given a tool call payload with name "test/echo" and args key only
|
|
When I detect the provider format
|
|
Then the detected format should be "langchain"
|
|
|
|
Scenario: Detect unknown format from empty payload
|
|
Given an empty tool call payload
|
|
When I detect the provider format
|
|
Then the detected format should be "unknown"
|
|
|
|
Scenario: Detect OpenAI format with dict arguments
|
|
Given a tool call payload with name "test/echo" and dict arguments {"a": 1}
|
|
When I detect the provider format
|
|
Then the detected format should be "openai"
|
|
|
|
# ---- Payload Normalization ----
|
|
|
|
Scenario: Normalize OpenAI payload
|
|
Given a tool call payload with name "test/echo" and arguments '{"key": "value"}'
|
|
When I normalize the tool call
|
|
Then the normalized request tool_name should be "test/echo"
|
|
And the normalized request arguments should have key "key"
|
|
And the normalized request provider_format should be "openai"
|
|
|
|
Scenario: Normalize Anthropic payload
|
|
Given a tool call payload with name "test/echo" and input {"msg": "hello"}
|
|
When I normalize the tool call
|
|
Then the normalized request tool_name should be "test/echo"
|
|
And the normalized request arguments should have key "msg"
|
|
And the normalized request provider_format should be "anthropic"
|
|
|
|
Scenario: Normalize LangChain payload
|
|
Given a tool call payload with name "test/echo" and type "tool_call" and args {"data": 42}
|
|
When I normalize the tool call
|
|
Then the normalized request tool_name should be "test/echo"
|
|
And the normalized request arguments should have key "data"
|
|
And the normalized request provider_format should be "langchain"
|
|
|
|
Scenario: Normalize payload with invalid JSON arguments raises error
|
|
Given a tool call with invalid JSON arguments
|
|
When I try to normalize the tool call
|
|
Then a router ValueError should be raised containing "parse error"
|
|
|
|
Scenario: Normalize payload without name raises error
|
|
Given a tool call payload without a name
|
|
When I try to normalize the tool call
|
|
Then a router ValueError should be raised containing "name"
|
|
|
|
Scenario: Normalize non-dict payload raises error
|
|
When I try to normalize a non-dict payload
|
|
Then a router ValueError should be raised containing "dict"
|
|
|
|
Scenario: Normalize unknown format with dict arguments
|
|
Given a tool call payload with name "test/echo" and parameters key {"x": 1}
|
|
When I normalize the tool call
|
|
Then the normalized request tool_name should be "test/echo"
|
|
And the normalized request arguments should have key "x"
|
|
|
|
# ---- Stable ID Generation ----
|
|
|
|
Scenario: Generate deterministic tool call ID
|
|
When I generate a tool call ID for plan "plan-001" sequence 0
|
|
Then the tool call ID should start with "tc_"
|
|
And the tool call ID should have length 27
|
|
|
|
Scenario: Same inputs produce same ID
|
|
When I generate a tool call ID for plan "plan-001" sequence 5
|
|
And I generate another tool call ID for plan "plan-001" sequence 5
|
|
Then both tool call IDs should be identical
|
|
|
|
Scenario: Different inputs produce different IDs
|
|
When I generate a tool call ID for plan "plan-001" sequence 0
|
|
And I generate another tool call ID for plan "plan-001" sequence 1
|
|
Then the tool call IDs should differ
|
|
|
|
Scenario: Empty plan_id raises ValueError
|
|
When I try to generate a tool call ID with empty plan_id
|
|
Then a router ValueError should be raised containing "plan_id"
|
|
|
|
Scenario: Negative sequence raises ValueError
|
|
When I try to generate a tool call ID with negative sequence
|
|
Then a router ValueError should be raised containing "sequence"
|
|
|
|
# ---- Router Execution ----
|
|
|
|
Scenario: Route OpenAI tool call to echo tool
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-001"
|
|
And an OpenAI-format tool call for "test/echo" with arguments '{"msg": "hi"}'
|
|
When I route the tool call
|
|
Then the normalized result should be successful
|
|
And the normalized result tool_name should be "test/echo"
|
|
And the normalized result provider_format should be "openai"
|
|
And the normalized result tool_call_id should start with "tc_"
|
|
|
|
Scenario: Route Anthropic tool call to echo tool
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-002"
|
|
And an Anthropic-format tool call for "test/echo" with input {"msg": "hi"}
|
|
When I route the tool call
|
|
Then the normalized result should be successful
|
|
And the normalized result provider_format should be "anthropic"
|
|
|
|
Scenario: Route LangChain tool call to echo tool
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-003"
|
|
And a LangChain-format tool call for "test/echo" with args {"msg": "hi"}
|
|
When I route the tool call
|
|
Then the normalized result should be successful
|
|
And the normalized result provider_format should be "langchain"
|
|
|
|
Scenario: Route tool call for missing tool
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-004"
|
|
And an OpenAI-format tool call for "test/missing" with arguments '{}'
|
|
When I route the tool call
|
|
Then the normalized result should not be successful
|
|
And the normalized result error_category should be "not_found"
|
|
|
|
Scenario: Route tool call with invalid payload
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-005"
|
|
And a tool call payload without a name
|
|
When I route the tool call
|
|
Then the normalized result should not be successful
|
|
And the normalized result error_category should be "parse"
|
|
|
|
Scenario: Route with provider metadata
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-006"
|
|
And an OpenAI-format tool call for "test/echo" with arguments '{"msg": "hi"}'
|
|
And provider metadata with model "gpt-4" and provider_id "openai"
|
|
When I route the tool call with provider metadata
|
|
Then the normalized result provider_metadata should contain key "model"
|
|
|
|
# ---- Batch Routing ----
|
|
|
|
Scenario: Route batch of tool calls
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-010"
|
|
And a batch of 3 OpenAI-format tool calls for "test/echo"
|
|
When I route the batch
|
|
Then the batch should return 3 results
|
|
And all batch results should be successful
|
|
|
|
Scenario: Route empty batch
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-011"
|
|
When I route an empty batch
|
|
Then the batch should return 0 results
|
|
|
|
# ---- Streaming Execution ----
|
|
|
|
Scenario: Route streaming tool call emits updates
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-020"
|
|
And an OpenAI-format tool call for "test/echo" with arguments '{"msg": "stream"}'
|
|
When I route the tool call with streaming
|
|
Then the stream should emit a pending update
|
|
And the stream should emit a running update
|
|
And the stream should emit a complete update
|
|
And the stream should emit a final result
|
|
|
|
Scenario: Route streaming tool call for failing tool
|
|
Given a tool registry with a failing tool "test/fail"
|
|
And a tool call router for plan "plan-021"
|
|
And an OpenAI-format tool call for "test/fail" with arguments '{}'
|
|
When I route the tool call with streaming
|
|
Then the stream should emit a complete update
|
|
And the stream should emit a final result that is not successful
|
|
|
|
Scenario: Route streaming with invalid payload
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-022"
|
|
And a tool call payload without a name
|
|
When I route the tool call with streaming
|
|
Then the stream should emit a final result that is not successful
|
|
|
|
# ---- Validation Tool Surfacing ----
|
|
|
|
Scenario: Route call to validation tool surfaces pass result
|
|
Given a tool registry with a passing validation tool "test/validator"
|
|
And a tool call router for plan "plan-030"
|
|
And an OpenAI-format tool call for "test/validator" with arguments '{}'
|
|
When I route the tool call
|
|
Then the normalized result is_validation should be True
|
|
And the normalized result validation_passed should be True
|
|
|
|
Scenario: Route call to validation tool surfaces fail result
|
|
Given a tool registry with a failing validation tool "test/validator-fail"
|
|
And a tool call router for plan "plan-031"
|
|
And an OpenAI-format tool call for "test/validator-fail" with arguments '{}'
|
|
When I route the tool call
|
|
Then the normalized result is_validation should be True
|
|
And the normalized result validation_passed should be False
|
|
|
|
# ---- Error Classification ----
|
|
|
|
Scenario: Classify timeout error
|
|
When I classify the error "Operation timed out after 30s"
|
|
Then the error category should be "timeout"
|
|
|
|
Scenario: Classify permission error
|
|
When I classify the error "Permission denied: cannot write"
|
|
Then the error category should be "permission"
|
|
|
|
Scenario: Classify not found error
|
|
When I classify the error "Tool 'x/y' not found"
|
|
Then the error category should be "not_found"
|
|
|
|
Scenario: Classify resource error
|
|
When I classify the error "Insufficient memory for operation"
|
|
Then the error category should be "resource"
|
|
|
|
Scenario: Classify schema error
|
|
When I classify the error "Schema validation failed"
|
|
Then the error category should be "schema"
|
|
|
|
Scenario: Classify parse error
|
|
When I classify the error "Failed to parse response"
|
|
Then the error category should be "parse"
|
|
|
|
Scenario: Classify generic execution error
|
|
When I classify the error "Something unexpected happened"
|
|
Then the error category should be "execution"
|
|
|
|
Scenario: Classify empty error
|
|
When I classify an empty error message
|
|
Then the error category should be "unknown"
|
|
|
|
# ---- Schema Normalization ----
|
|
|
|
Scenario: Normalize schema for OpenAI provider
|
|
Given a tool spec named "test/echo" with description "Echo tool"
|
|
When I normalize the schema for "openai" provider
|
|
Then the normalized schema should have key "parameters"
|
|
And the normalized schema should have name "test/echo"
|
|
|
|
Scenario: Normalize schema for Anthropic provider
|
|
Given a tool spec named "test/echo" with description "Echo tool"
|
|
When I normalize the schema for "anthropic" provider
|
|
Then the normalized schema should have key "input_schema"
|
|
|
|
Scenario: Normalize schema for LangChain provider
|
|
Given a tool spec named "test/echo" with description "Echo tool"
|
|
When I normalize the schema for "langchain" provider
|
|
Then the normalized schema should have key "args_schema"
|
|
|
|
Scenario: Normalize schema truncates long description
|
|
Given a tool spec named "test/echo" with a very long description
|
|
When I normalize the schema for "openai" provider with max length 50
|
|
Then the normalized schema description should be at most 50 characters
|
|
|
|
Scenario: Normalize schema with invalid max length raises error
|
|
Given a tool spec named "test/echo" with description "Echo tool"
|
|
When I try to normalize the schema with max_description_length 0
|
|
Then a router ValueError should be raised containing "max_description_length"
|
|
|
|
# ---- Schema Export ----
|
|
|
|
Scenario: Export schemas for OpenAI provider
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-040"
|
|
When I export schemas for "openai" provider
|
|
Then the exported schemas should contain 1 schema
|
|
And the first exported schema should have tool_type "tool"
|
|
|
|
# ---- Sequence Counter ----
|
|
|
|
Scenario: Router sequence increments on each route
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-050"
|
|
And an OpenAI-format tool call for "test/echo" with arguments '{}'
|
|
When I route the tool call
|
|
Then the router sequence should be 1
|
|
When I route the tool call
|
|
Then the router sequence should be 2
|
|
|
|
# ---- Router Construction Validation ----
|
|
|
|
Scenario: Router with empty plan_id raises ValueError
|
|
Given a tool registry with an echo tool "test/echo"
|
|
When I try to create a router with empty plan_id
|
|
Then a router ValueError should be raised containing "plan_id"
|
|
|
|
Scenario: Non-dict payload raises ValueError on route
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-060"
|
|
When I try to route a non-dict payload
|
|
Then a router ValueError should be raised containing "dict"
|
|
|
|
Scenario: Non-list payloads raises ValueError on route_batch
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-061"
|
|
When I try to route_batch a non-list payload
|
|
Then a router ValueError should be raised containing "list"
|
|
|
|
# ---- Detect format for non-dict ----
|
|
|
|
Scenario: Detect format for non-dict returns unknown
|
|
When I detect format for a non-dict value
|
|
Then the detected format should be "unknown"
|