freemo
2980b14e7b
Add ~60 new behave scenarios across output rendering, tool router, file ops, and skill search features targeting uncovered lines and branches. Key additions: - ElementHandle validation guards (empty id, type, negative index, None args) - Handle close/context-manager edge cases - Table list-row and batch-row coverage - Color/box-draw/JSON/YAML materializer edge cases - Format selection paths (detect capabilities, explicit flag, empty format) - Session ColumnDef, double-close guard, force-close open handles - Tool router schema export and provider format scenarios - File ops edge cases and search stat-failure/glob-include tests All nox checks pass: lint, typecheck (0 errors), unit_tests (4730 scenarios), coverage_report (97.0% >= 97% threshold).
449 lines
19 KiB
Gherkin
449 lines
19 KiB
Gherkin
Feature: Tool Call Router
|
|
As a developer integrating multiple LLM providers
|
|
I want a tool call router that normalizes provider-specific formats
|
|
So that tools execute uniformly regardless of the calling provider
|
|
|
|
# ---- Provider Format Detection ----
|
|
|
|
Scenario: Detect OpenAI format from arguments string
|
|
Given a tool call payload with name "test/echo" and arguments '{"key": "value"}'
|
|
When I detect the provider format
|
|
Then the detected format should be "openai"
|
|
|
|
Scenario: Detect Anthropic format from input dict
|
|
Given a tool call payload with name "test/echo" and input {"key": "value"}
|
|
When I detect the provider format
|
|
Then the detected format should be "anthropic"
|
|
|
|
Scenario: Detect LangChain format from type tool_call
|
|
Given a tool call payload with name "test/echo" and type "tool_call" and args {"key": "value"}
|
|
When I detect the provider format
|
|
Then the detected format should be "langchain"
|
|
|
|
Scenario: Detect LangChain format from args key
|
|
Given a tool call payload with name "test/echo" and args key only
|
|
When I detect the provider format
|
|
Then the detected format should be "langchain"
|
|
|
|
Scenario: Detect unknown format from empty payload
|
|
Given an empty tool call payload
|
|
When I detect the provider format
|
|
Then the detected format should be "unknown"
|
|
|
|
Scenario: Detect OpenAI format with dict arguments
|
|
Given a tool call payload with name "test/echo" and dict arguments {"a": 1}
|
|
When I detect the provider format
|
|
Then the detected format should be "openai"
|
|
|
|
# ---- Payload Normalization ----
|
|
|
|
Scenario: Normalize OpenAI payload
|
|
Given a tool call payload with name "test/echo" and arguments '{"key": "value"}'
|
|
When I normalize the tool call
|
|
Then the normalized request tool_name should be "test/echo"
|
|
And the normalized request arguments should have key "key"
|
|
And the normalized request provider_format should be "openai"
|
|
|
|
Scenario: Normalize Anthropic payload
|
|
Given a tool call payload with name "test/echo" and input {"msg": "hello"}
|
|
When I normalize the tool call
|
|
Then the normalized request tool_name should be "test/echo"
|
|
And the normalized request arguments should have key "msg"
|
|
And the normalized request provider_format should be "anthropic"
|
|
|
|
Scenario: Normalize LangChain payload
|
|
Given a tool call payload with name "test/echo" and type "tool_call" and args {"data": 42}
|
|
When I normalize the tool call
|
|
Then the normalized request tool_name should be "test/echo"
|
|
And the normalized request arguments should have key "data"
|
|
And the normalized request provider_format should be "langchain"
|
|
|
|
Scenario: Normalize payload with invalid JSON arguments raises error
|
|
Given a tool call with invalid JSON arguments
|
|
When I try to normalize the tool call
|
|
Then a router ValueError should be raised containing "parse error"
|
|
|
|
Scenario: Normalize payload without name raises error
|
|
Given a tool call payload without a name
|
|
When I try to normalize the tool call
|
|
Then a router ValueError should be raised containing "name"
|
|
|
|
Scenario: Normalize non-dict payload raises error
|
|
When I try to normalize a non-dict payload
|
|
Then a router ValueError should be raised containing "dict"
|
|
|
|
Scenario: Normalize unknown format with dict arguments
|
|
Given a tool call payload with name "test/echo" and parameters key {"x": 1}
|
|
When I normalize the tool call
|
|
Then the normalized request tool_name should be "test/echo"
|
|
And the normalized request arguments should have key "x"
|
|
|
|
# ---- Stable ID Generation ----
|
|
|
|
Scenario: Generate deterministic tool call ID
|
|
When I generate a tool call ID for plan "plan-001" sequence 0
|
|
Then the tool call ID should start with "tc_"
|
|
And the tool call ID should have length 27
|
|
|
|
Scenario: Same inputs produce same ID
|
|
When I generate a tool call ID for plan "plan-001" sequence 5
|
|
And I generate another tool call ID for plan "plan-001" sequence 5
|
|
Then both tool call IDs should be identical
|
|
|
|
Scenario: Different inputs produce different IDs
|
|
When I generate a tool call ID for plan "plan-001" sequence 0
|
|
And I generate another tool call ID for plan "plan-001" sequence 1
|
|
Then the tool call IDs should differ
|
|
|
|
Scenario: Empty plan_id raises ValueError
|
|
When I try to generate a tool call ID with empty plan_id
|
|
Then a router ValueError should be raised containing "plan_id"
|
|
|
|
Scenario: Negative sequence raises ValueError
|
|
When I try to generate a tool call ID with negative sequence
|
|
Then a router ValueError should be raised containing "sequence"
|
|
|
|
# ---- Router Execution ----
|
|
|
|
Scenario: Route OpenAI tool call to echo tool
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-001"
|
|
And an OpenAI-format tool call for "test/echo" with arguments '{"msg": "hi"}'
|
|
When I route the tool call
|
|
Then the normalized result should be successful
|
|
And the normalized result tool_name should be "test/echo"
|
|
And the normalized result provider_format should be "openai"
|
|
And the normalized result tool_call_id should start with "tc_"
|
|
|
|
Scenario: Route Anthropic tool call to echo tool
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-002"
|
|
And an Anthropic-format tool call for "test/echo" with input {"msg": "hi"}
|
|
When I route the tool call
|
|
Then the normalized result should be successful
|
|
And the normalized result provider_format should be "anthropic"
|
|
|
|
Scenario: Route LangChain tool call to echo tool
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-003"
|
|
And a LangChain-format tool call for "test/echo" with args {"msg": "hi"}
|
|
When I route the tool call
|
|
Then the normalized result should be successful
|
|
And the normalized result provider_format should be "langchain"
|
|
|
|
Scenario: Route tool call for missing tool
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-004"
|
|
And an OpenAI-format tool call for "test/missing" with arguments '{}'
|
|
When I route the tool call
|
|
Then the normalized result should not be successful
|
|
And the normalized result error_category should be "not_found"
|
|
|
|
Scenario: Route tool call with invalid payload
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-005"
|
|
And a tool call payload without a name
|
|
When I route the tool call
|
|
Then the normalized result should not be successful
|
|
And the normalized result error_category should be "parse"
|
|
|
|
Scenario: Route with provider metadata
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-006"
|
|
And an OpenAI-format tool call for "test/echo" with arguments '{"msg": "hi"}'
|
|
And provider metadata with model "gpt-4" and provider_id "openai"
|
|
When I route the tool call with provider metadata
|
|
Then the normalized result provider_metadata should contain key "model"
|
|
|
|
# ---- Batch Routing ----
|
|
|
|
Scenario: Route batch of tool calls
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-010"
|
|
And a batch of 3 OpenAI-format tool calls for "test/echo"
|
|
When I route the batch
|
|
Then the batch should return 3 results
|
|
And all batch results should be successful
|
|
|
|
Scenario: Route empty batch
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-011"
|
|
When I route an empty batch
|
|
Then the batch should return 0 results
|
|
|
|
# ---- Streaming Execution ----
|
|
|
|
Scenario: Route streaming tool call emits updates
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-020"
|
|
And an OpenAI-format tool call for "test/echo" with arguments '{"msg": "stream"}'
|
|
When I route the tool call with streaming
|
|
Then the stream should emit a pending update
|
|
And the stream should emit a running update
|
|
And the stream should emit a complete update
|
|
And the stream should emit a final result
|
|
|
|
Scenario: Route streaming tool call for failing tool
|
|
Given a tool registry with a failing tool "test/fail"
|
|
And a tool call router for plan "plan-021"
|
|
And an OpenAI-format tool call for "test/fail" with arguments '{}'
|
|
When I route the tool call with streaming
|
|
Then the stream should emit a complete update
|
|
And the stream should emit a final result that is not successful
|
|
|
|
Scenario: Route streaming with invalid payload
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-022"
|
|
And a tool call payload without a name
|
|
When I route the tool call with streaming
|
|
Then the stream should emit a final result that is not successful
|
|
|
|
# ---- Validation Tool Surfacing ----
|
|
|
|
Scenario: Route call to validation tool surfaces pass result
|
|
Given a tool registry with a passing validation tool "test/validator"
|
|
And a tool call router for plan "plan-030"
|
|
And an OpenAI-format tool call for "test/validator" with arguments '{}'
|
|
When I route the tool call
|
|
Then the normalized result is_validation should be True
|
|
And the normalized result validation_passed should be True
|
|
|
|
Scenario: Route call to validation tool surfaces fail result
|
|
Given a tool registry with a failing validation tool "test/validator-fail"
|
|
And a tool call router for plan "plan-031"
|
|
And an OpenAI-format tool call for "test/validator-fail" with arguments '{}'
|
|
When I route the tool call
|
|
Then the normalized result is_validation should be True
|
|
And the normalized result validation_passed should be False
|
|
|
|
# ---- Error Classification ----
|
|
|
|
Scenario: Classify timeout error
|
|
When I classify the error "Operation timed out after 30s"
|
|
Then the error category should be "timeout"
|
|
|
|
Scenario: Classify permission error
|
|
When I classify the error "Permission denied: cannot write"
|
|
Then the error category should be "permission"
|
|
|
|
Scenario: Classify not found error
|
|
When I classify the error "Tool 'x/y' not found"
|
|
Then the error category should be "not_found"
|
|
|
|
Scenario: Classify resource error
|
|
When I classify the error "Insufficient memory for operation"
|
|
Then the error category should be "resource"
|
|
|
|
Scenario: Classify schema error
|
|
When I classify the error "Schema validation failed"
|
|
Then the error category should be "schema"
|
|
|
|
Scenario: Classify parse error
|
|
When I classify the error "Failed to parse response"
|
|
Then the error category should be "parse"
|
|
|
|
Scenario: Classify generic execution error
|
|
When I classify the error "Something unexpected happened"
|
|
Then the error category should be "execution"
|
|
|
|
Scenario: Classify empty error
|
|
When I classify an empty error message
|
|
Then the error category should be "unknown"
|
|
|
|
# ---- Schema Normalization ----
|
|
|
|
Scenario: Normalize schema for OpenAI provider
|
|
Given a tool spec named "test/echo" with description "Echo tool"
|
|
When I normalize the schema for "openai" provider
|
|
Then the normalized schema should have key "parameters"
|
|
And the normalized schema should have name "test/echo"
|
|
|
|
Scenario: Normalize schema for Anthropic provider
|
|
Given a tool spec named "test/echo" with description "Echo tool"
|
|
When I normalize the schema for "anthropic" provider
|
|
Then the normalized schema should have key "input_schema"
|
|
|
|
Scenario: Normalize schema for LangChain provider
|
|
Given a tool spec named "test/echo" with description "Echo tool"
|
|
When I normalize the schema for "langchain" provider
|
|
Then the normalized schema should have key "args_schema"
|
|
|
|
Scenario: Normalize schema truncates long description
|
|
Given a tool spec named "test/echo" with a very long description
|
|
When I normalize the schema for "openai" provider with max length 50
|
|
Then the normalized schema description should be at most 50 characters
|
|
|
|
Scenario: Normalize schema with invalid max length raises error
|
|
Given a tool spec named "test/echo" with description "Echo tool"
|
|
When I try to normalize the schema with max_description_length 0
|
|
Then a router ValueError should be raised containing "max_description_length"
|
|
|
|
# ---- Schema Export ----
|
|
|
|
Scenario: Export schemas for OpenAI provider
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-040"
|
|
When I export schemas for "openai" provider
|
|
Then the exported schemas should contain 1 schema
|
|
And the first exported schema should have tool_type "tool"
|
|
|
|
# ---- Sequence Counter ----
|
|
|
|
Scenario: Router sequence increments on each route
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-050"
|
|
And an OpenAI-format tool call for "test/echo" with arguments '{}'
|
|
When I route the tool call
|
|
Then the router sequence should be 1
|
|
When I route the tool call
|
|
Then the router sequence should be 2
|
|
|
|
# ---- Router Construction Validation ----
|
|
|
|
Scenario: Router with empty plan_id raises ValueError
|
|
Given a tool registry with an echo tool "test/echo"
|
|
When I try to create a router with empty plan_id
|
|
Then a router ValueError should be raised containing "plan_id"
|
|
|
|
Scenario: Non-dict payload raises ValueError on route
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-060"
|
|
When I try to route a non-dict payload
|
|
Then a router ValueError should be raised containing "dict"
|
|
|
|
Scenario: Non-list payloads raises ValueError on route_batch
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-061"
|
|
When I try to route_batch a non-list payload
|
|
Then a router ValueError should be raised containing "list"
|
|
|
|
# ---- Detect format for non-dict ----
|
|
|
|
Scenario: Detect format for non-dict returns unknown
|
|
When I detect format for a non-dict value
|
|
Then the detected format should be "unknown"
|
|
|
|
# ---- Coverage: normalize with dict args, Anthropic/LangChain branches ----
|
|
|
|
Scenario: Normalize OpenAI payload with dict arguments
|
|
Given a tool call payload with name "test/echo" and dict arguments
|
|
When I normalize the tool call for "openai" format
|
|
Then the normalized request should have tool name "test/echo"
|
|
And the normalized arguments should be a dict with key "msg"
|
|
|
|
Scenario: Normalize LangChain payload with dict args
|
|
Given a LangChain payload with dict args
|
|
When I normalize the tool call for "langchain" format
|
|
Then the normalized request should have tool name "test/echo"
|
|
|
|
Scenario: Normalize LangChain payload with non-dict args raises ValueError
|
|
Given a LangChain payload with list args
|
|
When I try to normalize the invalid langchain args
|
|
Then a router ValueError should be raised containing "must be a dict"
|
|
|
|
Scenario: Normalize unknown payload with parameters key
|
|
Given an unknown format payload with parameters key
|
|
When I normalize the tool call for "unknown" format
|
|
Then the normalized arguments should have key "query"
|
|
|
|
Scenario: Normalize unknown payload with JSON string args
|
|
Given an unknown format payload with JSON string in args key
|
|
When I normalize the tool call for "unknown" format
|
|
Then the normalized arguments should have key "data"
|
|
|
|
Scenario: Normalize unknown payload with bad JSON skips to next key
|
|
Given an unknown format payload with bad JSON and valid fallback
|
|
When I normalize the tool call for "unknown" format
|
|
Then the normalized arguments should have key "ok"
|
|
|
|
# ---- Coverage: schema normalization LangChain and unknown ----
|
|
|
|
Scenario: Schema normalization for LangChain uses args_schema
|
|
Given a tool spec named "test/echo"
|
|
When I normalize the schema for "langchain" provider
|
|
Then the schema should have key "args_schema"
|
|
|
|
Scenario: Schema normalization for unknown uses input_schema
|
|
Given a tool spec named "test/echo"
|
|
When I normalize the schema for "unknown" provider
|
|
Then the schema should have key "input_schema"
|
|
|
|
# ---- Coverage: plan_id property ----
|
|
|
|
Scenario: Router plan_id property returns correct value
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-prop-test"
|
|
Then the router plan_id should be "plan-prop-test"
|
|
|
|
# ---- Coverage: route generic exception, validation surfacing, error classification ----
|
|
|
|
Scenario: Route catches generic RuntimeError during execution
|
|
Given a tool registry with a tool that raises RuntimeError
|
|
And a tool call router for plan "plan-runtime-err"
|
|
When I route an OpenAI payload for the error tool
|
|
Then the route result should have success false
|
|
And the route result error should contain "boom"
|
|
|
|
Scenario: Route returns error classification for failed ToolResult
|
|
Given a tool registry with a tool that returns failed result
|
|
And a tool call router for plan "plan-fail-result"
|
|
When I route an OpenAI payload for the fail tool
|
|
Then the route result should have success false
|
|
And the route result should have error_category set
|
|
|
|
Scenario: Route validates validation tool with passed result
|
|
Given a tool registry with a validation tool
|
|
And a tool call router for plan "plan-valid-check"
|
|
When I route an OpenAI payload for the validation tool
|
|
Then the route result should have is_validation true
|
|
And the route result should have validation_passed true
|
|
|
|
# ---- Coverage: streaming exception and validation ----
|
|
|
|
Scenario: Streaming route catches exception and yields ERROR
|
|
Given a tool registry with a tool that raises RuntimeError
|
|
And a tool call router for plan "plan-stream-err"
|
|
When I stream-route an OpenAI payload for the error tool
|
|
Then the streaming updates should include ERROR status
|
|
And the streaming final result should have success false
|
|
|
|
Scenario: Streaming route non-dict payload raises ValueError
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-stream-ndict"
|
|
When I try to stream-route a non-dict payload
|
|
Then a router ValueError should be raised containing "payload must be a dict"
|
|
|
|
Scenario: Streaming route validates validation tool result
|
|
Given a tool registry with a validation tool
|
|
And a tool call router for plan "plan-stream-valid"
|
|
When I stream-route an OpenAI payload for the validation tool
|
|
Then the streaming final result should have is_validation true
|
|
|
|
# ---- Coverage: get_tool_schemas validation annotation ----
|
|
|
|
Scenario: Export schemas annotates validation tools
|
|
Given a tool registry with a validation tool
|
|
And a tool call router for plan "plan-schema-export"
|
|
When I export tool schemas for "openai"
|
|
Then the exported schemas should include a tool with type "validation"
|
|
|
|
Scenario: Export schemas annotates regular tools
|
|
Given a tool registry with an echo tool "test/echo"
|
|
And a tool call router for plan "plan-schema-regular"
|
|
When I export tool schemas for "openai"
|
|
Then the exported schemas should include a tool with type "tool"
|
|
|
|
# ---- Coverage: _get_validation_mode ----
|
|
|
|
Scenario: Validation mode extracted from output_schema
|
|
Given a tool registry with a validation tool with mode "strict"
|
|
And a tool call router for plan "plan-vmode"
|
|
When I export tool schemas for "openai"
|
|
Then the exported validation schema should have mode "strict"
|
|
|
|
Scenario: Validation mode returns None for non-string mode
|
|
Given a tool registry with a validation tool with numeric mode
|
|
And a tool call router for plan "plan-vmode-num"
|
|
When I export tool schemas for "openai"
|
|
Then the exported validation schema should not have a mode
|