freemo
7c2f5a1c76
CI / lint (pull_request) Successful in 35s
CI / typecheck (pull_request) Successful in 58s
CI / security (pull_request) Successful in 1m0s
CI / quality (pull_request) Successful in 39s
CI / build (pull_request) Successful in 24s
CI / helm (pull_request) Successful in 23s
CI / unit_tests (pull_request) Successful in 7m4s
CI / e2e_tests (pull_request) Successful in 17m25s
CI / integration_tests (pull_request) Successful in 22m49s
CI / docker (pull_request) Successful in 1m29s
CI / coverage (pull_request) Successful in 11m6s
CI / status-check (pull_request) Successful in 1s
CI / benchmark-publish (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Successful in 57m30s
- What was implemented
- Updated MCPToolResult.data docstring to document the normalisation behaviour, clarifying that MCP 1.4.0 list-format content is normalised to a dict with a "content" key.
- Updated MCPToolAdapter.invoke() success path to normalise MCP 1.4.0 list-format content: when content is a list (as per MCP 1.4.0 spec), it is wrapped as {"content": [...]} so MCPToolResult.data is always dict[str, Any].
- Updated MockMCPTransport.call() to return MCP 1.4.0-compliant list-format content: invoke_results are now wrapped as [{"type": "text", "text": json.dumps(payload)}] instead of the non-standard dict format.
- Added json import to mock_mcp_transport.py to support JSON payload encoding.
- Updated existing mcp_adapter.feature scenario "Invoke a discovered tool successfully" to check for "content" key instead of "id" (since mock now returns MCP 1.4.0 list format).
- Added two new Behave scenarios: "Invoke tool returns MCP 1.4.0 list-format content normalised to dict" and "Invoke tool with MCP 1.4.0 content list stores content items".
- Added corresponding step definitions for the new scenarios.
- All 14,418 existing scenarios continue to pass; 2 new scenarios added; typecheck passes with 0 errors; lint passes.
- Why this approach
- Design decision: Normalize to dict (Option B) to maintain a consistent MCPToolResult.data type of dict[str, Any] and avoid breaking downstream code that accesses result.data["key"].
- Fallback path handles non-standard server responses gracefully, ensuring robustness when servers deviate from MCP 1.4.0 spec.
- Technical approach and affected components
- Core: MCPToolResult data handling and MCPToolAdapter.invoke() logic
- Mocks: mock_mcp_transport.py updated to emit MCP 1.4.0 list-format content
- Tests: updated mcp_adapter.feature expectations; added two Behave scenarios with new step definitions
- Dependencies: added import json to mock_mcp_transport.py
- Verification
- Comprehensive test suites: 14,418 existing scenarios pass
- 2 new scenarios added and pass
- Typechecking: 0 errors
- Linting: passes
ISSUES CLOSED: #2743
300 lines
13 KiB
Gherkin
300 lines
13 KiB
Gherkin
Feature: MCP Tool Adapter
|
|
As an actor runtime
|
|
I want an adapter that connects to MCP servers, discovers tools,
|
|
and invokes them with schema validation
|
|
So that external MCP tools are usable like any registered tool
|
|
|
|
# -------------------------------------------------------------------
|
|
# Adapter Lifecycle
|
|
# -------------------------------------------------------------------
|
|
|
|
Scenario: Create adapter from server config
|
|
Given an MCP server config for "github" with stdio transport
|
|
When I create an MCP adapter from the config
|
|
Then the adapter server name should be "github"
|
|
And the adapter transport should be "stdio"
|
|
And the adapter should not be connected
|
|
|
|
Scenario: Create adapter from SSE server config
|
|
Given an MCP server config for "remote-api" with sse transport and url "https://api.example.com/mcp"
|
|
When I create an MCP adapter from the config
|
|
Then the adapter server name should be "remote-api"
|
|
And the adapter transport should be "sse"
|
|
|
|
Scenario: Connect adapter starts handshake
|
|
Given an MCP adapter with a mock transport
|
|
When I connect the adapter
|
|
Then the adapter should be connected
|
|
And the adapter server capabilities should be set
|
|
|
|
Scenario: Disconnect adapter cleans up
|
|
Given a connected MCP adapter with a mock transport
|
|
When I disconnect the adapter
|
|
Then the adapter should not be connected
|
|
|
|
Scenario: Double connect is idempotent
|
|
Given a connected MCP adapter with a mock transport
|
|
When I connect the adapter
|
|
Then the adapter should be connected
|
|
|
|
Scenario: Disconnect when not connected is safe
|
|
Given an MCP adapter with a mock transport
|
|
When I disconnect the adapter
|
|
Then the adapter should not be connected
|
|
|
|
# -------------------------------------------------------------------
|
|
# Tool Discovery
|
|
# -------------------------------------------------------------------
|
|
|
|
Scenario: Discover tools from MCP server
|
|
Given a connected MCP adapter with 3 mock tools
|
|
When I discover tools from the adapter
|
|
Then the adapter should have 3 discovered tools
|
|
And discovered tool 0 should have a name
|
|
And discovered tool 0 should have an input schema
|
|
|
|
Scenario: Discover tools with include filter
|
|
Given a connected MCP adapter with 3 mock tools
|
|
And a tool filter including only "tool_0"
|
|
When I discover tools from the adapter with filter
|
|
Then the adapter should have 1 discovered tools
|
|
|
|
Scenario: Discover tools with exclude filter
|
|
Given a connected MCP adapter with 3 mock tools
|
|
And a tool filter excluding "tool_0"
|
|
When I discover tools from the adapter with filter
|
|
Then the adapter should have 2 discovered tools
|
|
|
|
Scenario: Discover returns empty when server has no tools
|
|
Given a connected MCP adapter with 0 mock tools
|
|
When I discover tools from the adapter
|
|
Then the adapter should have 0 discovered tools
|
|
|
|
Scenario: Discover when not connected raises error
|
|
Given an MCP adapter with a mock transport
|
|
When I discover tools expecting an error
|
|
Then the adapter error should mention "not connected"
|
|
|
|
# -------------------------------------------------------------------
|
|
# Tool Invocation
|
|
# -------------------------------------------------------------------
|
|
|
|
Scenario: Invoke a discovered tool successfully
|
|
Given a connected MCP adapter with a callable mock tool "create_issue"
|
|
When I invoke "create_issue" with arguments {"title": "Bug", "body": "Fix it"}
|
|
Then the invocation should succeed
|
|
And the invocation result should contain key "content"
|
|
|
|
Scenario: Invoke with valid input schema passes validation
|
|
Given a connected MCP adapter with a schema-validated mock tool "create_issue"
|
|
When I invoke "create_issue" with arguments {"title": "Bug", "body": "Fix it"}
|
|
Then the invocation should succeed
|
|
|
|
Scenario: Invoke with invalid input fails validation
|
|
Given a connected MCP adapter with a schema-validated mock tool "create_issue"
|
|
When I invoke "create_issue" with arguments {"wrong_field": 123}
|
|
Then the invocation should fail
|
|
And the invocation error should mention "validation"
|
|
|
|
Scenario: Invoke unknown tool raises error
|
|
Given a connected MCP adapter with a callable mock tool "create_issue"
|
|
When I invoke "nonexistent_tool" with arguments {}
|
|
Then the invocation should fail
|
|
And the invocation error should mention "not found"
|
|
|
|
Scenario: Invoke when not connected raises error
|
|
Given an MCP adapter with a mock transport
|
|
When I invoke MCP tool "create_issue" while disconnected
|
|
Then the adapter error should mention "not connected"
|
|
|
|
Scenario: Invoke tool that returns error from server
|
|
Given a connected MCP adapter with a failing mock tool "bad_tool"
|
|
When I invoke "bad_tool" with arguments {}
|
|
Then the invocation should fail
|
|
And the invocation error should mention "server error"
|
|
|
|
# -------------------------------------------------------------------
|
|
# Tool Registration in ToolRegistry
|
|
# -------------------------------------------------------------------
|
|
|
|
Scenario: Register discovered MCP tools in ToolRegistry
|
|
Given a connected MCP adapter with 2 mock tools
|
|
And an empty MCP tool registry
|
|
When I register MCP tools in the registry with namespace "mcp-github"
|
|
Then the registry should have 2 tools
|
|
And registry tool 0 name should start with "mcp-github/"
|
|
|
|
Scenario: Registered MCP tool is callable via registry
|
|
Given a connected MCP adapter with a callable mock tool "list_repos"
|
|
And an empty MCP tool registry
|
|
When I register MCP tools in the registry with namespace "mcp-github"
|
|
Then calling registry tool "mcp-github/list_repos" should succeed
|
|
|
|
Scenario: Re-registration after tool list changed
|
|
Given a connected MCP adapter with 2 mock tools
|
|
And an empty MCP tool registry
|
|
When I register MCP tools in the registry with namespace "mcp-github"
|
|
And the MCP server adds a new tool "deploy"
|
|
And I re-register MCP tools in the registry with namespace "mcp-github"
|
|
Then the registry should have 3 tools
|
|
|
|
Scenario: Registered tool handler returns error on failed invoke
|
|
Given a connected MCP adapter with a failing mock tool "bad_tool"
|
|
And an empty MCP tool registry
|
|
When I register MCP tools in the registry with namespace "mcp-fail"
|
|
Then calling registry tool "mcp-fail/bad_tool" should return an error
|
|
|
|
# -------------------------------------------------------------------
|
|
# Capability Inference
|
|
# -------------------------------------------------------------------
|
|
|
|
Scenario: Infer read_only for tool named list_repos
|
|
When I infer capabilities for tool name "list_repos"
|
|
Then the inferred capabilities should have read_only true
|
|
And the inferred capabilities should have writes false
|
|
|
|
Scenario: Infer writes for tool named create_issue
|
|
When I infer capabilities for tool name "create_issue"
|
|
Then the inferred capabilities should have read_only false
|
|
And the inferred capabilities should have writes true
|
|
|
|
Scenario: Infer writes for tool named delete_branch
|
|
When I infer capabilities for tool name "delete_branch"
|
|
Then the inferred capabilities should have read_only false
|
|
And the inferred capabilities should have writes true
|
|
|
|
Scenario: Infer default for unknown tool name
|
|
When I infer capabilities for tool name "run_pipeline"
|
|
Then the inferred capabilities should have read_only false
|
|
And the inferred capabilities should have writes false
|
|
|
|
Scenario: Infer read_only for tool with get prefix
|
|
When I infer capabilities for tool name "get-user-details"
|
|
Then the inferred capabilities should have read_only true
|
|
And the inferred capabilities should have writes false
|
|
|
|
Scenario: Write overrides read when both keywords present
|
|
When I infer capabilities for tool name "search_and_update"
|
|
Then the inferred capabilities should have writes true
|
|
|
|
Scenario: Registered MCP tool has inferred capabilities
|
|
Given a connected MCP adapter with a callable mock tool "list_repos"
|
|
And an empty MCP tool registry
|
|
When I register MCP tools in the registry with namespace "mcp-github"
|
|
Then registry tool "mcp-github/list_repos" should have read_only capability
|
|
|
|
Scenario: Registered write tool has writes capability
|
|
Given a connected MCP adapter with a callable mock tool "create_issue"
|
|
And an empty MCP tool registry
|
|
When I register MCP tools in the registry with namespace "mcp-github"
|
|
Then registry tool "mcp-github/create_issue" should have writes capability
|
|
|
|
# -------------------------------------------------------------------
|
|
# Edge Cases
|
|
# -------------------------------------------------------------------
|
|
|
|
Scenario: Access discovered_tools property after discovery
|
|
Given a connected MCP adapter with 2 mock tools
|
|
When I discover tools from the adapter
|
|
Then the adapter discovered_tools property should have 2 items
|
|
|
|
Scenario: Disconnect handles close errors gracefully
|
|
Given a connected MCP adapter with a transport that errors on close
|
|
When I disconnect the adapter
|
|
Then the adapter should not be connected
|
|
|
|
Scenario: Invoke tool that raises generic exception
|
|
Given a connected MCP adapter with a transport-error mock tool "crash_tool"
|
|
When I invoke "crash_tool" with arguments {}
|
|
Then the invocation should fail
|
|
And the invocation error should mention "server error"
|
|
|
|
# -------------------------------------------------------------------
|
|
# MCP 1.4.0 Content List Format
|
|
# -------------------------------------------------------------------
|
|
|
|
Scenario: Invoke tool returns MCP 1.4.0 list-format content normalised to dict
|
|
Given a connected MCP adapter with a callable mock tool "get_status"
|
|
When I invoke "get_status" with arguments {}
|
|
Then the invocation should succeed
|
|
And the invocation result data should have key "content"
|
|
And the invocation result content should be a list
|
|
|
|
Scenario: Invoke tool with MCP 1.4.0 content list stores content items
|
|
Given a connected MCP adapter with a raw list-content mock tool "fetch_data"
|
|
When I invoke "fetch_data" with arguments {}
|
|
Then the invocation should succeed
|
|
And the invocation result data should have key "content"
|
|
And the invocation result content list should have 1 item
|
|
|
|
# -------------------------------------------------------------------
|
|
# Error Classification
|
|
# -------------------------------------------------------------------
|
|
|
|
Scenario: Classify connection failure
|
|
Given an MCP adapter with a transport that fails to connect
|
|
When I connect the adapter expecting error
|
|
Then the adapter error should mention "connection"
|
|
|
|
Scenario: Classify timeout error
|
|
Given a connected MCP adapter with a tool that times out
|
|
When I invoke "slow_tool" with arguments {}
|
|
Then the invocation should fail
|
|
And the invocation error should mention "timeout"
|
|
|
|
# -------------------------------------------------------------------
|
|
# Server Config Validation
|
|
# -------------------------------------------------------------------
|
|
|
|
Scenario: Stdio transport requires command
|
|
Given an MCP server config for "bad" with stdio transport but no command
|
|
When I create an MCP adapter from the config expecting validation error
|
|
Then the adapter error should mention "command"
|
|
|
|
Scenario: SSE transport requires url
|
|
Given an MCP server config for "bad" with sse transport but no url
|
|
When I create an MCP adapter from the config expecting validation error
|
|
Then the adapter error should mention "url"
|
|
|
|
# -------------------------------------------------------------------
|
|
# Reconnect Lifecycle
|
|
# -------------------------------------------------------------------
|
|
|
|
Scenario: Reconnect restores connection after disconnect
|
|
Given a connected MCP adapter with a mock transport
|
|
When I disconnect the adapter
|
|
And I reconnect the adapter
|
|
Then the adapter should be connected
|
|
|
|
Scenario: Reconnect clears tool cache
|
|
Given a connected MCP adapter with 3 mock tools
|
|
When I discover tools from the adapter
|
|
And I reconnect the adapter
|
|
Then the adapter discovered_tools property should have 0 items
|
|
|
|
Scenario: Reconnect on unavailable server raises connection error
|
|
Given a connected MCP adapter with a transport that fails to reconnect
|
|
When I reconnect the adapter expecting error
|
|
Then the adapter error should mention "connection"
|
|
|
|
Scenario: Connect times out when transport hangs
|
|
Given an MCP adapter with a transport that hangs on connect
|
|
When I connect the adapter with timeout 0.05 expecting error
|
|
Then the adapter error should mention "connection"
|
|
|
|
# -------------------------------------------------------------------
|
|
# Source and Checkpointable Metadata
|
|
# -------------------------------------------------------------------
|
|
|
|
Scenario: Registered MCP tool has source mcp
|
|
Given a connected MCP adapter with a callable mock tool "list_repos"
|
|
And an empty MCP tool registry
|
|
When I register MCP tools in the registry with namespace "mcp-github"
|
|
Then registry tool "mcp-github/list_repos" should have source "mcp"
|
|
|
|
Scenario: Registered MCP tool has checkpointable false
|
|
Given a connected MCP adapter with a callable mock tool "create_issue"
|
|
And an empty MCP tool registry
|
|
When I register MCP tools in the registry with namespace "mcp-github"
|
|
Then registry tool "mcp-github/create_issue" should have checkpointable false
|