brent.edwards
3d2cd05f56
CI / lint (pull_request) Successful in 15s
CI / typecheck (pull_request) Successful in 28s
CI / quality (pull_request) Successful in 16s
CI / security (pull_request) Successful in 51s
CI / build (pull_request) Successful in 21s
CI / integration_tests (pull_request) Successful in 9m6s
CI / unit_tests (pull_request) Failing after 26m29s
CI / coverage (pull_request) Successful in 8m38s
CI / docker (pull_request) Has been skipped
364 lines
16 KiB
Gherkin
364 lines
16 KiB
Gherkin
Feature: Tool and Validation Domain Models
|
|
As a developer
|
|
I want tool and validation domain models for the tool registry
|
|
So that tools can be registered, validated, and managed via CLI
|
|
|
|
# ---- Tool Creation ----
|
|
|
|
Scenario: Create a custom tool with valid fields
|
|
When I create a tool with name "local/line-counter" source "custom" and code "print('hello')"
|
|
Then the tool model should be created
|
|
And the tool model name should be "local/line-counter"
|
|
And the tool model source should be "custom"
|
|
And the tool model type should be "tool"
|
|
And the tool model namespace should be "local"
|
|
And the tool model short name should be "line-counter"
|
|
|
|
Scenario: Create a builtin tool
|
|
When I create a tool with name "builtin/file-read" and source "builtin"
|
|
Then the tool model should be created
|
|
And the tool model source should be "builtin"
|
|
|
|
Scenario: Create an MCP tool with required fields
|
|
When I create an MCP tool with name "devops/docker-build" server "devops-mcp" and tool name "docker_build"
|
|
Then the tool model should be created
|
|
And the tool model source should be "mcp"
|
|
|
|
Scenario: Create an agent_skill tool
|
|
When I create an agent skill tool with name "skills/formatter" and path "/skills/formatter"
|
|
Then the tool model should be created
|
|
And the tool model source should be "agent_skill"
|
|
|
|
# ---- Tool Name Validation ----
|
|
|
|
Scenario: Tool name must match namespace/name pattern
|
|
When I try to create a tool with invalid name "no-namespace"
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "namespace/name"
|
|
|
|
Scenario: Tool name rejects spaces
|
|
When I try to create a tool with invalid name "local/bad name"
|
|
Then a tool model validation error should be raised
|
|
|
|
Scenario: Tool name rejects double slashes
|
|
When I try to create a tool with invalid name "local//bad"
|
|
Then a tool model validation error should be raised
|
|
|
|
Scenario: Tool name allows hyphens and underscores
|
|
When I create a tool with name "my-ns/my_tool" and source "builtin"
|
|
Then the tool model should be created
|
|
And the tool model namespace should be "my-ns"
|
|
And the tool model short name should be "my_tool"
|
|
|
|
# ---- Source-conditional field requirements ----
|
|
|
|
Scenario: Custom source requires code
|
|
When I try to create a tool with name "local/test" source "custom" without code
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "code is required"
|
|
|
|
Scenario: MCP source requires mcp_server
|
|
When I try to create an MCP tool without mcp_server
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "mcp_server is required"
|
|
|
|
Scenario: MCP source requires mcp_tool_name
|
|
When I try to create an MCP tool without mcp_tool_name
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "mcp_tool_name is required"
|
|
|
|
Scenario: Agent skill source requires agent_skill_path
|
|
When I try to create an agent skill tool without path
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "agent_skill_path is required"
|
|
|
|
# ---- Capability constraint enforcement ----
|
|
|
|
Scenario: Read-only capability enforces writes=false
|
|
When I try to create a tool capability with read_only true and writes true
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "read_only tool cannot have writes"
|
|
|
|
Scenario: Read-only capability enforces checkpointable=false
|
|
When I try to create a tool capability with read_only true and checkpointable true
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "read_only tool cannot have checkpointable"
|
|
|
|
Scenario: Non-read-only capability allows writes
|
|
When I create a tool capability with writes true
|
|
Then the tool capability should have writes true
|
|
|
|
# ---- ResourceSlot binding validation ----
|
|
|
|
Scenario: Static binding requires static_resource
|
|
When I try to create a resource slot with static binding and no static resource
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "static_resource is required"
|
|
|
|
Scenario: Non-static binding forbids static_resource
|
|
When I try to create a resource slot with contextual binding and static resource set
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "static_resource must not be set"
|
|
|
|
Scenario: Static binding with static_resource succeeds
|
|
When I create a resource slot with static binding and static resource "my-repo"
|
|
Then the tool resource slot should be created
|
|
|
|
Scenario: ResourceSlot name must be valid identifier
|
|
When I try to create a resource slot with invalid name "123bad"
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "valid Python identifier"
|
|
|
|
# ---- Validation model creation ----
|
|
|
|
Scenario: Create a custom validation with forced read-only
|
|
When I create a validation with name "qa/coverage-check" source "custom" and code "return True"
|
|
Then the tool model should be created
|
|
And the tool model type should be "validation"
|
|
And the tool validation capability should be read only
|
|
And the tool validation capability should not have writes
|
|
And the tool validation capability should not be checkpointable
|
|
|
|
Scenario: Validation forces tool_type to validation
|
|
When I create a validation with name "qa/test-check" source "builtin"
|
|
Then the tool model type should be "validation"
|
|
|
|
Scenario: Validation mode defaults to required
|
|
When I create a validation with name "qa/check" source "builtin"
|
|
Then the tool validation mode should be "required"
|
|
|
|
Scenario: Validation can be informational
|
|
When I create an informational validation with name "qa/info-check" source "builtin"
|
|
Then the tool validation mode should be "informational"
|
|
|
|
# ---- Validation wraps/transform requirements ----
|
|
|
|
Scenario: Wrapped validation requires transform
|
|
When I try to create a wrapped validation without transform
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "transform is required"
|
|
|
|
Scenario: Wrapped validation forbids code
|
|
When I try to create a wrapped validation with code
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "code must not be set"
|
|
|
|
Scenario: Wrapped validation requires source=wrapped
|
|
When I try to create a validation with wraps and wrong source
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "source must be 'wrapped'"
|
|
|
|
Scenario: argument_mapping only valid with wraps
|
|
When I try to create a validation with argument_mapping but no wraps
|
|
Then a tool model validation error should be raised
|
|
And the tool model error should mention "argument_mapping is only valid"
|
|
|
|
Scenario: Valid wrapped validation
|
|
When I create a valid wrapped validation
|
|
Then the tool model should be created
|
|
And the tool model type should be "validation"
|
|
And the tool validation wraps should be "devops/run-linter"
|
|
|
|
# ---- from_config loading ----
|
|
|
|
Scenario: Load tool from config dict
|
|
When I load a tool from config with name "local/my-tool" source "builtin"
|
|
Then the tool model should be created
|
|
And the tool model name should be "local/my-tool"
|
|
|
|
Scenario: Load tool from config missing name raises error
|
|
When I try to load a tool from config missing "name"
|
|
Then a tool model config error should be raised with "name"
|
|
|
|
Scenario: Load tool from config missing description raises error
|
|
When I try to load a tool from config missing "description"
|
|
Then a tool model config error should be raised with "description"
|
|
|
|
Scenario: Load tool from config missing source raises error
|
|
When I try to load a tool from config missing "source"
|
|
Then a tool model config error should be raised with "source"
|
|
|
|
Scenario: Load validation from config with wraps
|
|
When I load a validation from config with wraps "devops/linter"
|
|
Then the tool model should be created
|
|
And the tool model source should be "wrapped"
|
|
And the tool validation wraps should be "devops/linter"
|
|
|
|
Scenario: Load validation from config missing name raises error
|
|
When I try to load a validation config missing "name"
|
|
Then a tool model config error should be raised with "name"
|
|
|
|
# ---- as_cli_dict output stability ----
|
|
|
|
Scenario: Tool as_cli_dict has required keys
|
|
When I create a tool and call as_cli_dict
|
|
Then the tool cli dict should have key "name"
|
|
And the tool cli dict should have key "description"
|
|
And the tool cli dict should have key "source"
|
|
And the tool cli dict should have key "tool_type"
|
|
And the tool cli dict should have key "namespace"
|
|
And the tool cli dict should have key "short_name"
|
|
And the tool cli dict should have key "capability"
|
|
And the tool cli dict should have key "timeout"
|
|
|
|
Scenario: Validation as_cli_dict includes mode
|
|
When I create a validation and call as_cli_dict
|
|
Then the tool cli dict should have key "mode"
|
|
|
|
# ---- Tool timeout validation ----
|
|
|
|
Scenario: Tool timeout must be >= 1
|
|
When I try to create a tool with timeout 0
|
|
Then a tool model validation error should be raised
|
|
|
|
Scenario: Tool timeout default is 300
|
|
When I create a tool with name "local/default-timeout" and source "builtin"
|
|
Then the tool model timeout should be 300
|
|
|
|
# ---- ToolLifecycle ----
|
|
|
|
Scenario: ToolLifecycle accepts optional hooks
|
|
When I create a tool lifecycle with discover "my.discover"
|
|
Then the tool lifecycle discover should be "my.discover"
|
|
|
|
# ---- Enum values ----
|
|
|
|
Scenario: ToolSource enum has all expected values
|
|
Then the ToolSource enum should have values "mcp,agent_skill,builtin,custom,wrapped"
|
|
|
|
Scenario: ToolType enum has all expected values
|
|
Then the ToolType enum should have values "tool,validation"
|
|
|
|
Scenario: ValidationMode enum has all expected values
|
|
Then the ValidationMode enum should have values "required,informational"
|
|
|
|
Scenario: CheckpointScope enum has all expected values
|
|
Then the CheckpointScope enum should have values "file,transaction,snapshot,composite"
|
|
|
|
Scenario: BindingMode enum has all expected values
|
|
Then the BindingMode enum should have values "contextual,static,parameter"
|
|
|
|
Scenario: ResourceAccessMode enum has all expected values
|
|
Then the ResourceAccessMode enum should have values "read_only,read_write"
|
|
|
|
# ---- Tool with resource slots via from_config ----
|
|
|
|
Scenario: Load tool from config with resource slots
|
|
When I load a tool from config with resource slots
|
|
Then the tool model should be created
|
|
And the tool model should have 1 resource slot
|
|
|
|
# ---- Validation from_config missing description ----
|
|
|
|
Scenario: Load validation from config missing description raises error
|
|
When I try to load a validation config missing "description"
|
|
Then a tool model config error should be raised with "description"
|
|
|
|
# ---- Tool description validation ----
|
|
|
|
Scenario: Tool description must not be empty
|
|
When I try to create a tool with empty description
|
|
Then a tool model validation error should be raised
|
|
|
|
# ---- Invalid scenarios ----
|
|
|
|
Scenario: Tool with conflicting capability flags
|
|
When I try to create a tool capability with read_only true and writes true
|
|
Then a tool model validation error should be raised
|
|
|
|
Scenario: Validation wraps with custom source is rejected
|
|
When I try to create a validation with wraps and wrong source
|
|
Then a tool model validation error should be raised
|
|
|
|
# ---- as_cli_dict coverage for optional fields ----
|
|
|
|
Scenario: Tool as_cli_dict includes code when set
|
|
When I create a custom tool and call as_cli_dict
|
|
Then the tool cli dict should have key "code"
|
|
|
|
Scenario: Tool as_cli_dict includes mcp fields when set
|
|
When I create an MCP tool and call as_cli_dict
|
|
Then the tool cli dict should have key "mcp_server"
|
|
And the tool cli dict should have key "mcp_tool_name"
|
|
|
|
Scenario: Tool as_cli_dict includes agent_skill_path when set
|
|
When I create an agent skill tool and call as_cli_dict
|
|
Then the tool cli dict should have key "agent_skill_path"
|
|
|
|
Scenario: Tool as_cli_dict includes schemas when set
|
|
When I create a tool with schemas and call as_cli_dict
|
|
Then the tool cli dict should have key "input_schema"
|
|
And the tool cli dict should have key "output_schema"
|
|
|
|
Scenario: Tool as_cli_dict includes side_effects and cost_profile
|
|
When I create a tool with side effects and cost profile and call as_cli_dict
|
|
Then the tool cli dict capability should have key "side_effects"
|
|
And the tool cli dict capability should have key "cost_profile"
|
|
|
|
Scenario: Tool as_cli_dict includes resource slots when set
|
|
When I create a tool with resource slots and call as_cli_dict
|
|
Then the tool cli dict should have key "resource_slots"
|
|
|
|
# ---- Validation as_cli_dict wraps fields ----
|
|
|
|
Scenario: Validation as_cli_dict includes wraps and transform
|
|
When I create a wrapped validation and call as_cli_dict
|
|
Then the tool cli dict should have key "wraps"
|
|
And the tool cli dict should have key "transform"
|
|
And the tool cli dict should have key "argument_mapping"
|
|
|
|
# ---- Validation from_config without wraps ----
|
|
|
|
Scenario: Load validation from config without wraps needs source
|
|
When I try to load a validation config without source or wraps
|
|
Then a tool model config error should be raised with "source"
|
|
|
|
Scenario: Load validation from config with explicit source
|
|
When I load a validation from config with source "builtin"
|
|
Then the tool model should be created
|
|
And the tool model source should be "builtin"
|
|
|
|
# ---- Validation forces read_only on writable capability ----
|
|
|
|
Scenario: Validation forces read_only on writable capability
|
|
When I create a validation with writes capability
|
|
Then the tool validation capability should be read only
|
|
And the tool validation capability should not have writes
|
|
And the tool validation capability should not be checkpointable
|
|
|
|
# ---- YAML loader output alignment (mirrors Robot smoke suite) ----
|
|
|
|
Scenario: Load custom tool from example YAML config
|
|
When I load a tool from YAML file "examples/tools/custom-tool.yaml"
|
|
Then the tool model should be created
|
|
And the tool model name should be "local/line-counter"
|
|
And the tool model source should be "custom"
|
|
And the tool model should have 1 resource slot
|
|
And the tool model timeout should be 60
|
|
|
|
Scenario: Load MCP tool from example YAML config
|
|
When I load a tool from YAML file "examples/tools/mcp-tool.yaml"
|
|
Then the tool model should be created
|
|
And the tool model name should be "devops/docker-build"
|
|
And the tool model source should be "mcp"
|
|
And the tool model timeout should be 600
|
|
|
|
Scenario: Load wrapped validation from example YAML config
|
|
When I load a validation from YAML file "examples/validations/wrapped-validation.yaml"
|
|
Then the tool model should be created
|
|
And the tool model type should be "validation"
|
|
And the tool model source should be "wrapped"
|
|
And the tool validation wraps should be "devops/run-linter"
|
|
And the tool validation mode should be "required"
|
|
And the tool validation capability should be read only
|
|
And the tool validation capability should not have writes
|
|
|
|
Scenario: Load required validation from example YAML config
|
|
When I load a validation from YAML file "examples/validations/required-validation.yaml"
|
|
Then the tool model should be created
|
|
And the tool model type should be "validation"
|
|
And the tool model source should be "custom"
|
|
And the tool validation mode should be "required"
|
|
And the tool validation capability should be read only
|
|
And the tool validation capability should not have writes
|