86 lines
3.5 KiB
Gherkin
86 lines
3.5 KiB
Gherkin
Feature: Inline Tool Executor
|
|
As a developer
|
|
I want to execute inline tool code with safety constraints
|
|
So that skills can run embedded Python code safely and reliably
|
|
|
|
# ---- Successful Execution ----
|
|
|
|
Scenario: Execute simple inline tool successfully
|
|
Given an inline tool with code that prints "hello inline"
|
|
And a writable skill context with sandbox "/tmp/inline-sandbox"
|
|
When I execute the inline tool
|
|
Then the inline result should be successful
|
|
And the inline result output should contain "hello inline"
|
|
|
|
Scenario: Inline tool returns output data
|
|
Given an inline tool with code that prints "result: 42"
|
|
And a writable skill context with sandbox "/tmp/inline-sandbox"
|
|
When I execute the inline tool
|
|
Then the inline result should be successful
|
|
And the inline result output should contain "result: 42"
|
|
|
|
# ---- Timeout ----
|
|
|
|
Scenario: Inline tool times out and returns structured error
|
|
Given an inline tool with code that sleeps for 5 seconds
|
|
And a writable skill context with sandbox "/tmp/inline-sandbox"
|
|
And an inline executor with max runtime 0.1 seconds
|
|
When I execute the inline tool
|
|
Then the inline result should not be successful
|
|
And the inline result error message should contain "timed out"
|
|
|
|
# ---- Output Truncation ----
|
|
|
|
Scenario: Inline tool output exceeds max size and is truncated
|
|
Given an inline tool with code that prints 2000 bytes of output
|
|
And a writable skill context with sandbox "/tmp/inline-sandbox"
|
|
And an inline executor with max output 100 bytes
|
|
When I execute the inline tool
|
|
Then the inline result should be successful
|
|
And the inline result should be truncated
|
|
|
|
# ---- Write Guard ----
|
|
|
|
Scenario: Read-only context blocks inline tool with writes capability
|
|
Given an inline tool with writes capability and code that prints "write op"
|
|
And a read-only skill context with sandbox "/tmp/inline-sandbox"
|
|
When I execute the inline tool
|
|
Then the inline result should not be successful
|
|
And the inline result error message should contain "denied"
|
|
|
|
Scenario: Writable context allows inline tool with writes capability
|
|
Given an inline tool with writes capability and code that prints "write op"
|
|
And a writable skill context with sandbox "/tmp/inline-sandbox"
|
|
When I execute the inline tool
|
|
Then the inline result should be successful
|
|
|
|
# ---- Validation ----
|
|
|
|
Scenario: Validate inline tool with missing code
|
|
Given an inline tool with no code
|
|
When I validate the inline tool
|
|
Then the inline validation errors should contain "non-empty code"
|
|
|
|
Scenario: Validate inline tool with unsupported language
|
|
Given an inline tool with unsupported source type
|
|
When I validate the inline tool
|
|
Then the inline validation errors should contain "Unsupported source"
|
|
|
|
# ---- Change Tracking ----
|
|
|
|
Scenario: Tool invocation is recorded in change tracker
|
|
Given an inline tool with code that prints "tracked"
|
|
And a writable skill context with sandbox "/tmp/inline-sandbox"
|
|
When I execute the inline tool
|
|
Then the skill context change tracker should have 1 inline records
|
|
And the last inline tracker record tool_name should be "inline_tool"
|
|
|
|
# ---- Error Handling ----
|
|
|
|
Scenario: Inline tool execution error returns structured result
|
|
Given an inline tool with code that raises an exception
|
|
And a writable skill context with sandbox "/tmp/inline-sandbox"
|
|
When I execute the inline tool
|
|
Then the inline result should not be successful
|
|
And the inline result error message should contain "intentional error"
|