feat(validation): implement tool wrapping runtime (wraps + transform delegation)
CI / lint (pull_request) Successful in 14s
CI / typecheck (pull_request) Successful in 34s
CI / security (pull_request) Successful in 29s
CI / quality (pull_request) Successful in 15s
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 14s
CI / unit_tests (pull_request) Successful in 2m4s
CI / integration_tests (pull_request) Successful in 2m50s
CI / coverage (pull_request) Successful in 4m15s
CI / docker (pull_request) Successful in 42s
CI / lint (push) Successful in 12s
CI / typecheck (push) Successful in 31s
CI / quality (push) Successful in 16s
CI / security (push) Successful in 31s
CI / build (push) Successful in 16s
CI / unit_tests (push) Successful in 3m23s
CI / benchmark-regression (push) Has been skipped
CI / docker (push) Successful in 42s
CI / integration_tests (push) Successful in 4m11s
CI / coverage (push) Successful in 4m8s
CI / benchmark-publish (push) Successful in 16m5s
CI / benchmark-regression (pull_request) Successful in 29m59s
CI / lint (pull_request) Successful in 14s
CI / typecheck (pull_request) Successful in 34s
CI / security (pull_request) Successful in 29s
CI / quality (pull_request) Successful in 15s
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 14s
CI / unit_tests (pull_request) Successful in 2m4s
CI / integration_tests (pull_request) Successful in 2m50s
CI / coverage (pull_request) Successful in 4m15s
CI / docker (pull_request) Successful in 42s
CI / lint (push) Successful in 12s
CI / typecheck (push) Successful in 31s
CI / quality (push) Successful in 16s
CI / security (push) Successful in 31s
CI / build (push) Successful in 16s
CI / unit_tests (push) Successful in 3m23s
CI / benchmark-regression (push) Has been skipped
CI / docker (push) Successful in 42s
CI / integration_tests (push) Successful in 4m11s
CI / coverage (push) Successful in 4m8s
CI / benchmark-publish (push) Successful in 16m5s
CI / benchmark-regression (pull_request) Successful in 29m59s
Implement the runtime execution engine for validation tool wrapping, as specified in docs/specification.md § Tool Wrapping. WrappedToolExecutor resolves wraps references and delegates execution to wrapped tools, supporting composable wrapping chains with cycle detection and depth limiting (max 10 levels). ArgumentMapper translates arguments between wrapper and wrapped tool schemas using the argument_mapping configuration. Supports both forwarded parameter names and literal fixed values. TransformExecutor runs user-supplied transform functions in a sandboxed Python environment with restricted builtins (no imports, no filesystem, no network access). Validates that transforms return proper validation-format dicts with a passed boolean. Wired into the tool package public API via tool/__init__.py exports. All new error types (WrappedToolNotFoundError, WrappingCycleError, WrappingDepthExceededError, TransformExecutionError) provide clear diagnostic messages. Tests: 20 Behave scenarios covering argument mapping, transform execution, simple/chained delegation, error handling, and sandbox restrictions. 8 Robot Framework integration smoke tests. ASV benchmarks for delegation overhead measurement. ISSUES CLOSED: #543
This commit was merged in pull request #555.
This commit is contained in:
@@ -0,0 +1,220 @@
|
||||
Feature: Tool wrapping runtime
|
||||
As the validation execution engine
|
||||
I need to delegate execution to wrapped tools via wraps + transform
|
||||
So that validations can reuse existing tool implementations
|
||||
|
||||
# ── ArgumentMapper ────────────────────────────────────────────────
|
||||
|
||||
Scenario: ArgumentMapper with None mapping passes arguments through
|
||||
Given an argument mapper with no mapping
|
||||
When I apply the mapper to inputs {"path": "/src", "verbose": true}
|
||||
Then the mapped arguments should be {"path": "/src", "verbose": true}
|
||||
|
||||
Scenario: ArgumentMapper with mapping translates argument names
|
||||
Given an argument mapper with mapping {"test_directory": "source_dir", "coverage_enabled": true}
|
||||
When I apply the mapper to inputs {"source_dir": "/tests"}
|
||||
Then the mapped arguments should be {"test_directory": "/tests", "coverage_enabled": true}
|
||||
|
||||
Scenario: ArgumentMapper with literal values injects fixed values
|
||||
Given an argument mapper with mapping {"mode": "strict", "count": 42}
|
||||
When I apply the mapper to inputs {"extra": "ignored"}
|
||||
Then the mapped arguments should be {"mode": "strict", "count": 42}
|
||||
|
||||
Scenario: ArgumentMapper rejects non-dict inputs
|
||||
Given an argument mapper with no mapping
|
||||
When I try to apply the mapper to non-dict input "not_a_dict"
|
||||
Then a TypeError should be raised from the argument mapper
|
||||
|
||||
Scenario: ArgumentMapper rejects non-dict mapping at construction
|
||||
When I try to create an argument mapper with a non-dict mapping
|
||||
Then a TypeError should be raised from argument mapper construction
|
||||
|
||||
# ── TransformExecutor ─────────────────────────────────────────────
|
||||
|
||||
Scenario: TransformExecutor runs a valid transform function
|
||||
Given a transform executor with code that checks returncode equals zero
|
||||
When I execute the transform with tool output {"returncode": 0, "tests_run": 10}
|
||||
Then the transform result should have passed true
|
||||
And the transform result should have message "All tests passed"
|
||||
|
||||
Scenario: TransformExecutor handles failing transform output
|
||||
Given a transform executor with code that checks returncode equals zero
|
||||
When I execute the transform with tool output {"returncode": 1, "tests_run": 10}
|
||||
Then the transform result should have passed false
|
||||
|
||||
Scenario: TransformExecutor raises on missing transform function
|
||||
Given a transform executor with code that does not define a transform function
|
||||
When I try to execute the transform with any output
|
||||
Then a TransformExecutionError should be raised with message containing "callable"
|
||||
|
||||
Scenario: TransformExecutor raises on non-dict return value
|
||||
Given a transform executor with code that returns a non-dict value
|
||||
When I try to execute the transform with any output
|
||||
Then a TransformExecutionError should be raised with message containing "dict"
|
||||
|
||||
Scenario: TransformExecutor raises on missing passed key in result
|
||||
Given a transform executor with code that returns a dict without passed key
|
||||
When I try to execute the transform with any output
|
||||
Then a TransformExecutionError should be raised with message containing "passed"
|
||||
|
||||
Scenario: TransformExecutor raises on empty transform code
|
||||
When I try to create a transform executor with empty code
|
||||
Then a ValueError should be raised from transform construction
|
||||
|
||||
Scenario: TransformExecutor sandboxes dangerous operations
|
||||
Given a transform executor with code that attempts to import os
|
||||
When I try to execute the transform with any output
|
||||
Then a TransformExecutionError should be raised from sandbox restriction
|
||||
|
||||
# ── WrappedToolExecutor ───────────────────────────────────────────
|
||||
|
||||
Scenario: Simple wrapping delegates to the wrapped tool
|
||||
Given a tool registry with a tool "local/run-tests" that returns {"returncode": 0}
|
||||
And a validation "local/tests-pass" that wraps "local/run-tests" with a simple transform
|
||||
And a wrapped tool executor using the test registry
|
||||
When I execute the wrapping validation with inputs {"path": "/src"}
|
||||
Then the wrapped execution should succeed with passed true
|
||||
And the wrapped tool "local/run-tests" should have been called
|
||||
|
||||
Scenario: Argument mapping translates arguments to the wrapped tool
|
||||
Given a tool registry with a tool "local/run-tests" that returns {"returncode": 0}
|
||||
And a validation "local/mapped-check" that wraps "local/run-tests" with argument mapping {"test_directory": "source_dir", "coverage_enabled": true}
|
||||
And a wrapped tool executor using the test registry
|
||||
When I execute the wrapping validation with inputs {"source_dir": "/tests"}
|
||||
Then the wrapped tool "local/run-tests" should have received argument "test_directory" with value "/tests"
|
||||
And the wrapped tool "local/run-tests" should have received argument "coverage_enabled" with value true
|
||||
|
||||
Scenario: Chained wrapping delegates through the chain
|
||||
Given a tool registry with a tool "local/base-tool" that returns {"status": "ok"}
|
||||
And a validation "local/mid-wrapper" that wraps "local/base-tool" with a passthrough transform
|
||||
And a validation "local/outer-wrapper" that wraps "local/mid-wrapper" with a status transform
|
||||
And a wrapped tool executor using the test registry
|
||||
When I execute the outer wrapping validation with inputs {}
|
||||
Then the wrapped execution should succeed with passed true
|
||||
And the wrapped tool "local/base-tool" should have been called
|
||||
|
||||
Scenario: Missing wrapped tool raises WrappedToolNotFoundError
|
||||
Given a validation "local/broken-wrap" that wraps "local/nonexistent" with a simple transform
|
||||
And a wrapped tool executor with empty registry
|
||||
When I try to execute the wrapping validation with inputs {}
|
||||
Then a WrappedToolNotFoundError should be raised for "local/nonexistent"
|
||||
|
||||
Scenario: Circular wrapping chain raises WrappingCycleError
|
||||
Given a validation "local/wrap-a" that wraps "local/wrap-b" with a simple transform
|
||||
And a validation "local/wrap-b" that wraps "local/wrap-a" with a simple transform
|
||||
And a wrapped tool executor using the test registry with cycle
|
||||
When I try to execute the wrapping validation with cycle from "local/wrap-a"
|
||||
Then a WrappingCycleError should be raised
|
||||
|
||||
Scenario: Execution context is inherited from wrapping validation
|
||||
Given a tool registry with a tool "local/context-tool" that returns {"data": "value"}
|
||||
And a validation "local/context-wrap" that wraps "local/context-tool" with a data transform
|
||||
And a wrapped tool executor using the test registry
|
||||
When I execute the wrapping validation with inputs {"key": "value"}
|
||||
Then the wrapped tool should have received the validation inputs
|
||||
|
||||
Scenario: WrappedToolExecutor rejects non-Validation argument
|
||||
Given a wrapped tool executor with empty registry
|
||||
When I try to execute with a non-Validation object
|
||||
Then a TypeError should be raised from the executor
|
||||
|
||||
Scenario: WrappedToolExecutor rejects validation without wraps
|
||||
Given a validation "local/no-wrap" without wraps set
|
||||
And a wrapped tool executor with empty registry
|
||||
When I try to execute the non-wrapping validation
|
||||
Then a ValueError should be raised indicating wraps is not set
|
||||
|
||||
# ── Additional coverage: WrappingDepthExceededError ──────────────
|
||||
|
||||
Scenario: WrappingDepthExceededError is raised when chain is too deep
|
||||
Given a deep wrapping chain of 11 validations
|
||||
And a wrapped tool executor using the deep chain registry
|
||||
When I try to execute the deep chain wrapping validation
|
||||
Then a WrappingDepthExceededError should be raised with depth 10
|
||||
|
||||
# ── Additional coverage: ArgumentMapper.mapping property ─────────
|
||||
|
||||
Scenario: ArgumentMapper exposes the raw mapping via property
|
||||
Given an argument mapper with mapping {"x": "y"}
|
||||
Then the mapper mapping property should return {"x": "y"}
|
||||
|
||||
Scenario: ArgumentMapper mapping property returns None for identity mapper
|
||||
Given an argument mapper with no mapping
|
||||
Then the mapper mapping property should be None
|
||||
|
||||
# ── Additional coverage: TransformExecutor type checks ───────────
|
||||
|
||||
Scenario: TransformExecutor rejects non-string transform code
|
||||
When I try to create a transform executor with non-string code
|
||||
Then a TypeError should be raised from transform code type check
|
||||
|
||||
Scenario: TransformExecutor rejects non-string tool name
|
||||
When I try to create a transform executor with non-string tool name
|
||||
Then a TypeError should be raised from transform tool name check
|
||||
|
||||
Scenario: TransformExecutor raises on code that throws during exec
|
||||
Given a transform executor with code that raises an error during exec
|
||||
When I try to execute the transform with any output
|
||||
Then a TransformExecutionError should be raised with message containing "compile"
|
||||
|
||||
# ── Additional coverage: WrappedToolExecutor init checks ─────────
|
||||
|
||||
Scenario: WrappedToolExecutor rejects None tool_lookup
|
||||
When I try to create a wrapped tool executor with None tool_lookup
|
||||
Then a ValueError should be raised from executor construction for tool_lookup
|
||||
|
||||
Scenario: WrappedToolExecutor rejects None tool_executor
|
||||
When I try to create a wrapped tool executor with None tool_executor
|
||||
Then a ValueError should be raised from executor construction for tool_executor
|
||||
|
||||
Scenario: WrappedToolExecutor rejects non-callable tool_lookup
|
||||
When I try to create a wrapped tool executor with non-callable tool_lookup
|
||||
Then a TypeError should be raised from executor construction for tool_lookup
|
||||
|
||||
Scenario: WrappedToolExecutor rejects non-callable tool_executor
|
||||
When I try to create a wrapped tool executor with non-callable tool_executor
|
||||
Then a TypeError should be raised from executor construction for tool_executor
|
||||
|
||||
# ── Additional coverage: execute with non-dict inputs ────────────
|
||||
|
||||
Scenario: WrappedToolExecutor rejects non-dict inputs
|
||||
Given a tool registry with a tool "local/some-tool" that returns {"ok": true}
|
||||
And a validation "local/input-wrap" that wraps "local/some-tool" with a simple transform
|
||||
And a wrapped tool executor using the test registry
|
||||
When I try to execute wrapping validation with non-dict inputs
|
||||
Then a TypeError should be raised for non-dict inputs
|
||||
|
||||
# ── Additional coverage: chain with None wraps at end ────────────
|
||||
|
||||
Scenario: Chain resolution handles validation with wraps set to None in chain
|
||||
Given a tool registry with a tool "local/leaf" that returns {"value": 1}
|
||||
And a validation "local/wrapper-no-inner" that wraps "local/leaf" with a passthrough transform
|
||||
And a wrapped tool executor using the test registry
|
||||
When I execute the wrapping validation with inputs {}
|
||||
Then the wrapped execution should succeed with passed true
|
||||
|
||||
# ── Additional coverage: _execute_chain with None wraps target ────
|
||||
|
||||
Scenario: _execute_chain raises ValueError when leaf wraps is None
|
||||
Given a wrapped tool executor with empty registry
|
||||
When I directly call _execute_chain with a no-wraps leaf
|
||||
Then a ValueError should be raised for missing wraps target
|
||||
|
||||
# ── ToolRunner coverage: execution environment and error paths ───
|
||||
|
||||
Scenario: ToolRunner resolve_execution_environment delegates to resolver
|
||||
Given a ToolRunner with a mock registry
|
||||
When I call resolve_execution_environment on the runner
|
||||
Then the resolved environment should be local
|
||||
|
||||
Scenario: ToolRunner execute returns error when env resolver raises ValueError
|
||||
Given a ToolRunner with a value-error-raising env resolver
|
||||
When I execute a tool through the runner with env error
|
||||
Then the tool result should have success false
|
||||
And the tool result error should contain "Execution environment error"
|
||||
|
||||
Scenario: ToolRunner execute returns error for container environment
|
||||
Given a ToolRunner with a container-returning env resolver
|
||||
When I execute a tool through the runner with container env
|
||||
Then the tool result should have success false
|
||||
And the tool result error should contain "Container execution is not yet implemented"
|
||||
Reference in New Issue
Block a user