feat(validation): implement tool wrapping runtime (wraps + transform delegation)
CI / lint (pull_request) Successful in 14s
CI / typecheck (pull_request) Successful in 34s
CI / security (pull_request) Successful in 29s
CI / quality (pull_request) Successful in 15s
CI / benchmark-publish (pull_request) Has been skipped
CI / build (pull_request) Successful in 14s
CI / unit_tests (pull_request) Successful in 2m4s
CI / integration_tests (pull_request) Successful in 2m50s
CI / coverage (pull_request) Successful in 4m15s
CI / docker (pull_request) Successful in 42s
CI / lint (push) Successful in 12s
CI / typecheck (push) Successful in 31s
CI / quality (push) Successful in 16s
CI / security (push) Successful in 31s
CI / build (push) Successful in 16s
CI / unit_tests (push) Successful in 3m23s
CI / benchmark-regression (push) Has been skipped
CI / docker (push) Successful in 42s
CI / integration_tests (push) Successful in 4m11s
CI / coverage (push) Successful in 4m8s
CI / benchmark-publish (push) Successful in 16m5s
CI / benchmark-regression (pull_request) Successful in 29m59s

Implement the runtime execution engine for validation tool wrapping,
as specified in docs/specification.md § Tool Wrapping.

WrappedToolExecutor resolves wraps references and delegates execution
to wrapped tools, supporting composable wrapping chains with cycle
detection and depth limiting (max 10 levels).

ArgumentMapper translates arguments between wrapper and wrapped tool
schemas using the argument_mapping configuration. Supports both
forwarded parameter names and literal fixed values.

TransformExecutor runs user-supplied transform functions in a
sandboxed Python environment with restricted builtins (no imports,
no filesystem, no network access). Validates that transforms return
proper validation-format dicts with a passed boolean.

Wired into the tool package public API via tool/__init__.py exports.
All new error types (WrappedToolNotFoundError, WrappingCycleError,
WrappingDepthExceededError, TransformExecutionError) provide clear
diagnostic messages.

Tests: 20 Behave scenarios covering argument mapping, transform
execution, simple/chained delegation, error handling, and sandbox
restrictions. 8 Robot Framework integration smoke tests. ASV
benchmarks for delegation overhead measurement.

ISSUES CLOSED: #543
This commit was merged in pull request #555.
This commit is contained in:
2026-03-04 03:36:28 +00:00
committed by Forgejo
parent db5e5c974f
commit 93e3893d69
12 changed files with 1969 additions and 0 deletions
+220
View File
@@ -0,0 +1,220 @@
Feature: Tool wrapping runtime
As the validation execution engine
I need to delegate execution to wrapped tools via wraps + transform
So that validations can reuse existing tool implementations
# ArgumentMapper
Scenario: ArgumentMapper with None mapping passes arguments through
Given an argument mapper with no mapping
When I apply the mapper to inputs {"path": "/src", "verbose": true}
Then the mapped arguments should be {"path": "/src", "verbose": true}
Scenario: ArgumentMapper with mapping translates argument names
Given an argument mapper with mapping {"test_directory": "source_dir", "coverage_enabled": true}
When I apply the mapper to inputs {"source_dir": "/tests"}
Then the mapped arguments should be {"test_directory": "/tests", "coverage_enabled": true}
Scenario: ArgumentMapper with literal values injects fixed values
Given an argument mapper with mapping {"mode": "strict", "count": 42}
When I apply the mapper to inputs {"extra": "ignored"}
Then the mapped arguments should be {"mode": "strict", "count": 42}
Scenario: ArgumentMapper rejects non-dict inputs
Given an argument mapper with no mapping
When I try to apply the mapper to non-dict input "not_a_dict"
Then a TypeError should be raised from the argument mapper
Scenario: ArgumentMapper rejects non-dict mapping at construction
When I try to create an argument mapper with a non-dict mapping
Then a TypeError should be raised from argument mapper construction
# ── TransformExecutor ─────────────────────────────────────────────
Scenario: TransformExecutor runs a valid transform function
Given a transform executor with code that checks returncode equals zero
When I execute the transform with tool output {"returncode": 0, "tests_run": 10}
Then the transform result should have passed true
And the transform result should have message "All tests passed"
Scenario: TransformExecutor handles failing transform output
Given a transform executor with code that checks returncode equals zero
When I execute the transform with tool output {"returncode": 1, "tests_run": 10}
Then the transform result should have passed false
Scenario: TransformExecutor raises on missing transform function
Given a transform executor with code that does not define a transform function
When I try to execute the transform with any output
Then a TransformExecutionError should be raised with message containing "callable"
Scenario: TransformExecutor raises on non-dict return value
Given a transform executor with code that returns a non-dict value
When I try to execute the transform with any output
Then a TransformExecutionError should be raised with message containing "dict"
Scenario: TransformExecutor raises on missing passed key in result
Given a transform executor with code that returns a dict without passed key
When I try to execute the transform with any output
Then a TransformExecutionError should be raised with message containing "passed"
Scenario: TransformExecutor raises on empty transform code
When I try to create a transform executor with empty code
Then a ValueError should be raised from transform construction
Scenario: TransformExecutor sandboxes dangerous operations
Given a transform executor with code that attempts to import os
When I try to execute the transform with any output
Then a TransformExecutionError should be raised from sandbox restriction
# ── WrappedToolExecutor ───────────────────────────────────────────
Scenario: Simple wrapping delegates to the wrapped tool
Given a tool registry with a tool "local/run-tests" that returns {"returncode": 0}
And a validation "local/tests-pass" that wraps "local/run-tests" with a simple transform
And a wrapped tool executor using the test registry
When I execute the wrapping validation with inputs {"path": "/src"}
Then the wrapped execution should succeed with passed true
And the wrapped tool "local/run-tests" should have been called
Scenario: Argument mapping translates arguments to the wrapped tool
Given a tool registry with a tool "local/run-tests" that returns {"returncode": 0}
And a validation "local/mapped-check" that wraps "local/run-tests" with argument mapping {"test_directory": "source_dir", "coverage_enabled": true}
And a wrapped tool executor using the test registry
When I execute the wrapping validation with inputs {"source_dir": "/tests"}
Then the wrapped tool "local/run-tests" should have received argument "test_directory" with value "/tests"
And the wrapped tool "local/run-tests" should have received argument "coverage_enabled" with value true
Scenario: Chained wrapping delegates through the chain
Given a tool registry with a tool "local/base-tool" that returns {"status": "ok"}
And a validation "local/mid-wrapper" that wraps "local/base-tool" with a passthrough transform
And a validation "local/outer-wrapper" that wraps "local/mid-wrapper" with a status transform
And a wrapped tool executor using the test registry
When I execute the outer wrapping validation with inputs {}
Then the wrapped execution should succeed with passed true
And the wrapped tool "local/base-tool" should have been called
Scenario: Missing wrapped tool raises WrappedToolNotFoundError
Given a validation "local/broken-wrap" that wraps "local/nonexistent" with a simple transform
And a wrapped tool executor with empty registry
When I try to execute the wrapping validation with inputs {}
Then a WrappedToolNotFoundError should be raised for "local/nonexistent"
Scenario: Circular wrapping chain raises WrappingCycleError
Given a validation "local/wrap-a" that wraps "local/wrap-b" with a simple transform
And a validation "local/wrap-b" that wraps "local/wrap-a" with a simple transform
And a wrapped tool executor using the test registry with cycle
When I try to execute the wrapping validation with cycle from "local/wrap-a"
Then a WrappingCycleError should be raised
Scenario: Execution context is inherited from wrapping validation
Given a tool registry with a tool "local/context-tool" that returns {"data": "value"}
And a validation "local/context-wrap" that wraps "local/context-tool" with a data transform
And a wrapped tool executor using the test registry
When I execute the wrapping validation with inputs {"key": "value"}
Then the wrapped tool should have received the validation inputs
Scenario: WrappedToolExecutor rejects non-Validation argument
Given a wrapped tool executor with empty registry
When I try to execute with a non-Validation object
Then a TypeError should be raised from the executor
Scenario: WrappedToolExecutor rejects validation without wraps
Given a validation "local/no-wrap" without wraps set
And a wrapped tool executor with empty registry
When I try to execute the non-wrapping validation
Then a ValueError should be raised indicating wraps is not set
# ── Additional coverage: WrappingDepthExceededError ──────────────
Scenario: WrappingDepthExceededError is raised when chain is too deep
Given a deep wrapping chain of 11 validations
And a wrapped tool executor using the deep chain registry
When I try to execute the deep chain wrapping validation
Then a WrappingDepthExceededError should be raised with depth 10
# ── Additional coverage: ArgumentMapper.mapping property ─────────
Scenario: ArgumentMapper exposes the raw mapping via property
Given an argument mapper with mapping {"x": "y"}
Then the mapper mapping property should return {"x": "y"}
Scenario: ArgumentMapper mapping property returns None for identity mapper
Given an argument mapper with no mapping
Then the mapper mapping property should be None
# ── Additional coverage: TransformExecutor type checks ───────────
Scenario: TransformExecutor rejects non-string transform code
When I try to create a transform executor with non-string code
Then a TypeError should be raised from transform code type check
Scenario: TransformExecutor rejects non-string tool name
When I try to create a transform executor with non-string tool name
Then a TypeError should be raised from transform tool name check
Scenario: TransformExecutor raises on code that throws during exec
Given a transform executor with code that raises an error during exec
When I try to execute the transform with any output
Then a TransformExecutionError should be raised with message containing "compile"
# ── Additional coverage: WrappedToolExecutor init checks ─────────
Scenario: WrappedToolExecutor rejects None tool_lookup
When I try to create a wrapped tool executor with None tool_lookup
Then a ValueError should be raised from executor construction for tool_lookup
Scenario: WrappedToolExecutor rejects None tool_executor
When I try to create a wrapped tool executor with None tool_executor
Then a ValueError should be raised from executor construction for tool_executor
Scenario: WrappedToolExecutor rejects non-callable tool_lookup
When I try to create a wrapped tool executor with non-callable tool_lookup
Then a TypeError should be raised from executor construction for tool_lookup
Scenario: WrappedToolExecutor rejects non-callable tool_executor
When I try to create a wrapped tool executor with non-callable tool_executor
Then a TypeError should be raised from executor construction for tool_executor
# ── Additional coverage: execute with non-dict inputs ────────────
Scenario: WrappedToolExecutor rejects non-dict inputs
Given a tool registry with a tool "local/some-tool" that returns {"ok": true}
And a validation "local/input-wrap" that wraps "local/some-tool" with a simple transform
And a wrapped tool executor using the test registry
When I try to execute wrapping validation with non-dict inputs
Then a TypeError should be raised for non-dict inputs
# ── Additional coverage: chain with None wraps at end ────────────
Scenario: Chain resolution handles validation with wraps set to None in chain
Given a tool registry with a tool "local/leaf" that returns {"value": 1}
And a validation "local/wrapper-no-inner" that wraps "local/leaf" with a passthrough transform
And a wrapped tool executor using the test registry
When I execute the wrapping validation with inputs {}
Then the wrapped execution should succeed with passed true
# ── Additional coverage: _execute_chain with None wraps target ────
Scenario: _execute_chain raises ValueError when leaf wraps is None
Given a wrapped tool executor with empty registry
When I directly call _execute_chain with a no-wraps leaf
Then a ValueError should be raised for missing wraps target
# ── ToolRunner coverage: execution environment and error paths ───
Scenario: ToolRunner resolve_execution_environment delegates to resolver
Given a ToolRunner with a mock registry
When I call resolve_execution_environment on the runner
Then the resolved environment should be local
Scenario: ToolRunner execute returns error when env resolver raises ValueError
Given a ToolRunner with a value-error-raising env resolver
When I execute a tool through the runner with env error
Then the tool result should have success false
And the tool result error should contain "Execution environment error"
Scenario: ToolRunner execute returns error for container environment
Given a ToolRunner with a container-returning env resolver
When I execute a tool through the runner with container env
Then the tool result should have success false
And the tool result error should contain "Container execution is not yet implemented"