From 543248ffd4527bd85e3dfcc968f1b3c13a53d892 Mon Sep 17 00:00:00 2001 From: CoreRasurae Date: Tue, 4 Aug 2026 21:24:35 +0000 Subject: [PATCH] test(agents): capture inline sandbox __import__ escape regression (#107) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a failing-first Behave regression test proving issue #107: ToolAgent._execute_python_code (cleveractors.agents.tool) builds the inline-code sandbox's __builtins__ dict per docs/index.md §13.2.1's "Restricted Built-ins for Inline Code" table, which lists a single module-shaped facility (json) and states the table is exhaustive. §13.2.3 additionally prohibits "dynamic import of modules other than those explicitly listed". Despite this, safe_globals["__builtins__"] binds "__import__" directly to the real, unrestricted __import__ builtin, so inline code can write `import os` and reach exactly the filesystem/ network/process facilities the standard was written to keep out. Two Scenario Outlines drive the two documented entry points that share _execute_python_code -- a "type: tool" agent's inline `code:` body, and the exec_python-gated `python_exec` built-in tool -- through the public ToolAgent.process_message() API, each exercised against three dynamic- import forms that all resolve through the same unrestricted __import__ builtin: `import os`, `from os import getcwd`, and `__import__('os')`. Every case asserts an ExecutionError is raised instead of the import succeeding. Confirmed all assertions fail via AssertionError when @tdd_expected_fail is removed (os.getcwd() returns a real path today), and pass via TddExpectedFailPolicy's inversion with the tag present. All scenarios are tagged @tdd_issue, @tdd_issue_107, and @tdd_expected_fail per the TDD issue-capture workflow. The actual fix (a restricted __import__ shim permitting only `json`) lands separately on bugfix/m1-inline-sandbox-import-restriction per issue #107. Refs: #107, #108 --- features/application_coverage_gaps.feature | 2 +- .../application_interactive_coverage.feature | 2 +- features/bridge_coverage_gaps.feature | 2 +- features/bridge_final_coverage.feature | 2 +- features/bridge_remaining_coverage.feature | 2 +- features/config_parser_coverage_gaps.feature | 2 +- features/coverage_final.feature | 2 +- features/coverage_gaps.feature | 2 +- features/final_coverage_push.feature | 2 +- .../inline_sandbox_import_restriction.feature | 75 +++++++++++++++++++ features/llm_missing_coverage.feature | 2 +- features/llm_tools_coverage.feature | 2 +- features/massive_coverage.feature | 2 +- features/nodes_coverage.feature | 2 +- features/nodes_coverage_gaps.feature | 2 +- features/pure_graph_coverage.feature | 2 +- features/pure_graph_coverage_gaps_v2.feature | 2 +- features/registry_http_client.feature | 2 +- features/road_to_97.feature | 2 +- ...inline_sandbox_import_restriction_steps.py | 35 +++++++++ features/stream_router_coverage_gaps.feature | 2 +- features/stream_router_final_coverage.feature | 2 +- features/templates_base_remaining.feature | 2 +- features/tool_coverage_gaps.feature | 2 +- features/validate_dict.feature | 2 +- features/yte_remaining_coverage.feature | 2 +- robot/lib/registry_lib.py | 6 +- robot/registry_integration.robot | 2 +- 28 files changed, 140 insertions(+), 26 deletions(-) create mode 100644 features/inline_sandbox_import_restriction.feature create mode 100644 features/steps/inline_sandbox_import_restriction_steps.py diff --git a/features/application_coverage_gaps.feature b/features/application_coverage_gaps.feature index 7c255a1..c097175 100644 --- a/features/application_coverage_gaps.feature +++ b/features/application_coverage_gaps.feature @@ -239,4 +239,4 @@ Feature: Application Error Handling, Stream Routing, and Tool Execution Scenario: _handle_graph_command executes valid graph route Given an app with a graph route "my_graph" (apg) When _handle_graph_command is called with "my_graph hello graph" (apg) - Then the graph should be executed and result displayed (apg) \ No newline at end of file + Then the graph should be executed and result displayed (apg) diff --git a/features/application_interactive_coverage.feature b/features/application_interactive_coverage.feature index 0d783fd..be8672f 100644 --- a/features/application_interactive_coverage.feature +++ b/features/application_interactive_coverage.feature @@ -54,4 +54,4 @@ Feature: Interactive Session Input Handling and Error Recovery Scenario: _handle_stream_command with config=None handles safely Given an app without config but with streams for interactive (aic) When I call _handle_stream_command with "valid_stream test" (aic) - Then the message should be delivered to the stream (aic) \ No newline at end of file + Then the message should be delivered to the stream (aic) diff --git a/features/bridge_coverage_gaps.feature b/features/bridge_coverage_gaps.feature index 1a79114..f18a6f8 100644 --- a/features/bridge_coverage_gaps.feature +++ b/features/bridge_coverage_gaps.feature @@ -150,4 +150,4 @@ Feature: Bridge Event Loop Management, Task Cleanup, and Metadata Handling Given a fresh bridge gaps test context (brg) Given a bridge with active tasks where the loop is running (brg) When I call cleanup synchronously (brg) - Then tasks should be cancelled without blocking (brg) \ No newline at end of file + Then tasks should be cancelled without blocking (brg) diff --git a/features/bridge_final_coverage.feature b/features/bridge_final_coverage.feature index e24e34a..8650f16 100644 --- a/features/bridge_final_coverage.feature +++ b/features/bridge_final_coverage.feature @@ -89,4 +89,4 @@ Feature: Bridge Task Handler and Cleanup Edge Cases Scenario: Cleanup fallback catches general exception Given a bridge with active tasks where cleanup will fail (bgr) When I call cleanup and all operations raise exceptions (bgr) - Then tasks should be cancelled via fallback path (bgr) \ No newline at end of file + Then tasks should be cancelled via fallback path (bgr) diff --git a/features/bridge_remaining_coverage.feature b/features/bridge_remaining_coverage.feature index ec4e138..995745f 100644 --- a/features/bridge_remaining_coverage.feature +++ b/features/bridge_remaining_coverage.feature @@ -69,4 +69,4 @@ Feature: Bridge Executor and Connector Event Loop and Task Exception Handling Scenario: Cleanup inside running loop cancels tasks without blocking Given a bridge with active tasks where event loop is running (brc) When cleanup is called while loop is running (brc) - Then tasks should be cancelled without await (brc) \ No newline at end of file + Then tasks should be cancelled without await (brc) diff --git a/features/config_parser_coverage_gaps.feature b/features/config_parser_coverage_gaps.feature index 6e61ac3..ce27644 100644 --- a/features/config_parser_coverage_gaps.feature +++ b/features/config_parser_coverage_gaps.feature @@ -39,4 +39,4 @@ Feature: Config Parser List-Format Route Processing and Template Non-Dict Error Scenario: Non-dict restored config raises ValueError Given a ReactiveConfigParser instance (cpg) When parse_files receives a template file yielding a non-dict result (cpg) - Then a ValueError should be raised for non-dictionary config (cpg) \ No newline at end of file + Then a ValueError should be raised for non-dictionary config (cpg) diff --git a/features/coverage_final.feature b/features/coverage_final.feature index 85dc006..c25e441 100644 --- a/features/coverage_final.feature +++ b/features/coverage_final.feature @@ -16,4 +16,4 @@ Feature: Routing Adapter, Dynamic Router, and Message Router Integration Scenario: All MessageRouter code paths execute correctly Given a fresh integration test context When I exercise all MessageRouter code paths - Then all MessageRouter paths should execute correctly \ No newline at end of file + Then all MessageRouter paths should execute correctly diff --git a/features/coverage_gaps.feature b/features/coverage_gaps.feature index ae77633..5a82d2b 100644 --- a/features/coverage_gaps.feature +++ b/features/coverage_gaps.feature @@ -89,4 +89,4 @@ Feature: SAFE_BUILTINS, ConfigurationError Propagation, CLI Execution, and Messa And I resolve a message through the DynamicRouter And I resolve a non-matching message through the DynamicRouter Then the DynamicRouter should route to the correct target - And the DynamicRouter should return unchanged for non-matching content \ No newline at end of file + And the DynamicRouter should return unchanged for non-matching content diff --git a/features/final_coverage_push.feature b/features/final_coverage_push.feature index 081ee96..77fcd6b 100644 --- a/features/final_coverage_push.feature +++ b/features/final_coverage_push.feature @@ -29,4 +29,4 @@ Feature: ContextManager Lifecycle, ProgressBar, and Miscellaneous Module Coverag Scenario: Quick coverage boost for miscellaneous modules Given a fresh final coverage push test context (fcp) When I test miscellaneous uncovered paths (fcp) - Then miscellaneous coverage should succeed (fcp) \ No newline at end of file + Then miscellaneous coverage should succeed (fcp) diff --git a/features/inline_sandbox_import_restriction.feature b/features/inline_sandbox_import_restriction.feature new file mode 100644 index 0000000..aec7526 --- /dev/null +++ b/features/inline_sandbox_import_restriction.feature @@ -0,0 +1,75 @@ +Feature: Inline-code sandbox restricts imports to the json module only + As a host operating CleverAgents in safe mode + I want inline tool code to be unable to import any module other than json + So that a "type: tool" agent's code body or the python_exec built-in tool + cannot reach the filesystem, network, or process facilities that + §13.2.1's Restricted Built-ins table and §13.2.3's prohibited-capabilities + list were written to keep out + + # Regression test for issue #107: + # ToolAgent._execute_python_code (cleveractors.agents.tool) builds the + # inline-code sandbox's __builtins__ dict per docs/index.md §13.2.1's + # "Restricted Built-ins for Inline Code" table, which lists a single + # module-shaped facility — json — and states that table is exhaustive + # ("no other built-in facilities MAY be exposed"). §13.2.3 additionally + # prohibits, without qualification, "dynamic import of modules other than + # those explicitly listed". + # + # Despite this, safe_globals["__builtins__"]["__import__"] is bound + # directly to the real, unrestricted __import__ builtin (with the + # accompanying comment "Allow imports for json"). Because __import__ is + # the actual mechanism the `import` statement invokes, inline code can + # write `import os` (or any other module) and reach exactly the + # facilities the standard was written to keep out. Both documented + # entry points into _execute_python_code share this gap: a "type: tool" + # agent's inline `code:` body, and the exec_python-gated `python_exec` + # built-in tool. + # + # Will be fixed by replacing the raw __import__ reference with a + # restricted shim on bugfix/m1-inline-sandbox-import-restriction per + # issue #107. + + @tdd_issue @tdd_issue_107 @tdd_expected_fail + Scenario Outline: python_exec tool rejects "" of os instead of executing it + Given a ToolAgent is configured with name "python_exec_import_escape_agent" and config + """ + { + "tools": ["python_exec"], + "exec_python": true + } + """ + When I create the ToolAgent + And I process a JSON message with the ToolAgent: + """ + {"tool": "python_exec", "args": {"code": ""}} + """ + Then the inline code's import of "os" should have been rejected by the sandbox + + Examples: dynamic-import forms that all resolve through the unrestricted __import__ builtin + | import_form | code | + | import statement | import os\nresult = os.getcwd() | + | from-import statement | from os import getcwd\nresult = getcwd() | + | __import__ call | result = __import__('os').getcwd() | + + @tdd_issue @tdd_issue_107 @tdd_expected_fail + Scenario Outline: A tool agent's inline code body rejects "" of os instead of executing it + Given a ToolAgent is configured with name "inline_code_import_escape_agent" and config + """ + { + "tools": [ + { + "name": "escape_tool", + "code": "" + } + ] + } + """ + When I create the ToolAgent + And I process a message with the ToolAgent: "irrelevant trigger message" + Then the inline code's import of "os" should have been rejected by the sandbox + + Examples: dynamic-import forms that all resolve through the unrestricted __import__ builtin + | import_form | code | + | import statement | import os\nresult = os.getcwd() | + | from-import statement | from os import getcwd\nresult = getcwd() | + | __import__ call | result = __import__('os').getcwd() | diff --git a/features/llm_missing_coverage.feature b/features/llm_missing_coverage.feature index 52c9855..e7fab64 100644 --- a/features/llm_missing_coverage.feature +++ b/features/llm_missing_coverage.feature @@ -87,4 +87,4 @@ Feature: LLM Agent Temperature Override, Cleanup, and Context History Scenario: Non-numeric _temperature_override raises ConfigurationError in process_message Given I setup a test LLM agent for temperature override tests (llm_gaps) When I call process_message with _temperature_override "bad_string" (llm_gaps) - Then a ConfigurationError mentioning must be a number is raised (llm_gaps) \ No newline at end of file + Then a ConfigurationError mentioning must be a number is raised (llm_gaps) diff --git a/features/llm_tools_coverage.feature b/features/llm_tools_coverage.feature index dde448a..7815524 100644 --- a/features/llm_tools_coverage.feature +++ b/features/llm_tools_coverage.feature @@ -43,4 +43,4 @@ Feature: LLM Tools Coverage Given an llm_tools coverage test environment When I call normalize_tool_entry with string "http_request" Then the result description should mention "timeout" - And the result description should mention "config" \ No newline at end of file + And the result description should mention "config" diff --git a/features/massive_coverage.feature b/features/massive_coverage.feature index fcfc8d0..3aae8f5 100644 --- a/features/massive_coverage.feature +++ b/features/massive_coverage.feature @@ -6,4 +6,4 @@ Feature: Integration Smoke Test Across Pure Graph, Bridge, Stream Router, Config Scenario: Massive coverage exercise Given a massive coverage test environment When I exercise all major uncovered code paths - Then the massive coverage exercise should complete \ No newline at end of file + Then the massive coverage exercise should complete diff --git a/features/nodes_coverage.feature b/features/nodes_coverage.feature index 467043d..b3dab9c 100644 --- a/features/nodes_coverage.feature +++ b/features/nodes_coverage.feature @@ -6,4 +6,4 @@ Feature: Node Types and Execution Across Agent, Function, Conditional, Subgraph, Scenario: All Node code paths execute correctly Given a fresh nodes test context When I exercise all Node code paths - Then all node code paths should execute correctly \ No newline at end of file + Then all node code paths should execute correctly diff --git a/features/nodes_coverage_gaps.feature b/features/nodes_coverage_gaps.feature index 56ee702..0d4f3c6 100644 --- a/features/nodes_coverage_gaps.feature +++ b/features/nodes_coverage_gaps.feature @@ -176,4 +176,4 @@ Feature: Node Execution Without Event Loop, MESSAGE_ROUTER, and History Truncati Scenario: Evaluate edge condition with non-bool condition result Given a fresh nodes coverage gaps test context (nodes_gaps) When I evaluate edge condition returning a non-bool value (nodes_gaps) - Then it should coerce the result via bool conversion (nodes_gaps) \ No newline at end of file + Then it should coerce the result via bool conversion (nodes_gaps) diff --git a/features/pure_graph_coverage.feature b/features/pure_graph_coverage.feature index f4c3055..a96fb7d 100644 --- a/features/pure_graph_coverage.feature +++ b/features/pure_graph_coverage.feature @@ -211,4 +211,4 @@ Feature: Pure Graph Configuration Analysis, Cycle Detection, and Execution Scenario: process_message delegates to execute Given a fresh pure graph test context (pg_cov) When I call process_message on a PureLangGraph (pg_cov) - Then it should delegate to execute and return the result (pg_cov) \ No newline at end of file + Then it should delegate to execute and return the result (pg_cov) diff --git a/features/pure_graph_coverage_gaps_v2.feature b/features/pure_graph_coverage_gaps_v2.feature index cc650e5..98e1c96 100644 --- a/features/pure_graph_coverage_gaps_v2.feature +++ b/features/pure_graph_coverage_gaps_v2.feature @@ -144,4 +144,4 @@ Feature: Pure Graph AttributeError Handling, Output Logging, and Cycle Detection Scenario: create_pure_langgraph except path logs error Given a fresh pure graph gaps v2 test context (pgg) When PureLangGraph constructor raises an exception (pgg) - Then the error should be logged and re-raised during creation (pgg) \ No newline at end of file + Then the error should be logged and re-raised during creation (pgg) diff --git a/features/registry_http_client.feature b/features/registry_http_client.feature index c9d39fa..a65c98e 100644 --- a/features/registry_http_client.feature +++ b/features/registry_http_client.feature @@ -233,4 +233,4 @@ Feature: Registry HTTP Client Scenario: Client with allow_insecure suppresses HTTP warnings When I create an insecure RegistryClient pointed at "http://registry.example.com" and capture logs - Then no HTTPS warnings were logged \ No newline at end of file + Then no HTTPS warnings were logged diff --git a/features/road_to_97.feature b/features/road_to_97.feature index 4803cf4..67430ec 100644 --- a/features/road_to_97.feature +++ b/features/road_to_97.feature @@ -39,4 +39,4 @@ Feature: YAML Preprocessor, Route Graph Config, Renderer TemplateError, and Rout Scenario: Renderer raises TemplateError for Mustache engine without render Given a fresh road97 test context (r97) When I render a template with a Mustache engine missing render method (r97) - Then a TemplateError should be raised for missing render (r97) \ No newline at end of file + Then a TemplateError should be raised for missing render (r97) diff --git a/features/steps/inline_sandbox_import_restriction_steps.py b/features/steps/inline_sandbox_import_restriction_steps.py new file mode 100644 index 0000000..5f296d3 --- /dev/null +++ b/features/steps/inline_sandbox_import_restriction_steps.py @@ -0,0 +1,35 @@ +"""Step definitions for the inline-code sandbox import-escape regression (issue #107). + +Reuses the shared ToolAgent configuration/execution steps from +``tool_agent_steps.py`` and adds the one assertion specific to this +regression: that importing a non-``json`` module from inline code is +rejected with an :class:`ExecutionError`, the same category +``ToolAgent._execute_python_code`` already uses for other prohibited-name +access, rather than succeeding outright. See +``inline_sandbox_import_restriction.feature`` for the scenarios this backs. +""" + +from __future__ import annotations + +from typing import Any + +from behave import then + +from cleveractors.core.exceptions import ExecutionError + + +@then( + 'the inline code\'s import of "{module_name}" should have been rejected by the sandbox' +) +def step_assert_import_rejected(context: Any, module_name: str) -> None: + assert isinstance(context.error, ExecutionError), ( + f"Expected inline code executing 'import {module_name}' to be " + "rejected with an ExecutionError — the category " + "ToolAgent._execute_python_code already uses for other " + "prohibited-name access (docs/index.md §13.2.1 lists no import " + "facility beyond `json`; §13.2.3 prohibits dynamic import of any " + f"other module) — but got {context.error!r} with result " + f"{context.result!r} instead. The sandbox's __import__ builtin is " + f"the real, unrestricted import machinery and let {module_name!r} " + "through." + ) diff --git a/features/stream_router_coverage_gaps.feature b/features/stream_router_coverage_gaps.feature index ee253a1..1d429a8 100644 --- a/features/stream_router_coverage_gaps.feature +++ b/features/stream_router_coverage_gaps.feature @@ -256,4 +256,4 @@ Feature: Stream Operator Error Branches, Switch Routing, and Delay/Throttle Oper Scenario: Content not contains condition evaluates correctly Given I have a stream router ready for condition testing in srg When I test the content not contains condition with matching and non-matching inputs in srg - Then the content not contains condition should return true when text is absent in srg \ No newline at end of file + Then the content not contains condition should return true when text is absent in srg diff --git a/features/stream_router_final_coverage.feature b/features/stream_router_final_coverage.feature index d0d04bf..213dec0 100644 --- a/features/stream_router_final_coverage.feature +++ b/features/stream_router_final_coverage.feature @@ -65,4 +65,4 @@ Feature: StreamMessage Copy, Agent Mapper Context, and Accumulator Edge Cases Scenario: _apply_accumulator with unknown type returns accumulator Given a stream router configured for accumulator testing (srv) When _apply_accumulator receives unknown type (srv) - Then the accumulator should be returned unchanged (srv) \ No newline at end of file + Then the accumulator should be returned unchanged (srv) diff --git a/features/templates_base_remaining.feature b/features/templates_base_remaining.feature index a6d37fe..7dd2cc4 100644 --- a/features/templates_base_remaining.feature +++ b/features/templates_base_remaining.feature @@ -19,4 +19,4 @@ Feature: InstantiationContext and Template Variable Parse Error Handling Scenario: BaseTemplate _apply_template_vars YAML parse failure Given a fresh tbr test context (tbr) When I apply template vars with a conditional block producing broken YAML (tbr) - Then the raw rendered output should be returned (tbr) \ No newline at end of file + Then the raw rendered output should be returned (tbr) diff --git a/features/tool_coverage_gaps.feature b/features/tool_coverage_gaps.feature index dae449d..4bc75b7 100644 --- a/features/tool_coverage_gaps.feature +++ b/features/tool_coverage_gaps.feature @@ -426,4 +426,4 @@ Feature: Tool Agent Coverage Gaps Scenario: Process message handles JSONDecodeError Given a tool agent coverage test environment When I process a message with malformed JSON - Then an ExecutionError mentioning Invalid JSON is raised \ No newline at end of file + Then an ExecutionError mentioning Invalid JSON is raised diff --git a/features/validate_dict.feature b/features/validate_dict.feature index 3882656..6c44464 100644 --- a/features/validate_dict.feature +++ b/features/validate_dict.feature @@ -875,4 +875,4 @@ Feature: validate_dict public API — spec-conformant Actor Configuration valida Scenario: Route with duplicate subgraph references does not cause false too-complex error Given a config dict with a graph route containing two subgraph nodes referencing the same route When I call validate_dict with platform_limits max_subgraph_depth of 5 - Then no exception is raised \ No newline at end of file + Then no exception is raised diff --git a/features/yte_remaining_coverage.feature b/features/yte_remaining_coverage.feature index 03f0871..7d82eba 100644 --- a/features/yte_remaining_coverage.feature +++ b/features/yte_remaining_coverage.feature @@ -39,4 +39,4 @@ Feature: YAML Template Engine Remaining Uncovered Paths Scenario: Reconstruct with non-string template value Given a fresh yte remaining test context (ytr) When _reconstruct_from_structure encounters non-string _template_value (ytr) - Then the template value error should be raised (ytr) \ No newline at end of file + Then the template value error should be raised (ytr) diff --git a/robot/lib/registry_lib.py b/robot/lib/registry_lib.py index a6ca985..d867242 100644 --- a/robot/lib/registry_lib.py +++ b/robot/lib/registry_lib.py @@ -18,7 +18,11 @@ from cleveractors.registry.local_store import LocalPackageStore from cleveractors.registry.reference_resolver import PackageContentResolver from cleveractors.registry.resolver import ReferenceResolver from cleveractors.registry.types import PackageReference -from cleveractors.templates.base import ComponentReference, InstantiationContext, TemplateType +from cleveractors.templates.base import ( + ComponentReference, + InstantiationContext, + TemplateType, +) from cleveractors.templates.enhanced_registry import EnhancedTemplateRegistry from cleveractors.templates.renderer import TemplateEngine, TemplateRenderer diff --git a/robot/registry_integration.robot b/robot/registry_integration.robot index e4285c5..01e3d80 100644 --- a/robot/registry_integration.robot +++ b/robot/registry_integration.robot @@ -68,4 +68,4 @@ Discover Registry Metadata Registry Client Closes Cleanly [Documentation] close() shuts down the underlying httpx client. Create Registry Client ${SERVER_URL} - Close Client \ No newline at end of file + Close Client -- 2.52.0