fix(agents): restrict inline-code sandbox to a json-only __import__ #118

Merged
CoreRasurae merged 1 commit from bugfix/m1-inline-sandbox-import-restriction into master 2026-08-07 15:41:26 +00:00
Member

Summary

ToolAgent._execute_python_code bound the inline-code sandbox's __builtins__["__import__"] directly to the real, unrestricted __import__ builtin, so inline code (a type: tool agent's code: body, or the exec_python-gated python_exec tool) could import os/subprocess/socket and reach exactly the filesystem/network/process facilities docs/index.md §13.2.1's exhaustive Restricted Built-ins table and §13.2.3's prohibited-capabilities list were written to keep out.

Fixes it by replacing the raw reference with a new restricted shim, _sandboxed_import, that permits only json and rejects every other module name via the same NameErrorExecutionError translation _execute_python_code already uses for other prohibited-name access.

Collateral fixes required to keep the suite green (both depended on the exact gap being closed):

  • Retired features/stderr_suppression.feature/steps: it exercised inline code doing import sys to reach real stderr, which is no longer reachable by design.
  • Rewrote regex-based (import re) text extraction in several tests/fixtures/email_graph/*.yaml agent configs to use only sandbox-permitted str methods, preserving the same output keys.

Test plan

  • nox -s unit_tests — 2962 scenarios passed, including new positive (import json) and TDD regression (@tdd_issue_107, @tdd_expected_fail removed) scenarios in features/inline_sandbox_import_restriction.feature.
  • nox -s integration_tests — 345 Robot tests passed, including new robot/inline_sandbox_import_restriction.robot.
  • nox -s coverage_report — COVERAGE OK: 96.6% (threshold 96.5%).
  • nox -s lint, nox -s format -- --check, nox -s typecheck, nox -s security_scan, nox -s dead_code — all green.
  • Manually verified import json / from json import dumps succeed and import os/subprocess raise ExecutionError via the sandbox.

Closes #107

## Summary `ToolAgent._execute_python_code` bound the inline-code sandbox's `__builtins__["__import__"]` directly to the real, unrestricted `__import__` builtin, so inline code (a `type: tool` agent's `code:` body, or the `exec_python`-gated `python_exec` tool) could `import os`/`subprocess`/`socket` and reach exactly the filesystem/network/process facilities docs/index.md §13.2.1's exhaustive Restricted Built-ins table and §13.2.3's prohibited-capabilities list were written to keep out. Fixes it by replacing the raw reference with a new restricted shim, `_sandboxed_import`, that permits only `json` and rejects every other module name via the same `NameError` → `ExecutionError` translation `_execute_python_code` already uses for other prohibited-name access. **Collateral fixes required to keep the suite green** (both depended on the exact gap being closed): - Retired `features/stderr_suppression.feature`/steps: it exercised inline code doing `import sys` to reach real stderr, which is no longer reachable by design. - Rewrote regex-based (`import re`) text extraction in several `tests/fixtures/email_graph/*.yaml` agent configs to use only sandbox-permitted `str` methods, preserving the same output keys. ## Test plan - [x] `nox -s unit_tests` — 2962 scenarios passed, including new positive (`import json`) and TDD regression (`@tdd_issue_107`, `@tdd_expected_fail` removed) scenarios in `features/inline_sandbox_import_restriction.feature`. - [x] `nox -s integration_tests` — 345 Robot tests passed, including new `robot/inline_sandbox_import_restriction.robot`. - [x] `nox -s coverage_report` — COVERAGE OK: 96.6% (threshold 96.5%). - [x] `nox -s lint`, `nox -s format -- --check`, `nox -s typecheck`, `nox -s security_scan`, `nox -s dead_code` — all green. - [x] Manually verified `import json` / `from json import dumps` succeed and `import os`/`subprocess` raise `ExecutionError` via the sandbox. Closes #107
CoreRasurae added this to the v2.1.0 milestone 2026-08-06 17:55:15 +00:00
hurui200320 left a comment

PR Review: !118 (Ticket #107)

Verdict: Approve

The change correctly closes the security gap described in #107: the inline-code sandbox now routes every import statement through _sandboxed_import, which rejects any module other than json and surfaces the failure through the existing NameErrorExecutionError path. Both entry points (type: tool inline code: bodies and the exec_python-gated python_exec built-in tool) are covered, the expression sandbox is untouched, existing import json fixtures continue to work, and the collateral fixture updates remove the now-prohibited import re usage without introducing new imports.

Critical Issues

None

Major Issues

None

Minor Issues

  • Hardening: _sandboxed_import does not reject non-zero level arguments.
    File: src/cleveractors/agents/tool.py, lines 35–53
    The shim correctly blocks every module name except json, but it still passes a non-zero level through to the real __import__. For defense-in-depth, consider rejecting level != 0 with the same NameError, since the spec only intends the top-level json module to be reachable and relative imports are not part of the allowed surface.

  • The regression assertion only checks the error category, not the failure cause.
    File: features/steps/inline_sandbox_import_restriction_steps.py, lines 21–35
    The step asserts isinstance(context.error, ExecutionError), which is good, but it does not verify that the ExecutionError was actually triggered by the forbidden import (e.g., by checking that the module name appears in the message or that the post-import statement did not run). Consider tightening the assertion so a future regression that accidentally allows the import but fails later cannot masquerade as a passing test.

Nits

  • Duplicated string-parsing logic in email fixtures.
    Files: tests/fixtures/email_graph/*/{business_email,component_orders}.yaml
    The regex replacement logic for component-order part numbers and quantities is copy-pasted across six fixture files. If these fixtures need further semantic tweaks, a shared helper or template would reduce drift.

  • Robot library couples to a private method.
    File: robot/InlineSandboxImportRestrictionLib.py, lines 46–48
    Run Inline Code calls agent._python_exec_tool directly. This is fine for an integration test, but it creates a brittle coupling to a private API; consider documenting it or using the public process_message path.

Summary

This is a focused, well-scoped security fix. The shim is minimal, the regression coverage exercises both documented entry points and multiple dynamic-import forms, and the collateral cleanup (removing import re from fixtures and retiring the stderr_suppression feature that depended on import sys) is clearly justified. No critical or major issues were found; the minor items above are optional hardening/quality improvements.

## PR Review: !118 (Ticket #107) ### Verdict: Approve The change correctly closes the security gap described in #107: the inline-code sandbox now routes every `import` statement through `_sandboxed_import`, which rejects any module other than `json` and surfaces the failure through the existing `NameError` → `ExecutionError` path. Both entry points (`type: tool` inline `code:` bodies and the `exec_python`-gated `python_exec` built-in tool) are covered, the expression sandbox is untouched, existing `import json` fixtures continue to work, and the collateral fixture updates remove the now-prohibited `import re` usage without introducing new imports. ### Critical Issues None ### Major Issues None ### Minor Issues - **Hardening: `_sandboxed_import` does not reject non-zero `level` arguments.** File: `src/cleveractors/agents/tool.py`, lines 35–53 The shim correctly blocks every module name except `json`, but it still passes a non-zero `level` through to the real `__import__`. For defense-in-depth, consider rejecting `level != 0` with the same `NameError`, since the spec only intends the top-level `json` module to be reachable and relative imports are not part of the allowed surface. - **The regression assertion only checks the error category, not the failure cause.** File: `features/steps/inline_sandbox_import_restriction_steps.py`, lines 21–35 The step asserts `isinstance(context.error, ExecutionError)`, which is good, but it does not verify that the `ExecutionError` was actually triggered by the forbidden import (e.g., by checking that the module name appears in the message or that the post-import statement did not run). Consider tightening the assertion so a future regression that accidentally allows the import but fails later cannot masquerade as a passing test. ### Nits - **Duplicated string-parsing logic in email fixtures.** Files: `tests/fixtures/email_graph/*/{business_email,component_orders}.yaml` The regex replacement logic for component-order part numbers and quantities is copy-pasted across six fixture files. If these fixtures need further semantic tweaks, a shared helper or template would reduce drift. - **Robot library couples to a private method.** File: `robot/InlineSandboxImportRestrictionLib.py`, lines 46–48 `Run Inline Code` calls `agent._python_exec_tool` directly. This is fine for an integration test, but it creates a brittle coupling to a private API; consider documenting it or using the public `process_message` path. ### Summary This is a focused, well-scoped security fix. The shim is minimal, the regression coverage exercises both documented entry points and multiple dynamic-import forms, and the collateral cleanup (removing `import re` from fixtures and retiring the `stderr_suppression` feature that depended on `import sys`) is clearly justified. No critical or major issues were found; the minor items above are optional hardening/quality improvements.
CoreRasurae force-pushed bugfix/m1-inline-sandbox-import-restriction from 64cbb46f35
Some checks failed
CI / typecheck (pull_request) Successful in 1m41s
CI / lint (pull_request) Successful in 2m10s
CI / security (pull_request) Successful in 2m24s
CI / quality (pull_request) Successful in 2m31s
CI / build (pull_request) Successful in 2m25s
CI / integration_tests (pull_request) Successful in 3m11s
CI / unit_tests (pull_request) Successful in 5m58s
CI / coverage (pull_request) Successful in 5m32s
CI / status-check (pull_request) Successful in 10s
CI / benchmark (pull_request) Failing after 28m51s
to e99eb26579
Some checks failed
CI / lint (pull_request) Successful in 51s
CI / typecheck (pull_request) Successful in 1m20s
CI / security (pull_request) Successful in 1m12s
CI / quality (pull_request) Successful in 1m37s
CI / build (pull_request) Successful in 50s
CI / integration_tests (pull_request) Successful in 3m21s
CI / unit_tests (pull_request) Successful in 6m3s
CI / coverage (pull_request) Successful in 4m18s
CI / status-check (pull_request) Successful in 3s
CI / benchmark (pull_request) Failing after 27m43s
2026-08-07 14:08:58 +00:00
Compare
Author
Member

Thanks for the review, @hurui200320! Addressed the minor issues; the two nits were left as-is with reasoning below.

Applied:

  • _sandboxed_import now also rejects a non-zero level (relative import forms such as from .json import dumps) — §13.2.1 lists only the absolute json module, so relative forms aren't part of the allowed surface. Added a new scenario (python_exec tool rejects a relative import of json) covering this path.
  • The regression assertion in inline_sandbox_import_restriction_steps.py now also checks that the rejected module name appears in the ExecutionError message and that result stays unset (the statement after the rejected import never ran), so it can no longer pass on an unrelated error.

Left as-is (nits, not spec violations):

  • Duplicated str-based extraction logic across the email_graph fixture YAMLs: genuine deduplication isn't feasible here without either introducing a shared import the sandbox is specifically designed to forbid, or an unrelated fixture-templating refactor — disproportionate scope for this PR. Happy to open a follow-up issue if you'd like it tracked separately.
  • robot/InlineSandboxImportRestrictionLib.py calling agent._python_exec_tool directly: test-only code, not a maintainability concern for the production API surface.

All nox sessions green (lint, format, typecheck, security_scan, dead_code, unit_tests, coverage delta on the touched code, integration_tests, e2e_tests). Amended the existing commit (still unmerged, single-commit branch) rather than adding a new one, per project convention.

Thanks for the review, @hurui200320! Addressed the minor issues; the two nits were left as-is with reasoning below. **Applied:** - `_sandboxed_import` now also rejects a non-zero `level` (relative import forms such as `from .json import dumps`) — §13.2.1 lists only the absolute `json` module, so relative forms aren't part of the allowed surface. Added a new scenario (`python_exec tool rejects a relative import of json`) covering this path. - The regression assertion in `inline_sandbox_import_restriction_steps.py` now also checks that the rejected module name appears in the `ExecutionError` message and that `result` stays unset (the statement after the rejected import never ran), so it can no longer pass on an unrelated error. **Left as-is (nits, not spec violations):** - Duplicated str-based extraction logic across the `email_graph` fixture YAMLs: genuine deduplication isn't feasible here without either introducing a shared import the sandbox is specifically designed to forbid, or an unrelated fixture-templating refactor — disproportionate scope for this PR. Happy to open a follow-up issue if you'd like it tracked separately. - `robot/InlineSandboxImportRestrictionLib.py` calling `agent._python_exec_tool` directly: test-only code, not a maintainability concern for the production API surface. All nox sessions green (lint, format, typecheck, security_scan, dead_code, unit_tests, coverage delta on the touched code, integration_tests, e2e_tests). Amended the existing commit (still unmerged, single-commit branch) rather than adding a new one, per project convention.
fix(agents): restrict inline-code sandbox to a json-only __import__
Some checks failed
CI / lint (pull_request) Successful in 53s
CI / quality (pull_request) Successful in 59s
CI / security (pull_request) Successful in 2m6s
CI / typecheck (pull_request) Successful in 2m11s
CI / build (pull_request) Successful in 1m33s
CI / integration_tests (pull_request) Successful in 3m34s
CI / unit_tests (pull_request) Successful in 6m48s
CI / coverage (pull_request) Successful in 4m34s
CI / status-check (pull_request) Successful in 8s
CI / benchmark (pull_request) Failing after 26m36s
CI / security (push) Successful in 1m14s
CI / lint (push) Successful in 1m53s
CI / quality (push) Successful in 53s
CI / typecheck (push) Successful in 2m22s
CI / build (push) Successful in 1m48s
CI / integration_tests (push) Successful in 3m28s
CI / unit_tests (push) Successful in 5m38s
CI / coverage (push) Successful in 4m50s
CI / status-check (push) Successful in 10s
CI / benchmark (push) Failing after 20m4s
c5154701aa
ToolAgent._execute_python_code built the inline-code sandbox's
__builtins__ dict per docs/index.md §13.2.1's "Restricted Built-ins for
Inline Code" table (exhaustive per that section, and per §13.2.3's
"dynamic import of modules other than those explicitly listed" is
prohibited), but bound "__import__" directly to the real, unrestricted
__import__ builtin. Since __import__ is the mechanism the `import`
statement invokes, inline code -- a "type: tool" agent's `code:` body,
or the exec_python-gated python_exec tool, both of which share this
sandbox construction -- could write `import os` (or any other module)
and reach exactly the filesystem/network/process facilities the
standard was written to keep out.

Fixes it by replacing the raw __import__ reference with a new
module-level shim, _sandboxed_import, that permits only `json` (the
sole module-shaped facility §13.2.1 lists) and rejects every other
module name via NameError -- the same category _execute_python_code
already uses for other prohibited-name access, translated to
ExecutionError, rather than a bare Python traceback. `import json`
keeps working unchanged; the §13.2.2 expression sandbox is unaffected,
since it never exposed __import__.

Two collateral fixes were required to keep the existing suite green,
both because they depended on the exact `__import__` gap being closed:

- features/stderr_suppression.feature exercised inline code doing
  `import sys; print(..., file=sys.stderr)` to validate a debug-output
  suppression path in _execute_python_code. `sys` is not in §13.2.1's
  table, so this is now unreachable by design -- there is no longer any
  way for inline code to obtain a real stderr handle. Retired the
  feature and its steps; the underlying redirect logic in tool.py is
  untouched (harmless, and still guards defense-in-depth).

- Several tests/fixtures/email_graph/*.yaml agent configs used
  `import json, re` for regex-based text extraction (part numbers,
  quantities, meeting mentions). `re` is likewise not in §13.2.1's
  table. Rewrote the extraction logic in each affected fixture using
  only sandbox-permitted str methods (no regex needed for these
  fixed-shape patterns), preserving the same output keys.

Regression scenarios (tagged @tdd_issue, @tdd_issue_107) proving this
bug were merged separately in #108; this commit removes the
@tdd_expected_fail tag now that the fix is in place, and adds a
positive scenario plus a Robot integration test confirming `import
json` still works while os/socket/subprocess imports are rejected
end-to-end.

Addressed review feedback from @hurui200320 on PR !118:

- _sandboxed_import now also rejects a non-zero `level` (relative
  import forms such as `from .json import dumps`), since §13.2.1 lists
  only the absolute `json` module and relative imports are not part of
  that allowed surface. A new scenario in
  inline_sandbox_import_restriction.feature covers this path.
- The regression assertion (inline_sandbox_import_restriction_steps.py)
  now also checks that the rejected module name appears in the
  ExecutionError message and that the statement following the rejected
  import never ran (`result` stays unset), so the assertion can no
  longer pass on an unrelated error.
- Deduplicated the str-based extraction logic that was copy-pasted
  across seven tests/fixtures/email_graph/*.yaml agent configs (both
  the meeting/urgency detection and the part/quantity/BOM detection).
  Since the sandboxed `code:` bodies still cannot import a shared
  helper module (that is the exact restriction this commit adds), the
  two snippets now live once each as constants in robot/EmailGraphLib.py
  and are expanded from a placeholder token back into every fixture's
  parsed config right after load (yaml.safe_load, and after resolving a
  namespaced package via LocalPackageStore), so every consumer still
  executes byte-identical Python source assembled from one canonical
  copy. Verified equivalence by executing the old and new snippets
  against representative inputs and diffing the results, and re-ran the
  Robot email_graph suite end-to-end (345/345 passed).
- Left as-is (reviewer nit, not a spec violation): the Robot library's
  use of ToolAgent's private _python_exec_tool -- test-only code, not a
  maintainability concern for the production API surface.

ISSUES CLOSED: #107
CoreRasurae force-pushed bugfix/m1-inline-sandbox-import-restriction from e99eb26579
Some checks failed
CI / lint (pull_request) Successful in 51s
CI / typecheck (pull_request) Successful in 1m20s
CI / security (pull_request) Successful in 1m12s
CI / quality (pull_request) Successful in 1m37s
CI / build (pull_request) Successful in 50s
CI / integration_tests (pull_request) Successful in 3m21s
CI / unit_tests (pull_request) Successful in 6m3s
CI / coverage (pull_request) Successful in 4m18s
CI / status-check (pull_request) Successful in 3s
CI / benchmark (pull_request) Failing after 27m43s
to c5154701aa
Some checks failed
CI / lint (pull_request) Successful in 53s
CI / quality (pull_request) Successful in 59s
CI / security (pull_request) Successful in 2m6s
CI / typecheck (pull_request) Successful in 2m11s
CI / build (pull_request) Successful in 1m33s
CI / integration_tests (pull_request) Successful in 3m34s
CI / unit_tests (pull_request) Successful in 6m48s
CI / coverage (pull_request) Successful in 4m34s
CI / status-check (pull_request) Successful in 8s
CI / benchmark (pull_request) Failing after 26m36s
CI / security (push) Successful in 1m14s
CI / lint (push) Successful in 1m53s
CI / quality (push) Successful in 53s
CI / typecheck (push) Successful in 2m22s
CI / build (push) Successful in 1m48s
CI / integration_tests (push) Successful in 3m28s
CI / unit_tests (push) Successful in 5m38s
CI / coverage (push) Successful in 4m50s
CI / status-check (push) Successful in 10s
CI / benchmark (push) Failing after 20m4s
2026-08-07 14:30:55 +00:00
Compare
Author
Member

Follow-up: went ahead and deduplicated the fixture extraction logic too (code quality/maintainability took priority over leaving it as a nit).

Since the sandboxed code: bodies still can't import a shared helper module (that's the exact restriction this PR adds), the two duplicated snippets (business_email meeting/urgency detection, component_orders part/quantity/BOM detection) now live once each as constants in robot/EmailGraphLib.py, expanded from a placeholder token back into each fixture's config right after yaml.safe_load (and after resolving a namespaced package via LocalPackageStore) — before the config ever reaches create_executor. Verified equivalence by executing the old and new snippets against representative inputs and diffing the results, then re-ran the full Robot email_graph suite end-to-end (345/345 passed).

The private-method-coupling nit (InlineSandboxImportRestrictionLib.py calling agent._python_exec_tool) is still left as-is — test-only code, not a maintainability concern for the production API.

All nox sessions re-verified green after this change. Force-pushed the amended commit again.

Follow-up: went ahead and deduplicated the fixture extraction logic too (code quality/maintainability took priority over leaving it as a nit). Since the sandboxed `code:` bodies still can't import a shared helper module (that's the exact restriction this PR adds), the two duplicated snippets (business_email meeting/urgency detection, component_orders part/quantity/BOM detection) now live once each as constants in `robot/EmailGraphLib.py`, expanded from a placeholder token back into each fixture's config right after `yaml.safe_load` (and after resolving a namespaced package via `LocalPackageStore`) — before the config ever reaches `create_executor`. Verified equivalence by executing the old and new snippets against representative inputs and diffing the results, then re-ran the full Robot email_graph suite end-to-end (345/345 passed). The private-method-coupling nit (`InlineSandboxImportRestrictionLib.py` calling `agent._python_exec_tool`) is still left as-is — test-only code, not a maintainability concern for the production API. All nox sessions re-verified green after this change. Force-pushed the amended commit again.
CoreRasurae deleted branch bugfix/m1-inline-sandbox-import-restriction 2026-08-07 15:41:34 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
cleveragents/cleveractors-core!118
No description provided.