cleveragents/cleveractors-core

Fork

You've already forked cleveractors-core

Code Issues 16 Pull requests 1 Projects Releases Packages Wiki Activity Actions 373

test(agents): capture inline sandbox import escape regression (#107) #110

Merged

CoreRasurae merged 1 commit from tdd/m1-inline-sandbox-import-restriction into master

2026-08-06 14:31:20 +00:00

Conversation 3 Commits 1 Files changed 28 +140 -26

CoreRasurae commented

2026-08-04 21:26:22 +00:00

Member

Copy link

Summary

Adds a failing-first Behave regression test proving issue #107: ToolAgent._execute_python_code (cleveractors.agents.tool) builds the inline-code sandbox's __builtins__ dict per docs/index.md §13.2.1's "Restricted Built-ins for Inline Code" table, which lists a single module-shaped facility (json) and states that table is exhaustive. §13.2.3 additionally prohibits "dynamic import of modules other than those explicitly listed". Despite this, safe_globals["__builtins__"]["__import__"] is bound directly to the real, unrestricted __import__ builtin, so inline code can write import os and reach exactly the filesystem/network/process facilities the standard was written to keep out.

Two scenarios drive the two documented entry points that share _execute_python_code — a type: tool agent's inline code: body, and the exec_python-gated python_exec built-in tool — through the public ToolAgent.process_message() API with code that does import os; result = os.getcwd(), and assert an ExecutionError is raised instead of the import succeeding. Today it isn't: both return the real working directory path.

Both scenarios are tagged @tdd_issue, @tdd_issue_107, and @tdd_expected_fail per the TDD issue-capture workflow.

The actual fix (a restricted __import__ shim permitting only json) lands separately on bugfix/m1-inline-sandbox-import-restriction per issue #107.

Test plan

nox -s lint green
nox -s format -- --check green
nox -s typecheck green
nox -s security_scan green
nox -s dead_code green
nox -s unit_tests green — confirms both assertions fail via AssertionError without the @tdd_expected_fail tag (verified locally by temporarily removing it), and TddExpectedFailPolicy inverts them to a pass with the tag present
nox -s coverage_report — 96.7% (>= 96.5% threshold)
nox -s integration_tests — green
nox -s e2e_tests — green
nox -s complexity, nox -s benchmark_regression — informational, unaffected by this test-only change

Closes #108

## Summary Adds a failing-first Behave regression test proving issue #107: `ToolAgent._execute_python_code` (`cleveractors.agents.tool`) builds the inline-code sandbox's `__builtins__` dict per `docs/index.md` §13.2.1's "Restricted Built-ins for Inline Code" table, which lists a single module-shaped facility (`json`) and states that table is exhaustive. §13.2.3 additionally prohibits "dynamic import of modules other than those explicitly listed". Despite this, `safe_globals["__builtins__"]["__import__"]` is bound directly to the real, unrestricted `__import__` builtin, so inline code can write `import os` and reach exactly the filesystem/network/process facilities the standard was written to keep out. Two scenarios drive the two documented entry points that share `_execute_python_code` — a `type: tool` agent's inline `code:` body, and the `exec_python`-gated `python_exec` built-in tool — through the public `ToolAgent.process_message()` API with code that does `import os; result = os.getcwd()`, and assert an `ExecutionError` is raised instead of the import succeeding. Today it isn't: both return the real working directory path. Both scenarios are tagged `@tdd_issue`, `@tdd_issue_107`, and `@tdd_expected_fail` per the TDD issue-capture workflow. The actual fix (a restricted `__import__` shim permitting only `json`) lands separately on `bugfix/m1-inline-sandbox-import-restriction` per issue #107. ## Test plan - [x] `nox -s lint` green - [x] `nox -s format -- --check` green - [x] `nox -s typecheck` green - [x] `nox -s security_scan` green - [x] `nox -s dead_code` green - [x] `nox -s unit_tests` green — confirms both assertions fail via `AssertionError` without the `@tdd_expected_fail` tag (verified locally by temporarily removing it), and `TddExpectedFailPolicy` inverts them to a pass with the tag present - [x] `nox -s coverage_report` — 96.7% (>= 96.5% threshold) - [x] `nox -s integration_tests` — green - [x] `nox -s e2e_tests` — green - [x] `nox -s complexity`, `nox -s benchmark_regression` — informational, unaffected by this test-only change Closes #108

CoreRasurae added this to the v2.1.0 milestone

2026-08-04 21:26:22 +00:00

CoreRasurae added the

Type

Testing

label

2026-08-04 21:27:06 +00:00

CoreRasurae added a new dependency

2026-08-04 21:28:42 +00:00

#108 TDD: Inline-code sandbox exposes __import__, letting code bypass the Restricted Built-ins set entirely (e.g. import os)

hurui200320 approved these changes

2026-08-05 05:47:31 +00:00

hurui200320 left a comment

Copy link

PR Review: !110 (Ticket #108)

Verdict: Approve

The PR correctly implements the TDD regression test for issue #107 as specified in #108. The two Behave scenarios cover both documented entry points into ToolAgent._execute_python_code, carry the required @tdd_issue @tdd_issue_107 @tdd_expected_fail tags, and rely on AssertionError so the TddExpectedFailPolicy inversion works as intended. No critical or major technical issues were found.

Critical Issues

None

Major Issues

None

Minor Issues

Coverage threshold discrepancy — noxfile.py:27 sets COVERAGE_THRESHOLD = 96.5, while CONTRIBUTING.md §"Project-Specific Guidelines" and .gitea/workflows/ci.yml state the enforced coverage gate is 97%. The PR test plan reports 96.7% and passes noxfile.py's threshold, but this inherited project-level inconsistency could mask a real coverage regression. Recommend aligning noxfile.py with the documented 97% gate (separate from this PR, or clarify if 96.5% is intentional).

Nits

Import-form coverage gap — features/inline_sandbox_import_restriction.feature:44,56 only tests the import os form. Other dynamic-import forms (e.g., from os import getcwd or __import__('os')) share the same __import__ bypass and could be added in the fix PR (#107) to harden the restriction.

Summary

This is a clean, focused TDD capture PR. The scenarios correctly exercise the sandbox escape through both the python_exec built-in tool and a type: tool inline code: body, and they will flip from inverted-fail to outright pass once the fix in #107 replaces the raw __import__ reference. The code is statically typed, lint-clean, and follows the project's BDD/TDD conventions. Approving.

## PR Review: !110 (Ticket #108) ### Verdict: Approve The PR correctly implements the TDD regression test for issue #107 as specified in #108. The two Behave scenarios cover both documented entry points into `ToolAgent._execute_python_code`, carry the required `@tdd_issue @tdd_issue_107 @tdd_expected_fail` tags, and rely on `AssertionError` so the `TddExpectedFailPolicy` inversion works as intended. No critical or major technical issues were found. ### Critical Issues None ### Major Issues None ### Minor Issues 1. **Coverage threshold discrepancy** — `noxfile.py:27` sets `COVERAGE_THRESHOLD = 96.5`, while `CONTRIBUTING.md` §"Project-Specific Guidelines" and `.gitea/workflows/ci.yml` state the enforced coverage gate is **97%**. The PR test plan reports 96.7% and passes noxfile.py's threshold, but this inherited project-level inconsistency could mask a real coverage regression. Recommend aligning `noxfile.py` with the documented 97% gate (separate from this PR, or clarify if 96.5% is intentional). ### Nits 1. **Import-form coverage gap** — `features/inline_sandbox_import_restriction.feature:44,56` only tests the `import os` form. Other dynamic-import forms (e.g., `from os import getcwd` or `__import__('os')`) share the same `__import__` bypass and could be added in the fix PR (#107) to harden the restriction. ### Summary This is a clean, focused TDD capture PR. The scenarios correctly exercise the sandbox escape through both the `python_exec` built-in tool and a `type: tool` inline `code:` body, and they will flip from inverted-fail to outright pass once the fix in #107 replaces the raw `__import__` reference. The code is statically typed, lint-clean, and follows the project's BDD/TDD conventions. Approving.

CoreRasurae added a new dependency

2026-08-05 12:42:24 +00:00

#111 Allow the shell and http_request tool timeouts to be overridden per invocation via a timeout argument

CoreRasurae removed a dependency

2026-08-05 12:44:00 +00:00

#111 Allow the shell and http_request tool timeouts to be overridden per invocation via a timeout argument

CoreRasurae commented

2026-08-06 11:45:00 +00:00

Author

Member

Copy link

Minor issues number 1 - coverage, is not relevant, we have a noxfile threshold at 96.5% while having a 97% in the specification, but the 97% is the rounded value, which is also obtainable from a 96.5% real coverage.

CoreRasurae force-pushed tdd/m1-inline-sandbox-import-restriction from bac4868abd

Some checks failed

CI / lint (pull_request) Successful in 47s

Details

CI / typecheck (pull_request) Successful in 1m32s

Details

CI / security (pull_request) Successful in 1m16s

Details

CI / quality (pull_request) Successful in 1m34s

Details

CI / build (pull_request) Successful in 1m42s

Details

CI / integration_tests (pull_request) Successful in 3m38s

Details

CI / unit_tests (pull_request) Successful in 4m57s

Details

CI / coverage (pull_request) Successful in 4m2s

Details

CI / status-check (pull_request) Successful in 17s

Details

CI / benchmark (pull_request) Failing after 21m22s

Details

to 35e9f22bde

Some checks failed

CI / lint (pull_request) Successful in 1m48s

Details

CI / quality (pull_request) Successful in 1m49s

Details

CI / security (pull_request) Successful in 2m9s

Details

CI / typecheck (pull_request) Successful in 2m35s

Details

CI / build (pull_request) Successful in 51s

Details

CI / integration_tests (pull_request) Successful in 2m37s

Details

CI / unit_tests (pull_request) Successful in 4m26s

Details

CI / coverage (pull_request) Successful in 4m43s

Details

CI / status-check (pull_request) Failing after 11m50s

Details

CI / benchmark (pull_request) Failing after 20m24s

Details

2026-08-06 11:59:01 +00:00

Compare

CoreRasurae commented

2026-08-06 12:13:18 +00:00

Author

Member

Copy link

Minor Issue #1 (coverage threshold) is not being changed here — per my earlier comment it's a pre-existing, repo-wide noxfile.py setting unrelated to this test-only PR; revisiting it would require raising overall project coverage separately and is out of scope for #108.

Addressed Rui's Nit #1 directly on this branch (amended, not a separate PR): both scenarios are now Scenario Outlines exercising three dynamic-import forms that all resolve through the same unrestricted `__import__` — `import os`, `from os import getcwd`, and `__import__('os')` — across both entry points (6 scenarios total). Verified locally that all fail via `AssertionError` without `@tdd_expected_fail` and pass via the inversion with it present (`nox -s unit_tests -- features/inline_sandbox_import_restriction.feature`). `lint`, `typecheck`, `format --check`, `security_scan`, and `dead_code` all still green. Minor Issue #1 (coverage threshold) is not being changed here — per my earlier comment it's a pre-existing, repo-wide noxfile.py setting unrelated to this test-only PR; revisiting it would require raising overall project coverage separately and is out of scope for #108.

CoreRasurae added 1 commit

2026-08-06 12:28:02 +00:00

test(agents): capture inline sandbox __import__ escape regression (#107 )

Some checks failed

CI / build (pull_request) Successful in 49s

Details

CI / lint (pull_request) Successful in 1m13s

Details

CI / integration_tests (pull_request) Successful in 3m29s

Details

CI / benchmark (pull_request) Failing after 34m42s

Details

CI / typecheck (pull_request) Successful in 1m16s

Details

CI / security (pull_request) Successful in 2m34s

Details

CI / quality (pull_request) Successful in 54s

Details

CI / unit_tests (pull_request) Successful in 5m13s

Details

CI / coverage (pull_request) Successful in 4m27s

Details

CI / status-check (pull_request) Successful in 14s

Details

543248ffd4

Adds a failing-first Behave regression test proving issue #107:
ToolAgent._execute_python_code (cleveractors.agents.tool) builds the
inline-code sandbox's __builtins__ dict per docs/index.md §13.2.1's
"Restricted Built-ins for Inline Code" table, which lists a single
module-shaped facility (json) and states the table is exhaustive. §13.2.3
additionally prohibits "dynamic import of modules other than those
explicitly listed". Despite this, safe_globals["__builtins__"] binds
"__import__" directly to the real, unrestricted __import__ builtin, so
inline code can write `import os` and reach exactly the filesystem/
network/process facilities the standard was written to keep out.

Two Scenario Outlines drive the two documented entry points that share
_execute_python_code -- a "type: tool" agent's inline `code:` body, and
the exec_python-gated `python_exec` built-in tool -- through the public
ToolAgent.process_message() API, each exercised against three dynamic-
import forms that all resolve through the same unrestricted __import__
builtin: `import os`, `from os import getcwd`, and `__import__('os')`.
Every case asserts an ExecutionError is raised instead of the import
succeeding. Confirmed all assertions fail via AssertionError when
@tdd_expected_fail is removed (os.getcwd() returns a real path today),
and pass via TddExpectedFailPolicy's inversion with the tag present.

All scenarios are tagged @tdd_issue, @tdd_issue_107, and
@tdd_expected_fail per the TDD issue-capture workflow.

The actual fix (a restricted __import__ shim permitting only `json`)
lands separately on bugfix/m1-inline-sandbox-import-restriction per
issue #107.

Refs: #107, #108