forked from cleveragents/cleveragents-core
246 lines
13 KiB
Gherkin
246 lines
13 KiB
Gherkin
Feature: Validation test fixtures
|
||
As a QA engineer
|
||
I want to validate that code sanitization, AST security, model validation, and input coercion handle edge cases
|
||
So that invalid code samples, edge case project structures, and malformed input data are properly rejected
|
||
|
||
# ──────────────────────────────────────────────────
|
||
# Section 1: Invalid code samples - AST security validation
|
||
# ──────────────────────────────────────────────────
|
||
|
||
Scenario: Reject code containing import statement
|
||
When I validate code ast with "import os"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden construct"
|
||
|
||
Scenario: Reject code containing from-import statement
|
||
When I validate code ast with "from os import path"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden construct"
|
||
|
||
Scenario: Reject code containing global statement
|
||
When I validate code ast with "global x"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden construct"
|
||
|
||
Scenario: Reject code containing nonlocal statement
|
||
When I validate code ast with "def f():\n nonlocal x"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden construct"
|
||
|
||
Scenario: Reject code calling exec
|
||
When I validate code ast with "exec('print(1)')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden call"
|
||
|
||
Scenario: Reject code calling eval
|
||
When I validate code ast with "eval('1+1')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden call"
|
||
|
||
Scenario: Reject code calling compile
|
||
When I validate code ast with "compile('x', '', 'exec')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden call"
|
||
|
||
Scenario: Reject code calling __import__
|
||
When I validate code ast with "__import__('os')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden call"
|
||
|
||
Scenario: Reject code calling getattr
|
||
When I validate code ast with "getattr(obj, 'secret')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden call"
|
||
|
||
Scenario: Reject code calling setattr
|
||
When I validate code ast with "setattr(obj, 'key', 'val')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden call"
|
||
|
||
Scenario: Reject syntactically invalid code in AST validation
|
||
When I validate code ast with "def broken("
|
||
Then a StreamRoutingError should be raised mentioning "Invalid code syntax"
|
||
|
||
Scenario: Accept safe code in AST validation
|
||
When I validate code ast with "result = 1 + 2"
|
||
Then no error should be raised from ast validation
|
||
|
||
# ──────────────────────────────────────────────────
|
||
# Section 1b: RestrictedPython – dunder / subscript bypass vectors (C2)
|
||
# These verify that _compile_restricted_code blocks the attack patterns
|
||
# that the previous hand-rolled AST validator could not catch.
|
||
# ──────────────────────────────────────────────────
|
||
|
||
Scenario: Reject dunder attribute chain via __class__.__bases__
|
||
When I compile restricted code with "x = ().__class__.__bases__[0].__subclasses__()"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden construct"
|
||
|
||
Scenario: Reject dunder attribute access via __class__
|
||
When I compile restricted code with "x = ''.__class__"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden construct"
|
||
|
||
Scenario: Reject subscript access to __builtins__
|
||
When I compile restricted code with "x = __builtins__['__import__']('os')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden"
|
||
|
||
Scenario: Reject dunder __init__ attribute access
|
||
When I compile restricted code with "x = ().__class__.__bases__[0].__subclasses__()[0].__init__.__globals__"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden construct"
|
||
|
||
Scenario: Reject attribute-based getattr bypass at runtime
|
||
When I exec restricted code with "f = getattr; f([], '__class__')"
|
||
Then a runtime error should be raised
|
||
|
||
Scenario: Reject __import__ via name mangling
|
||
When I compile restricted code with "f = __import__; f('os')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden"
|
||
|
||
Scenario: Accept safe string operation through restricted compilation
|
||
When I compile restricted code with "result = 'hello'.upper()"
|
||
Then no error should be raised from restricted compilation
|
||
|
||
Scenario: Accept safe arithmetic through restricted compilation
|
||
When I compile restricted code with "result = sum(range(10))"
|
||
Then no error should be raised from restricted compilation
|
||
|
||
# ──────────────────────────────────────────────────
|
||
# Section 2: Invalid code samples - Lambda AST validation
|
||
# ──────────────────────────────────────────────────
|
||
|
||
Scenario: Accept valid lambda expression
|
||
When I validate lambda ast with "lambda x: x * 2"
|
||
Then no error should be raised from lambda validation
|
||
|
||
Scenario: Reject non-lambda expression
|
||
When I validate lambda ast with "1 + 2"
|
||
Then a StreamRoutingError should be raised mentioning "must be a lambda"
|
||
|
||
Scenario: Reject syntactically invalid lambda
|
||
When I validate lambda ast with "lambda x:"
|
||
Then a StreamRoutingError should be raised mentioning "Invalid transform function syntax"
|
||
|
||
Scenario: Reject function call instead of lambda
|
||
When I validate lambda ast with "print('hello')"
|
||
Then a StreamRoutingError should be raised mentioning "must be a lambda"
|
||
|
||
# ──────────────────────────────────────────────────
|
||
# Section 2b: Lambda body – forbidden call detection (H2)
|
||
# These verify that _validate_lambda_ast walks the lambda body and
|
||
# rejects calls to dangerous built-ins that RestrictedPython misses.
|
||
# ──────────────────────────────────────────────────
|
||
|
||
Scenario: Reject lambda calling compile in body
|
||
When I validate lambda ast with "lambda x: compile('x', '', 'exec')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden call"
|
||
|
||
Scenario: Reject lambda calling getattr in body
|
||
When I validate lambda ast with "lambda x: getattr(x, '__class__')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden call"
|
||
|
||
Scenario: Reject lambda calling setattr in body
|
||
When I validate lambda ast with "lambda x: setattr(x, 'k', 'v')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden call"
|
||
|
||
Scenario: Reject lambda calling eval in body
|
||
When I validate lambda ast with "lambda x: eval('1+1')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden call"
|
||
|
||
Scenario: Reject lambda calling __import__ in body
|
||
When I validate lambda ast with "lambda x: __import__('os')"
|
||
Then a StreamRoutingError should be raised mentioning "Forbidden call"
|
||
|
||
Scenario: Accept safe lambda with method call
|
||
When I validate lambda ast with "lambda x: x.upper()"
|
||
Then no error should be raised from lambda validation
|
||
|
||
# ──────────────────────────────────────────────────
|
||
# Section 3: Invalid code samples - Python content sanitization
|
||
# ──────────────────────────────────────────────────
|
||
|
||
Scenario: Sanitize valid Python passes through unchanged
|
||
Given I have a plan service for sanitization tests
|
||
When I sanitize the python content "x = 1 + 2"
|
||
Then the sanitized content should be "x = 1 + 2"
|
||
And the sanitization error should be None
|
||
And the sanitization reason should be None
|
||
|
||
Scenario: Sanitize Python with markdown code fences
|
||
Given I have a plan service for sanitization tests
|
||
When I sanitize the python content with code fences wrapping "x = 42"
|
||
Then the sanitized content should be "x = 42"
|
||
And the sanitization error should be None
|
||
And the sanitization reason should be "code_fence_removed"
|
||
|
||
Scenario: Sanitize non-Python prose falls back to docstring wrapping
|
||
Given I have a plan service for sanitization tests
|
||
When I sanitize the python content "This is just plain English text, not code."
|
||
Then the sanitization error should be None
|
||
And the sanitization reason should be "docstring_wrapped"
|
||
|
||
Scenario: Sanitize irrecoverable syntax returns error
|
||
Given I have a plan service for sanitization tests
|
||
When I sanitize python content containing a null byte
|
||
Then the sanitization error should not be None
|
||
|
||
# ──────────────────────────────────────────────────
|
||
# Section 4: Edge case project structures - Project model validation
|
||
# ──────────────────────────────────────────────────
|
||
|
||
Scenario: Project rejects name with special characters
|
||
When I try to create a project with name "my@project"
|
||
Then a project validation error should be raised mentioning "alphanumeric"
|
||
|
||
Scenario: Project rejects name with slashes
|
||
When I try to create a project with name "my/project"
|
||
Then a project validation error should be raised mentioning "alphanumeric"
|
||
|
||
Scenario: Project rejects name with exclamation
|
||
When I try to create a project with name "project!"
|
||
Then a project validation error should be raised mentioning "alphanumeric"
|
||
|
||
Scenario: Project accepts name with hyphens underscores and spaces
|
||
When I create a project fixture with name "my-project_v2 final"
|
||
Then the project should be created with that name
|
||
|
||
Scenario: Project resolves relative path to absolute
|
||
When I create a project with relative path "relative/path"
|
||
Then the project fixture path should be absolute
|
||
|
||
Scenario: Project rejects empty name
|
||
When I try to create a project with empty name
|
||
Then a project validation error should be raised
|
||
|
||
# ──────────────────────────────────────────────────
|
||
# Section 5: Malformed input data - Change list coercion
|
||
# ──────────────────────────────────────────────────
|
||
|
||
Scenario: Coerce empty change list returns empty list
|
||
Given I have a plan service for coercion tests
|
||
When I coerce an empty change list
|
||
Then the coerced result should be an empty list
|
||
|
||
Scenario: Coerce mixed dict and Change entries
|
||
Given I have a plan service for coercion tests
|
||
When I coerce a list with one dict entry and one Change entry
|
||
Then the coerced result should have 2 changes
|
||
|
||
Scenario: Coerce non-list input raises PlanError
|
||
Given I have a plan service for coercion tests
|
||
When I try to coerce a non-list input
|
||
Then a PlanError should be raised mentioning "invalid change payload"
|
||
|
||
Scenario: Coerce list with non-change non-dict entry raises PlanError
|
||
Given I have a plan service for coercion tests
|
||
When I try to coerce a list containing an integer
|
||
Then a PlanError should be raised mentioning "non-change entry"
|
||
|
||
# ──────────────────────────────────────────────────
|
||
# Section 6: Malformed input data - ActionArgument parsing
|
||
# ──────────────────────────────────────────────────
|
||
|
||
Scenario: Parse argument with too few parts
|
||
When I try to parse action argument fixture "onlyname"
|
||
Then an argument fixture parse error should be raised
|
||
|
||
Scenario: Parse argument with invalid type
|
||
When I try to parse action argument fixture "name:badtype:required:desc"
|
||
Then an argument fixture parse error should be raised
|
||
|
||
Scenario: Parse argument with invalid requirement
|
||
When I try to parse action argument fixture "name:str:sometimes:desc"
|
||
Then an argument fixture parse error should be raised
|
||
|
||
Scenario: Parse argument with reserved keyword name
|
||
When I try to create argument with name "class"
|
||
Then the argument name should be accepted as valid identifier
|