HAL9000
f808abff86
Fix JSON syntax errors in .devcontainer/devcontainer.json (removed
invalid JS-style // comments) and .devcontainer/opencode.json (removed
90+ trailing commas). Apply auto-fixes for end-of-file and trailing
whitespace issues across 100+ files. Fix SIM105 ruff violations in
benchmarks/core_circuit_breaker_bench.py (use contextlib.suppress).
Note: The security fix from issue #7478 (validate_path startswith bypass)
was already delivered to master in commit e18ac5f2. This PR as currently
structured is non-atomic (35 commits across 10+ issues) and needs
significant restructure before merge. This commit only addresses the
CI/pre-commit failures.
ISSUES CLOSED: #7478
1139 lines
45 KiB
Gherkin
1139 lines
45 KiB
Gherkin
Feature: Consolidated Security
|
|
Combined scenarios from: security_eval, security_exceptions, security_readonly, security_secrets, security_templates
|
|
|
|
# ============================================================
|
|
# Originally from: security_eval.feature
|
|
# Feature: SEC1 - eval/exec removal security
|
|
# ============================================================
|
|
|
|
Scenario: Config with code injection attempt is rejected
|
|
Given a SimpleToolAgent configured with a code injection payload
|
|
When I attempt to process content through the injected agent
|
|
Then the agent should reject the code block with a routing error
|
|
And the error message should mention code blocks are not supported
|
|
|
|
|
|
Scenario: Malicious transform expression does not execute
|
|
Given the CleverAgents reactive system is available
|
|
When I configure a transform with a malicious eval expression
|
|
Then the transform should be rejected with a routing error
|
|
And the error message should mention unknown transform
|
|
|
|
|
|
Scenario: Valid config still works after eval removal
|
|
Given a SimpleToolAgent configured with a named operation "uppercase"
|
|
When I process "hello" through the named operation agent
|
|
Then the named operation result should be "HELLO"
|
|
|
|
|
|
Scenario: Named transform from registry works after eval removal
|
|
Given the CleverAgents reactive system is available
|
|
When I configure a transform with a registered named function "uppercase"
|
|
Then the named transform should produce the uppercased result
|
|
|
|
|
|
Scenario: Code block with import attempt is rejected
|
|
Given a SimpleToolAgent configured with an import injection payload
|
|
When I attempt to process content through the injected agent
|
|
Then the agent should reject the code block with a routing error
|
|
|
|
|
|
Scenario: Eval expression mimicking registry name is still rejected
|
|
Given the CleverAgents reactive system is available
|
|
When I configure a transform with expression "lambda x: __import__('os').system('rm -rf /')"
|
|
Then the transform should be rejected with a routing error
|
|
|
|
|
|
Scenario: Custom registered operation works
|
|
Given a SimpleToolAgent with a custom registered operation "reverse"
|
|
When I process "abcde" through the custom operation agent
|
|
Then the custom operation result should be "edcba"
|
|
|
|
|
|
Scenario: Custom registered transform works
|
|
Given the CleverAgents reactive system is available
|
|
And a custom transform "double" is registered
|
|
When I configure a transform with a registered named function "double"
|
|
Then the custom transform should produce the doubled result
|
|
|
|
|
|
Scenario: Config scanner detects eval in content
|
|
Given config content containing "value: eval('hack')"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report at least 1 violation
|
|
And the scanner should report a violation with token "eval("
|
|
And the violation severity should be "CRITICAL"
|
|
|
|
|
|
Scenario: Config scanner detects exec in content
|
|
Given config content containing "action: exec('import os')"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report at least 1 violation
|
|
And the scanner should report a violation with token "exec("
|
|
|
|
|
|
Scenario: Config scanner passes clean content
|
|
Given config content containing "name: safe-project"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report 0 violations
|
|
|
|
|
|
Scenario: Config scanner reports correct line numbers
|
|
Given multiline config content with a violation on line 3
|
|
When I scan the config content for security violations
|
|
Then the scanner should report a violation on line 3
|
|
|
|
|
|
Scenario: Config safety validation raises on dangerous content
|
|
Given config content containing "run: exec('bad')"
|
|
When I validate the config content for safety
|
|
Then a ConfigurationError should be raised
|
|
|
|
|
|
Scenario: Config safety validation passes clean content
|
|
Given config content containing "name: safe-project"
|
|
When I validate the config content for safety
|
|
Then no configuration error should be raised
|
|
|
|
|
|
Scenario: Config scanner skips comment lines
|
|
Given config content containing "# eval('this is just a comment')"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report 0 violations
|
|
|
|
|
|
Scenario: Config scanner skips INI-style semicolon comment lines
|
|
Given config content containing "; eval('this is an INI comment')"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report 0 violations
|
|
|
|
|
|
Scenario: Config scanner skips inline comments after hash
|
|
Given config content containing "name: safe-project # eval('just a note')"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report 0 violations
|
|
|
|
|
|
Scenario: Config scanner preserves hash inside quoted strings
|
|
Given config content containing "value: \"eval('inside quotes') # not a comment\""
|
|
When I scan the config content for security violations
|
|
Then the scanner should report at least 1 violation
|
|
And the scanner should report a violation with token "eval("
|
|
|
|
|
|
Scenario: Config scanner detects multiple violation types
|
|
Given config content with eval and subprocess patterns
|
|
When I scan the config content for security violations
|
|
Then the scanner should report at least 2 violations
|
|
|
|
|
|
Scenario: Config scanner scans multiple files
|
|
Given two temporary config files with violations
|
|
When I scan the config files for security violations
|
|
Then the scanner should report at least 2 violations
|
|
|
|
|
|
Scenario: Config scanner CLI shows usage when no args given
|
|
When I run the config scanner CLI with no arguments
|
|
Then the scanner CLI should return exit code 2
|
|
|
|
|
|
Scenario: Config scanner CLI reports violations found
|
|
When I run the config scanner CLI with a dirty temp file
|
|
Then the scanner CLI should return exit code 1
|
|
And the scanner CLI output should contain "violation"
|
|
|
|
|
|
Scenario: Config scanner CLI reports clean file
|
|
When I run the config scanner CLI with a clean temp file
|
|
Then the scanner CLI should return exit code 0
|
|
|
|
|
|
Scenario: Config scanner scan_file raises for missing file
|
|
When I scan a non-existent file path
|
|
Then a scanner FileNotFoundError should be raised
|
|
|
|
|
|
Scenario: Config scanner scan_file handles non-UTF-8 files gracefully
|
|
Given a temporary config file with non-UTF-8 encoding
|
|
When I scan the non-UTF-8 file for violations
|
|
Then the scan should complete without crashing
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: security_exceptions.feature
|
|
# Feature: Explicit exception handling and error classification
|
|
# ============================================================
|
|
|
|
Scenario: Classify a ValidationError as 422
|
|
Given a security exceptions test environment
|
|
Given a ValidationError with message "field is invalid"
|
|
When I classify the security exception
|
|
Then the error code should be 422
|
|
And the error code name should be "VALIDATION_FAILED"
|
|
And the error handling category should be "CLIENT"
|
|
|
|
|
|
Scenario: Classify a ResourceNotFoundError as 404
|
|
Given a security exceptions test environment
|
|
Given a ResourceNotFoundError for resource "plan" with id "01PLAN001"
|
|
When I classify the security exception
|
|
Then the error code should be 404
|
|
And the error code name should be "NOT_FOUND"
|
|
|
|
|
|
Scenario: Classify an AuthenticationError as 401
|
|
Given a security exceptions test environment
|
|
Given an AuthenticationError with message "invalid token"
|
|
When I classify the security exception
|
|
Then the error code should be 401
|
|
And the error code name should be "UNAUTHORIZED"
|
|
|
|
|
|
Scenario: Classify an AuthorizationError as 403
|
|
Given a security exceptions test environment
|
|
Given an AuthorizationError with message "access denied"
|
|
When I classify the security exception
|
|
Then the error code should be 403
|
|
And the error code name should be "FORBIDDEN"
|
|
|
|
|
|
Scenario: Classify a RateLimitError as 429
|
|
Given a security exceptions test environment
|
|
Given a RateLimitError with message "slow down"
|
|
When I classify the security exception
|
|
Then the error code should be 429
|
|
And the error code name should be "RATE_LIMITED"
|
|
|
|
|
|
Scenario: Classify a DatabaseError as 522
|
|
Given a security exceptions test environment
|
|
Given a DatabaseError with message "connection lost"
|
|
When I classify the security exception
|
|
Then the error code should be 522
|
|
And the error handling category should be "SERVER"
|
|
|
|
|
|
Scenario: Classify a ConfigurationError as 525
|
|
Given a security exceptions test environment
|
|
Given a ConfigurationError with message "missing key"
|
|
When I classify the security exception
|
|
Then the error code should be 525
|
|
|
|
|
|
Scenario: Classify a NetworkError as 524
|
|
Given a security exceptions test environment
|
|
Given a NetworkError with message "timeout"
|
|
When I classify the security exception
|
|
Then the error code should be 524
|
|
|
|
|
|
Scenario: Classify a FileSystemError as 523
|
|
Given a security exceptions test environment
|
|
Given a FileSystemError with message "permission denied"
|
|
When I classify the security exception
|
|
Then the error code should be 523
|
|
|
|
|
|
Scenario: Classify a ProviderError as 520
|
|
Given a security exceptions test environment
|
|
Given a ProviderError with message "model failed"
|
|
When I classify the security exception
|
|
Then the error code should be 520
|
|
|
|
|
|
Scenario: Classify a TokenLimitExceededError as 526
|
|
Given a security exceptions test environment
|
|
Given a TokenLimitExceededError with message "context too long"
|
|
When I classify the security exception
|
|
Then the error code should be 526
|
|
|
|
|
|
Scenario: Classify a BusinessRuleViolation as 400
|
|
Given a security exceptions test environment
|
|
Given a BusinessRuleViolation with message "invalid transition"
|
|
When I classify the security exception
|
|
Then the error code should be 400
|
|
|
|
|
|
Scenario: Classify a PlanError as 400
|
|
Given a security exceptions test environment
|
|
Given a PlanError with message "plan stuck"
|
|
When I classify the security exception
|
|
Then the error code should be 400
|
|
|
|
|
|
Scenario: Classify an unknown Exception as 500
|
|
Given a security exceptions test environment
|
|
Given a bare Exception with message "something broke"
|
|
When I classify the security exception
|
|
Then the error code should be 500
|
|
And the error handling category should be "SERVER"
|
|
|
|
|
|
Scenario: Classify an ExecutionError as 521
|
|
Given a security exceptions test environment
|
|
Given an ExecutionError with message "agent failed"
|
|
When I classify the security exception
|
|
Then the error code should be 521
|
|
|
|
|
|
Scenario: Classify a StreamRoutingError as 528
|
|
Given a security exceptions test environment
|
|
Given a StreamRoutingError with message "routing failed"
|
|
When I classify the security exception
|
|
Then the error code should be 528
|
|
|
|
|
|
Scenario: Classify a ModelNotAvailableError as 527
|
|
Given a security exceptions test environment
|
|
Given a ModelNotAvailableError with message "deprecated"
|
|
When I classify the security exception
|
|
Then the error code should be 527
|
|
|
|
|
|
Scenario: Classify a ResourceConflictError as 409
|
|
Given a security exceptions test environment
|
|
Given a ResourceConflictError with message "already exists"
|
|
When I classify the security exception
|
|
Then the error code should be 409
|
|
|
|
|
|
Scenario: Classify a MissingConfigurationError as 525
|
|
Given a security exceptions test environment
|
|
Given a MissingConfigurationError with message "key not set"
|
|
When I classify the security exception
|
|
Then the error code should be 525
|
|
|
|
|
|
Scenario: Classify an ExternalServiceError as 503
|
|
Given a security exceptions test environment
|
|
Given an ExternalServiceError with message "service down"
|
|
When I classify the security exception
|
|
Then the error code should be 503
|
|
|
|
# -- Secret redaction -----------------------------------------------------
|
|
|
|
|
|
Scenario: Redact API key from error details
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "api_key" and value "sk-abc123secret"
|
|
When I redact the error details
|
|
Then the redacted detail "api_key" should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact password from error details
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "password" and value "my-secret-pass"
|
|
When I redact the error details
|
|
Then the redacted detail "password" should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact OpenAI key pattern from string value
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "message" and value "Failed with key sk-abcdefghijklmnopqrstuvwxyz"
|
|
When I redact the error details
|
|
Then the redacted detail "message" should contain "***REDACTED***"
|
|
And the redacted detail "message" should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
|
|
Scenario: Redact JWT token pattern from string value
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "log" and value "auth eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9 failed"
|
|
When I redact the error details
|
|
Then the redacted detail "log" should contain "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact nested dict values
|
|
Given a security exceptions test environment
|
|
Given redaction test details with nested key "config" containing secret_key "abc123"
|
|
When I redact the error details
|
|
Then the redacted nested detail "config" key "secret_key" should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Non-sensitive values pass through unchanged
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "plan_id" and value "01PLAN001"
|
|
When I redact the error details
|
|
Then the redacted detail "plan_id" should be "01PLAN001"
|
|
|
|
|
|
Scenario: Redact GitHub PAT pattern from string value
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "msg" and value "token ghp_abcdefghijklmnopqrstuvwxyz0123456 expired"
|
|
When I redact the error details
|
|
Then the redacted detail "msg" should contain "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact value with redact_value function
|
|
Given a security exceptions test environment
|
|
Given a redaction test string value "key=sk-abcdefghijklmnopqrstuvwxyz"
|
|
When I redact the string value
|
|
Then the redacted string should contain "***REDACTED***"
|
|
|
|
# -- Wrapping unexpected exceptions ----------------------------------------
|
|
|
|
|
|
Scenario: Wrap a bare Exception into CleverAgentsError
|
|
Given a security exceptions test environment
|
|
Given a bare Exception with message "internal crash"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error should be a CleverAgentsError
|
|
And the wrapped error message should be "An unexpected error occurred"
|
|
And the wrapped error details should have key "exception_type"
|
|
|
|
|
|
Scenario: Wrap preserves CleverAgentsError as-is
|
|
Given a security exceptions test environment
|
|
Given a CleverAgentsError with message "known issue"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error message should be "known issue"
|
|
|
|
|
|
Scenario: Wrap merges context into CleverAgentsError details
|
|
Given a security exceptions test environment
|
|
Given a CleverAgentsError with message "known issue"
|
|
And a wrapping context with key "plan_id" and value "01PLAN001"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error details should have key "plan_id"
|
|
|
|
|
|
Scenario: Wrap with custom safe message
|
|
Given a security exceptions test environment
|
|
Given a bare Exception with message "segfault"
|
|
When I wrap the unexpected exception with message "Operation failed"
|
|
Then the wrapped error message should be "Operation failed"
|
|
|
|
|
|
Scenario: Wrap redacts secrets in context
|
|
Given a security exceptions test environment
|
|
Given a bare Exception with message "oops"
|
|
And a wrapping context with key "api_key" and value "sk-secret123456789012345"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error details key "api_key" should be "***REDACTED***"
|
|
|
|
# -- CLI formatting -------------------------------------------------------
|
|
|
|
|
|
Scenario: Format error for CLI output
|
|
Given a security exceptions test environment
|
|
Given a ValidationError with message "name is required"
|
|
When I classify the security exception
|
|
And I format the error for CLI
|
|
Then the error CLI output should contain "422"
|
|
And the error CLI output should contain "VALIDATION_FAILED"
|
|
And the error CLI output should contain "name is required"
|
|
|
|
|
|
Scenario: Format error with details for CLI output
|
|
Given a security exceptions test environment
|
|
Given a detailed CleverAgentsError "db fail" with detail "table" as "plans"
|
|
When I classify the security exception
|
|
And I format the error for CLI
|
|
Then the error CLI output should contain "table: plans"
|
|
|
|
# -- Secret in message (S1) ------------------------------------------------
|
|
|
|
|
|
Scenario: classify_error redacts secrets embedded in exception message
|
|
Given a security exceptions test environment
|
|
Given a CleverAgentsError with message "key=sk-abcdefghijklmnopqrstuvwxyz leaked"
|
|
When I classify the security exception
|
|
Then the error info message should contain "***REDACTED***"
|
|
And the error info message should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
|
|
Scenario: Format CLI output redacts secrets from message
|
|
Given a security exceptions test environment
|
|
Given a CleverAgentsError with message "tok sk-abcdefghijklmnopqrstuvwxyz end"
|
|
When I classify the security exception
|
|
And I format the error for CLI
|
|
Then the error CLI output should contain "***REDACTED***"
|
|
And the error CLI output should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
# -- List value redaction (B2/T1) ------------------------------------------
|
|
|
|
|
|
Scenario: Redact secrets inside list values
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "keys" and list values "sk-abcdefghijklmnopqrstuvwxyz" and "normal"
|
|
When I redact the error details
|
|
Then the redacted detail "keys" should be a list
|
|
And the redacted list "keys" should contain "***REDACTED***"
|
|
And the redacted list "keys" should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
# -- Non-string non-dict passthrough (T4) ----------------------------------
|
|
|
|
|
|
Scenario: Integer values pass through redaction unchanged
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "count" and integer value 42
|
|
When I redact the error details
|
|
Then the redacted detail "count" should be integer 42
|
|
|
|
|
|
Scenario: Boolean values pass through redaction unchanged
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "enabled" and boolean value true
|
|
When I redact the error details
|
|
Then the redacted detail "enabled" should be boolean true
|
|
|
|
# -- Non-mutation of original details (B1) ---------------------------------
|
|
|
|
|
|
Scenario: Wrap does not mutate the original exception details dict
|
|
Given a security exceptions test environment
|
|
Given a tracked CleverAgentsError "original" with details key "a" value "1"
|
|
And a wrapping context with key "b" and value "2"
|
|
When I wrap the unexpected exception
|
|
Then the original exception details reference should not have key "b"
|
|
And the wrapped error details should have key "b"
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: security_readonly.feature
|
|
# Feature: Read-only action enforcement
|
|
# ============================================================
|
|
|
|
Scenario: Read-only plan blocks a write-capable tool with tool name in error
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/file-write" with writes=True and read_only=False
|
|
And a tool execution context with plan_id "plan-ro-1" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then a ToolAccessDeniedError should be raised for the tool
|
|
And the access-denied message should contain "builtin/file-write"
|
|
And the access-denied message should contain "read-only"
|
|
|
|
|
|
Scenario: Read-only plan allows a read-only tool
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/search" with writes=False and read_only=True
|
|
And a tool execution context with plan_id "plan-ro-2" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then no enforcement error should be raised
|
|
|
|
|
|
Scenario: Non-read-only plan allows a write-capable tool
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/file-write" with writes=True and read_only=False
|
|
And a tool execution context with plan_id "plan-rw-1" and read_only False
|
|
When I enforce capabilities on the tool
|
|
Then no enforcement error should be raised
|
|
|
|
|
|
Scenario: Tool with writes=True is blocked even when cap.read_only is unset
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/hybrid-tool" with writes=True and read_only=False
|
|
And a tool execution context with plan_id "plan-ro-3" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then a ToolAccessDeniedError should be raised for the tool
|
|
And the access-denied message should contain "builtin/hybrid-tool"
|
|
|
|
|
|
Scenario: Tool with neither read_only nor writes passes on read-only plan
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/noop" with writes=False and read_only=False
|
|
And a tool execution context with plan_id "plan-ro-4" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then no enforcement error should be raised
|
|
|
|
# ── SkillContext.enforce_write_guard ──────────────────────────────────
|
|
|
|
|
|
Scenario: SkillContext enforce_write_guard blocks write tool with tool name
|
|
Given the read-only enforcement modules are available
|
|
Given a read-only SkillContext for plan "plan-sk-1"
|
|
When I call enforce_write_guard with tool_name "skill/writer"
|
|
Then a SkillExecutionError should be raised for the guard
|
|
And the write-guard error message should contain "skill/writer"
|
|
And the write-guard error type should be PERMISSION_DENIED
|
|
|
|
|
|
Scenario: SkillContext enforce_write_guard allows on writable context
|
|
Given the read-only enforcement modules are available
|
|
Given a writable SkillContext for plan "plan-sk-2"
|
|
When I call enforce_write_guard with tool_name "skill/writer"
|
|
Then no write-guard error should be raised
|
|
|
|
# ── ChangeSetCapture read-only enforcement ───────────────────────────
|
|
|
|
|
|
Scenario: ChangeSet builder rejects write entry when plan is read-only
|
|
Given the read-only enforcement modules are available
|
|
Given a ChangeSetCapture with plan_id "plan-cs-1" and read_only True
|
|
When I try to wrap a write-capable tool spec
|
|
Then a ReadOnlyViolationError should be raised for the capture
|
|
And the capture error message should contain "read-only"
|
|
|
|
|
|
Scenario: ChangeSet builder allows wrapping when not read-only
|
|
Given the read-only enforcement modules are available
|
|
Given a ChangeSetCapture with plan_id "plan-cs-2" and read_only False
|
|
When I try to wrap a write-capable tool spec
|
|
Then no capture error should be raised
|
|
|
|
|
|
Scenario: ChangeSet builder passes read-only tool through unchanged
|
|
Given the read-only enforcement modules are available
|
|
Given a ChangeSetCapture with plan_id "plan-cs-3" and read_only True
|
|
When I wrap a read-only tool spec
|
|
Then the wrapped tool spec should be returned unchanged
|
|
|
|
# ── ExecuteStubActor read-only propagation ─────────────────────────────
|
|
|
|
|
|
Scenario: ExecuteStubActor propagates read_only to ChangeSetCapture
|
|
Given the read-only enforcement modules are available
|
|
When I execute via ExecuteStubActor with read_only True
|
|
Then the stub actor should create a read-only ChangeSetCapture
|
|
|
|
|
|
Scenario: ExecuteStubActor allows writes when not read-only
|
|
Given the read-only enforcement modules are available
|
|
When I execute via ExecuteStubActor with read_only False
|
|
Then the stub actor should create a writable ChangeSetCapture
|
|
|
|
# ── File write tool blocked ──────────────────────────────────────────
|
|
|
|
|
|
Scenario: File write tool blocked under read-only plan via ToolRuntime
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/file-write" with writes=True and read_only=False
|
|
And a tool execution context with plan_id "plan-fw-1" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then a ToolAccessDeniedError should be raised for the tool
|
|
And the access-denied message should contain "builtin/file-write"
|
|
|
|
# ── Git push tool blocked ────────────────────────────────────────────
|
|
|
|
|
|
Scenario: Git push tool blocked under read-only plan via ToolRuntime
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "git/push" with writes=True and read_only=False
|
|
And a tool execution context with plan_id "plan-gp-1" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then a ToolAccessDeniedError should be raised for the tool
|
|
And the access-denied message should contain "git/push"
|
|
|
|
# ── Read-only SkillContext blocks write tool ───────────────────────────
|
|
|
|
|
|
Scenario: Read-only SkillContext blocks write tool via enforce_write_guard
|
|
Given the read-only enforcement modules are available
|
|
Given a read-only SkillContext for plan "plan-compat-1"
|
|
When I call enforce_write_guard with tool_name "test/write-tool"
|
|
Then a SkillExecutionError should be raised for the guard
|
|
And the write-guard error message should contain "test/write-tool"
|
|
|
|
|
|
Scenario: Writable SkillContext allows write tool
|
|
Given the read-only enforcement modules are available
|
|
Given a writable SkillContext for plan "plan-compat-2"
|
|
When I call enforce_write_guard with tool_name "test/write-tool"
|
|
Then no write-guard error should be raised
|
|
|
|
# ── Plan -> ToolExecutionContext propagation ──────────────────────────
|
|
|
|
|
|
Scenario: ToolExecutionContext carries plan_read_only from plan
|
|
Given the read-only enforcement modules are available
|
|
Given a Plan domain model with read_only=True and plan_id "plan-ex-1"
|
|
When I create a ToolExecutionContext from the plan
|
|
Then the propagated context plan_read_only should be True
|
|
|
|
|
|
Scenario: ToolExecutionContext writable for writable plan
|
|
Given the read-only enforcement modules are available
|
|
Given a Plan domain model with read_only=False and plan_id "plan-ex-2"
|
|
When I create a ToolExecutionContext from the plan
|
|
Then the propagated context plan_read_only should be False
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: security_secrets.feature
|
|
# Feature: SEC5 - Secrets masking and validation
|
|
# ============================================================
|
|
|
|
Scenario: Redact OpenAI API key pattern
|
|
Given a string containing "sk-proj-abc123def456ghi789"
|
|
When I redact the string
|
|
Then the redacted value should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact Anthropic API key pattern
|
|
Given a string containing "sk-ant-api03-xyzzy1234567890abcdef"
|
|
When I redact the string
|
|
Then the redacted value should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact token pattern
|
|
Given a string containing "tok_01HXYZ1234567890ABCDEF"
|
|
When I redact the string
|
|
Then the redacted value should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact Google/Gemini API key pattern
|
|
Given a string containing "AIzaSyA1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q"
|
|
When I redact the string
|
|
Then the redacted value should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Mixed text with Gemini API key is partially redacted
|
|
Given a string containing "Using key AIzaSyA1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q for Gemini"
|
|
When I redact the string
|
|
Then the redacted value should contain "***REDACTED***"
|
|
And the redacted value should contain "Using key"
|
|
And the redacted value should contain "for Gemini"
|
|
And the redacted value should not contain "AIzaSy"
|
|
|
|
|
|
Scenario: Redact generic bearer token
|
|
Given a string containing "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJz"
|
|
When I redact the string
|
|
Then the redacted value should contain "***REDACTED***"
|
|
|
|
|
|
Scenario: Non-secret strings are not redacted
|
|
Given a string containing "hello world this is normal text"
|
|
When I redact the string
|
|
Then the redacted value should be "hello world this is normal text"
|
|
|
|
|
|
Scenario: Mixed text with embedded secret is partially redacted
|
|
Given a string containing "My key is sk-proj-abc123def456ghi789 and name is Bob"
|
|
When I redact the string
|
|
Then the redacted value should contain "***REDACTED***"
|
|
And the redacted value should contain "My key is"
|
|
And the redacted value should contain "and name is Bob"
|
|
|
|
|
|
Scenario: Empty string is returned unchanged
|
|
Given a string containing ""
|
|
When I redact the string
|
|
Then the redacted value should be empty
|
|
|
|
# --- Dict redaction ---
|
|
|
|
|
|
Scenario: Dict with sensitive key names are masked
|
|
Given a dict with key "api_key" and value "sk-proj-abc123"
|
|
When I redact the dict
|
|
Then the key "api_key" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with password key is masked
|
|
Given a dict with key "password" and value "mysecretpass"
|
|
When I redact the dict
|
|
Then the key "password" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with secret key is masked
|
|
Given a dict with key "secret" and value "topsecretvalue"
|
|
When I redact the dict
|
|
Then the key "secret" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with token key is masked
|
|
Given a dict with key "token" and value "sometoken123"
|
|
When I redact the dict
|
|
Then the key "token" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with credential key is masked
|
|
Given a dict with key "credential" and value "cred-value"
|
|
When I redact the dict
|
|
Then the key "credential" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with non-sensitive key is not masked
|
|
Given a dict with key "username" and value "bob"
|
|
When I redact the dict
|
|
Then the key "username" should have value "bob"
|
|
|
|
|
|
Scenario: Nested dict values are redacted recursively
|
|
Given a nested dict with inner key "api_key" and value "sk-proj-nested"
|
|
When I redact the dict
|
|
Then the nested key "api_key" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with pattern-matched value is masked regardless of key name
|
|
Given a dict with key "config_value" and value "sk-ant-api03-somekey123"
|
|
When I redact the dict
|
|
Then the key "config_value" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict redaction respects show_secrets flag
|
|
Given a dict with key "api_key" and value "sk-proj-abc123"
|
|
When I redact the dict with show_secrets enabled
|
|
Then the key "api_key" should have value "sk-proj-abc123"
|
|
|
|
# --- Sensitive key detection ---
|
|
|
|
|
|
Scenario Outline: Sensitive key names are detected
|
|
When I check if "<key>" is a sensitive key
|
|
Then the sensitivity check should be true
|
|
|
|
Examples:
|
|
| key |
|
|
| api_key |
|
|
| API_KEY |
|
|
| password |
|
|
| secret |
|
|
| token |
|
|
| credential |
|
|
| openai_api_key |
|
|
| ANTHROPIC_API_KEY |
|
|
| access_token |
|
|
| auth_token |
|
|
| private_key |
|
|
|
|
|
|
Scenario Outline: Non-sensitive key names are not flagged
|
|
When I check if "<key>" is a sensitive key
|
|
Then the sensitivity check should be false
|
|
|
|
Examples:
|
|
| key |
|
|
| username |
|
|
| email |
|
|
| name |
|
|
| description |
|
|
| project_id |
|
|
| token_count |
|
|
|
|
# --- Database URL masking ---
|
|
|
|
|
|
Scenario: SQLite URL is not masked
|
|
Given a database URL "sqlite:///path/to/db.sqlite"
|
|
When I mask the database URL
|
|
Then the masked URL should be "sqlite:///path/to/db.sqlite"
|
|
|
|
|
|
Scenario: PostgreSQL URL with credentials is masked
|
|
Given a database URL "postgresql://admin:s3cret@localhost:5432/mydb"
|
|
When I mask the database URL
|
|
Then the masked URL should contain "***"
|
|
And the masked URL should contain "localhost"
|
|
And the masked URL should not contain "s3cret"
|
|
|
|
# --- Global show_secrets flag ---
|
|
|
|
|
|
Scenario: Global show_secrets defaults to false
|
|
When I check the global show_secrets flag
|
|
Then the flag should be false
|
|
|
|
|
|
Scenario: Global show_secrets can be toggled
|
|
When I set the global show_secrets flag to true
|
|
Then the flag should be true
|
|
When I set the global show_secrets flag to false
|
|
Then the flag should be false
|
|
|
|
# --- structlog processor ---
|
|
|
|
|
|
Scenario: Structlog masking processor redacts event dict values
|
|
Given a structlog event dict with key "api_key" and value "sk-proj-secret123"
|
|
When the masking processor is applied
|
|
Then the event dict key "api_key" should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Structlog masking processor redacts event message
|
|
Given a structlog event dict with event "Failed with key sk-ant-api03-xyz"
|
|
When the masking processor is applied
|
|
Then the event message should contain "***REDACTED***"
|
|
And the event message should not contain "sk-ant-api03-xyz"
|
|
|
|
|
|
Scenario: Structlog masking processor passes non-secret events
|
|
Given a structlog event dict with event "User logged in successfully"
|
|
When the masking processor is applied
|
|
Then the event message should be "User logged in successfully"
|
|
|
|
# --- CLI error detail redaction ---
|
|
|
|
|
|
Scenario: Error details with secret values are redacted
|
|
Given error details with key "api_key" and value "sk-proj-leaked"
|
|
When I redact the error details for CLI display
|
|
Then the redacted details key "api_key" should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Error details with non-secret values are preserved
|
|
Given error details with key "plan_id" and value "01HXR123"
|
|
When I redact the error details for CLI display
|
|
Then the redacted details key "plan_id" should be "01HXR123"
|
|
|
|
# --- Secret pattern registration ---
|
|
|
|
|
|
Scenario: Custom secret pattern can be registered
|
|
Given the default secret patterns
|
|
When I register a custom pattern "custom_[a-z0-9]{16}"
|
|
And I redact the string "my custom_abcdef1234567890 key"
|
|
Then the redacted value should contain "***REDACTED***"
|
|
And the redacted value should not contain "custom_abcdef1234567890"
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: security_templates.feature
|
|
# Feature: Secure template rendering
|
|
# ============================================================
|
|
|
|
Scenario: Render a simple placeholder
|
|
Given a secure template test environment
|
|
Given the secure template text "{greeting} {name}"
|
|
And a secure template context key "greeting" with value "Hello"
|
|
And a secure template context key "name" with value "World"
|
|
When I render the secure template
|
|
Then the secure template output should be "Hello World"
|
|
|
|
|
|
Scenario: Render with missing placeholder leaves it intact
|
|
Given a secure template test environment
|
|
Given the secure template text "Hello {name}, your role is {role}"
|
|
And a secure template context key "name" with value "Alice"
|
|
When I render the secure template
|
|
Then the secure template output should be "Hello Alice, your role is {role}"
|
|
|
|
|
|
Scenario: Render with empty context returns template as-is
|
|
Given a secure template test environment
|
|
Given the secure template text "No placeholders here"
|
|
When I render the secure template
|
|
Then the secure template output should be "No placeholders here"
|
|
|
|
|
|
Scenario: Render with numeric values converts to string
|
|
Given a secure template test environment
|
|
Given the secure template text "Count: {count}"
|
|
And a secure template context key "count" with value "42"
|
|
When I render the secure template
|
|
Then the secure template output should be "Count: 42"
|
|
|
|
# -- Security rejection ---------------------------------------------------
|
|
|
|
|
|
Scenario: Reject attribute access in template
|
|
Given a secure template test environment
|
|
Given the secure template text "{obj.__class__.__name__}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject index access in template
|
|
Given a secure template test environment
|
|
Given the secure template text "{items[0]}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject format spec in template
|
|
Given a secure template test environment
|
|
Given the secure template text "{value:>10}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject conversion flag in template
|
|
Given a secure template test environment
|
|
Given the secure template text "{value!r}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject function call in template
|
|
Given a secure template test environment
|
|
Given the secure template text "{fn()}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject Jinja2 expression delimiters
|
|
Given a secure template test environment
|
|
Given the secure template text "{{ config }}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject Jinja2 block delimiters
|
|
Given a secure template test environment
|
|
Given the secure template text "{% for x in items %}x{% endfor %}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
# -- Size limits -----------------------------------------------------------
|
|
|
|
|
|
Scenario: Reject template exceeding max length
|
|
Given a secure template test environment
|
|
Given a secure template that is 11000 characters long
|
|
When I render the secure template expecting a size error
|
|
Then the secure template error should mention "exceeds maximum"
|
|
|
|
|
|
Scenario: Reject output exceeding max length
|
|
Given a secure template test environment
|
|
Given the secure template text "{big}"
|
|
And a secure template config with max output length of 20
|
|
And a secure template context key "big" with value "This value is definitely longer than twenty characters"
|
|
When I render the secure template expecting a size error
|
|
Then the secure template error should mention "exceeds maximum"
|
|
|
|
# -- Allowlist enforcement ------------------------------------------------
|
|
|
|
|
|
Scenario: Reject template key not in allowlist
|
|
Given a secure template test environment
|
|
Given the secure template text "{name} {secret}"
|
|
And a secure template with allowed keys "name"
|
|
And a secure template context key "name" with value "Alice"
|
|
And a secure template context key "secret" with value "hidden"
|
|
When I render the secure template expecting a validation error
|
|
Then the secure template error should mention "disallowed"
|
|
|
|
|
|
Scenario: Accept template key that is in allowlist
|
|
Given a secure template test environment
|
|
Given the secure template text "{name}"
|
|
And a secure template with allowed keys "name"
|
|
And a secure template context key "name" with value "Alice"
|
|
When I render the secure template
|
|
Then the secure template output should be "Alice"
|
|
|
|
|
|
Scenario: Open allowlist allows any simple key
|
|
Given a secure template test environment
|
|
Given the secure template text "{anything}"
|
|
And the secure template has no key restrictions
|
|
And a secure template context key "anything" with value "works"
|
|
When I render the secure template
|
|
Then the secure template output should be "works"
|
|
|
|
# -- Pre-validation -------------------------------------------------------
|
|
|
|
|
|
Scenario: Validate clean template returns no errors
|
|
Given a secure template test environment
|
|
Given the secure template text "{name} {role}"
|
|
When I validate the secure template
|
|
Then the secure template validation should be empty
|
|
|
|
|
|
Scenario: Validate unsafe template returns errors
|
|
Given a secure template test environment
|
|
Given the secure template text "{obj.__class__}"
|
|
When I validate the secure template
|
|
Then the secure template validation should not be empty
|
|
And the secure template validation should mention "Unsafe"
|
|
|
|
|
|
Scenario: Validate template with unknown keys returns errors
|
|
Given a secure template test environment
|
|
Given the secure template text "{name} {secret}"
|
|
And a secure template with allowed keys "name"
|
|
When I validate the secure template
|
|
Then the secure template validation should not be empty
|
|
And the secure template validation should mention "Unknown"
|
|
|
|
|
|
Scenario: Validate oversized template returns errors
|
|
Given a secure template test environment
|
|
Given a secure template that is 11000 characters long
|
|
When I validate the secure template
|
|
Then the secure template validation should not be empty
|
|
And the secure template validation should mention "exceeds"
|
|
|
|
# -- Convenience function -------------------------------------------------
|
|
|
|
|
|
Scenario: render_template_secure convenience function works
|
|
Given a secure template test environment
|
|
Given the secure template text "{greeting}"
|
|
And a secure template context key "greeting" with value "Hi"
|
|
When I render via the secure convenience function
|
|
Then the secure template output should be "Hi"
|
|
|
|
|
|
Scenario: render_template_secure rejects unsafe input
|
|
Given a secure template test environment
|
|
Given the secure template text "{obj.attr}"
|
|
When I render via the secure convenience function expecting an error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
# -- Config property access ------------------------------------------------
|
|
|
|
|
|
Scenario: Renderer exposes active configuration
|
|
Given a secure template test environment
|
|
Given a secure template with allowed keys "name"
|
|
When I create a renderer with those settings
|
|
Then the renderer config allowed keys should contain "name"
|
|
|
|
# -- Validate with matching allowlist (no unknown keys) -------------------
|
|
|
|
|
|
Scenario: Validate template where all keys match allowlist
|
|
Given a secure template test environment
|
|
Given the secure template text "{name}"
|
|
And a secure template with allowed keys "name"
|
|
When I validate the secure template
|
|
Then the secure template validation should be empty
|
|
|
|
# -- Rejected keys without logging ----------------------------------------
|
|
|
|
|
|
Scenario: Reject unknown key without logging when log_rejected is false
|
|
Given a secure template test environment
|
|
Given the secure template text "{name} {secret}"
|
|
And a secure template config with logging disabled
|
|
And a secure template context key "name" with value "Alice"
|
|
And a secure template context key "secret" with value "hidden"
|
|
When I render the secure template expecting a validation error
|
|
Then the secure template error should mention "disallowed"
|
|
|
|
# -- reject_unknown_keys=False path ----------------------------------------
|
|
|
|
|
|
Scenario: Allow unknown keys when reject_unknown_keys is disabled
|
|
Given a secure template test environment
|
|
Given the secure template text "{name} {extra}"
|
|
And a secure template config that permits unknown keys
|
|
And a secure template context key "name" with value "Alice"
|
|
And a secure template context key "extra" with value "data"
|
|
When I render the secure template
|
|
Then the secure template output should be "Alice data"
|
|
|
|
# -- Legacy renderer with config passthrough --------------------------------
|
|
|
|
|
|
Scenario: Legacy TemplateRenderer accepts config passthrough
|
|
Given a secure template test environment
|
|
Given the secure template text "{name}"
|
|
And a secure template with allowed keys "name"
|
|
And a secure template context key "name" with value "Carol"
|
|
When I render via the legacy TemplateRenderer with config
|
|
Then the secure template output should be "Carol"
|
|
|
|
# -- Legacy renderer backward compat --------------------------------------
|
|
|
|
|
|
Scenario: Legacy TemplateRenderer returns raw on unsafe template
|
|
Given a secure template test environment
|
|
Given the secure template text "{obj.__class__}"
|
|
And a secure template context key "obj" with value "test"
|
|
When I render via the legacy TemplateRenderer
|
|
Then the secure template output should be "{obj.__class__}"
|
|
|
|
|
|
Scenario: Legacy TemplateRenderer renders safe templates
|
|
Given a secure template test environment
|
|
Given the secure template text "{name}"
|
|
And a secure template context key "name" with value "Bob"
|
|
When I render via the legacy TemplateRenderer
|
|
Then the secure template output should be "Bob"
|