cbf8bcc993
CI / lint (push) Successful in 15s
CI / build (push) Successful in 26s
CI / quality (push) Successful in 35s
CI / typecheck (push) Successful in 40s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 43s
CI / unit_tests (push) Successful in 3m5s
CI / integration_tests (push) Successful in 3m35s
CI / docker (push) Successful in 1m3s
CI / e2e_tests (push) Successful in 5m28s
CI / coverage (push) Successful in 8m5s
CI / benchmark-publish (push) Successful in 20m11s
## Summary Add `robot/e2e/m5_acceptance.robot` with **21 zero-mock E2E test cases** (in addition to existing M5 test suite) covering all M5 (v3.4.0) acceptance criteria: 1. **Context Assembly** — add/list/show/clear files in the context pipeline 2. **Context Scaling** — 10,000+ file project setup with simulate plumbing *(structural)* 3. **Context Policy Configuration** — per-view include/exclude paths, file-size limits 4. **Budget Enforcement** — max_file_size / max_total_size constraint storage *(structural)* 5. **Context Analysis** — ACMS pipeline inspect (tier schema) and simulate (JSON schema) *(structural)* 6. **Plan Execution** — real LLM calls via `openai/gpt-4o-mini` (`plan use` + `plan resume`) ### Structural vs. Behavioural Scope Tests in sections 1b–4 that use `project context simulate` or `inspect` are **structural / plumbing validations** — they verify CLI execution, JSON serialization, and stored configuration but do **not** exercise actual ACMS indexing or budget enforcement because the `ContextTierService` is an in-memory singleton that starts empty per CLI process. Each affected test has a `[Documentation]` note explaining this limitation. Behavioural ACMS validation is deferred until the full indexing pipeline is wired. ### Production Bug Fixes | Fix | File | Description | |-----|------|-------------| | `session.flush()` → `session.commit()` | `project_context.py` | Policy changes silently lost on `session.close()` | | `contextlib.suppress` rollback wrapper | `project_context.py` | Prevents rollback failure from masking original commit exception | | Add `session_factory` DI provider | `container.py` | `project context` commands hit `AttributeError` | | `providers.Factory` → `providers.Singleton` | `container.py` | Avoid creating duplicate engines per call | | Add Gemini API key pattern | `redaction.py` | `AIzaSy...` keys now redacted in logs | ### Review Feedback Addressed (Tenth Pass — @CoreRasurae Review #2410) | # | Severity | Finding | Fix | |---|----------|---------|-----| | P3-1 | Medium | "Clear Context" test tautological — never asserts files were present before clearing | Added `Should Contain ${list_before.stdout} config.py` precondition check after `context-load` and before `clear` | | P3-2 | Medium | Policy/budget verification uses substring matching (`Should Contain 262144`) | Replaced with `Extract JSON From Stdout` + `$rv.get('max_file_size') == 262144` parsed JSON assertions using `resolved_view` dict access | | P3-3 | Medium | Plan resume doesn't verify `phase` value, only existence | Added `Should Not Be Equal As Strings ${phase} queued` assertion to verify plan transitioned from queued | | P3-4 | Medium | Plan JSON extraction inconsistency (`rindex` vs `Extract JSON From Stdout`) | Replaced fragile `rindex`-based extraction with `Extract JSON From Stdout` keyword for consistency | | P3-6 | Medium | Context show summary weak content assertions | Added `Should Not Contain` guards against traceback/error output to reject false positives | | P3-8 | Medium | `_SafeSession` singleton may accumulate dirty state after rollback | Changed `_SafeSession.close()` from pure no-op to `real.rollback()` to reset session state between calls | | P3-14 | Medium | No test for `_save_policy_json` rollback path | Added BDD scenario "Save policy rollback re-raises after commit failure" with monkey-patched commit | | P3-15 | Medium | No test for `_save_policy_json` on nonexistent project | Added BDD scenario "Save policy on nonexistent project row updates zero rows" verifying silent 0-row behavior | | P4-1 | Low | Plan resume TRY/EXCEPT swallows assertion details | Moved field assertions outside TRY block; TRY only guards JSON extraction | | P4-2 | Low | `Safe Parse Json Field` logs stale error context | Fixed to track and report both Strategy 1 and Strategy 2 error contexts separately | | P4-4 | Low | SQLite WAL/SHM files not cleaned in regression test | Added cleanup loop for `-wal` and `-shm` suffixes alongside `.db` file | ### Deferred Items (Out of Scope) | ID | Severity | Reason | |----|----------|--------| | P2-1 | High | `execution_environment` silently dropped on subsequent `context set` — pre-existing production code bug in `_write_policy()`, not introduced by this PR | | P2-2 | High | Unhandled `ValidationError` on corrupt policy blob — pre-existing `_read_policy()` code, not changed by this PR | | P2-3 | High | Silent no-op UPDATE when `ns_projects` row missing — pre-existing `_save_policy_json` logic; this PR only changed error handling | | P3-5 | Medium | Structural tests cannot detect regressions — already honestly documented in every affected test's `[Documentation]` block | | P3-7 | Medium | View inheritance/override behavior not tested — nice-to-have, not in ticket acceptance criteria | | P3-9 | Medium | `context_set` double-writes when `execution_environment` set — pre-existing production logic | | P3-10 | Medium | `budget_tokens=0` silently replaced by default (falsy `or`) — pre-existing production code bug | | P3-11 | Medium | `context set` replaces entire view instead of merging — pre-existing design choice | | P3-12 | Medium | GEMINI_API_KEY propagated but potentially unused — security-first: propagating for redaction testing | | P3-13 | Medium | `reset_container()` doesn't dispose Singleton resources — pre-existing container lifecycle issue | | M5 | Medium | `_build_session_factory` engine never disposed — production code architecture, out of scope for testing ticket | | M6 | Medium | Missing `check_same_thread`/`isolation_level` — production code architecture, out of scope for testing ticket | | L1 | Low | `plan resume` not in spec CLI synopsis — informational | | L2 | Low | Context summary assertions depend on exact CLI wording — acceptable stability risk | | L3 | Low | Gemini regex minimum length slightly loose — acceptable security-first trade-off | | L4 | Low | Missing Google OAuth2 credential patterns — out of scope for this PR | | P4-3 | Low | `Run CLI` keyword duplicated — different purpose (uses `${WS}` as default cwd), not a true duplicate | | P4-5–P4-9 | Low | Various additional E2E coverage gaps — nice-to-have, not in ticket acceptance criteria | ### Quality Gates | Gate | Result | |------|--------| | lint | PASS | | typecheck | PASS (0 errors) | | unit_tests | **393/393** features, 11,210 scenarios | | integration_tests | **1,576/1,576** | | e2e_tests | **37/37** (21 M5 + 12 M6 + 2 smoke + 2 M1) | | coverage_report | **97%** (threshold: 97%) | ### Files Changed | File | Change | |------|--------| | `robot/e2e/m5_acceptance.robot` | **NEW** — 21 E2E test cases with honest structural documentation, parsed JSON assertions, prerequisite skip guards on all sections, safe assertion messages | | `robot/e2e/common_e2e.resource` | `on_timeout=kill` + return code checks + safe key evaluation via `os.environ.get` + fixed stale error logging in `Safe Parse Json Field` | | `robot/e2e/m1_acceptance.robot` | `on_timeout=kill` on git log | | `robot/e2e/m2_acceptance.robot` | `on_timeout=kill` + return code checks + safe assertion messages (no stderr embedding) | | `src/cleveragents/application/container.py` | Add `_build_session_factory` + `session_factory` Singleton | | `src/cleveragents/cli/commands/project_context.py` | `flush()` → `commit()` + `contextlib.suppress` rollback | | `src/cleveragents/shared/redaction.py` | Add Gemini API key pattern | | `noxfile.py` | Propagate `GEMINI_API_KEY` in e2e_tests | | `CHANGELOG.md` | 4 entries for #745 | | `features/application_container_coverage_boost.feature` | Updated title + 3 scenarios | | `features/steps/application_container_coverage_boost_steps.py` | Step defs for `_build_session_factory` | | `features/consolidated_security.feature` | 2 Gemini API key redaction scenarios | | `features/project_context_cli_coverage_boost.feature` | `flush()→commit()` regression test + rollback path + nonexistent project tests | | `features/steps/project_context_cli_coverage_boost_steps.py` | Separate engines for regression test + `try/finally` cleanup + `_SafeSession.close()` state reset + rollback/nonexistent test steps + WAL/SHM cleanup | Closes #745 ISSUES CLOSED: #745 Reviewed-on: #811 Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com> Co-authored-by: Rui Hu <rui.hu@cleverthis.com> Co-committed-by: Rui Hu <rui.hu@cleverthis.com>
1140 lines
45 KiB
Gherkin
1140 lines
45 KiB
Gherkin
Feature: Consolidated Security
|
|
Combined scenarios from: security_eval, security_exceptions, security_readonly, security_secrets, security_templates
|
|
|
|
# ============================================================
|
|
# Originally from: security_eval.feature
|
|
# Feature: SEC1 - eval/exec removal security
|
|
# ============================================================
|
|
|
|
Scenario: Config with code injection attempt is rejected
|
|
Given a SimpleToolAgent configured with a code injection payload
|
|
When I attempt to process content through the injected agent
|
|
Then the agent should reject the code block with a routing error
|
|
And the error message should mention code blocks are not supported
|
|
|
|
|
|
Scenario: Malicious transform expression does not execute
|
|
Given the CleverAgents reactive system is available
|
|
When I configure a transform with a malicious eval expression
|
|
Then the transform should be rejected with a routing error
|
|
And the error message should mention unknown transform
|
|
|
|
|
|
Scenario: Valid config still works after eval removal
|
|
Given a SimpleToolAgent configured with a named operation "uppercase"
|
|
When I process "hello" through the named operation agent
|
|
Then the named operation result should be "HELLO"
|
|
|
|
|
|
Scenario: Named transform from registry works after eval removal
|
|
Given the CleverAgents reactive system is available
|
|
When I configure a transform with a registered named function "uppercase"
|
|
Then the named transform should produce the uppercased result
|
|
|
|
|
|
Scenario: Code block with import attempt is rejected
|
|
Given a SimpleToolAgent configured with an import injection payload
|
|
When I attempt to process content through the injected agent
|
|
Then the agent should reject the code block with a routing error
|
|
|
|
|
|
Scenario: Eval expression mimicking registry name is still rejected
|
|
Given the CleverAgents reactive system is available
|
|
When I configure a transform with expression "lambda x: __import__('os').system('rm -rf /')"
|
|
Then the transform should be rejected with a routing error
|
|
|
|
|
|
Scenario: Custom registered operation works
|
|
Given a SimpleToolAgent with a custom registered operation "reverse"
|
|
When I process "abcde" through the custom operation agent
|
|
Then the custom operation result should be "edcba"
|
|
|
|
|
|
Scenario: Custom registered transform works
|
|
Given the CleverAgents reactive system is available
|
|
And a custom transform "double" is registered
|
|
When I configure a transform with a registered named function "double"
|
|
Then the custom transform should produce the doubled result
|
|
|
|
|
|
Scenario: Config scanner detects eval in content
|
|
Given config content containing "value: eval('hack')"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report at least 1 violation
|
|
And the scanner should report a violation with token "eval("
|
|
And the violation severity should be "CRITICAL"
|
|
|
|
|
|
Scenario: Config scanner detects exec in content
|
|
Given config content containing "action: exec('import os')"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report at least 1 violation
|
|
And the scanner should report a violation with token "exec("
|
|
|
|
|
|
Scenario: Config scanner passes clean content
|
|
Given config content containing "name: safe-project"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report 0 violations
|
|
|
|
|
|
Scenario: Config scanner reports correct line numbers
|
|
Given multiline config content with a violation on line 3
|
|
When I scan the config content for security violations
|
|
Then the scanner should report a violation on line 3
|
|
|
|
|
|
Scenario: Config safety validation raises on dangerous content
|
|
Given config content containing "run: exec('bad')"
|
|
When I validate the config content for safety
|
|
Then a ConfigurationError should be raised
|
|
|
|
|
|
Scenario: Config safety validation passes clean content
|
|
Given config content containing "name: safe-project"
|
|
When I validate the config content for safety
|
|
Then no configuration error should be raised
|
|
|
|
|
|
Scenario: Config scanner skips comment lines
|
|
Given config content containing "# eval('this is just a comment')"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report 0 violations
|
|
|
|
|
|
Scenario: Config scanner skips INI-style semicolon comment lines
|
|
Given config content containing "; eval('this is an INI comment')"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report 0 violations
|
|
|
|
|
|
Scenario: Config scanner skips inline comments after hash
|
|
Given config content containing "name: safe-project # eval('just a note')"
|
|
When I scan the config content for security violations
|
|
Then the scanner should report 0 violations
|
|
|
|
|
|
Scenario: Config scanner preserves hash inside quoted strings
|
|
Given config content containing "value: \"eval('inside quotes') # not a comment\""
|
|
When I scan the config content for security violations
|
|
Then the scanner should report at least 1 violation
|
|
And the scanner should report a violation with token "eval("
|
|
|
|
|
|
Scenario: Config scanner detects multiple violation types
|
|
Given config content with eval and subprocess patterns
|
|
When I scan the config content for security violations
|
|
Then the scanner should report at least 2 violations
|
|
|
|
|
|
Scenario: Config scanner scans multiple files
|
|
Given two temporary config files with violations
|
|
When I scan the config files for security violations
|
|
Then the scanner should report at least 2 violations
|
|
|
|
|
|
Scenario: Config scanner CLI shows usage when no args given
|
|
When I run the config scanner CLI with no arguments
|
|
Then the scanner CLI should return exit code 2
|
|
|
|
|
|
Scenario: Config scanner CLI reports violations found
|
|
When I run the config scanner CLI with a dirty temp file
|
|
Then the scanner CLI should return exit code 1
|
|
And the scanner CLI output should contain "violation"
|
|
|
|
|
|
Scenario: Config scanner CLI reports clean file
|
|
When I run the config scanner CLI with a clean temp file
|
|
Then the scanner CLI should return exit code 0
|
|
|
|
|
|
Scenario: Config scanner scan_file raises for missing file
|
|
When I scan a non-existent file path
|
|
Then a scanner FileNotFoundError should be raised
|
|
|
|
|
|
Scenario: Config scanner scan_file handles non-UTF-8 files gracefully
|
|
Given a temporary config file with non-UTF-8 encoding
|
|
When I scan the non-UTF-8 file for violations
|
|
Then the scan should complete without crashing
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: security_exceptions.feature
|
|
# Feature: Explicit exception handling and error classification
|
|
# ============================================================
|
|
|
|
Scenario: Classify a ValidationError as 422
|
|
Given a security exceptions test environment
|
|
Given a ValidationError with message "field is invalid"
|
|
When I classify the security exception
|
|
Then the error code should be 422
|
|
And the error code name should be "VALIDATION_FAILED"
|
|
And the error handling category should be "CLIENT"
|
|
|
|
|
|
Scenario: Classify a ResourceNotFoundError as 404
|
|
Given a security exceptions test environment
|
|
Given a ResourceNotFoundError for resource "plan" with id "01PLAN001"
|
|
When I classify the security exception
|
|
Then the error code should be 404
|
|
And the error code name should be "NOT_FOUND"
|
|
|
|
|
|
Scenario: Classify an AuthenticationError as 401
|
|
Given a security exceptions test environment
|
|
Given an AuthenticationError with message "invalid token"
|
|
When I classify the security exception
|
|
Then the error code should be 401
|
|
And the error code name should be "UNAUTHORIZED"
|
|
|
|
|
|
Scenario: Classify an AuthorizationError as 403
|
|
Given a security exceptions test environment
|
|
Given an AuthorizationError with message "access denied"
|
|
When I classify the security exception
|
|
Then the error code should be 403
|
|
And the error code name should be "FORBIDDEN"
|
|
|
|
|
|
Scenario: Classify a RateLimitError as 429
|
|
Given a security exceptions test environment
|
|
Given a RateLimitError with message "slow down"
|
|
When I classify the security exception
|
|
Then the error code should be 429
|
|
And the error code name should be "RATE_LIMITED"
|
|
|
|
|
|
Scenario: Classify a DatabaseError as 522
|
|
Given a security exceptions test environment
|
|
Given a DatabaseError with message "connection lost"
|
|
When I classify the security exception
|
|
Then the error code should be 522
|
|
And the error handling category should be "SERVER"
|
|
|
|
|
|
Scenario: Classify a ConfigurationError as 525
|
|
Given a security exceptions test environment
|
|
Given a ConfigurationError with message "missing key"
|
|
When I classify the security exception
|
|
Then the error code should be 525
|
|
|
|
|
|
Scenario: Classify a NetworkError as 524
|
|
Given a security exceptions test environment
|
|
Given a NetworkError with message "timeout"
|
|
When I classify the security exception
|
|
Then the error code should be 524
|
|
|
|
|
|
Scenario: Classify a FileSystemError as 523
|
|
Given a security exceptions test environment
|
|
Given a FileSystemError with message "permission denied"
|
|
When I classify the security exception
|
|
Then the error code should be 523
|
|
|
|
|
|
Scenario: Classify a ProviderError as 520
|
|
Given a security exceptions test environment
|
|
Given a ProviderError with message "model failed"
|
|
When I classify the security exception
|
|
Then the error code should be 520
|
|
|
|
|
|
Scenario: Classify a TokenLimitExceededError as 526
|
|
Given a security exceptions test environment
|
|
Given a TokenLimitExceededError with message "context too long"
|
|
When I classify the security exception
|
|
Then the error code should be 526
|
|
|
|
|
|
Scenario: Classify a BusinessRuleViolation as 400
|
|
Given a security exceptions test environment
|
|
Given a BusinessRuleViolation with message "invalid transition"
|
|
When I classify the security exception
|
|
Then the error code should be 400
|
|
|
|
|
|
Scenario: Classify a PlanError as 400
|
|
Given a security exceptions test environment
|
|
Given a PlanError with message "plan stuck"
|
|
When I classify the security exception
|
|
Then the error code should be 400
|
|
|
|
|
|
Scenario: Classify an unknown Exception as 500
|
|
Given a security exceptions test environment
|
|
Given a bare Exception with message "something broke"
|
|
When I classify the security exception
|
|
Then the error code should be 500
|
|
And the error handling category should be "SERVER"
|
|
|
|
|
|
Scenario: Classify an ExecutionError as 521
|
|
Given a security exceptions test environment
|
|
Given an ExecutionError with message "agent failed"
|
|
When I classify the security exception
|
|
Then the error code should be 521
|
|
|
|
|
|
Scenario: Classify a StreamRoutingError as 528
|
|
Given a security exceptions test environment
|
|
Given a StreamRoutingError with message "routing failed"
|
|
When I classify the security exception
|
|
Then the error code should be 528
|
|
|
|
|
|
Scenario: Classify a ModelNotAvailableError as 527
|
|
Given a security exceptions test environment
|
|
Given a ModelNotAvailableError with message "deprecated"
|
|
When I classify the security exception
|
|
Then the error code should be 527
|
|
|
|
|
|
Scenario: Classify a ResourceConflictError as 409
|
|
Given a security exceptions test environment
|
|
Given a ResourceConflictError with message "already exists"
|
|
When I classify the security exception
|
|
Then the error code should be 409
|
|
|
|
|
|
Scenario: Classify a MissingConfigurationError as 525
|
|
Given a security exceptions test environment
|
|
Given a MissingConfigurationError with message "key not set"
|
|
When I classify the security exception
|
|
Then the error code should be 525
|
|
|
|
|
|
Scenario: Classify an ExternalServiceError as 503
|
|
Given a security exceptions test environment
|
|
Given an ExternalServiceError with message "service down"
|
|
When I classify the security exception
|
|
Then the error code should be 503
|
|
|
|
# -- Secret redaction -----------------------------------------------------
|
|
|
|
|
|
Scenario: Redact API key from error details
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "api_key" and value "sk-abc123secret"
|
|
When I redact the error details
|
|
Then the redacted detail "api_key" should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact password from error details
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "password" and value "my-secret-pass"
|
|
When I redact the error details
|
|
Then the redacted detail "password" should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact OpenAI key pattern from string value
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "message" and value "Failed with key sk-abcdefghijklmnopqrstuvwxyz"
|
|
When I redact the error details
|
|
Then the redacted detail "message" should contain "***REDACTED***"
|
|
And the redacted detail "message" should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
|
|
Scenario: Redact JWT token pattern from string value
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "log" and value "auth eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9 failed"
|
|
When I redact the error details
|
|
Then the redacted detail "log" should contain "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact nested dict values
|
|
Given a security exceptions test environment
|
|
Given redaction test details with nested key "config" containing secret_key "abc123"
|
|
When I redact the error details
|
|
Then the redacted nested detail "config" key "secret_key" should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Non-sensitive values pass through unchanged
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "plan_id" and value "01PLAN001"
|
|
When I redact the error details
|
|
Then the redacted detail "plan_id" should be "01PLAN001"
|
|
|
|
|
|
Scenario: Redact GitHub PAT pattern from string value
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "msg" and value "token ghp_abcdefghijklmnopqrstuvwxyz0123456 expired"
|
|
When I redact the error details
|
|
Then the redacted detail "msg" should contain "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact value with redact_value function
|
|
Given a security exceptions test environment
|
|
Given a redaction test string value "key=sk-abcdefghijklmnopqrstuvwxyz"
|
|
When I redact the string value
|
|
Then the redacted string should contain "***REDACTED***"
|
|
|
|
# -- Wrapping unexpected exceptions ----------------------------------------
|
|
|
|
|
|
Scenario: Wrap a bare Exception into CleverAgentsError
|
|
Given a security exceptions test environment
|
|
Given a bare Exception with message "internal crash"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error should be a CleverAgentsError
|
|
And the wrapped error message should be "An unexpected error occurred"
|
|
And the wrapped error details should have key "exception_type"
|
|
|
|
|
|
Scenario: Wrap preserves CleverAgentsError as-is
|
|
Given a security exceptions test environment
|
|
Given a CleverAgentsError with message "known issue"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error message should be "known issue"
|
|
|
|
|
|
Scenario: Wrap merges context into CleverAgentsError details
|
|
Given a security exceptions test environment
|
|
Given a CleverAgentsError with message "known issue"
|
|
And a wrapping context with key "plan_id" and value "01PLAN001"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error details should have key "plan_id"
|
|
|
|
|
|
Scenario: Wrap with custom safe message
|
|
Given a security exceptions test environment
|
|
Given a bare Exception with message "segfault"
|
|
When I wrap the unexpected exception with message "Operation failed"
|
|
Then the wrapped error message should be "Operation failed"
|
|
|
|
|
|
Scenario: Wrap redacts secrets in context
|
|
Given a security exceptions test environment
|
|
Given a bare Exception with message "oops"
|
|
And a wrapping context with key "api_key" and value "sk-secret123456789012345"
|
|
When I wrap the unexpected exception
|
|
Then the wrapped error details key "api_key" should be "***REDACTED***"
|
|
|
|
# -- CLI formatting -------------------------------------------------------
|
|
|
|
|
|
Scenario: Format error for CLI output
|
|
Given a security exceptions test environment
|
|
Given a ValidationError with message "name is required"
|
|
When I classify the security exception
|
|
And I format the error for CLI
|
|
Then the error CLI output should contain "422"
|
|
And the error CLI output should contain "VALIDATION_FAILED"
|
|
And the error CLI output should contain "name is required"
|
|
|
|
|
|
Scenario: Format error with details for CLI output
|
|
Given a security exceptions test environment
|
|
Given a detailed CleverAgentsError "db fail" with detail "table" as "plans"
|
|
When I classify the security exception
|
|
And I format the error for CLI
|
|
Then the error CLI output should contain "table: plans"
|
|
|
|
# -- Secret in message (S1) ------------------------------------------------
|
|
|
|
|
|
Scenario: classify_error redacts secrets embedded in exception message
|
|
Given a security exceptions test environment
|
|
Given a CleverAgentsError with message "key=sk-abcdefghijklmnopqrstuvwxyz leaked"
|
|
When I classify the security exception
|
|
Then the error info message should contain "***REDACTED***"
|
|
And the error info message should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
|
|
Scenario: Format CLI output redacts secrets from message
|
|
Given a security exceptions test environment
|
|
Given a CleverAgentsError with message "tok sk-abcdefghijklmnopqrstuvwxyz end"
|
|
When I classify the security exception
|
|
And I format the error for CLI
|
|
Then the error CLI output should contain "***REDACTED***"
|
|
And the error CLI output should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
# -- List value redaction (B2/T1) ------------------------------------------
|
|
|
|
|
|
Scenario: Redact secrets inside list values
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "keys" and list values "sk-abcdefghijklmnopqrstuvwxyz" and "normal"
|
|
When I redact the error details
|
|
Then the redacted detail "keys" should be a list
|
|
And the redacted list "keys" should contain "***REDACTED***"
|
|
And the redacted list "keys" should not contain "sk-abcdefghijklmnopqrstuvwxyz"
|
|
|
|
# -- Non-string non-dict passthrough (T4) ----------------------------------
|
|
|
|
|
|
Scenario: Integer values pass through redaction unchanged
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "count" and integer value 42
|
|
When I redact the error details
|
|
Then the redacted detail "count" should be integer 42
|
|
|
|
|
|
Scenario: Boolean values pass through redaction unchanged
|
|
Given a security exceptions test environment
|
|
Given redaction test details with key "enabled" and boolean value true
|
|
When I redact the error details
|
|
Then the redacted detail "enabled" should be boolean true
|
|
|
|
# -- Non-mutation of original details (B1) ---------------------------------
|
|
|
|
|
|
Scenario: Wrap does not mutate the original exception details dict
|
|
Given a security exceptions test environment
|
|
Given a tracked CleverAgentsError "original" with details key "a" value "1"
|
|
And a wrapping context with key "b" and value "2"
|
|
When I wrap the unexpected exception
|
|
Then the original exception details reference should not have key "b"
|
|
And the wrapped error details should have key "b"
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: security_readonly.feature
|
|
# Feature: Read-only action enforcement
|
|
# ============================================================
|
|
|
|
Scenario: Read-only plan blocks a write-capable tool with tool name in error
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/file-write" with writes=True and read_only=False
|
|
And a tool execution context with plan_id "plan-ro-1" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then a ToolAccessDeniedError should be raised for the tool
|
|
And the access-denied message should contain "builtin/file-write"
|
|
And the access-denied message should contain "read-only"
|
|
|
|
|
|
Scenario: Read-only plan allows a read-only tool
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/search" with writes=False and read_only=True
|
|
And a tool execution context with plan_id "plan-ro-2" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then no enforcement error should be raised
|
|
|
|
|
|
Scenario: Non-read-only plan allows a write-capable tool
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/file-write" with writes=True and read_only=False
|
|
And a tool execution context with plan_id "plan-rw-1" and read_only False
|
|
When I enforce capabilities on the tool
|
|
Then no enforcement error should be raised
|
|
|
|
|
|
Scenario: Tool with writes=True is blocked even when cap.read_only is unset
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/hybrid-tool" with writes=True and read_only=False
|
|
And a tool execution context with plan_id "plan-ro-3" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then a ToolAccessDeniedError should be raised for the tool
|
|
And the access-denied message should contain "builtin/hybrid-tool"
|
|
|
|
|
|
Scenario: Tool with neither read_only nor writes passes on read-only plan
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/noop" with writes=False and read_only=False
|
|
And a tool execution context with plan_id "plan-ro-4" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then no enforcement error should be raised
|
|
|
|
# ── SkillContext.enforce_write_guard ──────────────────────────────────
|
|
|
|
|
|
Scenario: SkillContext enforce_write_guard blocks write tool with tool name
|
|
Given the read-only enforcement modules are available
|
|
Given a read-only SkillContext for plan "plan-sk-1"
|
|
When I call enforce_write_guard with tool_name "skill/writer"
|
|
Then a SkillExecutionError should be raised for the guard
|
|
And the write-guard error message should contain "skill/writer"
|
|
And the write-guard error type should be PERMISSION_DENIED
|
|
|
|
|
|
Scenario: SkillContext enforce_write_guard allows on writable context
|
|
Given the read-only enforcement modules are available
|
|
Given a writable SkillContext for plan "plan-sk-2"
|
|
When I call enforce_write_guard with tool_name "skill/writer"
|
|
Then no write-guard error should be raised
|
|
|
|
# ── ChangeSetCapture read-only enforcement ───────────────────────────
|
|
|
|
|
|
Scenario: ChangeSet builder rejects write entry when plan is read-only
|
|
Given the read-only enforcement modules are available
|
|
Given a ChangeSetCapture with plan_id "plan-cs-1" and read_only True
|
|
When I try to wrap a write-capable tool spec
|
|
Then a ReadOnlyViolationError should be raised for the capture
|
|
And the capture error message should contain "read-only"
|
|
|
|
|
|
Scenario: ChangeSet builder allows wrapping when not read-only
|
|
Given the read-only enforcement modules are available
|
|
Given a ChangeSetCapture with plan_id "plan-cs-2" and read_only False
|
|
When I try to wrap a write-capable tool spec
|
|
Then no capture error should be raised
|
|
|
|
|
|
Scenario: ChangeSet builder passes read-only tool through unchanged
|
|
Given the read-only enforcement modules are available
|
|
Given a ChangeSetCapture with plan_id "plan-cs-3" and read_only True
|
|
When I wrap a read-only tool spec
|
|
Then the wrapped tool spec should be returned unchanged
|
|
|
|
# ── ExecuteStubActor read-only propagation ─────────────────────────────
|
|
|
|
|
|
Scenario: ExecuteStubActor propagates read_only to ChangeSetCapture
|
|
Given the read-only enforcement modules are available
|
|
When I execute via ExecuteStubActor with read_only True
|
|
Then the stub actor should create a read-only ChangeSetCapture
|
|
|
|
|
|
Scenario: ExecuteStubActor allows writes when not read-only
|
|
Given the read-only enforcement modules are available
|
|
When I execute via ExecuteStubActor with read_only False
|
|
Then the stub actor should create a writable ChangeSetCapture
|
|
|
|
# ── File write tool blocked ──────────────────────────────────────────
|
|
|
|
|
|
Scenario: File write tool blocked under read-only plan via ToolRuntime
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "builtin/file-write" with writes=True and read_only=False
|
|
And a tool execution context with plan_id "plan-fw-1" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then a ToolAccessDeniedError should be raised for the tool
|
|
And the access-denied message should contain "builtin/file-write"
|
|
|
|
# ── Git push tool blocked ────────────────────────────────────────────
|
|
|
|
|
|
Scenario: Git push tool blocked under read-only plan via ToolRuntime
|
|
Given the read-only enforcement modules are available
|
|
Given a tool "git/push" with writes=True and read_only=False
|
|
And a tool execution context with plan_id "plan-gp-1" and read_only True
|
|
When I enforce capabilities on the tool
|
|
Then a ToolAccessDeniedError should be raised for the tool
|
|
And the access-denied message should contain "git/push"
|
|
|
|
# ── Read-only SkillContext blocks write tool ───────────────────────────
|
|
|
|
|
|
Scenario: Read-only SkillContext blocks write tool via enforce_write_guard
|
|
Given the read-only enforcement modules are available
|
|
Given a read-only SkillContext for plan "plan-compat-1"
|
|
When I call enforce_write_guard with tool_name "test/write-tool"
|
|
Then a SkillExecutionError should be raised for the guard
|
|
And the write-guard error message should contain "test/write-tool"
|
|
|
|
|
|
Scenario: Writable SkillContext allows write tool
|
|
Given the read-only enforcement modules are available
|
|
Given a writable SkillContext for plan "plan-compat-2"
|
|
When I call enforce_write_guard with tool_name "test/write-tool"
|
|
Then no write-guard error should be raised
|
|
|
|
# ── Plan -> ToolExecutionContext propagation ──────────────────────────
|
|
|
|
|
|
Scenario: ToolExecutionContext carries plan_read_only from plan
|
|
Given the read-only enforcement modules are available
|
|
Given a Plan domain model with read_only=True and plan_id "plan-ex-1"
|
|
When I create a ToolExecutionContext from the plan
|
|
Then the propagated context plan_read_only should be True
|
|
|
|
|
|
Scenario: ToolExecutionContext writable for writable plan
|
|
Given the read-only enforcement modules are available
|
|
Given a Plan domain model with read_only=False and plan_id "plan-ex-2"
|
|
When I create a ToolExecutionContext from the plan
|
|
Then the propagated context plan_read_only should be False
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: security_secrets.feature
|
|
# Feature: SEC5 - Secrets masking and validation
|
|
# ============================================================
|
|
|
|
Scenario: Redact OpenAI API key pattern
|
|
Given a string containing "sk-proj-abc123def456ghi789"
|
|
When I redact the string
|
|
Then the redacted value should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact Anthropic API key pattern
|
|
Given a string containing "sk-ant-api03-xyzzy1234567890abcdef"
|
|
When I redact the string
|
|
Then the redacted value should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact token pattern
|
|
Given a string containing "tok_01HXYZ1234567890ABCDEF"
|
|
When I redact the string
|
|
Then the redacted value should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Redact Google/Gemini API key pattern
|
|
Given a string containing "AIzaSyA1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q"
|
|
When I redact the string
|
|
Then the redacted value should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Mixed text with Gemini API key is partially redacted
|
|
Given a string containing "Using key AIzaSyA1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q for Gemini"
|
|
When I redact the string
|
|
Then the redacted value should contain "***REDACTED***"
|
|
And the redacted value should contain "Using key"
|
|
And the redacted value should contain "for Gemini"
|
|
And the redacted value should not contain "AIzaSy"
|
|
|
|
|
|
Scenario: Redact generic bearer token
|
|
Given a string containing "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJz"
|
|
When I redact the string
|
|
Then the redacted value should contain "***REDACTED***"
|
|
|
|
|
|
Scenario: Non-secret strings are not redacted
|
|
Given a string containing "hello world this is normal text"
|
|
When I redact the string
|
|
Then the redacted value should be "hello world this is normal text"
|
|
|
|
|
|
Scenario: Mixed text with embedded secret is partially redacted
|
|
Given a string containing "My key is sk-proj-abc123def456ghi789 and name is Bob"
|
|
When I redact the string
|
|
Then the redacted value should contain "***REDACTED***"
|
|
And the redacted value should contain "My key is"
|
|
And the redacted value should contain "and name is Bob"
|
|
|
|
|
|
Scenario: Empty string is returned unchanged
|
|
Given a string containing ""
|
|
When I redact the string
|
|
Then the redacted value should be empty
|
|
|
|
# --- Dict redaction ---
|
|
|
|
|
|
Scenario: Dict with sensitive key names are masked
|
|
Given a dict with key "api_key" and value "sk-proj-abc123"
|
|
When I redact the dict
|
|
Then the key "api_key" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with password key is masked
|
|
Given a dict with key "password" and value "mysecretpass"
|
|
When I redact the dict
|
|
Then the key "password" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with secret key is masked
|
|
Given a dict with key "secret" and value "topsecretvalue"
|
|
When I redact the dict
|
|
Then the key "secret" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with token key is masked
|
|
Given a dict with key "token" and value "sometoken123"
|
|
When I redact the dict
|
|
Then the key "token" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with credential key is masked
|
|
Given a dict with key "credential" and value "cred-value"
|
|
When I redact the dict
|
|
Then the key "credential" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with non-sensitive key is not masked
|
|
Given a dict with key "username" and value "bob"
|
|
When I redact the dict
|
|
Then the key "username" should have value "bob"
|
|
|
|
|
|
Scenario: Nested dict values are redacted recursively
|
|
Given a nested dict with inner key "api_key" and value "sk-proj-nested"
|
|
When I redact the dict
|
|
Then the nested key "api_key" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict with pattern-matched value is masked regardless of key name
|
|
Given a dict with key "config_value" and value "sk-ant-api03-somekey123"
|
|
When I redact the dict
|
|
Then the key "config_value" should have value "***REDACTED***"
|
|
|
|
|
|
Scenario: Dict redaction respects show_secrets flag
|
|
Given a dict with key "api_key" and value "sk-proj-abc123"
|
|
When I redact the dict with show_secrets enabled
|
|
Then the key "api_key" should have value "sk-proj-abc123"
|
|
|
|
# --- Sensitive key detection ---
|
|
|
|
|
|
Scenario Outline: Sensitive key names are detected
|
|
When I check if "<key>" is a sensitive key
|
|
Then the sensitivity check should be true
|
|
|
|
Examples:
|
|
| key |
|
|
| api_key |
|
|
| API_KEY |
|
|
| password |
|
|
| secret |
|
|
| token |
|
|
| credential |
|
|
| openai_api_key |
|
|
| ANTHROPIC_API_KEY |
|
|
| access_token |
|
|
| auth_token |
|
|
| private_key |
|
|
|
|
|
|
Scenario Outline: Non-sensitive key names are not flagged
|
|
When I check if "<key>" is a sensitive key
|
|
Then the sensitivity check should be false
|
|
|
|
Examples:
|
|
| key |
|
|
| username |
|
|
| email |
|
|
| name |
|
|
| description |
|
|
| project_id |
|
|
| token_count |
|
|
|
|
# --- Database URL masking ---
|
|
|
|
|
|
Scenario: SQLite URL is not masked
|
|
Given a database URL "sqlite:///path/to/db.sqlite"
|
|
When I mask the database URL
|
|
Then the masked URL should be "sqlite:///path/to/db.sqlite"
|
|
|
|
|
|
Scenario: PostgreSQL URL with credentials is masked
|
|
Given a database URL "postgresql://admin:s3cret@localhost:5432/mydb"
|
|
When I mask the database URL
|
|
Then the masked URL should contain "***"
|
|
And the masked URL should contain "localhost"
|
|
And the masked URL should not contain "s3cret"
|
|
|
|
# --- Global show_secrets flag ---
|
|
|
|
|
|
Scenario: Global show_secrets defaults to false
|
|
When I check the global show_secrets flag
|
|
Then the flag should be false
|
|
|
|
|
|
Scenario: Global show_secrets can be toggled
|
|
When I set the global show_secrets flag to true
|
|
Then the flag should be true
|
|
When I set the global show_secrets flag to false
|
|
Then the flag should be false
|
|
|
|
# --- structlog processor ---
|
|
|
|
|
|
Scenario: Structlog masking processor redacts event dict values
|
|
Given a structlog event dict with key "api_key" and value "sk-proj-secret123"
|
|
When the masking processor is applied
|
|
Then the event dict key "api_key" should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Structlog masking processor redacts event message
|
|
Given a structlog event dict with event "Failed with key sk-ant-api03-xyz"
|
|
When the masking processor is applied
|
|
Then the event message should contain "***REDACTED***"
|
|
And the event message should not contain "sk-ant-api03-xyz"
|
|
|
|
|
|
Scenario: Structlog masking processor passes non-secret events
|
|
Given a structlog event dict with event "User logged in successfully"
|
|
When the masking processor is applied
|
|
Then the event message should be "User logged in successfully"
|
|
|
|
# --- CLI error detail redaction ---
|
|
|
|
|
|
Scenario: Error details with secret values are redacted
|
|
Given error details with key "api_key" and value "sk-proj-leaked"
|
|
When I redact the error details for CLI display
|
|
Then the redacted details key "api_key" should be "***REDACTED***"
|
|
|
|
|
|
Scenario: Error details with non-secret values are preserved
|
|
Given error details with key "plan_id" and value "01HXR123"
|
|
When I redact the error details for CLI display
|
|
Then the redacted details key "plan_id" should be "01HXR123"
|
|
|
|
# --- Secret pattern registration ---
|
|
|
|
|
|
Scenario: Custom secret pattern can be registered
|
|
Given the default secret patterns
|
|
When I register a custom pattern "custom_[a-z0-9]{16}"
|
|
And I redact the string "my custom_abcdef1234567890 key"
|
|
Then the redacted value should contain "***REDACTED***"
|
|
And the redacted value should not contain "custom_abcdef1234567890"
|
|
|
|
|
|
# ============================================================
|
|
# Originally from: security_templates.feature
|
|
# Feature: Secure template rendering
|
|
# ============================================================
|
|
|
|
Scenario: Render a simple placeholder
|
|
Given a secure template test environment
|
|
Given the secure template text "{greeting} {name}"
|
|
And a secure template context key "greeting" with value "Hello"
|
|
And a secure template context key "name" with value "World"
|
|
When I render the secure template
|
|
Then the secure template output should be "Hello World"
|
|
|
|
|
|
Scenario: Render with missing placeholder leaves it intact
|
|
Given a secure template test environment
|
|
Given the secure template text "Hello {name}, your role is {role}"
|
|
And a secure template context key "name" with value "Alice"
|
|
When I render the secure template
|
|
Then the secure template output should be "Hello Alice, your role is {role}"
|
|
|
|
|
|
Scenario: Render with empty context returns template as-is
|
|
Given a secure template test environment
|
|
Given the secure template text "No placeholders here"
|
|
When I render the secure template
|
|
Then the secure template output should be "No placeholders here"
|
|
|
|
|
|
Scenario: Render with numeric values converts to string
|
|
Given a secure template test environment
|
|
Given the secure template text "Count: {count}"
|
|
And a secure template context key "count" with value "42"
|
|
When I render the secure template
|
|
Then the secure template output should be "Count: 42"
|
|
|
|
# -- Security rejection ---------------------------------------------------
|
|
|
|
|
|
Scenario: Reject attribute access in template
|
|
Given a secure template test environment
|
|
Given the secure template text "{obj.__class__.__name__}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject index access in template
|
|
Given a secure template test environment
|
|
Given the secure template text "{items[0]}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject format spec in template
|
|
Given a secure template test environment
|
|
Given the secure template text "{value:>10}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject conversion flag in template
|
|
Given a secure template test environment
|
|
Given the secure template text "{value!r}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject function call in template
|
|
Given a secure template test environment
|
|
Given the secure template text "{fn()}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject Jinja2 expression delimiters
|
|
Given a secure template test environment
|
|
Given the secure template text "{{ config }}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
|
|
Scenario: Reject Jinja2 block delimiters
|
|
Given a secure template test environment
|
|
Given the secure template text "{% for x in items %}x{% endfor %}"
|
|
When I render the secure template expecting a security error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
# -- Size limits -----------------------------------------------------------
|
|
|
|
|
|
Scenario: Reject template exceeding max length
|
|
Given a secure template test environment
|
|
Given a secure template that is 11000 characters long
|
|
When I render the secure template expecting a size error
|
|
Then the secure template error should mention "exceeds maximum"
|
|
|
|
|
|
Scenario: Reject output exceeding max length
|
|
Given a secure template test environment
|
|
Given the secure template text "{big}"
|
|
And a secure template config with max output length of 20
|
|
And a secure template context key "big" with value "This value is definitely longer than twenty characters"
|
|
When I render the secure template expecting a size error
|
|
Then the secure template error should mention "exceeds maximum"
|
|
|
|
# -- Allowlist enforcement ------------------------------------------------
|
|
|
|
|
|
Scenario: Reject template key not in allowlist
|
|
Given a secure template test environment
|
|
Given the secure template text "{name} {secret}"
|
|
And a secure template with allowed keys "name"
|
|
And a secure template context key "name" with value "Alice"
|
|
And a secure template context key "secret" with value "hidden"
|
|
When I render the secure template expecting a validation error
|
|
Then the secure template error should mention "disallowed"
|
|
|
|
|
|
Scenario: Accept template key that is in allowlist
|
|
Given a secure template test environment
|
|
Given the secure template text "{name}"
|
|
And a secure template with allowed keys "name"
|
|
And a secure template context key "name" with value "Alice"
|
|
When I render the secure template
|
|
Then the secure template output should be "Alice"
|
|
|
|
|
|
Scenario: Open allowlist allows any simple key
|
|
Given a secure template test environment
|
|
Given the secure template text "{anything}"
|
|
And the secure template has no key restrictions
|
|
And a secure template context key "anything" with value "works"
|
|
When I render the secure template
|
|
Then the secure template output should be "works"
|
|
|
|
# -- Pre-validation -------------------------------------------------------
|
|
|
|
|
|
Scenario: Validate clean template returns no errors
|
|
Given a secure template test environment
|
|
Given the secure template text "{name} {role}"
|
|
When I validate the secure template
|
|
Then the secure template validation should be empty
|
|
|
|
|
|
Scenario: Validate unsafe template returns errors
|
|
Given a secure template test environment
|
|
Given the secure template text "{obj.__class__}"
|
|
When I validate the secure template
|
|
Then the secure template validation should not be empty
|
|
And the secure template validation should mention "Unsafe"
|
|
|
|
|
|
Scenario: Validate template with unknown keys returns errors
|
|
Given a secure template test environment
|
|
Given the secure template text "{name} {secret}"
|
|
And a secure template with allowed keys "name"
|
|
When I validate the secure template
|
|
Then the secure template validation should not be empty
|
|
And the secure template validation should mention "Unknown"
|
|
|
|
|
|
Scenario: Validate oversized template returns errors
|
|
Given a secure template test environment
|
|
Given a secure template that is 11000 characters long
|
|
When I validate the secure template
|
|
Then the secure template validation should not be empty
|
|
And the secure template validation should mention "exceeds"
|
|
|
|
# -- Convenience function -------------------------------------------------
|
|
|
|
|
|
Scenario: render_template_secure convenience function works
|
|
Given a secure template test environment
|
|
Given the secure template text "{greeting}"
|
|
And a secure template context key "greeting" with value "Hi"
|
|
When I render via the secure convenience function
|
|
Then the secure template output should be "Hi"
|
|
|
|
|
|
Scenario: render_template_secure rejects unsafe input
|
|
Given a secure template test environment
|
|
Given the secure template text "{obj.attr}"
|
|
When I render via the secure convenience function expecting an error
|
|
Then the secure template error should mention "Unsafe"
|
|
|
|
# -- Config property access ------------------------------------------------
|
|
|
|
|
|
Scenario: Renderer exposes active configuration
|
|
Given a secure template test environment
|
|
Given a secure template with allowed keys "name"
|
|
When I create a renderer with those settings
|
|
Then the renderer config allowed keys should contain "name"
|
|
|
|
# -- Validate with matching allowlist (no unknown keys) -------------------
|
|
|
|
|
|
Scenario: Validate template where all keys match allowlist
|
|
Given a secure template test environment
|
|
Given the secure template text "{name}"
|
|
And a secure template with allowed keys "name"
|
|
When I validate the secure template
|
|
Then the secure template validation should be empty
|
|
|
|
# -- Rejected keys without logging ----------------------------------------
|
|
|
|
|
|
Scenario: Reject unknown key without logging when log_rejected is false
|
|
Given a secure template test environment
|
|
Given the secure template text "{name} {secret}"
|
|
And a secure template config with logging disabled
|
|
And a secure template context key "name" with value "Alice"
|
|
And a secure template context key "secret" with value "hidden"
|
|
When I render the secure template expecting a validation error
|
|
Then the secure template error should mention "disallowed"
|
|
|
|
# -- reject_unknown_keys=False path ----------------------------------------
|
|
|
|
|
|
Scenario: Allow unknown keys when reject_unknown_keys is disabled
|
|
Given a secure template test environment
|
|
Given the secure template text "{name} {extra}"
|
|
And a secure template config that permits unknown keys
|
|
And a secure template context key "name" with value "Alice"
|
|
And a secure template context key "extra" with value "data"
|
|
When I render the secure template
|
|
Then the secure template output should be "Alice data"
|
|
|
|
# -- Legacy renderer with config passthrough --------------------------------
|
|
|
|
|
|
Scenario: Legacy TemplateRenderer accepts config passthrough
|
|
Given a secure template test environment
|
|
Given the secure template text "{name}"
|
|
And a secure template with allowed keys "name"
|
|
And a secure template context key "name" with value "Carol"
|
|
When I render via the legacy TemplateRenderer with config
|
|
Then the secure template output should be "Carol"
|
|
|
|
# -- Legacy renderer backward compat --------------------------------------
|
|
|
|
|
|
Scenario: Legacy TemplateRenderer returns raw on unsafe template
|
|
Given a secure template test environment
|
|
Given the secure template text "{obj.__class__}"
|
|
And a secure template context key "obj" with value "test"
|
|
When I render via the legacy TemplateRenderer
|
|
Then the secure template output should be "{obj.__class__}"
|
|
|
|
|
|
Scenario: Legacy TemplateRenderer renders safe templates
|
|
Given a secure template test environment
|
|
Given the secure template text "{name}"
|
|
And a secure template context key "name" with value "Bob"
|
|
When I render via the legacy TemplateRenderer
|
|
Then the secure template output should be "Bob"
|
|
|