Files
cleveragents-core/features/consolidated_security.feature
T
hurui200320 cbf8bcc993
CI / lint (push) Successful in 15s
CI / build (push) Successful in 26s
CI / quality (push) Successful in 35s
CI / typecheck (push) Successful in 40s
CI / benchmark-regression (push) Has been skipped
CI / security (push) Successful in 43s
CI / unit_tests (push) Successful in 3m5s
CI / integration_tests (push) Successful in 3m35s
CI / docker (push) Successful in 1m3s
CI / e2e_tests (push) Successful in 5m28s
CI / coverage (push) Successful in 8m5s
CI / benchmark-publish (push) Successful in 20m11s
test(e2e): E2E acceptance criteria for M5 (v3.4.0) — ACMS v1 and context scaling (#811)
## Summary

Add `robot/e2e/m5_acceptance.robot` with **21 zero-mock E2E test cases** (in addition to existing M5 test suite) covering all M5 (v3.4.0) acceptance criteria:

1. **Context Assembly** — add/list/show/clear files in the context pipeline
2. **Context Scaling** — 10,000+ file project setup with simulate plumbing *(structural)*
3. **Context Policy Configuration** — per-view include/exclude paths, file-size limits
4. **Budget Enforcement** — max_file_size / max_total_size constraint storage *(structural)*
5. **Context Analysis** — ACMS pipeline inspect (tier schema) and simulate (JSON schema) *(structural)*
6. **Plan Execution** — real LLM calls via `openai/gpt-4o-mini` (`plan use` + `plan resume`)

### Structural vs. Behavioural Scope

Tests in sections 1b–4 that use `project context simulate` or `inspect` are **structural / plumbing validations** — they verify CLI execution, JSON serialization, and stored configuration but do **not** exercise actual ACMS indexing or budget enforcement because the `ContextTierService` is an in-memory singleton that starts empty per CLI process. Each affected test has a `[Documentation]` note explaining this limitation. Behavioural ACMS validation is deferred until the full indexing pipeline is wired.

### Production Bug Fixes

| Fix | File | Description |
|-----|------|-------------|
| `session.flush()` → `session.commit()` | `project_context.py` | Policy changes silently lost on `session.close()` |
| `contextlib.suppress` rollback wrapper | `project_context.py` | Prevents rollback failure from masking original commit exception |
| Add `session_factory` DI provider | `container.py` | `project context` commands hit `AttributeError` |
| `providers.Factory` → `providers.Singleton` | `container.py` | Avoid creating duplicate engines per call |
| Add Gemini API key pattern | `redaction.py` | `AIzaSy...` keys now redacted in logs |

### Review Feedback Addressed (Tenth Pass — @CoreRasurae Review #2410)

| # | Severity | Finding | Fix |
|---|----------|---------|-----|
| P3-1 | Medium | "Clear Context" test tautological — never asserts files were present before clearing | Added `Should Contain ${list_before.stdout} config.py` precondition check after `context-load` and before `clear` |
| P3-2 | Medium | Policy/budget verification uses substring matching (`Should Contain 262144`) | Replaced with `Extract JSON From Stdout` + `$rv.get('max_file_size') == 262144` parsed JSON assertions using `resolved_view` dict access |
| P3-3 | Medium | Plan resume doesn't verify `phase` value, only existence | Added `Should Not Be Equal As Strings ${phase} queued` assertion to verify plan transitioned from queued |
| P3-4 | Medium | Plan JSON extraction inconsistency (`rindex` vs `Extract JSON From Stdout`) | Replaced fragile `rindex`-based extraction with `Extract JSON From Stdout` keyword for consistency |
| P3-6 | Medium | Context show summary weak content assertions | Added `Should Not Contain` guards against traceback/error output to reject false positives |
| P3-8 | Medium | `_SafeSession` singleton may accumulate dirty state after rollback | Changed `_SafeSession.close()` from pure no-op to `real.rollback()` to reset session state between calls |
| P3-14 | Medium | No test for `_save_policy_json` rollback path | Added BDD scenario "Save policy rollback re-raises after commit failure" with monkey-patched commit |
| P3-15 | Medium | No test for `_save_policy_json` on nonexistent project | Added BDD scenario "Save policy on nonexistent project row updates zero rows" verifying silent 0-row behavior |
| P4-1 | Low | Plan resume TRY/EXCEPT swallows assertion details | Moved field assertions outside TRY block; TRY only guards JSON extraction |
| P4-2 | Low | `Safe Parse Json Field` logs stale error context | Fixed to track and report both Strategy 1 and Strategy 2 error contexts separately |
| P4-4 | Low | SQLite WAL/SHM files not cleaned in regression test | Added cleanup loop for `-wal` and `-shm` suffixes alongside `.db` file |

### Deferred Items (Out of Scope)

| ID | Severity | Reason |
|----|----------|--------|
| P2-1 | High | `execution_environment` silently dropped on subsequent `context set` — pre-existing production code bug in `_write_policy()`, not introduced by this PR |
| P2-2 | High | Unhandled `ValidationError` on corrupt policy blob — pre-existing `_read_policy()` code, not changed by this PR |
| P2-3 | High | Silent no-op UPDATE when `ns_projects` row missing — pre-existing `_save_policy_json` logic; this PR only changed error handling |
| P3-5 | Medium | Structural tests cannot detect regressions — already honestly documented in every affected test's `[Documentation]` block |
| P3-7 | Medium | View inheritance/override behavior not tested — nice-to-have, not in ticket acceptance criteria |
| P3-9 | Medium | `context_set` double-writes when `execution_environment` set — pre-existing production logic |
| P3-10 | Medium | `budget_tokens=0` silently replaced by default (falsy `or`) — pre-existing production code bug |
| P3-11 | Medium | `context set` replaces entire view instead of merging — pre-existing design choice |
| P3-12 | Medium | GEMINI_API_KEY propagated but potentially unused — security-first: propagating for redaction testing |
| P3-13 | Medium | `reset_container()` doesn't dispose Singleton resources — pre-existing container lifecycle issue |
| M5 | Medium | `_build_session_factory` engine never disposed — production code architecture, out of scope for testing ticket |
| M6 | Medium | Missing `check_same_thread`/`isolation_level` — production code architecture, out of scope for testing ticket |
| L1 | Low | `plan resume` not in spec CLI synopsis — informational |
| L2 | Low | Context summary assertions depend on exact CLI wording — acceptable stability risk |
| L3 | Low | Gemini regex minimum length slightly loose — acceptable security-first trade-off |
| L4 | Low | Missing Google OAuth2 credential patterns — out of scope for this PR |
| P4-3 | Low | `Run CLI` keyword duplicated — different purpose (uses `${WS}` as default cwd), not a true duplicate |
| P4-5–P4-9 | Low | Various additional E2E coverage gaps — nice-to-have, not in ticket acceptance criteria |

### Quality Gates

| Gate | Result |
|------|--------|
| lint | PASS |
| typecheck | PASS (0 errors) |
| unit_tests | **393/393** features, 11,210 scenarios |
| integration_tests | **1,576/1,576** |
| e2e_tests | **37/37** (21 M5 + 12 M6 + 2 smoke + 2 M1) |
| coverage_report | **97%** (threshold: 97%) |

### Files Changed

| File | Change |
|------|--------|
| `robot/e2e/m5_acceptance.robot` | **NEW** — 21 E2E test cases with honest structural documentation, parsed JSON assertions, prerequisite skip guards on all sections, safe assertion messages |
| `robot/e2e/common_e2e.resource` | `on_timeout=kill` + return code checks + safe key evaluation via `os.environ.get` + fixed stale error logging in `Safe Parse Json Field` |
| `robot/e2e/m1_acceptance.robot` | `on_timeout=kill` on git log |
| `robot/e2e/m2_acceptance.robot` | `on_timeout=kill` + return code checks + safe assertion messages (no stderr embedding) |
| `src/cleveragents/application/container.py` | Add `_build_session_factory` + `session_factory` Singleton |
| `src/cleveragents/cli/commands/project_context.py` | `flush()` → `commit()` + `contextlib.suppress` rollback |
| `src/cleveragents/shared/redaction.py` | Add Gemini API key pattern |
| `noxfile.py` | Propagate `GEMINI_API_KEY` in e2e_tests |
| `CHANGELOG.md` | 4 entries for #745 |
| `features/application_container_coverage_boost.feature` | Updated title + 3 scenarios |
| `features/steps/application_container_coverage_boost_steps.py` | Step defs for `_build_session_factory` |
| `features/consolidated_security.feature` | 2 Gemini API key redaction scenarios |
| `features/project_context_cli_coverage_boost.feature` | `flush()→commit()` regression test + rollback path + nonexistent project tests |
| `features/steps/project_context_cli_coverage_boost_steps.py` | Separate engines for regression test + `try/finally` cleanup + `_SafeSession.close()` state reset + rollback/nonexistent test steps + WAL/SHM cleanup |

Closes #745

ISSUES CLOSED: #745

Reviewed-on: #811
Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com>
Co-authored-by: Rui Hu <rui.hu@cleverthis.com>
Co-committed-by: Rui Hu <rui.hu@cleverthis.com>
2026-03-19 06:53:06 +00:00

1140 lines
45 KiB
Gherkin

Feature: Consolidated Security
Combined scenarios from: security_eval, security_exceptions, security_readonly, security_secrets, security_templates
# ============================================================
# Originally from: security_eval.feature
# Feature: SEC1 - eval/exec removal security
# ============================================================
Scenario: Config with code injection attempt is rejected
Given a SimpleToolAgent configured with a code injection payload
When I attempt to process content through the injected agent
Then the agent should reject the code block with a routing error
And the error message should mention code blocks are not supported
Scenario: Malicious transform expression does not execute
Given the CleverAgents reactive system is available
When I configure a transform with a malicious eval expression
Then the transform should be rejected with a routing error
And the error message should mention unknown transform
Scenario: Valid config still works after eval removal
Given a SimpleToolAgent configured with a named operation "uppercase"
When I process "hello" through the named operation agent
Then the named operation result should be "HELLO"
Scenario: Named transform from registry works after eval removal
Given the CleverAgents reactive system is available
When I configure a transform with a registered named function "uppercase"
Then the named transform should produce the uppercased result
Scenario: Code block with import attempt is rejected
Given a SimpleToolAgent configured with an import injection payload
When I attempt to process content through the injected agent
Then the agent should reject the code block with a routing error
Scenario: Eval expression mimicking registry name is still rejected
Given the CleverAgents reactive system is available
When I configure a transform with expression "lambda x: __import__('os').system('rm -rf /')"
Then the transform should be rejected with a routing error
Scenario: Custom registered operation works
Given a SimpleToolAgent with a custom registered operation "reverse"
When I process "abcde" through the custom operation agent
Then the custom operation result should be "edcba"
Scenario: Custom registered transform works
Given the CleverAgents reactive system is available
And a custom transform "double" is registered
When I configure a transform with a registered named function "double"
Then the custom transform should produce the doubled result
Scenario: Config scanner detects eval in content
Given config content containing "value: eval('hack')"
When I scan the config content for security violations
Then the scanner should report at least 1 violation
And the scanner should report a violation with token "eval("
And the violation severity should be "CRITICAL"
Scenario: Config scanner detects exec in content
Given config content containing "action: exec('import os')"
When I scan the config content for security violations
Then the scanner should report at least 1 violation
And the scanner should report a violation with token "exec("
Scenario: Config scanner passes clean content
Given config content containing "name: safe-project"
When I scan the config content for security violations
Then the scanner should report 0 violations
Scenario: Config scanner reports correct line numbers
Given multiline config content with a violation on line 3
When I scan the config content for security violations
Then the scanner should report a violation on line 3
Scenario: Config safety validation raises on dangerous content
Given config content containing "run: exec('bad')"
When I validate the config content for safety
Then a ConfigurationError should be raised
Scenario: Config safety validation passes clean content
Given config content containing "name: safe-project"
When I validate the config content for safety
Then no configuration error should be raised
Scenario: Config scanner skips comment lines
Given config content containing "# eval('this is just a comment')"
When I scan the config content for security violations
Then the scanner should report 0 violations
Scenario: Config scanner skips INI-style semicolon comment lines
Given config content containing "; eval('this is an INI comment')"
When I scan the config content for security violations
Then the scanner should report 0 violations
Scenario: Config scanner skips inline comments after hash
Given config content containing "name: safe-project # eval('just a note')"
When I scan the config content for security violations
Then the scanner should report 0 violations
Scenario: Config scanner preserves hash inside quoted strings
Given config content containing "value: \"eval('inside quotes') # not a comment\""
When I scan the config content for security violations
Then the scanner should report at least 1 violation
And the scanner should report a violation with token "eval("
Scenario: Config scanner detects multiple violation types
Given config content with eval and subprocess patterns
When I scan the config content for security violations
Then the scanner should report at least 2 violations
Scenario: Config scanner scans multiple files
Given two temporary config files with violations
When I scan the config files for security violations
Then the scanner should report at least 2 violations
Scenario: Config scanner CLI shows usage when no args given
When I run the config scanner CLI with no arguments
Then the scanner CLI should return exit code 2
Scenario: Config scanner CLI reports violations found
When I run the config scanner CLI with a dirty temp file
Then the scanner CLI should return exit code 1
And the scanner CLI output should contain "violation"
Scenario: Config scanner CLI reports clean file
When I run the config scanner CLI with a clean temp file
Then the scanner CLI should return exit code 0
Scenario: Config scanner scan_file raises for missing file
When I scan a non-existent file path
Then a scanner FileNotFoundError should be raised
Scenario: Config scanner scan_file handles non-UTF-8 files gracefully
Given a temporary config file with non-UTF-8 encoding
When I scan the non-UTF-8 file for violations
Then the scan should complete without crashing
# ============================================================
# Originally from: security_exceptions.feature
# Feature: Explicit exception handling and error classification
# ============================================================
Scenario: Classify a ValidationError as 422
Given a security exceptions test environment
Given a ValidationError with message "field is invalid"
When I classify the security exception
Then the error code should be 422
And the error code name should be "VALIDATION_FAILED"
And the error handling category should be "CLIENT"
Scenario: Classify a ResourceNotFoundError as 404
Given a security exceptions test environment
Given a ResourceNotFoundError for resource "plan" with id "01PLAN001"
When I classify the security exception
Then the error code should be 404
And the error code name should be "NOT_FOUND"
Scenario: Classify an AuthenticationError as 401
Given a security exceptions test environment
Given an AuthenticationError with message "invalid token"
When I classify the security exception
Then the error code should be 401
And the error code name should be "UNAUTHORIZED"
Scenario: Classify an AuthorizationError as 403
Given a security exceptions test environment
Given an AuthorizationError with message "access denied"
When I classify the security exception
Then the error code should be 403
And the error code name should be "FORBIDDEN"
Scenario: Classify a RateLimitError as 429
Given a security exceptions test environment
Given a RateLimitError with message "slow down"
When I classify the security exception
Then the error code should be 429
And the error code name should be "RATE_LIMITED"
Scenario: Classify a DatabaseError as 522
Given a security exceptions test environment
Given a DatabaseError with message "connection lost"
When I classify the security exception
Then the error code should be 522
And the error handling category should be "SERVER"
Scenario: Classify a ConfigurationError as 525
Given a security exceptions test environment
Given a ConfigurationError with message "missing key"
When I classify the security exception
Then the error code should be 525
Scenario: Classify a NetworkError as 524
Given a security exceptions test environment
Given a NetworkError with message "timeout"
When I classify the security exception
Then the error code should be 524
Scenario: Classify a FileSystemError as 523
Given a security exceptions test environment
Given a FileSystemError with message "permission denied"
When I classify the security exception
Then the error code should be 523
Scenario: Classify a ProviderError as 520
Given a security exceptions test environment
Given a ProviderError with message "model failed"
When I classify the security exception
Then the error code should be 520
Scenario: Classify a TokenLimitExceededError as 526
Given a security exceptions test environment
Given a TokenLimitExceededError with message "context too long"
When I classify the security exception
Then the error code should be 526
Scenario: Classify a BusinessRuleViolation as 400
Given a security exceptions test environment
Given a BusinessRuleViolation with message "invalid transition"
When I classify the security exception
Then the error code should be 400
Scenario: Classify a PlanError as 400
Given a security exceptions test environment
Given a PlanError with message "plan stuck"
When I classify the security exception
Then the error code should be 400
Scenario: Classify an unknown Exception as 500
Given a security exceptions test environment
Given a bare Exception with message "something broke"
When I classify the security exception
Then the error code should be 500
And the error handling category should be "SERVER"
Scenario: Classify an ExecutionError as 521
Given a security exceptions test environment
Given an ExecutionError with message "agent failed"
When I classify the security exception
Then the error code should be 521
Scenario: Classify a StreamRoutingError as 528
Given a security exceptions test environment
Given a StreamRoutingError with message "routing failed"
When I classify the security exception
Then the error code should be 528
Scenario: Classify a ModelNotAvailableError as 527
Given a security exceptions test environment
Given a ModelNotAvailableError with message "deprecated"
When I classify the security exception
Then the error code should be 527
Scenario: Classify a ResourceConflictError as 409
Given a security exceptions test environment
Given a ResourceConflictError with message "already exists"
When I classify the security exception
Then the error code should be 409
Scenario: Classify a MissingConfigurationError as 525
Given a security exceptions test environment
Given a MissingConfigurationError with message "key not set"
When I classify the security exception
Then the error code should be 525
Scenario: Classify an ExternalServiceError as 503
Given a security exceptions test environment
Given an ExternalServiceError with message "service down"
When I classify the security exception
Then the error code should be 503
# -- Secret redaction -----------------------------------------------------
Scenario: Redact API key from error details
Given a security exceptions test environment
Given redaction test details with key "api_key" and value "sk-abc123secret"
When I redact the error details
Then the redacted detail "api_key" should be "***REDACTED***"
Scenario: Redact password from error details
Given a security exceptions test environment
Given redaction test details with key "password" and value "my-secret-pass"
When I redact the error details
Then the redacted detail "password" should be "***REDACTED***"
Scenario: Redact OpenAI key pattern from string value
Given a security exceptions test environment
Given redaction test details with key "message" and value "Failed with key sk-abcdefghijklmnopqrstuvwxyz"
When I redact the error details
Then the redacted detail "message" should contain "***REDACTED***"
And the redacted detail "message" should not contain "sk-abcdefghijklmnopqrstuvwxyz"
Scenario: Redact JWT token pattern from string value
Given a security exceptions test environment
Given redaction test details with key "log" and value "auth eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9 failed"
When I redact the error details
Then the redacted detail "log" should contain "***REDACTED***"
Scenario: Redact nested dict values
Given a security exceptions test environment
Given redaction test details with nested key "config" containing secret_key "abc123"
When I redact the error details
Then the redacted nested detail "config" key "secret_key" should be "***REDACTED***"
Scenario: Non-sensitive values pass through unchanged
Given a security exceptions test environment
Given redaction test details with key "plan_id" and value "01PLAN001"
When I redact the error details
Then the redacted detail "plan_id" should be "01PLAN001"
Scenario: Redact GitHub PAT pattern from string value
Given a security exceptions test environment
Given redaction test details with key "msg" and value "token ghp_abcdefghijklmnopqrstuvwxyz0123456 expired"
When I redact the error details
Then the redacted detail "msg" should contain "***REDACTED***"
Scenario: Redact value with redact_value function
Given a security exceptions test environment
Given a redaction test string value "key=sk-abcdefghijklmnopqrstuvwxyz"
When I redact the string value
Then the redacted string should contain "***REDACTED***"
# -- Wrapping unexpected exceptions ----------------------------------------
Scenario: Wrap a bare Exception into CleverAgentsError
Given a security exceptions test environment
Given a bare Exception with message "internal crash"
When I wrap the unexpected exception
Then the wrapped error should be a CleverAgentsError
And the wrapped error message should be "An unexpected error occurred"
And the wrapped error details should have key "exception_type"
Scenario: Wrap preserves CleverAgentsError as-is
Given a security exceptions test environment
Given a CleverAgentsError with message "known issue"
When I wrap the unexpected exception
Then the wrapped error message should be "known issue"
Scenario: Wrap merges context into CleverAgentsError details
Given a security exceptions test environment
Given a CleverAgentsError with message "known issue"
And a wrapping context with key "plan_id" and value "01PLAN001"
When I wrap the unexpected exception
Then the wrapped error details should have key "plan_id"
Scenario: Wrap with custom safe message
Given a security exceptions test environment
Given a bare Exception with message "segfault"
When I wrap the unexpected exception with message "Operation failed"
Then the wrapped error message should be "Operation failed"
Scenario: Wrap redacts secrets in context
Given a security exceptions test environment
Given a bare Exception with message "oops"
And a wrapping context with key "api_key" and value "sk-secret123456789012345"
When I wrap the unexpected exception
Then the wrapped error details key "api_key" should be "***REDACTED***"
# -- CLI formatting -------------------------------------------------------
Scenario: Format error for CLI output
Given a security exceptions test environment
Given a ValidationError with message "name is required"
When I classify the security exception
And I format the error for CLI
Then the error CLI output should contain "422"
And the error CLI output should contain "VALIDATION_FAILED"
And the error CLI output should contain "name is required"
Scenario: Format error with details for CLI output
Given a security exceptions test environment
Given a detailed CleverAgentsError "db fail" with detail "table" as "plans"
When I classify the security exception
And I format the error for CLI
Then the error CLI output should contain "table: plans"
# -- Secret in message (S1) ------------------------------------------------
Scenario: classify_error redacts secrets embedded in exception message
Given a security exceptions test environment
Given a CleverAgentsError with message "key=sk-abcdefghijklmnopqrstuvwxyz leaked"
When I classify the security exception
Then the error info message should contain "***REDACTED***"
And the error info message should not contain "sk-abcdefghijklmnopqrstuvwxyz"
Scenario: Format CLI output redacts secrets from message
Given a security exceptions test environment
Given a CleverAgentsError with message "tok sk-abcdefghijklmnopqrstuvwxyz end"
When I classify the security exception
And I format the error for CLI
Then the error CLI output should contain "***REDACTED***"
And the error CLI output should not contain "sk-abcdefghijklmnopqrstuvwxyz"
# -- List value redaction (B2/T1) ------------------------------------------
Scenario: Redact secrets inside list values
Given a security exceptions test environment
Given redaction test details with key "keys" and list values "sk-abcdefghijklmnopqrstuvwxyz" and "normal"
When I redact the error details
Then the redacted detail "keys" should be a list
And the redacted list "keys" should contain "***REDACTED***"
And the redacted list "keys" should not contain "sk-abcdefghijklmnopqrstuvwxyz"
# -- Non-string non-dict passthrough (T4) ----------------------------------
Scenario: Integer values pass through redaction unchanged
Given a security exceptions test environment
Given redaction test details with key "count" and integer value 42
When I redact the error details
Then the redacted detail "count" should be integer 42
Scenario: Boolean values pass through redaction unchanged
Given a security exceptions test environment
Given redaction test details with key "enabled" and boolean value true
When I redact the error details
Then the redacted detail "enabled" should be boolean true
# -- Non-mutation of original details (B1) ---------------------------------
Scenario: Wrap does not mutate the original exception details dict
Given a security exceptions test environment
Given a tracked CleverAgentsError "original" with details key "a" value "1"
And a wrapping context with key "b" and value "2"
When I wrap the unexpected exception
Then the original exception details reference should not have key "b"
And the wrapped error details should have key "b"
# ============================================================
# Originally from: security_readonly.feature
# Feature: Read-only action enforcement
# ============================================================
Scenario: Read-only plan blocks a write-capable tool with tool name in error
Given the read-only enforcement modules are available
Given a tool "builtin/file-write" with writes=True and read_only=False
And a tool execution context with plan_id "plan-ro-1" and read_only True
When I enforce capabilities on the tool
Then a ToolAccessDeniedError should be raised for the tool
And the access-denied message should contain "builtin/file-write"
And the access-denied message should contain "read-only"
Scenario: Read-only plan allows a read-only tool
Given the read-only enforcement modules are available
Given a tool "builtin/search" with writes=False and read_only=True
And a tool execution context with plan_id "plan-ro-2" and read_only True
When I enforce capabilities on the tool
Then no enforcement error should be raised
Scenario: Non-read-only plan allows a write-capable tool
Given the read-only enforcement modules are available
Given a tool "builtin/file-write" with writes=True and read_only=False
And a tool execution context with plan_id "plan-rw-1" and read_only False
When I enforce capabilities on the tool
Then no enforcement error should be raised
Scenario: Tool with writes=True is blocked even when cap.read_only is unset
Given the read-only enforcement modules are available
Given a tool "builtin/hybrid-tool" with writes=True and read_only=False
And a tool execution context with plan_id "plan-ro-3" and read_only True
When I enforce capabilities on the tool
Then a ToolAccessDeniedError should be raised for the tool
And the access-denied message should contain "builtin/hybrid-tool"
Scenario: Tool with neither read_only nor writes passes on read-only plan
Given the read-only enforcement modules are available
Given a tool "builtin/noop" with writes=False and read_only=False
And a tool execution context with plan_id "plan-ro-4" and read_only True
When I enforce capabilities on the tool
Then no enforcement error should be raised
# ── SkillContext.enforce_write_guard ──────────────────────────────────
Scenario: SkillContext enforce_write_guard blocks write tool with tool name
Given the read-only enforcement modules are available
Given a read-only SkillContext for plan "plan-sk-1"
When I call enforce_write_guard with tool_name "skill/writer"
Then a SkillExecutionError should be raised for the guard
And the write-guard error message should contain "skill/writer"
And the write-guard error type should be PERMISSION_DENIED
Scenario: SkillContext enforce_write_guard allows on writable context
Given the read-only enforcement modules are available
Given a writable SkillContext for plan "plan-sk-2"
When I call enforce_write_guard with tool_name "skill/writer"
Then no write-guard error should be raised
# ── ChangeSetCapture read-only enforcement ───────────────────────────
Scenario: ChangeSet builder rejects write entry when plan is read-only
Given the read-only enforcement modules are available
Given a ChangeSetCapture with plan_id "plan-cs-1" and read_only True
When I try to wrap a write-capable tool spec
Then a ReadOnlyViolationError should be raised for the capture
And the capture error message should contain "read-only"
Scenario: ChangeSet builder allows wrapping when not read-only
Given the read-only enforcement modules are available
Given a ChangeSetCapture with plan_id "plan-cs-2" and read_only False
When I try to wrap a write-capable tool spec
Then no capture error should be raised
Scenario: ChangeSet builder passes read-only tool through unchanged
Given the read-only enforcement modules are available
Given a ChangeSetCapture with plan_id "plan-cs-3" and read_only True
When I wrap a read-only tool spec
Then the wrapped tool spec should be returned unchanged
# ── ExecuteStubActor read-only propagation ─────────────────────────────
Scenario: ExecuteStubActor propagates read_only to ChangeSetCapture
Given the read-only enforcement modules are available
When I execute via ExecuteStubActor with read_only True
Then the stub actor should create a read-only ChangeSetCapture
Scenario: ExecuteStubActor allows writes when not read-only
Given the read-only enforcement modules are available
When I execute via ExecuteStubActor with read_only False
Then the stub actor should create a writable ChangeSetCapture
# ── File write tool blocked ──────────────────────────────────────────
Scenario: File write tool blocked under read-only plan via ToolRuntime
Given the read-only enforcement modules are available
Given a tool "builtin/file-write" with writes=True and read_only=False
And a tool execution context with plan_id "plan-fw-1" and read_only True
When I enforce capabilities on the tool
Then a ToolAccessDeniedError should be raised for the tool
And the access-denied message should contain "builtin/file-write"
# ── Git push tool blocked ────────────────────────────────────────────
Scenario: Git push tool blocked under read-only plan via ToolRuntime
Given the read-only enforcement modules are available
Given a tool "git/push" with writes=True and read_only=False
And a tool execution context with plan_id "plan-gp-1" and read_only True
When I enforce capabilities on the tool
Then a ToolAccessDeniedError should be raised for the tool
And the access-denied message should contain "git/push"
# ── Read-only SkillContext blocks write tool ───────────────────────────
Scenario: Read-only SkillContext blocks write tool via enforce_write_guard
Given the read-only enforcement modules are available
Given a read-only SkillContext for plan "plan-compat-1"
When I call enforce_write_guard with tool_name "test/write-tool"
Then a SkillExecutionError should be raised for the guard
And the write-guard error message should contain "test/write-tool"
Scenario: Writable SkillContext allows write tool
Given the read-only enforcement modules are available
Given a writable SkillContext for plan "plan-compat-2"
When I call enforce_write_guard with tool_name "test/write-tool"
Then no write-guard error should be raised
# ── Plan -> ToolExecutionContext propagation ──────────────────────────
Scenario: ToolExecutionContext carries plan_read_only from plan
Given the read-only enforcement modules are available
Given a Plan domain model with read_only=True and plan_id "plan-ex-1"
When I create a ToolExecutionContext from the plan
Then the propagated context plan_read_only should be True
Scenario: ToolExecutionContext writable for writable plan
Given the read-only enforcement modules are available
Given a Plan domain model with read_only=False and plan_id "plan-ex-2"
When I create a ToolExecutionContext from the plan
Then the propagated context plan_read_only should be False
# ============================================================
# Originally from: security_secrets.feature
# Feature: SEC5 - Secrets masking and validation
# ============================================================
Scenario: Redact OpenAI API key pattern
Given a string containing "sk-proj-abc123def456ghi789"
When I redact the string
Then the redacted value should be "***REDACTED***"
Scenario: Redact Anthropic API key pattern
Given a string containing "sk-ant-api03-xyzzy1234567890abcdef"
When I redact the string
Then the redacted value should be "***REDACTED***"
Scenario: Redact token pattern
Given a string containing "tok_01HXYZ1234567890ABCDEF"
When I redact the string
Then the redacted value should be "***REDACTED***"
Scenario: Redact Google/Gemini API key pattern
Given a string containing "AIzaSyA1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q"
When I redact the string
Then the redacted value should be "***REDACTED***"
Scenario: Mixed text with Gemini API key is partially redacted
Given a string containing "Using key AIzaSyA1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q for Gemini"
When I redact the string
Then the redacted value should contain "***REDACTED***"
And the redacted value should contain "Using key"
And the redacted value should contain "for Gemini"
And the redacted value should not contain "AIzaSy"
Scenario: Redact generic bearer token
Given a string containing "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJz"
When I redact the string
Then the redacted value should contain "***REDACTED***"
Scenario: Non-secret strings are not redacted
Given a string containing "hello world this is normal text"
When I redact the string
Then the redacted value should be "hello world this is normal text"
Scenario: Mixed text with embedded secret is partially redacted
Given a string containing "My key is sk-proj-abc123def456ghi789 and name is Bob"
When I redact the string
Then the redacted value should contain "***REDACTED***"
And the redacted value should contain "My key is"
And the redacted value should contain "and name is Bob"
Scenario: Empty string is returned unchanged
Given a string containing ""
When I redact the string
Then the redacted value should be empty
# --- Dict redaction ---
Scenario: Dict with sensitive key names are masked
Given a dict with key "api_key" and value "sk-proj-abc123"
When I redact the dict
Then the key "api_key" should have value "***REDACTED***"
Scenario: Dict with password key is masked
Given a dict with key "password" and value "mysecretpass"
When I redact the dict
Then the key "password" should have value "***REDACTED***"
Scenario: Dict with secret key is masked
Given a dict with key "secret" and value "topsecretvalue"
When I redact the dict
Then the key "secret" should have value "***REDACTED***"
Scenario: Dict with token key is masked
Given a dict with key "token" and value "sometoken123"
When I redact the dict
Then the key "token" should have value "***REDACTED***"
Scenario: Dict with credential key is masked
Given a dict with key "credential" and value "cred-value"
When I redact the dict
Then the key "credential" should have value "***REDACTED***"
Scenario: Dict with non-sensitive key is not masked
Given a dict with key "username" and value "bob"
When I redact the dict
Then the key "username" should have value "bob"
Scenario: Nested dict values are redacted recursively
Given a nested dict with inner key "api_key" and value "sk-proj-nested"
When I redact the dict
Then the nested key "api_key" should have value "***REDACTED***"
Scenario: Dict with pattern-matched value is masked regardless of key name
Given a dict with key "config_value" and value "sk-ant-api03-somekey123"
When I redact the dict
Then the key "config_value" should have value "***REDACTED***"
Scenario: Dict redaction respects show_secrets flag
Given a dict with key "api_key" and value "sk-proj-abc123"
When I redact the dict with show_secrets enabled
Then the key "api_key" should have value "sk-proj-abc123"
# --- Sensitive key detection ---
Scenario Outline: Sensitive key names are detected
When I check if "<key>" is a sensitive key
Then the sensitivity check should be true
Examples:
| key |
| api_key |
| API_KEY |
| password |
| secret |
| token |
| credential |
| openai_api_key |
| ANTHROPIC_API_KEY |
| access_token |
| auth_token |
| private_key |
Scenario Outline: Non-sensitive key names are not flagged
When I check if "<key>" is a sensitive key
Then the sensitivity check should be false
Examples:
| key |
| username |
| email |
| name |
| description |
| project_id |
| token_count |
# --- Database URL masking ---
Scenario: SQLite URL is not masked
Given a database URL "sqlite:///path/to/db.sqlite"
When I mask the database URL
Then the masked URL should be "sqlite:///path/to/db.sqlite"
Scenario: PostgreSQL URL with credentials is masked
Given a database URL "postgresql://admin:s3cret@localhost:5432/mydb"
When I mask the database URL
Then the masked URL should contain "***"
And the masked URL should contain "localhost"
And the masked URL should not contain "s3cret"
# --- Global show_secrets flag ---
Scenario: Global show_secrets defaults to false
When I check the global show_secrets flag
Then the flag should be false
Scenario: Global show_secrets can be toggled
When I set the global show_secrets flag to true
Then the flag should be true
When I set the global show_secrets flag to false
Then the flag should be false
# --- structlog processor ---
Scenario: Structlog masking processor redacts event dict values
Given a structlog event dict with key "api_key" and value "sk-proj-secret123"
When the masking processor is applied
Then the event dict key "api_key" should be "***REDACTED***"
Scenario: Structlog masking processor redacts event message
Given a structlog event dict with event "Failed with key sk-ant-api03-xyz"
When the masking processor is applied
Then the event message should contain "***REDACTED***"
And the event message should not contain "sk-ant-api03-xyz"
Scenario: Structlog masking processor passes non-secret events
Given a structlog event dict with event "User logged in successfully"
When the masking processor is applied
Then the event message should be "User logged in successfully"
# --- CLI error detail redaction ---
Scenario: Error details with secret values are redacted
Given error details with key "api_key" and value "sk-proj-leaked"
When I redact the error details for CLI display
Then the redacted details key "api_key" should be "***REDACTED***"
Scenario: Error details with non-secret values are preserved
Given error details with key "plan_id" and value "01HXR123"
When I redact the error details for CLI display
Then the redacted details key "plan_id" should be "01HXR123"
# --- Secret pattern registration ---
Scenario: Custom secret pattern can be registered
Given the default secret patterns
When I register a custom pattern "custom_[a-z0-9]{16}"
And I redact the string "my custom_abcdef1234567890 key"
Then the redacted value should contain "***REDACTED***"
And the redacted value should not contain "custom_abcdef1234567890"
# ============================================================
# Originally from: security_templates.feature
# Feature: Secure template rendering
# ============================================================
Scenario: Render a simple placeholder
Given a secure template test environment
Given the secure template text "{greeting} {name}"
And a secure template context key "greeting" with value "Hello"
And a secure template context key "name" with value "World"
When I render the secure template
Then the secure template output should be "Hello World"
Scenario: Render with missing placeholder leaves it intact
Given a secure template test environment
Given the secure template text "Hello {name}, your role is {role}"
And a secure template context key "name" with value "Alice"
When I render the secure template
Then the secure template output should be "Hello Alice, your role is {role}"
Scenario: Render with empty context returns template as-is
Given a secure template test environment
Given the secure template text "No placeholders here"
When I render the secure template
Then the secure template output should be "No placeholders here"
Scenario: Render with numeric values converts to string
Given a secure template test environment
Given the secure template text "Count: {count}"
And a secure template context key "count" with value "42"
When I render the secure template
Then the secure template output should be "Count: 42"
# -- Security rejection ---------------------------------------------------
Scenario: Reject attribute access in template
Given a secure template test environment
Given the secure template text "{obj.__class__.__name__}"
When I render the secure template expecting a security error
Then the secure template error should mention "Unsafe"
Scenario: Reject index access in template
Given a secure template test environment
Given the secure template text "{items[0]}"
When I render the secure template expecting a security error
Then the secure template error should mention "Unsafe"
Scenario: Reject format spec in template
Given a secure template test environment
Given the secure template text "{value:>10}"
When I render the secure template expecting a security error
Then the secure template error should mention "Unsafe"
Scenario: Reject conversion flag in template
Given a secure template test environment
Given the secure template text "{value!r}"
When I render the secure template expecting a security error
Then the secure template error should mention "Unsafe"
Scenario: Reject function call in template
Given a secure template test environment
Given the secure template text "{fn()}"
When I render the secure template expecting a security error
Then the secure template error should mention "Unsafe"
Scenario: Reject Jinja2 expression delimiters
Given a secure template test environment
Given the secure template text "{{ config }}"
When I render the secure template expecting a security error
Then the secure template error should mention "Unsafe"
Scenario: Reject Jinja2 block delimiters
Given a secure template test environment
Given the secure template text "{% for x in items %}x{% endfor %}"
When I render the secure template expecting a security error
Then the secure template error should mention "Unsafe"
# -- Size limits -----------------------------------------------------------
Scenario: Reject template exceeding max length
Given a secure template test environment
Given a secure template that is 11000 characters long
When I render the secure template expecting a size error
Then the secure template error should mention "exceeds maximum"
Scenario: Reject output exceeding max length
Given a secure template test environment
Given the secure template text "{big}"
And a secure template config with max output length of 20
And a secure template context key "big" with value "This value is definitely longer than twenty characters"
When I render the secure template expecting a size error
Then the secure template error should mention "exceeds maximum"
# -- Allowlist enforcement ------------------------------------------------
Scenario: Reject template key not in allowlist
Given a secure template test environment
Given the secure template text "{name} {secret}"
And a secure template with allowed keys "name"
And a secure template context key "name" with value "Alice"
And a secure template context key "secret" with value "hidden"
When I render the secure template expecting a validation error
Then the secure template error should mention "disallowed"
Scenario: Accept template key that is in allowlist
Given a secure template test environment
Given the secure template text "{name}"
And a secure template with allowed keys "name"
And a secure template context key "name" with value "Alice"
When I render the secure template
Then the secure template output should be "Alice"
Scenario: Open allowlist allows any simple key
Given a secure template test environment
Given the secure template text "{anything}"
And the secure template has no key restrictions
And a secure template context key "anything" with value "works"
When I render the secure template
Then the secure template output should be "works"
# -- Pre-validation -------------------------------------------------------
Scenario: Validate clean template returns no errors
Given a secure template test environment
Given the secure template text "{name} {role}"
When I validate the secure template
Then the secure template validation should be empty
Scenario: Validate unsafe template returns errors
Given a secure template test environment
Given the secure template text "{obj.__class__}"
When I validate the secure template
Then the secure template validation should not be empty
And the secure template validation should mention "Unsafe"
Scenario: Validate template with unknown keys returns errors
Given a secure template test environment
Given the secure template text "{name} {secret}"
And a secure template with allowed keys "name"
When I validate the secure template
Then the secure template validation should not be empty
And the secure template validation should mention "Unknown"
Scenario: Validate oversized template returns errors
Given a secure template test environment
Given a secure template that is 11000 characters long
When I validate the secure template
Then the secure template validation should not be empty
And the secure template validation should mention "exceeds"
# -- Convenience function -------------------------------------------------
Scenario: render_template_secure convenience function works
Given a secure template test environment
Given the secure template text "{greeting}"
And a secure template context key "greeting" with value "Hi"
When I render via the secure convenience function
Then the secure template output should be "Hi"
Scenario: render_template_secure rejects unsafe input
Given a secure template test environment
Given the secure template text "{obj.attr}"
When I render via the secure convenience function expecting an error
Then the secure template error should mention "Unsafe"
# -- Config property access ------------------------------------------------
Scenario: Renderer exposes active configuration
Given a secure template test environment
Given a secure template with allowed keys "name"
When I create a renderer with those settings
Then the renderer config allowed keys should contain "name"
# -- Validate with matching allowlist (no unknown keys) -------------------
Scenario: Validate template where all keys match allowlist
Given a secure template test environment
Given the secure template text "{name}"
And a secure template with allowed keys "name"
When I validate the secure template
Then the secure template validation should be empty
# -- Rejected keys without logging ----------------------------------------
Scenario: Reject unknown key without logging when log_rejected is false
Given a secure template test environment
Given the secure template text "{name} {secret}"
And a secure template config with logging disabled
And a secure template context key "name" with value "Alice"
And a secure template context key "secret" with value "hidden"
When I render the secure template expecting a validation error
Then the secure template error should mention "disallowed"
# -- reject_unknown_keys=False path ----------------------------------------
Scenario: Allow unknown keys when reject_unknown_keys is disabled
Given a secure template test environment
Given the secure template text "{name} {extra}"
And a secure template config that permits unknown keys
And a secure template context key "name" with value "Alice"
And a secure template context key "extra" with value "data"
When I render the secure template
Then the secure template output should be "Alice data"
# -- Legacy renderer with config passthrough --------------------------------
Scenario: Legacy TemplateRenderer accepts config passthrough
Given a secure template test environment
Given the secure template text "{name}"
And a secure template with allowed keys "name"
And a secure template context key "name" with value "Carol"
When I render via the legacy TemplateRenderer with config
Then the secure template output should be "Carol"
# -- Legacy renderer backward compat --------------------------------------
Scenario: Legacy TemplateRenderer returns raw on unsafe template
Given a secure template test environment
Given the secure template text "{obj.__class__}"
And a secure template context key "obj" with value "test"
When I render via the legacy TemplateRenderer
Then the secure template output should be "{obj.__class__}"
Scenario: Legacy TemplateRenderer renders safe templates
Given a secure template test environment
Given the secure template text "{name}"
And a secure template context key "name" with value "Bob"
When I render via the legacy TemplateRenderer
Then the secure template output should be "Bob"