118 lines
4.1 KiB
Markdown
118 lines
4.1 KiB
Markdown
# `cleveragents.shared` — Secret Redaction Utilities
|
|
|
|
The `cleveragents.shared` package exposes cross-cutting helpers that can be
|
|
used from any layer without violating architecture boundaries. At the moment
|
|
its public surface focuses on a single responsibility: deterministic masking of
|
|
secrets before they reach logs, error payloads, or tool output.
|
|
|
|
All exports live in `cleveragents.shared.redaction` and are re-exported from the
|
|
package root for convenience.
|
|
|
|
---
|
|
|
|
## Quick start
|
|
|
|
```python
|
|
from cleveragents.shared import redact_dict, register_pattern
|
|
|
|
# redact nested structures before logging
|
|
payload = {
|
|
"provider_api_key": "sk-ant-api03-secret-1234567890",
|
|
"token_usage": 1280,
|
|
"upstream": {
|
|
"authorization": "Bearer 0123456789ABCDEFGHIJKLMNOPQRSTUV",
|
|
},
|
|
}
|
|
print(redact_dict(payload))
|
|
# {'provider_api_key': '***REDACTED***', 'token_usage': 1280,
|
|
# 'upstream': {'authorization': '***REDACTED***'}}
|
|
|
|
# extend detection with an internal pattern
|
|
register_pattern(r"my-secret-[0-9a-f]{16}")
|
|
```
|
|
|
|
---
|
|
|
|
## Global redaction controls
|
|
|
|
| Symbol | Description |
|
|
|--------|-------------|
|
|
| `REDACTED` | Constant string (`***REDACTED***`) used as the replacement token. |
|
|
| `get_show_secrets() -> bool` | Returns the global flag that determines whether redaction is bypassed. |
|
|
| `set_show_secrets(value: bool) -> None` | Toggles the flag. Accepts only boolean values and is thread-safe. |
|
|
|
|
When `get_show_secrets()` returns `True`, the redaction helpers shortcut and
|
|
return the original data unmodified—useful for debugging in trusted
|
|
environments. The flag is protected by a lock so it can be flipped safely from
|
|
CLI options or unit tests.
|
|
|
|
---
|
|
|
|
## Key helpers
|
|
|
|
### `is_sensitive_key(name: str) -> bool`
|
|
|
|
Heuristically classifies dictionary keys as sensitive (e.g. containing
|
|
"token", "secret", "password"). False positives such as `token_count`
|
|
are explicitly whitelisted in the implementation.
|
|
|
|
### `redact_value(value: str) -> str`
|
|
|
|
Scans a string for known secret patterns (OpenAI, Anthropic, Google keys,
|
|
`tok_` IDs, bearer tokens, long hex strings) and replaces each match with
|
|
`REDACTED`. Additional patterns can be registered at runtime via
|
|
`register_pattern()`.
|
|
|
|
### `redact_dict(data: Mapping[str, Any], *, show_secrets: bool | None = None) -> dict`
|
|
|
|
Recursively traverses nested dictionaries and lists, redacting sensitive keys
|
|
with `REDACTED` and scrubbing string values via `redact_value`. By default it
|
|
honours the global `get_show_secrets()` flag; override it per call with
|
|
`show_secrets=True` when you explicitly need the original values.
|
|
|
|
### `mask_database_url(url: str) -> str`
|
|
|
|
Masks the password segment of SQL connection URLs. SQLite URLs are returned
|
|
unchanged; any `scheme://user:password@host` pattern is rewritten as
|
|
`scheme://user:***@host`.
|
|
|
|
### `register_pattern(pattern: str) -> None`
|
|
|
|
Allows projects or extensions to add custom regular expressions to the
|
|
redaction list. Patterns are compiled under a lock so multiple workers can
|
|
register safely. Passing an empty pattern raises `ValueError`.
|
|
|
|
---
|
|
|
|
## Structlog integration
|
|
|
|
`secrets_masking_processor(logger, method_name, event_dict)` implements the
|
|
structlog processor contract. It walks every value in the event dictionary,
|
|
redacting keys detected by `is_sensitive_key` and values using `redact_value`
|
|
and `redact_dict`. The processor is designed to be inserted near the top of the
|
|
structlog pipeline so that downstream formatters never observe raw secrets.
|
|
|
|
```python
|
|
import structlog
|
|
from cleveragents.shared import secrets_masking_processor
|
|
|
|
structlog.configure(
|
|
processors=[
|
|
secrets_masking_processor,
|
|
structlog.processors.JSONRenderer(),
|
|
],
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## Thread-safety notes
|
|
|
|
The module guards both the global `show_secrets` flag and the list of compiled
|
|
patterns with locks. You can safely call `set_show_secrets()` or
|
|
`register_pattern()` from concurrent tasks without corrupting shared state.
|
|
|
|
Because all helpers return new dictionaries (never mutating inputs), you may
|
|
confidently hand the redacted payloads to logging subsystems or telemetry
|
|
pipelines without affecting the original objects used by business logic.
|