khyari hamza
d815339c52
Implement centralized redaction utility that masks API keys, tokens, and credentials across CLI output, structlog logs, and error messages. - Add shared/redaction.py with pattern-based secret detection (sk-*, sk-ant-*, tok_*, Bearer tokens), sensitive key name detection, database URL masking, custom pattern registration, and thread-safe global show_secrets flag - Add config/logging.py with structlog configuration integrating the secrets_masking_processor into the processor chain - Add --show-secrets global CLI option to reveal secrets when needed - Redact error details in main.py, project.py, and auto_debug.py error handlers before printing - Wrap format_output() in formatting.py with automatic dict redaction - Add show_secrets field and safe __repr__ to Settings model - Add 43-scenario Behave feature (features/security_secrets.feature) - Add 10 Robot Framework smoke tests (robot/security_secrets.robot) - Add ASV benchmarks (benchmarks/security_secrets_bench.py) - Add reference docs (docs/reference/secrets_handling.md)
113 lines
3.2 KiB
Markdown
113 lines
3.2 KiB
Markdown
# Secrets Handling (SEC5)
|
|
|
|
CleverAgents automatically masks API keys, tokens, and credentials in
|
|
CLI output, structured logs, and error messages. This document covers
|
|
the design and usage of the redaction subsystem.
|
|
|
|
## Overview
|
|
|
|
The `cleveragents.shared.redaction` module provides centralised,
|
|
pattern-based secret detection and masking. It is integrated into:
|
|
|
|
- **CLI output** via `format_output()` in `cleveragents.cli.formatting`
|
|
- **Error handlers** in `main.py`, `project.py`, and `auto_debug.py`
|
|
- **structlog processors** via `secrets_masking_processor`
|
|
- **Settings repr** via `Settings.__repr__`
|
|
|
|
## Detected patterns
|
|
|
|
| Pattern family | Regex summary | Example |
|
|
|---|---|---|
|
|
| OpenAI keys | `sk-(?:proj-)?[A-Za-z0-9_-]{10,}` | `sk-proj-abc123…` |
|
|
| Anthropic keys | `sk-ant-[A-Za-z0-9_-]{10,}` | `sk-ant-api03-xyz…` |
|
|
| Token IDs | `tok_[A-Za-z0-9]{10,}` | `tok_01HXYZ…` |
|
|
| Bearer tokens | `Bearer\s+[A-Za-z0-9._~+/=-]{20,}` | `Bearer eyJhb…` |
|
|
| Generic keys | `(?:key\|KEY)-[A-Za-z0-9]{20,}` | `KEY-abcdef…` |
|
|
|
|
Sensitive **key names** (dictionary keys, config fields) are also
|
|
detected by substring matching:
|
|
|
|
`api_key`, `apikey`, `password`, `passwd`, `secret`, `token`,
|
|
`credential`, `private_key`, `access_key`, `auth`
|
|
|
|
False-positive key names like `token_count`, `max_tokens`, etc. are
|
|
excluded.
|
|
|
|
## Replacement token
|
|
|
|
All detected secrets are replaced with `***REDACTED***`.
|
|
|
|
## Global `--show-secrets` flag
|
|
|
|
Pass `--show-secrets` to any CLI command to reveal secrets in that
|
|
invocation:
|
|
|
|
```bash
|
|
agents --show-secrets info
|
|
agents --show-secrets diagnostics
|
|
```
|
|
|
|
The flag sets a global thread-safe boolean that the redaction layer
|
|
checks. When `True`, all masking is bypassed.
|
|
|
|
Programmatic access:
|
|
|
|
```python
|
|
from cleveragents.shared.redaction import get_show_secrets, set_show_secrets
|
|
|
|
set_show_secrets(True) # reveal secrets
|
|
set_show_secrets(False) # re-enable masking (default)
|
|
```
|
|
|
|
## Custom patterns
|
|
|
|
Register additional secret patterns at runtime:
|
|
|
|
```python
|
|
from cleveragents.shared.redaction import register_pattern
|
|
|
|
register_pattern(r"myorg_[a-z0-9]{32}")
|
|
```
|
|
|
|
Patterns are compiled once and appended to the global list.
|
|
|
|
## Database URL masking
|
|
|
|
`mask_database_url()` replaces embedded passwords in connection
|
|
strings while preserving the host and database name:
|
|
|
|
```python
|
|
from cleveragents.shared.redaction import mask_database_url
|
|
|
|
mask_database_url("postgresql://user:pass@host/db")
|
|
# → "postgresql://user:***@host/db"
|
|
|
|
mask_database_url("sqlite:///data.db")
|
|
# → "sqlite:///data.db" (unchanged)
|
|
```
|
|
|
|
## structlog integration
|
|
|
|
The `secrets_masking_processor` is inserted into the structlog
|
|
processor chain by `configure_structlog()`:
|
|
|
|
```python
|
|
from cleveragents.config.logging import configure_structlog
|
|
|
|
configure_structlog(env="production", log_level="INFO")
|
|
```
|
|
|
|
Every log event is scanned for secret patterns before rendering.
|
|
|
|
## Settings safety
|
|
|
|
`Settings.__repr__()` masks any field whose name matches a sensitive
|
|
key pattern, so accidental `print(settings)` or debug output never
|
|
leaks credentials.
|
|
|
|
## Testing
|
|
|
|
- **Behave**: `features/security_secrets.feature` (43 scenarios)
|
|
- **Robot Framework**: `robot/security_secrets.robot` (10 smoke tests)
|
|
- **ASV benchmarks**: `benchmarks/security_secrets_bench.py`
|