khyari hamza
d815339c52
Implement centralized redaction utility that masks API keys, tokens, and credentials across CLI output, structlog logs, and error messages. - Add shared/redaction.py with pattern-based secret detection (sk-*, sk-ant-*, tok_*, Bearer tokens), sensitive key name detection, database URL masking, custom pattern registration, and thread-safe global show_secrets flag - Add config/logging.py with structlog configuration integrating the secrets_masking_processor into the processor chain - Add --show-secrets global CLI option to reveal secrets when needed - Redact error details in main.py, project.py, and auto_debug.py error handlers before printing - Wrap format_output() in formatting.py with automatic dict redaction - Add show_secrets field and safe __repr__ to Settings model - Add 43-scenario Behave feature (features/security_secrets.feature) - Add 10 Robot Framework smoke tests (robot/security_secrets.robot) - Add ASV benchmarks (benchmarks/security_secrets_bench.py) - Add reference docs (docs/reference/secrets_handling.md)
3.2 KiB
Secrets Handling (SEC5)
CleverAgents automatically masks API keys, tokens, and credentials in CLI output, structured logs, and error messages. This document covers the design and usage of the redaction subsystem.
Overview
The cleveragents.shared.redaction module provides centralised,
pattern-based secret detection and masking. It is integrated into:
- CLI output via
format_output()incleveragents.cli.formatting - Error handlers in
main.py,project.py, andauto_debug.py - structlog processors via
secrets_masking_processor - Settings repr via
Settings.__repr__
Detected patterns
| Pattern family | Regex summary | Example |
|---|---|---|
| OpenAI keys | sk-(?:proj-)?[A-Za-z0-9_-]{10,} |
sk-proj-abc123… |
| Anthropic keys | sk-ant-[A-Za-z0-9_-]{10,} |
sk-ant-api03-xyz… |
| Token IDs | tok_[A-Za-z0-9]{10,} |
tok_01HXYZ… |
| Bearer tokens | Bearer\s+[A-Za-z0-9._~+/=-]{20,} |
Bearer eyJhb… |
| Generic keys | (?:key|KEY)-[A-Za-z0-9]{20,} |
KEY-abcdef… |
Sensitive key names (dictionary keys, config fields) are also detected by substring matching:
api_key, apikey, password, passwd, secret, token,
credential, private_key, access_key, auth
False-positive key names like token_count, max_tokens, etc. are
excluded.
Replacement token
All detected secrets are replaced with ***REDACTED***.
Global --show-secrets flag
Pass --show-secrets to any CLI command to reveal secrets in that
invocation:
agents --show-secrets info
agents --show-secrets diagnostics
The flag sets a global thread-safe boolean that the redaction layer
checks. When True, all masking is bypassed.
Programmatic access:
from cleveragents.shared.redaction import get_show_secrets, set_show_secrets
set_show_secrets(True) # reveal secrets
set_show_secrets(False) # re-enable masking (default)
Custom patterns
Register additional secret patterns at runtime:
from cleveragents.shared.redaction import register_pattern
register_pattern(r"myorg_[a-z0-9]{32}")
Patterns are compiled once and appended to the global list.
Database URL masking
mask_database_url() replaces embedded passwords in connection
strings while preserving the host and database name:
from cleveragents.shared.redaction import mask_database_url
mask_database_url("postgresql://user:pass@host/db")
# → "postgresql://user:***@host/db"
mask_database_url("sqlite:///data.db")
# → "sqlite:///data.db" (unchanged)
structlog integration
The secrets_masking_processor is inserted into the structlog
processor chain by configure_structlog():
from cleveragents.config.logging import configure_structlog
configure_structlog(env="production", log_level="INFO")
Every log event is scanned for secret patterns before rendering.
Settings safety
Settings.__repr__() masks any field whose name matches a sensitive
key pattern, so accidental print(settings) or debug output never
leaks credentials.
Testing
- Behave:
features/security_secrets.feature(43 scenarios) - Robot Framework:
robot/security_secrets.robot(10 smoke tests) - ASV benchmarks:
benchmarks/security_secrets_bench.py