Secrets Handling (SEC5)

CleverAgents automatically masks API keys, tokens, and credentials in CLI output, structured logs, and error messages. This document covers the design and usage of the redaction subsystem.

Overview

The cleveragents.shared.redaction module provides centralised, pattern-based secret detection and masking. It is integrated into:

CLI output via format_output() in cleveragents.cli.formatting
Error handlers in main.py, project.py, and auto_debug.py
structlog processors via secrets_masking_processor
Settings repr via Settings.__repr__

Detected patterns

Pattern family	Regex summary	Example
OpenAI keys	`sk-(?:proj-)?[A-Za-z0-9_-]{10,}`	`sk-proj-abc123…`
Anthropic keys	`sk-ant-[A-Za-z0-9_-]{10,}`	`sk-ant-api03-xyz…`
Token IDs	`tok_[A-Za-z0-9]{10,}`	`tok_01HXYZ…`
Bearer tokens	`Bearer\s+[A-Za-z0-9._~+/=-]{20,}`	`Bearer eyJhb…`
Generic keys	`(?:key\|KEY)-[A-Za-z0-9]{20,}`	`KEY-abcdef…`

Sensitive key names (dictionary keys, config fields) are also detected by substring matching:

api_key, apikey, password, passwd, secret, token, credential, private_key, access_key, auth

False-positive key names like token_count, max_tokens, etc. are excluded.

Replacement token

All detected secrets are replaced with ***REDACTED***.

Global `--show-secrets` flag

Pass --show-secrets to any CLI command to reveal secrets in that invocation:

agents --show-secrets info
agents --show-secrets diagnostics

The flag sets a global thread-safe boolean that the redaction layer checks. When True, all masking is bypassed.

Programmatic access:

from cleveragents.shared.redaction import get_show_secrets, set_show_secrets

set_show_secrets(True)   # reveal secrets
set_show_secrets(False)  # re-enable masking (default)

Custom patterns

from cleveragents.shared.redaction import register_pattern

register_pattern(r"myorg_[a-z0-9]{32}")

Patterns are compiled once and appended to the global list.

Database URL masking

mask_database_url() replaces embedded passwords in connection strings while preserving the host and database name:

from cleveragents.shared.redaction import mask_database_url

mask_database_url("postgresql://user:pass@host/db")
# → "postgresql://user:***@host/db"

mask_database_url("sqlite:///data.db")
# → "sqlite:///data.db"  (unchanged)

structlog integration

The secrets_masking_processor is inserted into the structlog processor chain by configure_structlog():

from cleveragents.config.logging import configure_structlog

configure_structlog(env="production", log_level="INFO")

Every log event is scanned for secret patterns before rendering.

Settings safety

Settings.__repr__() masks any field whose name matches a sensitive key pattern, so accidental print(settings) or debug output never leaks credentials.

Testing

Behave: features/security_secrets.feature (43 scenarios)
Robot Framework: robot/security_secrets.robot (10 smoke tests)
ASV benchmarks: benchmarks/security_secrets_bench.py

3.2 KiB Raw Permalink Blame History