docs: add configuration reference guide

- Create comprehensive Configuration Reference Guide in docs/guides/configuration-reference.md
- Document all environment variables organized by functional area (runtime, providers, storage, observability, etc.)
- Include configuration loading order, file locations, and best practices
- Add security considerations for secrets management, database, and API keys
- Provide troubleshooting guide for common configuration issues
- Include practical configuration examples for development, production, and Kubernetes deployments
- Update mkdocs.yml to include new Guides section with Configuration Reference link

CHANGELOG.md

@@ -28,6 +28,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
   correctly in all deployment modes: Docker containers, local pip installs
   (wheel or editable), and development environments.
 - **TDD Non-AssertionError Guard Visibility** (#8294): `apply_tdd_inversion` in
+- **bug-hunt-pool-supervisor Non-Blocking Tracking** (#8835): The automation-tracking-manager
+  call in step 5 was blocking the main loop indefinitely, causing 3+ consecutive initialization
+  failures. Step 5 now explicitly marks tracking as best-effort -- if the call does not complete
+  within a reasonable time or fails, it is skipped and the supervisor continues to the next
+  cycle. A new Rule 9 reinforces that tracking must never block the main loop; core
+  functionality (module mapping, worker dispatch, monitoring) takes priority over status
+  reporting.
+
   `features/environment.py` now emits its non-assertion exception guard warning to
   both the structured logger and `stderr` via a new `_warning_with_stderr` helper.
   This makes the guard firing visible in standard Behave console output and CI log

docs/guides/configuration-reference.md

@@ -0,0 +1,453 @@
+# Configuration Reference Guide
+
+This guide provides a comprehensive reference for configuring CleverAgents. Configuration is primarily managed through environment variables, with sensible defaults for most settings.
+
+## Overview
+
+CleverAgents uses a **Pydantic-based settings system** that reads configuration from environment variables with the `CLEVERAGENTS_` prefix. The configuration system is designed to be:
+
+- **Environment-first**: All configuration comes from environment variables, making it ideal for containerized and cloud-native deployments
+- **Hierarchical**: Settings are organized by functional area (runtime, providers, storage, observability, etc.)
+- **Validated**: All configuration values are validated at startup with clear error messages
+- **Sensible defaults**: Most settings have reasonable defaults suitable for development and production
+
+### Configuration Loading Order
+
+1. **Environment variables** (highest priority) — `CLEVERAGENTS_*` prefixed variables
+2. **Provider-specific environment variables** — Native provider variables (e.g., `OPENAI_API_KEY`)
+3. **Default values** — Hardcoded defaults in the settings schema
+
+When a setting is not explicitly configured, the system falls back to the next level in the order above.
+
+### Configuration File Locations
+
+CleverAgents stores configuration and data in the following locations:
+
+| Purpose | Default Location | Environment Variable |
+|---------|------------------|----------------------|
+| User data (sessions, personas, etc.) | `~/.cleveragents/` | `CLEVERAGENTS_DATA_DIR` |
+| Database | `~/.cleveragents/cleveragents.db` | `CLEVERAGENTS_DATABASE_URL` |
+| Logs | `./logs/` | `CLEVERAGENTS_LOG_DIR` |
+| Storage (plans, resources, etc.) | `./data/` | `CLEVERAGENTS_STORAGE_BASE_PATH` |
+| Vector store | `./.cleveragents/vector_store/` | `CLEVERAGENTS_VECTOR_STORE_PATH` |
+
+## Environment Variables Reference
+
+All environment variables are organized by functional area. Each entry includes:
+
+- **Variable name** — The full `CLEVERAGENTS_*` environment variable name
+- **Type** — The expected data type (string, integer, boolean, etc.)
+- **Default** — The default value if not set
+- **Description** — What the setting controls
+- **Example** — A practical example value
+
+### Runtime & Server Configuration
+
+#### `CLEVERAGENTS_ENV`
+- **Type:** String
+- **Default:** `development`
+- **Valid values:** `development`, `production`, `staging`, `test`
+- **Description:** The runtime environment. Controls logging verbosity, debug features, and validation strictness.
+- **Example:** `export CLEVERAGENTS_ENV=production`
+
+#### `CLEVERAGENTS_DEBUG_ENABLED`
+- **Type:** Boolean
+- **Default:** `false`
+- **Description:** Enable debug mode with verbose logging and additional runtime checks. Should never be enabled in production.
+- **Example:** `export CLEVERAGENTS_DEBUG_ENABLED=true`
+
+#### `CLEVERAGENTS_SERVER_HOST`
+- **Type:** String (IP address)
+- **Default:** `0.0.0.0`
+- **Description:** The host address for the CleverAgents server to bind to.
+- **Example:** `export CLEVERAGENTS_SERVER_HOST=127.0.0.1`
+
+#### `CLEVERAGENTS_SERVER_PORT`
+- **Type:** Integer
+- **Default:** `8080`
+- **Valid range:** 1-65535
+- **Description:** The port for the CleverAgents server to listen on.
+- **Example:** `export CLEVERAGENTS_SERVER_PORT=9000`
+
+#### `CLEVERAGENTS_SERVER_RELOAD`
+- **Type:** Boolean
+- **Default:** `false`
+- **Description:** Enable auto-reload on code changes (development only). Never use in production.
+- **Example:** `export CLEVERAGENTS_SERVER_RELOAD=true`
+
+#### `CLEVERAGENTS_SERVER_URL`
+- **Type:** String (URL)
+- **Default:** `None` (local mode)
+- **Description:** URL of a remote CleverAgents server to connect to. When set, the CLI operates in server mode.
+- **Example:** `export CLEVERAGENTS_SERVER_URL=https://cleveragents.example.com`
+
+#### `CLEVERAGENTS_SERVER_TOKEN`
+- **Type:** String
+- **Default:** `None`
+- **Description:** Authentication token for connecting to a remote CleverAgents server. Required when `CLEVERAGENTS_SERVER_URL` is set.
+- **Example:** `export CLEVERAGENTS_SERVER_TOKEN=sk-1234567890abcdef`
+
+### AI Provider Configuration
+
+#### Provider API Keys
+
+Each AI provider requires its API key to be configured. CleverAgents supports multiple providers simultaneously and automatically selects the best available one.
+
+| Provider | Primary Variable | Fallback Variables | Example |
+|----------|------------------|-------------------|---------|
+| OpenAI | `OPENAI_API_KEY` | `CLEVERAGENTS_OPENAI_API_KEY` | `sk-proj-...` |
+| Anthropic | `ANTHROPIC_API_KEY` | `CLEVERAGENTS_ANTHROPIC_API_KEY` | `sk-ant-...` |
+| Google | `GOOGLE_API_KEY` | `GOOGLE_GENAI_API_KEY`, `CLEVERAGENTS_GOOGLE_API_KEY` | `AIza...` |
+| Azure OpenAI | `AZURE_OPENAI_API_KEY` | `AZURE_API_KEY`, `CLEVERAGENTS_AZURE_API_KEY` | `...` |
+| OpenRouter | `OPENROUTER_API_KEY` | `CLEVERAGENTS_OPENROUTER_API_KEY` | `sk-or-...` |
+| Gemini | `GEMINI_API_KEY` | `GOOGLE_GEMINI_API_KEY`, `CLEVERAGENTS_GEMINI_API_KEY` | `AIza...` |
+| Cohere | `COHERE_API_KEY` | `CLEVERAGENTS_COHERE_API_KEY` | `...` |
+| Groq | `GROQ_API_KEY` | `CLEVERAGENTS_GROQ_API_KEY` | `gsk_...` |
+| Together | `TOGETHER_API_KEY` | `CLEVERAGENTS_TOGETHER_API_KEY` | `...` |
+| HuggingFace | `HF_TOKEN` | `HUGGINGFACEHUB_API_TOKEN`, `HUGGING_FACE_HUB_TOKEN` | `hf_...` |
+| Perplexity | `PERPLEXITY_API_KEY` | `CLEVERAGENTS_PERPLEXITY_API_KEY` | `pplx-...` |
+
+#### `CLEVERAGENTS_DEFAULT_PROVIDER`
+- **Type:** String
+- **Default:** `None` (auto-detect from configured providers)
+- **Valid values:** `openai`, `anthropic`, `google`, `azure`, `openrouter`, `gemini`, `cohere`, `groq`, `together`, `huggingface`, `perplexity`
+- **Description:** Pin the default AI provider. When not set, the system automatically selects the first available configured provider in the fallback order.
+- **Example:** `export CLEVERAGENTS_DEFAULT_PROVIDER=anthropic`
+
+#### `CLEVERAGENTS_DEFAULT_MODEL`
+- **Type:** String
+- **Default:** `None` (provider-specific default)
+- **Description:** Pin the default model for the selected provider. When not set, each provider's published default is used (e.g., `gpt-4o` for OpenAI, `claude-sonnet-4-20250514` for Anthropic).
+- **Example:** `export CLEVERAGENTS_DEFAULT_MODEL=gpt-4-turbo`
+
+#### `CLEVERAGENTS_MOCK_PROVIDERS`
+- **Type:** Boolean
+- **Default:** `false`
+- **Description:** Enable mock AI providers for testing. Must not be used in production. Useful for development and CI/CD pipelines.
+- **Example:** `export CLEVERAGENTS_MOCK_PROVIDERS=true`
+
+#### `CLEVERAGENTS_TESTING_USE_MOCK_AI`
+- **Type:** Boolean
+- **Default:** `false`
+- **Description:** Force the in-repo mock provider for Behave/Robot test suites to prevent external API calls.
+- **Example:** `export CLEVERAGENTS_TESTING_USE_MOCK_AI=true`
+
+### Logging & Paths
+
+#### `CLEVERAGENTS_LOG_LEVEL`
+- **Type:** String
+- **Default:** `INFO`
+- **Valid values:** `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`
+- **Description:** The logging level for standard output and file logs.
+- **Example:** `export CLEVERAGENTS_LOG_LEVEL=DEBUG`
+
+#### `CLEVERAGENTS_LOG_DIR`
+- **Type:** Path
+- **Default:** `./logs/`
+- **Description:** Directory where log files are written.
+- **Example:** `export CLEVERAGENTS_LOG_DIR=/var/log/cleveragents`
+
+#### `CLEVERAGENTS_DATA_DIR`
+- **Type:** Path
+- **Default:** `~/.cleveragents/`
+- **Description:** Directory for user data, sessions, personas, and other persistent state.
+- **Example:** `export CLEVERAGENTS_DATA_DIR=/data/cleveragents`
+
+#### `CLEVERAGENTS_STORAGE_BASE_PATH`
+- **Type:** Path
+- **Default:** `./data/`
+- **Description:** Base directory for plan storage, checkpoints, and other runtime data.
+- **Example:** `export CLEVERAGENTS_STORAGE_BASE_PATH=/storage/cleveragents`
+
+### Database & Persistence
+
+#### `CLEVERAGENTS_DATABASE_URL`
+- **Type:** String (database URL)
+- **Default:** `sqlite:///$HOME/.cleveragents/cleveragents.db`
+- **Supported formats:**
+  - SQLite: `sqlite:///path/to/database.db`
+  - PostgreSQL: `postgresql://user:password@host:port/database`
+  - MySQL: `mysql+pymysql://user:password@host:port/database`
+  - DuckDB: `duckdb:///path/to/database.duckdb`
+- **Description:** The database URL for storing sessions, plans, and other persistent data.
+- **Example:** `export CLEVERAGENTS_DATABASE_URL=postgresql://user:pass@localhost/cleveragents`
+
+### Cost Controls & Budgets
+
+#### `CLEVERAGENTS_SESSION_MAX_COST_USD`
+- **Type:** Float
+- **Default:** `None` (unlimited)
+- **Valid range:** >= 0.0
+- **Description:** Maximum USD spend per session across all plans. `None` means unlimited.
+- **Example:** `export CLEVERAGENTS_SESSION_MAX_COST_USD=100.00`
+
+#### `CLEVERAGENTS_ORG_MAX_COST_USD`
+- **Type:** Float
+- **Default:** `None` (unlimited)
+- **Valid range:** >= 0.0
+- **Description:** Maximum USD spend per organization across all sessions. `None` means unlimited.
+- **Example:** `export CLEVERAGENTS_ORG_MAX_COST_USD=1000.00`
+
+#### `CLEVERAGENTS_BUDGET_PER_PLAN`
+- **Type:** Float
+- **Default:** `None` (unlimited)
+- **Valid range:** >= 0.0
+- **Description:** Maximum USD spend per plan execution. `None` means unlimited.
+- **Example:** `export CLEVERAGENTS_BUDGET_PER_PLAN=50.00`
+
+#### `CLEVERAGENTS_BUDGET_PER_DAY`
+- **Type:** Float
+- **Default:** `None` (unlimited)
+- **Valid range:** >= 0.0
+- **Description:** Maximum USD spend per calendar day across all plans. `None` means unlimited.
+- **Example:** `export CLEVERAGENTS_BUDGET_PER_DAY=500.00`
+
+### Observability & Logging
+
+#### LangSmith Tracing
+
+#### `CLEVERAGENTS_LANGSMITH_ENABLED`
+- **Type:** Boolean
+- **Default:** `false`
+- **Aliases:** `LANGCHAIN_TRACING_V2`, `LANGSMITH_TRACING_V2`
+- **Description:** Enable LangSmith tracing for LangChain/LangGraph agents.
+- **Example:** `export CLEVERAGENTS_LANGSMITH_ENABLED=true`
+
+#### `CLEVERAGENTS_LANGSMITH_API_KEY`
+- **Type:** String
+- **Default:** `None`
+- **Aliases:** `LANGSMITH_API_KEY`, `LANGCHAIN_API_KEY`
+- **Description:** API key for LangSmith. Required when tracing is enabled.
+- **Example:** `export CLEVERAGENTS_LANGSMITH_API_KEY=ls_...`
+
+#### `CLEVERAGENTS_LANGSMITH_PROJECT`
+- **Type:** String
+- **Default:** `None`
+- **Aliases:** `LANGSMITH_PROJECT`, `LANGCHAIN_PROJECT`
+- **Description:** LangSmith project name. Required when tracing is enabled.
+- **Example:** `export CLEVERAGENTS_LANGSMITH_PROJECT=my-project`
+
+### Cleanup & Retention Policies
+
+#### `CLEVERAGENTS_CLEANUP_SANDBOX_MAX_AGE_HOURS`
+- **Type:** Integer
+- **Default:** `48`
+- **Valid range:** >= 1
+- **Description:** Maximum age (in hours) before stale sandboxes are eligible for cleanup.
+- **Example:** `export CLEVERAGENTS_CLEANUP_SANDBOX_MAX_AGE_HOURS=72`
+
+#### `CLEVERAGENTS_CHECKPOINT_MAX`
+- **Type:** Integer
+- **Default:** `50`
+- **Valid range:** >= 2
+- **Description:** Maximum number of checkpoints to keep per plan. Oldest checkpoints are pruned first, but the first and most recent are always preserved.
+- **Example:** `export CLEVERAGENTS_CHECKPOINT_MAX=100`
+
+### Audit Logging
+
+#### `CLEVERAGENTS_AUDIT_RETENTION_DAYS`
+- **Type:** Integer
+- **Default:** `0` (keep indefinitely)
+- **Valid range:** >= 0
+- **Description:** Days to retain audit log entries before pruning. `0` means keep indefinitely (spec default for compliance).
+- **Example:** `export CLEVERAGENTS_AUDIT_RETENTION_DAYS=365`
+
+#### `CLEVERAGENTS_AUDIT_ASYNC`
+- **Type:** Boolean
+- **Default:** `true`
+- **Description:** When true, audit entries are written asynchronously via a write-behind queue on a background thread. Set to false for synchronous behavior (useful for debugging).
+- **Example:** `export CLEVERAGENTS_AUDIT_ASYNC=false`
+
+### Metrics & Observability
+
+#### `CLEVERAGENTS_METRICS_ENABLED`
+- **Type:** Boolean
+- **Default:** `true`
+- **Description:** Enable structured metric collection and emission.
+- **Example:** `export CLEVERAGENTS_METRICS_ENABLED=false`
+
+#### `CLEVERAGENTS_METRICS_EXPORT_PROMETHEUS`
+- **Type:** Boolean
+- **Default:** `false`
+- **Description:** Enable Prometheus metrics export endpoint.
+- **Example:** `export CLEVERAGENTS_METRICS_EXPORT_PROMETHEUS=true`
+
+### Retry & Circuit Breaker Configuration
+
+#### `CLEVERAGENTS_RETRY_MAX_ATTEMPTS`
+- **Type:** Integer
+- **Default:** `3`
+- **Valid range:** 1-50
+- **Description:** Default maximum retry attempts for service operations.
+- **Example:** `export CLEVERAGENTS_RETRY_MAX_ATTEMPTS=5`
+
+#### `CLEVERAGENTS_RETRY_BASE_DELAY`
+- **Type:** Float
+- **Default:** `1.0`
+- **Valid range:** 0.0-300.0
+- **Description:** Default base delay in seconds between retries.
+- **Example:** `export CLEVERAGENTS_RETRY_BASE_DELAY=0.5`
+
+#### `CLEVERAGENTS_RETRY_MAX_DELAY`
+- **Type:** Float
+- **Default:** `60.0`
+- **Valid range:** 0.0-3600.0
+- **Description:** Default maximum delay in seconds between retries.
+- **Example:** `export CLEVERAGENTS_RETRY_MAX_DELAY=120.0`
+
+#### `CLEVERAGENTS_RETRY_BACKOFF_STRATEGY`
+- **Type:** String
+- **Default:** `exponential`
+- **Valid values:** `exponential`, `linear`, `fixed`, `jitter`, `none`
+- **Description:** Default backoff strategy for retries.
+- **Example:** `export CLEVERAGENTS_RETRY_BACKOFF_STRATEGY=linear`
+
+### Security & Secrets
+
+#### `CLEVERAGENTS_SHOW_SECRETS`
+- **Type:** Boolean
+- **Default:** `false`
+- **Description:** When true, secrets are shown in CLI output and logs. Should never be enabled in production.
+- **Example:** `export CLEVERAGENTS_SHOW_SECRETS=true`
+
+### Automation & Output
+
+#### `CLEVERAGENTS_AUTOMATION_PROFILE`
+- **Type:** String
+- **Default:** `` (empty, manual mode)
+- **Valid values:** `manual`, `auto`, `full-auto` (or custom profile names)
+- **Description:** Default automation profile name controlling plan execution behavior.
+- **Example:** `export CLEVERAGENTS_AUTOMATION_PROFILE=auto`
+
+#### `CLEVERAGENTS_FORMAT`
+- **Type:** String
+- **Default:** `None` (uses default format)
+- **Valid values:** `json`, `markdown`, `text`, `yaml`
+- **Description:** Default output format for CLI commands.
+- **Example:** `export CLEVERAGENTS_FORMAT=json`
+
+## Configuration Best Practices
+
+### Development Environment
+
+For local development, create a `.env` file in your project root:
+
+```bash
+# .env
+CLEVERAGENTS_ENV=development
+CLEVERAGENTS_DEBUG_ENABLED=true
+CLEVERAGENTS_LOG_LEVEL=DEBUG
+OPENAI_API_KEY=sk-proj-...
+CLEVERAGENTS_DATABASE_URL=sqlite:///./dev.db
+```
+
+### Production Environment
+
+For production deployments:
+
+```bash
+# Use strong, randomly generated secrets
+export CLEVERAGENTS_ENV=production
+export CLEVERAGENTS_DEBUG_ENABLED=false
+export CLEVERAGENTS_LOG_LEVEL=WARNING
+export CLEVERAGENTS_SHOW_SECRETS=false
+
+# Use a production database
+export CLEVERAGENTS_DATABASE_URL=postgresql://user:password@db.example.com/cleveragents
+
+# Configure cost controls
+export CLEVERAGENTS_SESSION_MAX_COST_USD=100
+export CLEVERAGENTS_ORG_MAX_COST_USD=1000
+
+# Enable observability
+export CLEVERAGENTS_METRICS_ENABLED=true
+export CLEVERAGENTS_LANGSMITH_ENABLED=true
+export CLEVERAGENTS_LANGSMITH_API_KEY=ls_...
+export CLEVERAGENTS_LANGSMITH_PROJECT=production
+```
+
+### Multi-Provider Setup
+
+To support multiple AI providers with automatic fallback:
+
+```bash
+# Configure multiple providers
+export OPENAI_API_KEY=sk-proj-...
+export ANTHROPIC_API_KEY=sk-ant-...
+export GOOGLE_API_KEY=AIza...
+
+# Set primary provider (optional)
+export CLEVERAGENTS_DEFAULT_PROVIDER=anthropic
+```
+
+## Security Considerations
+
+### Secrets Management
+
+1. **Never commit secrets** — Use environment variables or a secrets manager (Vault, AWS Secrets Manager, etc.)
+2. **Use strong API keys** — Ensure all provider API keys are strong and rotated regularly
+3. **Restrict access** — Limit who can view or modify configuration in your deployment
+4. **Audit logging** — Enable audit logging to track configuration changes and API usage
+5. **Redaction** — CleverAgents automatically redacts secrets in logs and output
+
+### Database Security
+
+- **Use encrypted connections** — Always use TLS/SSL for database connections
+- **Strong credentials** — Use strong, randomly generated database passwords
+- **Network isolation** — Restrict database access to authorized services only
+
+### API Key Security
+
+- **Rotate keys regularly** — Implement automated key rotation (e.g., monthly)
+- **Use scoped keys** — If your provider supports it, use keys with minimal required permissions
+- **Monitor usage** — Track API usage and set up alerts for unusual activity
+
+## Troubleshooting Configuration Issues
+
+### No AI Provider Configured
+
+**Error:** `ValueError: No AI providers configured and mock_providers is disabled.`
+
+**Solution:**
+1. Set at least one provider API key: `export OPENAI_API_KEY=sk-proj-...`
+2. Or enable mock providers for testing: `export CLEVERAGENTS_MOCK_PROVIDERS=true`
+3. Verify the key is set: `echo $OPENAI_API_KEY`
+
+### LangSmith Configuration Incomplete
+
+**Error:** `LangSmith API key is required` or `LangSmith project name is required`
+
+**Solution:**
+1. Set both required variables:
+   ```bash
+   export CLEVERAGENTS_LANGSMITH_API_KEY=ls_...
+   export CLEVERAGENTS_LANGSMITH_PROJECT=my-project
+   ```
+2. Or disable LangSmith: `export CLEVERAGENTS_LANGSMITH_ENABLED=false`
+
+### Database Connection Failed
+
+**Error:** `sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server`
+
+**Solution:**
+1. Verify the database URL: `echo $CLEVERAGENTS_DATABASE_URL`
+2. Check database credentials and host
+3. Ensure the database is running and accessible
+
+### Configuration Not Taking Effect
+
+**Problem:** Environment variable changes not reflected
+
+**Solution:**
+1. Verify the variable is set: `env | grep CLEVERAGENTS`
+2. Restart the application (singleton settings are cached)
+3. Check for typos in variable names (case-sensitive)
+4. Ensure the variable is exported: `export CLEVERAGENTS_VAR=value`
+
+## Related Documentation
+
+- [Architecture Guide](../architecture.md) — System design and component overview
+- [API Reference - Configuration](../api/config.md) — Detailed API documentation
+- [Development Guide](../development/quality-automation.md) — Development setup and workflows
+- [Specification](../specification.md) — Authoritative design documentation
+- [FAQ](../faq.md) — Frequently asked questions

mkdocs.yml

@@ -24,6 +24,8 @@ nav:
     - AI Providers: api/providers.md
     - TUI: api/tui.md
     - ACMS / UKO: api/acms.md
+  - Guides:
+    - Configuration Reference: guides/configuration-reference.md
   - Modules:
     - Shell Safety: modules/shell-safety.md
     - UKO Provenance Tracking: modules/uko-provenance.md