docs(api): link shared and LSP references

docs/api/index.md

@@ -9,10 +9,12 @@ classes, functions, and exceptions with signatures and usage examples.
 | Module | Description |
 |--------|-------------|
 | [`cleveragents.core`](core.md) | Exception hierarchy, error handling, retry patterns, async cleanup |
+| [`cleveragents.shared`](shared.md) | Secret redaction utilities for logs, telemetry, and structured errors |
 | [`cleveragents.a2a`](a2a.md) | Agent-to-Agent (A2A) protocol facade, transport, models, and events |
 | [`cleveragents.actor`](actor.md) | Actor registry, configuration, loader, and compiler |
 | [`cleveragents.skills`](skills.md) | Skill framework — schema, protocol, registry, discovery, and inline executor |
 | [`cleveragents.tool`](tool.md) | Tool runtime, lifecycle, registry, router, and container executor |
+| [`cleveragents.lsp`](lsp.md) | Language Server Protocol integration — registry, runtime, client, and tool adapter |
 | [`cleveragents.mcp`](mcp.md) | Model Context Protocol (MCP) adapter, client, registry, and sandbox |
 | [`cleveragents.resource`](resource.md) | Resource schema, handlers, and type-inheritance system |
 | [`cleveragents.config`](config.md) | Application settings, logging, metrics, and security scanning |

docs/api/lsp.md

@@ -0,0 +1,252 @@
+# `cleveragents.lsp` — Language Server Protocol Integration
+
+The `cleveragents.lsp` package wires Language Server Protocol (LSP) features
+into CleverAgents. It provides:
+
+- Pydantic models describing servers, transports, and bindings.
+- A thread-safe registry for language server configurations.
+- Runtime orchestration (start, health-check, restart, stop) with automatic
+  reference counting.
+- A synchronous protocol client and lifecycle manager around stdio transports.
+- A tool adapter that exposes LSP capabilities to the generic tool system.
+- A multi-layer language discovery engine that selects servers for files.
+
+All exports listed below are available directly from `cleveragents.lsp`.
+
+---
+
+## Quick start
+
+```python
+from cleveragents.lsp import (
+    LanguageDiscovery,
+    LspCapability,
+    LspRegistry,
+    LspRuntime,
+    LspServerConfig,
+    LspToolAdapter,
+)
+
+registry = LspRegistry()
+registry.register(
+    LspServerConfig(
+        name="local/pyright",
+        description="Pyright language server",
+        command="pyright-langserver",
+        languages=["python"],
+        capabilities=[
+            LspCapability.DIAGNOSTICS,
+            LspCapability.COMPLETIONS,
+            LspCapability.HOVER,
+        ],
+    )
+)
+
+runtime = LspRuntime(registry=registry)
+runtime.start_server("local/pyright", workspace_path="/workspace")
+
+adapter = LspToolAdapter(runtime)
+pyright_tools = adapter.generate_tool_specs(registry.get_or_raise("local/pyright"))
+print([spec["name"] for spec in pyright_tools])
+# ['local/pyright/diagnostics', 'local/pyright/completions', 'local/pyright/hover']
+
+language = LanguageDiscovery().detect_file_language("/workspace/app/models.py")
+print(language)  # 'python'
+```
+
+---
+
+## Models
+
+### `LspTransport`
+
+`StrEnum` describing how the runtime communicates with a server:
+
+- `STDIO` (default) — spawn the server and communicate via stdin/stdout JSON-RPC.
+- `TCP` — connect to a server listening on a TCP socket.
+
+### `LspCapability`
+
+Enum of LSP features that can be surfaced as ToolSpecs. Members map directly to
+protocol operations:
+
+| Capability | Description |
+|------------|-------------|
+| `DIAGNOSTICS` | Fetch diagnostics, warnings, and errors for a file. |
+| `HOVER` | Retrieve hover information (type hints, docs) at a position. |
+| `COMPLETIONS` | Request completion candidates at a position. |
+| `DEFINITIONS` | Go to definition for a symbol. |
+| `REFERENCES` | Find references to a symbol. |
+| `RENAME` | Request a rename operation across the workspace. |
+| `CODE_ACTIONS` | Obtain code actions for a range. |
+| `FORMATTING` | Format a document or selection. |
+| `SIGNATURE_HELP` | Fetch call signature help. |
+| `DOCUMENT_SYMBOLS` | List symbols defined in a document. |
+| `WORKSPACE_SYMBOLS` | Search for symbols across the workspace. |
+
+> **Note:** The runtime currently provides concrete handlers for diagnostics,
+> completions, hover, and definitions. Other capabilities raise
+> `LspNotAvailableError` until their protocol flows are implemented.
+
+### `LspServerConfig`
+
+Pydantic model describing a language server.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | `str` | Namespaced identifier (`namespace/name`); validated to contain a slash. |
+| `description` | `str` | Optional human-readable description. |
+| `languages` | `list[str]` | Lowercased language identifiers served by the process. |
+| `command` | `str` | Executable used to start the server. |
+| `args` | `list[str]` | Extra CLI arguments. |
+| `transport` | `LspTransport` | Communication mode (`STDIO` by default). |
+| `env` | `dict[str, str]` | Environment overrides for the process. |
+| `capabilities` | `list[LspCapability]` | Enabled protocol features. |
+| `initialization` | `dict[str, Any]` | `initialize` request `initializationOptions`. |
+| `workspace_settings` | `dict[str, Any]` | Settings pushed after initialization. |
+
+Helper properties:
+
+- `namespace` — portion before the last `/`.
+- `short_name` — portion after the last `/`.
+- `to_dict()` — JSON-compatible serialisation respecting model validators.
+
+### `LspBinding`
+
+Binds a server to an actor graph node. Used by the compiler/runtime to know
+which servers to activate. Validates that `lsp_server_name` is namespaced and
+supports optional per-node language filters.
+
+---
+
+## Registry and discovery
+
+### `LspRegistry`
+
+Thread-safe registry for `LspServerConfig` objects. Public methods acquire a
+re-entrant lock, making concurrent registration and lookup safe.
+
+- `register(config)` — add a new server; rejects duplicates.
+- `get(name)` — return a config or `None` when missing.
+- `get_or_raise(name)` — raise `LspServerNotFoundError` if absent.
+- `list_servers(namespace=None, language=None)` — filter by namespace prefix or
+  supported language, sorted by name.
+- `remove(name)` — unregister a server; returns `True` if removed.
+
+### `LanguageDiscovery`
+
+Four-layer language detection engine used by the runtime and compiler. Layers
+are consulted in priority order:
+
+1. File extension (`.py`, `.tsx`, …) via a built-in lookup table.
+2. Shebang inspection for ambiguous or extension-less files.
+3. Optional UKO (Universal Knowledge Ontology) classifications supplied at
+   construction.
+4. Explicit project-level language declarations passed to the constructor.
+
+Key methods:
+
+- `detect_file_language(path)` — returns a language ID (falls back to
+  `"plaintext"`). Results are cached per path.
+- `detect_directory_languages(path)` — walk a tree and return sorted unique
+  languages (ignoring `plaintext`).
+- `invalidate(path)` / `invalidate_all()` — manage the detection cache.
+- `get_servers_for_language(language, configs)` — helper to match registered
+  servers to a detected language.
+
+---
+
+## Runtime orchestration
+
+### `LspLifecycleManager`
+
+Manages running server processes with reference counting. Guarantees that the
+underlying transport is started only once per server/workspace combination and
+shutdown occurs when the last reference is released. Provides:
+
+- `start_server(config, workspace_path)` — start or reuse a server, returning an
+  `LspClient`.
+- `stop_server(name)` — release a reference; raises `LspServerNotFoundError` if
+  the server was never acquired.
+- `health_check(name)` — return `True` if the process is alive.
+- `restart_server(name)` — restart a crashed server while preserving reference
+  counts.
+- `stop_all()` / `list_running()` — maintenance helpers for graceful shutdown
+  and observability.
+
+### `LspClient`
+
+Synchronous JSON-RPC client layered on top of `StdioTransport`. Exposes typed
+methods for the protocol flows currently needed by CleverAgents:
+
+- `initialize(workspace_path, initialization_options=None)` and `shutdown()` for
+  lifecycle management.
+- Document operations: `did_open`, `did_close`, `get_diagnostics`,
+  `get_completions`, `get_hover`, `get_definitions`.
+- Accessors `is_initialized` and `server_capabilities` for telemetry or testing.
+
+Internally the client keeps a bounded queue of notifications, correlates
+responses by ID, and raises `LspError` on timeouts or protocol failures.
+
+### `LspRuntime`
+
+High-level facade combining the registry and lifecycle manager. It performs
+input validation, logs significant events, and exposes convenience methods:
+
+- `start_server(name, workspace_path)` / `stop_server(name)` — manage process
+  lifecycle.
+- `get_diagnostics`, `get_completions`, `get_hover`, `get_definitions` — delegate
+  to the underlying `LspClient`, reading file contents and translating between
+  filesystem paths and LSP URIs.
+- `activate_bindings(bindings, workspace_path)` and `deactivate_bindings(bindings)` —
+  start/stop all servers referenced by `LspBinding` objects produced during actor
+  compilation.
+- `stop_all()` — forward to `LspLifecycleManager.stop_all()`.
+
+If a server crashes, `LspRuntime` attempts to restart it automatically before
+servicing the next request.
+
+### `StdioTransport`
+
+Process wrapper that launches language servers via subprocess, reads/writes
+JSON-RPC messages, and exposes `is_alive`, `start()`, `stop()`, `send_message()`,
+`read_message()`. It is the default transport used by the lifecycle manager and
+client.
+
+---
+
+## Tool adapter
+
+### `LspToolAdapter`
+
+Generates tool specifications from `LspServerConfig.capabilities` so the
+standard tool runtime can invoke language-server functionality.
+
+```python
+from cleveragents.lsp import LspToolAdapter
+
+adapter = LspToolAdapter(runtime)
+for spec in adapter.generate_tool_specs(config):
+    tool_registry.register(spec)
+```
+
+- When constructed with an `LspRuntime`, capability handlers delegate to runtime
+  methods (diagnostics, completions, hover, definitions today). Missing runtime
+  capabilities raise `LspNotAvailableError` with a structured payload.
+- Without a runtime (local/offline mode), generated handlers always raise
+  `LspNotAvailableError`, signalling that remote execution is required.
+- Each spec includes a minimal JSON Schema input contract tailored to the
+  capability (file-only, position-based, rename, or query-based).
+
+---
+
+## Error types
+
+The package exports structured exceptions for callers to handle:
+
+- `LspError` — generic protocol or transport failure.
+- `LspNotAvailableError` — capability invoked without an available runtime.
+- `LspServerNotFoundError` — registry or lifecycle lookup of an unknown server.
+
+Catch these errors in higher layers to provide user-facing remediation (e.g.
+retry prompts, error banners in the TUI, or fallback tool behaviour).

docs/api/shared.md

@@ -0,0 +1,117 @@
+# `cleveragents.shared` — Secret Redaction Utilities
+
+The `cleveragents.shared` package exposes cross-cutting helpers that can be
+used from any layer without violating architecture boundaries. At the moment
+its public surface focuses on a single responsibility: deterministic masking of
+secrets before they reach logs, error payloads, or tool output.
+
+All exports live in `cleveragents.shared.redaction` and are re-exported from the
+package root for convenience.
+
+---
+
+## Quick start
+
+```python
+from cleveragents.shared import redact_dict, register_pattern
+
+# redact nested structures before logging
+payload = {
+    "provider_api_key": "sk-ant-api03-secret-1234567890",
+    "token_usage": 1280,
+    "upstream": {
+        "authorization": "Bearer 0123456789ABCDEFGHIJKLMNOPQRSTUV",
+    },
+}
+print(redact_dict(payload))
+# {'provider_api_key': '***REDACTED***', 'token_usage': 1280,
+#  'upstream': {'authorization': '***REDACTED***'}}
+
+# extend detection with an internal pattern
+register_pattern(r"my-secret-[0-9a-f]{16}")
+```
+
+---
+
+## Global redaction controls
+
+| Symbol | Description |
+|--------|-------------|
+| `REDACTED` | Constant string (`***REDACTED***`) used as the replacement token. |
+| `get_show_secrets() -> bool` | Returns the global flag that determines whether redaction is bypassed. |
+| `set_show_secrets(value: bool) -> None` | Toggles the flag. Accepts only boolean values and is thread-safe. |
+
+When `get_show_secrets()` returns `True`, the redaction helpers shortcut and
+return the original data unmodified—useful for debugging in trusted
+environments. The flag is protected by a lock so it can be flipped safely from
+CLI options or unit tests.
+
+---
+
+## Key helpers
+
+### `is_sensitive_key(name: str) -> bool`
+
+Heuristically classifies dictionary keys as sensitive (e.g. containing
+"token", "secret", "password"). False positives such as `token_count`
+are explicitly whitelisted in the implementation.
+
+### `redact_value(value: str) -> str`
+
+Scans a string for known secret patterns (OpenAI, Anthropic, Google keys,
+`tok_` IDs, bearer tokens, long hex strings) and replaces each match with
+`REDACTED`. Additional patterns can be registered at runtime via
+`register_pattern()`.
+
+### `redact_dict(data: Mapping[str, Any], *, show_secrets: bool | None = None) -> dict`
+
+Recursively traverses nested dictionaries and lists, redacting sensitive keys
+with `REDACTED` and scrubbing string values via `redact_value`. By default it
+honours the global `get_show_secrets()` flag; override it per call with
+`show_secrets=True` when you explicitly need the original values.
+
+### `mask_database_url(url: str) -> str`
+
+Masks the password segment of SQL connection URLs. SQLite URLs are returned
+unchanged; any `scheme://user:password@host` pattern is rewritten as
+`scheme://user:***@host`.
+
+### `register_pattern(pattern: str) -> None`
+
+Allows projects or extensions to add custom regular expressions to the
+redaction list. Patterns are compiled under a lock so multiple workers can
+register safely. Passing an empty pattern raises `ValueError`.
+
+---
+
+## Structlog integration
+
+`secrets_masking_processor(logger, method_name, event_dict)` implements the
+structlog processor contract. It walks every value in the event dictionary,
+redacting keys detected by `is_sensitive_key` and values using `redact_value`
+and `redact_dict`. The processor is designed to be inserted near the top of the
+structlog pipeline so that downstream formatters never observe raw secrets.
+
+```python
+import structlog
+from cleveragents.shared import secrets_masking_processor
+
+structlog.configure(
+    processors=[
+        secrets_masking_processor,
+        structlog.processors.JSONRenderer(),
+    ],
+)
+```
+
+---
+
+## Thread-safety notes
+
+The module guards both the global `show_secrets` flag and the list of compiled
+patterns with locks. You can safely call `set_show_secrets()` or
+`register_pattern()` from concurrent tasks without corrupting shared state.
+
+Because all helpers return new dictionaries (never mutating inputs), you may
+confidently hand the redacted payloads to logging subsystems or telemetry
+pipelines without affecting the original objects used by business logic.