diff --git a/docs/api/index.md b/docs/api/index.md index 4eee7341e..b195a19d7 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -9,10 +9,12 @@ classes, functions, and exceptions with signatures and usage examples. | Module | Description | |--------|-------------| | [`cleveragents.core`](core.md) | Exception hierarchy, error handling, retry patterns, async cleanup | +| [`cleveragents.shared`](shared.md) | Secret redaction utilities for logs, telemetry, and structured errors | | [`cleveragents.a2a`](a2a.md) | Agent-to-Agent (A2A) protocol facade, transport, models, and events | | [`cleveragents.actor`](actor.md) | Actor registry, configuration, loader, and compiler | | [`cleveragents.skills`](skills.md) | Skill framework — schema, protocol, registry, discovery, and inline executor | | [`cleveragents.tool`](tool.md) | Tool runtime, lifecycle, registry, router, and container executor | +| [`cleveragents.lsp`](lsp.md) | Language Server Protocol integration — registry, runtime, client, and tool adapter | | [`cleveragents.mcp`](mcp.md) | Model Context Protocol (MCP) adapter, client, registry, and sandbox | | [`cleveragents.resource`](resource.md) | Resource schema, handlers, and type-inheritance system | | [`cleveragents.config`](config.md) | Application settings, logging, metrics, and security scanning | diff --git a/docs/api/lsp.md b/docs/api/lsp.md new file mode 100644 index 000000000..9a37ec8ab --- /dev/null +++ b/docs/api/lsp.md @@ -0,0 +1,252 @@ +# `cleveragents.lsp` — Language Server Protocol Integration + +The `cleveragents.lsp` package wires Language Server Protocol (LSP) features +into CleverAgents. It provides: + +- Pydantic models describing servers, transports, and bindings. +- A thread-safe registry for language server configurations. +- Runtime orchestration (start, health-check, restart, stop) with automatic + reference counting. +- A synchronous protocol client and lifecycle manager around stdio transports. +- A tool adapter that exposes LSP capabilities to the generic tool system. +- A multi-layer language discovery engine that selects servers for files. + +All exports listed below are available directly from `cleveragents.lsp`. + +--- + +## Quick start + +```python +from cleveragents.lsp import ( + LanguageDiscovery, + LspCapability, + LspRegistry, + LspRuntime, + LspServerConfig, + LspToolAdapter, +) + +registry = LspRegistry() +registry.register( + LspServerConfig( + name="local/pyright", + description="Pyright language server", + command="pyright-langserver", + languages=["python"], + capabilities=[ + LspCapability.DIAGNOSTICS, + LspCapability.COMPLETIONS, + LspCapability.HOVER, + ], + ) +) + +runtime = LspRuntime(registry=registry) +runtime.start_server("local/pyright", workspace_path="/workspace") + +adapter = LspToolAdapter(runtime) +pyright_tools = adapter.generate_tool_specs(registry.get_or_raise("local/pyright")) +print([spec["name"] for spec in pyright_tools]) +# ['local/pyright/diagnostics', 'local/pyright/completions', 'local/pyright/hover'] + +language = LanguageDiscovery().detect_file_language("/workspace/app/models.py") +print(language) # 'python' +``` + +--- + +## Models + +### `LspTransport` + +`StrEnum` describing how the runtime communicates with a server: + +- `STDIO` (default) — spawn the server and communicate via stdin/stdout JSON-RPC. +- `TCP` — connect to a server listening on a TCP socket. + +### `LspCapability` + +Enum of LSP features that can be surfaced as ToolSpecs. Members map directly to +protocol operations: + +| Capability | Description | +|------------|-------------| +| `DIAGNOSTICS` | Fetch diagnostics, warnings, and errors for a file. | +| `HOVER` | Retrieve hover information (type hints, docs) at a position. | +| `COMPLETIONS` | Request completion candidates at a position. | +| `DEFINITIONS` | Go to definition for a symbol. | +| `REFERENCES` | Find references to a symbol. | +| `RENAME` | Request a rename operation across the workspace. | +| `CODE_ACTIONS` | Obtain code actions for a range. | +| `FORMATTING` | Format a document or selection. | +| `SIGNATURE_HELP` | Fetch call signature help. | +| `DOCUMENT_SYMBOLS` | List symbols defined in a document. | +| `WORKSPACE_SYMBOLS` | Search for symbols across the workspace. | + +> **Note:** The runtime currently provides concrete handlers for diagnostics, +> completions, hover, and definitions. Other capabilities raise +> `LspNotAvailableError` until their protocol flows are implemented. + +### `LspServerConfig` + +Pydantic model describing a language server. + +| Field | Type | Description | +|-------|------|-------------| +| `name` | `str` | Namespaced identifier (`namespace/name`); validated to contain a slash. | +| `description` | `str` | Optional human-readable description. | +| `languages` | `list[str]` | Lowercased language identifiers served by the process. | +| `command` | `str` | Executable used to start the server. | +| `args` | `list[str]` | Extra CLI arguments. | +| `transport` | `LspTransport` | Communication mode (`STDIO` by default). | +| `env` | `dict[str, str]` | Environment overrides for the process. | +| `capabilities` | `list[LspCapability]` | Enabled protocol features. | +| `initialization` | `dict[str, Any]` | `initialize` request `initializationOptions`. | +| `workspace_settings` | `dict[str, Any]` | Settings pushed after initialization. | + +Helper properties: + +- `namespace` — portion before the last `/`. +- `short_name` — portion after the last `/`. +- `to_dict()` — JSON-compatible serialisation respecting model validators. + +### `LspBinding` + +Binds a server to an actor graph node. Used by the compiler/runtime to know +which servers to activate. Validates that `lsp_server_name` is namespaced and +supports optional per-node language filters. + +--- + +## Registry and discovery + +### `LspRegistry` + +Thread-safe registry for `LspServerConfig` objects. Public methods acquire a +re-entrant lock, making concurrent registration and lookup safe. + +- `register(config)` — add a new server; rejects duplicates. +- `get(name)` — return a config or `None` when missing. +- `get_or_raise(name)` — raise `LspServerNotFoundError` if absent. +- `list_servers(namespace=None, language=None)` — filter by namespace prefix or + supported language, sorted by name. +- `remove(name)` — unregister a server; returns `True` if removed. + +### `LanguageDiscovery` + +Four-layer language detection engine used by the runtime and compiler. Layers +are consulted in priority order: + +1. File extension (`.py`, `.tsx`, …) via a built-in lookup table. +2. Shebang inspection for ambiguous or extension-less files. +3. Optional UKO (Universal Knowledge Ontology) classifications supplied at + construction. +4. Explicit project-level language declarations passed to the constructor. + +Key methods: + +- `detect_file_language(path)` — returns a language ID (falls back to + `"plaintext"`). Results are cached per path. +- `detect_directory_languages(path)` — walk a tree and return sorted unique + languages (ignoring `plaintext`). +- `invalidate(path)` / `invalidate_all()` — manage the detection cache. +- `get_servers_for_language(language, configs)` — helper to match registered + servers to a detected language. + +--- + +## Runtime orchestration + +### `LspLifecycleManager` + +Manages running server processes with reference counting. Guarantees that the +underlying transport is started only once per server/workspace combination and +shutdown occurs when the last reference is released. Provides: + +- `start_server(config, workspace_path)` — start or reuse a server, returning an + `LspClient`. +- `stop_server(name)` — release a reference; raises `LspServerNotFoundError` if + the server was never acquired. +- `health_check(name)` — return `True` if the process is alive. +- `restart_server(name)` — restart a crashed server while preserving reference + counts. +- `stop_all()` / `list_running()` — maintenance helpers for graceful shutdown + and observability. + +### `LspClient` + +Synchronous JSON-RPC client layered on top of `StdioTransport`. Exposes typed +methods for the protocol flows currently needed by CleverAgents: + +- `initialize(workspace_path, initialization_options=None)` and `shutdown()` for + lifecycle management. +- Document operations: `did_open`, `did_close`, `get_diagnostics`, + `get_completions`, `get_hover`, `get_definitions`. +- Accessors `is_initialized` and `server_capabilities` for telemetry or testing. + +Internally the client keeps a bounded queue of notifications, correlates +responses by ID, and raises `LspError` on timeouts or protocol failures. + +### `LspRuntime` + +High-level facade combining the registry and lifecycle manager. It performs +input validation, logs significant events, and exposes convenience methods: + +- `start_server(name, workspace_path)` / `stop_server(name)` — manage process + lifecycle. +- `get_diagnostics`, `get_completions`, `get_hover`, `get_definitions` — delegate + to the underlying `LspClient`, reading file contents and translating between + filesystem paths and LSP URIs. +- `activate_bindings(bindings, workspace_path)` and `deactivate_bindings(bindings)` — + start/stop all servers referenced by `LspBinding` objects produced during actor + compilation. +- `stop_all()` — forward to `LspLifecycleManager.stop_all()`. + +If a server crashes, `LspRuntime` attempts to restart it automatically before +servicing the next request. + +### `StdioTransport` + +Process wrapper that launches language servers via subprocess, reads/writes +JSON-RPC messages, and exposes `is_alive`, `start()`, `stop()`, `send_message()`, +`read_message()`. It is the default transport used by the lifecycle manager and +client. + +--- + +## Tool adapter + +### `LspToolAdapter` + +Generates tool specifications from `LspServerConfig.capabilities` so the +standard tool runtime can invoke language-server functionality. + +```python +from cleveragents.lsp import LspToolAdapter + +adapter = LspToolAdapter(runtime) +for spec in adapter.generate_tool_specs(config): + tool_registry.register(spec) +``` + +- When constructed with an `LspRuntime`, capability handlers delegate to runtime + methods (diagnostics, completions, hover, definitions today). Missing runtime + capabilities raise `LspNotAvailableError` with a structured payload. +- Without a runtime (local/offline mode), generated handlers always raise + `LspNotAvailableError`, signalling that remote execution is required. +- Each spec includes a minimal JSON Schema input contract tailored to the + capability (file-only, position-based, rename, or query-based). + +--- + +## Error types + +The package exports structured exceptions for callers to handle: + +- `LspError` — generic protocol or transport failure. +- `LspNotAvailableError` — capability invoked without an available runtime. +- `LspServerNotFoundError` — registry or lifecycle lookup of an unknown server. + +Catch these errors in higher layers to provide user-facing remediation (e.g. +retry prompts, error banners in the TUI, or fallback tool behaviour). diff --git a/docs/api/shared.md b/docs/api/shared.md new file mode 100644 index 000000000..cdf81886d --- /dev/null +++ b/docs/api/shared.md @@ -0,0 +1,117 @@ +# `cleveragents.shared` — Secret Redaction Utilities + +The `cleveragents.shared` package exposes cross-cutting helpers that can be +used from any layer without violating architecture boundaries. At the moment +its public surface focuses on a single responsibility: deterministic masking of +secrets before they reach logs, error payloads, or tool output. + +All exports live in `cleveragents.shared.redaction` and are re-exported from the +package root for convenience. + +--- + +## Quick start + +```python +from cleveragents.shared import redact_dict, register_pattern + +# redact nested structures before logging +payload = { + "provider_api_key": "sk-ant-api03-secret-1234567890", + "token_usage": 1280, + "upstream": { + "authorization": "Bearer 0123456789ABCDEFGHIJKLMNOPQRSTUV", + }, +} +print(redact_dict(payload)) +# {'provider_api_key': '***REDACTED***', 'token_usage': 1280, +# 'upstream': {'authorization': '***REDACTED***'}} + +# extend detection with an internal pattern +register_pattern(r"my-secret-[0-9a-f]{16}") +``` + +--- + +## Global redaction controls + +| Symbol | Description | +|--------|-------------| +| `REDACTED` | Constant string (`***REDACTED***`) used as the replacement token. | +| `get_show_secrets() -> bool` | Returns the global flag that determines whether redaction is bypassed. | +| `set_show_secrets(value: bool) -> None` | Toggles the flag. Accepts only boolean values and is thread-safe. | + +When `get_show_secrets()` returns `True`, the redaction helpers shortcut and +return the original data unmodified—useful for debugging in trusted +environments. The flag is protected by a lock so it can be flipped safely from +CLI options or unit tests. + +--- + +## Key helpers + +### `is_sensitive_key(name: str) -> bool` + +Heuristically classifies dictionary keys as sensitive (e.g. containing +"token", "secret", "password"). False positives such as `token_count` +are explicitly whitelisted in the implementation. + +### `redact_value(value: str) -> str` + +Scans a string for known secret patterns (OpenAI, Anthropic, Google keys, +`tok_` IDs, bearer tokens, long hex strings) and replaces each match with +`REDACTED`. Additional patterns can be registered at runtime via +`register_pattern()`. + +### `redact_dict(data: Mapping[str, Any], *, show_secrets: bool | None = None) -> dict` + +Recursively traverses nested dictionaries and lists, redacting sensitive keys +with `REDACTED` and scrubbing string values via `redact_value`. By default it +honours the global `get_show_secrets()` flag; override it per call with +`show_secrets=True` when you explicitly need the original values. + +### `mask_database_url(url: str) -> str` + +Masks the password segment of SQL connection URLs. SQLite URLs are returned +unchanged; any `scheme://user:password@host` pattern is rewritten as +`scheme://user:***@host`. + +### `register_pattern(pattern: str) -> None` + +Allows projects or extensions to add custom regular expressions to the +redaction list. Patterns are compiled under a lock so multiple workers can +register safely. Passing an empty pattern raises `ValueError`. + +--- + +## Structlog integration + +`secrets_masking_processor(logger, method_name, event_dict)` implements the +structlog processor contract. It walks every value in the event dictionary, +redacting keys detected by `is_sensitive_key` and values using `redact_value` +and `redact_dict`. The processor is designed to be inserted near the top of the +structlog pipeline so that downstream formatters never observe raw secrets. + +```python +import structlog +from cleveragents.shared import secrets_masking_processor + +structlog.configure( + processors=[ + secrets_masking_processor, + structlog.processors.JSONRenderer(), + ], +) +``` + +--- + +## Thread-safety notes + +The module guards both the global `show_secrets` flag and the list of compiled +patterns with locks. You can safely call `set_show_secrets()` or +`register_pattern()` from concurrent tasks without corrupting shared state. + +Because all helpers return new dictionaries (never mutating inputs), you may +confidently hand the redacted payloads to logging subsystems or telemetry +pipelines without affecting the original objects used by business logic.