diff --git a/CHANGELOG.md b/CHANGELOG.md index 36ca0cc2c..3b3e5eb36 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -291,3 +291,101 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). renders permission requests directly in the conversation stream for single-file operations. Users can allow/reject with single-key shortcuts (`a`/`A`/`r`/`R`), navigate with arrow keys, confirm with `Enter`, or press `v` to open the full + +## [3.1.0] — 2025-Q4 + +### Added + +- **Actor Compiler** — GRAPH-type actor YAML files now compile into live + LangGraph `StateGraph` structures via `ActorCompiler`. Compilation validates + node references, detects intra-graph and cross-actor subgraph cycles, maps + node types to LangGraph `NodeType` values, and returns a `CompiledActor` + bundle with nodes, edges, entry point, and `CompilationMetadata`. + +- **`agents actor add --config actor.yaml`** — New CLI command loads and + registers custom actors from YAML files. The loader parses, validates via + Pydantic, enforces graph topology rules, and persists to the actor registry. + Supports `--update`, `--unsafe`, and `--set-default` flags. + +- **Actor YAML `version: "3"` schema** — Actor YAML files now declare + `version: "3"` and support `type: llm | tool | graph`. All fields are + validated via Pydantic with precise error messages including dotted field + paths. + +- **Tool Call Router** — `ToolCallRouter` normalizes OpenAI, Anthropic, and + LangChain tool call formats into `NormalizedToolCallResult` objects. + Supports single routing, batch routing, and streaming execution. Generates + stable deterministic tool call IDs from `plan_id + sequence`. + +- **MCP Adapter** — `MCPToolAdapter` connects to external MCP (Model Context + Protocol) tool servers, discovers tools, validates inputs against JSON + Schema, and bulk-registers tools into the `ToolRegistry`. Supports `stdio`, + `sse`, and `streamable-http` transports. Thread-safe via `threading.RLock`. + +- **Validation Runner** — Executes resource-attached validations before the + apply gate. Classifies validations as required (blocking) or informational + (non-blocking). `ApplyValidationSummary.all_required_passed` returns `False` + when zero validations were run (empty-run guard). + +- **Multi-file ChangeSet generation** — The execution actor now correctly + produces `ChangeSet` records with `added_files`, `modified_files`, and + `deleted_files` for multi-file generation scenarios. + +- **Skill registry and tool lifecycle CLI** — `agents skill add/list/show/remove` + commands manage named skill collections. Skills provide tool sets to actors + during plan execution. + +--- + +## [3.0.0] — 2025-Q3 + +### Added + +- **`agents action create --config action.yaml`** — Register reusable action + templates from YAML files. Actions are persisted to SQLite and include + `name`, `description`, `strategy_actor`, `execution_actor`, + `definition_of_done`, optional `arguments`, `invariants`, `inputs_schema`, + and `automation_profile`. + +- **`agents resource add git-checkout`** — Register git repositories as + resources in the resource registry. The `git-checkout` resource type is the + primary resource type for the git worktree sandbox. + +- **`agents project create` and `agents project link-resource`** — Create + projects and link resources to them. Projects group resources together and + provide context for plan execution. + +- **`agents plan use`** — Create plan records from action templates and + projects. Plans are persisted to SQLite with ULID identifiers and start in + the `strategize/queued` state. + +- **`agents plan execute `** — Run the Strategize and Execute phases + of the plan lifecycle. Invokes the actor-based LLM path and writes generated + file changes to the git worktree sandbox. + +- **`agents plan diff `** — Show pending sandbox changes as a unified + diff for review before applying. + +- **`agents plan apply `** — Merge sandbox changes into the real git + repository via `git merge` from the isolated worktree branch. Non-git + projects fall back to flat file copy. + +- **Git worktree sandbox** — Isolated working directories for plan execution. + Each plan gets its own branch (`cleveragents/plan-`) and a + temporary worktree. On apply, the sandbox branch is merged back into the + original branch and the worktree is cleaned up. + +- **Pydantic v2 domain models with `frozen=True`** — All domain models + (`ActionRecord`, `ResourceRecord`, `ProjectRecord`, `PlanRecord`, + `ChangeSet`, etc.) use Pydantic v2 with `frozen=True` for immutability + guarantees. + +- **SQLite persistence with Alembic migrations** — All records are persisted + to a local SQLite database. Schema migrations are managed by Alembic and + run automatically on first use. + +- **Plan state machine** — Plans follow a strict state machine: + `strategize/queued` → `strategize/in_progress` → `strategize/complete` → + `execute/queued` → `execute/in_progress` → `execute/complete` → + `apply/queued` → `apply/in_progress` → `applied`. Terminal states: + `applied`, `cancelled`, `errored`. diff --git a/README.md b/README.md index 733986eda..13bb2528f 100644 --- a/README.md +++ b/README.md @@ -170,3 +170,41 @@ Set `CLEVERAGENTS_DEFAULT_PROVIDER` to pin the global provider (for example `exp - Built-in actors (`/`) are immutable, custom actors must be named `local/`, and the default actor cannot be removed. Use `--unsafe` when adding/updating configs marked unsafe; runtime only warns when invoking unsafe actors. - `CLEVERAGENTS_TESTING_USE_MOCK_AI=true` forces the in-repo mock provider so Behave/Robot suites never hit external APIs. - The full capability matrix (streaming, tool calls, JSON mode, etc.) is documented in `docs/reference/providers.md`. + +## Milestone History + +### v3.0.0 — M1: Minimal Local Source-Code Workflow + +The first minimally usable local-mode flow. Register an action from YAML, +link a git repository resource to a project, and run a plan end-to-end. + +**Key capabilities:** +- `agents action create --config action.yaml` — register action templates +- `agents resource add git-checkout` — register git repository resources +- `agents project create` / `agents project link-resource` — project management +- `agents plan use` — create plan records with state machine transitions +- `agents plan execute ` — invoke actor-based LLM path +- `agents plan diff ` — review pending sandbox changes +- `agents plan apply ` — merge sandbox changes with git commit +- Git worktree sandbox for isolated working directories +- Pydantic v2 domain models with `frozen=True` +- SQLite persistence with Alembic migrations + +See [`docs/modules/milestone-v3.0.0-local-workflow.md`](docs/modules/milestone-v3.0.0-local-workflow.md) for the full guide. + +### v3.1.0 — M2: Actor Compiler + Full LLM Integration + +Actor YAML files compile into live LangGraph graphs. Custom actors are fully +operational with a normalized tool router, validation runner, and MCP adapter. + +**Key capabilities:** +- Actor YAML files with `version: "3"`, `type: llm|tool|graph` — parse and validate via Pydantic +- GRAPH-type actors compile into LangGraph `StateGraph` structures +- `agents actor add --config actor.yaml` — load and register custom actors +- Skill registry and tool lifecycle via CLI (`agents skill add/list/show/remove`) +- MCP adapter discovers and connects to external tool servers +- Tool call router normalizes OpenAI / Anthropic / LangChain formats +- Validation runner executes required and informational validations +- Multi-file generation produces correct `ChangeSet` + +See [`docs/modules/milestone-v3.1.0-actor-compiler.md`](docs/modules/milestone-v3.1.0-actor-compiler.md) for the full guide. diff --git a/docs/modules/milestone-v3.0.0-local-workflow.md b/docs/modules/milestone-v3.0.0-local-workflow.md new file mode 100644 index 000000000..da796811c --- /dev/null +++ b/docs/modules/milestone-v3.0.0-local-workflow.md @@ -0,0 +1,265 @@ +# v3.0.0 — M1: Minimal Local Source-Code Workflow + +**Milestone:** v3.0.0 — M1 (CLOSED) +**Released:** 2025-Q3 + +This milestone delivered the first minimally usable local-mode flow for +CleverAgents. A user can register an action from YAML, link a git repository +resource to a project, and run a plan end-to-end — from creation through +execution, diff review, and final apply — using a sandboxed workspace with +tool-based change capture. + +--- + +## Overview + +The M1 milestone established the foundational plan lifecycle: + +``` +agents action create → agents resource add → agents project create + │ │ + └──────────────────────────────────────────────►│ + │ + agents plan use + │ + agents plan execute + │ + agents plan diff + │ + agents plan apply +``` + +All state is persisted to SQLite via Alembic-managed migrations. Domain +models use Pydantic v2 with `frozen=True` for immutability guarantees. + +--- + +## CLI Commands Introduced + +### `agents action create` + +Register a reusable action template from a YAML configuration file. + +```bash +agents action create --config action.yaml +``` + +**Minimal `action.yaml`:** + +```yaml +name: local/my-action +description: Describe what this action does +strategy_actor: openai/gpt-4 +execution_actor: openai/gpt-4 +definition_of_done: | + The task is complete when all acceptance criteria are met. +``` + +Actions are persisted to SQLite and can be listed with `agents action list`. +See [Action CLI Reference](../reference/action_cli.md) for the full field +listing and options. + +--- + +### `agents resource add git-checkout` + +Register a git repository as a resource in the resource registry. + +```bash +agents resource add git-checkout local/my-repo --path /home/user/my-repo +``` + +The `git-checkout` resource type represents a local git repository. It is +the primary resource type used by the git worktree sandbox during plan +execution. + +See [Resource CLI Reference](../reference/resource_cli.md) for all resource +commands. + +--- + +### `agents project create` and `agents project link-resource` + +Create a project and link one or more resources to it. + +```bash +agents project create local/my-project --description "My project" +agents project link-resource local/my-project local/my-repo +``` + +Projects group resources together and provide the context for plan execution. +Multiple resources can be linked to a single project. + +--- + +### `agents plan use` + +Create a plan record from an action template and one or more projects. + +```bash +agents plan use local/my-action local/my-project +``` + +This creates a plan in the `strategize/queued` state and persists it to +SQLite. The plan is assigned a ULID identifier. + +See [Plan CLI Reference](../reference/plan_cli.md) for full options including +`--arg`, `--automation-profile`, and `--invariant` flags. + +--- + +### `agents plan execute ` + +Run the Strategize and Execute phases of the plan lifecycle. This invokes +the actor-based LLM path and writes generated file changes to the sandbox. + +```bash +agents plan execute 01HXYZ1234567890ABCDEFGH +``` + +The executor: +1. Runs the **Strategize** phase — the strategy actor produces a decision + tree of tasks. +2. Runs the **Execute** phase — the execution actor generates file changes + and writes them to the git worktree sandbox. + +When the automation profile permits auto-apply (e.g. `ci` or `full-auto`), +`plan execute` drives the plan all the way to `applied` in a single call. + +--- + +### `agents plan diff ` + +Show pending changes in the sandbox as a unified diff. + +```bash +agents plan diff 01HXYZ1234567890ABCDEFGH +``` + +Displays the ChangeSet as a unified diff so you can review what the LLM +generated before committing it to the real repository. + +--- + +### `agents plan apply ` + +Merge sandbox changes into the real git repository with a commit. + +```bash +agents plan apply 01HXYZ1234567890ABCDEFGH +``` + +For git-checkout resources, this uses the **git worktree sandbox** strategy: +changes are merged via `git merge` from the isolated worktree branch into +the original branch. Non-git projects fall back to flat file copy. + +Pass `--yes` / `-y` to skip the confirmation prompt in CI pipelines. + +--- + +## Git Worktree Sandbox + +The git worktree sandbox provides isolated working directories for plan +execution. Each plan gets its own branch (`cleveragents/plan-`) +and a temporary worktree directory. + +**Lifecycle:** + +``` +PENDING → CREATED → ACTIVE → COMMITTED → CLEANED_UP + │ + └→ ROLLED_BACK → CLEANED_UP +``` + +On `plan apply`, the sandbox branch is merged back into the original branch +and the worktree is cleaned up. On failure or cancellation, the worktree is +discarded without touching the original branch. + +See [Git Worktree Sandbox](git-worktree-sandbox.md) for the full API +reference. + +--- + +## Domain Models + +All domain models introduced in M1 use **Pydantic v2** with `frozen=True`: + +| Model | Description | +|-------|-------------| +| `ActionRecord` | Persisted action template | +| `ActionArgument` | Typed argument definition for an action | +| `ResourceRecord` | Registered resource instance | +| `ProjectRecord` | Project grouping resources | +| `ProjectResourceLink` | Many-to-many link between projects and resources | +| `PlanRecord` | Plan instance with state machine | +| `ChangeSet` | Set of file changes produced by execution | + +Frozen models ensure that domain objects are never mutated after creation, +preventing accidental state corruption in multi-phase workflows. + +--- + +## SQLite Persistence and Alembic Migrations + +All records are persisted to a local SQLite database (default: +`~/.config/cleveragents/cleveragents.db`). Schema migrations are managed +by **Alembic** and run automatically on first use. + +The database path can be overridden via the `CLEVERAGENTS_DB_PATH` +environment variable or the `database.path` config key. + +--- + +## State Machine + +Plans follow a strict state machine with the following phases and states: + +| Phase | States | +|-------|--------| +| `strategize` | `queued` → `in_progress` → `complete` / `errored` | +| `execute` | `queued` → `in_progress` → `complete` / `errored` | +| `apply` | `queued` → `in_progress` → `applied` / `errored` | + +Terminal states: `applied`, `cancelled`, `errored`. + +--- + +## End-to-End Example + +```bash +# 1. Register an action +agents action create --config action.yaml + +# 2. Register a git repository resource +agents resource add git-checkout local/my-repo --path /home/user/my-repo + +# 3. Create a project and link the resource +agents project create local/my-project +agents project link-resource local/my-project local/my-repo + +# 4. Create a plan +agents plan use local/my-action local/my-project + +# 5. Execute the plan (Strategize + Execute phases) +agents plan execute 01HXYZ1234567890ABCDEFGH + +# 6. Review the generated changes +agents plan diff 01HXYZ1234567890ABCDEFGH + +# 7. Apply the changes to the real repository +agents plan apply 01HXYZ1234567890ABCDEFGH +``` + +--- + +## Related + +- [Plan CLI Reference](../reference/plan_cli.md) +- [Action CLI Reference](../reference/action_cli.md) +- [Resource CLI Reference](../reference/resource_cli.md) +- [Git Worktree Sandbox](git-worktree-sandbox.md) +- [ADR-006 Plan Lifecycle](../adr/ADR-006-plan-lifecycle.md) +- [ADR-008 Resource System](../adr/ADR-008-resource-system.md) +- [ADR-009 Project Model](../adr/ADR-009-project-model.md) +- [ADR-015 Sandbox & Checkpoint](../adr/ADR-015-sandbox-and-checkpoint.md) +- [ADR-019 Storage & Persistence](../adr/ADR-019-storage-and-persistence.md) diff --git a/docs/modules/milestone-v3.1.0-actor-compiler.md b/docs/modules/milestone-v3.1.0-actor-compiler.md new file mode 100644 index 000000000..652b8ae04 --- /dev/null +++ b/docs/modules/milestone-v3.1.0-actor-compiler.md @@ -0,0 +1,305 @@ +# v3.1.0 — M2: Actor Compiler + Full LLM Integration + +**Milestone:** v3.1.0 — M2 (CLOSED) +**Released:** 2025-Q4 + +This milestone made Actor YAML files fully operational. Custom actors compile +into live LangGraph graphs, the tool router normalizes calls across providers, +the validation runner enforces resource-attached validations, and the MCP +adapter connects to external tool servers. + +--- + +## Overview + +M2 built on the M1 plan lifecycle by introducing a full actor compilation +pipeline and LLM integration layer: + +``` +actor.yaml (version: "3") + │ + ▼ +ActorLoader (parse + validate via Pydantic) + │ + ▼ +ActorCompiler (GRAPH type → LangGraph StateGraph) + │ + ▼ +LangGraph Runtime + │ + ├── ToolCallRouter (OpenAI / Anthropic / LangChain normalization) + │ + ├── ValidationRunner (required + informational validations) + │ + └── MCPToolAdapter (external tool servers) +``` + +--- + +## Actor YAML Schema (version: "3") + +Actor YAML files use `version: "3"` and support three actor types: + +| Type | Description | +|------|-------------| +| `llm` | Single LLM invocation node | +| `tool` | Tool execution node | +| `graph` | Multi-node LangGraph workflow | + +**Minimal LLM actor:** + +```yaml +version: "3" +name: local/my-llm-actor +type: llm +description: A simple LLM actor +model: gpt-4 +``` + +**Minimal GRAPH actor:** + +```yaml +version: "3" +name: local/my-pipeline +type: graph +description: A multi-node pipeline +model: gpt-4 + +route: + nodes: + - id: planner + type: agent + name: Planner + description: Plans the work + - id: executor + type: agent + name: Executor + description: Executes the plan + edges: + - from_node: planner + to_node: executor + entry_node: planner + exit_nodes: + - executor +``` + +See [Actor Configuration Reference](../reference/actor_config.md) and +[Actor YAML Schema](../reference/actors_schema.md) for the full field listing. + +--- + +## `agents actor add --config actor.yaml` + +Load and register a custom actor from a YAML file. + +```bash +agents actor add local/my-actor --config ./actors/my-actor.yaml +``` + +The loader: +1. Parses the YAML file. +2. Validates the schema via Pydantic (all fields, types, and constraints). +3. For GRAPH actors, validates the route topology (no cycles, reachable nodes, + valid entry/exit points). +4. Persists the actor to the registry. + +Use `--update` to replace an existing actor, `--unsafe` to register actors +marked as unsafe, and `--set-default` to make the actor the default. + +See [Actor CLI Reference](../reference/actor_cli.md) for all options. + +--- +## Actor Compiler: GRAPH → LangGraph + +GRAPH-type actors compile into LangGraph `StateGraph` structures via the +`ActorCompiler`. The compilation pipeline: + +1. **Input validation** — Accepts `ActorConfigSchema` with `type=GRAPH`. +2. **Reference validation** — All node IDs in edges, entry, and exit points + are checked against the declared node set. +3. **Cycle detection** — Intra-graph and cross-actor subgraph cycles are + detected and rejected. +4. **Node mapping** — Each `NodeDefinition` maps to a LangGraph `NodeConfig`. +5. **Edge mapping** — Each `EdgeDefinition` maps to a LangGraph `Edge`. +6. **LSP binding extraction** — Per-node LSP bindings are extracted into + `LspBinding` objects. +7. **Metadata assembly** — Returns a `CompiledActor` with nodes, edges, and + `CompilationMetadata`. + +**Node type mapping:** + +| Actor Node Type | LangGraph NodeType | +|---|---| +| `agent` | `AGENT` | +| `tool` | `TOOL` | +| `conditional` | `CONDITIONAL` | +| `subgraph` | `SUBGRAPH` | + +See [Actor Compiler Reference](../reference/actor_compiler.md) for the full +API reference. + +--- + +## Skill Registry and Tool Lifecycle + +Skills provide named collections of tools that actors can use. The skill +registry manages the full lifecycle: + +```bash +# Add a skill +agents skill add local/my-skill --config skill.yaml + +# List registered skills +agents skill list + +# Show skill details +agents skill show local/my-skill + +# Remove a skill +agents skill remove local/my-skill +``` + +Tools within skills are registered in the `ToolRegistry` and made available +to actors during plan execution. See [Skill CLI Reference](../reference/skill_cli.md) +for all commands. + +--- +## MCP Adapter + +The `MCPToolAdapter` connects to external MCP (Model Context Protocol) tool +servers and registers their tools in the `ToolRegistry`. + +**Server configuration:** + +```yaml +name: local/my-mcp-server +transport: stdio +command: npx +args: ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/dir"] +``` + +**Lifecycle:** + +```python +adapter = MCPToolAdapter(config) +adapter.connect() +tools = adapter.discover_tools() +adapter.register_tools(registry, namespace="mcp") +# ... use tools in plan execution ... +adapter.disconnect() +``` + +Supported transports: `stdio` (child process), `sse`, `streamable-http`. + +See [MCP Tool Adapter Reference](../reference/mcp_adapter.md) for the full +API reference. + +--- + +## Tool Call Router + +The `ToolCallRouter` normalizes tool calls across LLM providers. It accepts +OpenAI, Anthropic, and LangChain tool call formats and produces normalized +`NormalizedToolCallResult` objects. + +**Provider format mapping:** + +| Provider | Arguments key | Arguments type | +|----------|---------------|----------------| +| OpenAI | `arguments` | JSON string | +| Anthropic | `input` | dict | +| LangChain | `args` | dict | + +The router automatically detects the provider format and normalizes the call +before dispatching to the `ToolRunner`. + +See [Tool Call Router Reference](../reference/tool_router.md) for the full +API reference. + +--- + +## Validation Runner + +The validation runner executes resource-attached validations before the +apply gate. Validations are classified as: + +- **Required** — must pass for `plan apply` to proceed. +- **Informational** — run and reported but do not block apply. + +The `ApplyValidationSummary.all_required_passed` property returns `False` +when zero validations were run (empty-run guard), ensuring apply is always +blocked unless at least one validation was executed. + +--- + +## Multi-File Generation and ChangeSet + +The execution actor generates a `ChangeSet` containing multiple file +operations. M2 ensures that multi-file generation produces a correct +`ChangeSet` with: + +- `added_files` — new files created by the actor. +- `modified_files` — existing files changed by the actor. +- `deleted_files` — files removed by the actor. + +The ChangeSet is written to the git worktree sandbox and can be reviewed +with `agents plan diff` before applying. + +--- +## End-to-End Example with Custom Actor + +```bash +# 1. Write an actor YAML file +cat > my-actor.yaml << 'EOF' +version: "3" +name: local/code-reviewer +type: graph +description: Multi-step code review pipeline +model: gpt-4 + +route: + nodes: + - id: analyzer + type: agent + name: Code Analyzer + description: Analyzes code quality + - id: reporter + type: agent + name: Report Writer + description: Writes the review report + edges: + - from_node: analyzer + to_node: reporter + entry_node: analyzer + exit_nodes: + - reporter +EOF + +# 2. Register the actor +agents actor add local/code-reviewer --config my-actor.yaml + +# 3. Create an action using the custom actor +agents action create --config action.yaml # references local/code-reviewer + +# 4. Run a plan using the custom actor +agents plan use local/review-action local/my-project +agents plan execute 01HXYZ1234567890ABCDEFGH +agents plan diff 01HXYZ1234567890ABCDEFGH +agents plan apply 01HXYZ1234567890ABCDEFGH +``` + +--- +## Related + +- [Actor CLI Reference](../reference/actor_cli.md) +- [Actor Configuration Reference](../reference/actor_config.md) +- [Actor YAML Schema](../reference/actors_schema.md) +- [Actor Compiler Reference](../reference/actor_compiler.md) +- [MCP Tool Adapter Reference](../reference/mcp_adapter.md) +- [Tool Call Router Reference](../reference/tool_router.md) +- [Skill CLI Reference](../reference/skill_cli.md) +- [ADR-010 Actor & Agent Architecture](../adr/ADR-010-actor-and-agent-architecture.md) +- [ADR-011 Tool System](../adr/ADR-011-tool-system.md) +- [ADR-022 LangChain/LangGraph Integration](../adr/ADR-022-langchain-langgraph-integration.md) +- [ADR-029 Model Context Protocol (MCP)](../adr/ADR-029-model-context-protocol.md) +- [ADR-031 Actor Abstraction Definition](../adr/ADR-031-actor-abstraction-definition.md) diff --git a/mkdocs.yml b/mkdocs.yml index 48a98aa71..d714f0f00 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,6 +30,8 @@ nav: - Invariant Reconciliation: modules/invariant-reconciliation.md - ACMS Context Hydration: modules/context-hydration.md - Git Worktree Sandbox: modules/git-worktree-sandbox.md + - M1: Minimal Local Workflow: modules/milestone-v3.0.0-local-workflow.md + - M2: Actor Compiler + LLM Integration: modules/milestone-v3.1.0-actor-compiler.md - Development: - Agent System Specification: development/agent-system-specification.md - CI/CD Pipeline: development/ci-cd.md