docs: Added significant details and finalized (hopefully) task assignments
This commit is contained in:
+2528
-359
File diff suppressed because it is too large
Load Diff
+382
-45
@@ -993,25 +993,90 @@ Execution should be treated like a transactional pipeline:
|
||||
|
||||
This is explicitly motivated by "partial failure leaves codebase inconsistent" and the need for transaction rollback.
|
||||
|
||||
### Parsing and File Generation Expectations
|
||||
### Tool-Based Resource Modification (Modern Architecture)
|
||||
|
||||
The notes call out planned improvements:
|
||||
**IMPORTANT: CleverAgents does NOT parse LLM output to extract code.** Instead, it uses the modern tool-based approach pioneered by Claude Code, Cursor, and Aider where:
|
||||
|
||||
* Extract code blocks properly (avoid markdown explanation text in output files).
|
||||
* Avoid generic file names like `generated.py`.
|
||||
* Improve file structure generation.
|
||||
* Handle invalid python responses better than wrapping in docstrings.
|
||||
1. **LLMs call tools/skills directly** (`edit_file()`, `write_file()`, `delete_file()`, etc.)
|
||||
2. **Tools operate on the sandbox** - each tool invocation modifies sandbox state directly
|
||||
3. **ChangeSet is built from tool invocations** - not by parsing LLM text output
|
||||
4. **Validation runs on sandbox state** - after tools execute, not on parsed output
|
||||
|
||||
Therefore, CleverAgents should define a **standard output parsing pipeline**:
|
||||
This architecture provides:
|
||||
|
||||
1. Parse model output.
|
||||
2. Identify code blocks and associated filenames (if present).
|
||||
3. If filename missing, infer from context and existing project structure.
|
||||
4. Validate syntax (when language known).
|
||||
5. If invalid, either:
|
||||
* **Atomic operations**: Each tool call is a discrete, trackable change
|
||||
* **No parsing ambiguity**: Tools have structured parameters (path, content, etc.)
|
||||
* **Resource-agnostic**: Same pattern works for files, databases, APIs, any resource type
|
||||
* **Safety by design**: Tools run in sandbox with defined capabilities and restrictions
|
||||
* **MCP compatibility**: Skills map directly to MCP tools for external integrations
|
||||
|
||||
* request correction from actor, or
|
||||
* quarantine the output as a "draft artifact" rather than writing it as real source.
|
||||
#### How It Works
|
||||
|
||||
```
|
||||
LLM Response (with tool calls)
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Skill/Tool Router │
|
||||
│ - Routes each tool call to handler │
|
||||
│ - Validates parameters │
|
||||
│ - Enforces capability restrictions │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Sandbox Execution │
|
||||
│ - Tool operates on sandboxed state │
|
||||
│ - Each invocation recorded │
|
||||
│ - Checkpoint created if needed │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ ChangeSet Accumulation │
|
||||
│ - Each resource-modifying call → │
|
||||
│ becomes a Change record │
|
||||
│ - ChangeSet = history of changes │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Validation & Review │
|
||||
│ - Run validators on sandbox state │
|
||||
│ - Generate diff from ChangeSet │
|
||||
│ - Present for review before Apply │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
#### Built-in Resource Skills
|
||||
|
||||
CleverAgents provides these core skills for resource manipulation:
|
||||
|
||||
| Skill | Description | Creates Change? |
|
||||
|-------|-------------|-----------------|
|
||||
| `read_file(path)` | Read file contents | No |
|
||||
| `write_file(path, content)` | Create/overwrite file | Yes |
|
||||
| `edit_file(path, changes)` | Apply targeted edits | Yes |
|
||||
| `delete_file(path)` | Remove file | Yes |
|
||||
| `move_file(src, dst)` | Rename/move file | Yes |
|
||||
| `create_directory(path)` | Create directory | Yes |
|
||||
| `list_files(pattern)` | List files matching glob | No |
|
||||
| `search_files(pattern, content)` | Search file contents | No |
|
||||
| `get_file_info(path)` | Get file metadata | No |
|
||||
|
||||
Each skill automatically:
|
||||
* Operates within sandbox boundaries
|
||||
* Records changes to the ChangeSet
|
||||
* Validates parameters against project configuration
|
||||
* Enforces deny-list patterns (`.git/`, `node_modules/`, etc.)
|
||||
|
||||
#### Why Not Parse LLM Output?
|
||||
|
||||
The obsolete approach of parsing markdown code fences has fundamental problems:
|
||||
|
||||
1. **Ambiguity**: Is text explanation or code? Where does one file end and another begin?
|
||||
2. **Fragility**: Models output varying formats; regex parsing is brittle
|
||||
3. **Loss of semantics**: You lose the intent (create vs modify vs delete)
|
||||
4. **No atomicity**: Can't rollback individual operations
|
||||
5. **Resource-limited**: Only works for files, not databases or other resources
|
||||
|
||||
The tool-based approach solves all of these by making each operation explicit, typed, and trackable.
|
||||
|
||||
## Semantic Error Prevention
|
||||
|
||||
@@ -1688,6 +1753,182 @@ This registry supports:
|
||||
* plan validation ("this plan requires checkpointable write skills; do we have them?")
|
||||
* safe automation ("don't ask permission for every tiny command—use sandbox/checkpoints instead")
|
||||
|
||||
### MCP Integration Architecture
|
||||
|
||||
CleverAgents fully integrates with the **Model Context Protocol (MCP)** while extending it for agentic workflows:
|
||||
|
||||
#### MCP Concepts Mapping
|
||||
|
||||
| MCP Concept | CleverAgents Equivalent | Extension |
|
||||
|------------|------------------------|-----------|
|
||||
| Tool | Skill | Extended metadata (write_scope, checkpointable) |
|
||||
| Resource | Resource | Read AND write operations |
|
||||
| Prompt | Action template | Full plan lifecycle |
|
||||
| Server | MCP Server (external) | Integrated via skill adapters |
|
||||
|
||||
#### Using External MCP Servers
|
||||
|
||||
CleverAgents can connect to any MCP server and expose its tools as skills:
|
||||
|
||||
```yaml
|
||||
actors:
|
||||
github_ops:
|
||||
type: tool
|
||||
config:
|
||||
mcp_servers:
|
||||
- name: github
|
||||
command: "npx @anthropic/mcp-github"
|
||||
env:
|
||||
GITHUB_TOKEN: "${GITHUB_TOKEN}"
|
||||
- name: filesystem
|
||||
command: "npx @anthropic/mcp-filesystem"
|
||||
args: ["--root", "/workspace"]
|
||||
```
|
||||
|
||||
When an actor specifies `mcp_servers`, all tools from those servers become available as skills within that actor's execution context.
|
||||
|
||||
#### MCP Tool → Skill Adapter
|
||||
|
||||
External MCP tools are automatically wrapped with CleverAgents skill semantics:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ MCP Server │
|
||||
│ - Exposes tools via JSON-RPC │
|
||||
│ - Has MCP metadata (read-only, etc) │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ MCPSkillAdapter │
|
||||
│ - Wraps MCP tool as Skill │
|
||||
│ - Infers extended metadata │
|
||||
│ - Intercepts calls for: │
|
||||
│ - Sandbox path rewriting │
|
||||
│ - Change tracking │
|
||||
│ - Permission enforcement │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Skill Execution │
|
||||
│ - Runs in plan's sandbox context │
|
||||
│ - Changes recorded to ChangeSet │
|
||||
│ - Checkpoints created as needed │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
#### Skill Execution Flow (Tool-Based Architecture)
|
||||
|
||||
When an LLM decides to use a skill, the following flow occurs:
|
||||
|
||||
```
|
||||
1. LLM generates tool call: edit_file(path="src/main.py", changes=[...])
|
||||
↓
|
||||
2. Tool Router receives call
|
||||
- Validates parameters against schema
|
||||
- Checks skill capability metadata
|
||||
- Enforces permission restrictions
|
||||
↓
|
||||
3. Sandbox Context Resolution
|
||||
- Maps logical path to sandbox path
|
||||
- Ensures sandbox exists for resource
|
||||
- Creates checkpoint if skill is checkpointable
|
||||
↓
|
||||
4. Skill Execution
|
||||
- Runs skill code (built-in or MCP)
|
||||
- Operations occur on sandboxed state
|
||||
- Result captured
|
||||
↓
|
||||
5. Change Recording
|
||||
- If skill modifies resources, create Change record
|
||||
- Append Change to plan's ChangeSet
|
||||
- Update sandbox state
|
||||
↓
|
||||
6. Return to LLM
|
||||
- Return skill result
|
||||
- LLM continues with next action
|
||||
```
|
||||
|
||||
#### Built-in Skills (Core Resource Operations)
|
||||
|
||||
CleverAgents provides these built-in skills that work with any resource through the unified abstraction layer:
|
||||
|
||||
**File Operations:**
|
||||
```python
|
||||
read_file(path: str) -> str
|
||||
write_file(path: str, content: str) -> None
|
||||
edit_file(path: str, edits: list[Edit]) -> None
|
||||
delete_file(path: str) -> None
|
||||
move_file(source: str, destination: str) -> None
|
||||
copy_file(source: str, destination: str) -> None
|
||||
```
|
||||
|
||||
**Directory Operations:**
|
||||
```python
|
||||
create_directory(path: str) -> None
|
||||
list_directory(path: str, pattern: str = "*") -> list[str]
|
||||
delete_directory(path: str, recursive: bool = False) -> None
|
||||
```
|
||||
|
||||
**Search Operations:**
|
||||
```python
|
||||
search_files(pattern: str, content_pattern: str = None) -> list[Match]
|
||||
find_definition(symbol: str) -> list[Location]
|
||||
find_references(symbol: str) -> list[Location]
|
||||
```
|
||||
|
||||
**Git Operations (when resource is git repository):**
|
||||
```python
|
||||
git_status() -> GitStatus
|
||||
git_diff(path: str = None) -> str
|
||||
git_log(count: int = 10) -> list[Commit]
|
||||
git_blame(path: str) -> list[BlameLine]
|
||||
```
|
||||
|
||||
Each built-in skill:
|
||||
* Has fully defined capability metadata
|
||||
* Operates through the resource abstraction layer
|
||||
* Automatically tracks changes to the ChangeSet
|
||||
* Respects sandbox boundaries and deny-lists
|
||||
|
||||
#### Change Tracking from Tool Invocations
|
||||
|
||||
**Critical Architecture Point:** The ChangeSet is NOT built by parsing LLM output. It is built by recording the effects of tool/skill invocations:
|
||||
|
||||
```python
|
||||
class SkillExecutionContext:
|
||||
"""Context provided to skill execution."""
|
||||
|
||||
def __init__(self, plan: Plan, sandbox: Sandbox):
|
||||
self.plan = plan
|
||||
self.sandbox = sandbox
|
||||
self.changes: list[Change] = []
|
||||
|
||||
def record_change(self, change: Change) -> None:
|
||||
"""Record a change made by a skill."""
|
||||
self.changes.append(change)
|
||||
self.plan.changeset.add_change(change)
|
||||
|
||||
|
||||
class WriteFileSkill:
|
||||
"""Built-in skill for writing files."""
|
||||
|
||||
def execute(self, path: str, content: str, ctx: SkillExecutionContext) -> None:
|
||||
# Get the resource handler for this path
|
||||
handler = ctx.sandbox.get_handler(path)
|
||||
|
||||
# Perform the write (returns Change record)
|
||||
change = handler.write(path, content, ctx.sandbox)
|
||||
|
||||
# Record the change
|
||||
ctx.record_change(change)
|
||||
```
|
||||
|
||||
This approach means:
|
||||
* Every resource modification is explicit and tracked
|
||||
* The ChangeSet accurately reflects what was done, not what was said
|
||||
* Rollback is precise (replay inverse of recorded changes)
|
||||
* Audit logs show exactly what each skill invocation did
|
||||
|
||||
## Session
|
||||
|
||||
### What a Session Is
|
||||
@@ -1882,6 +2123,85 @@ This enables:
|
||||
* better auditing
|
||||
* accurate sandbox scoping
|
||||
|
||||
### Unified Resource Abstraction Layer
|
||||
|
||||
CleverAgents provides a unified abstraction that allows skills to work with any resource type through a consistent interface. This enables:
|
||||
|
||||
1. **Resource-agnostic skills**: A skill like `read_content(path)` works whether the path refers to a file, database record, or API endpoint
|
||||
2. **Consistent sandbox semantics**: All resources support the same sandbox lifecycle (create, read, write, checkpoint, rollback)
|
||||
3. **Pluggable resource handlers**: New resource types can be added without modifying existing skills
|
||||
4. **Unified change tracking**: All resource modifications flow into the same ChangeSet model
|
||||
|
||||
#### Resource Handler Interface
|
||||
|
||||
Every resource type implements this interface:
|
||||
|
||||
```python
|
||||
class ResourceHandler(Protocol):
|
||||
"""Handler for a specific resource type."""
|
||||
|
||||
def read(self, path: str, sandbox: Sandbox) -> Content:
|
||||
"""Read content from the sandboxed resource."""
|
||||
...
|
||||
|
||||
def write(self, path: str, content: Content, sandbox: Sandbox) -> Change:
|
||||
"""Write content and return the Change record."""
|
||||
...
|
||||
|
||||
def delete(self, path: str, sandbox: Sandbox) -> Change:
|
||||
"""Delete resource and return the Change record."""
|
||||
...
|
||||
|
||||
def list(self, pattern: str, sandbox: Sandbox) -> list[str]:
|
||||
"""List paths matching pattern."""
|
||||
...
|
||||
|
||||
def diff(self, path: str, sandbox: Sandbox) -> str:
|
||||
"""Generate diff between sandbox and original state."""
|
||||
...
|
||||
|
||||
def supports_operation(self, operation: OperationType) -> bool:
|
||||
"""Check if this resource supports the given operation."""
|
||||
...
|
||||
```
|
||||
|
||||
#### Built-in Resource Handlers
|
||||
|
||||
| Resource Type | Handler | Read | Write | Delete | Sandbox Strategy |
|
||||
|--------------|---------|------|-------|--------|------------------|
|
||||
| Filesystem | `FilesystemHandler` | ✓ | ✓ | ✓ | copy_on_write |
|
||||
| Git Repository | `GitHandler` | ✓ | ✓ | ✓ | git_worktree |
|
||||
| PostgreSQL | `PostgresHandler` | ✓ | ✓ | ✓ | transaction |
|
||||
| SQLite | `SQLiteHandler` | ✓ | ✓ | ✓ | copy_on_write |
|
||||
| HTTP API | `HTTPHandler` | ✓ | ✓* | ✓* | none |
|
||||
| S3 Bucket | `S3Handler` | ✓ | ✓ | ✓ | versioning |
|
||||
|
||||
*HTTP writes may not be sandboxable depending on the API
|
||||
|
||||
#### Resource Path Resolution
|
||||
|
||||
Paths in skills are resolved through a resource routing system:
|
||||
|
||||
```
|
||||
path://resource-name/relative/path
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Resource Router │
|
||||
│ - Parses path scheme │
|
||||
│ - Looks up resource by name │
|
||||
│ - Routes to appropriate handler │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Handler (e.g., GitHandler) │
|
||||
│ - Resolves relative path │
|
||||
│ - Operates on sandboxed state │
|
||||
│ - Returns Change record │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
For convenience, paths without a scheme default to the project's primary filesystem resource.
|
||||
|
||||
## Context
|
||||
|
||||
Context in is not "dump all files into an LLM." It is a system that:
|
||||
@@ -2354,56 +2674,73 @@ Where `Change` includes:
|
||||
* `patch` (optional unified diff)
|
||||
* `language` (optional but helpful for validation)
|
||||
|
||||
**Key requirement:** the pipeline must stop treating a plan as "one output string"; it must treat the plan as "a structured set of file ops."
|
||||
**Key requirement:** the pipeline must stop treating a plan as "one output string"; it must treat the plan as "a structured set of resource ops."
|
||||
|
||||
#### B. Force a structured LLM output contract (JSON-first)
|
||||
#### B. Implement Tool-Based Resource Modification (Modern Architecture)
|
||||
|
||||
To get multi-file reliably, require the execution actor (or a dedicated "renderer" actor) to output **strict JSON** that conforms to a schema, for example:
|
||||
**CRITICAL: Do NOT parse LLM output to extract code.** Instead, use the modern tool-based approach:
|
||||
|
||||
```json
|
||||
{
|
||||
"changes": [
|
||||
{"operation": "create", "path": "src/main.py", "content": "..."},
|
||||
{"operation": "create", "path": "src/routes/users.py", "content": "..."},
|
||||
{"operation": "modify", "path": "README.md", "patch": "...unified diff..."}
|
||||
],
|
||||
"notes": ["Run tests: pytest", "Add env var: DATABASE_URL"]
|
||||
}
|
||||
```
|
||||
LLM → calls tools/skills directly → tools modify sandbox → ChangeSet built from tool invocations
|
||||
```
|
||||
|
||||
Implementation details:
|
||||
|
||||
* Define a JSON schema (or Pydantic model) and validate strictly.
|
||||
* If invalid JSON:
|
||||
1. **Provide built-in resource skills** that LLMs call directly:
|
||||
* `read_file(path)` - Read file contents
|
||||
* `write_file(path, content)` - Create/overwrite file
|
||||
* `edit_file(path, edits)` - Apply targeted edits (search/replace or line-based)
|
||||
* `delete_file(path)` - Remove file
|
||||
* `move_file(src, dst)` - Rename/move file
|
||||
* `create_directory(path)` - Create directory
|
||||
* `list_files(pattern)` - List files matching glob
|
||||
* `search_files(pattern)` - Search file contents
|
||||
|
||||
* run an automatic "repair" step (LLM or deterministic fixer),
|
||||
* if still invalid, fail the phase with a clear error and keep artifacts for review (don't write junk into files).
|
||||
2. **Each skill invocation that modifies resources** creates a `Change` record:
|
||||
```python
|
||||
# When write_file("src/main.py", content) is called:
|
||||
change = Change(
|
||||
operation=OperationType.CREATE,
|
||||
path="src/main.py",
|
||||
content=content,
|
||||
language="python"
|
||||
)
|
||||
changeset.add_change(change)
|
||||
```
|
||||
|
||||
**Why JSON-first matters:** the current "strip only the outer fences" behavior is not enough and will always regress into "explanations in code."
|
||||
3. **The ChangeSet is the accumulated history** of resource-modifying skill calls, NOT parsed LLM text.
|
||||
|
||||
#### C. Provide a fallback parser (markdown/code-fence tolerant) but never "raw dump"
|
||||
**Why tool-based is superior to parsing:**
|
||||
|
||||
Even with JSON-first, a fallback parser is practical. But the fallback must produce the same `ChangeSet` structure.
|
||||
| Aspect | Parsing Approach | Tool-Based Approach |
|
||||
|--------|------------------|---------------------|
|
||||
| Ambiguity | "Is this code or explanation?" | Each operation is explicit |
|
||||
| Atomicity | Parse entire response | Each tool call is discrete |
|
||||
| Rollback | Reconstruct from diff | Replay inverse of recorded changes |
|
||||
| Resource types | Files only | Any resource (files, DBs, APIs) |
|
||||
| Audit trail | What model said | What actually happened |
|
||||
|
||||
A robust fallback should:
|
||||
#### C. Support MCP Tools and Custom Skills
|
||||
|
||||
* Parse multiple fenced blocks
|
||||
* Detect per-block file paths from:
|
||||
Skills can come from multiple sources, all producing the same `Change` records:
|
||||
|
||||
* headings like `### path/to/file.py`
|
||||
* inline hints like `// file: ...`
|
||||
* or an explicit preamble list
|
||||
* Reject/strip explanatory prose **unless** it is explicitly routed to a non-code artifact (e.g., `notes.md` or `PLAN.md`)
|
||||
1. **Built-in skills**: Core file operations, git, search
|
||||
2. **MCP servers**: External tools wrapped with sandbox interception
|
||||
3. **Custom inline skills**: Python code in actor YAML
|
||||
|
||||
**Never again** write the entire model response into a source file.
|
||||
All skills operate through the unified resource abstraction layer:
|
||||
* Paths are resolved to sandboxed locations
|
||||
* Write operations create Change records
|
||||
* Capability metadata enforces safety restrictions
|
||||
|
||||
#### D. Implement directory creation and scaffolding rules
|
||||
|
||||
Once you can create multiple files, you need deterministic rules for directories:
|
||||
Skills automatically handle directory operations:
|
||||
|
||||
* auto-create directories for new files (safe, idempotent)
|
||||
* enforce project root boundaries (never escape the repo root)
|
||||
* enforce ignore patterns / deny lists (e.g., don't create inside `.git/`)
|
||||
* Auto-create parent directories for new file paths (safe, idempotent)
|
||||
* Enforce project root boundaries (never escape the sandbox)
|
||||
* Enforce deny-list patterns (`.git/`, `node_modules/`, etc.)
|
||||
* Respect project `.gitignore` and custom exclusion rules
|
||||
|
||||
This directly addresses the "create REST API" expectation (routes/, models/, etc.) that current code cannot meet.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user