feature/docs-spec-imp #140

Merged
freemo merged 2 commits from feature/docs-spec-imp into master 2026-02-22 06:37:47 +00:00
3 changed files with 265 additions and 11 deletions
+22
View File
@@ -550,9 +550,31 @@
});
}
// ── Desktop detection ────────────────────────────────────────────────
//
// This entire script is designed for the desktop persistent sidebar
// (≥76.25em). On mobile, Material uses a slide-out drawer with its
// own drill-down navigation, scroll management, and back-button
// behaviour. Running our DOM restructuring (convertPageItems,
// initCollapsibleToc, lockActiveAncestorSections) on mobile breaks
// the drawer's scroll container and makes it impossible to scroll
// past the visible items.
//
// We use matchMedia to detect the breakpoint and only bootstrap on
// desktop. On resize/orientation changes that cross the breakpoint,
// the page reloads via Material's SPA system anyway, so we don't need
// a live resize listener.
var desktopQuery = window.matchMedia("(min-width: 76.25em)");
function isDesktop() {
return desktopQuery.matches;
}
// ── Bootstrap ───────────────────────────────────────────────────────
function bootstrap() {
if (!isDesktop()) return; // skip all DOM manipulation on mobile
initCollapsibleToc();
convertPageItems();
lockActiveAncestorSections();
+171
View File
@@ -43300,3 +43300,174 @@ The following table shows which Protocol each pipeline slot implements and what
| **Pipeline: PreambleGenerator** | `config.toml` `context.pipeline.preamble-generator` or project/plan YAML | TOML/YAML + Python module | Configuration-driven (scope chain) |
| **Pipeline: SkeletonCompressor** | `config.toml` `context.pipeline.skeleton-compressor` or project/plan YAML | TOML/YAML + Python module | Configuration-driven (scope chain) |
## Implementation Timeline
This section provides a high-level overview of the CleverAgents implementation roadmap: what has been completed, what is in progress, and what remains across all major milestones. The project kicked off on **2026-02-09** and follows a milestone-driven schedule with six parallel workstreams converging at defined merge points.
### Current Status Summary
As of Day 14 (2026-02-22), approximately **42% of all implementation tasks are complete** (74 of 178 tracked tasks). The foundational layers — domain models, persistence, YAML schemas, CLI commands, quality automation, and CI/CD — are substantially built out. The primary blocker for the first end-to-end milestone is the **execute/apply pipeline**: the tool-aware actor runtime that drives LLM-based plan execution and the sandbox merge path that commits changes to real resources.
!!! warning "Critical Priority: v2 Feature Restoration"
During the transition from the earlier v2 architecture to the current v3 spec-aligned design, approximately 4,000 lines of LangGraph agent infrastructure were stubbed or removed. **Restoring these v2 capabilities** — specifically the LangGraph-based execution actors, tool-calling runtime, and agent graph compilation — is the highest-priority blocker for reaching a minimal working version. Until the actor runtime can invoke tools through the LLM and capture changes into a ChangeSet, the plan execute/apply lifecycle cannot function end-to-end.
### Parallel Workstreams
Development is organized into six concurrent workstreams plus continuous quality and testing tracks:
| Track | Focus Area | Status |
| :---- | :--------- | :----- |
| **A** — Plan Lifecycle | Action/plan domain models, persistence, CLI alignment, execute/apply wiring | Core models and persistence done; execute/apply pipeline in progress |
| **B** — Projects & Resources | Resource registry, project model, resource types, sandbox strategies, CLI | Registry tables, models, and basic CLI done; handler runtime and auto-discovery remaining |
| **C** — Actors, Tools, Skills | Actor YAML schema/loader/compiler, tool registry/runtime, skill framework, MCP adapter, validation runner | Schemas, loaders, tool runtime, and file/git/search tools done; actor compiler, MCP runtime, and validation runner remaining |
| **D** — Decisions & Apply | Decision tree model, change tracking, diff review, apply pipeline, correction engine | Change tracking models and diff artifacts done; decision persistence, correction engine, and apply pipeline remaining |
| **E** — Subplans | Subplan domain model, orchestration, parallel execution, three-way merge | Domain model done; orchestrator and merge logic remaining |
| **F** — ACMS & Context | UKO ontology, CRP protocol, context strategies, assembly pipeline, hot/warm/cold tiers | Not yet started; scheduled for later milestones |
| **Q** — Quality & CI | Pre-commit hooks, Forgejo CI/CD, coverage enforcement (97%), security scanning | Complete and ahead of schedule |
| **T** — Testing | Behave unit tests, Robot integration tests, ASV benchmarks, end-to-end smoke suites | Ongoing; currently carried by quality specialist and feature owners |
### What Has Been Completed
The following major areas are fully or substantially implemented:
- **Domain models**: Plan (with full lifecycle and subplan hierarchy), Action (with YAML schema and typed arguments), Tool and Validation (with resource bindings and capability metadata), Skill (with resolver and cycle detection), Session (with message ordering and token tracking), Resource (with DAG edges and type constraints), ChangeSet and ToolInvocation tracking models.
- **Persistence layer**: SQLite database with Alembic migrations for actions, plans, sessions, session messages, tools, tool resource bindings, validation attachments, resources, resource types, and resource edges. Repository classes with retry decorators and referential integrity guards.
- **YAML configuration schemas**: Action config schema with env var interpolation and key normalization, Actor config schema with three actor types (LLM/TOOL/GRAPH) and graph topology validation with cycle detection, Skill config schema with MCP server and Agent Skills folder support. All schemas include loader/validator Pydantic models with `from_yaml()` factory methods.
- **CLI commands**: Action (create/list/show/archive), Plan (use/execute/apply/status/list/cancel/diff/artifacts), Skill (add/remove/list/show/tools), Session (create/list/show/delete/tell/export/import), Actor (run/add/remove/list/show with context management), Tool, LSP, Resource, Validation, Config, and Invariant command groups — all with multi-format output support (rich/json/yaml/plain/table/color).
- **Tool runtime**: Tool registry with thread-safe operations, tool runner with four-stage lifecycle (discover/activate/execute/deactivate), built-in file tools (read/write/edit/delete/list/search), git tools (status/diff/log/blame), tool call router for multi-provider format normalization (OpenAI/Anthropic/LangChain), and path traversal prevention.
- **Sandbox infrastructure**: Sandbox protocol with status transitions, NoSandbox implementation for non-sandboxable resources, sandbox factory, thread-safe sandbox manager with plan-scoped tracking, and three merge strategies (git, sequential, JSON deep-merge).
- **Actor loader**: Thread-safe discovery with SHA-256 content-hash caching, namespace normalization, duplicate detection, and tool reference resolution.
- **Automation levels**: Manual, review-before-apply, and full-automation modes with plan-level overrides, settings resolution, pause/resume, and auto-progression.
- **Security hardening**: All `eval()`/`exec()` calls replaced with named operation registries, config security scanner detecting 15 disallowed patterns, AST-based code validation, Jinja2 sandboxed environments.
- **Quality automation**: Pre-commit hooks (12 hooks across 5 categories), Forgejo CI/CD pipeline with 10 nox-routed jobs, nightly quality monitoring, ADR compliance checking, 97% test coverage enforcement, and comprehensive security scanning (Bandit + Semgrep + Vulture).
### What Remains To Be Done
The following areas require completion to reach the full specification:
- **v2 LangGraph restoration and actor runtime**: The execution actor runtime that invokes LLMs via LangGraph, routes tool calls through the tool runner, and captures changes into a ChangeSet. This is the critical path for plan execution.
- **Actor compiler**: Compiling actor YAML configurations into runnable LangGraph StateGraph instances, including graph node wiring, conditional routing, and subgraph invocation.
- **Apply pipeline**: The sandbox merge and commit path that transitions execute-phase sandbox changes into real project resources, with validation gating hooks and conflict detection.
- **MCP adapter runtime**: Live connection to MCP servers for external tool discovery and invocation at runtime (config schemas exist; runtime adapter pending).
- **Agent Skills loader**: Runtime loading and execution of `SKILL.md`-based instruction-driven workflows.
- **Validation runner**: Executing required and informational validations attached to resources, with pass/fail gating on the apply phase.
- **Decision tree**: Domain model, persistence, CLI viewing (`plan tree`, `plan explain`), and recording of decisions during Strategize and Execute phases.
- **Invariant reconciliation**: Runtime enforcement of global/project/plan-scoped invariants with precedence resolution and the Invariant Reconciliation Actor.
- **Correction engine**: Revert and append correction modes for editing the decision tree and selectively recomputing affected subtrees.
- **Subplan orchestration**: Spawning child plans during Execute, parallel execution with configurable concurrency, three-way merge of subplan results, and failure handling.
- **ACMS v1**: Universal Knowledge Ontology (UKO) with RDF-based resource representation, Context Request Protocol (CRP), pluggable context strategies, the ten-component Context Assembly Pipeline, and hot/warm/cold tiered storage with per-actor scoped views.
- **Large-project context management**: Indexing 10,000+ file codebases, bounded memory context assembly, and skeleton compression for parent-to-child context propagation.
- **Resource handlers and auto-discovery**: Runtime handlers for built-in resource types (git-checkout, fs-mount), resource DAG auto-discovery, and resource/tool binding resolution.
- **LSP integration**: LSP Registry stubs, LSP Runtime lifecycle management, and LSPToolAdapter for exposing language intelligence as actor tools (server mode deferred).
- **Server connectivity stubs**: Client-side interfaces for future server communication (connect command, local/remote detection, NotImplementedError stubs). The server itself is a separate project.
- **Automation profiles**: Full implementation of the eight built-in profiles and custom profile registration with confidence-threshold gating.
- **Estimation actors**: Cost and risk estimation for plans before execution.
- **Checkpointing and rollback**: Per-plan checkpoint creation during Execute and rollback to specific checkpoints.
### Milestone Roadmap
#### Milestone 1: Minimal Plan Execution (~Day 16)
**Goal**: A minimally usable local-mode flow where a user can register an action from YAML, link a git repository resource to a project, and run a plan end-to-end (`plan use` -> `plan execute` -> `plan diff` -> `plan apply`) with a sandboxed workspace and tool-based change capture.
**Current status**: ~82% of M1 tasks complete. The primary gap is the tool-aware execution pipeline — the actor runtime must be able to invoke the LLM, route tool calls, and capture the resulting changes. This depends directly on restoring the v2 LangGraph execution infrastructure.
**Key remaining work**:
- Restore v2 LangGraph agent runtime with tool-calling support
- Wire plan execute to actor runtime with ChangeSet capture via tool invocations
- Implement sandbox merge/apply pipeline with validation gating hooks
- Complete resource handler runtime and tool/resource bindings
- End-to-end source-code smoke test suite
#### Milestone 2: Actor Compiler + Full LLM Integration (~Day 20)
**Goal**: Actor YAML files compile into live LangGraph graphs. Custom actors (strategy, execution, estimation) are fully operational. The tool router normalizes calls across providers, the validation runner enforces resource-attached validations, and the MCP adapter connects to external tool servers.
**Key remaining work**:
- Actor compiler (YAML -> LangGraph StateGraph)
- MCP adapter runtime and Agent Skills loader
- Validation runner with required/informational mode enforcement
- Full multi-provider tool routing (OpenAI, Anthropic, Google, OpenRouter)
#### Milestone 3: Decision Tree + Correction (~Day 26)
**Goal**: Decisions are recorded during Strategize and Execute phases and persisted to the database. Users can view the decision tree (`plan tree`), inspect individual decisions (`plan explain`), manage invariants (`invariant add/list/remove`), and correct decisions (`plan correct --mode revert|append`) with selective subtree recomputation.
**Key remaining work**:
- Decision domain model and persistence
- Decision recording during strategize/execute phases
- Invariant enforcement and reconciliation
- Correction engine (revert and append modes)
- Plan tree and explain CLI commands
#### Milestone 4: Subplans + Parallel Execution (~Day 30)
**Goal**: Plans can spawn child plans (subplans) during execution. Subplans execute in parallel with configurable concurrency limits. Results are merged back using three-way merge strategies. This milestone is the threshold for handling **large, complex codebases** — the system can decompose a large task into independent subtasks, execute them concurrently, and merge the results.
**Key remaining work**:
- Subplan orchestrator and scheduler
- Parallel execution with max_parallel limits and fail-fast behavior
- Three-way merge of subplan changesets with conflict detection
- Phase reversion (constrained apply -> back to strategize)
- Parent plan status aggregation across subplan tree
#### Milestone 5: ACMS + Large-Project Context (~Day 34)
**Goal**: The Advanced Context Management System v1 is operational. Projects with 10,000+ files can be indexed and queried. The context assembly pipeline produces scoped, budget-constrained context views for actors. Hot/warm/cold storage tiers manage context lifecycle.
**Key remaining work**:
- UKO ontology with RDF-based resource representation
- CRP protocol for actor context requests
- Context strategies (keyword, semantic, graph, temporal)
- Ten-component context assembly pipeline
- Hot/warm/cold tiered storage with skeleton compression
#### Milestone 6: Large-Scale Autonomous Execution (~Day 38)
**Goal**: The system can autonomously execute a large-scale task (e.g., porting a substantial codebase) using hierarchical plan decomposition with 4+ levels of subplans, decision correction with selective subtree recomputation, parallel execution scaling to 10+ concurrent subplans, and validation-gated apply. This is the target for demonstrating production-level autonomous capability in local mode.
**Key remaining work**:
- Hierarchical decomposition (recursive subplan planning)
- Autonomous estimation (cost/risk/time) before execution
- Performance tuning for large-project context assembly
- End-to-end porting run with validation gates
- Comprehensive integration test suite
### Beyond Day 38: Extended Roadmap
#### Server Connectivity — Client Side (~Days 39-43)
Client-side infrastructure for connecting to a future CleverAgents server. The server is a separate project; this work covers only the client stubs and interfaces.
- HTTP client infrastructure for server communication
- Plan sync client (push/pull plan state to server)
- WebSocket client for real-time event streaming
- Remote project support (request server-side execution)
#### Full Feature Polish (~Days 44-48)
Final refinements to achieve the complete specification:
- Automation profile refinements (all eight built-in profiles fully operational)
- Cost/risk estimation actor integration
- Checkpoint and rollback hardening
- LSP integration stubs for future IDE plugin support
- Performance tuning and benchmark validation across all subsystems
- Comprehensive documentation and release candidate preparation
### Schedule Risk Summary
The project is currently running approximately **7-9 days behind** the original aggressive schedule. The primary causes are: (1) the scope of v2-to-v3 architectural transition was larger than anticipated, requiring significant model realignment and test rewriting; (2) CI/CD pipeline stabilization consumed more effort than planned due to container resource constraints and cross-environment compatibility issues; and (3) the execute/apply pipeline — the critical vertical slice — was deferred in favor of breadth-first foundation work.
The gap is expected to **narrow** in later milestones because the foundational infrastructure (persistence, CLI patterns, quality gates, domain models) is now solid and reusable. Each subsequent milestone builds on proven patterns rather than establishing new ones.
!!! note "Server Mode"
Server mode implementation is explicitly deferred. The 30-day local-mode target focuses on a fully functional standalone client. Server connectivity is scoped as client-side stubs only — the server itself will be developed as a separate project. This boundary is enforced by the ACP (Agent Client Protocol) abstraction: in local mode, ACP maps to in-process service calls; adding server transport later requires no changes to core logic.
+72 -11
View File
@@ -105,11 +105,19 @@
* when unchecked. Without this, the <nav> container retains residual
* height from padding/grid-rows even when the checkbox is unchecked,
* causing visible whitespace below collapsed section headings.
*
* DESKTOP ONLY (≥76.25em): On mobile, Material uses a slide-out drawer
* with drill-down navigation that depends on nested <nav> elements being
* in the render tree. Applying display:none on mobile breaks both the
* drill-down animation and scroll-height computation, preventing users
* from scrolling to see all nav items.
* ----------------------------------------------------------------------- */
/* Fully hide children of unchecked primary nav sections */
.md-sidebar--primary .md-nav__item--nested > .md-nav__toggle:not(:checked) ~ .md-nav {
display: none;
/* Fully hide children of unchecked primary nav sections — desktop only */
@media screen and (min-width: 76.25em) {
.md-sidebar--primary .md-nav__item--nested > .md-nav__toggle:not(:checked) ~ .md-nav {
display: none;
}
}
/* --- Leaf-page nav items: section-like behaviour ----------------------------
@@ -219,16 +227,69 @@
transform: rotate(0deg);
}
/* Hide nested nav list when collapsed (TOC items only) */
.md-nav--secondary .md-nav__item--nested.toc-collapsed > .md-nav,
.md-nav--page-toc .md-nav__item--nested.toc-collapsed > .md-nav {
display: none;
/* Hide/show nested nav list when collapsed (TOC items only).
* Desktop only — on mobile the TOC is rendered within Material's drawer
* and must remain in the render tree for correct scroll behaviour. */
@media screen and (min-width: 76.25em) {
.md-nav--secondary .md-nav__item--nested.toc-collapsed > .md-nav,
.md-nav--page-toc .md-nav__item--nested.toc-collapsed > .md-nav {
display: none;
}
.md-nav--secondary .md-nav__item--nested > .md-nav,
.md-nav--page-toc .md-nav__item--nested > .md-nav {
display: block;
}
}
/* Show nested nav when expanded (TOC items only) */
.md-nav--secondary .md-nav__item--nested > .md-nav,
.md-nav--page-toc .md-nav__item--nested > .md-nav {
display: block;
/* --- Mobile sidebar scroll fix -----------------------------------------------
*
* On mobile (< 76.25em), Material renders the primary sidebar as a
* slide-out drawer. The custom toc-collapse.js script (which is
* desktop-only) restructures nav items and collapses sections, but on
* mobile we rely on Material's native drawer navigation. Ensure the
* drawer's content is scrollable when there are more items than fit on
* screen — this was broken because the desktop-only display:none rules
* were interfering with the scroll container height computation, and
* the nav containers lacked explicit overflow declarations.
* ----------------------------------------------------------------------- */
@media screen and (max-width: 76.1875em) {
/* Ensure the sidebar drawer inner wrapper scrolls vertically */
.md-sidebar--primary .md-sidebar__inner {
overflow-y: auto;
-webkit-overflow-scrolling: touch;
}
/* Ensure each nav level within the mobile drawer can scroll when it
* exceeds the viewport height. Material uses absolute positioning
* and transforms for the drill-down layers; this ensures each layer
* is independently scrollable. */
.md-sidebar--primary .md-nav__list {
overflow-y: auto;
-webkit-overflow-scrolling: touch;
}
/* Prevent the overall nav from clipping its children —
* Material's mobile drawer needs visible overflow on the
* horizontal axis for the slide animation, and auto on the
* vertical axis for scrolling. */
.md-sidebar--primary .md-nav--primary {
overflow: visible;
}
/* Undo any items hidden by the JS-driven toc-collapsed class that
* may have been applied before the mobile guard kicks in (race
* condition on resize or orientation change). */
.md-sidebar--primary .toc-collapsed > .md-nav {
display: block !important;
}
/* Undo the section-page forced display set by JS on active pages —
* let Material's native mobile styling handle it. */
.md-nav__item--section-page > .md-nav--secondary {
display: revert !important;
}
}
/* --- Diagram lightbox (fullscreen on click) --------------------------------