ClaudeFlow ported, needs cleanup

2025-08-10 12:00:13 -04:00
parent 08df0d26aa
commit 475b483850
61 changed files with 14025 additions and 3000 deletions
@@ -0,0 +1,572 @@
+# CleverClaude Architecture Overview
+
+This document provides a comprehensive overview of CleverClaude's architecture, design principles, and system components.
+
+## 🏗️ High-Level Architecture
+
+CleverClaude follows a modern, microservices-inspired architecture built on Python's AsyncIO ecosystem. The system is designed for scalability, maintainability, and extensibility.
+
+```mermaid
+graph TB
+    CLI[CLI Interface] --> Core[Core Application]
+    WebUI[Web Interface] --> API[FastAPI Server]
+    API --> Core
+    
+    Core --> AM[Agent Manager]
+    Core --> SC[Swarm Coordinator]  
+    Core --> MCP[MCP Client]
+    Core --> CM[Configuration Manager]
+    
+    AM --> Agents[Agent Pool]
+    SC --> Swarms[Swarm Pool]
+    MCP --> Tools[87+ MCP Tools]
+    
+    Core --> DB[(Database)]
+    Core --> Redis[(Redis Cache)]
+    Core --> Storage[(File Storage)]
+    
+    Agents --> Tasks[Task Queue]
+    Swarms --> Tasks
+    Tasks --> Results[Result Store]
+```
+
+## 🎯 Design Principles
+
+### 1. Async-First Architecture
+- Built on Python AsyncIO for high concurrency
+- Non-blocking I/O operations throughout
+- Event-driven task processing
+- Efficient resource utilization
+
+### 2. Modular Design
+- Clear separation of concerns
+- Dependency injection for loose coupling
+- Plugin architecture for extensibility
+- Interface-based abstractions
+
+### 3. Type Safety
+- Comprehensive type hints using Python 3.11+ features
+- Pydantic models for data validation
+- Runtime type checking in critical paths
+- IDE support and early error detection
+
+### 4. Configuration-Driven
+- YAML-based configuration files
+- Environment variable overrides
+- Hot-reloading of configuration
+- Environment-specific settings
+
+### 5. Observability
+- Structured logging with correlation IDs
+- Comprehensive metrics collection
+- Health checks and monitoring endpoints
+- Distributed tracing support
+
+## 🧩 Core Components
+
+### Core Application (`cleverclaude.core.app`)
+
+The `CleverClaudeApp` class is the central orchestrator that initializes and coordinates all system components.
+
+```python
+class CleverClaudeApp:
+    """Main application orchestrator."""
+    
+    def __init__(self, config_dir: Optional[Path] = None):
+        self.config_dir = config_dir or get_default_config_dir()
+        self.settings = load_settings(self.config_dir)
+        
+        # Core components
+        self.agent_manager: Optional[AgentManager] = None
+        self.swarm_coordinator: Optional[SwarmCoordinator] = None
+        self.mcp_client: Optional[MCPClient] = None
+        self.api_server: Optional[APIServer] = None
+    
+    async def initialize(self) -> None:
+        """Initialize all components in dependency order."""
+        await self._initialize_database()
+        await self._initialize_redis()
+        await self._initialize_agents()
+        await self._initialize_swarms()
+        await self._initialize_mcp()
+        await self._initialize_api()
+    
+    async def start(self) -> None:
+        """Start all services."""
+        await self.api_server.start()
+        self.logger.info("CleverClaude started successfully")
+```
+
+**Key Responsibilities:**
+- Component lifecycle management
+- Dependency injection setup
+- Configuration loading and validation
+- Graceful shutdown handling
+
+### Agent Manager (`cleverclaude.agents.manager`)
+
+Manages the lifecycle and execution of individual AI agents.
+
+```python
+class AgentManager:
+    """Manages agent lifecycle and task execution."""
+    
+    async def create_agent(
+        self,
+        agent_type: AgentType,
+        name: str,
+        capabilities: Optional[Set[str]] = None,
+        **kwargs
+    ) -> str:
+        """Create a new agent instance."""
+        
+    async def execute_task(
+        self,
+        task: Dict[str, Any],
+        agent_id: Optional[str] = None
+    ) -> Dict[str, Any]:
+        """Execute a task on an agent."""
+        
+    async def destroy_agent(self, agent_id: str) -> None:
+        """Remove an agent from the system."""
+```
+
+**Key Features:**
+- Agent type factory (Researcher, Coder, Analyst, etc.)
+- Task routing and load balancing
+- Health monitoring and failure recovery
+- Capability-based agent selection
+- Circuit breaker pattern for fault tolerance
+
+### Swarm Coordinator (`cleverclaude.coordination.swarm`)
+
+Orchestrates multi-agent coordination and swarm intelligence.
+
+```python
+class SwarmCoordinator:
+    """Coordinates agent swarms and task distribution."""
+    
+    async def create_swarm(
+        self,
+        name: str,
+        topology: SwarmTopology,
+        max_agents: int = 50
+    ) -> str:
+        """Create a new agent swarm."""
+        
+    async def submit_task(
+        self,
+        swarm_id: str,
+        task: SwarmTask
+    ) -> str:
+        """Submit a task to a swarm for execution."""
+        
+    async def scale_swarm(
+        self,
+        swarm_id: str,
+        target_size: int
+    ) -> None:
+        """Scale swarm to target size."""
+```
+
+**Supported Topologies:**
+- **Mesh**: Full connectivity, peer-to-peer coordination
+- **Hierarchical**: Tree structure with coordinators and workers
+- **Star**: Central coordinator with spoke workers
+- **Ring**: Circular communication patterns
+
+**Key Features:**
+- Dynamic scaling based on workload
+- Fault-tolerant task distribution
+- Load balancing across agents
+- Performance metrics and optimization
+- Cross-swarm coordination
+
+### MCP Client (`cleverclaude.mcp.client`)
+
+Implements the Model Context Protocol for external tool integration.
+
+```python
+class MCPClient:
+    """MCP (Model Context Protocol) client for tool execution."""
+    
+    async def execute_tool(
+        self,
+        tool_name: str,
+        parameters: Dict[str, Any]
+    ) -> MCPToolExecutionResult:
+        """Execute an MCP tool with given parameters."""
+        
+    async def get_available_tools(self) -> Dict[str, MCPToolInfo]:
+        """Get list of available tools with metadata."""
+        
+    async def batch_execute(
+        self,
+        requests: List[Dict[str, Any]]
+    ) -> List[MCPToolExecutionResult]:
+        """Execute multiple tools in parallel."""
+```
+
+**Tool Categories:**
+- **Swarm Management**: 15+ tools for swarm operations
+- **Neural Operations**: 20+ tools for AI model management
+- **Memory Management**: 10+ tools for persistent storage
+- **Performance Monitoring**: 15+ tools for metrics and analysis
+- **Workflow Automation**: 12+ tools for task orchestration
+- **GitHub Integration**: 8+ tools for repository management
+- **DAA Tools**: 10+ tools for autonomous agents
+- **System Tools**: 8+ tools for system operations
+
+## 📊 Data Flow Architecture
+
+### Task Execution Flow
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant API
+    participant Core
+    participant AM as Agent Manager
+    participant Agent
+    participant SC as Swarm Coordinator
+    participant MCP
+    
+    Client->>API: Submit Task Request
+    API->>Core: Route Request
+    
+    alt Single Agent Task
+        Core->>AM: Execute Task
+        AM->>Agent: Assign Task
+        Agent->>Agent: Process Task
+        Agent->>AM: Return Result
+        AM->>Core: Task Complete
+    else Swarm Task
+        Core->>SC: Submit to Swarm
+        SC->>SC: Select Agents
+        SC->>AM: Distribute Subtasks
+        AM->>Agent: Execute Subtasks
+        Agent->>AM: Return Results
+        AM->>SC: Aggregate Results
+        SC->>Core: Swarm Task Complete
+    end
+    
+    opt MCP Tool Usage
+        Agent->>MCP: Execute Tool
+        MCP->>MCP: Tool Processing
+        MCP->>Agent: Tool Result
+    end
+    
+    Core->>API: Return Response
+    API->>Client: Task Result
+```
+
+### Memory and State Management
+
+```mermaid
+graph LR
+    App[Application State] --> Memory[Memory Manager]
+    App --> Cache[Redis Cache]
+    App --> DB[PostgreSQL DB]
+    
+    Memory --> Namespace[Namespaced Storage]
+    Memory --> TTL[TTL Management]
+    Memory --> Persistence[Persistent Memory]
+    
+    Cache --> Session[Session Data]
+    Cache --> TaskQueue[Task Queues]
+    Cache --> Metrics[Real-time Metrics]
+    
+    DB --> Config[Configuration]
+    DB --> History[Task History]
+    DB --> Analytics[Analytics Data]
+```
+
+## 🔧 Configuration Architecture
+
+CleverClaude uses a hierarchical configuration system with multiple override levels:
+
+```yaml
+# config.yaml
+app:
+  name: "CleverClaude"
+  version: "2.0.0"
+  environment: "development"
+  debug: true
+
+database:
+  url: "postgresql+asyncpg://user:pass@localhost/cleverclaude"
+  pool_size: 10
+  max_overflow: 20
+
+redis:
+  url: "redis://localhost:6379/0"
+  connection_pool_size: 10
+
+agents:
+  max_agents: 100
+  default_timeout: 300
+  health_check_interval: 30
+  supported_types:
+    - researcher
+    - coder
+    - analyst
+    - coordinator
+    - reviewer
+    - tester
+
+swarm:
+  default_topology: "mesh"
+  max_swarm_size: 50
+  coordination_timeout: 60
+  scaling:
+    enabled: true
+    min_agents: 1
+    max_agents: 100
+    scale_up_threshold: 0.8
+    scale_down_threshold: 0.3
+
+mcp:
+  servers:
+    - name: "claude-flow-server"
+      url: "http://localhost:8001/mcp"
+      enabled: true
+    - name: "neural-server" 
+      url: "http://localhost:8002/mcp"
+      enabled: true
+
+api:
+  host: "127.0.0.1"
+  port: 8000
+  docs_enabled: true
+  cors_enabled: true
+
+monitoring:
+  metrics_enabled: true
+  log_level: "INFO"
+  log_format: "json"
+  tracing_enabled: false
+```
+
+**Configuration Priority (highest to lowest):**
+1. Command-line arguments
+2. Environment variables (`CLEVERCLAUDE_*`)
+3. Local configuration files
+4. Default values
+
+## 🏃‍♂️ Performance Architecture
+
+### Concurrency Model
+
+```python
+# AsyncIO-based concurrency
+async def handle_concurrent_tasks():
+    """Handle multiple tasks concurrently."""
+    tasks = [
+        process_task_async(task1),
+        process_task_async(task2),
+        process_task_async(task3)
+    ]
+    results = await asyncio.gather(*tasks, return_exceptions=True)
+    return results
+
+# Connection pooling
+async def get_database_connection():
+    """Get connection from pool."""
+    async with database_pool.acquire() as connection:
+        return await connection.fetch("SELECT * FROM agents")
+```
+
+### Caching Strategy
+
+- **L1 Cache**: In-memory Python dictionaries for hot data
+- **L2 Cache**: Redis for session data and temporary results
+- **L3 Cache**: Database query result caching
+- **CDN**: Static asset delivery (in production)
+
+### Resource Management
+
+```python
+class ResourceManager:
+    """Manages system resources and limits."""
+    
+    def __init__(self, config: ResourceConfig):
+        self.max_agents = config.max_agents
+        self.max_memory = config.max_memory_mb * 1024 * 1024
+        self.max_cpu_percent = config.max_cpu_percent
+    
+    async def allocate_agent(self) -> bool:
+        """Check if resources are available for new agent."""
+        current_usage = await self.get_current_usage()
+        return (
+            current_usage.agents < self.max_agents and
+            current_usage.memory < self.max_memory and
+            current_usage.cpu_percent < self.max_cpu_percent
+        )
+```
+
+## 🔐 Security Architecture
+
+### Authentication & Authorization
+
+```python
+class SecurityManager:
+    """Handles authentication and authorization."""
+    
+    async def authenticate_request(self, token: str) -> Optional[User]:
+        """Validate request token."""
+        
+    async def authorize_action(
+        self, 
+        user: User, 
+        action: str, 
+        resource: str
+    ) -> bool:
+        """Check if user can perform action on resource."""
+```
+
+### Security Features
+
+- **JWT-based authentication** for API access
+- **Role-based access control** (RBAC) for operations
+- **Rate limiting** to prevent abuse
+- **Input validation** using Pydantic schemas
+- **Secrets management** with environment variables
+- **Audit logging** for security events
+
+## 📈 Scalability Architecture
+
+### Horizontal Scaling
+
+```mermaid
+graph TB
+    LB[Load Balancer] --> API1[API Server 1]
+    LB --> API2[API Server 2]
+    LB --> API3[API Server N]
+    
+    API1 --> Redis[(Redis Cluster)]
+    API2 --> Redis
+    API3 --> Redis
+    
+    API1 --> DB[(PostgreSQL)]
+    API2 --> DB
+    API3 --> DB
+    
+    API1 --> MCP[MCP Servers]
+    API2 --> MCP
+    API3 --> MCP
+```
+
+### Auto-Scaling Triggers
+
+- **CPU utilization** > 80% for 5 minutes
+- **Memory usage** > 85% for 5 minutes  
+- **Queue depth** > 100 pending tasks
+- **Response latency** > 2 seconds average
+
+### Multi-Region Deployment
+
+```yaml
+regions:
+  primary:
+    name: "us-east-1"
+    database: "primary"
+    cache: "redis-cluster-east"
+    
+  secondary:
+    name: "eu-west-1" 
+    database: "read-replica"
+    cache: "redis-cluster-eu"
+    
+  disaster_recovery:
+    name: "us-west-2"
+    database: "backup"
+    cache: "redis-standalone"
+```
+
+## 🧪 Testing Architecture
+
+### Test Pyramid
+
+```mermaid
+pyramid
+    title Test Pyramid
+    
+    "E2E Tests" : 10
+    "Integration Tests" : 30
+    "Unit Tests" : 60
+```
+
+### Test Categories
+
+1. **Unit Tests** (60%): Fast, isolated component tests
+2. **Integration Tests** (30%): Component interaction tests
+3. **End-to-End Tests** (10%): Full system workflow tests
+4. **Property-Based Tests**: Hypothesis-driven edge case testing
+5. **Performance Tests**: Load testing and benchmarking
+
+### Test Infrastructure
+
+```python
+# Test fixtures and mocking
+@pytest.fixture
+async def agent_manager():
+    """Provide mocked agent manager for tests."""
+    manager = AgentManager(test_config, mock_session, mock_redis)
+    await manager.initialize()
+    yield manager
+    await manager.shutdown()
+
+# BDD test scenarios
+Feature: Agent Management
+  Scenario: Create and execute task
+    Given I have an agent manager
+    When I create a researcher agent
+    And I execute a research task
+    Then the task should complete successfully
+```
+
+## 📝 Extension Points
+
+### Custom Agent Types
+
+```python
+class CustomAnalystAgent(BaseAgent):
+    """Custom analyst with specialized capabilities."""
+    
+    async def _process_task(self, task: Dict[str, Any]) -> Dict[str, Any]:
+        """Custom task processing logic."""
+        if task["type"] == "custom_analysis":
+            return await self._perform_custom_analysis(task["data"])
+        return await super()._process_task(task)
+
+# Register custom agent type
+AgentFactory.register_agent_type("custom_analyst", CustomAnalystAgent)
+```
+
+### Custom MCP Tools
+
+```python
+@mcp_tool("custom_data_processor")
+async def process_custom_data(parameters: Dict[str, Any]) -> MCPToolResult:
+    """Custom data processing tool."""
+    data = parameters.get("data")
+    processed = await custom_processing_logic(data)
+    return MCPToolResult(success=True, result={"processed_data": processed})
+```
+
+### Custom Swarm Topologies
+
+```python
+class CustomTopology(SwarmTopology):
+    """Custom swarm coordination pattern."""
+    
+    async def distribute_task(
+        self,
+        task: SwarmTask,
+        agents: List[Agent]
+    ) -> List[SubTask]:
+        """Custom task distribution logic."""
+        return await self._custom_distribution_algorithm(task, agents)
+```
+
+This architecture provides a solid foundation for building scalable, maintainable AI agent orchestration systems while remaining extensible for future requirements.