cleverclaude-core/docs/architecture.md

# CleverClaude Architecture Overview

This document provides a comprehensive overview of CleverClaude's architecture, design principles, and system components.

## 🏗️ High-Level Architecture

CleverClaude follows a modern, microservices-inspired architecture built on Python's AsyncIO ecosystem. The system is designed for scalability, maintainability, and extensibility.

```mermaid
graph TB
    CLI[CLI Interface] --> Core[Core Application]
    WebUI[Web Interface] --> API[FastAPI Server]
    API --> Core
    
    Core --> AM[Agent Manager]
    Core --> SC[Swarm Coordinator]  
    Core --> MCP[MCP Client]
    Core --> CM[Configuration Manager]
    
    AM --> Agents[Agent Pool]
    SC --> Swarms[Swarm Pool]
    MCP --> Tools[87+ MCP Tools]
    
    Core --> DB[(Database)]
    Core --> Redis[(Redis Cache)]
    Core --> Storage[(File Storage)]
    
    Agents --> Tasks[Task Queue]
    Swarms --> Tasks
    Tasks --> Results[Result Store]
```

## 🎯 Design Principles

### 1. Async-First Architecture
- Built on Python AsyncIO for high concurrency
- Non-blocking I/O operations throughout
- Event-driven task processing
- Efficient resource utilization

### 2. Modular Design
- Clear separation of concerns
- Dependency injection for loose coupling
- Plugin architecture for extensibility
- Interface-based abstractions

### 3. Type Safety
- Comprehensive type hints using Python 3.11+ features
- Pydantic models for data validation
- Runtime type checking in critical paths
- IDE support and early error detection

### 4. Configuration-Driven
- YAML-based configuration files
- Environment variable overrides
- Hot-reloading of configuration
- Environment-specific settings

### 5. Observability
- Structured logging with correlation IDs
- Comprehensive metrics collection
- Health checks and monitoring endpoints
- Distributed tracing support

## 🧩 Core Components

### Core Application (`cleverclaude.core.app`)

The `CleverClaudeApp` class is the central orchestrator that initializes and coordinates all system components.

```python
class CleverClaudeApp:
    """Main application orchestrator."""
    
    def __init__(self, config_dir: Optional[Path] = None):
        self.config_dir = config_dir or get_default_config_dir()
        self.settings = load_settings(self.config_dir)
        
        # Core components
        self.agent_manager: Optional[AgentManager] = None
        self.swarm_coordinator: Optional[SwarmCoordinator] = None
        self.mcp_client: Optional[MCPClient] = None
        self.api_server: Optional[APIServer] = None
    
    async def initialize(self) -> None:
        """Initialize all components in dependency order."""
        await self._initialize_database()
        await self._initialize_redis()
        await self._initialize_agents()
        await self._initialize_swarms()
        await self._initialize_mcp()
        await self._initialize_api()
    
    async def start(self) -> None:
        """Start all services."""
        await self.api_server.start()
        self.logger.info("CleverClaude started successfully")
```

**Key Responsibilities:**
- Component lifecycle management
- Dependency injection setup
- Configuration loading and validation
- Graceful shutdown handling

### Agent Manager (`cleverclaude.agents.manager`)

Manages the lifecycle and execution of individual AI agents.

```python
class AgentManager:
    """Manages agent lifecycle and task execution."""
    
    async def create_agent(
        self,
        agent_type: AgentType,
        name: str,
        capabilities: Optional[Set[str]] = None,
        **kwargs
    ) -> str:
        """Create a new agent instance."""
        
    async def execute_task(
        self,
        task: Dict[str, Any],
        agent_id: Optional[str] = None
    ) -> Dict[str, Any]:
        """Execute a task on an agent."""
        
    async def destroy_agent(self, agent_id: str) -> None:
        """Remove an agent from the system."""
```

**Key Features:**
- Agent type factory (Researcher, Coder, Analyst, etc.)
- Task routing and load balancing
- Health monitoring and failure recovery
- Capability-based agent selection
- Circuit breaker pattern for fault tolerance

### Swarm Coordinator (`cleverclaude.coordination.swarm`)

Orchestrates multi-agent coordination and swarm intelligence.

```python
class SwarmCoordinator:
    """Coordinates agent swarms and task distribution."""
    
    async def create_swarm(
        self,
        name: str,
        topology: SwarmTopology,
        max_agents: int = 50
    ) -> str:
        """Create a new agent swarm."""
        
    async def submit_task(
        self,
        swarm_id: str,
        task: SwarmTask
    ) -> str:
        """Submit a task to a swarm for execution."""
        
    async def scale_swarm(
        self,
        swarm_id: str,
        target_size: int
    ) -> None:
        """Scale swarm to target size."""
```

**Supported Topologies:**
- **Mesh**: Full connectivity, peer-to-peer coordination
- **Hierarchical**: Tree structure with coordinators and workers
- **Star**: Central coordinator with spoke workers
- **Ring**: Circular communication patterns

**Key Features:**
- Dynamic scaling based on workload
- Fault-tolerant task distribution
- Load balancing across agents
- Performance metrics and optimization
- Cross-swarm coordination

### MCP Client (`cleverclaude.mcp.client`)

Implements the Model Context Protocol for external tool integration.

```python
class MCPClient:
    """MCP (Model Context Protocol) client for tool execution."""
    
    async def execute_tool(
        self,
        tool_name: str,
        parameters: Dict[str, Any]
    ) -> MCPToolExecutionResult:
        """Execute an MCP tool with given parameters."""
        
    async def get_available_tools(self) -> Dict[str, MCPToolInfo]:
        """Get list of available tools with metadata."""
        
    async def batch_execute(
        self,
        requests: List[Dict[str, Any]]
    ) -> List[MCPToolExecutionResult]:
        """Execute multiple tools in parallel."""
```

**Tool Categories:**
- **Swarm Management**: 15+ tools for swarm operations
- **Neural Operations**: 20+ tools for AI model management
- **Memory Management**: 10+ tools for persistent storage
- **Performance Monitoring**: 15+ tools for metrics and analysis
- **Workflow Automation**: 12+ tools for task orchestration
- **GitHub Integration**: 8+ tools for repository management
- **DAA Tools**: 10+ tools for autonomous agents
- **System Tools**: 8+ tools for system operations

## 📊 Data Flow Architecture

### Task Execution Flow

```mermaid
sequenceDiagram
    participant Client
    participant API
    participant Core
    participant AM as Agent Manager
    participant Agent
    participant SC as Swarm Coordinator
    participant MCP
    
    Client->>API: Submit Task Request
    API->>Core: Route Request
    
    alt Single Agent Task
        Core->>AM: Execute Task
        AM->>Agent: Assign Task
        Agent->>Agent: Process Task
        Agent->>AM: Return Result
        AM->>Core: Task Complete
    else Swarm Task
        Core->>SC: Submit to Swarm
        SC->>SC: Select Agents
        SC->>AM: Distribute Subtasks
        AM->>Agent: Execute Subtasks
        Agent->>AM: Return Results
        AM->>SC: Aggregate Results
        SC->>Core: Swarm Task Complete
    end
    
    opt MCP Tool Usage
        Agent->>MCP: Execute Tool
        MCP->>MCP: Tool Processing
        MCP->>Agent: Tool Result
    end
    
    Core->>API: Return Response
    API->>Client: Task Result
```

### Memory and State Management

```mermaid
graph LR
    App[Application State] --> Memory[Memory Manager]
    App --> Cache[Redis Cache]
    App --> DB[PostgreSQL DB]
    
    Memory --> Namespace[Namespaced Storage]
    Memory --> TTL[TTL Management]
    Memory --> Persistence[Persistent Memory]
    
    Cache --> Session[Session Data]
    Cache --> TaskQueue[Task Queues]
    Cache --> Metrics[Real-time Metrics]
    
    DB --> Config[Configuration]
    DB --> History[Task History]
    DB --> Analytics[Analytics Data]
```

## 🔧 Configuration Architecture

CleverClaude uses a hierarchical configuration system with multiple override levels:

```yaml
# config.yaml
app:
  name: "CleverClaude"
  version: "2.0.0"
  environment: "development"
  debug: true

database:
  url: "postgresql+asyncpg://user:pass@localhost/cleverclaude"
  pool_size: 10
  max_overflow: 20

redis:
  url: "redis://localhost:6379/0"
  connection_pool_size: 10

agents:
  max_agents: 100
  default_timeout: 300
  health_check_interval: 30
  supported_types:
    - researcher
    - coder
    - analyst
    - coordinator
    - reviewer
    - tester

swarm:
  default_topology: "mesh"
  max_swarm_size: 50
  coordination_timeout: 60
  scaling:
    enabled: true
    min_agents: 1
    max_agents: 100
    scale_up_threshold: 0.8
    scale_down_threshold: 0.3

mcp:
  servers:
    - name: "claude-flow-server"
      url: "http://localhost:8001/mcp"
      enabled: true
    - name: "neural-server" 
      url: "http://localhost:8002/mcp"
      enabled: true

api:
  host: "127.0.0.1"
  port: 8000
  docs_enabled: true
  cors_enabled: true

monitoring:
  metrics_enabled: true
  log_level: "INFO"
  log_format: "json"
  tracing_enabled: false
```

**Configuration Priority (highest to lowest):**
1. Command-line arguments
2. Environment variables (`CLEVERCLAUDE_*`)
3. Local configuration files
4. Default values

## 🏃‍♂️ Performance Architecture

### Concurrency Model

```python
# AsyncIO-based concurrency
async def handle_concurrent_tasks():
    """Handle multiple tasks concurrently."""
    tasks = [
        process_task_async(task1),
        process_task_async(task2),
        process_task_async(task3)
    ]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    return results

# Connection pooling
async def get_database_connection():
    """Get connection from pool."""
    async with database_pool.acquire() as connection:
        return await connection.fetch("SELECT * FROM agents")
```

### Caching Strategy

- **L1 Cache**: In-memory Python dictionaries for hot data
- **L2 Cache**: Redis for session data and temporary results
- **L3 Cache**: Database query result caching
- **CDN**: Static asset delivery (in production)

### Resource Management

```python
class ResourceManager:
    """Manages system resources and limits."""
    
    def __init__(self, config: ResourceConfig):
        self.max_agents = config.max_agents
        self.max_memory = config.max_memory_mb * 1024 * 1024
        self.max_cpu_percent = config.max_cpu_percent
    
    async def allocate_agent(self) -> bool:
        """Check if resources are available for new agent."""
        current_usage = await self.get_current_usage()
        return (
            current_usage.agents < self.max_agents and
            current_usage.memory < self.max_memory and
            current_usage.cpu_percent < self.max_cpu_percent
        )
```

## 🔐 Security Architecture

### Authentication & Authorization

```python
class SecurityManager:
    """Handles authentication and authorization."""
    
    async def authenticate_request(self, token: str) -> Optional[User]:
        """Validate request token."""
        
    async def authorize_action(
        self, 
        user: User, 
        action: str, 
        resource: str
    ) -> bool:
        """Check if user can perform action on resource."""
```

### Security Features

- **JWT-based authentication** for API access
- **Role-based access control** (RBAC) for operations
- **Rate limiting** to prevent abuse
- **Input validation** using Pydantic schemas
- **Secrets management** with environment variables
- **Audit logging** for security events

## 📈 Scalability Architecture

### Horizontal Scaling

```mermaid
graph TB
    LB[Load Balancer] --> API1[API Server 1]
    LB --> API2[API Server 2]
    LB --> API3[API Server N]
    
    API1 --> Redis[(Redis Cluster)]
    API2 --> Redis
    API3 --> Redis
    
    API1 --> DB[(PostgreSQL)]
    API2 --> DB
    API3 --> DB
    
    API1 --> MCP[MCP Servers]
    API2 --> MCP
    API3 --> MCP
```

### Auto-Scaling Triggers

- **CPU utilization** > 80% for 5 minutes
- **Memory usage** > 85% for 5 minutes  
- **Queue depth** > 100 pending tasks
- **Response latency** > 2 seconds average

### Multi-Region Deployment

```yaml
regions:
  primary:
    name: "us-east-1"
    database: "primary"
    cache: "redis-cluster-east"
    
  secondary:
    name: "eu-west-1" 
    database: "read-replica"
    cache: "redis-cluster-eu"
    
  disaster_recovery:
    name: "us-west-2"
    database: "backup"
    cache: "redis-standalone"
```

## 🧪 Testing Architecture

### Test Pyramid

```mermaid
pyramid
    title Test Pyramid
    
    "E2E Tests" : 10
    "Integration Tests" : 30
    "Unit Tests" : 60
```

### Test Categories

1. **Unit Tests** (60%): Fast, isolated component tests
2. **Integration Tests** (30%): Component interaction tests
3. **End-to-End Tests** (10%): Full system workflow tests
4. **Property-Based Tests**: Hypothesis-driven edge case testing
5. **Performance Tests**: Load testing and benchmarking

### Test Infrastructure

```python
# Test fixtures and mocking
@pytest.fixture
async def agent_manager():
    """Provide mocked agent manager for tests."""
    manager = AgentManager(test_config, mock_session, mock_redis)
    await manager.initialize()
    yield manager
    await manager.shutdown()

# BDD test scenarios
Feature: Agent Management
  Scenario: Create and execute task
    Given I have an agent manager
    When I create a researcher agent
    And I execute a research task
    Then the task should complete successfully
```

## 📝 Extension Points

### Custom Agent Types

```python
class CustomAnalystAgent(BaseAgent):
    """Custom analyst with specialized capabilities."""
    
    async def _process_task(self, task: Dict[str, Any]) -> Dict[str, Any]:
        """Custom task processing logic."""
        if task["type"] == "custom_analysis":
            return await self._perform_custom_analysis(task["data"])
        return await super()._process_task(task)

# Register custom agent type
AgentFactory.register_agent_type("custom_analyst", CustomAnalystAgent)
```

### Custom MCP Tools

```python
@mcp_tool("custom_data_processor")
async def process_custom_data(parameters: Dict[str, Any]) -> MCPToolResult:
    """Custom data processing tool."""
    data = parameters.get("data")
    processed = await custom_processing_logic(data)
    return MCPToolResult(success=True, result={"processed_data": processed})
```

### Custom Swarm Topologies

```python
class CustomTopology(SwarmTopology):
    """Custom swarm coordination pattern."""
    
    async def distribute_task(
        self,
        task: SwarmTask,
        agents: List[Agent]
    ) -> List[SubTask]:
        """Custom task distribution logic."""
        return await self._custom_distribution_algorithm(task, agents)
```

This architecture provides a solid foundation for building scalable, maintainable AI agent orchestration systems while remaining extensible for future requirements.