Your Name
15 KiB
15 KiB
CleverClaude Architecture Overview
This document provides a comprehensive overview of CleverClaude's architecture, design principles, and system components.
🏗️ High-Level Architecture
CleverClaude follows a modern, microservices-inspired architecture built on Python's AsyncIO ecosystem. The system is designed for scalability, maintainability, and extensibility.
graph TB
CLI[CLI Interface] --> Core[Core Application]
WebUI[Web Interface] --> API[FastAPI Server]
API --> Core
Core --> AM[Agent Manager]
Core --> SC[Swarm Coordinator]
Core --> MCP[MCP Client]
Core --> CM[Configuration Manager]
AM --> Agents[Agent Pool]
SC --> Swarms[Swarm Pool]
MCP --> Tools[87+ MCP Tools]
Core --> DB[(Database)]
Core --> Redis[(Redis Cache)]
Core --> Storage[(File Storage)]
Agents --> Tasks[Task Queue]
Swarms --> Tasks
Tasks --> Results[Result Store]
🎯 Design Principles
1. Async-First Architecture
- Built on Python AsyncIO for high concurrency
- Non-blocking I/O operations throughout
- Event-driven task processing
- Efficient resource utilization
2. Modular Design
- Clear separation of concerns
- Dependency injection for loose coupling
- Plugin architecture for extensibility
- Interface-based abstractions
3. Type Safety
- Comprehensive type hints using Python 3.11+ features
- Pydantic models for data validation
- Runtime type checking in critical paths
- IDE support and early error detection
4. Configuration-Driven
- YAML-based configuration files
- Environment variable overrides
- Hot-reloading of configuration
- Environment-specific settings
5. Observability
- Structured logging with correlation IDs
- Comprehensive metrics collection
- Health checks and monitoring endpoints
- Distributed tracing support
🧩 Core Components
Core Application (cleverclaude.core.app)
The CleverClaudeApp class is the central orchestrator that initializes and coordinates all system components.
class CleverClaudeApp:
"""Main application orchestrator."""
def __init__(self, config_dir: Optional[Path] = None):
self.config_dir = config_dir or get_default_config_dir()
self.settings = load_settings(self.config_dir)
# Core components
self.agent_manager: Optional[AgentManager] = None
self.swarm_coordinator: Optional[SwarmCoordinator] = None
self.mcp_client: Optional[MCPClient] = None
self.api_server: Optional[APIServer] = None
async def initialize(self) -> None:
"""Initialize all components in dependency order."""
await self._initialize_database()
await self._initialize_redis()
await self._initialize_agents()
await self._initialize_swarms()
await self._initialize_mcp()
await self._initialize_api()
async def start(self) -> None:
"""Start all services."""
await self.api_server.start()
self.logger.info("CleverClaude started successfully")
Key Responsibilities:
- Component lifecycle management
- Dependency injection setup
- Configuration loading and validation
- Graceful shutdown handling
Agent Manager (cleverclaude.agents.manager)
Manages the lifecycle and execution of individual AI agents.
class AgentManager:
"""Manages agent lifecycle and task execution."""
async def create_agent(
self,
agent_type: AgentType,
name: str,
capabilities: Optional[Set[str]] = None,
**kwargs
) -> str:
"""Create a new agent instance."""
async def execute_task(
self,
task: Dict[str, Any],
agent_id: Optional[str] = None
) -> Dict[str, Any]:
"""Execute a task on an agent."""
async def destroy_agent(self, agent_id: str) -> None:
"""Remove an agent from the system."""
Key Features:
- Agent type factory (Researcher, Coder, Analyst, etc.)
- Task routing and load balancing
- Health monitoring and failure recovery
- Capability-based agent selection
- Circuit breaker pattern for fault tolerance
Swarm Coordinator (cleverclaude.coordination.swarm)
Orchestrates multi-agent coordination and swarm intelligence.
class SwarmCoordinator:
"""Coordinates agent swarms and task distribution."""
async def create_swarm(
self,
name: str,
topology: SwarmTopology,
max_agents: int = 50
) -> str:
"""Create a new agent swarm."""
async def submit_task(
self,
swarm_id: str,
task: SwarmTask
) -> str:
"""Submit a task to a swarm for execution."""
async def scale_swarm(
self,
swarm_id: str,
target_size: int
) -> None:
"""Scale swarm to target size."""
Supported Topologies:
- Mesh: Full connectivity, peer-to-peer coordination
- Hierarchical: Tree structure with coordinators and workers
- Star: Central coordinator with spoke workers
- Ring: Circular communication patterns
Key Features:
- Dynamic scaling based on workload
- Fault-tolerant task distribution
- Load balancing across agents
- Performance metrics and optimization
- Cross-swarm coordination
MCP Client (cleverclaude.mcp.client)
Implements the Model Context Protocol for external tool integration.
class MCPClient:
"""MCP (Model Context Protocol) client for tool execution."""
async def execute_tool(
self,
tool_name: str,
parameters: Dict[str, Any]
) -> MCPToolExecutionResult:
"""Execute an MCP tool with given parameters."""
async def get_available_tools(self) -> Dict[str, MCPToolInfo]:
"""Get list of available tools with metadata."""
async def batch_execute(
self,
requests: List[Dict[str, Any]]
) -> List[MCPToolExecutionResult]:
"""Execute multiple tools in parallel."""
Tool Categories:
- Swarm Management: 15+ tools for swarm operations
- Neural Operations: 20+ tools for AI model management
- Memory Management: 10+ tools for persistent storage
- Performance Monitoring: 15+ tools for metrics and analysis
- Workflow Automation: 12+ tools for task orchestration
- GitHub Integration: 8+ tools for repository management
- DAA Tools: 10+ tools for autonomous agents
- System Tools: 8+ tools for system operations
📊 Data Flow Architecture
Task Execution Flow
sequenceDiagram
participant Client
participant API
participant Core
participant AM as Agent Manager
participant Agent
participant SC as Swarm Coordinator
participant MCP
Client->>API: Submit Task Request
API->>Core: Route Request
alt Single Agent Task
Core->>AM: Execute Task
AM->>Agent: Assign Task
Agent->>Agent: Process Task
Agent->>AM: Return Result
AM->>Core: Task Complete
else Swarm Task
Core->>SC: Submit to Swarm
SC->>SC: Select Agents
SC->>AM: Distribute Subtasks
AM->>Agent: Execute Subtasks
Agent->>AM: Return Results
AM->>SC: Aggregate Results
SC->>Core: Swarm Task Complete
end
opt MCP Tool Usage
Agent->>MCP: Execute Tool
MCP->>MCP: Tool Processing
MCP->>Agent: Tool Result
end
Core->>API: Return Response
API->>Client: Task Result
Memory and State Management
graph LR
App[Application State] --> Memory[Memory Manager]
App --> Cache[Redis Cache]
App --> DB[PostgreSQL DB]
Memory --> Namespace[Namespaced Storage]
Memory --> TTL[TTL Management]
Memory --> Persistence[Persistent Memory]
Cache --> Session[Session Data]
Cache --> TaskQueue[Task Queues]
Cache --> Metrics[Real-time Metrics]
DB --> Config[Configuration]
DB --> History[Task History]
DB --> Analytics[Analytics Data]
🔧 Configuration Architecture
CleverClaude uses a hierarchical configuration system with multiple override levels:
# config.yaml
app:
name: "CleverClaude"
version: "2.0.0"
environment: "development"
debug: true
database:
url: "postgresql+asyncpg://user:pass@localhost/cleverclaude"
pool_size: 10
max_overflow: 20
redis:
url: "redis://localhost:6379/0"
connection_pool_size: 10
agents:
max_agents: 100
default_timeout: 300
health_check_interval: 30
supported_types:
- researcher
- coder
- analyst
- coordinator
- reviewer
- tester
swarm:
default_topology: "mesh"
max_swarm_size: 50
coordination_timeout: 60
scaling:
enabled: true
min_agents: 1
max_agents: 100
scale_up_threshold: 0.8
scale_down_threshold: 0.3
mcp:
servers:
- name: "claude-flow-server"
url: "http://localhost:8001/mcp"
enabled: true
- name: "neural-server"
url: "http://localhost:8002/mcp"
enabled: true
api:
host: "127.0.0.1"
port: 8000
docs_enabled: true
cors_enabled: true
monitoring:
metrics_enabled: true
log_level: "INFO"
log_format: "json"
tracing_enabled: false
Configuration Priority (highest to lowest):
- Command-line arguments
- Environment variables (
CLEVERCLAUDE_*) - Local configuration files
- Default values
🏃♂️ Performance Architecture
Concurrency Model
# AsyncIO-based concurrency
async def handle_concurrent_tasks():
"""Handle multiple tasks concurrently."""
tasks = [
process_task_async(task1),
process_task_async(task2),
process_task_async(task3)
]
results = await asyncio.gather(*tasks, return_exceptions=True)
return results
# Connection pooling
async def get_database_connection():
"""Get connection from pool."""
async with database_pool.acquire() as connection:
return await connection.fetch("SELECT * FROM agents")
Caching Strategy
- L1 Cache: In-memory Python dictionaries for hot data
- L2 Cache: Redis for session data and temporary results
- L3 Cache: Database query result caching
- CDN: Static asset delivery (in production)
Resource Management
class ResourceManager:
"""Manages system resources and limits."""
def __init__(self, config: ResourceConfig):
self.max_agents = config.max_agents
self.max_memory = config.max_memory_mb * 1024 * 1024
self.max_cpu_percent = config.max_cpu_percent
async def allocate_agent(self) -> bool:
"""Check if resources are available for new agent."""
current_usage = await self.get_current_usage()
return (
current_usage.agents < self.max_agents and
current_usage.memory < self.max_memory and
current_usage.cpu_percent < self.max_cpu_percent
)
🔐 Security Architecture
Authentication & Authorization
class SecurityManager:
"""Handles authentication and authorization."""
async def authenticate_request(self, token: str) -> Optional[User]:
"""Validate request token."""
async def authorize_action(
self,
user: User,
action: str,
resource: str
) -> bool:
"""Check if user can perform action on resource."""
Security Features
- JWT-based authentication for API access
- Role-based access control (RBAC) for operations
- Rate limiting to prevent abuse
- Input validation using Pydantic schemas
- Secrets management with environment variables
- Audit logging for security events
📈 Scalability Architecture
Horizontal Scaling
graph TB
LB[Load Balancer] --> API1[API Server 1]
LB --> API2[API Server 2]
LB --> API3[API Server N]
API1 --> Redis[(Redis Cluster)]
API2 --> Redis
API3 --> Redis
API1 --> DB[(PostgreSQL)]
API2 --> DB
API3 --> DB
API1 --> MCP[MCP Servers]
API2 --> MCP
API3 --> MCP
Auto-Scaling Triggers
- CPU utilization > 80% for 5 minutes
- Memory usage > 85% for 5 minutes
- Queue depth > 100 pending tasks
- Response latency > 2 seconds average
Multi-Region Deployment
regions:
primary:
name: "us-east-1"
database: "primary"
cache: "redis-cluster-east"
secondary:
name: "eu-west-1"
database: "read-replica"
cache: "redis-cluster-eu"
disaster_recovery:
name: "us-west-2"
database: "backup"
cache: "redis-standalone"
🧪 Testing Architecture
Test Pyramid
pyramid
title Test Pyramid
"E2E Tests" : 10
"Integration Tests" : 30
"Unit Tests" : 60
Test Categories
- Unit Tests (60%): Fast, isolated component tests
- Integration Tests (30%): Component interaction tests
- End-to-End Tests (10%): Full system workflow tests
- Property-Based Tests: Hypothesis-driven edge case testing
- Performance Tests: Load testing and benchmarking
Test Infrastructure
# Test fixtures and mocking
@pytest.fixture
async def agent_manager():
"""Provide mocked agent manager for tests."""
manager = AgentManager(test_config, mock_session, mock_redis)
await manager.initialize()
yield manager
await manager.shutdown()
# BDD test scenarios
Feature: Agent Management
Scenario: Create and execute task
Given I have an agent manager
When I create a researcher agent
And I execute a research task
Then the task should complete successfully
📝 Extension Points
Custom Agent Types
class CustomAnalystAgent(BaseAgent):
"""Custom analyst with specialized capabilities."""
async def _process_task(self, task: Dict[str, Any]) -> Dict[str, Any]:
"""Custom task processing logic."""
if task["type"] == "custom_analysis":
return await self._perform_custom_analysis(task["data"])
return await super()._process_task(task)
# Register custom agent type
AgentFactory.register_agent_type("custom_analyst", CustomAnalystAgent)
Custom MCP Tools
@mcp_tool("custom_data_processor")
async def process_custom_data(parameters: Dict[str, Any]) -> MCPToolResult:
"""Custom data processing tool."""
data = parameters.get("data")
processed = await custom_processing_logic(data)
return MCPToolResult(success=True, result={"processed_data": processed})
Custom Swarm Topologies
class CustomTopology(SwarmTopology):
"""Custom swarm coordination pattern."""
async def distribute_task(
self,
task: SwarmTask,
agents: List[Agent]
) -> List[SubTask]:
"""Custom task distribution logic."""
return await self._custom_distribution_algorithm(task, agents)
This architecture provides a solid foundation for building scalable, maintainable AI agent orchestration systems while remaining extensible for future requirements.