diff --git a/README.md b/README.md new file mode 100644 index 00000000..5e775c61 --- /dev/null +++ b/README.md @@ -0,0 +1,306 @@ +# CleverAgents + +A powerful, reactive Agent Framework using RxPy streams for complex AI agent orchestration and message routing. + +[![PyPI Package](https://img.shields.io/pypi/v/cleveragents.svg)](https://pypi.org/project/cleveragents/) +[![Travis-CI Build Status](https://img.shields.io/travis/cleverthis/cleveragents.svg)](https://travis-ci.org/cleverthis/cleveragents) + +## ๐ŸŒŠ What is CleverAgents? + +CleverAgents is a **reactive agent orchestration framework** that combines the power of RxPy reactive streams with LangGraph stateful workflows to create sophisticated AI agent networks. It's designed for building complex, multi-agent systems that can handle real-time data processing, conversational AI, and workflow automation. + +### Key Features + +- **๐ŸŒŠ Reactive Architecture**: Built on RxPy streams for real-time data processing +- **๐Ÿค– Multiple Agent Types**: LLM agents, tool agents, composite agents, and more +- **๐Ÿ”„ Unified Routes**: Single configuration system for streams and graphs +- **๐Ÿง  Memory Management**: Conversation history and state persistence +- **๐Ÿ”— LangGraph Integration**: Stateful workflows with conditional logic +- **โšก Async Processing**: Full async/await support throughout +- **๐ŸŽฏ Template System**: Reusable agent and workflow templates + +## ๐Ÿ“ฆ Installation + +### Prerequisites + +- Python 3.9+ (recommended: Python 3.11+) +- pip package manager + +### Install from PyPI + +```bash +pip install cleveragents +``` + +### Install from Source + +```bash +git clone https://git.cleverthis.com/cleveragents/cleveragents-core +cd cleveragents +pip install -e . +``` + +### Dependencies + +CleverAgents automatically installs these dependencies: + +- `click` - Command-line interface +- `rx>=3.2.0` - Reactive extensions for Python +- `jinja2` - Template engine +- `pystache` - Mustache template support +- `pyyaml` - YAML configuration parsing +- `langchain-core>=0.3.0` - LangChain integration +- `langchain-openai>=0.2.0` - OpenAI integration +- `langchain-anthropic>=0.2.0` - Anthropic integration +- `langchain-google-genai>=2.0.0` - Google Gemini integration +- `aiohttp` - Async HTTP client + +## ๐Ÿš€ Quick Start + +### 1. Create a Configuration File + +Create a `config.yaml` file: + +```yaml +agents: + chat_agent: + type: llm + config: + provider: openai + model: gpt-4 + temperature: 0.7 + memory_enabled: true + +routes: + chat_stream: + type: stream + stream_type: cold + operators: + - type: map + params: + agent: chat_agent + publications: + - __output__ + +merges: + - sources: [__input__] + target: chat_stream +``` + +### 2. Set Up API Keys + +Configure your API keys via environment variables: + +```bash +export OPENAI_API_KEY="your-openai-key" +export ANTHROPIC_API_KEY="your-anthropic-key" +export GOOGLE_API_KEY="your-google-key" +``` + +Or include them directly in your configuration: + +```yaml +agents: + chat_agent: + type: llm + config: + provider: openai + model: gpt-4 + api_key: "your-openai-key" +``` + +### 3. Run CleverAgents + +**Interactive Mode:** +```bash +cleveragents interactive -c config.yaml +``` + +**Single-Shot Mode:** +```bash +cleveragents run -c config.yaml -p "Hello, how are you?" +``` + +## ๐Ÿ› ๏ธ Command Line Interface + +### Main Commands + +#### `cleveragents run` +Process a single prompt through the agent network. + +```bash +cleveragents run -c config.yaml -p "Your prompt here" +``` + +**Options:** +- `-c, --config`: Path to configuration file(s) (can be used multiple times) +- `-p, --prompt`: The prompt to send to the agent network +- `-o, --output`: Optional file to write output to +- `-v, --verbose`: Enable verbose output +- `-u, --unsafe`: Enable unsafe mode for code execution + +#### `cleveragents interactive` +Start an interactive chat session. + +```bash +cleveragents interactive -c config.yaml +``` + +**Options:** +- `-c, --config`: Path to configuration file(s) +- `-h, --history`: Optional file to load/save conversation history +- `-v, --verbose`: Enable verbose output +- `-u, --unsafe`: Enable unsafe mode + +#### `cleveragents generate-examples` +Generate example configuration files. + +```bash +cleveragents generate-examples -o ./my-examples +``` + +**Options:** +- `-o, --output`: Directory to write example files to (default: `./examples`) + +#### `cleveragents visualize` +Visualize the agent network. + +```bash +cleveragents visualize -c config.yaml -f mermaid +``` + +**Options:** +- `-c, --config`: Path to configuration file(s) +- `-o, --output`: Output file for the diagram +- `-f, --format`: Output format (`mermaid`, `dot`, `ascii`) + +### Interactive Session Commands + +When in interactive mode, you can use these commands: + +- `exit` - Exit the session +- `help` - Show available commands +- `/stream ` - Send message to specific stream +- `/graph ` - Execute a LangGraph with message + +## ๐Ÿ”ง API Key Configuration + +CleverAgents supports multiple LLM providers through LangChain. Configure API keys in two ways: + +### Environment Variables (Recommended) + +```bash +# OpenAI +export OPENAI_API_KEY="sk-your-key-here" + +# Anthropic +export ANTHROPIC_API_KEY="your-anthropic-key" + +# Google Gemini +export GOOGLE_API_KEY="your-google-key" +``` + +### In Configuration Files + +```yaml +agents: + my_agent: + type: llm + config: + provider: openai + model: gpt-4 + api_key: "sk-your-key-here" +``` + +### Supported Models + +**OpenAI:** +- GPT-4, GPT-4o, GPT-3.5-turbo +- o1-preview, o1-mini + +**Anthropic:** +- Claude-3.5-Sonnet, Claude-3-Opus, Claude-3-Haiku + +**Google:** +- Gemini-1.5-Pro, Gemini-1.5-Flash, Gemini-2.0-Flash + +## ๐Ÿ—๏ธ Architecture + +### Reactive Streams (RxPy) + +CleverAgents uses RxPy for reactive programming: + +- **Hot Streams**: Always active, replay last value +- **Cold Streams**: Start when subscribed +- **Replay Streams**: Replay all previous values +- **Operators**: map, filter, merge, split, buffer, throttle, debounce + +### Agent Types + +1. **LLM Agents**: Powered by large language models +2. **Tool Agents**: Execute functions and tools +3. **Composite Agents**: Combine multiple agents +4. **Chain Agents**: Sequential processing chains + +### Unified Routes System + +Single configuration system for different processing types: + +- **Stream Routes**: Reactive, stateless processing +- **Graph Routes**: Stateful workflows with conditional logic +- **Bridge Routes**: Dynamic type conversion + +## ๐Ÿ“š Examples + +### Basic Chat +```bash +cleveragents interactive -c examples/basic_chat.yaml +``` + +### Scientific Paper Writer +```bash +cleveragents interactive -c examples/scientific_paper_writer.yaml +``` + +### Multi-Agent Collaboration +```bash +cleveragents interactive -c examples/collaboration_reactive.yaml +``` + +## ๐Ÿงช Development + +### Running Tests + +```bash +# Run all tests +python -m behave tests/features + +# Run with coverage +python -m pytest --cov=cleveragents + +# Run with tox for multiple environments +tox +``` + +### Contributing + +1. Fork the repository +2. Create a feature branch +3. Add tests for new functionality +4. Ensure all tests pass +5. Submit a pull request + +## ๐Ÿ“„ License + +Apache License 2.0 + +## ๐Ÿ”— Links + +- **Documentation**: https://cleveragents.readthedocs.io/ +- **PyPI**: https://pypi.org/project/cleveragents/ +- **GitHub**: https://github.com/cleverthis/cleveragents +- **Issues**: https://github.com/cleverthis/cleveragents/issues + +## ๐Ÿค Support + +For questions, issues, or contributions, please visit our [GitHub repository](https://github.com/cleverthis/cleveragents) or contact us at [dev@cleverthis.com](mailto:dev@cleverthis.com). diff --git a/YAML_SYNTAX.md b/YAML_SYNTAX.md new file mode 100644 index 00000000..e88f2ad1 --- /dev/null +++ b/YAML_SYNTAX.md @@ -0,0 +1,596 @@ +# CleverAgents YAML Configuration Syntax + +This document provides comprehensive documentation for CleverAgents YAML configuration files, including all sections, syntax, and expected behaviors. + +## ๐Ÿ“‹ Table of Contents + +- [Configuration Structure](#configuration-structure) +- [Agents Section](#agents-section) +- [Routes Section](#routes-section) +- [Merges Section](#merges-section) +- [Splits Section](#splits-section) +- [Templates Section](#templates-section) +- [Context Section](#context-section) +- [Complete Examples](#complete-examples) +- [Best Practices](#best-practices) + +## ๐Ÿ—๏ธ Configuration Structure + +A CleverAgents configuration file follows this structure: + +```yaml +# Optional: Global configuration +cleveragents: + template_engine: JINJA2 # or MUSTACHE + +# Required: Agent definitions +agents: + agent_name: + type: agent_type + config: + # Agent-specific configuration + +# Required: Route definitions (unified streams and graphs) +routes: + route_name: + type: stream|graph|bridge + # Route-specific configuration + +# Optional: Stream operations +merges: + - sources: [source_streams] + target: target_stream + +splits: + - source: source_stream + targets: + target1: condition1 + target2: condition2 + +# Optional: Template definitions +templates: + agents: + template_name: + # Template definition + routes: + template_name: + # Template definition + +# Optional: Global context +context: + global: + variable_name: value +``` + +## ๐Ÿค– Agents Section + +The `agents` section defines AI agents that can process messages. + +### Agent Types + +#### LLM Agents +```yaml +agents: + chat_agent: + type: llm + config: + provider: openai|anthropic|google + model: gpt-4|claude-3-5-sonnet|gemini-1.5-pro + temperature: 0.7 + max_tokens: 1000 + system_prompt: "You are a helpful assistant" + memory_enabled: true + max_history: 10 + api_key: "optional-api-key" +``` + +**LLM Agent Configuration:** +- `provider`: LLM provider (openai, anthropic, google) +- `model`: Specific model name +- `temperature`: Creativity level (0.0-1.0) +- `max_tokens`: Maximum response length +- `system_prompt`: System message for the agent +- `memory_enabled`: Enable conversation memory +- `max_history`: Number of messages to remember +- `api_key`: Optional API key (can use environment variables) + +#### Tool Agents +```yaml +agents: + tool_agent: + type: tool + config: + tools: ["math", "http", "file", "json", "shell"] + safe_mode: true + timeout: 30 +``` + +**Tool Agent Configuration:** +- `tools`: List of available tools +- `safe_mode`: Enable safety restrictions +- `timeout`: Execution timeout in seconds + +#### Composite Agents +```yaml +agents: + composite_agent: + type: composite + config: + components: + agents: + - name: sub_agent1 + type: llm + graphs: + - name: sub_graph1 + type: graph + streams: + - name: sub_stream1 + type: stream + routing: + input: sub_agent1 + output: sub_stream1 +``` + +## ๐Ÿ›ค๏ธ Routes Section + +The `routes` section defines data flow through your agent network using a unified system. + +### Stream Routes + +```yaml +routes: + my_stream: + type: stream # REQUIRED + stream_type: cold|hot|replay + operators: + - type: map + params: + agent: agent_name + - type: filter + params: + condition: + type: content_contains + text: "keyword" + - type: debounce + params: + duration: 1.0 + - type: buffer + params: + count: 5 + - type: throttle + params: + duration: 2.0 + subscriptions: + - source_stream + publications: + - target_stream + - __output__ + agents: + - agent_name + initial_value: "default" + buffer_size: 10 +``` + +**Stream Types:** +- `cold`: Start processing when subscribed +- `hot`: Always active, replay last value +- `replay`: Replay all previous values + +**Operators:** +- `map`: Transform messages using agents or functions +- `filter`: Filter messages based on conditions +- `debounce`: Wait for quiet period before processing +- `buffer`: Collect messages before processing +- `throttle`: Limit processing rate +- `catch`: Handle errors +- `retry`: Retry failed operations + +### Graph Routes + +```yaml +routes: + my_graph: + type: graph # REQUIRED + entry_point: start + nodes: + process_node: + type: agent + agent: agent_name + conditional_node: + type: conditional + condition: + type: content_contains + text: "keyword" + function_node: + type: function + function: my_function + edges: + - source: start + target: process_node + - source: process_node + target: conditional_node + condition: + type: metadata_has + key: success + - source: conditional_node + target: end + checkpointing: true + checkpoint_dir: "./checkpoints" + enable_time_travel: true + parallel_execution: true + state_class: "my_module.MyState" +``` + +**Node Types:** +- `agent`: Process using an agent +- `conditional`: Conditional routing +- `function`: Custom function execution + +**Graph Features:** +- `checkpointing`: Save state for recovery +- `enable_time_travel`: Allow state rollback +- `parallel_execution`: Run nodes in parallel +- `state_class`: Custom state management + +### Bridge Routes + +```yaml +routes: + adaptive_route: + type: stream + # ... stream configuration ... + bridge: + upgrade_conditions: + needs_state: true + message_count: 5 + custom_predicate: "lambda msg, cfg: 'NEEDS_STATE' in msg.content" + downgrade_conditions: + idle_time: 300 + state_size: 2 + no_conditionals_used: true + state_extractor: "lambda msg: {'last_message': msg.content}" + state_flattener: "lambda state: state.data.get('summary', '')" + preserve_subscriptions: true + preserve_checkpointing: true +``` + +## ๐Ÿ”„ Merges Section + +Merge multiple streams into one: + +```yaml +merges: + - sources: [stream1, stream2, stream3] + target: combined_stream + - sources: [__input__] + target: main_processor +``` + +## ๐Ÿ”€ Splits Section + +Split one stream into multiple streams: + +```yaml +splits: + - source: input_stream + targets: + high_priority: + type: content_contains + text: "urgent" + normal_priority: + type: content_not_contains + text: "urgent" + error_stream: + type: metadata_has + key: error +``` + +**Split Conditions:** +- `content_contains`: Message contains text +- `content_not_contains`: Message doesn't contain text +- `metadata_has`: Message has metadata key +- `metadata_equals`: Metadata equals value +- `custom`: Custom condition function + +## ๐ŸŽจ Templates Section + +Define reusable templates: + +```yaml +templates: + agents: + basic_llm: + type: llm + config: + provider: openai + model: gpt-4 + temperature: 0.7 + system_prompt: "{{system_message}}" + + routes: + chat_stream: + type: stream + stream_type: cold + operators: + - type: map + params: + agent: "{{agent_name}}" + publications: + - __output__ + + graphs: + simple_workflow: + type: graph + nodes: + process: + type: agent + agent: "{{agent_name}}" + edges: + - source: start + target: process + - source: process + target: end + +# Use templates +agents: + my_agent: + template: basic_llm + params: + system_message: "You are a helpful assistant" + + my_chat: + template: chat_stream + params: + agent_name: my_agent +``` + +## ๐ŸŒ Context Section + +Global variables and configuration: + +```yaml +context: + global: + app_name: "My CleverAgents App" + debug_mode: true + max_retries: 3 + timeout: 30 + api_keys: + openai: "sk-your-key" + anthropic: "your-key" + user_preferences: + language: "en" + theme: "dark" +``` + +## ๐Ÿ“ Complete Examples + +### Basic Chat with Memory + +```yaml +agents: + chat_agent: + type: llm + config: + provider: openai + model: gpt-4 + temperature: 0.8 + memory_enabled: true + max_history: 10 + system_prompt: | + You are a helpful and friendly AI assistant with memory. + You can remember our conversation and refer back to previous topics. + +routes: + chat_stream: + type: stream + stream_type: cold + operators: + - type: map + params: + agent: chat_agent + publications: + - __output__ + +merges: + - sources: [__input__] + target: chat_stream + +context: + global: + conversation_mode: true + memory_enabled: true +``` + +### Multi-Agent Research Pipeline + +```yaml +agents: + researcher: + type: llm + config: + provider: openai + model: gpt-4 + system_prompt: "Research and gather information on topics" + + analyzer: + type: llm + config: + provider: openai + model: gpt-4 + system_prompt: "Analyze and synthesize research findings" + + writer: + type: llm + config: + provider: openai + model: gpt-4 + system_prompt: "Write comprehensive reports based on analysis" + +routes: + research_stream: + type: stream + stream_type: cold + operators: + - type: map + params: + agent: researcher + - type: buffer + params: + time: 2.0 + publications: + - analysis_stream + + analysis_stream: + type: stream + stream_type: cold + operators: + - type: map + params: + agent: analyzer + publications: + - writing_stream + + writing_stream: + type: stream + stream_type: cold + operators: + - type: map + params: + agent: writer + publications: + - __output__ + +merges: + - sources: [__input__] + target: research_stream +``` + +### Stateful Workflow with LangGraph + +```yaml +agents: + classifier: + type: llm + config: + provider: openai + model: gpt-3.5-turbo + system_prompt: "Classify the input as: question, command, or statement" + + question_handler: + type: llm + config: + provider: openai + model: gpt-4 + system_prompt: "Answer questions concisely and accurately" + + command_handler: + type: tool + config: + tools: ["execute", "search"] + +routes: + workflow_graph: + type: graph + entry_point: start + nodes: + classify: + type: agent + agent: classifier + handle_question: + type: agent + agent: question_handler + handle_command: + type: agent + agent: command_handler + edges: + - source: start + target: classify + - source: classify + target: handle_question + condition: + type: content_contains + text: "question" + - source: classify + target: handle_command + condition: + type: content_contains + text: "command" + - source: handle_question + target: end + - source: handle_command + target: end + checkpointing: true + + input_stream: + type: stream + stream_type: cold + operators: + - type: graph_execute + params: + graph: workflow_graph + publications: + - __output__ + +merges: + - sources: [__input__] + target: input_stream +``` + +## โœ… Best Practices + +### 1. Configuration Organization +- Use descriptive names for agents and routes +- Group related configurations together +- Use templates for reusable patterns +- Keep configurations modular + +### 2. Memory Management +- Enable memory for conversational agents +- Set appropriate `max_history` limits +- Use context variables for global state + +### 3. Error Handling +- Use `catch` operators for error handling +- Implement retry logic for unreliable operations +- Use splits to route errors to error handlers + +### 4. Performance +- Use `throttle` and `debounce` for rate limiting +- Implement `buffer` for batch processing +- Use `hot` streams for real-time updates + +### 5. Testing +- Start with simple configurations +- Test each component individually +- Use verbose mode for debugging +- Validate configurations before deployment + +## ๐Ÿšจ Common Issues + +### Configuration Errors +- Missing `type` field in routes (REQUIRED) +- Invalid agent types +- Circular dependencies in merges/splits +- Missing required configuration fields + +### Runtime Errors +- API key not configured +- Invalid model names +- Network connectivity issues +- Memory limits exceeded + +### Debugging Tips +- Use `--verbose` flag for detailed logging +- Check agent capabilities with `/help` in interactive mode +- Validate YAML syntax before running +- Test with simple configurations first + +## ๐Ÿ“š Additional Resources + +- [CleverAgents Documentation](https://cleveragents.readthedocs.io/) +- [RxPy Documentation](https://rxpy.readthedocs.io/) +- [LangGraph Documentation](https://langchain-ai.github.io/langgraph/) +- [YAML Syntax Reference](https://yaml.org/spec/1.2/spec.html) + +For more examples and advanced configurations, see the `examples/` directory in the CleverAgents repository.