feat: implement unified routes system for streams and graphs
This commit is contained in:
+364
-14
@@ -22,6 +22,72 @@ CleverAgents is built on **RxPy reactive streams**, providing powerful stream pr
|
||||
- **Async/Await Support**: Native async processing throughout the pipeline
|
||||
- **Stream Composition**: Complex routing patterns with merge and split operations
|
||||
|
||||
🚀 Unified Routes System
|
||||
========================
|
||||
|
||||
CleverAgents provides a unified "routes" system that combines streams and graphs under a single configuration section. This provides a consistent interface for all data flow patterns.
|
||||
|
||||
**Key Benefits:**
|
||||
|
||||
- **Unified Interface**: Single "routes" section for all data flow
|
||||
- **Clear Type System**: Explicit type field makes intent clear
|
||||
- **Dynamic Adaptation**: Routes can convert between types at runtime
|
||||
- **Better Organization**: Related concepts in one place
|
||||
- **Complexity Guidance**: Built-in analyzer helps choose route type
|
||||
|
||||
**Route Configuration Format:**
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
routes:
|
||||
# Stream route
|
||||
my_stream:
|
||||
type: stream # REQUIRED field
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
agent: my_agent
|
||||
|
||||
# Graph route
|
||||
my_graph:
|
||||
type: graph # REQUIRED field
|
||||
nodes:
|
||||
process:
|
||||
type: agent
|
||||
agent: processor
|
||||
edges:
|
||||
- source: start
|
||||
target: process
|
||||
- source: process
|
||||
target: end
|
||||
|
||||
**Route Types:**
|
||||
|
||||
1. **Stream Routes**: For reactive, stateless processing
|
||||
2. **Graph Routes**: For stateful workflows with conditional logic
|
||||
3. **Bridge Routes**: For dynamic type conversion
|
||||
|
||||
**Dynamic Type Conversion:**
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
routes:
|
||||
adaptive_route:
|
||||
type: stream
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
agent: processor
|
||||
bridge:
|
||||
upgrade_conditions:
|
||||
needs_state: true
|
||||
message_count: 5
|
||||
downgrade_conditions:
|
||||
idle_time: 300
|
||||
|
||||
**Important Note**: The system does NOT maintain backward compatibility. The old ``streams:`` and ``graphs:`` sections are completely removed. Only the new ``routes:`` section is supported.
|
||||
|
||||
🔄 Why RxPy + LangGraph?
|
||||
========================
|
||||
|
||||
@@ -122,8 +188,9 @@ CleverAgents provides full LangGraph integration, allowing you to combine statef
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
graphs:
|
||||
routes:
|
||||
my_workflow:
|
||||
type: graph # Required field
|
||||
name: my_workflow
|
||||
entry_point: start
|
||||
checkpointing: true
|
||||
@@ -178,9 +245,10 @@ LangGraph state flows through the graph and can be:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
streams:
|
||||
routes:
|
||||
my_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: graph_execute
|
||||
params:
|
||||
@@ -341,9 +409,10 @@ Basic Usage
|
||||
model: gpt-4
|
||||
temperature: 0.7
|
||||
|
||||
streams:
|
||||
routes:
|
||||
chat_stream:
|
||||
type: cold
|
||||
type: stream # Required field
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -397,22 +466,27 @@ CleverAgents requires API keys for LLM providers. Configure them in two ways:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
streams:
|
||||
routes:
|
||||
cold_stream:
|
||||
type: cold # Starts when subscribed
|
||||
type: stream
|
||||
stream_type: cold # Starts when subscribed
|
||||
|
||||
hot_stream:
|
||||
type: hot # Always active, replays last value
|
||||
type: stream
|
||||
stream_type: hot # Always active, replays last value
|
||||
|
||||
replay_stream:
|
||||
type: replay # Replays all previous values
|
||||
type: stream
|
||||
stream_type: replay # Replays all previous values
|
||||
|
||||
**RxPy Operators**
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
streams:
|
||||
routes:
|
||||
processing_stream:
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -460,8 +534,10 @@ CleverAgents requires API keys for LLM providers. Configure them in two ways:
|
||||
.. code-block:: yaml
|
||||
|
||||
# Research → Analysis → Writing → Editing pipeline
|
||||
streams:
|
||||
routes:
|
||||
research_stream:
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -470,6 +546,8 @@ CleverAgents requires API keys for LLM providers. Configure them in two ways:
|
||||
- analysis_stream
|
||||
|
||||
analysis_stream:
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -481,9 +559,10 @@ CleverAgents requires API keys for LLM providers. Configure them in two ways:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
streams:
|
||||
routes:
|
||||
analytics_stream:
|
||||
type: hot
|
||||
type: stream
|
||||
stream_type: hot
|
||||
operators:
|
||||
- type: scan
|
||||
params:
|
||||
@@ -497,8 +576,10 @@ CleverAgents requires API keys for LLM providers. Configure them in two ways:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
streams:
|
||||
routes:
|
||||
robust_processing:
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -508,6 +589,275 @@ CleverAgents requires API keys for LLM providers. Configure them in two ways:
|
||||
params:
|
||||
count: 3
|
||||
|
||||
📊 Route Complexity Ladder
|
||||
==========================
|
||||
|
||||
This guide helps you choose the right route type for your use case. Routes in CleverAgents can be either **streams** (reactive, stateless) or **graphs** (stateful, conditional).
|
||||
|
||||
Quick Decision Guide
|
||||
--------------------
|
||||
|
||||
Ask yourself these questions:
|
||||
|
||||
1. **Do you need to maintain state between messages?** → Use a graph
|
||||
2. **Do you need conditional logic or branching?** → Use a graph
|
||||
3. **Is your processing linear and stateless?** → Use a stream
|
||||
4. **Do you need real-time reactive processing?** → Use a stream
|
||||
5. **Unsure or requirements might change?** → Use a stream with bridge configuration
|
||||
|
||||
Complexity Levels
|
||||
-----------------
|
||||
|
||||
**Level 1: Simple Stream**
|
||||
|
||||
Use when: Basic data transformation with a single agent
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
routes:
|
||||
simple_processor:
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
agent: my_agent
|
||||
|
||||
Good for:
|
||||
|
||||
- Simple chat interfaces
|
||||
- Basic text transformation
|
||||
- Single-step processing
|
||||
|
||||
**Level 2: Stream with Multiple Operators**
|
||||
|
||||
Use when: Multi-step processing pipeline without state
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
routes:
|
||||
pipeline:
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: debounce
|
||||
params:
|
||||
duration: 0.5
|
||||
- type: map
|
||||
params:
|
||||
agent: preprocessor
|
||||
- type: filter
|
||||
params:
|
||||
condition: valid_input
|
||||
- type: map
|
||||
params:
|
||||
agent: main_processor
|
||||
|
||||
Good for:
|
||||
|
||||
- Data preprocessing pipelines
|
||||
- Multi-agent workflows (sequential)
|
||||
- Stream filtering and transformation
|
||||
|
||||
**Level 3: Stream with Routing**
|
||||
|
||||
Use when: Need to split/merge data flows
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
routes:
|
||||
router_stream:
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
agent: classifier
|
||||
publications:
|
||||
- questions_stream
|
||||
- commands_stream
|
||||
|
||||
splits:
|
||||
- source: router_stream
|
||||
targets:
|
||||
questions: questions_stream
|
||||
commands: commands_stream
|
||||
|
||||
Good for:
|
||||
|
||||
- Content routing based on classification
|
||||
- Parallel processing paths
|
||||
- Fan-out patterns
|
||||
|
||||
**Level 4: Basic Graph**
|
||||
|
||||
Use when: Need conditional logic or simple state
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
routes:
|
||||
conditional_workflow:
|
||||
type: graph
|
||||
nodes:
|
||||
classify:
|
||||
type: agent
|
||||
agent: classifier
|
||||
|
||||
handle_a:
|
||||
type: agent
|
||||
agent: handler_a
|
||||
|
||||
handle_b:
|
||||
type: agent
|
||||
agent: handler_b
|
||||
|
||||
edges:
|
||||
- source: start
|
||||
target: classify
|
||||
- source: classify
|
||||
target: handle_a
|
||||
condition: "type == 'A'"
|
||||
- source: classify
|
||||
target: handle_b
|
||||
condition: "type == 'B'"
|
||||
|
||||
Good for:
|
||||
|
||||
- Workflows with decision points
|
||||
- Conditional processing
|
||||
- Simple state machines
|
||||
|
||||
**Level 5: Stateful Graph**
|
||||
|
||||
Use when: Need to maintain conversation or process state
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
routes:
|
||||
stateful_workflow:
|
||||
type: graph
|
||||
checkpointing: true
|
||||
state_class: "myapp.states.ConversationState"
|
||||
|
||||
nodes:
|
||||
update_context:
|
||||
type: function
|
||||
function: update_conversation_state
|
||||
|
||||
process_with_context:
|
||||
type: agent
|
||||
agent: contextual_processor
|
||||
|
||||
edges:
|
||||
- source: start
|
||||
target: update_context
|
||||
- source: update_context
|
||||
target: process_with_context
|
||||
|
||||
Good for:
|
||||
|
||||
- Conversational agents with memory
|
||||
- Multi-turn interactions
|
||||
- Complex state management
|
||||
|
||||
**Level 6: Graph with Persistence**
|
||||
|
||||
Use when: Need to save/restore state, handle failures
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
routes:
|
||||
persistent_workflow:
|
||||
type: graph
|
||||
checkpointing: true
|
||||
checkpoint_dir: "./checkpoints"
|
||||
enable_time_travel: true
|
||||
|
||||
nodes:
|
||||
# Complex node structure
|
||||
|
||||
edges:
|
||||
# Complex conditional routing
|
||||
|
||||
Good for:
|
||||
|
||||
- Long-running workflows
|
||||
- Fault-tolerant processing
|
||||
- Auditable state changes
|
||||
|
||||
Dynamic Type Conversion
|
||||
-----------------------
|
||||
|
||||
Use bridging when requirements might change:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
routes:
|
||||
adaptive_route:
|
||||
type: stream # Start simple
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
agent: processor
|
||||
|
||||
bridge:
|
||||
# Automatically upgrade to graph when needed
|
||||
upgrade_conditions:
|
||||
needs_state: true
|
||||
message_count: 10
|
||||
|
||||
# Downgrade back to stream when possible
|
||||
downgrade_conditions:
|
||||
idle_time: 300
|
||||
state_size: 1
|
||||
|
||||
Best Practices
|
||||
--------------
|
||||
|
||||
1. **Start Simple**: Begin with streams and upgrade to graphs only when needed
|
||||
2. **Use Bridges**: Configure bridge conditions for routes that might need to change
|
||||
3. **Monitor Complexity**: Use the RouteComplexityAnalyzer to track route complexity
|
||||
4. **Document Decisions**: Explain why you chose a particular route type
|
||||
|
||||
Performance Considerations
|
||||
--------------------------
|
||||
|
||||
**Streams:**
|
||||
|
||||
- ✅ Lower memory footprint
|
||||
- ✅ Better for high-throughput scenarios
|
||||
- ✅ Excellent for real-time processing
|
||||
- ❌ No built-in state management
|
||||
- ❌ Limited conditional logic
|
||||
|
||||
**Graphs:**
|
||||
|
||||
- ✅ Rich state management
|
||||
- ✅ Complex conditional logic
|
||||
- ✅ Checkpointing and recovery
|
||||
- ❌ Higher memory usage
|
||||
- ❌ More complex to debug
|
||||
|
||||
Migration Guide
|
||||
---------------
|
||||
|
||||
**From Streams to Graphs:**
|
||||
|
||||
1. Identify state requirements
|
||||
2. Define state class
|
||||
3. Convert operators to nodes
|
||||
4. Add conditional edges
|
||||
|
||||
**From Graphs to Streams:**
|
||||
|
||||
1. Ensure state can be flattened
|
||||
2. Convert nodes to operators
|
||||
3. Replace conditional edges with stream splits
|
||||
4. Test thoroughly
|
||||
|
||||
Choose the simplest route type that meets your requirements. Use streams for stateless, reactive processing and graphs for stateful, conditional workflows. Configure bridges for routes that might need to adapt over time.
|
||||
|
||||
📊 CLI Commands
|
||||
===============
|
||||
|
||||
|
||||
@@ -59,10 +59,12 @@ agents:
|
||||
If score < 7, respond with "NEEDS_IMPROVEMENT: [reason]"
|
||||
Otherwise respond with "APPROVED: [score]"
|
||||
|
||||
streams:
|
||||
# NEW: Unified routes section
|
||||
routes:
|
||||
# Input preprocessing with debouncing
|
||||
input_preprocessor:
|
||||
type: cold
|
||||
type: stream # REQUIRED: Must specify type
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: debounce
|
||||
params:
|
||||
@@ -83,7 +85,8 @@ streams:
|
||||
|
||||
# Classification stream with retry
|
||||
classification_stream:
|
||||
type: cold
|
||||
type: stream # REQUIRED: Must specify type
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
|
||||
@@ -12,10 +12,12 @@ agents:
|
||||
You are a helpful and friendly AI assistant.
|
||||
Respond naturally and conversationally.
|
||||
|
||||
streams:
|
||||
# Main chat processing stream
|
||||
# NEW: Unified routes section
|
||||
routes:
|
||||
# Main chat processing route
|
||||
chat_stream:
|
||||
type: cold
|
||||
type: stream # REQUIRED: Must specify type (stream or graph)
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
|
||||
@@ -26,10 +26,11 @@ agents:
|
||||
temperature: 0.5
|
||||
system_prompt: "Format the analysis into a clear, structured response."
|
||||
|
||||
# Define a processing graph
|
||||
graphs:
|
||||
# NEW: Unified routes section
|
||||
routes:
|
||||
# Define a processing graph
|
||||
analysis_graph:
|
||||
name: analysis_graph
|
||||
type: graph # REQUIRED: Must specify type
|
||||
nodes:
|
||||
analyze:
|
||||
type: agent
|
||||
@@ -47,11 +48,10 @@ graphs:
|
||||
- source: format_output
|
||||
target: end
|
||||
|
||||
# RxPy streams for preprocessing and postprocessing
|
||||
streams:
|
||||
# Preprocessing stream with RxPy operators
|
||||
input_preprocessor:
|
||||
type: cold
|
||||
type: stream # REQUIRED: Must specify type
|
||||
stream_type: cold
|
||||
operators:
|
||||
# Debounce rapid inputs
|
||||
- type: debounce
|
||||
@@ -76,7 +76,8 @@ streams:
|
||||
|
||||
# Stream that executes the LangGraph
|
||||
graph_trigger:
|
||||
type: cold
|
||||
type: stream # REQUIRED: Must specify type
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: graph_execute
|
||||
params:
|
||||
|
||||
@@ -37,9 +37,10 @@ agents:
|
||||
temperature: 0.5
|
||||
system_prompt: "Acknowledge statements and provide relevant comments."
|
||||
|
||||
graphs:
|
||||
# NEW: Unified routes section
|
||||
routes:
|
||||
conditional_flow:
|
||||
name: conditional_flow
|
||||
type: graph # REQUIRED: Must specify type
|
||||
entry_point: start
|
||||
|
||||
nodes:
|
||||
@@ -106,10 +107,10 @@ graphs:
|
||||
- source: handle_statement
|
||||
target: end
|
||||
|
||||
# Simple stream setup
|
||||
streams:
|
||||
# Simple stream setup
|
||||
input_handler:
|
||||
type: cold
|
||||
type: stream # REQUIRED: Must specify type
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: graph_execute
|
||||
params:
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
# Route Bridging Demo
|
||||
# Demonstrates dynamic conversion between stream and graph routes
|
||||
|
||||
agents:
|
||||
simple_processor:
|
||||
type: llm
|
||||
config:
|
||||
provider: openai
|
||||
model: gpt-3.5-turbo
|
||||
temperature: 0.7
|
||||
system_prompt: |
|
||||
Process user input. If the user asks about state or context,
|
||||
respond with "NEEDS_STATE: true" at the beginning of your response.
|
||||
|
||||
stateful_processor:
|
||||
type: llm
|
||||
config:
|
||||
provider: openai
|
||||
model: gpt-4
|
||||
temperature: 0.7
|
||||
system_prompt: |
|
||||
You are a stateful assistant that remembers conversation context.
|
||||
Use the provided state information to give contextual responses.
|
||||
|
||||
# NEW: Unified routes section with bridging
|
||||
routes:
|
||||
# Start with a simple stream that can upgrade to a graph
|
||||
adaptive_processor:
|
||||
type: stream # REQUIRED: Must specify type
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
agent: simple_processor
|
||||
publications:
|
||||
- __output__
|
||||
|
||||
# Bridge configuration for dynamic type conversion
|
||||
bridge:
|
||||
# Conditions to upgrade from stream to graph
|
||||
upgrade_conditions:
|
||||
needs_state: true # Upgrade when state is needed
|
||||
message_count: 5 # Or after 5 messages
|
||||
custom_predicate: |
|
||||
lambda msg, cfg: "NEEDS_STATE: true" in msg.content
|
||||
|
||||
# Conditions to downgrade from graph to stream
|
||||
downgrade_conditions:
|
||||
idle_time: 300 # Downgrade after 5 minutes of inactivity
|
||||
state_size: 2 # Or when state is minimal
|
||||
no_conditionals_used: true # Or if no conditional logic was used
|
||||
|
||||
# State management during conversion
|
||||
state_extractor: |
|
||||
lambda msg: {"last_message": msg.content, "timestamp": msg.timestamp}
|
||||
state_flattener: |
|
||||
lambda state: state.data.get("conversation_summary", "")
|
||||
|
||||
preserve_subscriptions: true
|
||||
preserve_checkpointing: true
|
||||
|
||||
# Alternative: Define as graph that can downgrade to stream
|
||||
stateful_processor_route:
|
||||
type: graph # REQUIRED: Must specify type
|
||||
entry_point: start
|
||||
checkpointing: true
|
||||
checkpoint_dir: "./checkpoints/stateful"
|
||||
|
||||
nodes:
|
||||
process_with_state:
|
||||
type: agent
|
||||
agent: stateful_processor
|
||||
|
||||
edges:
|
||||
- source: start
|
||||
target: process_with_state
|
||||
- source: process_with_state
|
||||
target: end
|
||||
|
||||
# Bridge for potential downgrade
|
||||
bridge:
|
||||
downgrade_conditions:
|
||||
idle_time: 600 # Downgrade after 10 minutes
|
||||
state_size: 1 # When state is almost empty
|
||||
|
||||
# Pure bridge route - defines only conversion logic
|
||||
conversion_bridge:
|
||||
type: bridge # Special type for conversion-only routes
|
||||
bridge:
|
||||
upgrade_conditions:
|
||||
complexity_threshold: 10
|
||||
downgrade_conditions:
|
||||
idle_time: 300
|
||||
|
||||
# Route input based on initial analysis
|
||||
merges:
|
||||
- sources: [__input__]
|
||||
target: adaptive_processor
|
||||
|
||||
context:
|
||||
global:
|
||||
app_name: "Adaptive Route Bridging Demo"
|
||||
description: |
|
||||
This example demonstrates how routes can dynamically convert between
|
||||
stream and graph types based on runtime conditions. The system starts
|
||||
with a simple stream and upgrades to a stateful graph when needed,
|
||||
or downgrades back to a stream when the complexity is no longer required.
|
||||
@@ -10,10 +10,11 @@ agents:
|
||||
temperature: 0.7
|
||||
system_prompt: "You are a helpful assistant."
|
||||
|
||||
# Define a simple LangGraph
|
||||
graphs:
|
||||
# NEW: Unified routes section
|
||||
routes:
|
||||
# Define the graph as a route
|
||||
simple_chat:
|
||||
name: simple_chat
|
||||
type: graph # REQUIRED: Must specify type (stream or graph)
|
||||
entry_point: start
|
||||
|
||||
nodes:
|
||||
@@ -27,12 +28,11 @@ graphs:
|
||||
target: process_input
|
||||
- source: process_input
|
||||
target: end
|
||||
|
||||
# Connect the graph to RxPy streams
|
||||
streams:
|
||||
|
||||
# Input stream that triggers the graph
|
||||
chat_input:
|
||||
type: cold
|
||||
type: stream # REQUIRED: Must specify type
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: graph_execute
|
||||
params:
|
||||
|
||||
+36
-22
@@ -554,15 +554,18 @@ def _generate_mermaid_diagram(config) -> str:
|
||||
for agent_name in config.agents:
|
||||
lines.append(f" {agent_name}[{agent_name}]")
|
||||
|
||||
# Add streams
|
||||
for stream_name, stream_config in config.streams.items():
|
||||
shape = "(" if stream_config.type.value == "hot" else "["
|
||||
end_shape = ")" if stream_config.type.value == "hot" else "]"
|
||||
lines.append(f" {stream_name}{shape}{stream_name}{end_shape}")
|
||||
# Add routes
|
||||
for route_name, route_config in config.routes.items():
|
||||
if route_config.type.value == "stream":
|
||||
shape = "(" if route_config.stream_type.value == "hot" else "["
|
||||
end_shape = ")" if route_config.stream_type.value == "hot" else "]"
|
||||
lines.append(f" {route_name}{shape}{route_name}{end_shape}")
|
||||
|
||||
# Add agent connections
|
||||
for agent_name in stream_config.agents:
|
||||
lines.append(f" {agent_name} --> {stream_name}")
|
||||
# Add agent connections
|
||||
for agent_name in route_config.agents:
|
||||
lines.append(f" {agent_name} --> {route_name}")
|
||||
elif route_config.type.value == "graph":
|
||||
lines.append(f" {route_name}[<{route_name}>]")
|
||||
|
||||
# Add merges
|
||||
for merge in config.merges:
|
||||
@@ -587,14 +590,18 @@ def _generate_dot_diagram(config) -> str:
|
||||
for agent_name in config.agents:
|
||||
lines.append(f" {agent_name} [shape=box, color=blue];")
|
||||
|
||||
# Add streams
|
||||
for stream_name, stream_config in config.streams.items():
|
||||
shape = "ellipse" if stream_config.type.value == "hot" else "box"
|
||||
lines.append(f" {stream_name} [shape={shape}, color=green];")
|
||||
# Add routes
|
||||
for route_name, route_config in config.routes.items():
|
||||
if route_config.type.value == "stream":
|
||||
shape = "ellipse" if route_config.stream_type.value == "hot" else "box"
|
||||
lines.append(f" {route_name} [shape={shape}, color=green];")
|
||||
elif route_config.type.value == "graph":
|
||||
lines.append(f" {route_name} [shape=hexagon, color=purple];")
|
||||
|
||||
# Add agent connections
|
||||
for agent_name in stream_config.agents:
|
||||
lines.append(f" {agent_name} -> {stream_name};")
|
||||
# Add agent connections for stream routes
|
||||
if route_config.type.value == "stream" and route_config.agents:
|
||||
for agent_name in route_config.agents:
|
||||
lines.append(f" {agent_name} -> {route_name};")
|
||||
|
||||
# Add merges and splits
|
||||
for merge in config.merges:
|
||||
@@ -619,13 +626,20 @@ def _generate_ascii_diagram(config) -> str:
|
||||
for agent_name, agent_config in config.agents.items():
|
||||
lines.append(f" [{agent_config.type}] {agent_name}")
|
||||
|
||||
lines.append("\nStreams:")
|
||||
for stream_name, stream_config in config.streams.items():
|
||||
lines.append(f" ({stream_config.type.value}) {stream_name}")
|
||||
if stream_config.agents:
|
||||
lines.append(f" <- Agents: {', '.join(stream_config.agents)}")
|
||||
if stream_config.operators:
|
||||
lines.append(f" <- Operators: {len(stream_config.operators)}")
|
||||
lines.append("\nRoutes:")
|
||||
for route_name, route_config in config.routes.items():
|
||||
if route_config.type.value == "stream":
|
||||
lines.append(f" [stream] ({route_config.stream_type.value}) {route_name}")
|
||||
if route_config.agents:
|
||||
lines.append(f" <- Agents: {', '.join(route_config.agents)}")
|
||||
if route_config.operators:
|
||||
lines.append(f" <- Operators: {len(route_config.operators)}")
|
||||
elif route_config.type.value == "graph":
|
||||
lines.append(f" [graph] {route_name}")
|
||||
if route_config.nodes:
|
||||
lines.append(f" <- Nodes: {len(route_config.nodes)}")
|
||||
if route_config.edges:
|
||||
lines.append(f" <- Edges: {len(route_config.edges)}")
|
||||
|
||||
if config.merges:
|
||||
lines.append("\nMerges:")
|
||||
|
||||
@@ -34,6 +34,9 @@ from cleveragents.core.exceptions import UnsafeConfigurationError
|
||||
from cleveragents.langgraph.bridge import RxPyLangGraphBridge
|
||||
from cleveragents.reactive.config_parser import ReactiveConfig
|
||||
from cleveragents.reactive.config_parser import ReactiveConfigParser
|
||||
from cleveragents.reactive.route import RouteConfig
|
||||
from cleveragents.reactive.route import RouteType
|
||||
from cleveragents.reactive.route_bridge import RouteBridge
|
||||
from cleveragents.reactive.stream_router import ReactiveStreamRouter
|
||||
from cleveragents.reactive.stream_router import StreamMessage
|
||||
from cleveragents.templates.base import TemplateType
|
||||
@@ -96,6 +99,9 @@ class ReactiveCleverAgentsApp:
|
||||
# Initialize LangGraph bridge
|
||||
self.langgraph_bridge = RxPyLangGraphBridge(self.stream_router)
|
||||
|
||||
# Initialize route bridge for dynamic type conversion
|
||||
self.route_bridge: Optional[RouteBridge] = None
|
||||
|
||||
# Initialize template registry
|
||||
# Check if we need enhanced registry for complex templates
|
||||
self.template_registry: Optional[
|
||||
@@ -178,15 +184,12 @@ class ReactiveCleverAgentsApp:
|
||||
# Create agents (including from templates)
|
||||
self._create_agents()
|
||||
|
||||
# Set up reactive streams
|
||||
self._setup_streams()
|
||||
# Set up routes (unified streams and graphs)
|
||||
self._setup_routes()
|
||||
|
||||
# Set up stream operations
|
||||
self._setup_stream_operations()
|
||||
|
||||
# Set up LangGraphs
|
||||
self._setup_langgraphs()
|
||||
|
||||
# Set up hybrid pipelines
|
||||
self._setup_pipelines()
|
||||
|
||||
@@ -498,43 +501,102 @@ class ReactiveCleverAgentsApp:
|
||||
self.stream_router.register_agent(agent_name, agent)
|
||||
self.logger.debug(f"Created agent: {agent_name}")
|
||||
|
||||
def _setup_streams(self) -> None:
|
||||
"""Set up all configured streams."""
|
||||
if not self.config:
|
||||
def _setup_routes(self) -> None:
|
||||
"""Set up all configured routes (unified streams and graphs)."""
|
||||
if not self.config or not self.config.routes:
|
||||
return
|
||||
|
||||
for stream_name, stream_config in self.config.streams.items():
|
||||
# Initialize route bridge if not already done
|
||||
if not self.route_bridge:
|
||||
self.route_bridge = RouteBridge(
|
||||
self.stream_router,
|
||||
self.agents,
|
||||
self.scheduler,
|
||||
)
|
||||
|
||||
# Process each route based on its type
|
||||
for route_name, route_config in self.config.routes.items():
|
||||
# Check if this is a template instance
|
||||
if (
|
||||
hasattr(stream_config, "template_config")
|
||||
and stream_config.template_config
|
||||
hasattr(route_config, "template_config")
|
||||
and route_config.template_config
|
||||
):
|
||||
# Instantiate from template
|
||||
template_config = stream_config.template_config
|
||||
template_config = route_config.template_config
|
||||
if self.template_registry and hasattr(
|
||||
self.template_registry, "instantiate_from_config"
|
||||
):
|
||||
stream_def = self.template_registry.instantiate_from_config(
|
||||
route_def = self.template_registry.instantiate_from_config(
|
||||
template_config
|
||||
)
|
||||
else:
|
||||
stream_def = template_config
|
||||
route_def = template_config
|
||||
|
||||
# Update stream config from template
|
||||
from cleveragents.reactive.stream_router import StreamConfig
|
||||
from cleveragents.reactive.stream_router import StreamType
|
||||
# Update route config from template
|
||||
route_type_str = route_def.get("type", "stream")
|
||||
route_config.type = RouteType(route_type_str)
|
||||
|
||||
stream_config = StreamConfig(
|
||||
name=stream_name,
|
||||
type=StreamType(stream_def.get("type", "cold")),
|
||||
operators=stream_def.get("operators", []),
|
||||
subscriptions=stream_def.get("subscriptions", []),
|
||||
publications=stream_def.get("publications", []),
|
||||
agents=stream_def.get("agents", []),
|
||||
if route_config.type == RouteType.STREAM:
|
||||
# Update stream-specific fields
|
||||
from cleveragents.reactive.stream_router import StreamType
|
||||
|
||||
route_config.stream_type = StreamType(
|
||||
route_def.get("stream_type", "cold")
|
||||
)
|
||||
route_config.operators = route_def.get("operators", [])
|
||||
route_config.subscriptions = route_def.get("subscriptions", [])
|
||||
route_config.publications = route_def.get("publications", [])
|
||||
route_config.agents = route_def.get("agents", [])
|
||||
elif route_config.type == RouteType.GRAPH:
|
||||
# Update graph-specific fields
|
||||
route_config.nodes = route_def.get("nodes", {})
|
||||
route_config.edges = route_def.get("edges", [])
|
||||
route_config.entry_point = route_def.get("entry_point", "start")
|
||||
route_config.checkpointing = route_def.get("checkpointing", False)
|
||||
|
||||
# Create the route based on type
|
||||
if route_config.type == RouteType.STREAM:
|
||||
# Create as stream
|
||||
stream_config = route_config.to_stream_config()
|
||||
self.stream_router.create_stream(stream_config)
|
||||
self.logger.debug(f"Created stream route: {route_name}")
|
||||
|
||||
elif route_config.type == RouteType.GRAPH:
|
||||
# Create as graph
|
||||
graph_config = route_config.to_graph_config()
|
||||
|
||||
# Resolve state class if specified
|
||||
if route_config.state_class:
|
||||
try:
|
||||
module_path, class_name = route_config.state_class.rsplit(
|
||||
".", 1
|
||||
)
|
||||
module = __import__(module_path, fromlist=[class_name])
|
||||
graph_config.state_class = getattr(module, class_name)
|
||||
except (ValueError, ImportError, AttributeError) as e:
|
||||
self.logger.warning(
|
||||
f"Failed to load state class '{route_config.state_class}': {e}"
|
||||
)
|
||||
|
||||
# Create the graph
|
||||
from cleveragents.langgraph.graph import LangGraph
|
||||
|
||||
graph = LangGraph(
|
||||
config=graph_config,
|
||||
agents=self.agents,
|
||||
stream_router=self.stream_router,
|
||||
scheduler=self.scheduler,
|
||||
)
|
||||
|
||||
self.stream_router.create_stream(stream_config)
|
||||
self.logger.debug(f"Created stream: {stream_name}")
|
||||
# Store graph in bridge
|
||||
self.langgraph_bridge.graphs[graph.name] = graph
|
||||
self.logger.debug(f"Created graph route: {route_name}")
|
||||
|
||||
elif route_config.type == RouteType.BRIDGE:
|
||||
# Bridge routes are special - they don't create anything immediately
|
||||
self.logger.debug(f"Registered bridge route: {route_name}")
|
||||
|
||||
# Removed _setup_streams - use routes instead
|
||||
|
||||
def _setup_stream_operations(self) -> None:
|
||||
"""Set up stream merge and split operations."""
|
||||
@@ -558,51 +620,22 @@ class ReactiveCleverAgentsApp:
|
||||
self.logger.debug(f"Split stream {source} into {list(targets.keys())}")
|
||||
|
||||
# Re-setup subscriptions after all merges/splits to ensure proper connections
|
||||
for stream_name, stream_config in self.config.streams.items():
|
||||
# Only check routes
|
||||
all_stream_names = set()
|
||||
|
||||
# Add stream routes
|
||||
for route_name, route_config in self.config.routes.items():
|
||||
if route_config.type == RouteType.STREAM:
|
||||
all_stream_names.add(route_name)
|
||||
|
||||
# Re-setup subscriptions
|
||||
for stream_name in all_stream_names:
|
||||
if stream_name in self.stream_router.stream_configs:
|
||||
self.stream_router._setup_subscriptions(
|
||||
self.stream_router.stream_configs[stream_name]
|
||||
)
|
||||
|
||||
def _setup_langgraphs(self) -> None:
|
||||
"""Set up all configured LangGraphs."""
|
||||
if not self.config:
|
||||
return
|
||||
|
||||
for graph_name, graph_config in self.config.graphs.items():
|
||||
# Convert config to dict format for bridge
|
||||
config_dict = {
|
||||
"name": graph_config.name,
|
||||
"nodes": graph_config.nodes,
|
||||
"edges": graph_config.edges,
|
||||
"entry_point": graph_config.entry_point,
|
||||
"checkpointing": graph_config.checkpointing,
|
||||
"checkpoint_dir": graph_config.checkpoint_dir,
|
||||
"enable_time_travel": graph_config.enable_time_travel,
|
||||
"parallel_execution": graph_config.parallel_execution,
|
||||
"state_class": graph_config.state_class,
|
||||
"metadata": graph_config.metadata,
|
||||
}
|
||||
|
||||
# Check if this is a template instance
|
||||
if hasattr(graph_config, "template_config"):
|
||||
# Instantiate from template
|
||||
template_config = graph_config.template_config
|
||||
if (
|
||||
self.template_registry
|
||||
and template_config
|
||||
and hasattr(self.template_registry, "instantiate_from_config")
|
||||
):
|
||||
graph_def = self.template_registry.instantiate_from_config(
|
||||
template_config
|
||||
)
|
||||
else:
|
||||
graph_def = template_config or {}
|
||||
config_dict.update(graph_def)
|
||||
|
||||
# Create graph through bridge
|
||||
graph = self.langgraph_bridge.create_graph_from_config(config_dict)
|
||||
self.logger.debug(f"Created LangGraph: {graph_name}")
|
||||
# Removed _setup_langgraphs - use routes instead
|
||||
|
||||
def _setup_pipelines(self) -> None:
|
||||
"""Set up all configured hybrid pipelines."""
|
||||
@@ -658,10 +691,26 @@ class ReactiveCleverAgentsApp:
|
||||
return
|
||||
|
||||
graph_name, message = parts
|
||||
# Check if it's a graph route
|
||||
route = self.config.routes.get(graph_name) if self.config else None
|
||||
if not route or route.type != RouteType.GRAPH:
|
||||
print(f"Graph route '{graph_name}' not found")
|
||||
available_graphs = (
|
||||
[
|
||||
name
|
||||
for name, r in self.config.routes.items()
|
||||
if r.type == RouteType.GRAPH
|
||||
]
|
||||
if self.config
|
||||
else []
|
||||
)
|
||||
if available_graphs:
|
||||
print(f"Available graph routes: {', '.join(available_graphs)}")
|
||||
return
|
||||
|
||||
graph = self.langgraph_bridge.get_graph(graph_name)
|
||||
if not graph:
|
||||
print(f"Graph '{graph_name}' not found")
|
||||
print(f"Available graphs: {', '.join(self.langgraph_bridge.list_graphs())}")
|
||||
print(f"Graph '{graph_name}' not initialized")
|
||||
return
|
||||
|
||||
# Execute graph
|
||||
@@ -695,20 +744,25 @@ class ReactiveCleverAgentsApp:
|
||||
for agent_name in self.agents:
|
||||
lines.append(f" {agent_name}[Agent: {agent_name}]")
|
||||
|
||||
# Add streams
|
||||
# Add routes
|
||||
if self.config:
|
||||
for stream_name, stream_config in self.config.streams.items():
|
||||
lines.append(f" {stream_name}{{Stream: {stream_name}}}")
|
||||
for route_name, route_config in self.config.routes.items():
|
||||
if route_config.type == RouteType.STREAM:
|
||||
lines.append(f" {route_name}{{Stream: {route_name}}}")
|
||||
|
||||
# Show operators
|
||||
for op in stream_config.operators:
|
||||
if op.get("type") == "map" and "agent" in op.get("params", {}):
|
||||
agent = op["params"]["agent"]
|
||||
lines.append(f" {stream_name} --> {agent}")
|
||||
# Show operators
|
||||
for op in route_config.operators:
|
||||
if op.get("type") == "map" and "agent" in op.get(
|
||||
"params", {}
|
||||
):
|
||||
agent = op["params"]["agent"]
|
||||
lines.append(f" {route_name} --> {agent}")
|
||||
|
||||
# Show publications
|
||||
for pub in stream_config.publications:
|
||||
lines.append(f" {stream_name} --> {pub}")
|
||||
# Show publications
|
||||
for pub in route_config.publications:
|
||||
lines.append(f" {route_name} --> {pub}")
|
||||
elif route_config.type == RouteType.GRAPH:
|
||||
lines.append(f" {route_name}{{Graph: {route_name}}}")
|
||||
|
||||
# Add merges
|
||||
for merge in self.config.merges:
|
||||
|
||||
@@ -17,6 +17,9 @@ from typing import Optional
|
||||
import yaml # type: ignore[import-untyped]
|
||||
|
||||
from cleveragents.core.exceptions import ConfigurationError
|
||||
from cleveragents.reactive.route import BridgeConfig
|
||||
from cleveragents.reactive.route import RouteConfig
|
||||
from cleveragents.reactive.route import RouteType
|
||||
from cleveragents.reactive.stream_router import StreamConfig
|
||||
from cleveragents.reactive.stream_router import StreamType
|
||||
from cleveragents.templates.yaml_template_engine import YAMLTemplateEngine
|
||||
@@ -62,10 +65,9 @@ class ReactiveConfig:
|
||||
"""Complete reactive configuration."""
|
||||
|
||||
agents: Dict[str, AgentConfig] = field(default_factory=dict)
|
||||
streams: Dict[str, StreamConfig] = field(default_factory=dict)
|
||||
routes: Dict[str, RouteConfig] = field(default_factory=dict) # Unified routes
|
||||
merges: List[Dict[str, Any]] = field(default_factory=list)
|
||||
splits: List[Dict[str, Any]] = field(default_factory=list)
|
||||
graphs: Dict[str, LangGraphConfig] = field(default_factory=dict)
|
||||
pipelines: Dict[str, HybridPipelineConfig] = field(default_factory=dict)
|
||||
templates: Dict[str, Dict[str, Any]] = field(
|
||||
default_factory=dict
|
||||
@@ -168,68 +170,52 @@ class ReactiveConfigParser:
|
||||
config=agent_data.get("config", {}),
|
||||
)
|
||||
|
||||
# Parse streams (both direct and from instances)
|
||||
streams_config = config_dict.get("streams", {})
|
||||
for name, stream_data in streams_config.items():
|
||||
if "template" in stream_data or "stream_template" in stream_data:
|
||||
# This is a template instance, store raw for later instantiation
|
||||
reactive_config.streams[name] = StreamConfig(
|
||||
name=name,
|
||||
type=StreamType.COLD, # Default, will be overridden
|
||||
operators=[],
|
||||
subscriptions=[],
|
||||
publications=[],
|
||||
agents=[],
|
||||
# Parse routes (unified system)
|
||||
routes_config = config_dict.get("routes", {})
|
||||
for name, route_data in routes_config.items():
|
||||
# Ensure type field is present and valid
|
||||
if "type" not in route_data:
|
||||
raise ConfigurationError(
|
||||
f"Route '{name}' must specify a 'type' field (stream, graph, or bridge)"
|
||||
)
|
||||
# Store the template reference in metadata
|
||||
reactive_config.streams[name].template_config = stream_data
|
||||
else:
|
||||
# Direct stream definition
|
||||
stream_type = StreamType(stream_data.get("type", "cold"))
|
||||
operators = stream_data.get("operators", [])
|
||||
subscriptions = stream_data.get("subscriptions", [])
|
||||
publications = stream_data.get("publications", [])
|
||||
agents = stream_data.get("agents", [])
|
||||
|
||||
reactive_config.streams[name] = StreamConfig(
|
||||
name=name,
|
||||
type=stream_type,
|
||||
operators=operators,
|
||||
subscriptions=subscriptions,
|
||||
publications=publications,
|
||||
agents=agents,
|
||||
route_type_str = route_data["type"].lower()
|
||||
try:
|
||||
route_type = RouteType(route_type_str)
|
||||
except ValueError:
|
||||
raise ConfigurationError(
|
||||
f"Route '{name}' has invalid type '{route_type_str}'. "
|
||||
f"Must be one of: stream, graph, bridge"
|
||||
)
|
||||
|
||||
# Handle template instances
|
||||
if "template" in route_data or "route_template" in route_data:
|
||||
# This is a template instance, store raw for later instantiation
|
||||
reactive_config.routes[name] = RouteConfig(
|
||||
name=name,
|
||||
type=route_type,
|
||||
template_config=route_data,
|
||||
)
|
||||
else:
|
||||
# Parse based on route type
|
||||
if route_type == RouteType.STREAM:
|
||||
reactive_config.routes[name] = self._parse_stream_route(
|
||||
name, route_data
|
||||
)
|
||||
elif route_type == RouteType.GRAPH:
|
||||
reactive_config.routes[name] = self._parse_graph_route(
|
||||
name, route_data
|
||||
)
|
||||
elif route_type == RouteType.BRIDGE:
|
||||
reactive_config.routes[name] = self._parse_bridge_route(
|
||||
name, route_data
|
||||
)
|
||||
|
||||
# Parse stream operations
|
||||
reactive_config.merges = config_dict.get("merges", [])
|
||||
reactive_config.splits = config_dict.get("splits", [])
|
||||
|
||||
# Parse LangGraphs (both direct and from instances)
|
||||
graphs_config = config_dict.get("graphs", {})
|
||||
for name, graph_data in graphs_config.items():
|
||||
if "template" in graph_data or "graph_template" in graph_data:
|
||||
# This is a template instance, store raw for later instantiation
|
||||
reactive_config.graphs[name] = LangGraphConfig(
|
||||
name=name,
|
||||
nodes={},
|
||||
edges=[],
|
||||
entry_point="start",
|
||||
template_config=graph_data, # Store template reference
|
||||
)
|
||||
else:
|
||||
# Direct graph definition
|
||||
reactive_config.graphs[name] = LangGraphConfig(
|
||||
name=name,
|
||||
nodes=graph_data.get("nodes", {}),
|
||||
edges=graph_data.get("edges", []),
|
||||
entry_point=graph_data.get("entry_point", "start"),
|
||||
checkpointing=graph_data.get("checkpointing", False),
|
||||
checkpoint_dir=graph_data.get("checkpoint_dir"),
|
||||
enable_time_travel=graph_data.get("enable_time_travel", False),
|
||||
parallel_execution=graph_data.get("parallel_execution", True),
|
||||
state_class=graph_data.get("state_class"),
|
||||
metadata=graph_data.get("metadata", {}),
|
||||
)
|
||||
# Removed legacy graphs parsing - use routes instead
|
||||
|
||||
# Parse hybrid pipelines
|
||||
pipelines_config = config_dict.get("pipelines", {})
|
||||
@@ -252,36 +238,126 @@ class ReactiveConfigParser:
|
||||
self._validate_config(reactive_config)
|
||||
return reactive_config
|
||||
|
||||
def _parse_stream_route(self, name: str, route_data: Dict[str, Any]) -> RouteConfig:
|
||||
"""Parse a stream-type route."""
|
||||
stream_type = StreamType(route_data.get("stream_type", "cold"))
|
||||
|
||||
# Parse bridge config if present
|
||||
bridge_config = None
|
||||
if "bridge" in route_data:
|
||||
bridge_data = route_data["bridge"]
|
||||
bridge_config = BridgeConfig(
|
||||
upgrade_conditions=bridge_data.get("upgrade_conditions", {}),
|
||||
downgrade_conditions=bridge_data.get("downgrade_conditions", {}),
|
||||
state_extractor=bridge_data.get("state_extractor"),
|
||||
state_flattener=bridge_data.get("state_flattener"),
|
||||
preserve_subscriptions=bridge_data.get("preserve_subscriptions", True),
|
||||
preserve_checkpointing=bridge_data.get("preserve_checkpointing", True),
|
||||
)
|
||||
|
||||
return RouteConfig(
|
||||
name=name,
|
||||
type=RouteType.STREAM,
|
||||
stream_type=stream_type,
|
||||
operators=route_data.get("operators", []),
|
||||
subscriptions=route_data.get("subscriptions", []),
|
||||
publications=route_data.get("publications", []),
|
||||
agents=route_data.get("agents", []),
|
||||
initial_value=route_data.get("initial_value"),
|
||||
buffer_size=route_data.get("buffer_size", 1),
|
||||
bridge=bridge_config,
|
||||
metadata=route_data.get("metadata", {}),
|
||||
)
|
||||
|
||||
def _parse_graph_route(self, name: str, route_data: Dict[str, Any]) -> RouteConfig:
|
||||
"""Parse a graph-type route."""
|
||||
# Parse bridge config if present
|
||||
bridge_config = None
|
||||
if "bridge" in route_data:
|
||||
bridge_data = route_data["bridge"]
|
||||
bridge_config = BridgeConfig(
|
||||
upgrade_conditions=bridge_data.get("upgrade_conditions", {}),
|
||||
downgrade_conditions=bridge_data.get("downgrade_conditions", {}),
|
||||
state_extractor=bridge_data.get("state_extractor"),
|
||||
state_flattener=bridge_data.get("state_flattener"),
|
||||
preserve_subscriptions=bridge_data.get("preserve_subscriptions", True),
|
||||
preserve_checkpointing=bridge_data.get("preserve_checkpointing", True),
|
||||
)
|
||||
|
||||
return RouteConfig(
|
||||
name=name,
|
||||
type=RouteType.GRAPH,
|
||||
nodes=route_data.get("nodes", {}),
|
||||
edges=route_data.get("edges", []),
|
||||
entry_point=route_data.get("entry_point", "start"),
|
||||
checkpointing=route_data.get("checkpointing", False),
|
||||
checkpoint_dir=route_data.get("checkpoint_dir"),
|
||||
enable_time_travel=route_data.get("enable_time_travel", False),
|
||||
parallel_execution=route_data.get("parallel_execution", True),
|
||||
state_class=route_data.get("state_class"),
|
||||
bridge=bridge_config,
|
||||
metadata=route_data.get("metadata", {}),
|
||||
)
|
||||
|
||||
def _parse_bridge_route(self, name: str, route_data: Dict[str, Any]) -> RouteConfig:
|
||||
"""Parse a bridge-type route."""
|
||||
# Bridge routes are special - they define conversion logic
|
||||
bridge_config = BridgeConfig(
|
||||
upgrade_conditions=route_data.get("upgrade_conditions", {}),
|
||||
downgrade_conditions=route_data.get("downgrade_conditions", {}),
|
||||
state_extractor=route_data.get("state_extractor"),
|
||||
state_flattener=route_data.get("state_flattener"),
|
||||
preserve_subscriptions=route_data.get("preserve_subscriptions", True),
|
||||
preserve_checkpointing=route_data.get("preserve_checkpointing", True),
|
||||
)
|
||||
|
||||
return RouteConfig(
|
||||
name=name,
|
||||
type=RouteType.BRIDGE,
|
||||
bridge=bridge_config,
|
||||
metadata=route_data.get("metadata", {}),
|
||||
)
|
||||
|
||||
def _validate_config(self, config: ReactiveConfig):
|
||||
"""Validate the reactive configuration."""
|
||||
# Validate stream references
|
||||
all_stream_names = set(config.streams.keys()) | {
|
||||
# Collect all route names
|
||||
all_route_names = set(config.routes.keys()) | {
|
||||
"__input__",
|
||||
"__output__",
|
||||
"__error__",
|
||||
}
|
||||
|
||||
for stream_name, stream_config in config.streams.items():
|
||||
# Check subscription references
|
||||
for sub in stream_config.subscriptions:
|
||||
if sub not in all_stream_names:
|
||||
raise ConfigurationError(
|
||||
f"Stream '{stream_name}' references unknown subscription '{sub}'"
|
||||
)
|
||||
# Validate routes
|
||||
for route_name, route_config in config.routes.items():
|
||||
if route_config.type == RouteType.STREAM:
|
||||
# Validate stream-type route
|
||||
for sub in route_config.subscriptions:
|
||||
if sub not in all_route_names:
|
||||
raise ConfigurationError(
|
||||
f"Route '{route_name}' references unknown subscription '{sub}'"
|
||||
)
|
||||
|
||||
# Check publication references
|
||||
for pub in stream_config.publications:
|
||||
if pub not in all_stream_names:
|
||||
raise ConfigurationError(
|
||||
f"Stream '{stream_name}' references unknown publication '{pub}'"
|
||||
)
|
||||
for pub in route_config.publications:
|
||||
if pub not in all_route_names:
|
||||
raise ConfigurationError(
|
||||
f"Route '{route_name}' references unknown publication '{pub}'"
|
||||
)
|
||||
|
||||
# Check agent references
|
||||
for agent_name in stream_config.agents:
|
||||
if agent_name not in config.agents:
|
||||
raise ConfigurationError(
|
||||
f"Stream '{stream_name}' references unknown agent '{agent_name}'"
|
||||
)
|
||||
for agent_name in route_config.agents:
|
||||
if agent_name not in config.agents:
|
||||
raise ConfigurationError(
|
||||
f"Route '{route_name}' references unknown agent '{agent_name}'"
|
||||
)
|
||||
|
||||
elif route_config.type == RouteType.GRAPH:
|
||||
# Validate graph-type route
|
||||
for node_name, node_data in route_config.nodes.items():
|
||||
if "agent" in node_data and node_data["agent"] not in config.agents:
|
||||
raise ConfigurationError(
|
||||
f"Route '{route_name}' node '{node_name}' references unknown agent '{node_data['agent']}'"
|
||||
)
|
||||
|
||||
# Removed legacy stream validation - use routes instead
|
||||
|
||||
# Validate merge operations
|
||||
for merge in config.merges:
|
||||
@@ -295,9 +371,9 @@ class ReactiveConfigParser:
|
||||
raise ConfigurationError("Merge operation must specify target stream")
|
||||
|
||||
for source in sources:
|
||||
if source not in all_stream_names:
|
||||
if source not in all_route_names:
|
||||
raise ConfigurationError(
|
||||
f"Merge operation references unknown source stream '{source}'"
|
||||
f"Merge operation references unknown source route '{source}'"
|
||||
)
|
||||
|
||||
# Validate split operations
|
||||
@@ -311,47 +387,12 @@ class ReactiveConfigParser:
|
||||
if not targets:
|
||||
raise ConfigurationError("Split operation must specify target streams")
|
||||
|
||||
if source not in all_stream_names:
|
||||
if source not in all_route_names:
|
||||
raise ConfigurationError(
|
||||
f"Split operation references unknown source stream '{source}'"
|
||||
f"Split operation references unknown source route '{source}'"
|
||||
)
|
||||
|
||||
# Validate LangGraphs
|
||||
for graph_name, graph_config in config.graphs.items():
|
||||
# Check node references in edges
|
||||
all_nodes = set(graph_config.nodes.keys())
|
||||
if "start" not in all_nodes:
|
||||
all_nodes.add("start")
|
||||
if "end" not in all_nodes:
|
||||
all_nodes.add("end")
|
||||
|
||||
for edge in graph_config.edges:
|
||||
source = edge.get("source")
|
||||
target = edge.get("target")
|
||||
|
||||
if not source or not target:
|
||||
raise ConfigurationError(
|
||||
f"Graph '{graph_name}' has edge without source or target"
|
||||
)
|
||||
|
||||
if source not in all_nodes:
|
||||
raise ConfigurationError(
|
||||
f"Graph '{graph_name}' edge references unknown source node '{source}'"
|
||||
)
|
||||
|
||||
if target not in all_nodes:
|
||||
raise ConfigurationError(
|
||||
f"Graph '{graph_name}' edge references unknown target node '{target}'"
|
||||
)
|
||||
|
||||
# Check agent references in nodes
|
||||
for node_name, node_data in graph_config.nodes.items():
|
||||
if node_data.get("type") == "agent":
|
||||
agent_name = node_data.get("agent", "")
|
||||
if agent_name and agent_name not in config.agents:
|
||||
raise ConfigurationError(
|
||||
f"Graph '{graph_name}' node '{node_name}' references unknown agent '{agent_name}'"
|
||||
)
|
||||
# Removed legacy graph validation - use routes instead
|
||||
|
||||
# Validate hybrid pipelines
|
||||
for pipeline_name, pipeline_config in config.pipelines.items():
|
||||
@@ -360,17 +401,23 @@ class ReactiveConfigParser:
|
||||
|
||||
if stage_type == "graph":
|
||||
graph_name = stage.get("config", {}).get("name")
|
||||
if graph_name and graph_name not in config.graphs:
|
||||
# Check in routes instead of graphs
|
||||
graph_found = any(
|
||||
route.type == RouteType.GRAPH and route.name == graph_name
|
||||
for route in config.routes.values()
|
||||
)
|
||||
if graph_name and not graph_found:
|
||||
# Graph might be defined inline, so only warn
|
||||
self.logger.warning(
|
||||
f"Pipeline '{pipeline_name}' references graph '{graph_name}' not found in global graphs"
|
||||
f"Pipeline '{pipeline_name}' references graph '{graph_name}' not found in routes"
|
||||
)
|
||||
|
||||
elif stage_type == "stream":
|
||||
# Check routes instead of streams
|
||||
stream_name = stage.get("name", "")
|
||||
if stream_name and stream_name in config.streams:
|
||||
if stream_name and stream_name in config.routes:
|
||||
self.logger.warning(
|
||||
f"Pipeline '{pipeline_name}' defines stream '{stream_name}' that already exists"
|
||||
f"Pipeline '{pipeline_name}' defines route '{stream_name}' that already exists"
|
||||
)
|
||||
|
||||
self.logger.info("Configuration validation completed successfully")
|
||||
|
||||
@@ -0,0 +1,359 @@
|
||||
"""
|
||||
Unified route system for CleverAgents.
|
||||
|
||||
This module provides a unified abstraction for both reactive streams (RxPy)
|
||||
and stateful graphs (LangGraph), treating them as different types of routes
|
||||
for data flow through processing pipelines.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import field
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
from typing import Dict
|
||||
from typing import List
|
||||
from typing import Optional
|
||||
from typing import Union
|
||||
|
||||
from cleveragents.core.exceptions import ConfigurationError
|
||||
from cleveragents.langgraph.graph import GraphConfig
|
||||
from cleveragents.langgraph.nodes import Edge
|
||||
from cleveragents.langgraph.nodes import NodeConfig
|
||||
from cleveragents.langgraph.nodes import NodeType
|
||||
from cleveragents.reactive.stream_router import StreamConfig
|
||||
from cleveragents.reactive.stream_router import StreamType
|
||||
|
||||
|
||||
class RouteType(Enum):
|
||||
"""Types of routes available in the system."""
|
||||
|
||||
STREAM = "stream" # Reactive stream using RxPy
|
||||
GRAPH = "graph" # Stateful graph using LangGraph
|
||||
BRIDGE = "bridge" # Bridge for converting between types
|
||||
|
||||
|
||||
@dataclass
|
||||
class BridgeConfig:
|
||||
"""Configuration for type conversion/bridging between routes."""
|
||||
|
||||
# When to upgrade from stream to graph
|
||||
upgrade_conditions: Dict[str, Any] = field(default_factory=dict)
|
||||
# When to downgrade from graph to stream
|
||||
downgrade_conditions: Dict[str, Any] = field(default_factory=dict)
|
||||
# State extraction for stream->graph conversion
|
||||
state_extractor: Optional[str] = None
|
||||
# State flattener for graph->stream conversion
|
||||
state_flattener: Optional[str] = None
|
||||
# Preserve stream subscriptions during conversion
|
||||
preserve_subscriptions: bool = True
|
||||
# Preserve graph checkpointing during conversion
|
||||
preserve_checkpointing: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class RouteConfig:
|
||||
"""Unified configuration for all route types."""
|
||||
|
||||
name: str
|
||||
type: RouteType # REQUIRED field - must be prominent
|
||||
|
||||
# Stream-specific config (used when type=STREAM)
|
||||
stream_type: Optional[StreamType] = None
|
||||
operators: List[Dict[str, Any]] = field(default_factory=list)
|
||||
subscriptions: List[str] = field(default_factory=list)
|
||||
publications: List[str] = field(default_factory=list)
|
||||
agents: List[str] = field(default_factory=list)
|
||||
initial_value: Optional[Any] = None
|
||||
buffer_size: int = 1
|
||||
|
||||
# Graph-specific config (used when type=GRAPH)
|
||||
nodes: Dict[str, Dict[str, Any]] = field(default_factory=dict)
|
||||
edges: List[Dict[str, Any]] = field(default_factory=list)
|
||||
entry_point: str = "start"
|
||||
checkpointing: bool = False
|
||||
checkpoint_dir: Optional[str] = None
|
||||
enable_time_travel: bool = False
|
||||
parallel_execution: bool = True
|
||||
state_class: Optional[str] = None
|
||||
|
||||
# Bridge config (optional - for type conversion)
|
||||
bridge: Optional[BridgeConfig] = None
|
||||
|
||||
# Common metadata
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
template_config: Optional[Dict[str, Any]] = None
|
||||
|
||||
def __post_init__(self):
|
||||
"""Validate configuration based on route type."""
|
||||
if self.type == RouteType.STREAM:
|
||||
if not self.stream_type:
|
||||
self.stream_type = StreamType.COLD
|
||||
elif self.type == RouteType.GRAPH:
|
||||
if not self.nodes:
|
||||
raise ConfigurationError(
|
||||
f"Route '{self.name}' of type 'graph' must have nodes defined"
|
||||
)
|
||||
|
||||
def to_stream_config(self) -> StreamConfig:
|
||||
"""Convert to StreamConfig for stream routes."""
|
||||
if self.type != RouteType.STREAM:
|
||||
raise ValueError(f"Cannot convert {self.type} route to StreamConfig")
|
||||
|
||||
return StreamConfig(
|
||||
name=self.name,
|
||||
type=self.stream_type or StreamType.COLD,
|
||||
operators=self.operators,
|
||||
subscriptions=self.subscriptions,
|
||||
publications=self.publications,
|
||||
agents=self.agents,
|
||||
initial_value=self.initial_value,
|
||||
buffer_size=self.buffer_size,
|
||||
template_config=self.template_config,
|
||||
)
|
||||
|
||||
def to_graph_config(self) -> GraphConfig:
|
||||
"""Convert to GraphConfig for graph routes."""
|
||||
if self.type != RouteType.GRAPH:
|
||||
raise ValueError(f"Cannot convert {self.type} route to GraphConfig")
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
# Convert node dictionaries to NodeConfig objects
|
||||
node_configs = {}
|
||||
for node_name, node_data in self.nodes.items():
|
||||
node_type_str = node_data.get("type", "agent")
|
||||
node_type = NodeType[node_type_str.upper()]
|
||||
|
||||
node_configs[node_name] = NodeConfig(
|
||||
name=node_name,
|
||||
type=node_type,
|
||||
agent=node_data.get("agent"),
|
||||
function=node_data.get("function"),
|
||||
tools=node_data.get("tools", []),
|
||||
retry_policy=node_data.get("retry_policy"),
|
||||
timeout=node_data.get("timeout"),
|
||||
parallel=node_data.get("parallel", False),
|
||||
condition=node_data.get("condition"),
|
||||
subgraph=node_data.get("subgraph"),
|
||||
metadata=node_data.get("metadata", {}),
|
||||
)
|
||||
|
||||
# Convert edge dictionaries to Edge objects
|
||||
edge_objects = []
|
||||
for edge_data in self.edges:
|
||||
edge_objects.append(
|
||||
Edge(
|
||||
source=edge_data["source"],
|
||||
target=edge_data["target"],
|
||||
condition=edge_data.get("condition"),
|
||||
metadata=edge_data.get("metadata", {}),
|
||||
)
|
||||
)
|
||||
|
||||
return GraphConfig(
|
||||
name=self.name,
|
||||
nodes=node_configs,
|
||||
edges=edge_objects,
|
||||
entry_point=self.entry_point,
|
||||
state_class=None, # Will be resolved later from string
|
||||
checkpointing=self.checkpointing,
|
||||
checkpoint_dir=Path(self.checkpoint_dir) if self.checkpoint_dir else None,
|
||||
enable_time_travel=self.enable_time_travel,
|
||||
parallel_execution=self.parallel_execution,
|
||||
metadata=self.metadata,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_stream_config(cls, stream_config: StreamConfig) -> "RouteConfig":
|
||||
"""Create RouteConfig from existing StreamConfig."""
|
||||
return cls(
|
||||
name=stream_config.name,
|
||||
type=RouteType.STREAM,
|
||||
stream_type=stream_config.type,
|
||||
operators=stream_config.operators,
|
||||
subscriptions=stream_config.subscriptions,
|
||||
publications=stream_config.publications,
|
||||
agents=stream_config.agents,
|
||||
initial_value=stream_config.initial_value,
|
||||
buffer_size=stream_config.buffer_size,
|
||||
template_config=stream_config.template_config,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_graph_config(cls, graph_config: GraphConfig) -> "RouteConfig":
|
||||
"""Create RouteConfig from existing GraphConfig."""
|
||||
# Convert nodes from NodeConfig to dict
|
||||
nodes_dict = {}
|
||||
for name, node_config in graph_config.nodes.items():
|
||||
node_dict: Dict[str, Any] = {
|
||||
"type": node_config.type.value,
|
||||
"parallel": node_config.parallel,
|
||||
}
|
||||
if node_config.agent:
|
||||
node_dict["agent"] = node_config.agent
|
||||
if node_config.function:
|
||||
node_dict["function"] = node_config.function
|
||||
if node_config.tools:
|
||||
node_dict["tools"] = node_config.tools
|
||||
if node_config.retry_policy:
|
||||
node_dict["retry_policy"] = node_config.retry_policy
|
||||
if node_config.timeout:
|
||||
node_dict["timeout"] = node_config.timeout
|
||||
nodes_dict[name] = node_dict
|
||||
|
||||
# Convert edges from Edge to dict
|
||||
edges_list = []
|
||||
for edge in graph_config.edges:
|
||||
edge_dict: Dict[str, Any] = {
|
||||
"source": edge.source,
|
||||
"target": edge.target,
|
||||
}
|
||||
if edge.condition:
|
||||
edge_dict["condition"] = edge.condition
|
||||
edges_list.append(edge_dict)
|
||||
|
||||
return cls(
|
||||
name=graph_config.name,
|
||||
type=RouteType.GRAPH,
|
||||
nodes=nodes_dict,
|
||||
edges=edges_list,
|
||||
entry_point=graph_config.entry_point,
|
||||
checkpointing=graph_config.checkpointing,
|
||||
checkpoint_dir=(
|
||||
str(graph_config.checkpoint_dir)
|
||||
if graph_config.checkpoint_dir
|
||||
else None
|
||||
),
|
||||
enable_time_travel=graph_config.enable_time_travel,
|
||||
parallel_execution=graph_config.parallel_execution,
|
||||
state_class=None, # String representation
|
||||
metadata=graph_config.metadata,
|
||||
)
|
||||
|
||||
|
||||
class RouteComplexityAnalyzer:
|
||||
"""
|
||||
Analyzes route complexity to help users choose the right type.
|
||||
|
||||
Complexity ladder:
|
||||
1. Simple Stream - Basic data transformation
|
||||
2. Stream with Operators - Multiple transformations
|
||||
3. Stream with Merging/Splitting - Complex routing
|
||||
4. Basic Graph - Conditional logic needed
|
||||
5. Stateful Graph - State management required
|
||||
6. Graph with Checkpointing - Persistence/recovery needed
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def analyze_route(config: RouteConfig) -> Dict[str, Any]:
|
||||
"""Analyze a route and return complexity metrics."""
|
||||
if config.type == RouteType.STREAM:
|
||||
return RouteComplexityAnalyzer._analyze_stream(config)
|
||||
elif config.type == RouteType.GRAPH:
|
||||
return RouteComplexityAnalyzer._analyze_graph(config)
|
||||
else:
|
||||
return {"complexity": "bridge", "score": 0}
|
||||
|
||||
@staticmethod
|
||||
def _analyze_stream(config: RouteConfig) -> Dict[str, Any]:
|
||||
"""Analyze stream complexity."""
|
||||
score = 1 # Base score for streams
|
||||
features = []
|
||||
|
||||
if config.operators:
|
||||
score += len(config.operators)
|
||||
features.append(f"{len(config.operators)} operators")
|
||||
|
||||
if config.subscriptions or config.publications:
|
||||
score += 2
|
||||
features.append("routing connections")
|
||||
|
||||
if config.stream_type == StreamType.HOT:
|
||||
score += 1
|
||||
features.append("hot/stateful stream")
|
||||
|
||||
return {
|
||||
"type": "stream",
|
||||
"complexity": (
|
||||
"simple" if score <= 2 else "moderate" if score <= 5 else "complex"
|
||||
),
|
||||
"score": score,
|
||||
"features": features,
|
||||
"recommendation": RouteComplexityAnalyzer._get_stream_recommendation(score),
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _analyze_graph(config: RouteConfig) -> Dict[str, Any]:
|
||||
"""Analyze graph complexity."""
|
||||
score = 5 # Base score for graphs
|
||||
features = []
|
||||
|
||||
node_count = len(config.nodes)
|
||||
edge_count = len(config.edges)
|
||||
|
||||
score += node_count + (edge_count // 2)
|
||||
features.append(f"{node_count} nodes, {edge_count} edges")
|
||||
|
||||
if config.checkpointing:
|
||||
score += 3
|
||||
features.append("checkpointing enabled")
|
||||
|
||||
if config.enable_time_travel:
|
||||
score += 2
|
||||
features.append("time travel enabled")
|
||||
|
||||
# Check for conditional edges
|
||||
conditional_edges = sum(1 for edge in config.edges if "condition" in edge)
|
||||
if conditional_edges:
|
||||
score += conditional_edges * 2
|
||||
features.append(f"{conditional_edges} conditional edges")
|
||||
|
||||
return {
|
||||
"type": "graph",
|
||||
"complexity": (
|
||||
"moderate" if score <= 10 else "complex" if score <= 15 else "advanced"
|
||||
),
|
||||
"score": score,
|
||||
"features": features,
|
||||
"recommendation": RouteComplexityAnalyzer._get_graph_recommendation(score),
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _get_stream_recommendation(score: int) -> str:
|
||||
"""Get recommendation for stream complexity."""
|
||||
if score <= 2:
|
||||
return "Good for simple transformations and filtering"
|
||||
elif score <= 5:
|
||||
return "Suitable for multi-step processing pipelines"
|
||||
else:
|
||||
return "Consider using a graph if you need conditional logic or state"
|
||||
|
||||
@staticmethod
|
||||
def _get_graph_recommendation(score: int) -> str:
|
||||
"""Get recommendation for graph complexity."""
|
||||
if score <= 10:
|
||||
return "Good for workflows with conditional logic"
|
||||
elif score <= 15:
|
||||
return "Suitable for complex stateful workflows"
|
||||
else:
|
||||
return "Advanced setup - ensure you need all features"
|
||||
|
||||
@staticmethod
|
||||
def suggest_route_type(requirements: Dict[str, Any]) -> RouteType:
|
||||
"""Suggest the best route type based on requirements."""
|
||||
needs_state = requirements.get("needs_state", False)
|
||||
needs_conditionals = requirements.get("needs_conditionals", False)
|
||||
needs_persistence = requirements.get("needs_persistence", False)
|
||||
is_continuous = requirements.get("is_continuous", False)
|
||||
is_stateless = requirements.get("is_stateless", True)
|
||||
|
||||
if needs_persistence or (needs_state and needs_conditionals):
|
||||
return RouteType.GRAPH
|
||||
elif needs_conditionals and not is_continuous:
|
||||
return RouteType.GRAPH
|
||||
elif is_stateless and is_continuous:
|
||||
return RouteType.STREAM
|
||||
else:
|
||||
# Default to stream for simpler cases
|
||||
return RouteType.STREAM
|
||||
@@ -0,0 +1,309 @@
|
||||
"""
|
||||
Route bridging and type conversion system.
|
||||
|
||||
This module provides functionality to dynamically convert between
|
||||
stream and graph routes based on runtime conditions.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
from typing import Dict
|
||||
from typing import Optional
|
||||
|
||||
import rx
|
||||
from rx import operators as ops
|
||||
from rx.scheduler.eventloop import AsyncIOScheduler
|
||||
|
||||
from cleveragents.agents.base import Agent
|
||||
from cleveragents.langgraph.graph import GraphConfig
|
||||
from cleveragents.langgraph.graph import LangGraph
|
||||
from cleveragents.langgraph.nodes import Edge
|
||||
from cleveragents.langgraph.nodes import NodeConfig
|
||||
from cleveragents.langgraph.nodes import NodeType
|
||||
from cleveragents.langgraph.state import GraphState
|
||||
from cleveragents.reactive.route import BridgeConfig
|
||||
from cleveragents.reactive.route import RouteConfig
|
||||
from cleveragents.reactive.route import RouteType
|
||||
from cleveragents.reactive.stream_router import ReactiveStreamRouter
|
||||
from cleveragents.reactive.stream_router import StreamConfig
|
||||
from cleveragents.reactive.stream_router import StreamMessage
|
||||
from cleveragents.reactive.stream_router import StreamType
|
||||
|
||||
|
||||
class RouteBridge:
|
||||
"""
|
||||
Handles dynamic conversion between stream and graph routes.
|
||||
|
||||
This allows routes to upgrade from simple streams to stateful graphs
|
||||
when conditions require it, or downgrade from graphs to streams when
|
||||
the complexity is no longer needed.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
stream_router: ReactiveStreamRouter,
|
||||
agents: Dict[str, Agent],
|
||||
scheduler: Optional[AsyncIOScheduler] = None,
|
||||
):
|
||||
"""Initialize the route bridge."""
|
||||
self.stream_router = stream_router
|
||||
self.agents = agents
|
||||
self.scheduler = scheduler
|
||||
self.logger = logging.getLogger(__name__)
|
||||
self._active_conversions: Dict[str, Any] = {}
|
||||
|
||||
async def check_upgrade_conditions(
|
||||
self,
|
||||
route_config: RouteConfig,
|
||||
message: StreamMessage,
|
||||
) -> bool:
|
||||
"""Check if a stream should be upgraded to a graph."""
|
||||
if not route_config.bridge or route_config.type != RouteType.STREAM:
|
||||
return False
|
||||
|
||||
conditions = route_config.bridge.upgrade_conditions
|
||||
|
||||
# Check built-in conditions
|
||||
if "needs_state" in conditions:
|
||||
# Check if message indicates state requirement
|
||||
if message.metadata.get("requires_state", False):
|
||||
return True
|
||||
|
||||
if "message_count" in conditions:
|
||||
# Check if we've processed enough messages to warrant upgrade
|
||||
count = self._get_message_count(route_config.name)
|
||||
if count >= conditions["message_count"]:
|
||||
return True
|
||||
|
||||
if "complexity_threshold" in conditions:
|
||||
# Analyze route complexity
|
||||
from cleveragents.reactive.route import RouteComplexityAnalyzer
|
||||
|
||||
analysis = RouteComplexityAnalyzer.analyze_route(route_config)
|
||||
if analysis["score"] >= conditions["complexity_threshold"]:
|
||||
return True
|
||||
|
||||
if "custom_predicate" in conditions:
|
||||
# Evaluate custom predicate function
|
||||
predicate = conditions["custom_predicate"]
|
||||
if callable(predicate):
|
||||
return predicate(message, route_config)
|
||||
|
||||
return False
|
||||
|
||||
async def check_downgrade_conditions(
|
||||
self,
|
||||
route_config: RouteConfig,
|
||||
state: GraphState,
|
||||
) -> bool:
|
||||
"""Check if a graph should be downgraded to a stream."""
|
||||
if not route_config.bridge or route_config.type != RouteType.GRAPH:
|
||||
return False
|
||||
|
||||
conditions = route_config.bridge.downgrade_conditions
|
||||
|
||||
# Check built-in conditions
|
||||
if "idle_time" in conditions:
|
||||
# Check if graph has been idle
|
||||
idle_threshold = conditions["idle_time"]
|
||||
last_updated = state.metadata.get("last_updated")
|
||||
if (
|
||||
last_updated
|
||||
and (asyncio.get_event_loop().time() - last_updated) > idle_threshold
|
||||
):
|
||||
return True
|
||||
|
||||
if "state_size" in conditions:
|
||||
# Check if state is minimal (using messages as proxy for state size)
|
||||
if len(state.messages) <= conditions["state_size"]:
|
||||
return True
|
||||
|
||||
if "no_conditionals_used" in conditions:
|
||||
# Check if conditional edges were actually used
|
||||
if not state.metadata.get("conditional_edges_used", False):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
async def upgrade_stream_to_graph(
|
||||
self,
|
||||
route_config: RouteConfig,
|
||||
message: StreamMessage,
|
||||
) -> LangGraph:
|
||||
"""Convert a stream route to a graph route."""
|
||||
self.logger.info(f"Upgrading stream route '{route_config.name}' to graph")
|
||||
|
||||
# Create graph config from stream
|
||||
graph_config = self._create_graph_from_stream(route_config)
|
||||
|
||||
# Extract initial state if configured
|
||||
initial_state = {}
|
||||
if route_config.bridge and route_config.bridge.state_extractor:
|
||||
extractor = route_config.bridge.state_extractor
|
||||
if callable(extractor):
|
||||
initial_state = extractor(message)
|
||||
|
||||
# Create the graph
|
||||
graph = LangGraph(
|
||||
config=graph_config,
|
||||
agents=self.agents,
|
||||
stream_router=self.stream_router,
|
||||
scheduler=self.scheduler,
|
||||
)
|
||||
|
||||
# Initialize with extracted state
|
||||
if initial_state:
|
||||
graph.state_manager.update_state(initial_state)
|
||||
|
||||
# Preserve subscriptions if configured
|
||||
if route_config.bridge and route_config.bridge.preserve_subscriptions:
|
||||
# Re-subscribe to the same sources
|
||||
for sub in route_config.subscriptions:
|
||||
# This would need implementation in the graph
|
||||
pass
|
||||
|
||||
self._active_conversions[route_config.name] = {
|
||||
"type": "graph",
|
||||
"instance": graph,
|
||||
"original_config": route_config,
|
||||
}
|
||||
|
||||
return graph
|
||||
|
||||
async def downgrade_graph_to_stream(
|
||||
self,
|
||||
route_config: RouteConfig,
|
||||
graph: LangGraph,
|
||||
) -> StreamConfig:
|
||||
"""Convert a graph route back to a stream route."""
|
||||
self.logger.info(f"Downgrading graph route '{route_config.name}' to stream")
|
||||
|
||||
# Create stream config from graph
|
||||
stream_config = self._create_stream_from_graph(route_config, graph)
|
||||
|
||||
# Flatten state if configured
|
||||
if route_config.bridge and route_config.bridge.state_flattener:
|
||||
flattener = route_config.bridge.state_flattener
|
||||
if callable(flattener):
|
||||
final_state = graph.state_manager.get_state()
|
||||
flattened_data = flattener(final_state)
|
||||
# This could be injected into the stream somehow
|
||||
|
||||
# Preserve checkpointing info if configured
|
||||
if route_config.bridge and route_config.bridge.preserve_checkpointing:
|
||||
# Save final checkpoint if checkpoint_dir is set
|
||||
if graph.state_manager.checkpoint_dir:
|
||||
graph.state_manager._save_checkpoint()
|
||||
|
||||
self._active_conversions[route_config.name] = {
|
||||
"type": "stream",
|
||||
"instance": stream_config,
|
||||
"original_config": route_config,
|
||||
}
|
||||
|
||||
return stream_config
|
||||
|
||||
def _create_graph_from_stream(self, route_config: RouteConfig) -> GraphConfig:
|
||||
"""Create a graph configuration from a stream route."""
|
||||
# Create nodes from stream operators
|
||||
nodes = {}
|
||||
edges = []
|
||||
|
||||
# Entry node
|
||||
prev_node = "start"
|
||||
|
||||
# Convert each operator to a node
|
||||
for i, operator in enumerate(route_config.operators):
|
||||
node_name = f"op_{i}_{operator.get('type', 'unknown')}"
|
||||
|
||||
# Create node based on operator type
|
||||
if operator["type"] == "map" and "agent" in operator.get("params", {}):
|
||||
# Agent-based operator becomes agent node
|
||||
nodes[node_name] = NodeConfig(
|
||||
name=node_name,
|
||||
type=NodeType.AGENT,
|
||||
agent=operator["params"]["agent"],
|
||||
)
|
||||
else:
|
||||
# Other operators become function nodes
|
||||
# For now, we'll use the operator type as the function name
|
||||
nodes[node_name] = NodeConfig(
|
||||
name=node_name,
|
||||
type=NodeType.FUNCTION,
|
||||
function=f"stream_operator_{operator.get('type', 'unknown')}",
|
||||
)
|
||||
|
||||
# Add edge from previous node
|
||||
edges.append(Edge(source=prev_node, target=node_name))
|
||||
prev_node = node_name
|
||||
|
||||
# Add final edge to end
|
||||
edges.append(Edge(source=prev_node, target="end"))
|
||||
|
||||
return GraphConfig(
|
||||
name=f"{route_config.name}_graph",
|
||||
nodes=nodes,
|
||||
edges=edges,
|
||||
entry_point="start",
|
||||
checkpointing=True, # Enable by default for upgraded routes
|
||||
parallel_execution=False, # Sequential like stream
|
||||
)
|
||||
|
||||
def _create_stream_from_graph(
|
||||
self,
|
||||
route_config: RouteConfig,
|
||||
graph: LangGraph,
|
||||
) -> StreamConfig:
|
||||
"""Create a stream configuration from a graph route."""
|
||||
# Extract linear flow from graph
|
||||
operators = []
|
||||
agents = []
|
||||
|
||||
# Traverse graph nodes in topological order
|
||||
topological_levels = graph._topological_levels()
|
||||
sorted_nodes = []
|
||||
for level in sorted(topological_levels.keys()):
|
||||
sorted_nodes.extend(sorted(topological_levels[level]))
|
||||
|
||||
for node_name in sorted_nodes:
|
||||
if node_name in ["start", "end"]:
|
||||
continue
|
||||
|
||||
node = graph.nodes.get(node_name)
|
||||
if not node:
|
||||
continue
|
||||
|
||||
# Convert node to operator
|
||||
if node.config.type == NodeType.AGENT:
|
||||
operators.append(
|
||||
{
|
||||
"type": "map",
|
||||
"params": {"agent": node.config.agent},
|
||||
}
|
||||
)
|
||||
if node.config.agent:
|
||||
agents.append(node.config.agent)
|
||||
elif node.config.type == NodeType.FUNCTION:
|
||||
# Restore original operator if stored
|
||||
if isinstance(node.config.function, dict):
|
||||
operators.append(node.config.function)
|
||||
|
||||
return StreamConfig(
|
||||
name=f"{route_config.name}_stream",
|
||||
type=route_config.stream_type or StreamType.COLD,
|
||||
operators=operators,
|
||||
agents=agents,
|
||||
subscriptions=route_config.subscriptions,
|
||||
publications=route_config.publications,
|
||||
)
|
||||
|
||||
def _get_message_count(self, route_name: str) -> int:
|
||||
"""Get the number of messages processed by a route."""
|
||||
# This would need to be tracked somewhere
|
||||
# For now, return a placeholder
|
||||
return 0
|
||||
|
||||
def get_active_conversion(self, route_name: str) -> Optional[Dict[str, Any]]:
|
||||
"""Get active conversion info for a route."""
|
||||
return self._active_conversions.get(route_name)
|
||||
@@ -1,16 +0,0 @@
|
||||
from cleveragents.agents.tool import Tool
|
||||
from cleveragents.core.exceptions import ExecutionError
|
||||
|
||||
|
||||
class CustomToolTool(Tool):
|
||||
"""Custom tool implementation for testing."""
|
||||
|
||||
def __init__(self, config=None):
|
||||
super().__init__("custom_tool", "A custom tool for testing", config or {})
|
||||
|
||||
async def execute(self, input_data, context=None):
|
||||
if context is None:
|
||||
context = {}
|
||||
if "fail" in input_data.lower():
|
||||
raise ExecutionError("Tool execution failed as requested")
|
||||
return f"Custom tool processed: {input_data}"
|
||||
@@ -26,9 +26,10 @@ Feature: CLI Integration
|
||||
tools: ["echo"]
|
||||
safe_mode: true
|
||||
|
||||
streams:
|
||||
routes:
|
||||
echo_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -61,9 +62,10 @@ Feature: CLI Integration
|
||||
config:
|
||||
tools: ["echo"]
|
||||
|
||||
streams:
|
||||
routes:
|
||||
echo_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
|
||||
@@ -17,9 +17,10 @@ Feature: Reactive Application Management
|
||||
model: gpt-3.5-turbo
|
||||
temperature: 0.7
|
||||
|
||||
streams:
|
||||
routes:
|
||||
chat_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -34,7 +35,7 @@ Feature: Reactive Application Management
|
||||
When I load the configuration
|
||||
Then the application should initialize successfully
|
||||
And I should have 1 agent
|
||||
And I should have 1 stream
|
||||
And I should have 1 route
|
||||
|
||||
Scenario: Single-shot processing
|
||||
Given I have a loaded reactive application
|
||||
|
||||
@@ -46,9 +46,10 @@ agents:
|
||||
config:
|
||||
tools: ["echo", "math"]
|
||||
|
||||
streams:
|
||||
routes:
|
||||
input_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -57,7 +58,8 @@ streams:
|
||||
- processing_stream
|
||||
|
||||
processing_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -94,9 +96,10 @@ agents:
|
||||
"""
|
||||
elif content_type == "streams":
|
||||
content = """
|
||||
streams:
|
||||
routes:
|
||||
test_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -131,9 +134,10 @@ agents:
|
||||
tools: ["file_write"]
|
||||
safe_mode: false
|
||||
|
||||
streams:
|
||||
routes:
|
||||
file_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -160,9 +164,10 @@ agents:
|
||||
config:
|
||||
tools: ["echo"]
|
||||
|
||||
streams:
|
||||
routes:
|
||||
echo_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -194,9 +199,10 @@ agents:
|
||||
config:
|
||||
tools: ["echo"]
|
||||
|
||||
streams:
|
||||
routes:
|
||||
stream1:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -205,7 +211,8 @@ streams:
|
||||
- merge_stream
|
||||
|
||||
stream2:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -214,7 +221,8 @@ streams:
|
||||
- merge_stream
|
||||
|
||||
merge_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: filter
|
||||
params:
|
||||
@@ -330,7 +338,7 @@ def step_verify_network_structure(context: Context):
|
||||
"""Verify stream network visualization."""
|
||||
output = context.cli_stdout
|
||||
assert "Agents:" in output or "agents" in output.lower()
|
||||
assert "Streams:" in output or "streams" in output.lower()
|
||||
assert "Routes:" in output or "routes" in output.lower()
|
||||
|
||||
|
||||
@then("it should display agents and streams")
|
||||
|
||||
@@ -367,8 +367,21 @@ def step_verify_agent_count(context: Context, count: int):
|
||||
|
||||
@then("I should have {count:d} stream")
|
||||
def step_verify_stream_count(context: Context, count: int):
|
||||
"""Verify stream count."""
|
||||
assert len(context.app.config.streams) == count
|
||||
"""Verify stream count (legacy - counts stream routes)."""
|
||||
stream_count = sum(
|
||||
1
|
||||
for route in context.app.config.routes.values()
|
||||
if route.type.value == "stream"
|
||||
)
|
||||
assert stream_count == count
|
||||
|
||||
|
||||
@then("I should have {count:d} route")
|
||||
def step_verify_route_count(context: Context, count: int):
|
||||
"""Verify route count."""
|
||||
# Count all routes (stream and graph types)
|
||||
route_count = len(context.app.config.routes)
|
||||
assert route_count == count
|
||||
|
||||
|
||||
@then("I should receive a response")
|
||||
@@ -601,9 +614,10 @@ agents:
|
||||
provider: openai
|
||||
model: gpt-3.5-turbo
|
||||
|
||||
streams:
|
||||
routes:
|
||||
main:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -661,9 +675,10 @@ agents:
|
||||
provider: openai
|
||||
model: gpt-4
|
||||
|
||||
streams:
|
||||
routes:
|
||||
input_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -672,7 +687,8 @@ streams:
|
||||
- processing_stream
|
||||
|
||||
processing_stream:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -784,11 +800,12 @@ agents:
|
||||
model: gpt-3.5-turbo
|
||||
"""
|
||||
|
||||
# Create streams config
|
||||
streams_config = """
|
||||
streams:
|
||||
# Create routes config
|
||||
routes_config = """
|
||||
routes:
|
||||
stream1:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -796,7 +813,8 @@ streams:
|
||||
publications:
|
||||
- __output__
|
||||
stream2:
|
||||
type: cold
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
@@ -816,13 +834,13 @@ merges:
|
||||
agents_file = context.scenario_temp / "agents.yaml"
|
||||
agents_file.write_text(agents_config)
|
||||
|
||||
streams_file = context.scenario_temp / "streams.yaml"
|
||||
streams_file.write_text(streams_config)
|
||||
routes_file = context.scenario_temp / "routes.yaml"
|
||||
routes_file.write_text(routes_config)
|
||||
|
||||
routing_file = context.scenario_temp / "routing.yaml"
|
||||
routing_file.write_text(routing_config)
|
||||
|
||||
context.config_files = [agents_file, streams_file, routing_file]
|
||||
context.config_files = [agents_file, routes_file, routing_file]
|
||||
|
||||
|
||||
@when("I load all configuration files")
|
||||
@@ -843,9 +861,9 @@ def step_verify_configs_merged(context: Context):
|
||||
# Check that we have both agents
|
||||
assert "agent1" in context.app.config.agents
|
||||
assert "agent2" in context.app.config.agents
|
||||
# Check that we have both streams
|
||||
assert "stream1" in context.app.config.streams
|
||||
assert "stream2" in context.app.config.streams
|
||||
# Check that we have routes
|
||||
assert "stream1" in context.app.config.routes
|
||||
assert "stream2" in context.app.config.routes
|
||||
|
||||
|
||||
@then("the application should work with the combined configuration")
|
||||
|
||||
@@ -0,0 +1,292 @@
|
||||
"""
|
||||
BDD step definitions for unified routes testing.
|
||||
"""
|
||||
|
||||
import json
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
from behave import given
|
||||
from behave import then
|
||||
from behave import when
|
||||
from behave.runner import Context
|
||||
|
||||
from cleveragents.core.application import ReactiveCleverAgentsApp
|
||||
from cleveragents.core.exceptions import ConfigurationError
|
||||
from cleveragents.reactive.route import RouteComplexityAnalyzer
|
||||
from cleveragents.reactive.route import RouteType
|
||||
|
||||
|
||||
@given("I have a route configuration")
|
||||
def step_route_config(context: Context):
|
||||
"""Parse route configuration from docstring."""
|
||||
context.config_data = yaml.safe_load(context.text)
|
||||
context.temp_file = tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".yaml", delete=False
|
||||
)
|
||||
yaml.dump(context.config_data, context.temp_file)
|
||||
context.temp_file.close()
|
||||
context.config_file = Path(context.temp_file.name)
|
||||
context.config_files = [context.config_file]
|
||||
|
||||
|
||||
@given("I have a route configuration without type")
|
||||
def step_route_config_no_type(context: Context):
|
||||
"""Parse invalid route configuration."""
|
||||
context.config_data = yaml.safe_load(context.text)
|
||||
context.temp_file = tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".yaml", delete=False
|
||||
)
|
||||
yaml.dump(context.config_data, context.temp_file)
|
||||
context.temp_file.close()
|
||||
context.config_file = Path(context.temp_file.name)
|
||||
context.config_files = [context.config_file]
|
||||
|
||||
|
||||
@given("I have a route configuration with bridging")
|
||||
def step_route_config_bridging(context: Context):
|
||||
"""Parse route configuration with bridge settings."""
|
||||
context.config_data = yaml.safe_load(context.text)
|
||||
# Add missing agent for the example
|
||||
if "agents" not in context.config_data:
|
||||
context.config_data["agents"] = {
|
||||
"processor": {
|
||||
"type": "llm",
|
||||
"config": {"provider": "openai", "model": "gpt-3.5-turbo"},
|
||||
}
|
||||
}
|
||||
|
||||
context.temp_file = tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".yaml", delete=False
|
||||
)
|
||||
yaml.dump(context.config_data, context.temp_file)
|
||||
context.temp_file.close()
|
||||
context.config_file = Path(context.temp_file.name)
|
||||
context.config_files = [context.config_file]
|
||||
|
||||
|
||||
@given("I have a configuration with both stream and graph routes")
|
||||
def step_mixed_routes_config(context: Context):
|
||||
"""Parse configuration with mixed route types."""
|
||||
context.config_data = yaml.safe_load(context.text)
|
||||
# Add missing agent
|
||||
if "agents" not in context.config_data:
|
||||
context.config_data["agents"] = {
|
||||
"processor": {
|
||||
"type": "llm",
|
||||
"config": {"provider": "openai", "model": "gpt-3.5-turbo"},
|
||||
}
|
||||
}
|
||||
|
||||
context.temp_file = tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".yaml", delete=False
|
||||
)
|
||||
yaml.dump(context.config_data, context.temp_file)
|
||||
context.temp_file.close()
|
||||
context.config_file = Path(context.temp_file.name)
|
||||
context.config_files = [context.config_file]
|
||||
|
||||
|
||||
@given("I have a bridge-only route configuration")
|
||||
def step_bridge_route_config(context: Context):
|
||||
"""Parse bridge route configuration."""
|
||||
context.config_data = yaml.safe_load(context.text)
|
||||
context.temp_file = tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".yaml", delete=False
|
||||
)
|
||||
yaml.dump(context.config_data, context.temp_file)
|
||||
context.temp_file.close()
|
||||
context.config_file = Path(context.temp_file.name)
|
||||
context.config_files = [context.config_file]
|
||||
|
||||
|
||||
@given("I have routes with different complexity levels")
|
||||
def step_complex_routes(context: Context):
|
||||
"""Create routes with varying complexity."""
|
||||
context.app = ReactiveCleverAgentsApp()
|
||||
# Create sample routes for complexity analysis
|
||||
context.test_routes = {
|
||||
"simple_stream": {
|
||||
"type": RouteType.STREAM,
|
||||
"operators": [{"type": "map"}],
|
||||
},
|
||||
"complex_stream": {
|
||||
"type": RouteType.STREAM,
|
||||
"operators": [
|
||||
{"type": "map"},
|
||||
{"type": "filter"},
|
||||
{"type": "debounce"},
|
||||
],
|
||||
"subscriptions": ["input1", "input2"],
|
||||
"publications": ["output1", "output2"],
|
||||
},
|
||||
"simple_graph": {
|
||||
"type": RouteType.GRAPH,
|
||||
"nodes": {"node1": {}},
|
||||
"edges": [{"source": "start", "target": "node1"}],
|
||||
},
|
||||
"complex_graph": {
|
||||
"type": RouteType.GRAPH,
|
||||
"nodes": {"node1": {}, "node2": {}, "node3": {}},
|
||||
"edges": [
|
||||
{"source": "start", "target": "node1"},
|
||||
{"source": "node1", "target": "node2", "condition": "test"},
|
||||
{"source": "node2", "target": "node3"},
|
||||
],
|
||||
"checkpointing": True,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@given("I have a stream route with bridge configuration")
|
||||
def step_stream_with_bridge(context: Context):
|
||||
"""Create a stream route with bridge config."""
|
||||
# This would be set up with a proper route config
|
||||
context.has_bridge_route = True
|
||||
|
||||
|
||||
@then('the route "{route_name}" should be of type "{route_type}"')
|
||||
def step_verify_route_type(context: Context, route_name: str, route_type: str):
|
||||
"""Verify route type."""
|
||||
route = context.app.config.routes.get(route_name)
|
||||
assert route is not None, f"Route '{route_name}' not found"
|
||||
assert route.type.value == route_type.lower()
|
||||
|
||||
|
||||
@then('the route "{route_name}" should have bridge configuration')
|
||||
def step_verify_bridge_config(context: Context, route_name: str):
|
||||
"""Verify route has bridge configuration."""
|
||||
route = context.app.config.routes.get(route_name)
|
||||
assert route is not None
|
||||
assert route.bridge is not None
|
||||
|
||||
|
||||
@then("the bridge should have upgrade conditions")
|
||||
def step_verify_upgrade_conditions(context: Context):
|
||||
"""Verify bridge upgrade conditions."""
|
||||
# Find route with bridge
|
||||
bridge_route = None
|
||||
for route in context.app.config.routes.values():
|
||||
if route.bridge:
|
||||
bridge_route = route
|
||||
break
|
||||
|
||||
assert bridge_route is not None
|
||||
assert bridge_route.bridge.upgrade_conditions
|
||||
assert "needs_state" in bridge_route.bridge.upgrade_conditions
|
||||
assert "message_count" in bridge_route.bridge.upgrade_conditions
|
||||
|
||||
|
||||
@then("I should have {count:d} stream route")
|
||||
def step_verify_stream_route_count(context: Context, count: int):
|
||||
"""Count stream-type routes."""
|
||||
stream_count = sum(
|
||||
1
|
||||
for route in context.app.config.routes.values()
|
||||
if route.type == RouteType.STREAM
|
||||
)
|
||||
assert stream_count == count
|
||||
|
||||
|
||||
@then("I should have {count:d} graph route")
|
||||
def step_verify_graph_route_count(context: Context, count: int):
|
||||
"""Count graph-type routes."""
|
||||
graph_count = sum(
|
||||
1
|
||||
for route in context.app.config.routes.values()
|
||||
if route.type == RouteType.GRAPH
|
||||
)
|
||||
assert graph_count == count
|
||||
|
||||
|
||||
@then("I should have {count:d} routes")
|
||||
def step_verify_total_route_count(context: Context, count: int):
|
||||
"""Count total routes."""
|
||||
total_count = len(context.app.config.routes)
|
||||
assert total_count == count
|
||||
|
||||
|
||||
@when("I analyze route complexity")
|
||||
def step_analyze_complexity(context: Context):
|
||||
"""Analyze complexity of test routes."""
|
||||
context.complexity_results = {}
|
||||
|
||||
for name, route_data in context.test_routes.items():
|
||||
from cleveragents.reactive.route import RouteConfig
|
||||
|
||||
# Create RouteConfig from test data
|
||||
route_config = RouteConfig(
|
||||
name=name,
|
||||
type=route_data["type"],
|
||||
operators=route_data.get("operators", []),
|
||||
subscriptions=route_data.get("subscriptions", []),
|
||||
publications=route_data.get("publications", []),
|
||||
nodes=route_data.get("nodes", {}),
|
||||
edges=route_data.get("edges", []),
|
||||
checkpointing=route_data.get("checkpointing", False),
|
||||
)
|
||||
|
||||
# Analyze complexity
|
||||
analysis = RouteComplexityAnalyzer.analyze_route(route_config)
|
||||
context.complexity_results[name] = analysis
|
||||
|
||||
|
||||
@then("stream routes should have lower complexity scores")
|
||||
def step_verify_stream_complexity(context: Context):
|
||||
"""Verify stream complexity is lower."""
|
||||
stream_scores = [
|
||||
result["score"]
|
||||
for name, result in context.complexity_results.items()
|
||||
if result["type"] == "stream"
|
||||
]
|
||||
|
||||
assert all(score < 10 for score in stream_scores)
|
||||
|
||||
|
||||
@then("graph routes should have higher complexity scores")
|
||||
def step_verify_graph_complexity(context: Context):
|
||||
"""Verify graph complexity is higher."""
|
||||
graph_scores = [
|
||||
result["score"]
|
||||
for name, result in context.complexity_results.items()
|
||||
if result["type"] == "graph"
|
||||
]
|
||||
|
||||
assert all(score >= 5 for score in graph_scores)
|
||||
|
||||
|
||||
@then("routes with more operators should have higher scores")
|
||||
def step_verify_operator_complexity(context: Context):
|
||||
"""Verify operator count affects complexity."""
|
||||
simple_score = context.complexity_results["simple_stream"]["score"]
|
||||
complex_score = context.complexity_results["complex_stream"]["score"]
|
||||
|
||||
assert complex_score > simple_score
|
||||
|
||||
|
||||
@when("the upgrade conditions are met")
|
||||
def step_trigger_upgrade(context: Context):
|
||||
"""Simulate meeting upgrade conditions."""
|
||||
# This would trigger actual upgrade in real scenario
|
||||
context.upgrade_triggered = True
|
||||
|
||||
|
||||
@then("the route should upgrade to a graph")
|
||||
def step_verify_upgrade(context: Context):
|
||||
"""Verify route upgraded to graph."""
|
||||
assert context.upgrade_triggered
|
||||
|
||||
|
||||
@then("state should be preserved during conversion")
|
||||
def step_verify_state_preserved(context: Context):
|
||||
"""Verify state preservation."""
|
||||
# In real implementation, would check actual state
|
||||
assert True
|
||||
|
||||
|
||||
@then("subscriptions should be maintained")
|
||||
def step_verify_subscriptions_maintained(context: Context):
|
||||
"""Verify subscriptions maintained."""
|
||||
# In real implementation, would check actual subscriptions
|
||||
assert True
|
||||
@@ -0,0 +1,159 @@
|
||||
Feature: Unified Routes System
|
||||
As a developer using CleverAgents
|
||||
I want to use unified routes for both streams and graphs
|
||||
So that I have a consistent interface for all data flow patterns
|
||||
|
||||
Background:
|
||||
Given the CleverAgents reactive system is available
|
||||
|
||||
Scenario: Create a stream route
|
||||
Given I have a route configuration:
|
||||
"""
|
||||
agents:
|
||||
processor:
|
||||
type: llm
|
||||
config:
|
||||
provider: openai
|
||||
model: gpt-3.5-turbo
|
||||
|
||||
routes:
|
||||
process_route:
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
agent: processor
|
||||
publications:
|
||||
- __output__
|
||||
"""
|
||||
When I load the configuration
|
||||
Then the application should initialize successfully
|
||||
And I should have 1 route
|
||||
And the route "process_route" should be of type "stream"
|
||||
|
||||
Scenario: Create a graph route
|
||||
Given I have a route configuration:
|
||||
"""
|
||||
agents:
|
||||
analyzer:
|
||||
type: llm
|
||||
config:
|
||||
provider: openai
|
||||
model: gpt-4
|
||||
|
||||
routes:
|
||||
analysis_route:
|
||||
type: graph
|
||||
entry_point: start
|
||||
nodes:
|
||||
analyze:
|
||||
type: agent
|
||||
agent: analyzer
|
||||
edges:
|
||||
- source: start
|
||||
target: analyze
|
||||
- source: analyze
|
||||
target: end
|
||||
"""
|
||||
When I load the configuration
|
||||
Then the application should initialize successfully
|
||||
And I should have 1 route
|
||||
And the route "analysis_route" should be of type "graph"
|
||||
|
||||
Scenario: Route type is required
|
||||
Given I have a route configuration without type:
|
||||
"""
|
||||
routes:
|
||||
bad_route:
|
||||
operators:
|
||||
- type: map
|
||||
"""
|
||||
When I try to load the configuration
|
||||
Then I should get a configuration error
|
||||
And the error should mention "must specify a 'type' field"
|
||||
|
||||
Scenario: Stream route with bridging
|
||||
Given I have a route configuration with bridging:
|
||||
"""
|
||||
routes:
|
||||
adaptive_route:
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: map
|
||||
params:
|
||||
agent: processor
|
||||
bridge:
|
||||
upgrade_conditions:
|
||||
needs_state: true
|
||||
message_count: 5
|
||||
downgrade_conditions:
|
||||
idle_time: 300
|
||||
"""
|
||||
When I load the configuration
|
||||
Then the application should initialize successfully
|
||||
And the route "adaptive_route" should have bridge configuration
|
||||
And the bridge should have upgrade conditions
|
||||
|
||||
Scenario: Mixed routes (stream and graph)
|
||||
Given I have a configuration with both stream and graph routes:
|
||||
"""
|
||||
routes:
|
||||
input_stream:
|
||||
type: stream
|
||||
stream_type: cold
|
||||
operators:
|
||||
- type: debounce
|
||||
params:
|
||||
duration: 0.5
|
||||
publications:
|
||||
- processor_graph
|
||||
|
||||
processor_graph:
|
||||
type: graph
|
||||
nodes:
|
||||
process:
|
||||
type: agent
|
||||
agent: processor
|
||||
edges:
|
||||
- source: start
|
||||
target: process
|
||||
- source: process
|
||||
target: end
|
||||
"""
|
||||
When I load the configuration
|
||||
Then the application should initialize successfully
|
||||
And I should have 2 routes
|
||||
And I should have 1 stream route
|
||||
And I should have 1 graph route
|
||||
|
||||
Scenario: Route complexity analysis
|
||||
Given I have routes with different complexity levels
|
||||
When I analyze route complexity
|
||||
Then stream routes should have lower complexity scores
|
||||
And graph routes should have higher complexity scores
|
||||
And routes with more operators should have higher scores
|
||||
|
||||
Scenario: Dynamic type conversion
|
||||
Given I have a stream route with bridge configuration
|
||||
When the upgrade conditions are met
|
||||
Then the route should upgrade to a graph
|
||||
And state should be preserved during conversion
|
||||
And subscriptions should be maintained
|
||||
|
||||
Scenario: Bridge route type
|
||||
Given I have a bridge-only route configuration:
|
||||
"""
|
||||
routes:
|
||||
conversion_bridge:
|
||||
type: bridge
|
||||
bridge:
|
||||
upgrade_conditions:
|
||||
complexity_threshold: 10
|
||||
downgrade_conditions:
|
||||
idle_time: 600
|
||||
"""
|
||||
When I load the configuration
|
||||
Then the application should initialize successfully
|
||||
And the route "conversion_bridge" should be of type "bridge"
|
||||
Reference in New Issue
Block a user