Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| f5d187086c | |||
| 9a5ccc6b01 | |||
| f167098541 | |||
| 072f470212 | |||
| 1f95ea0c2a | |||
| 832d0b26ae | |||
| 89baa0a525 |
@@ -28,6 +28,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
correctly in all deployment modes: Docker containers, local pip installs
|
||||
(wheel or editable), and development environments.
|
||||
- **TDD Non-AssertionError Guard Visibility** (#8294): `apply_tdd_inversion` in
|
||||
- **bug-hunt-pool-supervisor Non-Blocking Tracking** (#8835): The automation-tracking-manager
|
||||
call in step 5 was blocking the main loop indefinitely, causing 3+ consecutive initialization
|
||||
failures. Step 5 now explicitly marks tracking as best-effort -- if the call does not complete
|
||||
within a reasonable time or fails, it is skipped and the supervisor continues to the next
|
||||
cycle. A new Rule 9 reinforces that tracking must never block the main loop; core
|
||||
functionality (module mapping, worker dispatch, monitoring) takes priority over status
|
||||
reporting.
|
||||
|
||||
`features/environment.py` now emits its non-assertion exception guard warning to
|
||||
both the structured logger and `stderr` via a new `_warning_with_stderr` helper.
|
||||
This makes the guard firing visible in standard Behave console output and CI log
|
||||
|
||||
@@ -0,0 +1,411 @@
|
||||
# Getting Started with CleverAgents
|
||||
|
||||
Welcome to CleverAgents! This guide will help you get up and running in about 5 minutes, walk you through your first project, and point you toward deeper learning resources.
|
||||
|
||||
## What is CleverAgents?
|
||||
|
||||
CleverAgents is a Python-first AI agent orchestration platform that lets you build, configure, and run intelligent automation workflows. It provides:
|
||||
|
||||
- **Unified CLI** (`agents` command) for all interactions
|
||||
- **Interactive TUI** (Terminal User Interface) for hands-on agent management
|
||||
- **Actor System** — composable AI agents with tools and skills
|
||||
- **Plan Lifecycle** — structured workflow from strategy to execution to application
|
||||
- **Resource Management** — handle files, databases, containers, and more
|
||||
- **Multi-Provider Support** — OpenAI, Anthropic, Google, Azure, and others
|
||||
|
||||
## Quick Start (5 Minutes)
|
||||
|
||||
### 1. Clone the Repository
|
||||
|
||||
```bash
|
||||
git clone https://git.cleverthis.com/cleveragents/cleveragents-core.git
|
||||
cd cleveragents-core
|
||||
```
|
||||
|
||||
### 2. Set Up Your Environment
|
||||
|
||||
```bash
|
||||
# Create a virtual environment
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate # On Windows: .venv\Scripts\activate
|
||||
|
||||
# Install CleverAgents with development dependencies
|
||||
pip install -e ".[dev,tests,docs]"
|
||||
|
||||
# Set up pre-commit hooks and verify tooling
|
||||
bash scripts/setup-dev.sh
|
||||
```
|
||||
|
||||
### 3. Verify Installation
|
||||
|
||||
```bash
|
||||
# Check the CLI is working
|
||||
agents --version
|
||||
agents --help
|
||||
|
||||
# Run diagnostics to check LLM provider configuration
|
||||
agents diagnostics
|
||||
```
|
||||
|
||||
### 4. Configure an LLM Provider
|
||||
|
||||
CleverAgents works with multiple LLM providers. Set up at least one:
|
||||
|
||||
**OpenAI:**
|
||||
```bash
|
||||
export OPENAI_API_KEY="sk-..."
|
||||
```
|
||||
|
||||
**Anthropic:**
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY="sk-ant-..."
|
||||
```
|
||||
|
||||
**Google:**
|
||||
```bash
|
||||
export GOOGLE_API_KEY="..."
|
||||
```
|
||||
|
||||
See [LLM Provider Configuration](#llm-provider-configuration) below for all supported providers.
|
||||
|
||||
### 5. Launch the Interactive TUI
|
||||
|
||||
```bash
|
||||
# Install the TUI extra if not already installed
|
||||
pip install -e ".[tui]"
|
||||
|
||||
# Launch the interactive terminal UI
|
||||
agents tui
|
||||
```
|
||||
|
||||
Inside the TUI, you can:
|
||||
- Type messages and press `Enter` to chat with the active actor
|
||||
- Press `/` to open the slash command overlay
|
||||
- Press `@` to insert file/resource references
|
||||
- Press `!` to enter shell mode
|
||||
- Press `F1` for context-sensitive help
|
||||
- Press `Ctrl+Q` to quit
|
||||
|
||||
## Your First Project: Hello World Agent
|
||||
|
||||
Let's create a simple agent that greets you and answers questions.
|
||||
|
||||
### Step 1: Create a Basic Actor Configuration
|
||||
|
||||
Create a file `my-first-actor.yaml`:
|
||||
|
||||
```yaml
|
||||
# Simple greeting actor
|
||||
name: local/hello-world
|
||||
entry_node: greeter
|
||||
nodes:
|
||||
greeter:
|
||||
model: gpt-4o # or your preferred model
|
||||
tool_sources: [builtin]
|
||||
system_prompt: |
|
||||
You are a friendly greeting agent.
|
||||
Respond warmly and helpfully to user messages.
|
||||
Keep responses concise and friendly.
|
||||
```
|
||||
|
||||
### Step 2: Use the Actor in the CLI
|
||||
|
||||
```bash
|
||||
# Tell the agent to do something
|
||||
agents tell --actor local/hello-world "Say hello and tell me what you can do"
|
||||
|
||||
# Or use the v3 plan workflow
|
||||
agents plan use local/hello-world my-project
|
||||
agents plan execute <PLAN_ID>
|
||||
agents plan apply <PLAN_ID>
|
||||
```
|
||||
|
||||
### Step 3: Use the Actor in the TUI
|
||||
|
||||
```bash
|
||||
# Launch the TUI
|
||||
agents tui
|
||||
|
||||
# Inside the TUI:
|
||||
# 1. Press Ctrl+T to cycle through available actors
|
||||
# 2. Select "local/hello-world"
|
||||
# 3. Type a message and press Enter
|
||||
```
|
||||
|
||||
## Basic Concepts
|
||||
|
||||
### Actors
|
||||
|
||||
**Actors** are the execution units of CleverAgents. Each actor:
|
||||
- Is defined in YAML with a name, entry node, and node graph
|
||||
- Binds an LLM, tools, and optional integrations (LSP, MCP)
|
||||
- Can be built-in (e.g., `openai/gpt-4o`) or custom (e.g., `local/my-actor`)
|
||||
|
||||
Example actor structure:
|
||||
```yaml
|
||||
name: local/my-actor
|
||||
entry_node: main
|
||||
nodes:
|
||||
main:
|
||||
model: gpt-4o
|
||||
tool_sources: [builtin, mcp://bash-tools]
|
||||
```
|
||||
|
||||
See [Actor System](../architecture.md#actor-system) for details.
|
||||
|
||||
### Tools
|
||||
|
||||
**Tools** are atomic capabilities available to actors. They include:
|
||||
- Built-in tools (file operations, shell commands)
|
||||
- MCP (Model Context Protocol) tools from external servers
|
||||
- LSP (Language Server Protocol) tools for code intelligence
|
||||
- Custom tools defined in your project
|
||||
|
||||
Tools are registered in the `ToolRegistry` and invoked by actors during execution.
|
||||
|
||||
### Skills
|
||||
|
||||
**Skills** are composable capability bundles that expose one or more tools. They:
|
||||
- Load from YAML or AgentSkills.io-compatible directories
|
||||
- Support progressive disclosure (discover → activate → deactivate)
|
||||
- Are tracked by the `SkillRegistry`
|
||||
|
||||
### Resources
|
||||
|
||||
**Resources** are managed external entities like files, databases, and containers. They:
|
||||
- Organize into a DAG (Directed Acyclic Graph) with dependency tracking
|
||||
- Support multiple types: `file`, `directory`, `sqlite`, `postgresql`, `container.docker`, etc.
|
||||
- Each type has a handler implementing CRUD, checkpoint, and rollback
|
||||
|
||||
### Plan Lifecycle
|
||||
|
||||
The **Plan Lifecycle** is the central workflow abstraction:
|
||||
|
||||
```
|
||||
Action → Strategize → Execute → Apply
|
||||
↑ ↓
|
||||
└──────────────────────┘
|
||||
(correction/rollback)
|
||||
```
|
||||
|
||||
| Phase | Description |
|
||||
|-------|-------------|
|
||||
| **Action** | User intent captured as a plan request |
|
||||
| **Strategize** | LLM generates a structured plan with operations |
|
||||
| **Execute** | Operations executed against resources via tools |
|
||||
| **Apply** | Validated changes committed; diff reviewed and approved |
|
||||
|
||||
See [Plan Lifecycle](../architecture.md#plan-lifecycle) for details.
|
||||
|
||||
### Personas
|
||||
|
||||
**Personas** are named identities that bind:
|
||||
- An actor (which LLM and tools to use)
|
||||
- Argument presets (default parameters)
|
||||
- Scope references (which resources are available)
|
||||
|
||||
Personas are persisted in `~/.config/cleveragents/personas/` and can be switched in the TUI with `Ctrl+T`.
|
||||
|
||||
### Sessions
|
||||
|
||||
**Sessions** are conversation histories. You can:
|
||||
- Create new sessions: `agents session create --actor openai/gpt-4o`
|
||||
- List sessions: `agents session list`
|
||||
- Export sessions: `agents session export --session-id <ID> --output session.json`
|
||||
- Import sessions: `agents session import --input session.json`
|
||||
|
||||
## LLM Provider Configuration
|
||||
|
||||
CleverAgents automatically discovers and uses configured LLM providers. Set environment variables for the providers you want to use:
|
||||
|
||||
| Provider | Environment Variable | Example |
|
||||
|----------|----------------------|---------|
|
||||
| OpenAI | `OPENAI_API_KEY` | `sk-...` |
|
||||
| Anthropic | `ANTHROPIC_API_KEY` | `sk-ant-...` |
|
||||
| Google | `GOOGLE_API_KEY` or `GOOGLE_GENAI_API_KEY` | `AIza...` |
|
||||
| Azure OpenAI | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT` | See Azure docs |
|
||||
| OpenRouter | `OPENROUTER_API_KEY` | `sk-or-...` |
|
||||
| Groq | `GROQ_API_KEY` | `gsk_...` |
|
||||
| Together | `TOGETHER_API_KEY` | `...` |
|
||||
| Cohere | `COHERE_API_KEY` | `...` |
|
||||
|
||||
### Setting a Default Provider
|
||||
|
||||
```bash
|
||||
# Pin the global provider
|
||||
export CLEVERAGENTS_DEFAULT_PROVIDER=openai
|
||||
|
||||
# Pin a specific model
|
||||
export CLEVERAGENTS_DEFAULT_MODEL=gpt-4o
|
||||
```
|
||||
|
||||
### Checking Your Configuration
|
||||
|
||||
```bash
|
||||
# See which providers are configured and which actor is selected
|
||||
agents diagnostics
|
||||
```
|
||||
|
||||
## Common First-Time Issues and Solutions
|
||||
|
||||
### Issue: "No LLM provider configured"
|
||||
|
||||
**Symptom:** Error message says no API keys found.
|
||||
|
||||
**Solution:**
|
||||
1. Verify you've set an environment variable: `echo $OPENAI_API_KEY`
|
||||
2. If empty, set it: `export OPENAI_API_KEY="sk-..."`
|
||||
3. Run `agents diagnostics` to verify the provider is detected
|
||||
4. Restart your terminal or shell session if you just set the variable
|
||||
|
||||
### Issue: "Actor not found"
|
||||
|
||||
**Symptom:** Error says `local/my-actor` doesn't exist.
|
||||
|
||||
**Solution:**
|
||||
1. Check the actor file exists: `ls my-first-actor.yaml`
|
||||
2. Verify the file is valid YAML (check indentation)
|
||||
3. Use the full path if the file is not in the current directory
|
||||
4. Built-in actors use the format `<provider>/<model>` (e.g., `openai/gpt-4o`)
|
||||
|
||||
### Issue: "TUI won't start"
|
||||
|
||||
**Symptom:** `agents tui` fails or shows a blank screen.
|
||||
|
||||
**Solution:**
|
||||
1. Ensure the TUI extra is installed: `pip install -e ".[tui]"`
|
||||
2. Check your terminal supports 256 colors: `echo $TERM`
|
||||
3. Try running with explicit terminal: `TERM=xterm-256color agents tui`
|
||||
4. Check for conflicting environment variables: `env | grep -i textual`
|
||||
|
||||
### Issue: "Tool execution fails"
|
||||
|
||||
**Symptom:** Actor tries to use a tool but gets an error.
|
||||
|
||||
**Solution:**
|
||||
1. Check the tool is available: `agents tools list` (if implemented)
|
||||
2. Verify tool permissions are granted (TUI shows permission overlay)
|
||||
3. Check tool configuration in the actor YAML
|
||||
4. Review tool documentation: see [Tool System](../architecture.md#tool-system)
|
||||
|
||||
### Issue: "Session not found"
|
||||
|
||||
**Symptom:** Error when trying to export or import a session.
|
||||
|
||||
**Solution:**
|
||||
1. List available sessions: `agents session list`
|
||||
2. Use the correct session ID from the list
|
||||
3. Check the session file exists (for import): `ls session.json`
|
||||
4. Verify the JSON is valid: `python -m json.tool session.json`
|
||||
|
||||
### Issue: "Permission denied" errors
|
||||
|
||||
**Symptom:** Actor can't read/write files or access resources.
|
||||
|
||||
**Solution:**
|
||||
1. Check file permissions: `ls -la <file>`
|
||||
2. Ensure the file is readable/writable by your user
|
||||
3. In the TUI, approve permission requests when prompted (press `y`)
|
||||
4. Check resource configuration in your project
|
||||
|
||||
## Next Learning Steps
|
||||
|
||||
Now that you're up and running, here's what to explore next:
|
||||
|
||||
### 1. **Understand the Architecture** (30 minutes)
|
||||
- Read [Architecture Overview](../architecture.md)
|
||||
- Learn about the layered design and key components
|
||||
- Understand the Plan Lifecycle in detail
|
||||
|
||||
### 2. **Build Your First Custom Actor** (1 hour)
|
||||
- Create a YAML actor configuration
|
||||
- Add tools and integrations
|
||||
- Test in the TUI and CLI
|
||||
- See [Actor System](../architecture.md#actor-system) for details
|
||||
|
||||
### 3. **Work with Resources** (1 hour)
|
||||
- Create file and database resources
|
||||
- Use resources in your actor
|
||||
- Understand the Resource DAG
|
||||
- See [Resource System](../architecture.md#resource-system)
|
||||
|
||||
### 4. **Explore Tools and Skills** (1 hour)
|
||||
- Discover available tools
|
||||
- Create custom tools
|
||||
- Load and manage skills
|
||||
- See [Tool System](../architecture.md#tool-system) and [Skill System](../architecture.md#skill-system)
|
||||
|
||||
### 5. **Master the Plan Lifecycle** (2 hours)
|
||||
- Create and execute plans
|
||||
- Understand phase transitions
|
||||
- Use correction and rollback
|
||||
- See [Plan Lifecycle](../architecture.md#plan-lifecycle)
|
||||
|
||||
### 6. **Integrate with External Services** (2 hours)
|
||||
- Set up MCP (Model Context Protocol) servers
|
||||
- Configure LSP (Language Server Protocol) integration
|
||||
- Use ACMS (Advanced Context Management System)
|
||||
- See [MCP Integration](../architecture.md#mcp-integration) and [LSP Integration](../architecture.md#lsp-integration)
|
||||
|
||||
### 7. **Deploy to Production** (2 hours)
|
||||
- Use server mode: `agents server connect`
|
||||
- Deploy with Kubernetes (see `k8s/`)
|
||||
- Configure observability and logging
|
||||
- See [Server Architecture](../development/agent-system-specification.md)
|
||||
|
||||
## Key Documentation References
|
||||
|
||||
| Topic | Document |
|
||||
|-------|----------|
|
||||
| **Architecture** | [Architecture Overview](../architecture.md) |
|
||||
| **Specification** | [Full Specification](../specification.md) |
|
||||
| **API Reference** | [API Docs](../api/index.md) |
|
||||
| **Development** | [Development Guide](../development/agent-system-specification.md) |
|
||||
| **Testing** | [Testing Guide](../development/testing.md) |
|
||||
| **FAQ** | [Frequently Asked Questions](../faq.md) |
|
||||
| **Design Decisions** | [Architecture Decision Records (ADRs)](../adr/index.md) |
|
||||
|
||||
## Tips for Success
|
||||
|
||||
1. **Start Small** — Create simple actors before complex ones
|
||||
2. **Use the TUI** — The interactive interface is great for learning and debugging
|
||||
3. **Read the Specification** — `docs/specification.md` is the authoritative source
|
||||
4. **Check the Examples** — See `examples/` for real-world configurations
|
||||
5. **Run Tests** — Use `nox -s unit_tests` to verify your changes
|
||||
6. **Ask for Help** — Check the FAQ and ADRs for common questions
|
||||
|
||||
## Troubleshooting Commands
|
||||
|
||||
```bash
|
||||
# Check your setup
|
||||
agents diagnostics
|
||||
|
||||
# List available actors
|
||||
agents actor list
|
||||
|
||||
# List available tools
|
||||
agents tools list # if implemented
|
||||
|
||||
# List sessions
|
||||
agents session list
|
||||
|
||||
# View help for any command
|
||||
agents <command> --help
|
||||
|
||||
# Run tests to verify everything works
|
||||
nox -s unit_tests
|
||||
|
||||
# Check code quality
|
||||
nox -s lint
|
||||
nox -s typecheck
|
||||
```
|
||||
|
||||
## What's Next?
|
||||
|
||||
- **Build an actor** — Create a custom actor for your use case
|
||||
- **Explore the TUI** — Spend time in the interactive interface
|
||||
- **Read the architecture** — Understand the design decisions
|
||||
- **Join the community** — Check out discussions and issues
|
||||
- **Contribute** — See [CONTRIBUTING.md](../../CONTRIBUTING.md) for guidelines
|
||||
|
||||
Happy automating! 🚀
|
||||
@@ -1,120 +0,0 @@
|
||||
# Provider Configuration Reference
|
||||
|
||||
CleverAgents supports multiple AI provider backends through the `ProviderRegistry`.
|
||||
Each provider is configured via environment variables or the `agents config set` command.
|
||||
|
||||
## Supported Providers
|
||||
|
||||
| Provider | `ProviderType` | API Key Environment Variable |
|
||||
|----------|---------------|------------------------------|
|
||||
| OpenAI | `openai` | `OPENAI_API_KEY` |
|
||||
| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` |
|
||||
| Google / Gemini | `google` / `gemini` | `GOOGLE_API_KEY` / `GEMINI_API_KEY` |
|
||||
| Azure OpenAI | `azure` | `AZURE_OPENAI_API_KEY` |
|
||||
| **OpenRouter** | **`openrouter`** | **`OPENROUTER_API_KEY`** |
|
||||
| Groq | `groq` | `GROQ_API_KEY` |
|
||||
| Together AI | `together` | `TOGETHER_API_KEY` |
|
||||
| Cohere | `cohere` | `COHERE_API_KEY` |
|
||||
|
||||
---
|
||||
|
||||
## OpenRouter
|
||||
|
||||
[OpenRouter](https://openrouter.ai) provides a unified API gateway to hundreds of
|
||||
models from different providers (Anthropic, OpenAI, Google, Meta, Mistral, and more).
|
||||
This makes it ideal for cost optimisation and model diversity without managing
|
||||
multiple API keys.
|
||||
|
||||
### Configuration
|
||||
|
||||
Set the following environment variables:
|
||||
|
||||
```bash
|
||||
# Required
|
||||
export OPENROUTER_API_KEY="sk-or-v1-..."
|
||||
|
||||
# Optional: default model (defaults to anthropic/claude-sonnet-4-20250514)
|
||||
export CLEVERAGENTS_DEFAULT_MODEL="openai/gpt-4o"
|
||||
|
||||
# Optional: set OpenRouter as the default provider
|
||||
export CLEVERAGENTS_DEFAULT_PROVIDER="openrouter"
|
||||
|
||||
# Optional: organisation identifier sent in HTTP headers
|
||||
export OPENROUTER_ORGANIZATION="myapp.example.com"
|
||||
```
|
||||
|
||||
Or use the CLI:
|
||||
|
||||
```bash
|
||||
agents config set provider.openrouter.api_key "sk-or-v1-..."
|
||||
agents config set provider.openrouter.model "anthropic/claude-3-haiku"
|
||||
```
|
||||
|
||||
### Config Keys
|
||||
|
||||
| Config Key | Environment Variable | Description |
|
||||
|------------|---------------------|-------------|
|
||||
| `provider.openrouter.api_key` | `OPENROUTER_API_KEY` | OpenRouter API key (required) |
|
||||
| `provider.openrouter.model` | `CLEVERAGENTS_DEFAULT_MODEL` | Default model slug (optional) |
|
||||
| `provider.openrouter.organization` | `OPENROUTER_ORGANIZATION` | Organisation name sent in `HTTP-Referer` / `X-Title` headers (optional) |
|
||||
|
||||
### Capabilities
|
||||
|
||||
| Feature | Supported |
|
||||
|---------|-----------|
|
||||
| Streaming | ✅ Yes |
|
||||
| Tool calls | ✅ Yes |
|
||||
| Vision / image input | ✅ Yes (model-dependent) |
|
||||
| JSON mode | ✅ Yes (model-dependent) |
|
||||
| Max context length | 128 000 tokens (varies by model) |
|
||||
|
||||
### Selecting a Model
|
||||
|
||||
OpenRouter model slugs follow the format `<provider>/<model-name>`. Examples:
|
||||
|
||||
```bash
|
||||
# Anthropic via OpenRouter
|
||||
agents config set provider.openrouter.model "anthropic/claude-sonnet-4-20250514"
|
||||
|
||||
# OpenAI via OpenRouter
|
||||
agents config set provider.openrouter.model "openai/gpt-4o"
|
||||
|
||||
# Meta Llama via OpenRouter
|
||||
agents config set provider.openrouter.model "meta-llama/llama-3.1-70b-instruct"
|
||||
```
|
||||
|
||||
Browse the full model catalogue at <https://openrouter.ai/models>.
|
||||
|
||||
### Fallback Priority
|
||||
|
||||
When no explicit provider is configured, CleverAgents selects the first available
|
||||
provider in the following fallback order:
|
||||
|
||||
1. OpenAI
|
||||
2. Anthropic
|
||||
3. Google
|
||||
4. Azure
|
||||
5. **OpenRouter** ← position 5
|
||||
6. Groq
|
||||
7. Together AI
|
||||
8. Cohere
|
||||
|
||||
### Example Usage
|
||||
|
||||
```python
|
||||
from cleveragents.providers.registry import ProviderRegistry, ProviderType
|
||||
|
||||
registry = ProviderRegistry()
|
||||
|
||||
# Create a LangChain LLM directly
|
||||
llm = registry.create_llm(
|
||||
provider_type=ProviderType.OPENROUTER,
|
||||
model_id="anthropic/claude-3-haiku",
|
||||
)
|
||||
|
||||
# Or create a full AIProviderInterface adapter
|
||||
provider = registry.create_ai_provider(
|
||||
provider_type="openrouter",
|
||||
model_id="openai/gpt-4o",
|
||||
)
|
||||
```
|
||||
+26
-5502
File diff suppressed because one or more lines are too long
@@ -1,87 +0,0 @@
|
||||
Feature: OpenRouter provider support in ProviderRegistry
|
||||
As a developer using CleverAgents
|
||||
I want ProviderRegistry to handle ProviderType.OPENROUTER in create_llm and _create_provider_llm
|
||||
So that OpenRouter models are accessible without raising ValueError
|
||||
|
||||
# Covers the core acceptance criterion:
|
||||
# ProviderRegistry.create_llm handles ProviderType.OPENROUTER without raising ValueError
|
||||
|
||||
@unit @providers @openrouter @registry
|
||||
Scenario: create_llm dispatches to OpenRouter via _create_provider_llm
|
||||
Given I have an openrouter registry with API key "sk-or-test-key"
|
||||
And the openrouter registry LangChain ChatOpenAI client is stubbed
|
||||
When I call create_llm on the openrouter registry for provider "openrouter" with model "anthropic/claude-3-haiku"
|
||||
Then the openrouter registry ChatOpenAI should be called with model "anthropic/claude-3-haiku"
|
||||
And the openrouter registry ChatOpenAI should use base url "https://openrouter.ai/api/v1"
|
||||
And the openrouter registry ChatOpenAI should use api key "sk-or-test-key"
|
||||
|
||||
@unit @providers @openrouter @registry
|
||||
Scenario: _create_provider_llm builds ChatOpenAI for OpenRouter with default model
|
||||
Given I have an openrouter registry with API key "sk-or-default"
|
||||
And the openrouter registry LangChain ChatOpenAI client is stubbed
|
||||
When I call _create_provider_llm on the openrouter registry for provider "openrouter" with model None
|
||||
Then the openrouter registry ChatOpenAI should be called with model "anthropic/claude-sonnet-4-20250514"
|
||||
And the openrouter registry ChatOpenAI should use base url "https://openrouter.ai/api/v1"
|
||||
|
||||
@unit @providers @openrouter @registry
|
||||
Scenario: _create_provider_llm injects organization headers when configured
|
||||
Given I have an openrouter registry with API key "sk-or-org" and organization "myapp.example.com"
|
||||
And the openrouter registry LangChain ChatOpenAI client is stubbed
|
||||
When I call _create_provider_llm on the openrouter registry for provider "openrouter" with model "openai/gpt-4o"
|
||||
Then the openrouter registry ChatOpenAI should include header "HTTP-Referer" with value "myapp.example.com"
|
||||
And the openrouter registry ChatOpenAI should include header "X-Title" with value "myapp.example.com"
|
||||
|
||||
@unit @providers @openrouter @registry
|
||||
Scenario: _create_provider_llm passes custom default_headers through
|
||||
Given I have an openrouter registry with API key "sk-or-headers"
|
||||
And the openrouter registry LangChain ChatOpenAI client is stubbed
|
||||
When I call _create_provider_llm on the openrouter registry for provider "openrouter" with model "openai/gpt-4o" and default_headers "X-Custom=trace-123"
|
||||
Then the openrouter registry ChatOpenAI should include header "X-Custom" with value "trace-123"
|
||||
|
||||
@unit @providers @openrouter @registry
|
||||
Scenario: create_llm raises ValueError when OpenRouter API key is missing
|
||||
Given I have an openrouter registry with no API keys
|
||||
When I try to call create_llm on the openrouter registry for provider "openrouter"
|
||||
Then an openrouter registry ValueError should be raised
|
||||
And the openrouter registry error should mention "OPENROUTER_API_KEY"
|
||||
|
||||
@unit @providers @openrouter @registry
|
||||
Scenario: ProviderType.OPENROUTER is in the registry fallback order
|
||||
Given I have the openrouter ProviderRegistry class
|
||||
When I check the openrouter FALLBACK_ORDER
|
||||
Then ProviderType.OPENROUTER should be in the fallback order
|
||||
|
||||
@unit @providers @openrouter @registry
|
||||
Scenario: OpenRouter default model is configured correctly
|
||||
Given I have the openrouter ProviderRegistry class
|
||||
When I check the openrouter DEFAULT_MODELS
|
||||
Then the openrouter default model should be "anthropic/claude-sonnet-4-20250514"
|
||||
|
||||
@unit @providers @openrouter @registry
|
||||
Scenario: OpenRouter capabilities are configured correctly
|
||||
Given I have the openrouter ProviderRegistry class
|
||||
When I check the openrouter DEFAULT_CAPABILITIES for OPENROUTER
|
||||
Then the openrouter provider supports_streaming should be True
|
||||
And the openrouter provider supports_tool_calls should be True
|
||||
And the openrouter provider supports_vision should be True
|
||||
And the openrouter provider max_context_length should be 128000
|
||||
And the openrouter provider supports_json_mode should be True
|
||||
|
||||
@unit @providers @openrouter @registry
|
||||
Scenario: OpenRouter provider is discovered when API key is set
|
||||
Given I have an openrouter registry with API key "sk-or-discover"
|
||||
When I call get_provider_info for OPENROUTER on the openrouter registry
|
||||
Then the openrouter provider info should be configured
|
||||
|
||||
@unit @providers @openrouter @registry
|
||||
Scenario: OpenRouter provider is not discovered when API key is missing
|
||||
Given I have an openrouter registry with no API keys
|
||||
When I call get_provider_info for OPENROUTER on the openrouter registry
|
||||
Then the openrouter provider info should not be configured
|
||||
|
||||
@unit @providers @openrouter @registry
|
||||
Scenario: create_llm selects OpenRouter as default when it is the only configured provider
|
||||
Given I have an openrouter registry with API key "sk-or-only"
|
||||
And the openrouter registry LangChain ChatOpenAI client is stubbed
|
||||
When I call create_llm on the openrouter registry without specifying a provider
|
||||
Then the openrouter registry ChatOpenAI should be called with model "anthropic/claude-sonnet-4-20250514"
|
||||
@@ -1,374 +0,0 @@
|
||||
"""Step definitions for openrouter_provider_registry.feature.
|
||||
|
||||
Tests that ProviderRegistry.create_llm and _create_provider_llm correctly
|
||||
dispatch to the OpenRouter ChatOpenAI backend without raising ValueError.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import types
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from behave import given, then, use_step_matcher, when # type: ignore[import-untyped]
|
||||
|
||||
from cleveragents.providers.registry import (
|
||||
ProviderRegistry,
|
||||
ProviderType,
|
||||
)
|
||||
|
||||
_OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_or_settings(
|
||||
openrouter: str | None = None,
|
||||
organization: str | None = None,
|
||||
) -> MagicMock:
|
||||
"""Return a minimal fake Settings object for OpenRouter tests."""
|
||||
settings = MagicMock()
|
||||
settings.openai_api_key = None
|
||||
settings.anthropic_api_key = None
|
||||
settings.google_api_key = None
|
||||
settings.gemini_api_key = None
|
||||
settings.azure_api_key = None
|
||||
settings.openrouter_api_key = openrouter
|
||||
settings.cohere_api_key = None
|
||||
settings.groq_api_key = None
|
||||
settings.together_api_key = None
|
||||
settings.default_provider = None
|
||||
settings.default_model = None
|
||||
settings.azure_openai_endpoint = None
|
||||
settings.azure_openai_api_version = None
|
||||
settings.azure_openai_deployment = None
|
||||
settings.openrouter_organization = organization
|
||||
return settings
|
||||
|
||||
|
||||
def _stub_chat_openai(context: Any) -> None:
|
||||
"""Replace langchain_openai.ChatOpenAI with a recording stub."""
|
||||
original = sys.modules.get("langchain_openai")
|
||||
calls: list[dict[str, Any]] = []
|
||||
|
||||
class FakeChatOpenAI:
|
||||
def __init__(self, **kwargs: Any) -> None:
|
||||
calls.append(kwargs)
|
||||
|
||||
class FakeAzureChatOpenAI:
|
||||
def __init__(self, **kwargs: Any) -> None:
|
||||
pass
|
||||
|
||||
stub = types.ModuleType("langchain_openai")
|
||||
stub.ChatOpenAI = FakeChatOpenAI # type: ignore[attr-defined]
|
||||
stub.AzureChatOpenAI = FakeAzureChatOpenAI # type: ignore[attr-defined]
|
||||
sys.modules["langchain_openai"] = stub
|
||||
|
||||
context._or_chat_openai_calls = calls
|
||||
context._or_chat_openai_cls = FakeChatOpenAI
|
||||
|
||||
if not hasattr(context, "_or_cleanup"):
|
||||
context._or_cleanup = []
|
||||
|
||||
def _restore() -> None:
|
||||
if original is None:
|
||||
sys.modules.pop("langchain_openai", None)
|
||||
else:
|
||||
sys.modules["langchain_openai"] = original
|
||||
|
||||
context._or_cleanup.append(_restore)
|
||||
|
||||
|
||||
def _parse_headers_string(headers_string: str) -> dict[str, str]:
|
||||
"""Parse 'Key=Value,Key2=Value2' into a dict."""
|
||||
result: dict[str, str] = {}
|
||||
for entry in headers_string.split(","):
|
||||
entry = entry.strip()
|
||||
if "=" in entry:
|
||||
key, value = entry.split("=", 1)
|
||||
result[key.strip()] = value.strip()
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Given
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
use_step_matcher("re")
|
||||
|
||||
|
||||
@given(
|
||||
r'I have an openrouter registry with API key "(?P<or_api_key>[^"]+)" and organization "(?P<or_org>[^"]+)"'
|
||||
)
|
||||
def step_or_registry_with_key_and_org(
|
||||
context: Any, or_api_key: str, or_org: str
|
||||
) -> None:
|
||||
context.or_registry = ProviderRegistry(
|
||||
settings=_make_or_settings(openrouter=or_api_key, organization=or_org)
|
||||
)
|
||||
|
||||
|
||||
@given(r'I have an openrouter registry with API key "(?P<or_api_key>[^"]+)"')
|
||||
def step_or_registry_with_key(context: Any, or_api_key: str) -> None:
|
||||
context.or_registry = ProviderRegistry(
|
||||
settings=_make_or_settings(openrouter=or_api_key)
|
||||
)
|
||||
|
||||
|
||||
@given(r"I have an openrouter registry with no API keys")
|
||||
def step_or_registry_no_keys(context: Any) -> None:
|
||||
context.or_registry = ProviderRegistry(settings=_make_or_settings())
|
||||
|
||||
|
||||
@given(r"I have the openrouter ProviderRegistry class")
|
||||
def step_or_registry_class(context: Any) -> None:
|
||||
context.or_registry_class = ProviderRegistry
|
||||
|
||||
|
||||
@given(r"the openrouter registry LangChain ChatOpenAI client is stubbed")
|
||||
def step_or_stub_chat_openai(context: Any) -> None:
|
||||
_stub_chat_openai(context)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# When
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@when(
|
||||
r'I call create_llm on the openrouter registry for provider "openrouter" with model "(?P<or_model>[^"]+)"'
|
||||
)
|
||||
def step_or_create_llm_with_model(context: Any, or_model: str) -> None:
|
||||
context.or_error = None
|
||||
try:
|
||||
context.or_result = context.or_registry.create_llm(
|
||||
provider_type="openrouter", model_id=or_model
|
||||
)
|
||||
except ValueError as exc:
|
||||
context.or_error = exc
|
||||
|
||||
|
||||
@when(r"I call create_llm on the openrouter registry without specifying a provider")
|
||||
def step_or_create_llm_default(context: Any) -> None:
|
||||
context.or_error = None
|
||||
try:
|
||||
context.or_result = context.or_registry.create_llm()
|
||||
except ValueError as exc:
|
||||
context.or_error = exc
|
||||
|
||||
|
||||
@when(
|
||||
r'I call _create_provider_llm on the openrouter registry for provider "openrouter" with model None'
|
||||
)
|
||||
def step_or_private_llm_none_model(context: Any) -> None:
|
||||
context.or_error = None
|
||||
try:
|
||||
context.or_result = context.or_registry._create_provider_llm(
|
||||
ProviderType.OPENROUTER, None
|
||||
)
|
||||
except ValueError as exc:
|
||||
context.or_error = exc
|
||||
|
||||
|
||||
@when(
|
||||
r'I call _create_provider_llm on the openrouter registry for provider "openrouter" with model "(?P<or_model>[^"]+)" and default_headers "(?P<or_headers_string>[^"]+)"'
|
||||
)
|
||||
def step_or_private_llm_with_headers(
|
||||
context: Any, or_model: str, or_headers_string: str
|
||||
) -> None:
|
||||
headers = _parse_headers_string(or_headers_string)
|
||||
context.or_error = None
|
||||
try:
|
||||
context.or_result = context.or_registry._create_provider_llm(
|
||||
ProviderType.OPENROUTER, or_model, default_headers=headers
|
||||
)
|
||||
except ValueError as exc:
|
||||
context.or_error = exc
|
||||
|
||||
|
||||
@when(
|
||||
r'I call _create_provider_llm on the openrouter registry for provider "openrouter" with model "(?P<or_model>[^"]+)"'
|
||||
)
|
||||
def step_or_private_llm_with_model(context: Any, or_model: str) -> None:
|
||||
context.or_error = None
|
||||
try:
|
||||
context.or_result = context.or_registry._create_provider_llm(
|
||||
ProviderType.OPENROUTER, or_model
|
||||
)
|
||||
except ValueError as exc:
|
||||
context.or_error = exc
|
||||
|
||||
|
||||
@when(r'I try to call create_llm on the openrouter registry for provider "openrouter"')
|
||||
def step_or_try_create_llm_no_key(context: Any) -> None:
|
||||
context.or_error = None
|
||||
try:
|
||||
context.or_registry.create_llm(provider_type="openrouter")
|
||||
except ValueError as exc:
|
||||
context.or_error = exc
|
||||
|
||||
|
||||
@when(r"I check the openrouter FALLBACK_ORDER")
|
||||
def step_or_check_fallback_order(context: Any) -> None:
|
||||
context.or_fallback_order = context.or_registry_class.FALLBACK_ORDER
|
||||
|
||||
|
||||
@when(r"I check the openrouter DEFAULT_MODELS")
|
||||
def step_or_check_default_models(context: Any) -> None:
|
||||
context.or_default_models = context.or_registry_class.DEFAULT_MODELS
|
||||
|
||||
|
||||
@when(r"I check the openrouter DEFAULT_CAPABILITIES for OPENROUTER")
|
||||
def step_or_check_capabilities(context: Any) -> None:
|
||||
context.or_capabilities = context.or_registry_class.DEFAULT_CAPABILITIES.get(
|
||||
ProviderType.OPENROUTER
|
||||
)
|
||||
|
||||
|
||||
@when(r"I call get_provider_info for OPENROUTER on the openrouter registry")
|
||||
def step_or_get_provider_info(context: Any) -> None:
|
||||
context.or_provider_info = context.or_registry.get_provider_info(
|
||||
ProviderType.OPENROUTER
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Then
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@then(
|
||||
r'the openrouter registry ChatOpenAI should be called with model "(?P<or_expected_model>[^"]+)"'
|
||||
)
|
||||
def step_or_assert_model(context: Any, or_expected_model: str) -> None:
|
||||
calls = getattr(context, "_or_chat_openai_calls", [])
|
||||
assert calls, "Expected ChatOpenAI to be instantiated"
|
||||
actual_model = calls[-1].get("model")
|
||||
assert actual_model == or_expected_model, (
|
||||
f"Expected model={or_expected_model!r}, got {actual_model!r}"
|
||||
)
|
||||
|
||||
|
||||
@then(
|
||||
r'the openrouter registry ChatOpenAI should use base url "(?P<or_expected_url>[^"]+)"'
|
||||
)
|
||||
def step_or_assert_base_url(context: Any, or_expected_url: str) -> None:
|
||||
calls = getattr(context, "_or_chat_openai_calls", [])
|
||||
assert calls, "Expected ChatOpenAI to be instantiated"
|
||||
actual_url = calls[-1].get("openai_api_base")
|
||||
assert actual_url == or_expected_url, (
|
||||
f"Expected openai_api_base={or_expected_url!r}, got {actual_url!r}"
|
||||
)
|
||||
|
||||
|
||||
@then(
|
||||
r'the openrouter registry ChatOpenAI should use api key "(?P<or_expected_key>[^"]+)"'
|
||||
)
|
||||
def step_or_assert_api_key(context: Any, or_expected_key: str) -> None:
|
||||
calls = getattr(context, "_or_chat_openai_calls", [])
|
||||
assert calls, "Expected ChatOpenAI to be instantiated"
|
||||
actual_key = calls[-1].get("openai_api_key")
|
||||
assert actual_key == or_expected_key, (
|
||||
f"Expected openai_api_key={or_expected_key!r}, got {actual_key!r}"
|
||||
)
|
||||
|
||||
|
||||
@then(
|
||||
r'the openrouter registry ChatOpenAI should include header "(?P<or_header_name>[^"]+)" with value "(?P<or_expected_value>[^"]+)"'
|
||||
)
|
||||
def step_or_assert_header(
|
||||
context: Any, or_header_name: str, or_expected_value: str
|
||||
) -> None:
|
||||
calls = getattr(context, "_or_chat_openai_calls", [])
|
||||
assert calls, "Expected ChatOpenAI to be instantiated"
|
||||
headers = calls[-1].get("default_headers")
|
||||
assert isinstance(headers, dict), (
|
||||
f"Expected default_headers to be a dict, got {type(headers)}"
|
||||
)
|
||||
actual_value = headers.get(or_header_name)
|
||||
assert actual_value == or_expected_value, (
|
||||
f"Expected header {or_header_name}={or_expected_value!r}, got {actual_value!r}"
|
||||
)
|
||||
|
||||
|
||||
@then(r"an openrouter registry ValueError should be raised")
|
||||
def step_or_assert_value_error(context: Any) -> None:
|
||||
assert context.or_error is not None, "Expected a ValueError to be raised"
|
||||
assert isinstance(context.or_error, ValueError), (
|
||||
f"Expected ValueError, got {type(context.or_error).__name__}"
|
||||
)
|
||||
|
||||
|
||||
@then(r'the openrouter registry error should mention "(?P<or_fragment>[^"]+)"')
|
||||
def step_or_assert_error_contains(context: Any, or_fragment: str) -> None:
|
||||
assert context.or_error is not None, "Expected an error to be set"
|
||||
message = str(context.or_error)
|
||||
assert or_fragment in message, (
|
||||
f"Expected error to contain {or_fragment!r}, got: {message!r}"
|
||||
)
|
||||
|
||||
|
||||
@then(r"ProviderType\.OPENROUTER should be in the fallback order")
|
||||
def step_or_assert_in_fallback(context: Any) -> None:
|
||||
assert ProviderType.OPENROUTER in context.or_fallback_order, (
|
||||
"Expected ProviderType.OPENROUTER to be in FALLBACK_ORDER"
|
||||
)
|
||||
|
||||
|
||||
@then(r'the openrouter default model should be "(?P<or_expected_model>[^"]+)"')
|
||||
def step_or_assert_default_model(context: Any, or_expected_model: str) -> None:
|
||||
actual = context.or_default_models.get(ProviderType.OPENROUTER)
|
||||
assert actual == or_expected_model, (
|
||||
f"Expected default model={or_expected_model!r}, got {actual!r}"
|
||||
)
|
||||
|
||||
|
||||
@then(r"the openrouter provider supports_streaming should be (?P<or_expected>\w+)")
|
||||
def step_or_assert_streaming(context: Any, or_expected: str) -> None:
|
||||
assert context.or_capabilities is not None
|
||||
assert context.or_capabilities.supports_streaming is (or_expected == "True")
|
||||
|
||||
|
||||
@then(r"the openrouter provider supports_tool_calls should be (?P<or_expected>\w+)")
|
||||
def step_or_assert_tool_calls(context: Any, or_expected: str) -> None:
|
||||
assert context.or_capabilities is not None
|
||||
assert context.or_capabilities.supports_tool_calls is (or_expected == "True")
|
||||
|
||||
|
||||
@then(r"the openrouter provider supports_vision should be (?P<or_expected>\w+)")
|
||||
def step_or_assert_vision(context: Any, or_expected: str) -> None:
|
||||
assert context.or_capabilities is not None
|
||||
assert context.or_capabilities.supports_vision is (or_expected == "True")
|
||||
|
||||
|
||||
@then(r"the openrouter provider max_context_length should be (?P<or_expected>\d+)")
|
||||
def step_or_assert_context_length(context: Any, or_expected: str) -> None:
|
||||
assert context.or_capabilities is not None
|
||||
expected_int = int(or_expected)
|
||||
assert context.or_capabilities.max_context_length == expected_int, (
|
||||
f"Expected max_context_length={expected_int}, got {context.or_capabilities.max_context_length}"
|
||||
)
|
||||
|
||||
|
||||
@then(r"the openrouter provider supports_json_mode should be (?P<or_expected>\w+)")
|
||||
def step_or_assert_json_mode(context: Any, or_expected: str) -> None:
|
||||
assert context.or_capabilities is not None
|
||||
assert context.or_capabilities.supports_json_mode is (or_expected == "True")
|
||||
|
||||
|
||||
@then(r"the openrouter provider info should be configured")
|
||||
def step_or_assert_provider_configured(context: Any) -> None:
|
||||
info = context.or_provider_info
|
||||
assert info is not None, "Expected provider info to exist"
|
||||
assert info.is_configured is True, "Expected provider to be configured"
|
||||
|
||||
|
||||
@then(r"the openrouter provider info should not be configured")
|
||||
def step_or_assert_provider_not_configured(context: Any) -> None:
|
||||
info = context.or_provider_info
|
||||
assert info is not None, "Expected provider info to exist"
|
||||
assert info.is_configured is False, "Expected provider to not be configured"
|
||||
@@ -11,6 +11,8 @@ site_dir: build/site
|
||||
nav:
|
||||
- Specification: specification.md
|
||||
- Architecture: architecture.md
|
||||
- Guides:
|
||||
- Getting Started: guides/getting-started.md
|
||||
- API Reference:
|
||||
- Overview: api/index.md
|
||||
- Core Utilities: api/core.md
|
||||
|
||||
@@ -525,32 +525,6 @@ class ProviderRegistry:
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
if provider_type == ProviderType.OPENROUTER:
|
||||
from langchain_openai import ChatOpenAI
|
||||
|
||||
from cleveragents.providers.llm.openrouter_provider import (
|
||||
OpenRouterChatProvider,
|
||||
)
|
||||
|
||||
key_attr = self.PROVIDER_KEY_ATTRS.get(provider_type)
|
||||
api_key = getattr(self._settings, key_attr, None) if key_attr else None
|
||||
organization = getattr(self._settings, "openrouter_organization", None)
|
||||
default_headers: dict[str, str] | None = kwargs.pop("default_headers", None)
|
||||
if organization:
|
||||
if default_headers is None:
|
||||
default_headers = {}
|
||||
default_headers.setdefault("HTTP-Referer", organization)
|
||||
default_headers.setdefault("X-Title", organization)
|
||||
openrouter_kwargs: dict[str, Any] = {
|
||||
"model": model_id or "anthropic/claude-sonnet-4-20250514",
|
||||
"openai_api_base": OpenRouterChatProvider._BASE_URL,
|
||||
"openai_api_key": api_key or "",
|
||||
}
|
||||
if default_headers:
|
||||
openrouter_kwargs["default_headers"] = default_headers
|
||||
openrouter_kwargs.update(kwargs)
|
||||
return ChatOpenAI(**openrouter_kwargs)
|
||||
|
||||
if provider_type == ProviderType.GROQ:
|
||||
from langchain_groq import ChatGroq
|
||||
|
||||
|
||||
Reference in New Issue
Block a user