Compare commits

..

7 Commits

Author SHA1 Message Date
HAL9000 f5d187086c docs: add getting started tutorial 2026-04-19 11:43:46 +00:00
HAL9000 9a5ccc6b01 Merge pull request 'Update timeline: milestone status for 2026-04-19' (#10684) from timeline-update-2026-04-19-final into master
CI / benchmark-regression (push) Failing after 0s
CI / benchmark-publish (push) Failing after 0s
CI / helm (push) Successful in 40s
CI / push-validation (push) Successful in 22s
CI / build (push) Successful in 3m57s
CI / lint (push) Successful in 4m13s
CI / quality (push) Successful in 4m34s
CI / typecheck (push) Successful in 4m56s
CI / security (push) Successful in 5m4s
CI / e2e_tests (push) Successful in 8m2s
CI / integration_tests (push) Successful in 8m8s
CI / unit_tests (push) Successful in 9m27s
CI / coverage (push) Successful in 14m53s
CI / docker (push) Successful in 1m49s
CI / status-check (push) Successful in 3s
2026-04-19 06:18:05 +00:00
HAL9000 f167098541 Merge branch 'master' into timeline-update-2026-04-19-final
CI / helm (pull_request) Successful in 42s
CI / lint (pull_request) Successful in 3m58s
CI / build (pull_request) Successful in 4m1s
CI / typecheck (pull_request) Successful in 4m40s
CI / quality (pull_request) Successful in 4m42s
CI / security (pull_request) Successful in 5m22s
CI / push-validation (pull_request) Successful in 23s
CI / integration_tests (pull_request) Successful in 8m7s
CI / e2e_tests (pull_request) Successful in 8m31s
CI / unit_tests (pull_request) Successful in 9m44s
CI / coverage (pull_request) Successful in 14m52s
CI / docker (pull_request) Successful in 2m3s
CI / status-check (pull_request) Successful in 3s
2026-04-19 05:27:54 +00:00
HAL9000 072f470212 fix(agents): make bug-hunt-pool-supervisor tracking non-blocking to prevent initialization hangs
CI / helm (pull_request) Successful in 40s
CI / push-validation (pull_request) Successful in 23s
CI / lint (pull_request) Successful in 4m0s
CI / typecheck (pull_request) Successful in 4m23s
CI / build (pull_request) Successful in 3m36s
CI / quality (pull_request) Successful in 4m15s
CI / security (pull_request) Successful in 4m37s
CI / e2e_tests (pull_request) Successful in 7m43s
CI / integration_tests (pull_request) Successful in 8m44s
CI / unit_tests (pull_request) Successful in 9m18s
CI / docker (pull_request) Successful in 1m37s
CI / coverage (pull_request) Successful in 14m53s
CI / benchmark-regression (push) Failing after 0s
CI / benchmark-publish (push) Failing after 0s
CI / status-check (pull_request) Successful in 5s
CI / helm (push) Successful in 32s
CI / build (push) Successful in 3m48s
CI / lint (push) Successful in 3m55s
CI / quality (push) Successful in 4m23s
CI / typecheck (push) Successful in 4m48s
CI / security (push) Successful in 4m51s
CI / push-validation (push) Successful in 23s
CI / e2e_tests (push) Successful in 7m30s
CI / integration_tests (push) Successful in 7m44s
CI / unit_tests (push) Successful in 9m11s
CI / docker (push) Successful in 1m38s
CI / coverage (push) Successful in 14m50s
CI / status-check (push) Successful in 13s
The automation-tracking-manager call in step 5 was blocking the main loop
indefinitely, causing 3+ consecutive initialization failures. This commit
documents the fix in CHANGELOG.md with the proper issue reference.

ISSUES CLOSED: #8835
2026-04-19 04:02:17 +00:00
HAL9000 1f95ea0c2a Update timeline: milestone status for 2026-04-19
CI / push-validation (pull_request) Successful in 35s
CI / helm (pull_request) Successful in 38s
CI / lint (pull_request) Successful in 4m2s
CI / quality (pull_request) Successful in 4m18s
CI / typecheck (pull_request) Successful in 4m43s
CI / security (pull_request) Successful in 4m48s
CI / build (pull_request) Successful in 3m48s
CI / e2e_tests (pull_request) Successful in 6m57s
CI / integration_tests (pull_request) Successful in 7m48s
CI / unit_tests (pull_request) Successful in 10m11s
CI / docker (pull_request) Successful in 1m37s
CI / coverage (pull_request) Successful in 14m56s
CI / status-check (pull_request) Successful in 5s
2026-04-19 04:00:21 +00:00
HAL9000 832d0b26ae Update timeline: milestone status for 2026-04-19 2026-04-19 03:59:57 +00:00
HAL9000 89baa0a525 Update timeline: milestone status for 2026-04-19 2026-04-19 03:59:35 +00:00
8 changed files with 447 additions and 6109 deletions
+8
View File
@@ -28,6 +28,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
correctly in all deployment modes: Docker containers, local pip installs
(wheel or editable), and development environments.
- **TDD Non-AssertionError Guard Visibility** (#8294): `apply_tdd_inversion` in
- **bug-hunt-pool-supervisor Non-Blocking Tracking** (#8835): The automation-tracking-manager
call in step 5 was blocking the main loop indefinitely, causing 3+ consecutive initialization
failures. Step 5 now explicitly marks tracking as best-effort -- if the call does not complete
within a reasonable time or fails, it is skipped and the supervisor continues to the next
cycle. A new Rule 9 reinforces that tracking must never block the main loop; core
functionality (module mapping, worker dispatch, monitoring) takes priority over status
reporting.
`features/environment.py` now emits its non-assertion exception guard warning to
both the structured logger and `stderr` via a new `_warning_with_stderr` helper.
This makes the guard firing visible in standard Behave console output and CI log
+411
View File
@@ -0,0 +1,411 @@
# Getting Started with CleverAgents
Welcome to CleverAgents! This guide will help you get up and running in about 5 minutes, walk you through your first project, and point you toward deeper learning resources.
## What is CleverAgents?
CleverAgents is a Python-first AI agent orchestration platform that lets you build, configure, and run intelligent automation workflows. It provides:
- **Unified CLI** (`agents` command) for all interactions
- **Interactive TUI** (Terminal User Interface) for hands-on agent management
- **Actor System** — composable AI agents with tools and skills
- **Plan Lifecycle** — structured workflow from strategy to execution to application
- **Resource Management** — handle files, databases, containers, and more
- **Multi-Provider Support** — OpenAI, Anthropic, Google, Azure, and others
## Quick Start (5 Minutes)
### 1. Clone the Repository
```bash
git clone https://git.cleverthis.com/cleveragents/cleveragents-core.git
cd cleveragents-core
```
### 2. Set Up Your Environment
```bash
# Create a virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install CleverAgents with development dependencies
pip install -e ".[dev,tests,docs]"
# Set up pre-commit hooks and verify tooling
bash scripts/setup-dev.sh
```
### 3. Verify Installation
```bash
# Check the CLI is working
agents --version
agents --help
# Run diagnostics to check LLM provider configuration
agents diagnostics
```
### 4. Configure an LLM Provider
CleverAgents works with multiple LLM providers. Set up at least one:
**OpenAI:**
```bash
export OPENAI_API_KEY="sk-..."
```
**Anthropic:**
```bash
export ANTHROPIC_API_KEY="sk-ant-..."
```
**Google:**
```bash
export GOOGLE_API_KEY="..."
```
See [LLM Provider Configuration](#llm-provider-configuration) below for all supported providers.
### 5. Launch the Interactive TUI
```bash
# Install the TUI extra if not already installed
pip install -e ".[tui]"
# Launch the interactive terminal UI
agents tui
```
Inside the TUI, you can:
- Type messages and press `Enter` to chat with the active actor
- Press `/` to open the slash command overlay
- Press `@` to insert file/resource references
- Press `!` to enter shell mode
- Press `F1` for context-sensitive help
- Press `Ctrl+Q` to quit
## Your First Project: Hello World Agent
Let's create a simple agent that greets you and answers questions.
### Step 1: Create a Basic Actor Configuration
Create a file `my-first-actor.yaml`:
```yaml
# Simple greeting actor
name: local/hello-world
entry_node: greeter
nodes:
greeter:
model: gpt-4o # or your preferred model
tool_sources: [builtin]
system_prompt: |
You are a friendly greeting agent.
Respond warmly and helpfully to user messages.
Keep responses concise and friendly.
```
### Step 2: Use the Actor in the CLI
```bash
# Tell the agent to do something
agents tell --actor local/hello-world "Say hello and tell me what you can do"
# Or use the v3 plan workflow
agents plan use local/hello-world my-project
agents plan execute <PLAN_ID>
agents plan apply <PLAN_ID>
```
### Step 3: Use the Actor in the TUI
```bash
# Launch the TUI
agents tui
# Inside the TUI:
# 1. Press Ctrl+T to cycle through available actors
# 2. Select "local/hello-world"
# 3. Type a message and press Enter
```
## Basic Concepts
### Actors
**Actors** are the execution units of CleverAgents. Each actor:
- Is defined in YAML with a name, entry node, and node graph
- Binds an LLM, tools, and optional integrations (LSP, MCP)
- Can be built-in (e.g., `openai/gpt-4o`) or custom (e.g., `local/my-actor`)
Example actor structure:
```yaml
name: local/my-actor
entry_node: main
nodes:
main:
model: gpt-4o
tool_sources: [builtin, mcp://bash-tools]
```
See [Actor System](../architecture.md#actor-system) for details.
### Tools
**Tools** are atomic capabilities available to actors. They include:
- Built-in tools (file operations, shell commands)
- MCP (Model Context Protocol) tools from external servers
- LSP (Language Server Protocol) tools for code intelligence
- Custom tools defined in your project
Tools are registered in the `ToolRegistry` and invoked by actors during execution.
### Skills
**Skills** are composable capability bundles that expose one or more tools. They:
- Load from YAML or AgentSkills.io-compatible directories
- Support progressive disclosure (discover → activate → deactivate)
- Are tracked by the `SkillRegistry`
### Resources
**Resources** are managed external entities like files, databases, and containers. They:
- Organize into a DAG (Directed Acyclic Graph) with dependency tracking
- Support multiple types: `file`, `directory`, `sqlite`, `postgresql`, `container.docker`, etc.
- Each type has a handler implementing CRUD, checkpoint, and rollback
### Plan Lifecycle
The **Plan Lifecycle** is the central workflow abstraction:
```
Action → Strategize → Execute → Apply
↑ ↓
└──────────────────────┘
(correction/rollback)
```
| Phase | Description |
|-------|-------------|
| **Action** | User intent captured as a plan request |
| **Strategize** | LLM generates a structured plan with operations |
| **Execute** | Operations executed against resources via tools |
| **Apply** | Validated changes committed; diff reviewed and approved |
See [Plan Lifecycle](../architecture.md#plan-lifecycle) for details.
### Personas
**Personas** are named identities that bind:
- An actor (which LLM and tools to use)
- Argument presets (default parameters)
- Scope references (which resources are available)
Personas are persisted in `~/.config/cleveragents/personas/` and can be switched in the TUI with `Ctrl+T`.
### Sessions
**Sessions** are conversation histories. You can:
- Create new sessions: `agents session create --actor openai/gpt-4o`
- List sessions: `agents session list`
- Export sessions: `agents session export --session-id <ID> --output session.json`
- Import sessions: `agents session import --input session.json`
## LLM Provider Configuration
CleverAgents automatically discovers and uses configured LLM providers. Set environment variables for the providers you want to use:
| Provider | Environment Variable | Example |
|----------|----------------------|---------|
| OpenAI | `OPENAI_API_KEY` | `sk-...` |
| Anthropic | `ANTHROPIC_API_KEY` | `sk-ant-...` |
| Google | `GOOGLE_API_KEY` or `GOOGLE_GENAI_API_KEY` | `AIza...` |
| Azure OpenAI | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT` | See Azure docs |
| OpenRouter | `OPENROUTER_API_KEY` | `sk-or-...` |
| Groq | `GROQ_API_KEY` | `gsk_...` |
| Together | `TOGETHER_API_KEY` | `...` |
| Cohere | `COHERE_API_KEY` | `...` |
### Setting a Default Provider
```bash
# Pin the global provider
export CLEVERAGENTS_DEFAULT_PROVIDER=openai
# Pin a specific model
export CLEVERAGENTS_DEFAULT_MODEL=gpt-4o
```
### Checking Your Configuration
```bash
# See which providers are configured and which actor is selected
agents diagnostics
```
## Common First-Time Issues and Solutions
### Issue: "No LLM provider configured"
**Symptom:** Error message says no API keys found.
**Solution:**
1. Verify you've set an environment variable: `echo $OPENAI_API_KEY`
2. If empty, set it: `export OPENAI_API_KEY="sk-..."`
3. Run `agents diagnostics` to verify the provider is detected
4. Restart your terminal or shell session if you just set the variable
### Issue: "Actor not found"
**Symptom:** Error says `local/my-actor` doesn't exist.
**Solution:**
1. Check the actor file exists: `ls my-first-actor.yaml`
2. Verify the file is valid YAML (check indentation)
3. Use the full path if the file is not in the current directory
4. Built-in actors use the format `<provider>/<model>` (e.g., `openai/gpt-4o`)
### Issue: "TUI won't start"
**Symptom:** `agents tui` fails or shows a blank screen.
**Solution:**
1. Ensure the TUI extra is installed: `pip install -e ".[tui]"`
2. Check your terminal supports 256 colors: `echo $TERM`
3. Try running with explicit terminal: `TERM=xterm-256color agents tui`
4. Check for conflicting environment variables: `env | grep -i textual`
### Issue: "Tool execution fails"
**Symptom:** Actor tries to use a tool but gets an error.
**Solution:**
1. Check the tool is available: `agents tools list` (if implemented)
2. Verify tool permissions are granted (TUI shows permission overlay)
3. Check tool configuration in the actor YAML
4. Review tool documentation: see [Tool System](../architecture.md#tool-system)
### Issue: "Session not found"
**Symptom:** Error when trying to export or import a session.
**Solution:**
1. List available sessions: `agents session list`
2. Use the correct session ID from the list
3. Check the session file exists (for import): `ls session.json`
4. Verify the JSON is valid: `python -m json.tool session.json`
### Issue: "Permission denied" errors
**Symptom:** Actor can't read/write files or access resources.
**Solution:**
1. Check file permissions: `ls -la <file>`
2. Ensure the file is readable/writable by your user
3. In the TUI, approve permission requests when prompted (press `y`)
4. Check resource configuration in your project
## Next Learning Steps
Now that you're up and running, here's what to explore next:
### 1. **Understand the Architecture** (30 minutes)
- Read [Architecture Overview](../architecture.md)
- Learn about the layered design and key components
- Understand the Plan Lifecycle in detail
### 2. **Build Your First Custom Actor** (1 hour)
- Create a YAML actor configuration
- Add tools and integrations
- Test in the TUI and CLI
- See [Actor System](../architecture.md#actor-system) for details
### 3. **Work with Resources** (1 hour)
- Create file and database resources
- Use resources in your actor
- Understand the Resource DAG
- See [Resource System](../architecture.md#resource-system)
### 4. **Explore Tools and Skills** (1 hour)
- Discover available tools
- Create custom tools
- Load and manage skills
- See [Tool System](../architecture.md#tool-system) and [Skill System](../architecture.md#skill-system)
### 5. **Master the Plan Lifecycle** (2 hours)
- Create and execute plans
- Understand phase transitions
- Use correction and rollback
- See [Plan Lifecycle](../architecture.md#plan-lifecycle)
### 6. **Integrate with External Services** (2 hours)
- Set up MCP (Model Context Protocol) servers
- Configure LSP (Language Server Protocol) integration
- Use ACMS (Advanced Context Management System)
- See [MCP Integration](../architecture.md#mcp-integration) and [LSP Integration](../architecture.md#lsp-integration)
### 7. **Deploy to Production** (2 hours)
- Use server mode: `agents server connect`
- Deploy with Kubernetes (see `k8s/`)
- Configure observability and logging
- See [Server Architecture](../development/agent-system-specification.md)
## Key Documentation References
| Topic | Document |
|-------|----------|
| **Architecture** | [Architecture Overview](../architecture.md) |
| **Specification** | [Full Specification](../specification.md) |
| **API Reference** | [API Docs](../api/index.md) |
| **Development** | [Development Guide](../development/agent-system-specification.md) |
| **Testing** | [Testing Guide](../development/testing.md) |
| **FAQ** | [Frequently Asked Questions](../faq.md) |
| **Design Decisions** | [Architecture Decision Records (ADRs)](../adr/index.md) |
## Tips for Success
1. **Start Small** — Create simple actors before complex ones
2. **Use the TUI** — The interactive interface is great for learning and debugging
3. **Read the Specification**`docs/specification.md` is the authoritative source
4. **Check the Examples** — See `examples/` for real-world configurations
5. **Run Tests** — Use `nox -s unit_tests` to verify your changes
6. **Ask for Help** — Check the FAQ and ADRs for common questions
## Troubleshooting Commands
```bash
# Check your setup
agents diagnostics
# List available actors
agents actor list
# List available tools
agents tools list # if implemented
# List sessions
agents session list
# View help for any command
agents <command> --help
# Run tests to verify everything works
nox -s unit_tests
# Check code quality
nox -s lint
nox -s typecheck
```
## What's Next?
- **Build an actor** — Create a custom actor for your use case
- **Explore the TUI** — Spend time in the interactive interface
- **Read the architecture** — Understand the design decisions
- **Join the community** — Check out discussions and issues
- **Contribute** — See [CONTRIBUTING.md](../../CONTRIBUTING.md) for guidelines
Happy automating! 🚀
-120
View File
@@ -1,120 +0,0 @@
# Provider Configuration Reference
CleverAgents supports multiple AI provider backends through the `ProviderRegistry`.
Each provider is configured via environment variables or the `agents config set` command.
## Supported Providers
| Provider | `ProviderType` | API Key Environment Variable |
|----------|---------------|------------------------------|
| OpenAI | `openai` | `OPENAI_API_KEY` |
| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` |
| Google / Gemini | `google` / `gemini` | `GOOGLE_API_KEY` / `GEMINI_API_KEY` |
| Azure OpenAI | `azure` | `AZURE_OPENAI_API_KEY` |
| **OpenRouter** | **`openrouter`** | **`OPENROUTER_API_KEY`** |
| Groq | `groq` | `GROQ_API_KEY` |
| Together AI | `together` | `TOGETHER_API_KEY` |
| Cohere | `cohere` | `COHERE_API_KEY` |
---
## OpenRouter
[OpenRouter](https://openrouter.ai) provides a unified API gateway to hundreds of
models from different providers (Anthropic, OpenAI, Google, Meta, Mistral, and more).
This makes it ideal for cost optimisation and model diversity without managing
multiple API keys.
### Configuration
Set the following environment variables:
```bash
# Required
export OPENROUTER_API_KEY="sk-or-v1-..."
# Optional: default model (defaults to anthropic/claude-sonnet-4-20250514)
export CLEVERAGENTS_DEFAULT_MODEL="openai/gpt-4o"
# Optional: set OpenRouter as the default provider
export CLEVERAGENTS_DEFAULT_PROVIDER="openrouter"
# Optional: organisation identifier sent in HTTP headers
export OPENROUTER_ORGANIZATION="myapp.example.com"
```
Or use the CLI:
```bash
agents config set provider.openrouter.api_key "sk-or-v1-..."
agents config set provider.openrouter.model "anthropic/claude-3-haiku"
```
### Config Keys
| Config Key | Environment Variable | Description |
|------------|---------------------|-------------|
| `provider.openrouter.api_key` | `OPENROUTER_API_KEY` | OpenRouter API key (required) |
| `provider.openrouter.model` | `CLEVERAGENTS_DEFAULT_MODEL` | Default model slug (optional) |
| `provider.openrouter.organization` | `OPENROUTER_ORGANIZATION` | Organisation name sent in `HTTP-Referer` / `X-Title` headers (optional) |
### Capabilities
| Feature | Supported |
|---------|-----------|
| Streaming | ✅ Yes |
| Tool calls | ✅ Yes |
| Vision / image input | ✅ Yes (model-dependent) |
| JSON mode | ✅ Yes (model-dependent) |
| Max context length | 128 000 tokens (varies by model) |
### Selecting a Model
OpenRouter model slugs follow the format `<provider>/<model-name>`. Examples:
```bash
# Anthropic via OpenRouter
agents config set provider.openrouter.model "anthropic/claude-sonnet-4-20250514"
# OpenAI via OpenRouter
agents config set provider.openrouter.model "openai/gpt-4o"
# Meta Llama via OpenRouter
agents config set provider.openrouter.model "meta-llama/llama-3.1-70b-instruct"
```
Browse the full model catalogue at <https://openrouter.ai/models>.
### Fallback Priority
When no explicit provider is configured, CleverAgents selects the first available
provider in the following fallback order:
1. OpenAI
2. Anthropic
3. Google
4. Azure
5. **OpenRouter** ← position 5
6. Groq
7. Together AI
8. Cohere
### Example Usage
```python
from cleveragents.providers.registry import ProviderRegistry, ProviderType
registry = ProviderRegistry()
# Create a LangChain LLM directly
llm = registry.create_llm(
provider_type=ProviderType.OPENROUTER,
model_id="anthropic/claude-3-haiku",
)
# Or create a full AIProviderInterface adapter
provider = registry.create_ai_provider(
provider_type="openrouter",
model_id="openai/gpt-4o",
)
```
+26 -5502
View File
File diff suppressed because one or more lines are too long
@@ -1,87 +0,0 @@
Feature: OpenRouter provider support in ProviderRegistry
As a developer using CleverAgents
I want ProviderRegistry to handle ProviderType.OPENROUTER in create_llm and _create_provider_llm
So that OpenRouter models are accessible without raising ValueError
# Covers the core acceptance criterion:
# ProviderRegistry.create_llm handles ProviderType.OPENROUTER without raising ValueError
@unit @providers @openrouter @registry
Scenario: create_llm dispatches to OpenRouter via _create_provider_llm
Given I have an openrouter registry with API key "sk-or-test-key"
And the openrouter registry LangChain ChatOpenAI client is stubbed
When I call create_llm on the openrouter registry for provider "openrouter" with model "anthropic/claude-3-haiku"
Then the openrouter registry ChatOpenAI should be called with model "anthropic/claude-3-haiku"
And the openrouter registry ChatOpenAI should use base url "https://openrouter.ai/api/v1"
And the openrouter registry ChatOpenAI should use api key "sk-or-test-key"
@unit @providers @openrouter @registry
Scenario: _create_provider_llm builds ChatOpenAI for OpenRouter with default model
Given I have an openrouter registry with API key "sk-or-default"
And the openrouter registry LangChain ChatOpenAI client is stubbed
When I call _create_provider_llm on the openrouter registry for provider "openrouter" with model None
Then the openrouter registry ChatOpenAI should be called with model "anthropic/claude-sonnet-4-20250514"
And the openrouter registry ChatOpenAI should use base url "https://openrouter.ai/api/v1"
@unit @providers @openrouter @registry
Scenario: _create_provider_llm injects organization headers when configured
Given I have an openrouter registry with API key "sk-or-org" and organization "myapp.example.com"
And the openrouter registry LangChain ChatOpenAI client is stubbed
When I call _create_provider_llm on the openrouter registry for provider "openrouter" with model "openai/gpt-4o"
Then the openrouter registry ChatOpenAI should include header "HTTP-Referer" with value "myapp.example.com"
And the openrouter registry ChatOpenAI should include header "X-Title" with value "myapp.example.com"
@unit @providers @openrouter @registry
Scenario: _create_provider_llm passes custom default_headers through
Given I have an openrouter registry with API key "sk-or-headers"
And the openrouter registry LangChain ChatOpenAI client is stubbed
When I call _create_provider_llm on the openrouter registry for provider "openrouter" with model "openai/gpt-4o" and default_headers "X-Custom=trace-123"
Then the openrouter registry ChatOpenAI should include header "X-Custom" with value "trace-123"
@unit @providers @openrouter @registry
Scenario: create_llm raises ValueError when OpenRouter API key is missing
Given I have an openrouter registry with no API keys
When I try to call create_llm on the openrouter registry for provider "openrouter"
Then an openrouter registry ValueError should be raised
And the openrouter registry error should mention "OPENROUTER_API_KEY"
@unit @providers @openrouter @registry
Scenario: ProviderType.OPENROUTER is in the registry fallback order
Given I have the openrouter ProviderRegistry class
When I check the openrouter FALLBACK_ORDER
Then ProviderType.OPENROUTER should be in the fallback order
@unit @providers @openrouter @registry
Scenario: OpenRouter default model is configured correctly
Given I have the openrouter ProviderRegistry class
When I check the openrouter DEFAULT_MODELS
Then the openrouter default model should be "anthropic/claude-sonnet-4-20250514"
@unit @providers @openrouter @registry
Scenario: OpenRouter capabilities are configured correctly
Given I have the openrouter ProviderRegistry class
When I check the openrouter DEFAULT_CAPABILITIES for OPENROUTER
Then the openrouter provider supports_streaming should be True
And the openrouter provider supports_tool_calls should be True
And the openrouter provider supports_vision should be True
And the openrouter provider max_context_length should be 128000
And the openrouter provider supports_json_mode should be True
@unit @providers @openrouter @registry
Scenario: OpenRouter provider is discovered when API key is set
Given I have an openrouter registry with API key "sk-or-discover"
When I call get_provider_info for OPENROUTER on the openrouter registry
Then the openrouter provider info should be configured
@unit @providers @openrouter @registry
Scenario: OpenRouter provider is not discovered when API key is missing
Given I have an openrouter registry with no API keys
When I call get_provider_info for OPENROUTER on the openrouter registry
Then the openrouter provider info should not be configured
@unit @providers @openrouter @registry
Scenario: create_llm selects OpenRouter as default when it is the only configured provider
Given I have an openrouter registry with API key "sk-or-only"
And the openrouter registry LangChain ChatOpenAI client is stubbed
When I call create_llm on the openrouter registry without specifying a provider
Then the openrouter registry ChatOpenAI should be called with model "anthropic/claude-sonnet-4-20250514"
@@ -1,374 +0,0 @@
"""Step definitions for openrouter_provider_registry.feature.
Tests that ProviderRegistry.create_llm and _create_provider_llm correctly
dispatch to the OpenRouter ChatOpenAI backend without raising ValueError.
"""
from __future__ import annotations
import sys
import types
from typing import Any
from unittest.mock import MagicMock
from behave import given, then, use_step_matcher, when # type: ignore[import-untyped]
from cleveragents.providers.registry import (
ProviderRegistry,
ProviderType,
)
_OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_or_settings(
openrouter: str | None = None,
organization: str | None = None,
) -> MagicMock:
"""Return a minimal fake Settings object for OpenRouter tests."""
settings = MagicMock()
settings.openai_api_key = None
settings.anthropic_api_key = None
settings.google_api_key = None
settings.gemini_api_key = None
settings.azure_api_key = None
settings.openrouter_api_key = openrouter
settings.cohere_api_key = None
settings.groq_api_key = None
settings.together_api_key = None
settings.default_provider = None
settings.default_model = None
settings.azure_openai_endpoint = None
settings.azure_openai_api_version = None
settings.azure_openai_deployment = None
settings.openrouter_organization = organization
return settings
def _stub_chat_openai(context: Any) -> None:
"""Replace langchain_openai.ChatOpenAI with a recording stub."""
original = sys.modules.get("langchain_openai")
calls: list[dict[str, Any]] = []
class FakeChatOpenAI:
def __init__(self, **kwargs: Any) -> None:
calls.append(kwargs)
class FakeAzureChatOpenAI:
def __init__(self, **kwargs: Any) -> None:
pass
stub = types.ModuleType("langchain_openai")
stub.ChatOpenAI = FakeChatOpenAI # type: ignore[attr-defined]
stub.AzureChatOpenAI = FakeAzureChatOpenAI # type: ignore[attr-defined]
sys.modules["langchain_openai"] = stub
context._or_chat_openai_calls = calls
context._or_chat_openai_cls = FakeChatOpenAI
if not hasattr(context, "_or_cleanup"):
context._or_cleanup = []
def _restore() -> None:
if original is None:
sys.modules.pop("langchain_openai", None)
else:
sys.modules["langchain_openai"] = original
context._or_cleanup.append(_restore)
def _parse_headers_string(headers_string: str) -> dict[str, str]:
"""Parse 'Key=Value,Key2=Value2' into a dict."""
result: dict[str, str] = {}
for entry in headers_string.split(","):
entry = entry.strip()
if "=" in entry:
key, value = entry.split("=", 1)
result[key.strip()] = value.strip()
return result
# ---------------------------------------------------------------------------
# Given
# ---------------------------------------------------------------------------
use_step_matcher("re")
@given(
r'I have an openrouter registry with API key "(?P<or_api_key>[^"]+)" and organization "(?P<or_org>[^"]+)"'
)
def step_or_registry_with_key_and_org(
context: Any, or_api_key: str, or_org: str
) -> None:
context.or_registry = ProviderRegistry(
settings=_make_or_settings(openrouter=or_api_key, organization=or_org)
)
@given(r'I have an openrouter registry with API key "(?P<or_api_key>[^"]+)"')
def step_or_registry_with_key(context: Any, or_api_key: str) -> None:
context.or_registry = ProviderRegistry(
settings=_make_or_settings(openrouter=or_api_key)
)
@given(r"I have an openrouter registry with no API keys")
def step_or_registry_no_keys(context: Any) -> None:
context.or_registry = ProviderRegistry(settings=_make_or_settings())
@given(r"I have the openrouter ProviderRegistry class")
def step_or_registry_class(context: Any) -> None:
context.or_registry_class = ProviderRegistry
@given(r"the openrouter registry LangChain ChatOpenAI client is stubbed")
def step_or_stub_chat_openai(context: Any) -> None:
_stub_chat_openai(context)
# ---------------------------------------------------------------------------
# When
# ---------------------------------------------------------------------------
@when(
r'I call create_llm on the openrouter registry for provider "openrouter" with model "(?P<or_model>[^"]+)"'
)
def step_or_create_llm_with_model(context: Any, or_model: str) -> None:
context.or_error = None
try:
context.or_result = context.or_registry.create_llm(
provider_type="openrouter", model_id=or_model
)
except ValueError as exc:
context.or_error = exc
@when(r"I call create_llm on the openrouter registry without specifying a provider")
def step_or_create_llm_default(context: Any) -> None:
context.or_error = None
try:
context.or_result = context.or_registry.create_llm()
except ValueError as exc:
context.or_error = exc
@when(
r'I call _create_provider_llm on the openrouter registry for provider "openrouter" with model None'
)
def step_or_private_llm_none_model(context: Any) -> None:
context.or_error = None
try:
context.or_result = context.or_registry._create_provider_llm(
ProviderType.OPENROUTER, None
)
except ValueError as exc:
context.or_error = exc
@when(
r'I call _create_provider_llm on the openrouter registry for provider "openrouter" with model "(?P<or_model>[^"]+)" and default_headers "(?P<or_headers_string>[^"]+)"'
)
def step_or_private_llm_with_headers(
context: Any, or_model: str, or_headers_string: str
) -> None:
headers = _parse_headers_string(or_headers_string)
context.or_error = None
try:
context.or_result = context.or_registry._create_provider_llm(
ProviderType.OPENROUTER, or_model, default_headers=headers
)
except ValueError as exc:
context.or_error = exc
@when(
r'I call _create_provider_llm on the openrouter registry for provider "openrouter" with model "(?P<or_model>[^"]+)"'
)
def step_or_private_llm_with_model(context: Any, or_model: str) -> None:
context.or_error = None
try:
context.or_result = context.or_registry._create_provider_llm(
ProviderType.OPENROUTER, or_model
)
except ValueError as exc:
context.or_error = exc
@when(r'I try to call create_llm on the openrouter registry for provider "openrouter"')
def step_or_try_create_llm_no_key(context: Any) -> None:
context.or_error = None
try:
context.or_registry.create_llm(provider_type="openrouter")
except ValueError as exc:
context.or_error = exc
@when(r"I check the openrouter FALLBACK_ORDER")
def step_or_check_fallback_order(context: Any) -> None:
context.or_fallback_order = context.or_registry_class.FALLBACK_ORDER
@when(r"I check the openrouter DEFAULT_MODELS")
def step_or_check_default_models(context: Any) -> None:
context.or_default_models = context.or_registry_class.DEFAULT_MODELS
@when(r"I check the openrouter DEFAULT_CAPABILITIES for OPENROUTER")
def step_or_check_capabilities(context: Any) -> None:
context.or_capabilities = context.or_registry_class.DEFAULT_CAPABILITIES.get(
ProviderType.OPENROUTER
)
@when(r"I call get_provider_info for OPENROUTER on the openrouter registry")
def step_or_get_provider_info(context: Any) -> None:
context.or_provider_info = context.or_registry.get_provider_info(
ProviderType.OPENROUTER
)
# ---------------------------------------------------------------------------
# Then
# ---------------------------------------------------------------------------
@then(
r'the openrouter registry ChatOpenAI should be called with model "(?P<or_expected_model>[^"]+)"'
)
def step_or_assert_model(context: Any, or_expected_model: str) -> None:
calls = getattr(context, "_or_chat_openai_calls", [])
assert calls, "Expected ChatOpenAI to be instantiated"
actual_model = calls[-1].get("model")
assert actual_model == or_expected_model, (
f"Expected model={or_expected_model!r}, got {actual_model!r}"
)
@then(
r'the openrouter registry ChatOpenAI should use base url "(?P<or_expected_url>[^"]+)"'
)
def step_or_assert_base_url(context: Any, or_expected_url: str) -> None:
calls = getattr(context, "_or_chat_openai_calls", [])
assert calls, "Expected ChatOpenAI to be instantiated"
actual_url = calls[-1].get("openai_api_base")
assert actual_url == or_expected_url, (
f"Expected openai_api_base={or_expected_url!r}, got {actual_url!r}"
)
@then(
r'the openrouter registry ChatOpenAI should use api key "(?P<or_expected_key>[^"]+)"'
)
def step_or_assert_api_key(context: Any, or_expected_key: str) -> None:
calls = getattr(context, "_or_chat_openai_calls", [])
assert calls, "Expected ChatOpenAI to be instantiated"
actual_key = calls[-1].get("openai_api_key")
assert actual_key == or_expected_key, (
f"Expected openai_api_key={or_expected_key!r}, got {actual_key!r}"
)
@then(
r'the openrouter registry ChatOpenAI should include header "(?P<or_header_name>[^"]+)" with value "(?P<or_expected_value>[^"]+)"'
)
def step_or_assert_header(
context: Any, or_header_name: str, or_expected_value: str
) -> None:
calls = getattr(context, "_or_chat_openai_calls", [])
assert calls, "Expected ChatOpenAI to be instantiated"
headers = calls[-1].get("default_headers")
assert isinstance(headers, dict), (
f"Expected default_headers to be a dict, got {type(headers)}"
)
actual_value = headers.get(or_header_name)
assert actual_value == or_expected_value, (
f"Expected header {or_header_name}={or_expected_value!r}, got {actual_value!r}"
)
@then(r"an openrouter registry ValueError should be raised")
def step_or_assert_value_error(context: Any) -> None:
assert context.or_error is not None, "Expected a ValueError to be raised"
assert isinstance(context.or_error, ValueError), (
f"Expected ValueError, got {type(context.or_error).__name__}"
)
@then(r'the openrouter registry error should mention "(?P<or_fragment>[^"]+)"')
def step_or_assert_error_contains(context: Any, or_fragment: str) -> None:
assert context.or_error is not None, "Expected an error to be set"
message = str(context.or_error)
assert or_fragment in message, (
f"Expected error to contain {or_fragment!r}, got: {message!r}"
)
@then(r"ProviderType\.OPENROUTER should be in the fallback order")
def step_or_assert_in_fallback(context: Any) -> None:
assert ProviderType.OPENROUTER in context.or_fallback_order, (
"Expected ProviderType.OPENROUTER to be in FALLBACK_ORDER"
)
@then(r'the openrouter default model should be "(?P<or_expected_model>[^"]+)"')
def step_or_assert_default_model(context: Any, or_expected_model: str) -> None:
actual = context.or_default_models.get(ProviderType.OPENROUTER)
assert actual == or_expected_model, (
f"Expected default model={or_expected_model!r}, got {actual!r}"
)
@then(r"the openrouter provider supports_streaming should be (?P<or_expected>\w+)")
def step_or_assert_streaming(context: Any, or_expected: str) -> None:
assert context.or_capabilities is not None
assert context.or_capabilities.supports_streaming is (or_expected == "True")
@then(r"the openrouter provider supports_tool_calls should be (?P<or_expected>\w+)")
def step_or_assert_tool_calls(context: Any, or_expected: str) -> None:
assert context.or_capabilities is not None
assert context.or_capabilities.supports_tool_calls is (or_expected == "True")
@then(r"the openrouter provider supports_vision should be (?P<or_expected>\w+)")
def step_or_assert_vision(context: Any, or_expected: str) -> None:
assert context.or_capabilities is not None
assert context.or_capabilities.supports_vision is (or_expected == "True")
@then(r"the openrouter provider max_context_length should be (?P<or_expected>\d+)")
def step_or_assert_context_length(context: Any, or_expected: str) -> None:
assert context.or_capabilities is not None
expected_int = int(or_expected)
assert context.or_capabilities.max_context_length == expected_int, (
f"Expected max_context_length={expected_int}, got {context.or_capabilities.max_context_length}"
)
@then(r"the openrouter provider supports_json_mode should be (?P<or_expected>\w+)")
def step_or_assert_json_mode(context: Any, or_expected: str) -> None:
assert context.or_capabilities is not None
assert context.or_capabilities.supports_json_mode is (or_expected == "True")
@then(r"the openrouter provider info should be configured")
def step_or_assert_provider_configured(context: Any) -> None:
info = context.or_provider_info
assert info is not None, "Expected provider info to exist"
assert info.is_configured is True, "Expected provider to be configured"
@then(r"the openrouter provider info should not be configured")
def step_or_assert_provider_not_configured(context: Any) -> None:
info = context.or_provider_info
assert info is not None, "Expected provider info to exist"
assert info.is_configured is False, "Expected provider to not be configured"
+2
View File
@@ -11,6 +11,8 @@ site_dir: build/site
nav:
- Specification: specification.md
- Architecture: architecture.md
- Guides:
- Getting Started: guides/getting-started.md
- API Reference:
- Overview: api/index.md
- Core Utilities: api/core.md
-26
View File
@@ -525,32 +525,6 @@ class ProviderRegistry:
**kwargs,
)
if provider_type == ProviderType.OPENROUTER:
from langchain_openai import ChatOpenAI
from cleveragents.providers.llm.openrouter_provider import (
OpenRouterChatProvider,
)
key_attr = self.PROVIDER_KEY_ATTRS.get(provider_type)
api_key = getattr(self._settings, key_attr, None) if key_attr else None
organization = getattr(self._settings, "openrouter_organization", None)
default_headers: dict[str, str] | None = kwargs.pop("default_headers", None)
if organization:
if default_headers is None:
default_headers = {}
default_headers.setdefault("HTTP-Referer", organization)
default_headers.setdefault("X-Title", organization)
openrouter_kwargs: dict[str, Any] = {
"model": model_id or "anthropic/claude-sonnet-4-20250514",
"openai_api_base": OpenRouterChatProvider._BASE_URL,
"openai_api_key": api_key or "",
}
if default_headers:
openrouter_kwargs["default_headers"] = default_headers
openrouter_kwargs.update(kwargs)
return ChatOpenAI(**openrouter_kwargs)
if provider_type == ProviderType.GROQ:
from langchain_groq import ChatGroq