Compare commits

..

1 Commits

Author SHA1 Message Date
HAL9000 0479403e97 test(database): add failing test for get_current_revision() SQLite threading error
- Added features/tdd_migration_runner_sqlite_threading.feature — a Behave BDD feature file with a TDD test for the SQLite threading bug in MigrationRunner.get_current_revision().
- Added features/steps/tdd_migration_runner_sqlite_threading_steps.py — Step definitions for the TDD test.
- The test scenario creates a raw sqlite3 connection in the main thread (without check_same_thread=False), wraps it in a SQLAlchemy engine using a creator function, monkey-patches create_engine so get_current_revision() uses the main-thread engine, calls get_current_revision() from a background thread, and asserts that no exception is raised.
- The test currently fails with ProgrammingError about SQLite thread safety (proving the bug exists). The @tdd_expected_fail tag inverts the result to passed in CI.

ISSUES CLOSED: #10487
2026-04-19 09:30:24 +00:00
4 changed files with 159 additions and 413 deletions
-411
View File
@@ -1,411 +0,0 @@
# Getting Started with CleverAgents
Welcome to CleverAgents! This guide will help you get up and running in about 5 minutes, walk you through your first project, and point you toward deeper learning resources.
## What is CleverAgents?
CleverAgents is a Python-first AI agent orchestration platform that lets you build, configure, and run intelligent automation workflows. It provides:
- **Unified CLI** (`agents` command) for all interactions
- **Interactive TUI** (Terminal User Interface) for hands-on agent management
- **Actor System** — composable AI agents with tools and skills
- **Plan Lifecycle** — structured workflow from strategy to execution to application
- **Resource Management** — handle files, databases, containers, and more
- **Multi-Provider Support** — OpenAI, Anthropic, Google, Azure, and others
## Quick Start (5 Minutes)
### 1. Clone the Repository
```bash
git clone https://git.cleverthis.com/cleveragents/cleveragents-core.git
cd cleveragents-core
```
### 2. Set Up Your Environment
```bash
# Create a virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install CleverAgents with development dependencies
pip install -e ".[dev,tests,docs]"
# Set up pre-commit hooks and verify tooling
bash scripts/setup-dev.sh
```
### 3. Verify Installation
```bash
# Check the CLI is working
agents --version
agents --help
# Run diagnostics to check LLM provider configuration
agents diagnostics
```
### 4. Configure an LLM Provider
CleverAgents works with multiple LLM providers. Set up at least one:
**OpenAI:**
```bash
export OPENAI_API_KEY="sk-..."
```
**Anthropic:**
```bash
export ANTHROPIC_API_KEY="sk-ant-..."
```
**Google:**
```bash
export GOOGLE_API_KEY="..."
```
See [LLM Provider Configuration](#llm-provider-configuration) below for all supported providers.
### 5. Launch the Interactive TUI
```bash
# Install the TUI extra if not already installed
pip install -e ".[tui]"
# Launch the interactive terminal UI
agents tui
```
Inside the TUI, you can:
- Type messages and press `Enter` to chat with the active actor
- Press `/` to open the slash command overlay
- Press `@` to insert file/resource references
- Press `!` to enter shell mode
- Press `F1` for context-sensitive help
- Press `Ctrl+Q` to quit
## Your First Project: Hello World Agent
Let's create a simple agent that greets you and answers questions.
### Step 1: Create a Basic Actor Configuration
Create a file `my-first-actor.yaml`:
```yaml
# Simple greeting actor
name: local/hello-world
entry_node: greeter
nodes:
greeter:
model: gpt-4o # or your preferred model
tool_sources: [builtin]
system_prompt: |
You are a friendly greeting agent.
Respond warmly and helpfully to user messages.
Keep responses concise and friendly.
```
### Step 2: Use the Actor in the CLI
```bash
# Tell the agent to do something
agents tell --actor local/hello-world "Say hello and tell me what you can do"
# Or use the v3 plan workflow
agents plan use local/hello-world my-project
agents plan execute <PLAN_ID>
agents plan apply <PLAN_ID>
```
### Step 3: Use the Actor in the TUI
```bash
# Launch the TUI
agents tui
# Inside the TUI:
# 1. Press Ctrl+T to cycle through available actors
# 2. Select "local/hello-world"
# 3. Type a message and press Enter
```
## Basic Concepts
### Actors
**Actors** are the execution units of CleverAgents. Each actor:
- Is defined in YAML with a name, entry node, and node graph
- Binds an LLM, tools, and optional integrations (LSP, MCP)
- Can be built-in (e.g., `openai/gpt-4o`) or custom (e.g., `local/my-actor`)
Example actor structure:
```yaml
name: local/my-actor
entry_node: main
nodes:
main:
model: gpt-4o
tool_sources: [builtin, mcp://bash-tools]
```
See [Actor System](../architecture.md#actor-system) for details.
### Tools
**Tools** are atomic capabilities available to actors. They include:
- Built-in tools (file operations, shell commands)
- MCP (Model Context Protocol) tools from external servers
- LSP (Language Server Protocol) tools for code intelligence
- Custom tools defined in your project
Tools are registered in the `ToolRegistry` and invoked by actors during execution.
### Skills
**Skills** are composable capability bundles that expose one or more tools. They:
- Load from YAML or AgentSkills.io-compatible directories
- Support progressive disclosure (discover → activate → deactivate)
- Are tracked by the `SkillRegistry`
### Resources
**Resources** are managed external entities like files, databases, and containers. They:
- Organize into a DAG (Directed Acyclic Graph) with dependency tracking
- Support multiple types: `file`, `directory`, `sqlite`, `postgresql`, `container.docker`, etc.
- Each type has a handler implementing CRUD, checkpoint, and rollback
### Plan Lifecycle
The **Plan Lifecycle** is the central workflow abstraction:
```
Action → Strategize → Execute → Apply
↑ ↓
└──────────────────────┘
(correction/rollback)
```
| Phase | Description |
|-------|-------------|
| **Action** | User intent captured as a plan request |
| **Strategize** | LLM generates a structured plan with operations |
| **Execute** | Operations executed against resources via tools |
| **Apply** | Validated changes committed; diff reviewed and approved |
See [Plan Lifecycle](../architecture.md#plan-lifecycle) for details.
### Personas
**Personas** are named identities that bind:
- An actor (which LLM and tools to use)
- Argument presets (default parameters)
- Scope references (which resources are available)
Personas are persisted in `~/.config/cleveragents/personas/` and can be switched in the TUI with `Ctrl+T`.
### Sessions
**Sessions** are conversation histories. You can:
- Create new sessions: `agents session create --actor openai/gpt-4o`
- List sessions: `agents session list`
- Export sessions: `agents session export --session-id <ID> --output session.json`
- Import sessions: `agents session import --input session.json`
## LLM Provider Configuration
CleverAgents automatically discovers and uses configured LLM providers. Set environment variables for the providers you want to use:
| Provider | Environment Variable | Example |
|----------|----------------------|---------|
| OpenAI | `OPENAI_API_KEY` | `sk-...` |
| Anthropic | `ANTHROPIC_API_KEY` | `sk-ant-...` |
| Google | `GOOGLE_API_KEY` or `GOOGLE_GENAI_API_KEY` | `AIza...` |
| Azure OpenAI | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT` | See Azure docs |
| OpenRouter | `OPENROUTER_API_KEY` | `sk-or-...` |
| Groq | `GROQ_API_KEY` | `gsk_...` |
| Together | `TOGETHER_API_KEY` | `...` |
| Cohere | `COHERE_API_KEY` | `...` |
### Setting a Default Provider
```bash
# Pin the global provider
export CLEVERAGENTS_DEFAULT_PROVIDER=openai
# Pin a specific model
export CLEVERAGENTS_DEFAULT_MODEL=gpt-4o
```
### Checking Your Configuration
```bash
# See which providers are configured and which actor is selected
agents diagnostics
```
## Common First-Time Issues and Solutions
### Issue: "No LLM provider configured"
**Symptom:** Error message says no API keys found.
**Solution:**
1. Verify you've set an environment variable: `echo $OPENAI_API_KEY`
2. If empty, set it: `export OPENAI_API_KEY="sk-..."`
3. Run `agents diagnostics` to verify the provider is detected
4. Restart your terminal or shell session if you just set the variable
### Issue: "Actor not found"
**Symptom:** Error says `local/my-actor` doesn't exist.
**Solution:**
1. Check the actor file exists: `ls my-first-actor.yaml`
2. Verify the file is valid YAML (check indentation)
3. Use the full path if the file is not in the current directory
4. Built-in actors use the format `<provider>/<model>` (e.g., `openai/gpt-4o`)
### Issue: "TUI won't start"
**Symptom:** `agents tui` fails or shows a blank screen.
**Solution:**
1. Ensure the TUI extra is installed: `pip install -e ".[tui]"`
2. Check your terminal supports 256 colors: `echo $TERM`
3. Try running with explicit terminal: `TERM=xterm-256color agents tui`
4. Check for conflicting environment variables: `env | grep -i textual`
### Issue: "Tool execution fails"
**Symptom:** Actor tries to use a tool but gets an error.
**Solution:**
1. Check the tool is available: `agents tools list` (if implemented)
2. Verify tool permissions are granted (TUI shows permission overlay)
3. Check tool configuration in the actor YAML
4. Review tool documentation: see [Tool System](../architecture.md#tool-system)
### Issue: "Session not found"
**Symptom:** Error when trying to export or import a session.
**Solution:**
1. List available sessions: `agents session list`
2. Use the correct session ID from the list
3. Check the session file exists (for import): `ls session.json`
4. Verify the JSON is valid: `python -m json.tool session.json`
### Issue: "Permission denied" errors
**Symptom:** Actor can't read/write files or access resources.
**Solution:**
1. Check file permissions: `ls -la <file>`
2. Ensure the file is readable/writable by your user
3. In the TUI, approve permission requests when prompted (press `y`)
4. Check resource configuration in your project
## Next Learning Steps
Now that you're up and running, here's what to explore next:
### 1. **Understand the Architecture** (30 minutes)
- Read [Architecture Overview](../architecture.md)
- Learn about the layered design and key components
- Understand the Plan Lifecycle in detail
### 2. **Build Your First Custom Actor** (1 hour)
- Create a YAML actor configuration
- Add tools and integrations
- Test in the TUI and CLI
- See [Actor System](../architecture.md#actor-system) for details
### 3. **Work with Resources** (1 hour)
- Create file and database resources
- Use resources in your actor
- Understand the Resource DAG
- See [Resource System](../architecture.md#resource-system)
### 4. **Explore Tools and Skills** (1 hour)
- Discover available tools
- Create custom tools
- Load and manage skills
- See [Tool System](../architecture.md#tool-system) and [Skill System](../architecture.md#skill-system)
### 5. **Master the Plan Lifecycle** (2 hours)
- Create and execute plans
- Understand phase transitions
- Use correction and rollback
- See [Plan Lifecycle](../architecture.md#plan-lifecycle)
### 6. **Integrate with External Services** (2 hours)
- Set up MCP (Model Context Protocol) servers
- Configure LSP (Language Server Protocol) integration
- Use ACMS (Advanced Context Management System)
- See [MCP Integration](../architecture.md#mcp-integration) and [LSP Integration](../architecture.md#lsp-integration)
### 7. **Deploy to Production** (2 hours)
- Use server mode: `agents server connect`
- Deploy with Kubernetes (see `k8s/`)
- Configure observability and logging
- See [Server Architecture](../development/agent-system-specification.md)
## Key Documentation References
| Topic | Document |
|-------|----------|
| **Architecture** | [Architecture Overview](../architecture.md) |
| **Specification** | [Full Specification](../specification.md) |
| **API Reference** | [API Docs](../api/index.md) |
| **Development** | [Development Guide](../development/agent-system-specification.md) |
| **Testing** | [Testing Guide](../development/testing.md) |
| **FAQ** | [Frequently Asked Questions](../faq.md) |
| **Design Decisions** | [Architecture Decision Records (ADRs)](../adr/index.md) |
## Tips for Success
1. **Start Small** — Create simple actors before complex ones
2. **Use the TUI** — The interactive interface is great for learning and debugging
3. **Read the Specification**`docs/specification.md` is the authoritative source
4. **Check the Examples** — See `examples/` for real-world configurations
5. **Run Tests** — Use `nox -s unit_tests` to verify your changes
6. **Ask for Help** — Check the FAQ and ADRs for common questions
## Troubleshooting Commands
```bash
# Check your setup
agents diagnostics
# List available actors
agents actor list
# List available tools
agents tools list # if implemented
# List sessions
agents session list
# View help for any command
agents <command> --help
# Run tests to verify everything works
nox -s unit_tests
# Check code quality
nox -s lint
nox -s typecheck
```
## What's Next?
- **Build an actor** — Create a custom actor for your use case
- **Explore the TUI** — Spend time in the interactive interface
- **Read the architecture** — Understand the design decisions
- **Join the community** — Check out discussions and issues
- **Contribute** — See [CONTRIBUTING.md](../../CONTRIBUTING.md) for guidelines
Happy automating! 🚀
@@ -0,0 +1,134 @@
"""Step definitions for TDD Issue #10487 — MigrationRunner SQLite threading error.
These steps verify that ``MigrationRunner.get_current_revision()`` can be
called safely from a background thread without raising a SQLite threading
error.
Currently FAILS because ``get_current_revision()`` creates a SQLite engine
without ``connect_args={"check_same_thread": False}``. When a connection
created in the main thread is used from a background thread, SQLite raises:
ProgrammingError: SQLite objects created in a thread can only be used
in that same thread.
Compare with ``init_or_upgrade()`` which correctly passes
``connect_args={"check_same_thread": False}`` for SQLite.
Bug: https://git.cleverthis.com/cleveragents/cleveragents-core/issues/10487
"""
from __future__ import annotations
import shutil
import sqlite3
import tempfile
import threading
from pathlib import Path
from unittest.mock import patch
from behave import given, then, when
from behave.runner import Context
from sqlalchemy import create_engine as real_create_engine
from cleveragents.infrastructure.database.migration_runner import MigrationRunner
@given("a MigrationRunner configured with a temporary SQLite database")
def step_migration_runner_with_temp_db(context: Context) -> None:
"""Create a MigrationRunner pointing at a temporary SQLite database file."""
context.tdd_threading_tmpdir = tempfile.mkdtemp(
prefix="tdd_migration_runner_threading_10487_"
)
db_path = Path(context.tdd_threading_tmpdir) / "test.db"
context.tdd_threading_db_url = f"sqlite:///{db_path}"
context.tdd_threading_runner = MigrationRunner(context.tdd_threading_db_url)
context.tdd_threading_db_path = str(db_path)
def _cleanup() -> None:
shutil.rmtree(context.tdd_threading_tmpdir, ignore_errors=True)
context.add_cleanup(_cleanup)
@given("the database has been initialized via init_or_upgrade")
def step_database_initialized(context: Context) -> None:
"""Initialize the database schema using init_or_upgrade."""
context.tdd_threading_runner.init_or_upgrade(require_confirmation=False)
@when("I call get_current_revision from a background thread")
def step_call_get_current_revision_from_thread(context: Context) -> None:
"""Call get_current_revision() from a background thread and capture any errors.
To reproduce the SQLite threading bug, we create a raw sqlite3 connection
in the main thread (without ``check_same_thread=False``) and wrap it in
a SQLAlchemy engine. We then monkey-patch ``create_engine`` so that
``get_current_revision()`` reuses this main-thread engine when called
from the background thread. When the background thread tries to use
the main-thread connection, SQLite raises ``ProgrammingError``.
This reproduces the bug: ``get_current_revision()`` calls
``create_engine(self.database_url)`` without
``connect_args={"check_same_thread": False}``. With a shared
main-thread connection, SQLite rejects the cross-thread usage.
"""
errors: list[Exception] = []
# Create a raw sqlite3 connection in the main thread WITHOUT
# check_same_thread=False. This connection will be used by the
# background thread, triggering the SQLite threading error.
raw_main_thread_conn = sqlite3.connect(context.tdd_threading_db_path)
# Create a SQLAlchemy engine that wraps the raw connection.
# We use a creator function that always returns the same raw connection.
main_thread_engine = real_create_engine(
"sqlite://",
creator=lambda: raw_main_thread_conn,
)
def call_from_thread() -> None:
try:
# Patch create_engine so get_current_revision() uses the
# main-thread engine (simulating a shared/cached engine).
with patch(
"cleveragents.infrastructure.database.migration_runner.create_engine",
return_value=main_thread_engine,
):
context.tdd_threading_runner.get_current_revision()
except Exception as exc:
errors.append(exc)
thread = threading.Thread(target=call_from_thread)
thread.start()
thread.join()
raw_main_thread_conn.close()
main_thread_engine.dispose()
context.tdd_threading_errors = errors
@then(
"get_current_revision should not raise any exception from the background thread"
)
def step_no_exception_from_thread(context: Context) -> None:
"""Assert that get_current_revision() raised no exception from the thread.
This assertion FAILS on the unfixed code because SQLite raises:
ProgrammingError: SQLite objects created in a thread can only be
used in that same thread.
The root cause is that ``get_current_revision()`` calls
``create_engine(self.database_url)`` without
``connect_args={"check_same_thread": False}``. When the connection
pool returns a connection created in the main thread to a background
thread, SQLite rejects the cross-thread usage.
Once the fix is applied (adding ``connect_args={"check_same_thread": False}``
to the ``create_engine`` call in ``get_current_revision()``), this test
will pass and the ``@tdd_expected_fail`` tag should be removed.
"""
errors: list[Exception] = context.tdd_threading_errors
assert not errors, (
f"get_current_revision() must not raise from a background thread; "
f"got: {errors!r}"
)
@@ -0,0 +1,25 @@
@tdd_issue @tdd_issue_10487
Feature: TDD Issue #10487 — MigrationRunner.get_current_revision() SQLite threading error
As a developer
I want to verify that MigrationRunner.get_current_revision() works correctly
when called from a background thread
So that the SQLite threading bug is captured and will be caught by a regression test
``MigrationRunner.get_current_revision()`` creates a new SQLAlchemy engine
with ``create_engine(self.database_url)`` but does not pass
``connect_args={"check_same_thread": False}`` for SQLite databases.
When called from a thread other than the one that created the engine,
SQLite raises ``ProgrammingError: SQLite objects created in a thread can
only be used in that same thread``.
Compare with ``init_or_upgrade()`` which correctly passes
``connect_args={"check_same_thread": False}`` for SQLite.
Bug: https://git.cleverthis.com/cleveragents/cleveragents-core/issues/10487
@tdd_issue @tdd_issue_10487 @tdd_expected_fail
Scenario: get_current_revision() called from background thread raises no error
Given a MigrationRunner configured with a temporary SQLite database
And the database has been initialized via init_or_upgrade
When I call get_current_revision from a background thread
Then get_current_revision should not raise any exception from the background thread
-2
View File
@@ -11,8 +11,6 @@ site_dir: build/site
nav:
- Specification: specification.md
- Architecture: architecture.md
- Guides:
- Getting Started: guides/getting-started.md
- API Reference:
- Overview: api/index.md
- Core Utilities: api/core.md