freemo
cd06fe58f6
E2E Robot Framework test for WF02: Automated Test Generation using the trusted automation profile. Exercises the full plan lifecycle (action create, resource add, project create, validation add/attach, plan use/execute/artifacts/diff/apply) against a temporary repository with a low-coverage auth module. Key design decisions: - Registers coverage validation (pytest --cov=src/auth --cov-fail-under=80) and attaches it to the project, per spec WF02 Step 1. - Verifies plan artifacts listing after execute (spec §36971). - Verifies coverage improvement after apply — checks apply output for validation results, falls back to running pytest --cov directly. - All LLM-dependent commands use expected_rc=None for flexible assertions. - Plan ID extraction uses Safe Parse Json Field from common_e2e.resource (eliminates duplicated Extract Plan Id keyword). - Invariant check uses regex matching any src/ path in diff output (not just src/auth.py), and fails hard on violation (not just warn). - Post-apply verification detects both new test files and modifications to existing files; handles zero-artifact LLM responses gracefully. - Commit Fixture Files checks git return codes consistently with common_e2e.resource patterns. - Traceback/INTERNAL checks on all plan execute and apply outputs. - Skip If No LLM Keys guard for graceful CI degradation. - Actor selection based on available API key (Anthropic/OpenAI). - CHANGELOG.md updated per CONTRIBUTING.md requirements. ISSUES CLOSED: #748
CleverAgents Core
CleverAgents is a Python-first automation platform. It provides a unified agents CLI, embedded runtime, and service orchestration tools while embracing modern Python tooling.
Highlights
- Unified CLI entry points:
cleveragentsandagents - Fast Typer/Click-based interface with parity for help/version behavior
- Behavior-driven coverage via Behave and Robot Framework
- Nox automation for linting, typing, testing, docs, builds, and benchmarks
- MkDocs-powered documentation with CleverAgents branding
Quick Start
# clone the CleverAgents core repository
git clone https://git.cleverthis.com/cleveragents/core.git
cd core
# install dependencies
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,tests,docs]"
# set up pre-commit hooks and verify tooling
bash scripts/setup-dev.sh
# verify the CLI
agents --help
agents --version
Developing
Pre-commit hooks run automatically on every git commit (formatting, linting, type
checking, security scanning). To run checks manually:
# core validation
nox -s format # ruff auto-formatting
nox -s lint # ruff linting
nox -s typecheck # pyright type checking
nox -s unit_tests # behave unit tests
nox -s integration_tests # robot integration tests
# quality & security
nox -s security_scan # bandit security scanning
nox -s dead_code # vulture dead code detection
nox -s complexity # radon complexity analysis
nox -s pre_commit # run all pre-commit hooks
nox -s adr_compliance # verify ADR compliance
For the full quality automation guide, see docs/development/quality-automation.md.
Documentation
nox -s docs
nox -s serve_docs
Tests
Behave feature scenarios live under features/ and Robot suites under robot/. Use the Nox sessions above to execute them in parity with the implementation plan.
Observability
LangSmith tracing is optional and off by default. Enable it by exporting CLEVERAGENTS_LANGSMITH_ENABLED=true along with a project name and API key (CLEVERAGENTS_LANGSMITH_PROJECT, CLEVERAGENTS_LANGSMITH_API_KEY). The settings module automatically mirrors these values to LANGCHAIN_TRACING_V2, LANGCHAIN_PROJECT, and LANGCHAIN_API_KEY, so LangChain/LangGraph agents emit traces without extra wiring. Additional knobs such as CLEVERAGENTS_LANGSMITH_ENDPOINT, CLEVERAGENTS_LANGSMITH_USER_ID, and CLEVERAGENTS_LANGSMITH_TAGS are documented in docs/observability.md.
LLM provider configuration
CleverAgents ships with a LangChain/LangGraph powered provider registry that discovers whichever API keys you export and automatically selects the best available provider. The CLI now uses actors: select an actor with --actor <name> (or set a default via agents actor set-default). Actors embed provider/model choices; built-in actors are seeded from CLEVERAGENTS_DEFAULT_PROVIDER / CLEVERAGENTS_DEFAULT_MODEL, then fall back to the built-in order (openai → anthropic → google → azure → openrouter → groq → together → cohere → gemini).
Required environment variables
| Provider | Primary variables |
|---|---|
| OpenAI | OPENAI_API_KEY |
| Anthropic | ANTHROPIC_API_KEY |
GOOGLE_API_KEY or GOOGLE_GENAI_API_KEY |
|
| Azure OpenAI | AZURE_OPENAI_API_KEY plus AZURE_OPENAI_ENDPOINT/AZURE_OPENAI_DEPLOYMENT |
| OpenRouter | OPENROUTER_API_KEY (+ optional CLEVERAGENTS_OPENROUTER_ORGANIZATION for sanitized headers) |
| Gemini | GEMINI_API_KEY or GOOGLE_GEMINI_API_KEY |
| Cohere | COHERE_API_KEY |
| Groq | GROQ_API_KEY |
| Together | TOGETHER_API_KEY |
Set CLEVERAGENTS_DEFAULT_PROVIDER to pin the global provider (for example export CLEVERAGENTS_DEFAULT_PROVIDER=openai) and CLEVERAGENTS_DEFAULT_MODEL to lock in a model ID. When unset, the registry picks the first configured provider and uses its published default model such as gpt-4o for OpenAI or claude-sonnet-4-20250514 for Anthropic.
Diagnostics and testing shortcuts
agents diagnosticsprints whether the registry can see your credentials and which actor/provider is selected.agents tellandagents buildrequire--actor <name>unless a default actor is set; useagents actor set-default <name>to configure one.- Built-in actors (
<provider>/<model>) are immutable, custom actors must be namedlocal/<id>, and the default actor cannot be removed. Use--unsafewhen adding/updating configs marked unsafe; runtime only warns when invoking unsafe actors. CLEVERAGENTS_TESTING_USE_MOCK_AI=trueforces the in-repo mock provider so Behave/Robot suites never hit external APIs.- The full capability matrix (streaming, tool calls, JSON mode, etc.) is documented in
docs/reference/providers.md.