Implemented Robot Framework E2E test suite for M6 autonomy hardening
acceptance criteria. Tests exercise the real CleverAgents CLI with zero
mocking, covering session lifecycle, automation profiles, project setup,
plan lifecycle via A2A facade, guard enforcement, and a full autonomy
acceptance flow. LLM-dependent tests use Skip If No LLM Keys for
graceful degradation when API keys are unavailable.
Hardened shared E2E keywords (common_e2e.resource):
- Safe JSON parsing with rfind-bounded extraction and error wrapping (C1, M1)
- Multi-object fallback: last-line reverse scan for multi-JSON output (M5)
- Moved Safe Parse Json Field to common_e2e.resource for reuse (L1)
- Migrated deprecated Run Keyword If to IF/ELSE blocks (M4)
- Added cwd parameter to Run CleverAgents Command (M6)
- API key protection via inline evaluation instead of RF variables (S1)
- Git return-code assertions in Create Temp Git Repo (L1)
- Removed unused Collections library import (L2)
- Warning log on directory removal failure instead of silent ignore (L6)
Hardened m6_acceptance.robot:
- Force Tags E2E instead of per-test [Tags] (L3)
- Per-test [Teardown] for resource cleanup, including Init test (M2, M9, L5)
- Collision-safe uuid4 hex suffix instead of randint (L7)
- Initialized session_id to EMPTY before test body for safe teardown (L2)
- ELSE branches with WARN log on conditional assertions (M3, M4, M6)
- Guard enforcement checks specific automation-profile fields (M5)
- Strengthened ci assertions with case-sensitive matching and JSON
field parsing for config get verification (M9)
- Eliminated Python code injection via string interpolation (H2)
- Added assertions to Full Flow Apply Step keyword (H4)
- Removed redundant config set in Guard test (M10)
- Accurate CHANGELOG entry describing actual test scope (C2)
Post-review fixes applied:
- Verify all 8 built-in profiles (manual, review, supervised, cautious,
trusted, auto, ci, full-auto) instead of 4 (M1)
- Session delete confirms removal via re-list (M2)
- Full Flow Apply Step verifies plan phase transition (M3)
- Plan execute output asserts plan_id presence + phase parsing (M4)
- Safe Parse Json Field gains two-strategy approach: outer-bracket
extraction then last-line reverse scan fallback (M5)
- Removed redundant automation profile reset in Config test (L1)
- JSON-quoted assertions ("ci", "auto") prevent false-positive
substring matches on short profile names (L3)
New E2E tests covering remaining acceptance criteria:
- Guard enforcement with custom profile: registers a profile with
explicit denylist, budget cap, and tool-call limits, then verifies
all guard fields via automation-profile show (AC-4 / H1)
- Profile precedence resolution: sets global profile to "review",
creates plan with --automation-profile trusted, asserts plan output
shows "trusted" not "review" (AC-5 / C2)
- Event queue via plan lifecycle transitions: creates plan, captures
initial state, executes, verifies state transition proving domain
event bus delivered and processed events (AC-3 / C1)
- Hierarchical decomposition via plan tree: creates plan with full-auto
profile, executes, runs plan tree --format json, verifies decision
nodes and children structure (AC-6 / C3)
Robot Framework uses dots as hierarchy separators in suite names
(e.g. "E2E.M6 Acceptance"). The E2E Suite Setup keyword replaced
spaces with underscores but preserved dots, producing directory names
like "E2E.M6_Acceptance". The CLI init command derives the project
name from Path.cwd().name, and Project.validate_name() rejects dots
("Name must be alphanumeric with hyphens, underscores, or spaces").
Added a second Replace String call to convert dots to underscores so
the sanitized suite name passes Project name validation.
Also increased subprocess timeouts in m3_e2e_verification.robot (60s->120s)
and m4_e2e_verification.robot (30s->120s) to prevent flaky CI failures from
Python startup overhead.
The plan lifecycle E2E tests call "plan use local/code-review" but
actions are user-defined entities that must be explicitly registered
via "action create --config <yaml>" before use. Without the action,
plan use failed with "Action 'local/code-review' not found" and all
LLM-dependent tests skipped.
Added action registration to M6 Suite Setup that dynamically selects
the actor matching the available API key (anthropic/claude-sonnet-4
when ANTHROPIC_API_KEY is set, openai/gpt-4o otherwise) and creates
the action YAML inline.
ISSUES CLOSED: #746
CleverAgents Core
CleverAgents is a Python-first automation platform. It provides a unified agents CLI, embedded runtime, and service orchestration tools while embracing modern Python tooling.
Highlights
- Unified CLI entry points:
cleveragentsandagents - Fast Typer/Click-based interface with parity for help/version behavior
- Behavior-driven coverage via Behave and Robot Framework
- Nox automation for linting, typing, testing, docs, builds, and benchmarks
- MkDocs-powered documentation with CleverAgents branding
Quick Start
# clone the CleverAgents core repository
git clone https://git.cleverthis.com/cleveragents/core.git
cd core
# install dependencies
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,tests,docs]"
# set up pre-commit hooks and verify tooling
bash scripts/setup-dev.sh
# verify the CLI
agents --help
agents --version
Developing
Pre-commit hooks run automatically on every git commit (formatting, linting, type
checking, security scanning). To run checks manually:
# core validation
nox -s format # ruff auto-formatting
nox -s lint # ruff linting
nox -s typecheck # pyright type checking
nox -s unit_tests # behave unit tests
nox -s integration_tests # robot integration tests
# quality & security
nox -s security_scan # bandit security scanning
nox -s dead_code # vulture dead code detection
nox -s complexity # radon complexity analysis
nox -s pre_commit # run all pre-commit hooks
nox -s adr_compliance # verify ADR compliance
For the full quality automation guide, see docs/development/quality-automation.md.
Documentation
nox -s docs
nox -s serve_docs
Tests
Behave feature scenarios live under features/ and Robot suites under robot/. Use the Nox sessions above to execute them in parity with the implementation plan.
Observability
LangSmith tracing is optional and off by default. Enable it by exporting CLEVERAGENTS_LANGSMITH_ENABLED=true along with a project name and API key (CLEVERAGENTS_LANGSMITH_PROJECT, CLEVERAGENTS_LANGSMITH_API_KEY). The settings module automatically mirrors these values to LANGCHAIN_TRACING_V2, LANGCHAIN_PROJECT, and LANGCHAIN_API_KEY, so LangChain/LangGraph agents emit traces without extra wiring. Additional knobs such as CLEVERAGENTS_LANGSMITH_ENDPOINT, CLEVERAGENTS_LANGSMITH_USER_ID, and CLEVERAGENTS_LANGSMITH_TAGS are documented in docs/observability.md.
LLM provider configuration
CleverAgents ships with a LangChain/LangGraph powered provider registry that discovers whichever API keys you export and automatically selects the best available provider. The CLI now uses actors: select an actor with --actor <name> (or set a default via agents actor set-default). Actors embed provider/model choices; built-in actors are seeded from CLEVERAGENTS_DEFAULT_PROVIDER / CLEVERAGENTS_DEFAULT_MODEL, then fall back to the built-in order (openai → anthropic → google → azure → openrouter → groq → together → cohere → gemini).
Required environment variables
| Provider | Primary variables |
|---|---|
| OpenAI | OPENAI_API_KEY |
| Anthropic | ANTHROPIC_API_KEY |
GOOGLE_API_KEY or GOOGLE_GENAI_API_KEY |
|
| Azure OpenAI | AZURE_OPENAI_API_KEY plus AZURE_OPENAI_ENDPOINT/AZURE_OPENAI_DEPLOYMENT |
| OpenRouter | OPENROUTER_API_KEY (+ optional CLEVERAGENTS_OPENROUTER_ORGANIZATION for sanitized headers) |
| Gemini | GEMINI_API_KEY or GOOGLE_GEMINI_API_KEY |
| Cohere | COHERE_API_KEY |
| Groq | GROQ_API_KEY |
| Together | TOGETHER_API_KEY |
Set CLEVERAGENTS_DEFAULT_PROVIDER to pin the global provider (for example export CLEVERAGENTS_DEFAULT_PROVIDER=openai) and CLEVERAGENTS_DEFAULT_MODEL to lock in a model ID. When unset, the registry picks the first configured provider and uses its published default model such as gpt-4o for OpenAI or claude-sonnet-4-20250514 for Anthropic.
Diagnostics and testing shortcuts
agents diagnosticsprints whether the registry can see your credentials and which actor/provider is selected.agents tellandagents buildrequire--actor <name>unless a default actor is set; useagents actor set-default <name>to configure one.- Built-in actors (
<provider>/<model>) are immutable, custom actors must be namedlocal/<id>, and the default actor cannot be removed. Use--unsafewhen adding/updating configs marked unsafe; runtime only warns when invoking unsafe actors. CLEVERAGENTS_TESTING_USE_MOCK_AI=trueforces the in-repo mock provider so Behave/Robot suites never hit external APIs.- The full capability matrix (streaming, tool calls, JSON mode, etc.) is documented in
docs/reference/providers.md.