hurui200320 09d92ac67b test(e2e): validate M4 acceptance criteria for v3.3.0 milestone closure (#560)
## Summary

Validates all M4 acceptance criteria for v3.3.0 milestone closure. Adds CLI-exercising integration tests for the three subplan commands that lacked actual CLI invocation, fixes a pre-existing unit test failure, and corrects CONTRIBUTORS.md ordering.

## Changes

### New CLI Integration Tests (Robot Framework)

Three new test cases in `robot/m4_e2e_verification.robot` with helper functions in `robot/helper_m4_e2e_verification.py`:

| Test | CLI Command | Mocking Approach |
|------|-------------|-----------------|
| `CLI Plan Use Creates Plan With Subplan Config` | `plan use local/refactor-action local/monorepo` | Patches `_get_lifecycle_service`; mocks `get_action_by_name` + `use_action` |
| `CLI Plan Execute Transitions With Subplans` | `plan execute <plan_id>` | Patches `_get_lifecycle_service`; mocks `get_plan` + `execute_plan` |
| `CLI Plan Tree Displays Subplan Hierarchy` | `plan tree <plan_id> --format json` | Patches `get_container`; real `Decision` objects with `SUBPLAN_SPAWN`/`SUBPLAN_PARALLEL_SPAWN` types |

All three follow the same pattern as the existing `plan-diff` test: Typer `CliRunner.invoke()` with mocked services.

### Pre-existing Unit Test Fix

**File:** `features/steps/repositories_uncovered_branches_steps.py`
**Scenario:** `repo branch cov upsert profile with schema version mismatch` (line 110)
**Root cause:** Plain `sessionmaker` returns a new session per call. The `Given` step inserted via session A and committed session B (different session), so the `When` step's session C couldn't see the uncommitted data.
**Fix:** Changed to `scoped_session(sessionmaker(...))` so all factory calls return the same thread-local session.

### Other Fixes
- **CONTRIBUTORS.md**: Moved "Rui Hu" to correct alphabetical position (between Freeman and Khyari)
- **CHANGELOG.md**: Updated #495 entry to document CLI test additions

## M4 Acceptance Criteria Verification

All 7 criteria exercised by 10 E2E tests (7 existing + 3 new) + 8 smoke tests:

| # | Criterion | Test(s) | Status |
|---|-----------|---------|--------|
| 1 | Subplans spawned during Execute via SubplanConfig | `spawn-subplans`, **`cli-plan-use`**, **`cli-plan-execute`** | PASS |
| 2 | Parallel execution with max_parallel bounds | `parallel-exec` | PASS |
| 3 | Three-way merge combines non-conflicting changes | `merge-clean` | PASS |
| 4 | Merge conflicts surfaced with git markers | `merge-conflict` | PASS |
| 5 | Parent plan tracks subplan statuses | `parent-tracking` | PASS |
| 6 | Plan tree displays subplan hierarchy | `plan-tree`, **`cli-plan-tree`** | PASS |
| 7 | Plan diff shows merged results | `plan-diff` (already uses CLI) | PASS |

## Quality Gates

| Stage | Result |
|-------|--------|
| lint | pass |
| format | pass (1074 files unchanged) |
| typecheck | pass (0 errors) |
| unit_tests | 8524 scenarios, 0 failures |
| integration_tests | 1110/1118 pass (8 pre-existing failures in `cli_plan_context_commands.robot`, identical on master) |
| coverage_report | 97% (threshold: 97%) |
| security_scan | pass |
| dead_code | pass |
| docs | pass |
| build | pass |
| benchmark | pass |

## Files Changed

- `CHANGELOG.md` — Updated #495 entry
- `CONTRIBUTORS.md` — Fixed alphabetical ordering
- `features/steps/repositories_uncovered_branches_steps.py` — `scoped_session` fix
- `robot/helper_m4_e2e_verification.py` — 3 new helper functions + imports
- `robot/m4_e2e_verification.robot` — 3 new test cases

ISSUES CLOSED: #495

Reviewed-on: cleveragents/cleveragents-core#560
Reviewed-by: Luis Mendes <luis.mendes@cleverthis.com>
Co-authored-by: Rui Hu <rui.hu@cleverthis.com>
Co-committed-by: Rui Hu <rui.hu@cleverthis.com>
2026-03-06 03:22:45 +00:00
2026-02-22 14:18:24 -05:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2026-02-23 22:20:03 +00:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:14:00 -05:00
2024-01-25 23:14:00 -05:00

CleverAgents Core

CleverAgents is a Python-first automation platform. It provides a unified agents CLI, embedded runtime, and service orchestration tools while embracing modern Python tooling.

Highlights

  • Unified CLI entry points: cleveragents and agents
  • Fast Typer/Click-based interface with parity for help/version behavior
  • Behavior-driven coverage via Behave and Robot Framework
  • Nox automation for linting, typing, testing, docs, builds, and benchmarks
  • MkDocs-powered documentation with CleverAgents branding

Quick Start

# clone the CleverAgents core repository
git clone https://git.cleverthis.com/cleveragents/core.git
cd core

# install dependencies
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,tests,docs]"

# set up pre-commit hooks and verify tooling
bash scripts/setup-dev.sh

# verify the CLI
agents --help
agents --version

Developing

Pre-commit hooks run automatically on every git commit (formatting, linting, type checking, security scanning). To run checks manually:

# core validation
nox -s format            # ruff auto-formatting
nox -s lint              # ruff linting
nox -s typecheck         # pyright type checking
nox -s unit_tests        # behave unit tests
nox -s integration_tests # robot integration tests

# quality & security
nox -s security_scan     # bandit security scanning
nox -s dead_code         # vulture dead code detection
nox -s complexity        # radon complexity analysis
nox -s pre_commit        # run all pre-commit hooks
nox -s adr_compliance    # verify ADR compliance

For the full quality automation guide, see docs/development/quality-automation.md.

Documentation

nox -s docs
nox -s serve_docs

Tests

Behave feature scenarios live under features/ and Robot suites under robot/. Use the Nox sessions above to execute them in parity with the implementation plan.

Observability

LangSmith tracing is optional and off by default. Enable it by exporting CLEVERAGENTS_LANGSMITH_ENABLED=true along with a project name and API key (CLEVERAGENTS_LANGSMITH_PROJECT, CLEVERAGENTS_LANGSMITH_API_KEY). The settings module automatically mirrors these values to LANGCHAIN_TRACING_V2, LANGCHAIN_PROJECT, and LANGCHAIN_API_KEY, so LangChain/LangGraph agents emit traces without extra wiring. Additional knobs such as CLEVERAGENTS_LANGSMITH_ENDPOINT, CLEVERAGENTS_LANGSMITH_USER_ID, and CLEVERAGENTS_LANGSMITH_TAGS are documented in docs/observability.md.

LLM provider configuration

CleverAgents ships with a LangChain/LangGraph powered provider registry that discovers whichever API keys you export and automatically selects the best available provider. The CLI now uses actors: select an actor with --actor <name> (or set a default via agents actor set-default). Actors embed provider/model choices; built-in actors are seeded from CLEVERAGENTS_DEFAULT_PROVIDER / CLEVERAGENTS_DEFAULT_MODEL, then fall back to the built-in order (openai → anthropic → google → azure → openrouter → groq → together → cohere → gemini).

Required environment variables

Provider Primary variables
OpenAI OPENAI_API_KEY
Anthropic ANTHROPIC_API_KEY
Google GOOGLE_API_KEY or GOOGLE_GENAI_API_KEY
Azure OpenAI AZURE_OPENAI_API_KEY plus AZURE_OPENAI_ENDPOINT/AZURE_OPENAI_DEPLOYMENT
OpenRouter OPENROUTER_API_KEY (+ optional CLEVERAGENTS_OPENROUTER_ORGANIZATION for sanitized headers)
Gemini GEMINI_API_KEY or GOOGLE_GEMINI_API_KEY
Cohere COHERE_API_KEY
Groq GROQ_API_KEY
Together TOGETHER_API_KEY

Set CLEVERAGENTS_DEFAULT_PROVIDER to pin the global provider (for example export CLEVERAGENTS_DEFAULT_PROVIDER=openai) and CLEVERAGENTS_DEFAULT_MODEL to lock in a model ID. When unset, the registry picks the first configured provider and uses its published default model such as gpt-4o for OpenAI or claude-sonnet-4-20250514 for Anthropic.

Diagnostics and testing shortcuts

  • agents diagnostics prints whether the registry can see your credentials and which actor/provider is selected.
  • agents tell and agents build require --actor <name> unless a default actor is set; use agents actor set-default <name> to configure one.
  • Built-in actors (<provider>/<model>) are immutable, custom actors must be named local/<id>, and the default actor cannot be removed. Use --unsafe when adding/updating configs marked unsafe; runtime only warns when invoking unsafe actors.
  • CLEVERAGENTS_TESTING_USE_MOCK_AI=true forces the in-repo mock provider so Behave/Robot suites never hit external APIs.
  • The full capability matrix (streaming, tool calls, JSON mode, etc.) is documented in docs/reference/providers.md.
S
Description
No description provided
Readme 58 MiB
Languages
Python 76%
Gherkin 18.4%
RobotFramework 5.5%