hurui200320 774dfedc6b test: add TDD bug-capture test for #967 — plan execute phase processing (#1050)
## Summary

This PR adds TDD bug-capture tests for bug #967 — `plan execute` only transitions state without running strategize or execute phase processing.

### Motivation

Bug #967 describes that the `plan execute` CLI command originally only called `service.execute_plan(plan_id)`, which is a state transition only (Strategize/COMPLETE → Execute/QUEUED). When a plan was in Strategize/QUEUED state (immediately after `plan use`), the command failed because `execute_plan()` requires Strategize/COMPLETE. The CLI should detect the plan's current phase and run `PlanExecutor.run_strategize()` before transitioning.

Per the project's TDD Bug Fix Workflow (`CONTRIBUTING.md`), the first step in fixing any bug is to write a test that captures the buggy behavior. Since the fix for #967 is already present in the codebase (the CLI handler already orchestrates properly), the `@tdd_expected_fail` tags have been removed and these tests serve as **permanent regression guards** ensuring the fix is never reverted.

### Design Approach

**All `@tdd_expected_fail` scenarios were rewritten to exercise the CLI orchestration layer** — the actual code path affected by bug #967. This satisfies AC4: "The test is specific enough that it will pass normally (without the tag) only when the bug is genuinely fixed."

- **Scenarios 1, 2, 4** (previously `@tdd_expected_fail`): Use Typer's `CliRunner` with mocked services to invoke the `plan execute` CLI command handler directly. This tests the orchestration logic in `plan.py` — the exact code that was buggy.
- **Scenario 3** (positive control): Uses real `PlanLifecycleService` (in-memory) and `PlanExecutor` (stub actors) to demonstrate that proper service-level orchestration works.
- **Robot tests**: Replicate the CLI orchestration logic using real services to verify at the integration level.

### Changes

#### Behave Unit Tests
- `features/tdd_plan_execute_phase_processing.feature` — 4 scenarios
- `features/steps/tdd_plan_execute_phase_processing_steps.py` — Step definitions using CliRunner and mocked services

**Scenarios:**
1. **CLI execute command handles plan in Strategize/QUEUED state** — Invokes `plan execute` via CliRunner on a QUEUED plan. Verifies the CLI succeeds and the plan reaches Execute phase.
2. **CLI execute command orchestrates full lifecycle for QUEUED plan** — Verifies `run_strategize()` and `run_execute()` are both called by the CLI handler.
3. **Positive control — proper orchestration transitions QUEUED plan to Execute** — Demonstrates that `run_strategize()` → `execute_plan()` works correctly at the service level. Always passes.
4. **CLI auto-discovery finds plans in Strategize/QUEUED state** — Invokes `plan execute` with no plan_id. Verifies the auto-discovery filter includes QUEUED plans.

#### Robot Integration Tests
- `robot/tdd_plan_execute_phase_processing.robot` — 4 test cases matching the Behave scenarios
- `robot/helper_tdd_plan_execute_phase_processing.py` — Helper script replicating CLI orchestration logic

#### CHANGELOG
- `CHANGELOG.md` — Added entry under "## Unreleased" describing the new tests.

### Quality Gates
- `nox -e lint`:  passed
- `nox -e typecheck`:  passed (0 errors)
- `nox -e unit_tests`:  passed (391 features, 11177 scenarios, 0 failures)
- `nox -e integration_tests`:  passed (1572 tests, 0 failures)
- `nox -e e2e_tests`:  passed (16 tests, 0 failures)
- `nox -e coverage_report`:  97% coverage

### Review Cycle 2 Fixes
- **Critical #1**: Rewrote `@tdd_expected_fail` scenarios to exercise the CLI orchestration layer via CliRunner instead of testing service/executor APIs that are correct by design. Removed `@tdd_expected_fail` tags since the bug fix is already in the codebase.
- **Major #2**: Added CHANGELOG entry.
- **Major #3**: Rebased onto current `master`.
- **Minor #4**: Added `ProcessingState.QUEUED` assertion to positive control scenario.
- **Minor #5**: Narrowed exception handler from bare `except Exception` to `except (PlanError, PlanNotReadyError)`.
- **Minor #7**: Changed Robot suite to use `Setup Test Environment With Database Isolation`.
- **Minor #8**: Added `on_timeout=kill` to all Robot `Run Process` calls.
- **Minor #12**: Added docstring to `_fail()` helper.
- **Minor #13**: Removed redundant `Settings()` instantiation.

### Known Limitations
- The `@tdd_expected_fail` tags were removed because the bug fix for #967 is already in the codebase. If this PR is merged before the #967 fix PR, the tags would need to be re-added. However, the CHANGELOG and existing CLI code confirm the fix is already on `master`.

Closes #977

Reviewed-on: cleveragents/cleveragents-core#1050
Co-authored-by: Rui Hu <rui.hu@cleverthis.com>
Co-committed-by: Rui Hu <rui.hu@cleverthis.com>
2026-03-19 06:41:26 +00:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2026-02-23 22:20:03 +00:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:14:00 -05:00
2024-01-25 23:14:00 -05:00

CleverAgents Core

CleverAgents is a Python-first automation platform. It provides a unified agents CLI, embedded runtime, and service orchestration tools while embracing modern Python tooling.

Highlights

  • Unified CLI entry points: cleveragents and agents
  • Fast Typer/Click-based interface with parity for help/version behavior
  • Behavior-driven coverage via Behave and Robot Framework
  • Nox automation for linting, typing, testing, docs, builds, and benchmarks
  • MkDocs-powered documentation with CleverAgents branding

Quick Start

# clone the CleverAgents core repository
git clone https://git.cleverthis.com/cleveragents/core.git
cd core

# install dependencies
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,tests,docs]"

# set up pre-commit hooks and verify tooling
bash scripts/setup-dev.sh

# verify the CLI
agents --help
agents --version

Developing

Pre-commit hooks run automatically on every git commit (formatting, linting, type checking, security scanning). To run checks manually:

# core validation
nox -s format            # ruff auto-formatting
nox -s lint              # ruff linting
nox -s typecheck         # pyright type checking
nox -s unit_tests        # behave unit tests
nox -s integration_tests # robot integration tests

# quality & security
nox -s security_scan     # bandit security scanning
nox -s dead_code         # vulture dead code detection
nox -s complexity        # radon complexity analysis
nox -s pre_commit        # run all pre-commit hooks
nox -s adr_compliance    # verify ADR compliance

For the full quality automation guide, see docs/development/quality-automation.md.

Documentation

nox -s docs
nox -s serve_docs

Tests

Behave feature scenarios live under features/ and Robot suites under robot/. Use the Nox sessions above to execute them in parity with the implementation plan.

Observability

LangSmith tracing is optional and off by default. Enable it by exporting CLEVERAGENTS_LANGSMITH_ENABLED=true along with a project name and API key (CLEVERAGENTS_LANGSMITH_PROJECT, CLEVERAGENTS_LANGSMITH_API_KEY). The settings module automatically mirrors these values to LANGCHAIN_TRACING_V2, LANGCHAIN_PROJECT, and LANGCHAIN_API_KEY, so LangChain/LangGraph agents emit traces without extra wiring. Additional knobs such as CLEVERAGENTS_LANGSMITH_ENDPOINT, CLEVERAGENTS_LANGSMITH_USER_ID, and CLEVERAGENTS_LANGSMITH_TAGS are documented in docs/observability.md.

LLM provider configuration

CleverAgents ships with a LangChain/LangGraph powered provider registry that discovers whichever API keys you export and automatically selects the best available provider. The CLI now uses actors: select an actor with --actor <name> (or set a default via agents actor set-default). Actors embed provider/model choices; built-in actors are seeded from CLEVERAGENTS_DEFAULT_PROVIDER / CLEVERAGENTS_DEFAULT_MODEL, then fall back to the built-in order (openai → anthropic → google → azure → openrouter → groq → together → cohere → gemini).

Required environment variables

Provider Primary variables
OpenAI OPENAI_API_KEY
Anthropic ANTHROPIC_API_KEY
Google GOOGLE_API_KEY or GOOGLE_GENAI_API_KEY
Azure OpenAI AZURE_OPENAI_API_KEY plus AZURE_OPENAI_ENDPOINT/AZURE_OPENAI_DEPLOYMENT
OpenRouter OPENROUTER_API_KEY (+ optional CLEVERAGENTS_OPENROUTER_ORGANIZATION for sanitized headers)
Gemini GEMINI_API_KEY or GOOGLE_GEMINI_API_KEY
Cohere COHERE_API_KEY
Groq GROQ_API_KEY
Together TOGETHER_API_KEY

Set CLEVERAGENTS_DEFAULT_PROVIDER to pin the global provider (for example export CLEVERAGENTS_DEFAULT_PROVIDER=openai) and CLEVERAGENTS_DEFAULT_MODEL to lock in a model ID. When unset, the registry picks the first configured provider and uses its published default model such as gpt-4o for OpenAI or claude-sonnet-4-20250514 for Anthropic.

Diagnostics and testing shortcuts

  • agents diagnostics prints whether the registry can see your credentials and which actor/provider is selected.
  • agents tell and agents build require --actor <name> unless a default actor is set; use agents actor set-default <name> to configure one.
  • Built-in actors (<provider>/<model>) are immutable, custom actors must be named local/<id>, and the default actor cannot be removed. Use --unsafe when adding/updating configs marked unsafe; runtime only warns when invoking unsafe actors.
  • CLEVERAGENTS_TESTING_USE_MOCK_AI=true forces the in-repo mock provider so Behave/Robot suites never hit external APIs.
  • The full capability matrix (streaming, tool calls, JSON mode, etc.) is documented in docs/reference/providers.md.
S
Description
No description provided
Readme 58 MiB
Languages
Python 76%
Gherkin 18.4%
RobotFramework 5.5%