## Summary Adds TDD bug-capture tests for bug #1038 — the `agents validation add` command is missing `--required`/`--informational` flags that the specification describes at line 22334 and in numerous walkthrough examples. ### Behave BDD Tests Four Behave BDD scenarios verify the correct expected behavior: 1. **`--required` flag is accepted** — invokes `validation add --config <file> --required` and asserts exit code 0 and mode `"required"` in output. 2. **`--informational` flag is accepted** — same pattern with `--informational`. 3. **`--required` overrides YAML config mode** — creates a YAML config with `mode: informational`, invokes with `--required`, and asserts the CLI output and service layer both reflect `mode: "required"`. 4. **`--informational` overrides YAML config mode** — creates a YAML config with `mode: required`, invokes with `--informational`, and asserts the CLI output and service layer both reflect `mode: "informational"`. ### Robot Framework Integration Tests Four Robot Framework integration tests exercise the same CLI paths as the Behave scenarios via a subprocess helper script: 1. **TDD Validation Add Required Flag Accepted** — verifies `--required` is recognised by the CLI. 2. **TDD Validation Add Informational Flag Accepted** — verifies `--informational` is recognised. 3. **TDD Validation Add Required Flag Overrides YAML Config** — verifies `--required` overrides a YAML config with `mode: informational`. 4. **TDD Validation Add Informational Flag Overrides YAML Config** — verifies `--informational` overrides a YAML config with `mode: required`. The helper script (`robot/helper_tdd_validation_required_flag.py`) uses `typer.testing.CliRunner` with mocked services, following the same pattern as other TDD Robot helpers (e.g., `helper_tdd_plan_apply_yes_flag.py`). ### Tags All scenarios (Behave and Robot) are tagged `@tdd_bug @tdd_bug_1038 @tdd_expected_fail`. Because the bug is still present, the underlying assertions fail (proving the bug exists) and the `@tdd_expected_fail` tag inverts the result so CI passes. Once bug #1038 is fixed, the tag must be removed. ### Spec Contradiction Note Rui Hu's investigation (issue #1038 comment #70755) found that the formal CLI reference (specification.md lines 9279–9290) does NOT include `--required`/`--informational` flags — they appear only in walkthrough examples and specification.md line 22334. Additionally, specification.md line 30761 states: "For entity registration commands (`actor add`, `skill add`, `tool add`, `validation add`, …), the YAML configuration file is the sole source of truth — the `--config` file fully defines the entity and no CLI override flags are accepted." The resolution may be to add the flags to the CLI OR to clean up the spec. This is documented in both the feature file header and step definition module docstring. ### Key Design Decisions - **YAML configs use flat `mode` key** — matches what `Validation.from_config()` actually reads (`config.get("mode", "required")`). The specification's Validation Configuration schema (specification.md line 34109) nests mode under a `validation` key, but the current `from_config()` reads a flat top-level `mode` key. A comment in the step definitions documents this discrepancy. - **Mock uses `side_effect = lambda v: v`** — the `register_tool` mock returns the actual Validation object passed by the CLI, so the CLI output reflects real CLI processing rather than hard-coded mock configuration. This prevents false positives where the mock masks an incomplete fix. - **False-positive guard** — the Then step verifying mode checks both CLI output (via `f"mode: {expected_mode}"` prefix match) AND `register_tool` call args to prevent the mock from masking an incomplete fix. - **Robot helper follows established pattern** — uses `typer.testing.CliRunner` with mocked `_get_tool_registry_service`, matching the pattern from `helper_tdd_plan_apply_yes_flag.py`. Reports real outcome (exit 0 + sentinel on bug fixed, exit 1 on bug present) and relies on `tdd_expected_fail_listener` for result inversion. - **Symmetric override scenarios** — both directions are tested: `--required` overriding informational YAML config, and `--informational` overriding required YAML config. - **Positional NAME omitted** — the bug report includes a positional NAME argument, but that is a separate spec inconsistency not under test here. - **Mutual exclusivity deferred** — testing what happens when both `--required` and `--informational` are passed simultaneously is deferred to the bug-fix PR for #1038. Closes #1102 Reviewed-on: #1133 Reviewed-by: Jeffrey Phillips Freeman <jeffrey.freeman@cleverthis.com> Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
CleverAgents Core
CleverAgents is a Python-first automation platform. It provides a unified agents CLI, embedded runtime, and service orchestration tools while embracing modern Python tooling.
Highlights
- Unified CLI entry points:
cleveragentsandagents - Fast Typer/Click-based interface with parity for help/version behavior
- Behavior-driven coverage via Behave and Robot Framework
- Nox automation for linting, typing, testing, docs, builds, and benchmarks
- MkDocs-powered documentation with CleverAgents branding
Quick Start
# clone the CleverAgents core repository
git clone https://git.cleverthis.com/cleveragents/core.git
cd core
# install dependencies
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,tests,docs]"
# set up pre-commit hooks and verify tooling
bash scripts/setup-dev.sh
# verify the CLI
agents --help
agents --version
Developing
Pre-commit hooks run automatically on every git commit (formatting, linting, type
checking, security scanning). To run checks manually:
# core validation
nox -s format # ruff auto-formatting
nox -s lint # ruff linting
nox -s typecheck # pyright type checking
nox -s unit_tests # behave unit tests
nox -s integration_tests # robot integration tests
# quality & security
nox -s security_scan # bandit security scanning
nox -s dead_code # vulture dead code detection
nox -s complexity # radon complexity analysis
nox -s pre_commit # run all pre-commit hooks
nox -s adr_compliance # verify ADR compliance
For the full quality automation guide, see docs/development/quality-automation.md.
Documentation
nox -s docs
nox -s serve_docs
Tests
Behave feature scenarios live under features/ and Robot suites under robot/. Use the Nox sessions above to execute them in parity with the implementation plan.
Observability
LangSmith tracing is optional and off by default. Enable it by exporting CLEVERAGENTS_LANGSMITH_ENABLED=true along with a project name and API key (CLEVERAGENTS_LANGSMITH_PROJECT, CLEVERAGENTS_LANGSMITH_API_KEY). The settings module automatically mirrors these values to LANGCHAIN_TRACING_V2, LANGCHAIN_PROJECT, and LANGCHAIN_API_KEY, so LangChain/LangGraph agents emit traces without extra wiring. Additional knobs such as CLEVERAGENTS_LANGSMITH_ENDPOINT, CLEVERAGENTS_LANGSMITH_USER_ID, and CLEVERAGENTS_LANGSMITH_TAGS are documented in docs/observability.md.
LLM provider configuration
CleverAgents ships with a LangChain/LangGraph powered provider registry that discovers whichever API keys you export and automatically selects the best available provider. The CLI now uses actors: select an actor with --actor <name> (or set a default via agents actor set-default). Actors embed provider/model choices; built-in actors are seeded from CLEVERAGENTS_DEFAULT_PROVIDER / CLEVERAGENTS_DEFAULT_MODEL, then fall back to the built-in order (openai → anthropic → google → azure → openrouter → groq → together → cohere → gemini).
Required environment variables
| Provider | Primary variables |
|---|---|
| OpenAI | OPENAI_API_KEY |
| Anthropic | ANTHROPIC_API_KEY |
GOOGLE_API_KEY or GOOGLE_GENAI_API_KEY |
|
| Azure OpenAI | AZURE_OPENAI_API_KEY plus AZURE_OPENAI_ENDPOINT/AZURE_OPENAI_DEPLOYMENT |
| OpenRouter | OPENROUTER_API_KEY (+ optional CLEVERAGENTS_OPENROUTER_ORGANIZATION for sanitized headers) |
| Gemini | GEMINI_API_KEY or GOOGLE_GEMINI_API_KEY |
| Cohere | COHERE_API_KEY |
| Groq | GROQ_API_KEY |
| Together | TOGETHER_API_KEY |
Set CLEVERAGENTS_DEFAULT_PROVIDER to pin the global provider (for example export CLEVERAGENTS_DEFAULT_PROVIDER=openai) and CLEVERAGENTS_DEFAULT_MODEL to lock in a model ID. When unset, the registry picks the first configured provider and uses its published default model such as gpt-4o for OpenAI or claude-sonnet-4-20250514 for Anthropic.
Diagnostics and testing shortcuts
agents diagnosticsprints whether the registry can see your credentials and which actor/provider is selected.agents tellandagents buildrequire--actor <name>unless a default actor is set; useagents actor set-default <name>to configure one.- Built-in actors (
<provider>/<model>) are immutable, custom actors must be namedlocal/<id>, and the default actor cannot be removed. Use--unsafewhen adding/updating configs marked unsafe; runtime only warns when invoking unsafe actors. CLEVERAGENTS_TESTING_USE_MOCK_AI=trueforces the in-repo mock provider so Behave/Robot suites never hit external APIs.- The full capability matrix (streaming, tool calls, JSON mode, etc.) is documented in
docs/reference/providers.md.