brent.edwards b88bc0ec1b
CI / build (push) Successful in 19s
CI / lint (push) Successful in 3m19s
CI / quality (push) Successful in 3m43s
CI / typecheck (push) Successful in 3m55s
CI / security (push) Successful in 4m2s
CI / unit_tests (push) Successful in 6m39s
CI / integration_tests (push) Successful in 6m48s
CI / docker (push) Successful in 1m7s
CI / e2e_tests (push) Successful in 9m7s
CI / benchmark-regression (push) Has been skipped
CI / coverage (push) Failing after 16m36s
CI / benchmark-publish (push) Successful in 25m51s
CI / status-check (push) Failing after 4s
feat(perf): large project scaling tests (#984)
## Summary

Add large project scaling benchmarks and tests at production scale (10K–100K files).

### New ASV Benchmarks

**IndexingScalingSuite** (`large_project_scaling_bench.py`):
- `time_walk_and_index` at 1K/10K/50K/100K files
- `time_incremental_refresh` (1% modified files)
- `track_indexed_file_count`, `track_tokens_per_second`

**ContextAssemblyScalingSuite** (`context_assembly_scaling_bench.py`):
- `time_full_pipeline` at 100/1K/5K/10K fragments
- `time_tiered_strategy`, `time_recency_strategy`
- `track_assembled_tokens`, `track_fragments_per_second`

**ExecutionThroughputSuite** (`execution_throughput_bench.py`):
- `time_sequential_plans` at 10/50/100 plans
- `time_executor_construction`, `time_decision_tree_scaling`

### Scale Fixture Updates

- Added `xlarge` (50K files) and `xxlarge` (100K files) profiles to `scale_metadata.json`
- Added 50K/100K thresholds to `baseline_thresholds.json`
- Added `context_assembly` and `execution_throughput` threshold sections

### Tests & Documentation

- 15 Behave scenarios validating profiles, thresholds, monotonicity, memory budgets
- 6 Robot integration tests including live 1K-file indexing throughput check
- `docs/reference/scaling_baselines.md` documenting all baseline metrics

### Quality Gates

| Session | Result |
|---|---|
| `nox -s lint` | PASS |
| `nox -s typecheck` | PASS (0 errors) |
| `nox -s unit_tests` | PASS (10,910 scenarios) |
| `nox -s integration_tests` | PASS (1,526 tests) |
| `nox -s coverage_report` | 97% (>= 97%) |

Closes #859

Reviewed-on: #984
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
2026-03-21 04:46:45 +00:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2026-02-23 22:20:03 +00:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:10:04 -05:00
2024-01-25 23:14:00 -05:00
2024-01-25 23:14:00 -05:00

CleverAgents Core

CleverAgents is a Python-first automation platform. It provides a unified agents CLI, embedded runtime, and service orchestration tools while embracing modern Python tooling.

Highlights

  • Unified CLI entry points: cleveragents and agents
  • Fast Typer/Click-based interface with parity for help/version behavior
  • Behavior-driven coverage via Behave and Robot Framework
  • Nox automation for linting, typing, testing, docs, builds, and benchmarks
  • MkDocs-powered documentation with CleverAgents branding

Quick Start

# clone the CleverAgents core repository
git clone https://git.cleverthis.com/cleveragents/core.git
cd core

# install dependencies
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,tests,docs]"

# set up pre-commit hooks and verify tooling
bash scripts/setup-dev.sh

# verify the CLI
agents --help
agents --version

Developing

Pre-commit hooks run automatically on every git commit (formatting, linting, type checking, security scanning). To run checks manually:

# core validation
nox -s format            # ruff auto-formatting
nox -s lint              # ruff linting
nox -s typecheck         # pyright type checking
nox -s unit_tests        # behave unit tests
nox -s integration_tests # robot integration tests

# quality & security
nox -s security_scan     # bandit security scanning
nox -s dead_code         # vulture dead code detection
nox -s complexity        # radon complexity analysis
nox -s pre_commit        # run all pre-commit hooks
nox -s adr_compliance    # verify ADR compliance

For the full quality automation guide, see docs/development/quality-automation.md.

Documentation

nox -s docs
nox -s serve_docs

Tests

Behave feature scenarios live under features/ and Robot suites under robot/. Use the Nox sessions above to execute them in parity with the implementation plan.

Observability

LangSmith tracing is optional and off by default. Enable it by exporting CLEVERAGENTS_LANGSMITH_ENABLED=true along with a project name and API key (CLEVERAGENTS_LANGSMITH_PROJECT, CLEVERAGENTS_LANGSMITH_API_KEY). The settings module automatically mirrors these values to LANGCHAIN_TRACING_V2, LANGCHAIN_PROJECT, and LANGCHAIN_API_KEY, so LangChain/LangGraph agents emit traces without extra wiring. Additional knobs such as CLEVERAGENTS_LANGSMITH_ENDPOINT, CLEVERAGENTS_LANGSMITH_USER_ID, and CLEVERAGENTS_LANGSMITH_TAGS are documented in docs/observability.md.

LLM provider configuration

CleverAgents ships with a LangChain/LangGraph powered provider registry that discovers whichever API keys you export and automatically selects the best available provider. The CLI now uses actors: select an actor with --actor <name> (or set a default via agents actor set-default). Actors embed provider/model choices; built-in actors are seeded from CLEVERAGENTS_DEFAULT_PROVIDER / CLEVERAGENTS_DEFAULT_MODEL, then fall back to the built-in order (openai → anthropic → google → azure → openrouter → groq → together → cohere → gemini).

Required environment variables

Provider Primary variables
OpenAI OPENAI_API_KEY
Anthropic ANTHROPIC_API_KEY
Google GOOGLE_API_KEY or GOOGLE_GENAI_API_KEY
Azure OpenAI AZURE_OPENAI_API_KEY plus AZURE_OPENAI_ENDPOINT/AZURE_OPENAI_DEPLOYMENT
OpenRouter OPENROUTER_API_KEY (+ optional CLEVERAGENTS_OPENROUTER_ORGANIZATION for sanitized headers)
Gemini GEMINI_API_KEY or GOOGLE_GEMINI_API_KEY
Cohere COHERE_API_KEY
Groq GROQ_API_KEY
Together TOGETHER_API_KEY

Set CLEVERAGENTS_DEFAULT_PROVIDER to pin the global provider (for example export CLEVERAGENTS_DEFAULT_PROVIDER=openai) and CLEVERAGENTS_DEFAULT_MODEL to lock in a model ID. When unset, the registry picks the first configured provider and uses its published default model such as gpt-4o for OpenAI or claude-sonnet-4-20250514 for Anthropic.

Diagnostics and testing shortcuts

  • agents diagnostics prints whether the registry can see your credentials and which actor/provider is selected.
  • agents tell and agents build require --actor <name> unless a default actor is set; use agents actor set-default <name> to configure one.
  • Built-in actors (<provider>/<model>) are immutable, custom actors must be named local/<id>, and the default actor cannot be removed. Use --unsafe when adding/updating configs marked unsafe; runtime only warns when invoking unsafe actors.
  • CLEVERAGENTS_TESTING_USE_MOCK_AI=true forces the in-repo mock provider so Behave/Robot suites never hit external APIs.
  • The full capability matrix (streaming, tool calls, JSON mode, etc.) is documented in docs/reference/providers.md.
S
Description
test
Readme 84 MiB
Languages
Python 75.8%
Gherkin 18.3%
RobotFramework 4.8%
TypeScript 0.9%
Shell 0.2%