forked from HAL9000/cleveragents-core
brent.edwards
b88bc0ec1b
## Summary Add large project scaling benchmarks and tests at production scale (10K–100K files). ### New ASV Benchmarks **IndexingScalingSuite** (`large_project_scaling_bench.py`): - `time_walk_and_index` at 1K/10K/50K/100K files - `time_incremental_refresh` (1% modified files) - `track_indexed_file_count`, `track_tokens_per_second` **ContextAssemblyScalingSuite** (`context_assembly_scaling_bench.py`): - `time_full_pipeline` at 100/1K/5K/10K fragments - `time_tiered_strategy`, `time_recency_strategy` - `track_assembled_tokens`, `track_fragments_per_second` **ExecutionThroughputSuite** (`execution_throughput_bench.py`): - `time_sequential_plans` at 10/50/100 plans - `time_executor_construction`, `time_decision_tree_scaling` ### Scale Fixture Updates - Added `xlarge` (50K files) and `xxlarge` (100K files) profiles to `scale_metadata.json` - Added 50K/100K thresholds to `baseline_thresholds.json` - Added `context_assembly` and `execution_throughput` threshold sections ### Tests & Documentation - 15 Behave scenarios validating profiles, thresholds, monotonicity, memory budgets - 6 Robot integration tests including live 1K-file indexing throughput check - `docs/reference/scaling_baselines.md` documenting all baseline metrics ### Quality Gates | Session | Result | |---|---| | `nox -s lint` | PASS | | `nox -s typecheck` | PASS (0 errors) | | `nox -s unit_tests` | PASS (10,910 scenarios) | | `nox -s integration_tests` | PASS (1,526 tests) | | `nox -s coverage_report` | 97% (>= 97%) | Closes #859 Reviewed-on: cleveragents/cleveragents-core#984 Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
4.0 KiB
4.0 KiB
Scaling Baselines
Performance baselines and expectations for CleverAgents at production scale.
Overview
CleverAgents is designed to handle repositories ranging from small microservices (1K files) to large enterprise monorepos (100K+ files). This document captures the baseline performance thresholds and scaling expectations used by the CI benchmark suite.
Scale Profiles
| Profile | Files | Size (MB) | Index Time (s) | Decompose Time (s) |
|---|---|---|---|---|
| small | 1,000 | 50 | 5 | 10 |
| medium | 5,000 | 250 | 25 | 50 |
| large | 10,000 | 500 | 50 | 100 |
| xlarge | 50,000 | 2,500 | 200 | 400 |
| xxlarge | 100,000 | 5,000 | 450 | 900 |
Indexing Thresholds
Latency percentiles for walk_and_index on local SSD with 8+ GB RAM:
| File Count | p50 (ms) | p95 (ms) | p99 (ms) | Max (ms) |
|---|---|---|---|---|
| 1,000 | 2,000 | 5,000 | 8,000 | 12,000 |
| 5,000 | 10,000 | 25,000 | 40,000 | 60,000 |
| 10,000 | 18,000 | 45,000 | 72,000 | 110,000 |
| 50,000 | 75,000 | 180,000 | 290,000 | 450,000 |
| 100,000 | 140,000 | 340,000 | 550,000 | 850,000 |
Decomposition Thresholds
Decomposition typically takes 2x indexing time:
| File Count | p50 (ms) | p95 (ms) | p99 (ms) | Max (ms) |
|---|---|---|---|---|
| 1,000 | 4,000 | 10,000 | 16,000 | 24,000 |
| 5,000 | 20,000 | 50,000 | 80,000 | 120,000 |
| 10,000 | 40,000 | 100,000 | 160,000 | 240,000 |
| 50,000 | 160,000 | 400,000 | 640,000 | 960,000 |
| 100,000 | 300,000 | 750,000 | 1,200,000 | 1,800,000 |
Context Assembly Thresholds
ACMS pipeline latency for varying fragment counts:
| Fragments | p50 (ms) | p95 (ms) | p99 (ms) |
|---|---|---|---|
| 100 | 50 | 150 | 300 |
| 1,000 | 500 | 1,500 | 3,000 |
| 5,000 | 2,500 | 7,500 | 15,000 |
| 10,000 | 5,000 | 15,000 | 30,000 |
Execution Throughput Thresholds
Plan execution dispatch latency:
| Plans | p50 (ms) | p95 (ms) | p99 (ms) |
|---|---|---|---|
| 10 | 200 | 500 | 1,000 |
| 50 | 1,000 | 2,500 | 5,000 |
| 100 | 2,000 | 5,000 | 10,000 |
Memory Usage
| File Count | Peak (MB) | Steady State (MB) |
|---|---|---|
| 1,000 | 256 | 128 |
| 5,000 | 768 | 384 |
| 10,000 | 1,536 | 768 |
| 50,000 | 6,144 | 3,072 |
| 100,000 | 12,288 | 6,144 |
Scaling Expectations
- Indexing: O(n log n) — sub-linear scaling expected
- Decomposition: O(n log n) — sub-linear scaling expected
- Context Assembly: O(n log n) — sub-linear for large fragment sets
- Execution: O(n) — linear scaling expected
- Memory: O(n) — linear scaling expected
Sub-linear Scaling Invariants
The benchmark suite enforces these invariants:
- 10K indexing p50 < 10x the 1K p50
- 50K indexing p50 < 5x the 10K p50
- 100K indexing p50 < 2.5x the 50K p50
ASV Benchmark Suites
| Suite | Params | Timeout |
|---|---|---|
IndexingScalingSuite |
1K, 10K, 50K, 100K | 600s |
ContextAssemblyScalingSuite |
100, 1K, 5K, 10K | 300s |
ExecutionThroughputSuite |
10, 50, 100 | 300s |
Fixture Files
features/fixtures/scale/scale_metadata.json— profile definitionsfeatures/fixtures/scale/baseline_thresholds.json— p50/p95/p99 thresholdsfeatures/fixtures/scale/generator_instructions.md— fixture generation guide