brent.edwards
b88bc0ec1b
CI / build (push) Successful in 19s
CI / lint (push) Successful in 3m19s
CI / quality (push) Successful in 3m43s
CI / typecheck (push) Successful in 3m55s
CI / security (push) Successful in 4m2s
CI / unit_tests (push) Successful in 6m39s
CI / integration_tests (push) Successful in 6m48s
CI / docker (push) Successful in 1m7s
CI / e2e_tests (push) Successful in 9m7s
CI / benchmark-regression (push) Has been skipped
CI / coverage (push) Failing after 16m36s
CI / benchmark-publish (push) Successful in 25m51s
CI / status-check (push) Failing after 4s
## Summary Add large project scaling benchmarks and tests at production scale (10K–100K files). ### New ASV Benchmarks **IndexingScalingSuite** (`large_project_scaling_bench.py`): - `time_walk_and_index` at 1K/10K/50K/100K files - `time_incremental_refresh` (1% modified files) - `track_indexed_file_count`, `track_tokens_per_second` **ContextAssemblyScalingSuite** (`context_assembly_scaling_bench.py`): - `time_full_pipeline` at 100/1K/5K/10K fragments - `time_tiered_strategy`, `time_recency_strategy` - `track_assembled_tokens`, `track_fragments_per_second` **ExecutionThroughputSuite** (`execution_throughput_bench.py`): - `time_sequential_plans` at 10/50/100 plans - `time_executor_construction`, `time_decision_tree_scaling` ### Scale Fixture Updates - Added `xlarge` (50K files) and `xxlarge` (100K files) profiles to `scale_metadata.json` - Added 50K/100K thresholds to `baseline_thresholds.json` - Added `context_assembly` and `execution_throughput` threshold sections ### Tests & Documentation - 15 Behave scenarios validating profiles, thresholds, monotonicity, memory budgets - 6 Robot integration tests including live 1K-file indexing throughput check - `docs/reference/scaling_baselines.md` documenting all baseline metrics ### Quality Gates | Session | Result | |---|---| | `nox -s lint` | PASS | | `nox -s typecheck` | PASS (0 errors) | | `nox -s unit_tests` | PASS (10,910 scenarios) | | `nox -s integration_tests` | PASS (1,526 tests) | | `nox -s coverage_report` | 97% (>= 97%) | Closes #859 Reviewed-on: #984 Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
4.0 KiB
4.0 KiB
Scaling Baselines
Performance baselines and expectations for CleverAgents at production scale.
Overview
CleverAgents is designed to handle repositories ranging from small microservices (1K files) to large enterprise monorepos (100K+ files). This document captures the baseline performance thresholds and scaling expectations used by the CI benchmark suite.
Scale Profiles
| Profile | Files | Size (MB) | Index Time (s) | Decompose Time (s) |
|---|---|---|---|---|
| small | 1,000 | 50 | 5 | 10 |
| medium | 5,000 | 250 | 25 | 50 |
| large | 10,000 | 500 | 50 | 100 |
| xlarge | 50,000 | 2,500 | 200 | 400 |
| xxlarge | 100,000 | 5,000 | 450 | 900 |
Indexing Thresholds
Latency percentiles for walk_and_index on local SSD with 8+ GB RAM:
| File Count | p50 (ms) | p95 (ms) | p99 (ms) | Max (ms) |
|---|---|---|---|---|
| 1,000 | 2,000 | 5,000 | 8,000 | 12,000 |
| 5,000 | 10,000 | 25,000 | 40,000 | 60,000 |
| 10,000 | 18,000 | 45,000 | 72,000 | 110,000 |
| 50,000 | 75,000 | 180,000 | 290,000 | 450,000 |
| 100,000 | 140,000 | 340,000 | 550,000 | 850,000 |
Decomposition Thresholds
Decomposition typically takes 2x indexing time:
| File Count | p50 (ms) | p95 (ms) | p99 (ms) | Max (ms) |
|---|---|---|---|---|
| 1,000 | 4,000 | 10,000 | 16,000 | 24,000 |
| 5,000 | 20,000 | 50,000 | 80,000 | 120,000 |
| 10,000 | 40,000 | 100,000 | 160,000 | 240,000 |
| 50,000 | 160,000 | 400,000 | 640,000 | 960,000 |
| 100,000 | 300,000 | 750,000 | 1,200,000 | 1,800,000 |
Context Assembly Thresholds
ACMS pipeline latency for varying fragment counts:
| Fragments | p50 (ms) | p95 (ms) | p99 (ms) |
|---|---|---|---|
| 100 | 50 | 150 | 300 |
| 1,000 | 500 | 1,500 | 3,000 |
| 5,000 | 2,500 | 7,500 | 15,000 |
| 10,000 | 5,000 | 15,000 | 30,000 |
Execution Throughput Thresholds
Plan execution dispatch latency:
| Plans | p50 (ms) | p95 (ms) | p99 (ms) |
|---|---|---|---|
| 10 | 200 | 500 | 1,000 |
| 50 | 1,000 | 2,500 | 5,000 |
| 100 | 2,000 | 5,000 | 10,000 |
Memory Usage
| File Count | Peak (MB) | Steady State (MB) |
|---|---|---|
| 1,000 | 256 | 128 |
| 5,000 | 768 | 384 |
| 10,000 | 1,536 | 768 |
| 50,000 | 6,144 | 3,072 |
| 100,000 | 12,288 | 6,144 |
Scaling Expectations
- Indexing: O(n log n) — sub-linear scaling expected
- Decomposition: O(n log n) — sub-linear scaling expected
- Context Assembly: O(n log n) — sub-linear for large fragment sets
- Execution: O(n) — linear scaling expected
- Memory: O(n) — linear scaling expected
Sub-linear Scaling Invariants
The benchmark suite enforces these invariants:
- 10K indexing p50 < 10x the 1K p50
- 50K indexing p50 < 5x the 10K p50
- 100K indexing p50 < 2.5x the 50K p50
ASV Benchmark Suites
| Suite | Params | Timeout |
|---|---|---|
IndexingScalingSuite |
1K, 10K, 50K, 100K | 600s |
ContextAssemblyScalingSuite |
100, 1K, 5K, 10K | 300s |
ExecutionThroughputSuite |
10, 50, 100 | 300s |
Fixture Files
features/fixtures/scale/scale_metadata.json— profile definitionsfeatures/fixtures/scale/baseline_thresholds.json— p50/p95/p99 thresholdsfeatures/fixtures/scale/generator_instructions.md— fixture generation guide