forked from cleveragents/cleveragents-core
brent.edwards
b88bc0ec1b
## Summary Add large project scaling benchmarks and tests at production scale (10K–100K files). ### New ASV Benchmarks **IndexingScalingSuite** (`large_project_scaling_bench.py`): - `time_walk_and_index` at 1K/10K/50K/100K files - `time_incremental_refresh` (1% modified files) - `track_indexed_file_count`, `track_tokens_per_second` **ContextAssemblyScalingSuite** (`context_assembly_scaling_bench.py`): - `time_full_pipeline` at 100/1K/5K/10K fragments - `time_tiered_strategy`, `time_recency_strategy` - `track_assembled_tokens`, `track_fragments_per_second` **ExecutionThroughputSuite** (`execution_throughput_bench.py`): - `time_sequential_plans` at 10/50/100 plans - `time_executor_construction`, `time_decision_tree_scaling` ### Scale Fixture Updates - Added `xlarge` (50K files) and `xxlarge` (100K files) profiles to `scale_metadata.json` - Added 50K/100K thresholds to `baseline_thresholds.json` - Added `context_assembly` and `execution_throughput` threshold sections ### Tests & Documentation - 15 Behave scenarios validating profiles, thresholds, monotonicity, memory budgets - 6 Robot integration tests including live 1K-file indexing throughput check - `docs/reference/scaling_baselines.md` documenting all baseline metrics ### Quality Gates | Session | Result | |---|---| | `nox -s lint` | PASS | | `nox -s typecheck` | PASS (0 errors) | | `nox -s unit_tests` | PASS (10,910 scenarios) | | `nox -s integration_tests` | PASS (1,526 tests) | | `nox -s coverage_report` | 97% (>= 97%) | Closes #859 Reviewed-on: cleveragents/cleveragents-core#984 Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
106 lines
2.5 KiB
JSON
106 lines
2.5 KiB
JSON
{
|
|
"schema_version": "1.0",
|
|
"description": "Scale test fixture metadata for CleverAgents performance testing",
|
|
"scale_profiles": [
|
|
{
|
|
"name": "small",
|
|
"file_count": 1000,
|
|
"total_size_mb": 50,
|
|
"language_mix": {
|
|
"python": 0.6,
|
|
"javascript": 0.3,
|
|
"yaml": 0.1
|
|
},
|
|
"expected_index_time_s": 5.0,
|
|
"expected_decomposition_time_s": 10.0,
|
|
"description": "Small repository representative of a single microservice"
|
|
},
|
|
{
|
|
"name": "medium",
|
|
"file_count": 5000,
|
|
"total_size_mb": 250,
|
|
"language_mix": {
|
|
"python": 0.5,
|
|
"javascript": 0.25,
|
|
"typescript": 0.15,
|
|
"yaml": 0.1
|
|
},
|
|
"expected_index_time_s": 25.0,
|
|
"expected_decomposition_time_s": 50.0,
|
|
"description": "Medium repository representative of a monorepo with several services"
|
|
},
|
|
{
|
|
"name": "large",
|
|
"file_count": 10000,
|
|
"total_size_mb": 500,
|
|
"language_mix": {
|
|
"python": 0.4,
|
|
"javascript": 0.2,
|
|
"typescript": 0.2,
|
|
"yaml": 0.1,
|
|
"json": 0.1
|
|
},
|
|
"expected_index_time_s": 50.0,
|
|
"expected_decomposition_time_s": 100.0,
|
|
"description": "Large repository representative of a full enterprise monorepo"
|
|
},
|
|
{
|
|
"name": "xlarge",
|
|
"file_count": 50000,
|
|
"total_size_mb": 2500,
|
|
"language_mix": {
|
|
"python": 0.35,
|
|
"javascript": 0.2,
|
|
"typescript": 0.2,
|
|
"yaml": 0.1,
|
|
"json": 0.1,
|
|
"markdown": 0.05
|
|
},
|
|
"expected_index_time_s": 200.0,
|
|
"expected_decomposition_time_s": 400.0,
|
|
"description": "Extra-large repository representative of a multi-team enterprise monorepo"
|
|
},
|
|
{
|
|
"name": "xxlarge",
|
|
"file_count": 100000,
|
|
"total_size_mb": 5000,
|
|
"language_mix": {
|
|
"python": 0.3,
|
|
"javascript": 0.2,
|
|
"typescript": 0.2,
|
|
"yaml": 0.1,
|
|
"json": 0.1,
|
|
"markdown": 0.05,
|
|
"html": 0.05
|
|
},
|
|
"expected_index_time_s": 450.0,
|
|
"expected_decomposition_time_s": 900.0,
|
|
"description": "Maximum-scale repository representative of a large enterprise codebase"
|
|
}
|
|
],
|
|
"supported_languages": [
|
|
"python",
|
|
"javascript",
|
|
"typescript",
|
|
"yaml",
|
|
"json",
|
|
"markdown",
|
|
"html",
|
|
"css",
|
|
"sql",
|
|
"shell"
|
|
],
|
|
"avg_file_size_kb": {
|
|
"python": 8.5,
|
|
"javascript": 6.2,
|
|
"typescript": 7.8,
|
|
"yaml": 2.1,
|
|
"json": 3.4,
|
|
"markdown": 4.0,
|
|
"html": 12.0,
|
|
"css": 5.5,
|
|
"sql": 3.0,
|
|
"shell": 1.5
|
|
}
|
|
}
|