brent.edwards
b88bc0ec1b
CI / build (push) Successful in 19s
CI / lint (push) Successful in 3m19s
CI / quality (push) Successful in 3m43s
CI / typecheck (push) Successful in 3m55s
CI / security (push) Successful in 4m2s
CI / unit_tests (push) Successful in 6m39s
CI / integration_tests (push) Successful in 6m48s
CI / docker (push) Successful in 1m7s
CI / e2e_tests (push) Successful in 9m7s
CI / benchmark-regression (push) Has been skipped
CI / coverage (push) Failing after 16m36s
CI / benchmark-publish (push) Successful in 25m51s
CI / status-check (push) Failing after 4s
## Summary Add large project scaling benchmarks and tests at production scale (10K–100K files). ### New ASV Benchmarks **IndexingScalingSuite** (`large_project_scaling_bench.py`): - `time_walk_and_index` at 1K/10K/50K/100K files - `time_incremental_refresh` (1% modified files) - `track_indexed_file_count`, `track_tokens_per_second` **ContextAssemblyScalingSuite** (`context_assembly_scaling_bench.py`): - `time_full_pipeline` at 100/1K/5K/10K fragments - `time_tiered_strategy`, `time_recency_strategy` - `track_assembled_tokens`, `track_fragments_per_second` **ExecutionThroughputSuite** (`execution_throughput_bench.py`): - `time_sequential_plans` at 10/50/100 plans - `time_executor_construction`, `time_decision_tree_scaling` ### Scale Fixture Updates - Added `xlarge` (50K files) and `xxlarge` (100K files) profiles to `scale_metadata.json` - Added 50K/100K thresholds to `baseline_thresholds.json` - Added `context_assembly` and `execution_throughput` threshold sections ### Tests & Documentation - 15 Behave scenarios validating profiles, thresholds, monotonicity, memory budgets - 6 Robot integration tests including live 1K-file indexing throughput check - `docs/reference/scaling_baselines.md` documenting all baseline metrics ### Quality Gates | Session | Result | |---|---| | `nox -s lint` | PASS | | `nox -s typecheck` | PASS (0 errors) | | `nox -s unit_tests` | PASS (10,910 scenarios) | | `nox -s integration_tests` | PASS (1,526 tests) | | `nox -s coverage_report` | 97% (>= 97%) | Closes #859 Reviewed-on: #984 Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
101 lines
5.8 KiB
Gherkin
101 lines
5.8 KiB
Gherkin
Feature: Large project scaling performance baselines
|
|
Validates that benchmark fixtures, threshold definitions, and scaling
|
|
expectations are present and self-consistent for 50K and 100K file
|
|
scales, and that context assembly and execution throughput thresholds
|
|
are defined.
|
|
|
|
Background:
|
|
Given a scaling performance test environment is ready
|
|
|
|
# ──────────────────────────────────────────────────
|
|
# Section 1: Extended scale profile validation
|
|
# ──────────────────────────────────────────────────
|
|
|
|
Scenario: Scale metadata includes xlarge and xxlarge profiles
|
|
When I scaling perf load the scale metadata fixture
|
|
Then the scaling perf metadata should contain profile "xlarge"
|
|
And the scaling perf metadata should contain profile "xxlarge"
|
|
|
|
Scenario: XLarge profile has correct file count
|
|
When I scaling perf load the scale metadata fixture
|
|
Then the scaling perf profile "xlarge" should have file count 50000
|
|
|
|
Scenario: XXLarge profile has correct file count
|
|
When I scaling perf load the scale metadata fixture
|
|
Then the scaling perf profile "xxlarge" should have file count 100000
|
|
|
|
Scenario: New profiles have valid language mixes
|
|
When I scaling perf load the scale metadata fixture
|
|
Then the scaling perf language mix should sum to 1.0 for profile "xlarge"
|
|
And the scaling perf language mix should sum to 1.0 for profile "xxlarge"
|
|
|
|
# ──────────────────────────────────────────────────
|
|
# Section 2: Extended threshold validation
|
|
# ──────────────────────────────────────────────────
|
|
|
|
Scenario: Indexing thresholds cover 50K and 100K files
|
|
When I scaling perf load the baseline thresholds fixture
|
|
Then the scaling perf indexing thresholds should cover "50000_files"
|
|
And the scaling perf indexing thresholds should cover "100000_files"
|
|
|
|
Scenario: Decomposition thresholds cover 50K and 100K files
|
|
When I scaling perf load the baseline thresholds fixture
|
|
Then the scaling perf decomposition thresholds should cover "50000_files"
|
|
And the scaling perf decomposition thresholds should cover "100000_files"
|
|
|
|
Scenario: Extended thresholds are monotonically increasing
|
|
When I scaling perf load the baseline thresholds fixture
|
|
Then scaling perf p50 should be less than p95 for "50000_files" indexing
|
|
And scaling perf p95 should be less than p99 for "50000_files" indexing
|
|
And scaling perf p50 should be less than p95 for "100000_files" indexing
|
|
And scaling perf p95 should be less than p99 for "100000_files" indexing
|
|
|
|
Scenario: 50K indexing threshold scales sub-linearly from 10K
|
|
When I scaling perf load the baseline thresholds fixture
|
|
Then scaling perf 50K indexing p50 should be less than 5x the 10K p50
|
|
|
|
Scenario: 100K indexing threshold scales sub-linearly from 50K
|
|
When I scaling perf load the baseline thresholds fixture
|
|
Then scaling perf 100K indexing p50 should be less than 2.5x the 50K p50
|
|
|
|
# ──────────────────────────────────────────────────
|
|
# Section 3: Context assembly thresholds
|
|
# ──────────────────────────────────────────────────
|
|
|
|
Scenario: Context assembly thresholds section exists
|
|
When I scaling perf load the baseline thresholds fixture
|
|
Then the scaling perf thresholds should have context assembly section
|
|
|
|
Scenario: Context assembly covers 5K and 10K fragments
|
|
When I scaling perf load the baseline thresholds fixture
|
|
Then the scaling perf context assembly thresholds should cover "5000_fragments"
|
|
And the scaling perf context assembly thresholds should cover "10000_fragments"
|
|
|
|
# ──────────────────────────────────────────────────
|
|
# Section 4: Execution throughput thresholds
|
|
# ──────────────────────────────────────────────────
|
|
|
|
Scenario: Execution throughput thresholds section exists
|
|
When I scaling perf load the baseline thresholds fixture
|
|
Then the scaling perf thresholds should have execution throughput section
|
|
|
|
Scenario: Execution throughput covers all plan counts
|
|
When I scaling perf load the baseline thresholds fixture
|
|
Then the scaling perf execution thresholds should cover "10_plans"
|
|
And the scaling perf execution thresholds should cover "50_plans"
|
|
And the scaling perf execution thresholds should cover "100_plans"
|
|
|
|
# ──────────────────────────────────────────────────
|
|
# Section 5: Memory thresholds at production scale
|
|
# ──────────────────────────────────────────────────
|
|
|
|
Scenario: Memory thresholds cover 50K and 100K files
|
|
When I scaling perf load the baseline thresholds fixture
|
|
Then the scaling perf memory thresholds should cover "50000_files"
|
|
And the scaling perf memory thresholds should cover "100000_files"
|
|
|
|
Scenario: Memory peak exceeds steady state for extended profiles
|
|
When I scaling perf load the baseline thresholds fixture
|
|
Then scaling perf memory peak should exceed steady state for "50000_files"
|
|
And scaling perf memory peak should exceed steady state for "100000_files"
|