brent.edwards
b88bc0ec1b
CI / build (push) Successful in 19s
CI / lint (push) Successful in 3m19s
CI / quality (push) Successful in 3m43s
CI / typecheck (push) Successful in 3m55s
CI / security (push) Successful in 4m2s
CI / unit_tests (push) Successful in 6m39s
CI / integration_tests (push) Successful in 6m48s
CI / docker (push) Successful in 1m7s
CI / e2e_tests (push) Successful in 9m7s
CI / benchmark-regression (push) Has been skipped
CI / coverage (push) Failing after 16m36s
CI / benchmark-publish (push) Successful in 25m51s
CI / status-check (push) Failing after 4s
## Summary Add large project scaling benchmarks and tests at production scale (10K–100K files). ### New ASV Benchmarks **IndexingScalingSuite** (`large_project_scaling_bench.py`): - `time_walk_and_index` at 1K/10K/50K/100K files - `time_incremental_refresh` (1% modified files) - `track_indexed_file_count`, `track_tokens_per_second` **ContextAssemblyScalingSuite** (`context_assembly_scaling_bench.py`): - `time_full_pipeline` at 100/1K/5K/10K fragments - `time_tiered_strategy`, `time_recency_strategy` - `track_assembled_tokens`, `track_fragments_per_second` **ExecutionThroughputSuite** (`execution_throughput_bench.py`): - `time_sequential_plans` at 10/50/100 plans - `time_executor_construction`, `time_decision_tree_scaling` ### Scale Fixture Updates - Added `xlarge` (50K files) and `xxlarge` (100K files) profiles to `scale_metadata.json` - Added 50K/100K thresholds to `baseline_thresholds.json` - Added `context_assembly` and `execution_throughput` threshold sections ### Tests & Documentation - 15 Behave scenarios validating profiles, thresholds, monotonicity, memory budgets - 6 Robot integration tests including live 1K-file indexing throughput check - `docs/reference/scaling_baselines.md` documenting all baseline metrics ### Quality Gates | Session | Result | |---|---| | `nox -s lint` | PASS | | `nox -s typecheck` | PASS (0 errors) | | `nox -s unit_tests` | PASS (10,910 scenarios) | | `nox -s integration_tests` | PASS (1,526 tests) | | `nox -s coverage_report` | 97% (>= 97%) | Closes #859 Reviewed-on: #984 Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com> Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
61 lines
3.0 KiB
Plaintext
61 lines
3.0 KiB
Plaintext
*** Settings ***
|
|
Documentation Large-project scaling performance integration tests.
|
|
... Validates that benchmark fixtures exist, thresholds are
|
|
... reasonable, and a small-scale indexing run meets minimum
|
|
... throughput targets.
|
|
Resource ${CURDIR}/common.resource
|
|
Suite Setup Setup Test Environment
|
|
Suite Teardown Cleanup Test Environment
|
|
|
|
*** Variables ***
|
|
${HELPER} ${CURDIR}/helper_scaling_performance.py
|
|
|
|
*** Test Cases ***
|
|
Verify Extended Scale Profiles Exist
|
|
[Documentation] Verify xlarge and xxlarge profiles are present in metadata
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-extended-profiles cwd=${WORKSPACE} timeout=120s on_timeout=kill
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} scaling-profiles-ok
|
|
|
|
Verify Extended Indexing Thresholds Exist
|
|
[Documentation] Verify 50K and 100K indexing thresholds are present
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-extended-thresholds cwd=${WORKSPACE} timeout=120s on_timeout=kill
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} scaling-thresholds-ok
|
|
|
|
Verify Context Assembly Thresholds Exist
|
|
[Documentation] Verify context assembly threshold section and entries
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-context-assembly cwd=${WORKSPACE} timeout=120s on_timeout=kill
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} scaling-ctx-assembly-ok
|
|
|
|
Verify Execution Throughput Thresholds Exist
|
|
[Documentation] Verify execution throughput threshold section and entries
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-execution-throughput cwd=${WORKSPACE} timeout=120s on_timeout=kill
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} scaling-exec-throughput-ok
|
|
|
|
Verify Sub-Linear Scaling Of Extended Thresholds
|
|
[Documentation] Verify 50K and 100K thresholds scale sub-linearly
|
|
${result}= Run Process ${PYTHON} ${HELPER} check-sublinear cwd=${WORKSPACE} timeout=120s on_timeout=kill
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} scaling-sublinear-ok
|
|
|
|
Small Scale Indexing Throughput Check
|
|
[Documentation] Create a 1K-file fixture, index it, verify throughput
|
|
${result}= Run Process ${PYTHON} ${HELPER} run-small-index cwd=${WORKSPACE} timeout=180s on_timeout=kill
|
|
Log ${result.stdout}
|
|
Log ${result.stderr}
|
|
Should Be Equal As Integers ${result.rc} 0
|
|
Should Contain ${result.stdout} scaling-index-ok
|