169 lines
5.4 KiB
Markdown
169 lines
5.4 KiB
Markdown
# Scale Testing Runbook
|
|
|
|
## Overview
|
|
|
|
Scale testing validates that CleverAgents performs within acceptable bounds when
|
|
indexing and decomposing repositories of varying sizes (1K, 5K, and 10K files).
|
|
The test suite uses pre-defined fixtures rather than live repositories to ensure
|
|
repeatable, deterministic results.
|
|
|
|
### Approach
|
|
|
|
1. **Fixture-based**: Scale profiles are defined in JSON metadata files that
|
|
describe repository characteristics (file count, size, language mix) without
|
|
requiring actual large repositories.
|
|
2. **Threshold matrices**: Baseline performance thresholds are maintained in a
|
|
separate JSON file with p50/p95/p99 percentile targets for indexing and
|
|
decomposition operations.
|
|
3. **Simulation**: File distribution generation simulates how files would be
|
|
allocated across languages for each profile, validating the distribution
|
|
logic without creating actual files.
|
|
|
|
## Environment Requirements
|
|
|
|
- Python 3.13+
|
|
- At least 8 GB RAM (for large profile validation)
|
|
- Local SSD storage recommended
|
|
- All dependencies installed via `uv sync`
|
|
|
|
## Fixture Files
|
|
|
|
All scale fixtures live in `features/fixtures/scale/`:
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `scale_metadata.json` | Scale profile definitions (file count, size, language mix) |
|
|
| `baseline_thresholds.json` | Performance threshold matrices (p50/p95/p99 per operation) |
|
|
| `generator_instructions.md` | Manual instructions for generating synthetic test repos |
|
|
|
|
## Running Scale Tests
|
|
|
|
### Behave (Unit Tests)
|
|
|
|
```bash
|
|
# Run scale test scenarios only
|
|
nox -s unit_tests -- features/scale_test.feature
|
|
|
|
# Run all unit tests including scale tests
|
|
nox -s unit_tests
|
|
```
|
|
|
|
### Robot Framework (Integration Tests)
|
|
|
|
```bash
|
|
# Run scale test integration suite only
|
|
nox -s integration_tests -- --suite robot/scale_test.robot
|
|
|
|
# Run all integration tests including scale tests
|
|
nox -s integration_tests
|
|
```
|
|
|
|
### ASV Benchmarks
|
|
|
|
```bash
|
|
# Run all benchmarks including scale fixture benchmarks
|
|
nox -s benchmark
|
|
```
|
|
|
|
## Interpreting Results
|
|
|
|
### Behave Results
|
|
|
|
All 20 scenarios should pass. Failures indicate:
|
|
|
|
- **Fixture loading failures**: Missing or malformed JSON files
|
|
- **Threshold validation failures**: Percentile ordering violations (p50 >= p95)
|
|
- **Language mix failures**: Distribution ratios don't sum to 1.0
|
|
- **Distribution generation failures**: File count allocation doesn't match profile
|
|
|
|
### Robot Results
|
|
|
|
All 6 test cases should pass. Each test prints a sentinel string on success:
|
|
|
|
| Sentinel | Meaning |
|
|
|----------|---------|
|
|
| `scale-metadata-ok` | Metadata fixture loaded and has valid structure |
|
|
| `scale-profiles-ok` | All profiles have required fields and valid language mixes |
|
|
| `scale-thresholds-ok` | Threshold matrix has all required sections and percentiles |
|
|
| `scale-monotonicity-ok` | All thresholds satisfy p50 < p95 < p99 |
|
|
| `scale-distribution-ok` | File distributions match language mix ratios |
|
|
| `scale-generator-docs-ok` | Generator instructions document is present |
|
|
|
|
### ASV Benchmarks
|
|
|
|
The `ScaleFixtureSuite` measures:
|
|
|
|
| Benchmark | Description | Expected Range |
|
|
|-----------|-------------|---------------|
|
|
| `time_metadata_loading` | JSON parse time for metadata | < 1 ms |
|
|
| `time_threshold_loading` | JSON parse time for thresholds | < 1 ms |
|
|
| `time_threshold_validation` | Threshold matrix validation | < 0.1 ms |
|
|
| `time_language_mix_validation` | Language mix sum checks | < 0.1 ms |
|
|
| `time_file_distribution_generation` | Distribution calculation | < 0.5 ms |
|
|
| `time_full_fixture_roundtrip` | Complete load + validate + distribute | < 2 ms |
|
|
|
|
## Threshold Adjustment Guidelines
|
|
|
|
### When to Adjust Thresholds
|
|
|
|
- After significant indexing algorithm changes
|
|
- When targeting new hardware profiles (e.g., CI runners with different specs)
|
|
- When adding support for new file types that affect parsing performance
|
|
- After profiling reveals consistent threshold violations
|
|
|
|
### How to Adjust
|
|
|
|
1. Run benchmarks on the target environment:
|
|
```bash
|
|
nox -s benchmark
|
|
```
|
|
|
|
2. Collect actual p50/p95/p99 values from multiple runs (minimum 10 runs recommended).
|
|
|
|
3. Update `features/fixtures/scale/baseline_thresholds.json` with new values.
|
|
|
|
4. Ensure the monotonicity invariant holds: `p50 < p95 < p99` for all entries.
|
|
|
|
5. Update the `scaling_expectations` section if the algorithmic complexity has changed.
|
|
|
|
6. Run the full test suite to verify:
|
|
```bash
|
|
nox -s unit_tests -- features/scale_test.feature
|
|
nox -s integration_tests -- --suite robot/scale_test.robot
|
|
```
|
|
|
|
### Threshold Safety Margins
|
|
|
|
Current thresholds include a 2x safety margin over measured values to account for:
|
|
|
|
- CI runner variability
|
|
- Concurrent workload interference
|
|
- OS-level caching effects
|
|
- GC pauses in the Python runtime
|
|
|
|
When adjusting, maintain at least a 1.5x margin over observed p99 values.
|
|
|
|
## Adding New Scale Profiles
|
|
|
|
To add a new scale profile (e.g., "extra-large" for 50K files):
|
|
|
|
1. Add the profile entry to `scale_metadata.json`:
|
|
```json
|
|
{
|
|
"name": "extra-large",
|
|
"file_count": 50000,
|
|
"total_size_mb": 2500,
|
|
"language_mix": { ... },
|
|
"expected_index_time_s": 250.0,
|
|
"expected_decomposition_time_s": 500.0,
|
|
"description": "Extra-large enterprise monorepo"
|
|
}
|
|
```
|
|
|
|
2. Add threshold entries to `baseline_thresholds.json` for `"50000_files"`.
|
|
|
|
3. Update Behave scenarios in `features/scale_test.feature` if exact profile
|
|
count assertions need updating.
|
|
|
|
4. Run tests to verify the new profile integrates correctly.
|