# Scale Testing Runbook

## Overview

Scale testing validates that CleverAgents performs within acceptable bounds when
indexing and decomposing repositories of varying sizes (1K, 5K, and 10K files).
The test suite uses pre-defined fixtures rather than live repositories to ensure
repeatable, deterministic results.

### Approach

1. **Fixture-based**: Scale profiles are defined in JSON metadata files that
   describe repository characteristics (file count, size, language mix) without
   requiring actual large repositories.
2. **Threshold matrices**: Baseline performance thresholds are maintained in a
   separate JSON file with p50/p95/p99 percentile targets for indexing and
   decomposition operations.
3. **Simulation**: File distribution generation simulates how files would be
   allocated across languages for each profile, validating the distribution
   logic without creating actual files.

## Environment Requirements

- Python 3.13+
- At least 8 GB RAM (for large profile validation)
- Local SSD storage recommended
- All dependencies installed via `uv sync`

## Fixture Files

All scale fixtures live in `features/fixtures/scale/`:

| File | Purpose |
|------|---------|
| `scale_metadata.json` | Scale profile definitions (file count, size, language mix) |
| `baseline_thresholds.json` | Performance threshold matrices (p50/p95/p99 per operation) |
| `generator_instructions.md` | Manual instructions for generating synthetic test repos |

## Running Scale Tests

### Behave (Unit Tests)

```bash
# Run scale test scenarios only
nox -s unit_tests -- features/scale_test.feature

# Run all unit tests including scale tests
nox -s unit_tests
```

### Robot Framework (Integration Tests)

```bash
# Run scale test integration suite only
nox -s integration_tests -- --suite robot/scale_test.robot

# Run all integration tests including scale tests
nox -s integration_tests
```

### ASV Benchmarks

```bash
# Run all benchmarks including scale fixture benchmarks
nox -s benchmark
```

## Interpreting Results

### Behave Results

All 20 scenarios should pass. Failures indicate:

- **Fixture loading failures**: Missing or malformed JSON files
- **Threshold validation failures**: Percentile ordering violations (p50 >= p95)
- **Language mix failures**: Distribution ratios don't sum to 1.0
- **Distribution generation failures**: File count allocation doesn't match profile

### Robot Results

All 6 test cases should pass. Each test prints a sentinel string on success:

| Sentinel | Meaning |
|----------|---------|
| `scale-metadata-ok` | Metadata fixture loaded and has valid structure |
| `scale-profiles-ok` | All profiles have required fields and valid language mixes |
| `scale-thresholds-ok` | Threshold matrix has all required sections and percentiles |
| `scale-monotonicity-ok` | All thresholds satisfy p50 < p95 < p99 |
| `scale-distribution-ok` | File distributions match language mix ratios |
| `scale-generator-docs-ok` | Generator instructions document is present |

### ASV Benchmarks

The `ScaleFixtureSuite` measures:

| Benchmark | Description | Expected Range |
|-----------|-------------|---------------|
| `time_metadata_loading` | JSON parse time for metadata | < 1 ms |
| `time_threshold_loading` | JSON parse time for thresholds | < 1 ms |
| `time_threshold_validation` | Threshold matrix validation | < 0.1 ms |
| `time_language_mix_validation` | Language mix sum checks | < 0.1 ms |
| `time_file_distribution_generation` | Distribution calculation | < 0.5 ms |
| `time_full_fixture_roundtrip` | Complete load + validate + distribute | < 2 ms |

## Threshold Adjustment Guidelines

### When to Adjust Thresholds

- After significant indexing algorithm changes
- When targeting new hardware profiles (e.g., CI runners with different specs)
- When adding support for new file types that affect parsing performance
- After profiling reveals consistent threshold violations

### How to Adjust

1. Run benchmarks on the target environment:
   ```bash
   nox -s benchmark
   ```

2. Collect actual p50/p95/p99 values from multiple runs (minimum 10 runs recommended).

3. Update `features/fixtures/scale/baseline_thresholds.json` with new values.

4. Ensure the monotonicity invariant holds: `p50 < p95 < p99` for all entries.

5. Update the `scaling_expectations` section if the algorithmic complexity has changed.

6. Run the full test suite to verify:
   ```bash
   nox -s unit_tests -- features/scale_test.feature
   nox -s integration_tests -- --suite robot/scale_test.robot
   ```

### Threshold Safety Margins

Current thresholds include a 2x safety margin over measured values to account for:

- CI runner variability
- Concurrent workload interference
- OS-level caching effects
- GC pauses in the Python runtime

When adjusting, maintain at least a 1.5x margin over observed p99 values.

## Adding New Scale Profiles

To add a new scale profile (e.g., "extra-large" for 50K files):

1. Add the profile entry to `scale_metadata.json`:
   ```json
   {
     "name": "extra-large",
     "file_count": 50000,
     "total_size_mb": 2500,
     "language_mix": { ... },
     "expected_index_time_s": 250.0,
     "expected_decomposition_time_s": 500.0,
     "description": "Extra-large enterprise monorepo"
   }
   ```

2. Add threshold entries to `baseline_thresholds.json` for `"50000_files"`.

3. Update Behave scenarios in `features/scale_test.feature` if exact profile
   count assertions need updating.

4. Run tests to verify the new profile integrates correctly.