123 lines
6.8 KiB
Gherkin
123 lines
6.8 KiB
Gherkin
Feature: Scale test fixtures and performance baselines
|
|
As a performance engineer
|
|
I want to validate scale test fixtures and threshold matrices
|
|
So that performance baselines are well-formed and realistic
|
|
|
|
Background:
|
|
Given a scale test environment is ready
|
|
|
|
# ──────────────────────────────────────────────────
|
|
# Section 1: Fixture loading and metadata validation
|
|
# ──────────────────────────────────────────────────
|
|
|
|
Scenario: Load scale metadata fixture from disk
|
|
When I scale test load the scale metadata fixture
|
|
Then the scale test metadata should contain a schema version
|
|
And the scale test metadata should contain scale profiles
|
|
|
|
Scenario: Scale metadata contains all required profiles
|
|
When I scale test load the scale metadata fixture
|
|
Then the scale test metadata should have exactly 3 profiles
|
|
And the scale test profile names should be "small, medium, large"
|
|
|
|
Scenario: Each scale profile has required fields
|
|
When I scale test load the scale metadata fixture
|
|
Then each scale test profile should have a file count
|
|
And each scale test profile should have a total size in megabytes
|
|
And each scale test profile should have a language mix
|
|
And each scale test profile should have expected index time
|
|
And each scale test profile should have expected decomposition time
|
|
|
|
Scenario: Scale metadata lists supported languages
|
|
When I scale test load the scale metadata fixture
|
|
Then the scale test metadata should list supported languages
|
|
And the scale test supported languages should include "python"
|
|
And the scale test supported languages should include "javascript"
|
|
|
|
# ──────────────────────────────────────────────────
|
|
# Section 2: Threshold matrix validation
|
|
# ──────────────────────────────────────────────────
|
|
|
|
Scenario: Load baseline thresholds fixture from disk
|
|
When I scale test load the baseline thresholds fixture
|
|
Then the scale test thresholds should contain a schema version
|
|
And the scale test thresholds should have indexing section
|
|
And the scale test thresholds should have decomposition section
|
|
|
|
Scenario: Indexing thresholds cover all file counts
|
|
When I scale test load the baseline thresholds fixture
|
|
Then the scale test indexing thresholds should cover "1000_files"
|
|
And the scale test indexing thresholds should cover "5000_files"
|
|
And the scale test indexing thresholds should cover "10000_files"
|
|
|
|
Scenario: Each indexing threshold has percentile fields
|
|
When I scale test load the baseline thresholds fixture
|
|
Then each scale test indexing threshold should have p50 p95 and p99
|
|
|
|
Scenario: Decomposition thresholds cover all file counts
|
|
When I scale test load the baseline thresholds fixture
|
|
Then the scale test decomposition thresholds should cover "1000_files"
|
|
And the scale test decomposition thresholds should cover "5000_files"
|
|
And the scale test decomposition thresholds should cover "10000_files"
|
|
|
|
Scenario: Each decomposition threshold has percentile fields
|
|
When I scale test load the baseline thresholds fixture
|
|
Then each scale test decomposition threshold should have p50 p95 and p99
|
|
|
|
Scenario: Thresholds are monotonically increasing across percentiles
|
|
When I scale test load the baseline thresholds fixture
|
|
Then scale test indexing p50 should be less than p95 for all file counts
|
|
And scale test indexing p95 should be less than p99 for all file counts
|
|
|
|
# ──────────────────────────────────────────────────
|
|
# Section 3: Language mix distribution validation
|
|
# ──────────────────────────────────────────────────
|
|
|
|
Scenario: Language mix sums to 1.0 for each profile
|
|
When I scale test load the scale metadata fixture
|
|
Then the scale test language mix should sum to 1.0 for each profile
|
|
|
|
Scenario: Language mix values are between 0 and 1
|
|
When I scale test load the scale metadata fixture
|
|
Then each scale test language mix value should be between 0 and 1
|
|
|
|
Scenario: All languages in mix are in supported languages list
|
|
When I scale test load the scale metadata fixture
|
|
Then all scale test language mix keys should be in supported languages
|
|
|
|
# ──────────────────────────────────────────────────
|
|
# Section 4: Scale profile simulation
|
|
# ──────────────────────────────────────────────────
|
|
|
|
Scenario: Generate file count distribution for small profile
|
|
When I scale test generate file distribution for the "small" profile
|
|
Then the scale test total generated file count should be 1000
|
|
And the scale test generated distribution should match the language mix
|
|
|
|
Scenario: Generate file count distribution for medium profile
|
|
When I scale test generate file distribution for the "medium" profile
|
|
Then the scale test total generated file count should be 5000
|
|
And the scale test generated distribution should match the language mix
|
|
|
|
Scenario: Generate file count distribution for large profile
|
|
When I scale test generate file distribution for the "large" profile
|
|
Then the scale test total generated file count should be 10000
|
|
And the scale test generated distribution should match the language mix
|
|
|
|
# ──────────────────────────────────────────────────
|
|
# Section 5: Baseline threshold realism checks
|
|
# ──────────────────────────────────────────────────
|
|
|
|
Scenario: Indexing thresholds scale sub-linearly with file count
|
|
When I scale test load the baseline thresholds fixture
|
|
Then scale test 10K indexing p50 should be less than 10x the 1K p50
|
|
|
|
Scenario: Decomposition thresholds exceed indexing thresholds
|
|
When I scale test load the baseline thresholds fixture
|
|
Then scale test decomposition p50 should be greater than indexing p50 for each file count
|
|
|
|
Scenario: Memory thresholds are present and realistic
|
|
When I scale test load the baseline thresholds fixture
|
|
Then the scale test thresholds should have memory usage section
|
|
And scale test memory peak should exceed steady state for each file count
|