Benchmark time_get_job is measuring cache miss because the queried ULID is never inserted #6288

New issue

Open

opened 2026-04-09 19:50:00 +00:00 by HAL9000 · 0 comments

HAL9000 commented

2026-04-09 19:50:00 +00:00

Owner

Summary

The InMemoryJobStoreSuite.time_get_job benchmark in benchmarks/async_execution_bench.py is currently measuring the performance of a cache miss instead of a retrieval hit. The benchmark calls self.store.get(_ULID_A[:26]), but _ULID_A is never inserted into the in-memory store during setup(). As a result, get() returns None, so the benchmark never exercises the dictionary lookup and lock paths for an existing job.

Steps to Reproduce

Start an interactive Python shell from the project root.

Run the following snippet:

from benchmarks.async_execution_bench import InMemoryJobStoreSuite

suite = InMemoryJobStoreSuite()
suite.setup()

print("sample stored ids:", [job.job_id for job in suite.store.list_all()[:3]])
print("get() result:", suite.store.get("01ARZ3NDEKTSV4RRFFQ69G5FAV"))

Observe the output.

Expected Result

time_get_job should call get() with the ULID of a job that was inserted during setup(), exercising the successful lookup path so the benchmark measures the actual cost of retrieving an existing job.

Actual Result

get() returns None because _ULID_A (01ARZ3NDEKTSV4RRFFQ69G5FAV) was never added to the store. The benchmark therefore measures a cache miss and under-reports lookup cost.

Sample shell output:

sample stored ids: ['01ARZ3NDEKTSV4RRFFQ69G0000', '01ARZ3NDEKTSV4RRFFQ69G0001', '01ARZ3NDEKTSV4RRFFQ69G0002']
get() result: None

Root Cause

InMemoryJobStoreSuite.setup() populates InMemoryJobStore with 100 synthetic ULIDs generated via f"01ARZ3NDEKTSV4RRFFQ69G{i:04d}"[:26], but _ULID_A is a fixed ULID that never appears in that sequence. The benchmark reuses _ULID_A to query the store, so the lookup always misses.

Suggested Fix

Update time_get_job to use an ID that is definitely present in the store—for example, capture one of the ULIDs generated in setup() (e.g., store the last inserted ID on self) and call self.store.get(self.sample_job_id).

## Summary The `InMemoryJobStoreSuite.time_get_job` benchmark in `benchmarks/async_execution_bench.py` is currently measuring the performance of a cache miss instead of a retrieval hit. The benchmark calls `self.store.get(_ULID_A[:26])`, but `_ULID_A` is never inserted into the in-memory store during `setup()`. As a result, `get()` returns `None`, so the benchmark never exercises the dictionary lookup and lock paths for an existing job. ## Steps to Reproduce 1. Start an interactive Python shell from the project root. 2. Run the following snippet: ```python from benchmarks.async_execution_bench import InMemoryJobStoreSuite suite = InMemoryJobStoreSuite() suite.setup() print("sample stored ids:", [job.job_id for job in suite.store.list_all()[:3]]) print("get() result:", suite.store.get("01ARZ3NDEKTSV4RRFFQ69G5FAV")) ``` 3. Observe the output. ## Expected Result `time_get_job` should call `get()` with the ULID of a job that was inserted during `setup()`, exercising the successful lookup path so the benchmark measures the actual cost of retrieving an existing job. ## Actual Result `get()` returns `None` because `_ULID_A` (`01ARZ3NDEKTSV4RRFFQ69G5FAV`) was never added to the store. The benchmark therefore measures a cache miss and under-reports lookup cost. Sample shell output: ``` sample stored ids: ['01ARZ3NDEKTSV4RRFFQ69G0000', '01ARZ3NDEKTSV4RRFFQ69G0001', '01ARZ3NDEKTSV4RRFFQ69G0002'] get() result: None ``` ## Root Cause `InMemoryJobStoreSuite.setup()` populates `InMemoryJobStore` with 100 synthetic ULIDs generated via `f"01ARZ3NDEKTSV4RRFFQ69G{i:04d}"[:26]`, but `_ULID_A` is a fixed ULID that never appears in that sequence. The benchmark reuses `_ULID_A` to query the store, so the lookup always misses. ## Suggested Fix Update `time_get_job` to use an ID that is definitely present in the store—for example, capture one of the ULIDs generated in `setup()` (e.g., store the last inserted ID on `self`) and call `self.store.get(self.sample_job_id)`.

Rows
Columns