Odeinjul/deep-100m-static-search-eval

deep-100m-static-search-eval

Static 100M search-evaluation dataset with original vectors, original query/ground-truth files, a rebuilt hnswlib HNSW index, and generated PQ artifacts.

Files

• —Base: base.fbin
• —HNSW index: index_m_32_ef_500
• —Query: orig_query_10k.fbin
• —Ground truth: groundtruth.bin
• —Additional held-out 500K query/GT set: queries/heldout_deep1b_random500k_seed20260922/
• —Checksums: checksums.sha256
• —Kaggle metadata is provided via dataset-metadata.json during upload and is stored by Kaggle as dataset configuration, not as a listed data file.

Data Sources

• —base.fbin -> base.fbin
• —unif_query_10k.fbin -> orig_query_10k.fbin
• —GT_100M/deep-100M -> groundtruth.bin from https://dl.fbaipublicfiles.com/billion-scale-ann-benchmarks/GT_100M/deep-100M

Source license/terms note: Original source terms apply; Kaggle metadata uses license other.

Held-out random 500K query set

queries/heldout_deep1b_random500k_seed20260922/ contains a separate query set sampled from real DEEP1B vectors, not synthetic coordinates. NumPy PCG64 with seed 20260922 selected 500,000 distinct source row IDs uniformly without replacement from DEEP1B rows [100000000, 1000000000). Their random draw order is the query order. None of the selected source row IDs is in the first 100M rows used as the indexed base.

• —deep100m_heldout_deep1b_random500k_seed20260922.fbin: little-endian uint32 (500000, 96) header followed by row-major float32 vectors.
• —deep100m_heldout_deep1b_random500k_seed20260922_top100.bin: little-endian uint32 (500000, 100) header, then all uint32 base-row neighbor IDs, then all float32 squared-L2 distances. Each query has its exact top 100 neighbors among the 100M base vectors.
• —source_ids.npy maps each query row to its DEEP1B source row. sample.json, query.json, groundtruth.json, spec.json, and manifest.json record sampling, file checksums, pinned source, computation, and validation. logs/ contains the eight workers' progress bars and the independent source-query and exact-GT verification traces.

Generation and verification use `generate_random_fbin_groundtruth.py`, also committed locally on the vectordb-cxl branch feat/random-fbin-exact-gt at a1912a8d105ff55c0a4a78d448cabf68d2307d92. Computation uses eight nodes, 32 search threads per node, and batches of 512 queries. The original 10K query and GT files remain separate and unchanged.

Dataset Properties

• —Vector count: 100000000
• —Dimension: 96
• —Dtype: float32
• —Metric: l2
• —Original query set label: 10k
• —Original query count: 10000

Environment

• —CPU threads used: 96
• —Python: Python 3.10.12
• —hnswlib package version: unknown
• —hnswlib git commit: fe873b1cee484795bd254088c0d4579f567f1979
• —faiss: 1.14.3
• —numpy: 2.2.6
• —kaggle: unknown
• —kagglehub: 1.0.2
• —vectordb-cxl commit: 73e82318a4c346b64b8c645d030ac86696351632

HNSW Index

• —Parameters: M=32, ef_construction=500
• —Index dtype: float32
• —hnswlib space: l2
• —Thread count: 96
• —Build start: 2026-07-05 15:07:30 UTC
• —Build end: 2026-07-05 17:39:00 UTC
• —Build elapsed seconds: 9090.091
• —Built from original base vectors: true
• —Prebuilt Kaggle index reused: false

Primary Search

• —Parameters: ef_search=150, k=10
• —Thread count: 96
• —Recall@10: 0.975170
• —Batch throughput qps: 8784.062140
• —Latency mean_ms: 10.821620
• —Latency p50_ms: 2.060002
• —Latency p95_ms: 18.078470
• —Latency p99_ms: 38.452937
• —Instrumented latency from batch run: True
• —Search elapsed seconds: 1.138425

Optional ef_search Sweep

Not run unless a search_sweep.json file is included and referenced in the status log.

PQ

Implementation path: cpp

Fallback used: False

Thread count: 96

Elapsed seconds: 2811.184

Final m values: m24, m48, m96

Kaggle upload file names: pq_m24.pqcodes/.pqmeta/.pqcodebook, pq_m48.pqcodes/.pqmeta/.pqcodebook, pq_m96.pqcodes/.pqmeta/.pqcodebook

PQ row order

All pq_m*.pqcodes files are ordered by HNSW internal ID. For internal row i, PQ row i encodes the base vector whose external label is stored in HNSW row i.

Exhaustive validation confirmed that the M24, M48, and M96 PQ files use this hnsw_internal_id row order for all 100,000,000 rows.

This is required because a parallel hnswlib build does not guarantee that an internal ID equals its external/base-row ID. The original PQ export was in base-row order, which assigns the wrong code wherever those IDs differ and can reduce graph-routing recall even when final candidates are reranked with full vectors. The PQ payloads were therefore reordered against index_m_32_ef_500; codebooks and metadata were not retrained or changed.

pq_manifest.json records the row-order contract. The standalone tools/reorder_pq_codes_for_index.py script reproduces the transformation:

python tools/reorder_pq_codes_for_index.py \
  --source-pq-codes-path pq_m48.base-row.pqcodes \
  --source-code-order base_row \
  --target-id-mapping-path layout-sidecar/index_m_32_ef_500.id_mapping.u32 \
  --output-pq-codes-path pq_m48.pqcodes \
  --chunk-rows 250000 \
  --preload-source \
  --workers "$(nproc)"

Consumption and validation

Treat code_order as a contract to verify against the index, not as evidence by itself. Compare the sidecar mapping to the actual labels in the serialized index. When deriving PQ for a different index, map its internal row to its external label and then through the inverse source-index mapping to the source PQ row. Index-internal IDs, external labels, and base_permuted insert-vector row offsets are separate coordinates. Merely copying the manifest or matching file sizes does not establish this relation.

pq_manifest.json declares the L2 metric stored in every PQ binary header. checksums.sha256 also covers all three layout-sidecar files. Preparation and restore workflows should verify headers, hashes, actual index/sidecar labels, and deterministic vector re-encodings including nonidentity mappings before use.

Attribution

The original DEEP-1B vectors are credited to Yandex and were obtained from the Big ANN 2021 benchmark collection.

The original source data is subject to Creative Commons Attribution 4.0 International (CC BY 4.0); the original terms are reproduced in LICENSE.

We generated the indexes ourselves.