Odeinjul/bigann-100m-static-search-eval

bigann-100m-static-search-eval

Static 100M search-evaluation dataset with original vectors, original query/ground-truth files, a rebuilt hnswlib HNSW index, and generated PQ artifacts.

Files

• —Base: base.u8bin
• —HNSW index: index_m_32_ef_500
• —Query: orig_query_10k.u8bin
• —Ground truth: groundtruth.bin
• —Checksums: checksums.sha256
• —Kaggle metadata is provided via dataset-metadata.json during upload and is stored by Kaggle as dataset configuration, not as a listed data file.

Data Sources

• —base.u8bin -> base.u8bin
• —unif_query_10k.u8bin -> orig_query_10k.u8bin
• —unif_groundtruth_10k.bin -> groundtruth.bin

Source license/terms note: Original source terms apply; Kaggle metadata uses license other.

Dataset Properties

• —Vector count: 100000000
• —Dimension: 128
• —Dtype: uint8
• —Metric: l2
• —Query count label: 10k
• —Query count: 10000

Environment

• —CPU threads used: 96
• —Python: Python 3.10.12
• —hnswlib package version: unknown
• —hnswlib git commit: fe873b1cee484795bd254088c0d4579f567f1979
• —faiss: 1.14.3
• —numpy: 2.2.6
• —kaggle: unknown
• —kagglehub: 1.0.2
• —vectordb-cxl commit: 73e82318a4c346b64b8c645d030ac86696351632

HNSW Index

• —Parameters: M=32, ef_construction=500
• —Index dtype: uint8
• —hnswlib space: l2
• —Thread count: 96
• —Build start: 2026-07-04 23:38:36 UTC
• —Build end: 2026-07-05 01:29:06 UTC
• —Build elapsed seconds: 6630.319
• —Built from original base vectors: true
• —Prebuilt Kaggle index reused: false

Primary Search

• —Parameters: ef_search=150, k=10
• —Thread count: 96
• —Recall@10: 0.984250
• —Batch throughput qps: 5487.035622
• —Latency mean_ms: 17.166175
• —Latency p50_ms: 1.833859
• —Latency p95_ms: 2.784008
• —Latency p99_ms: 22.797722
• —Instrumented latency from batch run: True
• —Search elapsed seconds: 1.822478

Optional ef_search Sweep

Not run unless a search_sweep.json file is included and referenced in the status log.

PQ

Implementation path: cpp

Fallback used: False

Thread count: 96

Elapsed seconds: 1502.026

Final m values: m8, m16, m32

Kaggle upload file names: pq_m8.pqcodes/.pqmeta/.pqcodebook, pq_m16.pqcodes/.pqmeta/.pqcodebook, pq_m32.pqcodes/.pqmeta/.pqcodebook

PQ row order

All pq_m*.pqcodes files are ordered by HNSW internal ID. For internal row i, PQ row i encodes the base vector whose external label is stored in HNSW row i.

This is required because a parallel hnswlib build does not guarantee that an internal ID equals its external/base-row ID. The original PQ export was in base-row order, which assigns the wrong code wherever those IDs differ and can reduce graph-routing recall even when final candidates are reranked with full vectors. The PQ payloads were therefore reordered against index_m_32_ef_500; codebooks and metadata were not retrained or changed.

pq_manifest.json records the row-order contract. The standalone tools/reorder_pq_codes_for_index.py script reproduces the transformation:

python tools/reorder_pq_codes_for_index.py \
  --source-pq-codes-path pq_m32.base-row.pqcodes \
  --source-code-order base_row \
  --target-id-mapping-path layout-sidecar/index_m_32_ef_500.id_mapping.u32 \
  --output-pq-codes-path pq_m32.pqcodes \
  --chunk-rows 250000 \
  --preload-source \
  --workers "$(nproc)"

Attribution

The original BIGANN (SIFT) vectors are credited to the original BIGANN dataset contributors and were obtained from the Big ANN 2021 benchmark collection.

The original source data is subject to CC0 1.0 Universal; the original terms are reproduced in LICENSE.

We generated the indexes ourselves.