Datasetpublic

Nanvivi/SIFT1B-DiskANN

SIFT-1B Dataset & Disk Index The SIFT-1B (BigANN) dataset and pre-built disk-based ANN index. Built February 2026 on Intel Xeon 8462Y+ (Sapphire Rapids) with 800GB RAM. Build Parameters Parameter Value Dataset SIFT-1B (1,000,000,000 vectors, 128-dim, uint8) Graph R 128 (max degree) Build L 200 (search list size during construction) PQ chunks 32 (4 dimensions per sub-quantizer) Build time ~2 days Files Raw Data… See the full description on the dataset page: https://huggingface.co/datasets/Nanvivi/SIFT1B-DiskANN.

sourceHugging Faceupdated 7mo agoView on Hugging Face

0likes2.9kdownloads

Dataset Card

SIFT-1B Dataset & Disk Index

The SIFT-1B (BigANN) dataset and pre-built disk-based ANN index. Built February 2026 on Intel Xeon 8462Y+ (Sapphire Rapids) with 800GB RAM.

Build Parameters

Parameter	Value
Dataset	SIFT-1B (1,000,000,000 vectors, 128-dim, uint8)
Graph R	128 (max degree)
Build L	200 (search list size during construction)
PQ chunks	32 (4 dimensions per sub-quantizer)
Build time	~2 days

Files

Raw Data

File	Size	Description
`base.bin`	120GB	Raw SIFT-1B vectors. Binary format: `[uint32 num_points][uint32 dim][uint8 vectors...]`. 1B x 128-dim x uint8.
`query.bin`	1.3MB	10,000 query vectors in the same format. Standard BigANN query set.
`gt.bin`	39MB	Ground truth (100 nearest neighbors per query). Binary format: `[uint32 num_queries][uint32 K][uint32 ids...]`.

Disk Index

File	Size	Description
`disk_diskidx_parts/`	636GB total	Sector-aligned Vamana graph, split into 250GB chunks for upload. Each 4096-byte sector contains one or more nodes. Each node stores: `[float32 coords[128]][uint32 num_neighbors][uint32 neighbor_ids[R]]`. Primary structure for graph-based disk ANN search. Reassemble with: `cat disk_diskidx_parts/disk.diskidx.*.part > disk.diskidx`
`disk.pq`	129KB	PQ codebook (trained centroids). 32 sub-quantizers, each with 256 centroids of 4 float32 values. Used to compute approximate distances without reading full vectors from disk.
`disk.pqcodes`	30GB	PQ-compressed representation of all 1B vectors. Each vector encoded as 32 bytes (one centroid ID per sub-quantizer). Loaded into memory for fast approximate distance computation during search.

In-Memory Sample Index

File	Size	Description
`mem`	3.2GB	In-memory Vamana graph built on a 1% sample (~10M points). Provides fast entry point discovery — search this small graph first, then jump into the full disk graph at promising locations.
`mem.data`	1.2GB	Raw vectors for the 1% sample points. Kept in memory for exact distance computation during sample graph traversal.
`mem.tags`	39MB	Maps sample graph node IDs to their corresponding IDs in the full 1B dataset, so sample graph results can be used as entry points into the disk graph.