CoolFace
Datasetpublic

brady0127/sift1m-fanns-normal

SIFT1M Dataset to Evaluate (Filtered) Approximate Nearest Neighbor Search — Normal Variant This dataset is intended to benchmark Approximate Nearest Neighbor Search (ANNS) and Filtered Approximate Nearest Neighbor Search (FANNS) algorithms. It is based on the classic SIFT1M dataset (1,000,000 base vectors and 10,000 query vectors, each 128-dimensional). Because SIFT1M ships without any structured metadata, we augment it with synthetic attributes drawn from a normal (Gaussian)… See the full description on the dataset page: https://huggingface.co/datasets/brady0127/sift1m-fanns-normal.

sourceHugging Facemitupdated 4mo agoView on Hugging Face
0likes24downloads
Dataset Card

SIFT1M Dataset to Evaluate (Filtered) Approximate Nearest Neighbor Search — Normal Variant

This dataset is intended to benchmark Approximate Nearest Neighbor Search (ANNS) and Filtered Approximate Nearest Neighbor Search (FANNS) algorithms. It is based on the classic SIFT1M dataset (1,000,000 base vectors and 10,000 query vectors, each 128-dimensional). Because SIFT1M ships without any structured metadata, we augment it with synthetic attributes drawn from a normal (Gaussian) distribution: every base vector receives one categorical label and one continuous range_value. Query attributes for three different filter types are sampled from the same normal distribution as the base attributes. The ground truth for the up to k=100 nearest neighbors was computed for unfiltered ANNS and for filtered ANNS with the three different filter types. Please note that if fewer than 100 database items match the filter, then the ground truth can contain fewer than k=100 entries. This variant is the Gaussian counterpart of sift1m-fanns-uniform: the design is identical and only the sampling distribution changes, so that the two can be compared directly to study the effect of attribute distribution (homogeneous vs. heterogeneous selectivity).

Filter Types

FilterPredicateDescription
labellabel == required_labelExact match on the categorical attribute.
rangerange_low <= range_value <= range_highContainment in a fixed-width interval.
bothlabel AND rangeConjunction of the two predicates above.

Parameters

ParameterValueMeaning
N1,000,000Number of base (database) vectors
nq10,000Number of query vectors
vec_dim128Embedding dimension (from SIFT1M)
NUM_LABELS12label ~ round(Normal(5.5, 2.0)) clipped to [0, 11]
LABELMEAN / LABELSTD5.5 / 2.0Mean and std of the label histogram
range_valueNormal(0.5, 0.15) clipped to [0, 1]Continuous base attribute
RANGEMEAN / RANGESTD0.5 / 0.15Mean and std of range_value and of query window centers
RANGE_WIDTH0.2Width of every query range window (range_high - range_low)
GT_K100Max number of ground-truth neighbors per query
RANDOM_SEED42Fixed seed; the dataset is bit-reproducible

Expected Selectivity

Under the normal distribution, the average selectivity is higher than in the uniform variant, but more importantly the per-query selectivity has high variance (heterogeneous): common labels and central range windows are far less selective than rare labels and edge windows. A single benchmark therefore spans a wide range of difficulties, which is closer to real-world skewed attributes.

FilterAvg. selectivity (approx.)Per-query spreadNote
label~14%~0.6% (label 0/11) → ~19% (label 5/6)Σ pₓ² exceeds the uniform 8.33%
range~30–40%~8% (edge window) → ~50% (central window)Windows concentrate near the dense center
both~5%<0.5% (rare label + edge window) → ~10%Highest variance; a few queries may yield < 100 GT entries

Files and Description

FileDescription
database_vectors.fvecs128-dimensional base vectors. One vector per database item.
database_attributes.jsonlJSON objects with {id, label, range_value} for each item. One JSON object per database item.
query_vectors.fvecs128-dimensional query vectors. One vector per query.
ground_truth.ivecsGround truth for unfiltered nearest neighbor search. One vector per query.
label_query_attributes.jsonlQuery attributes for label (exact match) filtering. One JSON object per query.
ground_truth_label.ivecsGround truth for label-filtered NN search. One vector per query.
range_query_attributes.jsonlQuery attributes for range filtering. One JSON object per query.
ground_truth_range.ivecsGround truth for range-filtered NN search. One vector per query.
label_and_range_query_attributes.jsonlQuery attributes for the joint (label AND range) filter. One JSON object per query.
ground_truth_label_and_range.ivecsGround truth for joint-filtered NN search. One vector per query.

Formats

  • —.fvecs: Binary format for 32-bit floating point numbers (used for embedding vectors).
  • —.ivecs: Binary format for 32-bit signed integers (used for ground truth).
  • —.jsonl: Each line contains a JSON object (used for attributes).