datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SIFT1B-DiskANN
SIFT-1B Dataset & Disk Index
The SIFT-1B (BigANN) dataset and pre-built disk-based ANN index.
Built February 2026 on Intel Xeon 8462Y+ (Sapphire Rapids) with 800GB RAM.
Build Parameters
Parameter
Value
Dataset
SIFT-1B (1,000,000,000 vectors, 128-dim, uint8)
Graph R
128 (max degree)
Build L
200 (search list size during construction)
PQ chunks
32 (4 dimensions per sub-quantizer)
Build time
~2 days
Files
Raw Data… See the full description on the dataset page: https://huggingface.co/datasets/Nanvivi/SIFT1B-DiskANN.sift1b
SIFT1B - Sharded DiskANN Indices
Pre-built DiskANN indices for the SIFT1B (BigANN) dataset, sharded for distributed vector search.
Dataset Info
Source: BigANN Benchmarks
Vectors: 1,000,000,000 (1 billion)
Dimensions: 128
Data type: uint8
Queries: 10,000
Distance: L2
DiskANN Parameters
R (graph degree): 64
L (build beam width): 100
PQ bytes: 32
Shard Configurations
shard_2: 2 shards x 500,000,000 vectors
shard_3: 3 shards x ~333,333,333 vectors… See the full description on the dataset page: https://huggingface.co/datasets/makneeeee/sift1b.SIFT-50M
Dataset Card for SIFT-50M
SIFT-50M (Speech Instruction Fine-Tuning) is a 50-million-example dataset designed for instruction fine-tuning and pre-training of speech-text large language models (LLMs). It is built from publicly available speech corpora containing a total of 14K hours of speech and leverages LLMs and off-the-shelf expert models. The dataset spans five languages, covering diverse aspects of speech understanding and controllable speech generation instructions. SIFT-50M… See the full description on the dataset page: https://huggingface.co/datasets/amazon-agi/SIFT-50M.sift1m
sift1m
sift1m data, copied from http://corpus-texmex.irisa.fr/, published:
Jégou H, Douze M, Schmid C. Improving bag-of-features for large scale image search[J]. International journal of computer vision, 2010, 87(3): 316-336.
siftsmallsift100msift-audio
SIFT Audio Dataset
Self-Instruction Fine-Tuning (SIFT) dataset for training audio understanding models.
Dataset Description
This dataset contains audio samples paired with LLM-generated responses following the
AZeroS multi-mode approach. Each audio sample is processed in three different modes
to train models that can both respond conversationally AND describe/analyze audio.
SIFT Modes
Each audio sample generates three training samples with different behaviors:… See the full description on the dataset page: https://huggingface.co/datasets/mazesmazes/sift-audio.siftformer2-data
SiftFormer2 Data
K400 / SSv2 video classification 학습용 데이터.
구성
siftformer2-data/
├── k400/
│ ├── train_manifest.jsonl
│ ├── val_manifest.jsonl
│ ├── label_map.json (400 classes)
│ ├── train/ # ★ 부분만 업로드 (~10K mp4)
│ └── val/ # 19,881 mp4
├── ssv2/
│ ├── train_manifest.jsonl
│ ├── val_manifest.jsonl
│ ├── label_map.json (174 classes)
│ └── videos/ # ★ 부분만 업로드 (~10K webm)
└── yolo_heatmaps/
├── k400.tar # 6.4GB, 259K .npy… See the full description on the dataset page: https://huggingface.co/datasets/koreashin/siftformer2-data.sift-128-euclidean
Dataset Overview
dataset: sift-128-euclidean
Metadata
Creation Time: 2025-01-07 11:37:52+0000
Update Time: 2025-01-07 11:38:07+0000
Source: https://github.com/erikbern/ann-benchmarks
Task: N/A
Train Samples: N/A
Test Samples: N/A
License: DISCLAIMER AND LICENSE NOTICE:
This dataset is intended for benchmarking and research purposes only.
The source data used in this dataset retains its original license and copyright. Users must comply with the respective licenses of… See the full description on the dataset page: https://huggingface.co/datasets/open-vdb/sift-128-euclidean.SIFT100M-DiskANN
SIFT-100M Dataset & Disk Index
The SIFT-100M dataset (first 100M vectors of BigANN SIFT-1B) and pre-built disk-based ANN index.
Built February 2026 on Intel Xeon 8462Y+ (Sapphire Rapids).
Build Parameters
Parameter
Value
Dataset
SIFT-100M (100,000,000 vectors, 128-dim, uint8)
Graph R
96 (max degree)
Build L
128 (search list size during construction)
PQ chunks
32 (4 dimensions per sub-quantizer)
Files
Raw Data
File
Size… See the full description on the dataset page: https://huggingface.co/datasets/Nanvivi/SIFT100M-DiskANN.sift1m
SIFT1M Vector Search Dataset
1 million 128-dimensional SIFT descriptor vectors, a standard ANN benchmark.
Dataset Details
Vectors: 1,000,000
Dimensions: 128
Queries: 10,000
Source: texmex corpus
Shard Configurations
Config
Shards
Vectors/Shard
.indices
.vectors
shard_3
3
333,333
651MB
163MB
shard_5
5
200,000
391MB
98MB
shard_7
7
142,857
279MB
70MB
shard_10
10
100,000
195MB
49MB
DiskANN Parameters
R: 64, L: 100, Distance:… See the full description on the dataset page: https://huggingface.co/datasets/maknee/sift1m.mtg-cards-SIFT-Features
MTG Card SIFT Features Dataset (v5.1)
This dataset contains the latest incremental MTG card SIFT + RootSIFT feature extraction pipeline. It is designed for server-side production inference, enabling additive updates to the FAISS index and id_map.json without retraining or reindexing from scratch.
Note: This version aligns with a daily resources-nightly.zip Hugging Face upload workflow for reliable continuous deployment via my production server.
What’s New in v5.1?… See the full description on the dataset page: https://huggingface.co/datasets/JakeTurner616/mtg-cards-SIFT-Features.sift-archive
Sift — 研究数据归档
Sift 是一个 CPU/DDR-primary + GPU-assisted 的分层内存 MoE + 长上下文推理系统研究项目
（用便宜的大容量 DDR/CXL 承载放不进 HBM 的大型稀疏 MoE + 长上下文；头条指标是 tokens-per-dollar / tokens-per-Joule）。
本仓是该项目自产实验数据的归档，用于把数据从本地磁盘卸下来。
这里没有模型权重 —— 模型是上游公开 GGUF，见 MODELS.manifest.json + restore_models.sh。
取数据
hf download yil384/sift-archive --repo-type dataset fetch_archive.sh --local-dir .
bash fetch_archive.sh # 列出仓里有什么
bash fetch_archive.sh ssd2/traces/v2lite #… See the full description on the dataset page: https://huggingface.co/datasets/yil384/sift-archive.ANN_SIFTsift1mSIFTsift1m-fanns-uniform
SIFT1M Dataset to Evaluate (Filtered) Approximate Nearest Neighbor Search — Uniform Variant
This dataset is intended to benchmark Approximate Nearest Neighbor Search (ANNS) and
Filtered Approximate Nearest Neighbor Search (FANNS) algorithms. It is based on the
classic SIFT1M dataset (1,000,000 base vectors and 10,000 query vectors, each
128-dimensional). Because SIFT1M ships without any structured metadata, we augment it
with synthetic attributes drawn from a uniform… See the full description on the dataset page: https://huggingface.co/datasets/brady0127/sift1m-fanns-uniform.sifta-document-forgery-datasetsift1beq-esconv-sifted
EQ-ESConv-Sifted: Elo-Ranked Emotional Support Conversations
The ESConv dataset (Liu et al., ACL 2021) ranked by empathetic quality via Swiss-style Elo tournament. All 1,300 conversations scored and sorted.
Why this exists
ESConv is a widely-used emotional support dataset but quality varies significantly — some conversations have excellent empathetic support, others are low-effort or off-topic. This dataset adds Elo rankings so you can filter by quality.
For… See the full description on the dataset page: https://huggingface.co/datasets/nivvis/eq-esconv-sifted.SIFTsift1bsift1m-fanns-normal
SIFT1M Dataset to Evaluate (Filtered) Approximate Nearest Neighbor Search — Normal Variant
This dataset is intended to benchmark Approximate Nearest Neighbor Search (ANNS) and
Filtered Approximate Nearest Neighbor Search (FANNS) algorithms. It is based on the
classic SIFT1M dataset (1,000,000 base vectors and 10,000 query vectors, each
128-dimensional). Because SIFT1M ships without any structured metadata, we augment it
with synthetic attributes drawn from a normal (Gaussian)… See the full description on the dataset page: https://huggingface.co/datasets/brady0127/sift1m-fanns-normal.kannolo-sift1Msift-extensionANTON-SIFTAANNS_SIFT1Mdiskann_sift0.1b_filessift1msift10m-modify