datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
OpenThoughts-1k-sample
[!NOTE]
We have released a paper for OpenThoughts! See our paper here.
Open-Thoughts-1k-sample
This is a 1k sample of the OpenThoughts-114k dataset.
Open synthetic reasoning dataset with high-quality examples covering math, science, code, and puzzles!
Inspect the content with rich formatting with Curator Viewer.
Available Subsets
default subset containing ready-to-train data used to finetune the OpenThinker-7B and OpenThinker-32B models:
ds =… See the full description on the dataset page: https://huggingface.co/datasets/ryanmarten/OpenThoughts-1k-sample.cosmos_qaevolkit-logprobs-pipeline-75k-v2-sampleessential-web-1t-sample-fdc-partitioned
🌐 Essential-Web: FDC Level-2 Partitioned Dataset
📋 Dataset Description
This dataset contains a 1 trillion token sample from Essential-Web, partitioned by Free Decimal Correspondence (FDC) level-2 categories. Essential-Web is a 24-trillion-token web dataset with extensive document-level metadata designed to enable rapid dataset curation through SQL-like filtering.
🔍 Free Decimal Correspondence (FDC)
The FDC taxonomy is an open classification system… See the full description on the dataset page: https://huggingface.co/datasets/Research-EAI/essential-web-1t-sample-fdc-partitioned.babilong-1k-samples
BABILong (1000 samples) : a long-context needle-in-a-haystack benchmark for LLMs
Preprint is on arXiv and code for LLM evaluation is available on GitHub.
BABILong Leaderboard with top-performing long-context models.
bAbI + Books = BABILong
BABILong is a novel generative benchmark for evaluating the performance of NLP models in
processing arbitrarily long documents with distributed facts.
It contains 9 configs, corresponding to different sequence lengths in tokens: 0k… See the full description on the dataset page: https://huggingface.co/datasets/RMT-team/babilong-1k-samples.rpj-v2-sampleThis is a mirror of the sample-10B subset of RedPajama-Data-V2 which we have re-uploaded in order to resolve issues with the original download script.
Getting Started
RedPajama-V2 is an open dataset for training large language models. The dataset includes over 100B text
documents coming from 84 CommonCrawl snapshots and processed using
the CCNet pipeline. Out of these, there are 30B documents in the corpus
that additionally come with quality signals. In addition, we also provide the… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/rpj-v2-sample.olm-CC-MAIN-2022-21-sampling-ratio-0.14775510204
Dataset Card for OLM May 2022 Common Crawl
Cleaned and deduplicated pretraining dataset, created with the OLM repo here from 15% of the May 2022 Common Crawl snapshot.
Note: last_modified_timestamp was parsed from whatever a website returned in it's Last-Modified header; there are likely a small number of outliers that are incorrect, so we recommend removing the outliers before doing statistics with last_modified_timestamp.
cats_vs_dogs_sampleolm-CC-MAIN-2022-27-sampling-ratio-0.16142697881
Dataset Card for OLM June/July 2022 Common Crawl
Cleaned and deduplicated pretraining dataset, created with the OLM repo here from 16% of the June/July 2022 Common Crawl snapshot.
Note: last_modified_timestamp was parsed from whatever a website returned in it's Last-Modified header; there are likely a small number of outliers that are incorrect, so we recommend removing the outliers before doing statistics with last_modified_timestamp.
BirdSet
BirdSet (Mirror)
This dataset repository is a convenience mirror of BirdSet.
Attribution
Please credit the original BirdSet authors and resources:
Original dataset card: https://huggingface.co/datasets/DBD-research-group/BirdSet
Original project repository: https://github.com/DBD-research-group/BirdSet
Paper: https://arxiv.org/abs/2403.10380
Citation
If you use this dataset, please cite the original BirdSet paper:… See the full description on the dataset page: https://huggingface.co/datasets/samuelstevens/BirdSet.olm-CC-MAIN-2017-22-sampling-ratio-0.16178770949
Dataset Card for OLM May 2017 Common Crawl
Cleaned and deduplicated pretraining dataset, created with the OLM repo here from 16% of the May 2017 Common Crawl snapshot.
Note: last_modified_timestamp was parsed from whatever a website returned in it's Last-Modified header; there are likely a small number of outliers that are incorrect, so we recommend removing the outliers before doing statistics with last_modified_timestamp.
wildchat_creative_writing_annotated_10ksamanantar
Dataset Card for Samanantar
Dataset Summary
Samanantar is the largest publicly available parallel corpora collection for Indic language: Assamese, Bengali,
Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu.
The corpus has 49.6M sentence pairs between English to Indian Languages.
Supported Tasks and Leaderboards
[More Information Needed]
Languages
Samanantar contains parallel sentences between English (en) and 11 Indic… See the full description on the dataset page: https://huggingface.co/datasets/ai4bharat/samanantar.olm-CC-MAIN-2022-33-sampling-ratio-0.20
Dataset Card for OLM August 2022 Common Crawl
Cleaned and deduplicated pretraining dataset, created with the OLM repo here from 20% of the August 2022 Common Crawl snapshot.
Note: last_modified_timestamp was parsed from whatever a website returned in it's Last-Modified header; there are likely a small number of outliers that are incorrect, so we recommend removing the outliers before doing statistics with last_modified_timestamp.
wiki-sample
Loading dataset without vector embeddings
You can load the raw dataset without vectors, like this:
from datasets import load_dataset
dataset = load_dataset("weaviate/wiki-sample", split="train", streaming=True)
Loading dataset with vector embeddings
You can also load the dataset with vectors, like this:
from datasets import load_dataset
dataset = load_dataset("weaviate/wiki-sample", "weaviate-snowflake-arctic-v2", split="train", streaming=True)
for item in dataset:… See the full description on the dataset page: https://huggingface.co/datasets/weaviate/wiki-sample.redbajama-sampledminipile_100_samplesRedPajama-Data-1T-Sample-BackupSampleToHiyori
SampleToHiyori
'모모세 히요리(桃瀬 ひより)' 페르소나 학습용 한국어 데이터셋. config 두 개로 이루어진다.
config
split
행 수
내용
default
train
4,837
단일 턴 한국어 페르소나 대화 (instruction / response)
tools
train / eval
5,495 / 322
도구 호출(function calling) 대화
히요리는 상대를 항상 "오빠" 라고 부르고, 일인칭은 "히요리", 말투는 "인걸" / "인거야" 다.
tools
OpenMascotAI 마스코트의 자비스 모드(윈도우를 실제로 조작하는 모드)에서 쓰기 위한 도구 호출
학습 데이터. 페르소나 LoRA를 얹으면 베이스 모델이 도구를 전혀 호출하지 않게 되는 현상을 고치려고
만들었다. 시나리오(도구·인자·결과·브리프)는 자매 데이터셋 MelissaJ/ProjectLucia_Hera… See the full description on the dataset page: https://huggingface.co/datasets/MelissaJ/SampleToHiyori.dolma3_300B_sampleOpenThoughts-1k-Sampledlldms-associative-memory-samples
LLDMs Associative Memory — Generated Samples
Model-generated text for the paper:
Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data
Bao Pham, Mohammed J. Zaki, Luca Ambrogioni, Dmitry Krotov, Matteo Negri
Accepted to EMNLP 2026 (Main Conference).
arXiv:2604.26841 · paper · code · checkpoints
29.5 million generated sequences (~3.8B tokens) sampled from the released checkpoints — one
generation run per (model size, training-set fraction). These… See the full description on the dataset page: https://huggingface.co/datasets/lemoncmd/lldms-associative-memory-samples.olm-CC-MAIN-2022-49-sampling-ratio-olm-0.15114822547
Dataset Card for OLM November/December 2022 Common Crawl
Cleaned and deduplicated pretraining dataset, created with the OLM repo here from 15% of the November/December 2022 Common Crawl snapshot.
Note: last_modified_timestamp was parsed from whatever a website returned in it's Last-Modified header; there are likely a small number of outliers that are incorrect, so we recommend removing the outliers before doing statistics with last_modified_timestamp.
eye-web-breachesdrone-audio-detection-samples
Dataset Description
Drone Audio Detection Samples (DADS) is currently the largest publicly available drone audio database, specifically designed for developing drone detection systems using deep learning techniques. All audio files are standardized to a sample rate of 16,000 Hz, 16-bit depth, mono-channel, and vary in length from 500 milliseconds to several minutes.
Most drone audio files were manually trimmed to ensure that a drone was always present in the recording. However, some… See the full description on the dataset page: https://huggingface.co/datasets/geronimobasso/drone-audio-detection-samples.eye-web-passwordsqwen3-8b-activations-l20-l36
Qwen3 8B Activations for Layers 20 and 36
This dataset contains assistant-token residual activations harvested from Qwen/Qwen3-8B over 980000 training conversations from lmsys/lmsys-chat-1m.
We only generated for Layer 20 and 36 because each one costs 2TB and we simply cannot afford to store more :)
You can use this dataset to train SAEs, linear probes, other mech interp models etc, for Qwen3 8B.
We picked Qwen3 8B because this is a small part of a larger experiment to use feature… See the full description on the dataset page: https://huggingface.co/datasets/sammyliu/qwen3-8b-activations-l20-l36.riddles-dedupedDSCodeBench
DSCodeBench
Task-grouped, multidimensional code-generation quality estimation data derived from DSCodeBench.
Dataset contents
The release contains 24,972 complete artifact rows from 999 tasks. The source commit is e75ef26fedea7415bdffd3e1cbff95ddad89e7e2.
Each row contains the task instruction, released 200-case test generator, generated Python code, generator identity, sandbox execution context, the independently collected 200-element correctness vector, and four… See the full description on the dataset page: https://huggingface.co/datasets/Samsoup/DSCodeBench.olm-CC-MAIN-2022-40-sampling-ratio-0.15894621295-seed-69