datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
filtered_articles_by_year
Dataset Card for Filtered Articles by Year
Dataset Summary
The Filtered Articles by Year dataset contains yearly-segmented web articles from the FineWeb dataset, specifically filtered and processed for temporal language analysis and Word2Vec model training. This dataset spans 21 years (2005-2025) and serves as the foundation for research into semantic change, concept emergence, and language evolution over time.
Supported Tasks and Leaderboards
This dataset… See the full description on the dataset page: https://huggingface.co/datasets/adameubanks/filtered_articles_by_year.HinMix
HinMix (https://arxiv.org/abs/2512.18834) is a Hindi pretraining corpus containing 76 billion tokens across 60 million documents (in the minhash subset). Rather than scraping the web again, HinMix combines six publicly available Hindi datasets, applies Hindi-specific quality filtering, and performs cross-dataset deduplication.
We train a 1.4B parameter language model through nanotron on 30 billion tokens to show that HinMix outperforms the previous state-of-the-art, CulturaX… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/HinMix.TurMix
TurMix (https://arxiv.org/abs/2512.18834) is a Turkish pretraining corpus containing 168 billion tokens across 219 million documents (in the minhash subset). Rather than scraping the web again, TurMix combines five publicly available Turkish datasets, applies Turkish-specific quality filtering, and performs cross-dataset deduplication.
We train a 1.4B parameter language model through nanotron on 30 billion tokens to show that the matched subset of TurMix outperforms the… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/TurMix.AraMix-HQ
AraMix family: AraMix (minhash and matched) | AraMix-domain-classified (with domain labels) | AraMix-HQ (model-filtered)
AraMix-HQ is a high-quality subset of AraMix-MinHash created using model-based quality scoring. We adapt the approach from FineWeb2-HQ but replace the XLM-Roberta encoder with mmBERT, which provides better Arabic language understanding. We release the model at AdaMLLab/mmBERT-Arabic-Quality-Classifier.
AraMix-HQ outperforms both AraMix-Matched and FineWeb2-HQ… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/AraMix-HQ.AraMix
AraMix family: AraMix (minhash and matched) | AraMix-domain-classified (with domain labels) | AraMix-HQ (model-filtered)
AraMix (https://arxiv.org/abs/2512.18834) is an Arabic pretraining corpus containing 178 billion tokens across 179 million documents (in the minhash subset). Rather than scraping the web again, AraMix combines seven publicly available Arabic datasets, applies Arabic-specific quality filtering, and performs cross-dataset deduplication.We train a 1.4B parameter… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/AraMix.ThaiMix
ThaiMix (https://arxiv.org/abs/2512.18834) is a Thai pretraining corpus containing 70 billion tokens across 81 million documents (in the minhash subset). Rather than scraping the web again, ThaiMix combines five publicly available Thai datasets, applies Thai-specific quality filtering, and performs cross-dataset deduplication.
Subsets
Subset
Documents
Tokens
Description
minhash_deduped
81.3M
70.5B
Document-level MinHash deduplication
matched
10.9M… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/ThaiMix.AraMix-domain-classified
AraMix Domain-Classified
AraMix family: AraMix (minhash and matched) | AraMix-domain-classified (with domain labels) | AraMix-HQ (model-filtered)
This is AraMix with per-document domain labels from nvidia/multilingual-domain-classifier.
Usage
from datasets import load_dataset
ds = load_dataset("AdaMLLab/AraMix-domain-classified", "minhash_deduped")
ds = load_dataset("AdaMLLab/AraMix-domain-classified", "sentence_deduped")
Schema
Field… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/AraMix-domain-classified.VieMix
VieMix (https://arxiv.org/abs/2512.18834) is a Vietnamese pretraining corpus built by combining six publicly available Vietnamese datasets, applying Vietnamese-specific quality filtering, and performing cross-dataset deduplication.
Subsets
Subset
Description
quality_filtered
Quality-filtered data before deduplication
minhash_deduped
Document-level MinHash deduplication
matched
Documents appearing in 2+ source datasets
The matched subset uses… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/VieMix.KorMix
KorMix (https://arxiv.org/abs/2512.18834) is a Korean pretraining corpus built by combining five publicly available Korean datasets, applying Korean-specific quality filtering, and performing cross-dataset deduplication.
Subsets
Subset
Description
quality_filtered
Quality-filtered data before deduplication
minhash_deduped
Document-level MinHash deduplication
matched
Documents appearing in 2+ source datasets
The matched subset uses cross-dataset… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/KorMix.WebTerminal
Terminal/CLI Web Text
A filtered extract of terminal and command-line content from two large web-text corpora, designed for upsampling agentic-adjacent data during pretraining.
Subsets
Subset
Rows
Tokens
Size
Quality
clean (default)
2.33M
4.6B
11 GB
~98% terminal content
unfiltered
61.3M
359B
962 GB
~15% terminal content
from datasets import load_dataset
# Load the clean subset (default)
ds = load_dataset("AdaMLLab/WebTerminal")
# Load the unfiltered… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/WebTerminal.smolkalam-arabic-conversational-sft
SmolKalam
SmolKalam is a quality-filtered Arabic SFT dataset of 1,790,478 examples (~2.45B tokens), built as an ensemble translation of SmolTalk2. It covers multi-turn dialogue (23% of rows), reasoning traces (19% carry <think>), tool and function calling (4.4%), and long context, categories that are underrepresented in existing Arabic post-training data. The SmolTalk2 source mixtures are kept as subsets.
Released with the paper SmolKalam: Ensemble Quality-Filtered Translation… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/smolkalam-arabic-conversational-sft.IndMix
IndMix (https://arxiv.org/abs/2512.18834) is an Indonesian pretraining corpus built by combining six publicly available Indonesian datasets, applying Indonesian-specific quality filtering, and performing cross-dataset deduplication.
Subsets
Subset
Description
quality_filtered
Quality-filtered data before deduplication
minhash_deduped
Document-level MinHash deduplication
matched
Documents appearing in 2+ source datasets
The matched subset uses… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/IndMix.nyu-aco-ocr-full
nyu-aco-ocr-full
Scanned Arabic books from the Arabic Collections Online (ACO) archive, OCR'd page by page with the dots.ocr vision-language model served through vLLM. The archive spans seven partner collections: NYU, Princeton, Cornell, Columbia, AUB, AUC, and UAE National Archives.
Each PDF page is rendered at 200 DPI, OCR'd individually, and the pages of a book are joined into one markdown document separated by \n\n---\n\n. Each row is one complete book.
Schema… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/nyu-aco-ocr-full.OpenSakura-DS-260220-LN-ja-zh-PT-Adam
OpenSakura Adam LN Pretrain Dataset
OpenSakura-DS-260220-LN-ja-zh-PT-Adam is a large-scale pretraining corpus built from light-novel source shards and filtered to Japanese/Chinese scripts.
This export is intended for PT/CPT-style language modeling.
Dataset Summary
Metric
Value
Dataset ID
OpenSakura/OpenSakura-DS-260220-LN-ja-zh-PT-Adam
Total rows
9,515,512
Total parquet files
480
Total size
63,621,025,693 bytes (~63.62 GB, ~59.25 GiB)
Languages (BCP-47)… See the full description on the dataset page: https://huggingface.co/datasets/OpenSakura/OpenSakura-DS-260220-LN-ja-zh-PT-Adam.ThaMix
ThaMix (https://arxiv.org/abs/2512.18834) is a Thai pretraining corpus built by combining seven publicly available Thai datasets, applying Thai-specific quality filtering, and performing cross-dataset deduplication.
Subsets
Subset
Description
quality_filtered
Quality-filtered data before deduplication
minhash_deduped
Document-level MinHash deduplication
matched
Documents appearing in 2+ source datasets
The matched subset uses cross-dataset agreement… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/ThaMix.JpnMix
JpnMix (https://arxiv.org/abs/2512.18834) is a Japanese pretraining corpus built by combining five publicly available Japanese datasets, applying Japanese-specific quality filtering, and performing cross-dataset deduplication.
Subsets
Subset
Description
quality_filtered
Quality-filtered data before deduplication
minhash_deduped
Document-level MinHash deduplication
matched
Documents appearing in 2+ source datasets
The matched subset uses… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/JpnMix.ItaMix
ItaMix (https://arxiv.org/abs/2512.18834) is an Italian pretraining corpus built by combining five publicly available Italian datasets, applying Italian-specific quality filtering, and performing cross-dataset deduplication.
Subsets
Subset
Description
minhash_deduped
Document-level MinHash deduplication
matched
Documents appearing in 2+ source datasets
The matched subset uses cross-dataset agreement as a signal for quality.
Usage… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/ItaMix.Fable-5-Max-Reasoning-Filtered-250x
Dataset Description
This dataset contains 25. highly detailed architectural traces mapping out security implementations for hybrid global banking systems encompassing both fiat and cryptocurrency infrastructures.
This is 10,000,000+ estimated tokens of fable 5 data, filtered and classified to remove low-quality entries by qwen 2.5 7B, and improved by GLM 5.2. The dataset bypasses basic conversational filler and is engineered to advance the domain precision, strict formatting… See the full description on the dataset page: https://huggingface.co/datasets/adamm-hf/Fable-5-Max-Reasoning-Filtered-250x.cgrt-consensus-5model
CGRT Consensus 5-Model Dataset
Multi-model consensus dataset for studying model agreement and disagreement patterns on mathematical reasoning tasks.
Dataset Description
61,678 math problems evaluated by 5 frontier LLMs with full reasoning traces and extracted answers.
Models Used
Model
Provider
Version
Claude
Anthropic
claude-3-5-sonnet-20241022
Codex/GPT-4
OpenAI
gpt-4o
Gemini
Google
gemini-1.5-flash
DeepSeek
DeepSeek
deepseek-chat
Qwen… See the full description on the dataset page: https://huggingface.co/datasets/Adam1010/cgrt-consensus-5model.youtube-titles
Youtube Title & Descriptions Dataset
About
4941 videos across 50 YouTube Channels
List of sampled channels here
Splits:
Train: 4199
Validation: 493
Test: 249
Data was shuffled and sampled evenly from all channels to create splits.
Additionally, has a column ready to go for gemma-2-9b-it fine tuning formatting! Potentially more model formats to come.
About the Data:
Label
Description
channel_name
The… See the full description on the dataset page: https://huggingface.co/datasets/AdamLucek/youtube-titles.aprm-sft-thoughts-tau2-retail-policy_best-adamw30-lp0
Act-PRM SFT thoughts — tau2-bench retail
Act-PRM (Action Process Reward Models) infers the latent thoughts behind
logged, action-only agent demonstrations via an offline EM. For each
logged action x in state s we sample G=4 candidate thoughts z,
score each by the length-penalized action likelihood
reward(z) = p(x | s, z)
(len_frac grows with the thought's token length), and mark the best thought
(argmax reward). The (thought + action) span is then what downstream SFT / RL… See the full description on the dataset page: https://huggingface.co/datasets/mzio/aprm-sft-thoughts-tau2-retail-policy_best-adamw30-lp0.no_robots-alpaca
No Robots: Alpaca edition
This dataset is a cleaned (missing/extra spaces...) and reformatted version of the No Robots dataset from HuggingFaceH4, adapted to conform with the Alpaca instruction set.
Notably, it diverges from the original dataset in the way the 'Chat' category is handled; it has been decomposed into single-turn conversations to align with Alpaca's limitations regarding multi-turn interactions. The dataset's IDs have been generated using the SHA256 algorithm.… See the full description on the dataset page: https://huggingface.co/datasets/AdamCodd/no_robots-alpaca.aprm-sft-thoughts-snorkel-insurance-policy_best-adamw30-lp0
Act-PRM SFT thoughts — snorkel-insurance insurance
Act-PRM (Action Process Reward Models) infers the latent thoughts behind
logged, action-only agent demonstrations via an offline EM. For each
logged action x in state s we sample G=4 candidate thoughts z,
score each by the length-penalized action likelihood
reward(z) = p(x | s, z)
(len_frac grows with the thought's token length), and mark the best thought
(argmax reward). The (thought + action) span is then what downstream SFT… See the full description on the dataset page: https://huggingface.co/datasets/mzio/aprm-sft-thoughts-snorkel-insurance-policy_best-adamw30-lp0.pii-masking-300k
Purpose and Features
🌍 World's largest open dataset for privacy masking 🌎
The dataset is useful to train and evaluate models to remove personally identifiable and sensitive information from text, especially in the context of AI assistants and LLMs.
Key facts:
OpenPII-220k text entries have 27 PII classes (types of sensitive data), targeting 749 discussion subjects / use cases split across education, health, and psychology. FinPII contains an additional ~20 types tailored to… See the full description on the dataset page: https://huggingface.co/datasets/AdamiTitus/pii-masking-300k.aprm-sft-thoughts-tau2-airline-policy_best-adamw30-lp0
Act-PRM SFT thoughts — tau2-bench airline
Act-PRM (Action Process Reward Models) infers the latent thoughts behind
logged, action-only agent demonstrations via an offline EM. For each
logged action x in state s we sample G=4 candidate thoughts z,
score each by the length-penalized action likelihood
reward(z) = p(x | s, z)
(len_frac grows with the thought's token length), and mark the best thought
(argmax reward). The (thought + action) span is then what downstream SFT / RL… See the full description on the dataset page: https://huggingface.co/datasets/mzio/aprm-sft-thoughts-tau2-airline-policy_best-adamw30-lp0.startup-advisor-dataset
🚀 Startup Advisor Dataset
A high-quality instruction-following dataset distilled from 8 foundational business and startup books, structured as actionable advice with real-world 2025 examples. Designed for fine-tuning large language models (e.g., Qwen, LLaMA, Mistral) to become expert startup advisors.
📖 Dataset Summary
Property
Value
Total Entries
1,564
Format
JSONL — ChatML (messages array)
Language
English
License
CreativeML OpenRAIL-M
Avg. Tokens… See the full description on the dataset page: https://huggingface.co/datasets/adamabuhamdan/startup-advisor-dataset.claudish-pairs
Claudish Pairs
The first open parallel corpus of English ↔ Claudish — the characteristic prose
style of Claude and Claude Code. 10,227 pairs, each an English text and its Claudish
restyling, authored and quality-controlled for faithfulness.
This is the v3 training set of
adamrotmil/claudish-style-adapter;
pipeline code at
github.com/adamrotmil/claudish-style-adapter.
Fields
Field
Meaning
english
source text (plain English)
claudish
the restyling… See the full description on the dataset page: https://huggingface.co/datasets/adamrotmil/claudish-pairs.aprm-sft-thoughts-snorkel-finance-policy_best-adamw30-lp0
Act-PRM SFT thoughts — snorkel-finance finance
Act-PRM (Action Process Reward Models) infers the latent thoughts behind
logged, action-only agent demonstrations via an offline EM. For each
logged action x in state s we sample G=4 candidate thoughts z,
score each by the length-penalized action likelihood
reward(z) = p(x | s, z)
(len_frac grows with the thought's token length), and mark the best thought
(argmax reward). The (thought + action) span is then what downstream SFT / RL… See the full description on the dataset page: https://huggingface.co/datasets/mzio/aprm-sft-thoughts-snorkel-finance-policy_best-adamw30-lp0.aprm-sft-thoughts-tau2-airline-base_best-adamw30-lp0
Act-PRM SFT thoughts — tau2-bench airline
Act-PRM (Action Process Reward Models) infers the latent thoughts behind
logged, action-only agent demonstrations via an offline EM. For each
logged action x in state s we sample G=4 candidate thoughts z,
score each by the length-penalized action likelihood
reward(z) = p(x | s, z)
(len_frac grows with the thought's token length), and mark the best thought
(argmax reward). The (thought + action) span is then what downstream SFT / RL… See the full description on the dataset page: https://huggingface.co/datasets/mzio/aprm-sft-thoughts-tau2-airline-base_best-adamw30-lp0.goodhart-gap-benchmark
Goodhart Gap Benchmark
Detecting the gap between understanding and execution in language models
Overview
The Goodhart Gap Benchmark tests whether language models can correctly execute multi-step reasoning tasks that they can correctly explain. Named after Goodhart's Law ("When a measure becomes a target, it ceases to be a good measure"), this benchmark reveals a critical failure mode: models that understand procedures but fail to execute them.
Key Finding
In… See the full description on the dataset page: https://huggingface.co/datasets/Adam1010/goodhart-gap-benchmark.