datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
eai-taxonomy-code-w-dclm
💻 EAI-Taxonomy Code w/ DCLM
🏆 Website | 🖥️ Code | 📖 Paper
A 564 billion token dataset of high-quality code curated from web data using taxonomy-based filtering.
🎯 Dataset Overview
This dataset is part of the Essential-Web project, which introduces a new paradigm for dataset curation using expressive metadata and simple semantic filters. Unlike traditional code datasets that require complex domain-specific pipelines, our approach leverages a 12-category taxonomy… See the full description on the dataset page: https://huggingface.co/datasets/EssentialAI/eai-taxonomy-code-w-dclm.dclm-dedup
DCLM-Deduped
DCLM is a recently released high quality dataset that uses model-based quality filtering to filter a large subset of common-crawl for similarity to OpenHermes and other instruction-tuning datasets. For reference see the DCLM paper.
The original authors of DCLM did not release fully deduplicated version of their dataset, claiming that full deduplication did not improve performance. The released version was partially deduplicated in shards.
Nevertheless, when performing… See the full description on the dataset page: https://huggingface.co/datasets/Zyphra/dclm-dedup.DCLM-pro
📚 DCLM-pro
ArXiv | Models | Code
DCLM-pro is refined from DCLM using the ProX refining framework.
It contains about >500B high quality tokens, ready for general language model pre-training.
License
DCLM-pro is based on DCLM, which is made available under an cc-by-4.0 license.
Citation
@article{zhou2024programming,
title={Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale},
author={Zhou, Fan and Wang, Zengzhi… See the full description on the dataset page: https://huggingface.co/datasets/gair-prox/DCLM-pro.dclm-pro-arabic
dclm-pro-arabic
Arabic translation of DCLM-Pro (global shards 01 and 05), translated with Seed-X-PPO-7B using greedy decoding. Documents were split into ~490-token chunks at sentence boundaries, translated, and reassembled. Each row is one complete document. A companion corpus translated with the same pipeline is available at fineweb-edu-arabic.
Details
Documents: 33,245,503 (22.7% of the two source shards, uniformly sampled)
Arabic tokens: ~93B (Seed-X… See the full description on the dataset page: https://huggingface.co/datasets/SultanR/dclm-pro-arabic.dclm-edu-ar-500k
DCLM-Edu Arabic 500K
Description
This is an Arabic translation of the HuggingFaceTB/dclm-edu dataset. The original dataset is a filtered version of the DCLM (DataComp for Language Models) corpus, keeping only samples with an educational quality score of 2 or higher based on the FineWeb-Edu classifier.
I used google/translategemma-12b-it to translate 500K samples from English to Arabic. The goal is to provide high-quality Arabic pretraining data for training small language… See the full description on the dataset page: https://huggingface.co/datasets/SultanR/dclm-edu-ar-500k.finepdfs-dclm-fineweb-edu-fi
FinePDFs · DCLM · FineWeb-Edu — Finnish (machine-translated)
Finnish machine translation of an English pretraining mixture drawn from
FinePDFs, DCLM, and FineWeb-Edu (documents up to ~900 tokens),
produced as continued-pretraining data for Finnish LLMs.
Translation model: translategemma-27b (Gemma-based 27B translation model)
Documents: ~4,000,000 (80 shards × 50,000)
Language: Finnish (fi)
Format: plain text, one document per row
Originally stored as TFDS-style ArrayRecord… See the full description on the dataset page: https://huggingface.co/datasets/Finnish-NLP/finepdfs-dclm-fineweb-edu-fi.dclm-baseline-1.0-parquet_urls
Dataset Card for dclm-baseline-1.0-parquet_urls
This dataset provides the URLs and top-level domains associated with training records in mlfoundations/dclm-baseline-1.0-parquet. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/dclm-baseline-1.0-parquet_urls.DCLM_German
DCLM German Dataset
This dataset contains German language data processed for LLM pretraining, filtered using FastText language detection.
Usage
from datasets import load_dataset
# Load the entire dataset
dataset = load_dataset("faidrap/DCLM_German")
# Stream for large datasets (recommended)
dataset = load_dataset("faidrap/DCLM_German", streaming=True)
# Access the data
for example in dataset['train']:
print(example['text'][:100]) # Print first 100 chars… See the full description on the dataset page: https://huggingface.co/datasets/faidrap/DCLM_German.dclm-replay.seq-4096.n-262144-olmo3
dclm-replay.seq-4096.n-262144-olmo3
OLMo 3-tokenized CPT-replay sequences for prestar, the OLMo analogue of JackHsieh/dclm-replay.seq-4096.tokens-32B (Qwen3).
Source: mlfoundations/dclm-baseline-1.0, pin global-shard_01_of_10/local-shard_0_of_10/*.jsonl.zst.
Tokenizer: allenai/Olmo-3-1025-7B; EOD token id 100257 (<|endoftext|>).
262,144 sequences of exactly 4096 tokens each (docs concatenated and packed; EOD-separated).
Same builder/pin as the Qwen3 replay — corpus is the same… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/dclm-replay.seq-4096.n-262144-olmo3.dclm-crossover-source
DCLM Cross-Over Source
Subset of DCLM-Baseline
selected for synthetic augmentation with format-aware prompt routing.
Selection
Picked every 3th shard (9313 of 27938 shards)
Word count filter: 50-8000
Per-site cap: 10,000
Format detection: skip prompts that duplicate native document format
Stats
Metric
Value
Source docs scanned
54,947,699
Selected
54,017,165
Total words
44,119,449,000
Avg words/doc
816
Length filtered
930,534… See the full description on the dataset page: https://huggingface.co/datasets/essobi/dclm-crossover-source.dclm-dedup_urls
Dataset Card for dclm-dedup_urls
This dataset provides the URLs and top-level domains associated with training records in Zyphra/dclm-dedup. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing so… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/dclm-dedup_urls.dclm-sample-13k-en-et-translationDocument level translations from English to Estonian derived from a sample from the DLCM dataset as present in the dolmino mix translated with google/gemma-3-27b-it .
codelion-finemix-pdf-dclm-edu-1024-seq-len-15897-samples
FIneMix Dataset:
This is a 16,278,528 token, 15,897 text sample, 1094 sequence length (padded) subset of the curriculum used to train codelion/gpt-2-70m.
It is designed for smoke testing, hyperparameter optimization, and baseline comparison on novel small language model architectures before scaling up and burning more resources.
Data Sources and selection:
50% - FinePDFs (500M tokens): High-quality PDF content: ~ 7948 rows selected from… See the full description on the dataset page: https://huggingface.co/datasets/david-thrower/codelion-finemix-pdf-dclm-edu-1024-seq-len-15897-samples.KYS-DCLM-Refinedweb-100M-Scored
KYS-DCLM-Refinedweb-100M-Scored
The candidate document pool behind Know Your Sources: Data Selection Matters when Rewriting for
Data-Constrained Pretraining — 99,949,162 web documents sampled from
DCLM-RefinedWeb, each annotated
with three independent quality scores, their tie-aware global percentiles, a 24-way
WebOrganizer topic label, and a Llama-2 token count.
Every source-selection strategy in the paper is a different way of ranking this table.
Contents
200… See the full description on the dataset page: https://huggingface.co/datasets/blab-jhu/KYS-DCLM-Refinedweb-100M-Scored.