datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-tokenized
FineWeb Tokenized
> 4 trillion tokens of the pre-tokenized data the 🌐 web has to offer
What is it?
This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards.
By distributing the pre-tokenized corpus, we eliminate… See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.Claw-SWE-Bench
Claw-SWE-Bench
Paper: Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-Style Agent Harnesses on Coding Tasks
A multilingual issue-resolving benchmark with two evaluation configs:
full — 350 instances (300 from SWE-bench Multilingual + 50 Python from
SWEBench-verified-mini's size_optimized_sample).
lite — 80-instance calibrated subset (10 per language across 8
languages: Java, Go, Rust, JS/TS, C/C++, Ruby, PHP, Python). Designed for
low-cost iteration on harness… See the full description on the dataset page: https://huggingface.co/datasets/TokenRhythm/Claw-SWE-Bench.Stack_TokenizedFineWeb-Edu-10B-Tokens-NPY
FineWeb-Edu 10B Tokens (NPY Format)
数据集概述
这是一个预处理好的教育文本数据集，包含约100亿个tokens，专门为训练小型语言模型（如GPT-2 124M）而设计。数据来源于高质量的FineWeb-Edu数据集，已经使用GPT-2的tiktoken分词器进行预处理，并保存为numpy格式以提高训练效率。
Followed by Let's reproduce GPT-2 (124M). Thanks to Andrej Karpathy!!!
🎯 适用场景
小型语言模型训练：特别适合GPT-2 124M/350M等参数规模的模型
教育研究：高质量教育内容，适合教学和学术研究
快速原型开发：预处理完成，可直接用于训练间
📊 数据统计
总token数量：~10,000,000,000 tokens
分片大小：100M tokens/分片
数据格式：numpy (.npy) uint16数组
分词器：GPT-2 tiktoken
语言：英语… See the full description on the dataset page: https://huggingface.co/datasets/ShallowU/FineWeb-Edu-10B-Tokens-NPY.subliminal-transfer-token-replacement
Subliminal transfer: token replacement vs masking (artifacts)
Teachers, training data, per-token divergence scores and evaluation outputs for
brendanlong/subliminal-transfer-token-replacement.
The experiment asks whether replacing attribution-flagged tokens suppresses a
subliminally transmitted trait better than masking them from the loss, and
whether any advantage is specific to those tokens. Everything here is for the
one studied cell: Llama-3.2-1B-Instruct, target animal… See the full description on the dataset page: https://huggingface.co/datasets/brendanlong/subliminal-transfer-token-replacement.fineweb-tokenized-fake
What is it?
It's similar to anisolai/fineweb-tokenized but fake.
I don't understand why I did that :)
WARNING:
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN… See the full description on the dataset page: https://huggingface.co/datasets/mondk/fineweb-tokenized-fake.token-counts
Marin Token Counts
Token counts for all datasets used in Marin pretraining runs.
Schema
Column
Type
Description
dataset
string
Dataset identifier
marin_tokens
int
Number of tokens after tokenization
category
string
Content domain (web, code, math, academic, books, etc.)
synthetic
bool
Whether the data is LLM-generated or LLM-translated
Categories
web — Quality-classified Common Crawl text (Nemotron-CC)
code — Source code and… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/token-counts.Scientific_Research_Tokenized
NexaSci Scientific Research Tokenized
This dataset repository now holds the active NexaSci scientific pretraining reservoir, the NexaMat controller fine-tuning pack, and archived legacy reservoir builds. The current production reservoir is the 10B-token Apache Arrow release under nexasci_reservoir_v3_10b_prod_rust/.
Current Status
The active large-scale training artifact is:
nexasci_reservoir_v3_10b_prod_rust/
It was produced from the NexaSci 10B data-engineering campaign… See the full description on the dataset page: https://huggingface.co/datasets/AethronPhantom/Scientific_Research_Tokenized.Qwen-Terminal-ToolBench-Processed-Tokenized
Qwen Terminal ToolBench Processed Datasets
Qwen-family processed/template-applied and selected tokenized terminal datasets.
Contents
qwen35_2b_full_terminal_toolcall_processed_v1: 1011776 rows, 36.26 GiB, features: text
qwen35_2b_terminal_toolcall_fullconv_from_samecount_v1: 327383 rows, 25.11 GiB, features: text
qwen35_27b_hf_fsdp_1024: 15335 rows, 0.17 GiB, features: input_ids, attention_mask, labels
qwen35_27b_processed_2bdata: 23472 rows, 2.77 GiB, features:… See the full description on the dataset page: https://huggingface.co/datasets/LLM-OS-Models/Qwen-Terminal-ToolBench-Processed-Tokenized.code_x_glue_cc_code_completion_token
Dataset Card for "code_x_glue_cc_code_completion_token"
Dataset Summary
CodeXGLUE CodeCompletion-token dataset, available at https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/CodeCompletion-token
Predict next code token given context of previous tokens. Models are evaluated by token level accuracy.
Code completion is a one of the most widely used features in software development through IDEs. An effective code completion tool could improve software… See the full description on the dataset page: https://huggingface.co/datasets/google/code_x_glue_cc_code_completion_token.tokenization-multiplicity-data
Dataset: Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service
This dataset contains the official experiment inference traces for the paper Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service by Ivi Chatzi, Nina Corvelo Benz, Stratis Tsirtsis and Manuel Gomez-Rodriguez.
📂 Dataset Structure
The dataset is organized into folders as follows:
.\{model}\{task}\{lang}\{seed}_{10*temperature}.jsonl
where {model}… See the full description on the dataset page: https://huggingface.co/datasets/Human-Centric-Machine-Learning/tokenization-multiplicity-data.exp-pool-repository-code-dolma2-tokenized
Locus EXP Repository Code - Dolma 2 tokenized
Pretokenized experiment pool for reproducible proxy-training runs.
MANIFEST.json is the authoritative schema, provenance, checksums, and train/holdout assignment.
shards/shard-NNNNN/tokens.bin stores little-endian int32 token IDs.
offsets.bin stores little-endian int64 document boundaries.
index.parquet stores document IDs, offsets, and compact filter fields.
metadata.parquet stores complete source metadata and is downloaded only… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/exp-pool-repository-code-dolma2-tokenized.carbon-tokenized-corpus
Dataset Summary
AINovice2005/carbon-tokenized-corpus is the tokenized representation of sequence intervals processed in the Carbon enrichment pipeline.
Schema
The current dataset contains the following fields:
Field
Type
Description
record_id
string
Source/reference sequence identifier
start
int64
Start coordinate of the sequence interval
end
int64
End coordinate of the sequence interval
token_ids
list
Integer token IDs produced by the tokenizer… See the full description on the dataset page: https://huggingface.co/datasets/AINovice2005/carbon-tokenized-corpus.clt_gpt2_tokenized_control
Fresh multilingual GPT-2 CLT control data
Sequential, unshuffled control sample for CLT null experiments. For each language,
complete source documents were tokenized with CausalNLP/gpt2-hf_multilingual-20 at revision
0afbb31b2db3f394270d42d6a4cb7f8fceeca3d8. The first 100,000,000 tokenizer tokens were discarded
(including the complete document that crossed the threshold), after which complete
documents were retained until at least 100,000,000 tokens were collected.
Data are… See the full description on the dataset page: https://huggingface.co/datasets/abir-hr196/clt_gpt2_tokenized_control.climbmix-tokenized-20480-diloco
ClimbMix, retokenized and shuffled for three-worker DiLoCo
This is a document-preserving, three-way split of NVIDIA's
Nemotron-ClimbMix,
retokenized with a 20,480-entry byte-level BPE tokenizer. Each document ends in
<|endoftext|>. The Arrow IPC streams use transparent Zstandard buffer
compression. A deterministic whole-shard holdout is shared by every worker for
validation and is excluded from training.
Training part
Documents
Tokens
Files
Compressed size
000
15,709… See the full description on the dataset page: https://huggingface.co/datasets/Sambarboi/climbmix-tokenized-20480-diloco.urls-tokenized
URLs (tokenized)
ks46/urls-sampled run through a byte-level
BPE built for URLs, stored as flat uint16 token streams that memory-map
directly into a training loop.
Shards
512
URLs
18,729,786,698
Tokens
664,731,047,208
Vocabulary
8,192
Token dtype
uint16, little-endian
There is no parquet here and the dataset viewer will not render it. These
are raw token bins; see Reading the data below.
Layout
tokenizer/ the exact vocabulary… See the full description on the dataset page: https://huggingface.co/datasets/ks46/urls-tokenized.Pt-Corpus-Instruct-tokenized
Portuguese-Corpus Instruct (tokenized)
Dataset Summary
This repository has a tokenized version (using the TeenyTinyLlama tokenizer) of the Portuguese-Corpus Instruct dataset. All sequences are 2048 tokens long. All sequences are 2048 tokens long. This dataset was used in "TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese".
For more information, see the original dataset card.
Languages
Portuguese.
Dataset Structure… See the full description on the dataset page: https://huggingface.co/datasets/nicholasKluge/Pt-Corpus-Instruct-tokenized.retok-noncanonical-tokenization
Non-canonical tokenization in LLM generations
Per-generation records from seven language models, capturing the token IDs each
model actually emitted alongside the canonical re-encoding of its own output —
plus the trained toy-model checkpoints from the accompanying controlled
experiment.
Code, writeup and full run log:
https://github.com/brendanlong/tokenization-hidden-computation-experiment
Tokenization is many-to-one: many token sequences decode to the same string, but… See the full description on the dataset page: https://huggingface.co/datasets/brendanlong/retok-noncanonical-tokenization.tokenizers
Polygl0t Tokenizers
Dataset Summary
This dataset contains several subsets for training multilingual tokenizers. Every subset possesses a collection of curated text samples in different languages.
Supported Tasks and Leaderboards
This dataset can be used for the task of text generation, specifically for training and evaluating tokenizers in multiple languages.
Languages
Hindi, Bengali, English, Portuguese, and Code (a mixture of 36 programming… See the full description on the dataset page: https://huggingface.co/datasets/Polygl0t/tokenizers.dolma-v1_7-305B-tokenized-llama3-nanosetTokenized (Llama 3) verison of NousResearch/dolma-v1_7-305B as a Nanotron dataset split into 10 GB chunks.
To download:
huggingface-cli download --repo-type dataset --local-dir dolma-v1_7-305B-tokenized-llama3-nanoset --local-dir-use-symlinks False NousResearch/dolma-v1_7-305B-tokenized-llama3-nanoset
To recombine:
cat dolma-v1_7-305B-tokenized-llama3-nanoset/dolma-v1_7-305B-tokenized-llama3-nanoset.npy.* > dolma-v1_7-305B-tokenized-llama3-nanoset.npy
rm -rf… See the full description on the dataset page: https://huggingface.co/datasets/emozilla/dolma-v1_7-305B-tokenized-llama3-nanoset.exp-pool-commit-code-dolma2-tokenized
Locus EXP Commit Code - Dolma 2 tokenized
Pretokenized experiment pool for reproducible proxy-training runs.
MANIFEST.json is the authoritative schema, provenance, checksums, and train/holdout assignment.
shards/shard-NNNNN/tokens.bin stores little-endian int32 token IDs.
offsets.bin stores little-endian int64 document boundaries.
index.parquet stores document IDs, offsets, and compact filter fields.
metadata.parquet stores complete source metadata and is downloaded only for… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/exp-pool-commit-code-dolma2-tokenized.african-multilingual-tokenizer-challenge
African Multilingual Tokenizer Challenge dataset
The frozen public corpus for the African Multilingual Tokenizer Challenge. It contains one balanced multilingual train split and one balanced validation split.
Split
Per language
Total
Train
40,000
240,000
Validation
4,000
24,000
Languages are English (en), French (fr), Hausa (ha), Swahili (sw), Yoruba (yo) and Amharic (am). Official public-test and private-test text are deliberately absent from this repository.… See the full description on the dataset page: https://huggingface.co/datasets/Similoluwa/african-multilingual-tokenizer-challenge.damr-zyda2-64k-tokenized
DAMR Zyda2 64K Tokenized
This repository contains a deterministic tokenized representation of a
1,749,978,785,641-token subset of
Zyda-2 for language-model
pretraining.
Format
Files under data/ contain contiguous little-endian uint16 token IDs.
Concatenate shards in numeric order to reproduce the original stream. The
shared 64K BPE tokenizer is stored under tokenizer/tokenizer.json. Shard
manifests provide byte offsets, token counts, and SHA-256 checksums.
The… See the full description on the dataset page: https://huggingface.co/datasets/zee-drytis/damr-zyda2-64k-tokenized.ultrafineweb-100B-tokens
Tokenized Ultra-FineWeb (100B English Tokens)
This repository provides a tokenized version of the English split of the openbmb/Ultra-FineWeb dataset, prepared for large-scale language model training. The dataset consists of 100 billion high-quality tokens, processed with a custom tokenizer.
The data is sharded into 100 files, each containing exactly 1 billion tokens, making it easy to stream and use in distributed training setups.
Dataset Details
Source Dataset:… See the full description on the dataset page: https://huggingface.co/datasets/meryyllebr543/ultrafineweb-100B-tokens.fineweb_10BT_tokenized
Dataset card for FineWeb-Edu 10B tokenized dataset
This dataset contains tokenized texts from FineWeb-Edu sample-10B HuggingFaceFW/fineweb-edu.
The data was tokenized using the OpenAI's tiktoken tokenizer, and structured for efficient streaming and distributed (DDP) training.
Structure
The dataset follows Hugging Face’s recommended structure for efficient streaming in multi-GPU environments.
It consists of two splits, where each split contains a number of shards… See the full description on the dataset page: https://huggingface.co/datasets/nikolina-p/fineweb_10BT_tokenized.llm0to1-pt-tokenized-en-edu-2025
LLM0to1 사전학습 토큰화본 — 영어 교육(2025 덤프)
10B 규모 한/영 이중언어 LLM LLM0to1-10b 를 바닥부터 학습할 때 실제로 투입된
영어 교육(2025 덤프) 코퍼스의 토큰화본. 총 28종 / 138.4B 토큰.
왜 원문 텍스트가 아니라 토큰화본인가
이 코퍼스들은 공개 데이터셋을 스트리밍으로 받아 곧바로 토큰화했고 중간 텍스트를 보관하지 않았다.
따라서 이 .ds 파일이 학습에 들어간 데이터의 유일한 사본이다.
단점만 있는 건 아니다. 토큰화본은 학습 입력 그 자체이므로,
재토큰화 과정에서 생길 수 있는 차이 없이 학습을 그대로 재현할 수 있다.
원본 출처
HuggingFaceFW/fineweb-edu 2025 덤프
영어 벤치마크 하락에 대응해 뒤늦게 편입한 '다양성 영어' 보강분이다. g00~`g15` 는 원본 shard 를 균등 분할한 것으로, 서로 다른 문서 집합이다.… See the full description on the dataset page: https://huggingface.co/datasets/izlley2/llm0to1-pt-tokenized-en-edu-2025.clt_gpt2_tokenized_control
Fresh multilingual GPT-2 CLT control data
Sequential, unshuffled control sample for CLT null experiments. For each language,
complete source documents were tokenized with CausalNLP/gpt2-hf_multilingual-20 at revision
0afbb31b2db3f394270d42d6a4cb7f8fceeca3d8. The first 100,000,000 tokenizer tokens were discarded
(including the complete document that crossed the threshold), after which complete
documents were retained until at least 100,000,000 tokens were collected.
Data are… See the full description on the dataset page: https://huggingface.co/datasets/CausalNLP/clt_gpt2_tokenized_control.llm0to1-pt-tokenized-code
LLM0to1 사전학습 토큰화본 — 코드
10B 규모 한/영 이중언어 LLM LLM0to1-10b 를 바닥부터 학습할 때 실제로 투입된
코드 코퍼스의 토큰화본. 총 27종 / 59.1B 토큰.
왜 원문 텍스트가 아니라 토큰화본인가
이 코퍼스들은 공개 데이터셋을 스트리밍으로 받아 곧바로 토큰화했고 중간 텍스트를 보관하지 않았다.
따라서 이 .ds 파일이 학습에 들어간 데이터의 유일한 사본이다.
단점만 있는 건 아니다. 토큰화본은 학습 입력 그 자체이므로,
재토큰화 과정에서 생길 수 있는 차이 없이 학습을 그대로 재현할 수 있다.
원본 출처
bigcode/starcoderdata(언어별 서브셋) · bigcode/commitpackft · bigcode/jupyter-code-text-pairs · deepmind/code_contests
code_c 와 code_c2 처럼 2 가 붙은 것은 정제기… See the full description on the dataset page: https://huggingface.co/datasets/izlley2/llm0to1-pt-tokenized-code.exp-pool-academic-dolma2-tokenized
Locus EXP Academic - OLMo 2 tokenized
Pretokenized experiment pool for reproducible proxy-training runs.
MANIFEST.json is the authoritative schema, provenance, checksums, and train/holdout assignment.
shards/shard-NNNNN/tokens.bin stores little-endian int32 token IDs.
offsets.bin stores little-endian int64 document boundaries.
index.parquet stores document IDs, offsets, and compact filter fields.
metadata.parquet stores complete source metadata and is downloaded only for… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/exp-pool-academic-dolma2-tokenized.code-switching-tokenizer-robustness
Code-Switching Dataset for Tokenizer Robustness Analysis
Dataset Description
This dataset is designed for tokenizer robustness testing in multilingual and code-switching contexts. It contains identical content expressed across 16 different language variants, including pure English and 15 English-X code-switching pairs, allowing researchers to isolate tokenization effects from semantic differences when evaluating language models.
Purpose
Tokenizer Comparison:… See the full description on the dataset page: https://huggingface.co/datasets/Malikeh1375/code-switching-tokenizer-robustness.