datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
jat-dataset-tokenized
Dataset Card for "jat-dataset-tokenized"
More Information needed
tokenizer-wiki-bench
Multilingual Tokenizer Benchmark
This dataset includes pre-processed wikipedia data for tokenizer evaluation in 45 languages. We provide more information on the evaluation task in general this blogpost.
Usage
The dataset allows us to easily calculate tokenizer fertility and the proportion of continued words on any of the supported languages. In the example below we take the Mistral tokenizer and evaluate its performance on Slovak.
from transformers import AutoTokenizer… See the full description on the dataset page: https://huggingface.co/datasets/occiglot/tokenizer-wiki-bench.fineweb-tokenized
FineWeb Tokenized
> 4 trillion tokens of the pre-tokenized data the 🌐 web has to offer
What is it?
This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards.
By distributing the pre-tokenized corpus, we eliminate… See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.gia-dataset-tokenized-2024-2
Dataset Card for "gia-dataset-tokenized-2024-2"
More Information needed
regent-subset-of-jat-dataset-tokenizedc4-10k-mini-tokenized-16-ctx-gelu-1l-testsregent-subset-of-jat-dataset-tokenizedThis is the dataset for REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context In New Environments.
The REGENT dataset includes a subset of the JAT (tokenized) dataset (from https://huggingface.co/datasets/jat-project) for the REGENT training environments.
It has around a 100k transitions from each of the 145 training environments (45 metaworld, 52 atari, 9 mujoco, 39 babyai).
Please find this in the *_subset folders.
It also has distance values for input sequences used in… See the full description on the dataset page: https://huggingface.co/datasets/regent-research/regent-subset-of-jat-dataset-tokenized.c4-tokenized-2b
Dataset Card for "c4-tokenized-2b"
More Information needed
tokenized-falcon2-dutch-2048yarn-train-tokenized-16k-mistral
Dataset Card for "yarn-train-tokenized-16k-mistral"
More Information needed
Claw-SWE-Bench
Claw-SWE-Bench
Paper: Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-Style Agent Harnesses on Coding Tasks
A multilingual issue-resolving benchmark with two evaluation configs:
full — 350 instances (300 from SWE-bench Multilingual + 50 Python from
SWEBench-verified-mini's size_optimized_sample).
lite — 80-instance calibrated subset (10 per language across 8
languages: Java, Go, Rust, JS/TS, C/C++, Ruby, PHP, Python). Designed for
low-cost iteration on harness… See the full description on the dataset page: https://huggingface.co/datasets/TokenRhythm/Claw-SWE-Bench.Skylion007-openwebtext-tokenizer-gpt2Stack_Tokenizedtokenizers-dependents
tokenizers metrics
This dataset contains metrics about the huggingface/tokenizers package.
Number of repositories in the dataset: 11460
Number of packages in the dataset: 124
Package dependents
This contains the data available in the used-by
tab on GitHub.
Package & Repository star count
This section shows the package and repository star count, individually.
Package
Repository
There are 14 packages that have more than 1000 stars.
There are 41… See the full description on the dataset page: https://huggingface.co/datasets/open-source-metrics/tokenizers-dependents.v0-tinystories-v2-clean-tokenizedv0 refers to our tokenization process, v2 refers to stories generated by GPT4 only
tokenized_enwiki
Dataset Card for "tokenized_enwiki"
More Information needed
tokenized-llama3-dutch-2048Healix-2.8B-Token-Medical-Shot
Dataset Card for "Healix-2.8B-Token-Medical-Shot"
More Information needed
seamless-align-enA-jaA.tokenized.encodecPD-3M-Tokenized-Cosmos-Tokenizer-DI8x8I can't get the dataset viewer to work, sorry. There's about 3M images and captions from
Spawning/PD3M.
They are resized and center-cropped to 512x512, and then tokenized into discrete tokens with
NVIDIA Cosmos-Tokenizer-DI8x8,
which reduces the spatial dimension by a factor of 8, resulting in 64 x 64 = 4096 discrete tokens per image.
You can use these tokenized images to train an auto-regressive image model, or a MaskGIT. Or probably
other things I don't know about. :) License is the same… See the full description on the dataset page: https://huggingface.co/datasets/andersonbcdefg/PD-3M-Tokenized-Cosmos-Tokenizer-DI8x8.fineweb-tokenized-fake
What is it?
It's similar to anisolai/fineweb-tokenized but fake.
I don't understand why I did that :)
WARNING:
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN… See the full description on the dataset page: https://huggingface.co/datasets/mondk/fineweb-tokenized-fake.imaginative-perception-token-pet-ipt
Citation
Released with the paper Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models (arXiv:2606.03988):
@misc{bigverdi2026imaginativeperceptiontokensenhance,
title={Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models},
author={Mahtab Bigverdi and Linjie Li and Weikai Huang and Yiming Liu and Jaemin Cho and Jieyu Zhang and Tuhin Kundu and Chris Dangjoo Kim and Zelun Luo and Linda Shapiro and Ranjay… See the full description on the dataset page: https://huggingface.co/datasets/weikaih/imaginative-perception-token-pet-ipt.natural-instructions-tokenized
Dataset Card for "natural-instructions-tokenized"
Here is the script used to tokenize the dataset:
import multiprocessing
from typing import Union
from datasets import DatasetDict, load_dataset
from transformers import LlamaTokenizer
# Find your available cores
num_cores = multiprocessing.cpu_count()
cutoff_len = 2048
tokenizer = LlamaTokenizer.from_pretrained("chainyo/alpaca-lora-7b")
tokenizer.padding_side = "left"
tokenizer.pad_token_id = (0)
prompt_template = {… See the full description on the dataset page: https://huggingface.co/datasets/chainyo/natural-instructions-tokenized.PD-3M-Tokenized-Cosmos-Tokenizer-DI16x16c4-code-tokenized-2b
Dataset Card for "c4-code-tokenized-2b"
More Information needed
pg_books-tokenized-bos-eos-chunked-65536
Dataset Card for "pg_books-tokenized-bos-eos-chunked-65536"
The pg19 dataset tokenized under LLaMA into 64k chunks, bookended with BOS and EOS
monology-pile-uncopyrighted-tokenizer-gpt2olm-october-2022-tokenized-1024
Dataset Card for "olm-october-2022-tokenized-1024"
More Information needed
seamless-align-enA-hiA.tokenized.encodecud-treebank-tokens
Dataset Card for Dataset Name
Dataset Summary
This is a subset of the Universal Dependencies Treebanks dataset (version 2.12) which only contains raw sentences and their corresponding tokenized form.
This dataset is licensed under the Universal Dependencies v2.10 License Agreement. The user is reminded that some subsets only allow non-commercial use. Each row contains the license of the dataset it originated from, and a full listing of all included subsets and… See the full description on the dataset page: https://huggingface.co/datasets/tokenizer-eval/ud-treebank-tokens.