datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
obelics_seed2_tokensPart of the OBELISC data set, including 32 Million samples, please refer to dataset.py to use this data
tokenspace
tokenspace directory
This directory contains utilities for the purpose of browsing the
"token space" of CLIP ViT-L/14
Primary tools are:
"calculate-distances.py": allows command-line browsing of words and their neighbours
"graph-embeddings.py": plots graph of full values of two embeddings
(clipmodel,cliptextmodel)-calculate-distances.py
Loads the generated embeddings, reads in a word, calculates "distance" to every
embedding, and then shows the closest "neighbours".
To… See the full description on the dataset page: https://huggingface.co/datasets/ppbrown/tokenspace.loracle-pretrain-v5-qwen14b-tokensud-treebank-tokens
Dataset Card for Dataset Name
Dataset Summary
This is a subset of the Universal Dependencies Treebanks dataset (version 2.12) which only contains raw sentences and their corresponding tokenized form.
This dataset is licensed under the Universal Dependencies v2.10 License Agreement. The user is reminded that some subsets only allow non-commercial use. Each row contains the license of the dataset it originated from, and a full listing of all included subsets and… See the full description on the dataset page: https://huggingface.co/datasets/tokenizer-eval/ud-treebank-tokens.ERC20-Tokens-Ethereum-Cryptocurrency-Data
ERC20-Tokens-Ethereum-Cryptocurrency-Data
Hive-partitioned Parquet export of BlockDB erc20_tokens (Ethereum).
Load
from datasets import load_dataset
ds = load_dataset("BlockDB/ERC20-Tokens-Ethereum-Cryptocurrency-Data", split="train")
Files live under data/year=YYYY/month=MM/part-NNNN.parquet.
Range: 2023-08 .. 2026-06 (UTC calendar months).
Schema
column
type
block_timestamp
timestamp
block_number
int64
tx_index
int32
contract_id… See the full description on the dataset page: https://huggingface.co/datasets/BlockDB/ERC20-Tokens-Ethereum-Cryptocurrency-Data.loracle-eval-direction-tokensvivoice-libris-mls-eng-10k-tokens-v0.1ichigo_tokens_v1dclm-replay.seq-4096.tokens-32B2^35 tokens of replay data from DCLM-baseline, concatenated into 2^23 sequences of 4096 tokens each with <|endoftext|> separators.
ERC721-Tokens-Ethereum-Cryptocurrency-Data
ERC721-Tokens-Ethereum-Cryptocurrency-Data
Hive-partitioned Parquet export of BlockDB erc721_tokens (Ethereum).
Load
from datasets import load_dataset
ds = load_dataset("BlockDB/ERC721-Tokens-Ethereum-Cryptocurrency-Data", split="train")
Files live under data/year=YYYY/month=MM/part-NNNN.parquet.
Range: 2023-08 .. 2026-06 (UTC calendar months).
Schema
column
type
block_timestamp
timestamp
block_number
int64
tx_index
int32… See the full description on the dataset page: https://huggingface.co/datasets/BlockDB/ERC721-Tokens-Ethereum-Cryptocurrency-Data.the-stack-march-sample-special-tokens-stripped
Dataset Card for "the-stack-march-sample-special-tokens-stripped"
More Information needed
ERC1155-Tokens-Ethereum-Cryptocurrency-Data
ERC1155-Tokens-Ethereum-Cryptocurrency-Data
Hive-partitioned Parquet export of BlockDB erc1155_tokens (Ethereum).
Load
from datasets import load_dataset
ds = load_dataset("BlockDB/ERC1155-Tokens-Ethereum-Cryptocurrency-Data", split="train")
Files live under data/year=YYYY/month=MM/part-NNNN.parquet.
Range: 2015-08 .. 2026-06 (UTC calendar months).
Schema
column
type
block_timestamp
timestamp
block_number
int64
tx_index
int32… See the full description on the dataset page: https://huggingface.co/datasets/BlockDB/ERC1155-Tokens-Ethereum-Cryptocurrency-Data.dclm_10B_tokensGrounded_3D_LLM_with_Referent_Tokens_Dataset
Grounded 3D-LLM Dataset
For detailed information and resources, please visit the following links:
Paper
Arxiv
Project Website
Dataset Access
Code
We are in the process of releasing our data incrementally:
Processed ScanNet200 PCD(~7G):
Each .npyfile represents a N*12 array with the following structure:
coordinates, color, normals, segments, labels = (
points[:, :3],
points[:, 3:6],
points[:, 6:9],
points[:, 9]… See the full description on the dataset page: https://huggingface.co/datasets/ShuaiYang03/Grounded_3D_LLM_with_Referent_Tokens_Dataset.Qwen3.8-27B-Distill-1M-3.12B-Tokens
Qwen3.8-27B-Distill-1M-4.83B-Tokens
A unified, globally deduplicated, large-scale supervised distillation corpus built from 992,318 conversations generated by Qwen/Qwen3.8-27B, containing 4,834,771,862 target output tokens (3,570,459,498 reasoning tokens + 1,264,312,364 final response tokens) and 5,104,980,053 total sequence tokens.
1. Dataset Overview
This dataset merges, aligns, and deduplicates the two primary high-quality Qwen3.8-27B generation corpora on… See the full description on the dataset page: https://huggingface.co/datasets/MaxDevv/Qwen3.8-27B-Distill-1M-3.12B-Tokens.speech-dac-tokens-3cb
Speech DAC Tokens (3 Codebooks)
Pre-tokenized speech dataset using the Descript Audio Codec (DAC). Each audio clip has been encoded into discrete codebook tokens from DAC's first 3 residual vector quantization codebooks, paired with its text transcription.
Dataset Summary
Stat
Value
Total samples
241,451
Total audio
~780 hours
Language
English
Codebooks
3 (of DAC's 9)
Codebook size
1,024 entries each
DAC model
44kHz
Tokens per second
~258 (86 frames… See the full description on the dataset page: https://huggingface.co/datasets/treadon/speech-dac-tokens-3cb.webui-tokens-unlabelled
Dataset Card for WebUI tokens (unlabelled)
Every token over 5 characters long from gbenson/webui-dom-snapshots.
Curated by: Gary Benson
License: CC0 1.0 Universal
Uses
I'm using it to develop a DOM-aware tokenizer for HTML.
Bias, Risks, and Limitations
87% of the source dataset was English language websites, with no other language exceeding 2% of the total
Non-ASCII tokens have been coerced to ASCII using Unidecode where the result appears visually… See the full description on the dataset page: https://huggingface.co/datasets/gbenson/webui-tokens-unlabelled.mixed-speech-instruction-ichigo-tokens-vi-enopen-thoughts-4-30k-math-qwen3-32b-annotated-32768-tokens-n8
Open Thoughts 4 - Math (Qwen3-32B, 32K tokens, n=8)
This dataset contains math reasoning problems with 8 independent responses generated by Qwen3-32B.
Overview
Source: marin-community/open-thoughts-4-30k-math-qwen3-32b-annotated-32768-tokens
Model: Qwen/Qwen3-32B
Temperature: 0.8
Max tokens: 32,768
Columns
Column
Description
instruction_seed
The math problem prompt
_source
Source dataset identifier
gpt41_mini_response
Reference response from… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/open-thoughts-4-30k-math-qwen3-32b-annotated-32768-tokens-n8.midi-tokens-dyadic-tu0.01-vb32-mts1.0-vocab512-epr-labs
midi-tokens-dyadic-tu0.01-vb32-mts1.0-vocab512-epr-labs
Pre-tokenized MIDI pieces for IsoFLOP scaling-law runs. Each row is one full
piece (no time-windowing); training crops sequences from packed token bins.
The source column is the original piece metadata as JSON so a row can be
traced back to its EPR Labs source dataset.
Based on MIDI datasets gathered by EPR Labs.
Codec
name: dyadic
tokenizer vocab size: 512
max_time_step: 1.0
n_velocity_bins: 32… See the full description on the dataset page: https://huggingface.co/datasets/wmatejuk/midi-tokens-dyadic-tu0.01-vb32-mts1.0-vocab512-epr-labs.QuRatedPajama-1B_tokens_for_analysis
QuRatedPajama
Paper: QuRating: Selecting High-Quality Data for Training Language Models
This dataset is a 1B token subset derived from princeton-nlp/QuRatedPajama-260B, which is a subset of cerebras/SlimPajama-627B annotated by princeton-nlp/QuRater-1.3B with sequence-level quality ratings across 4 criteria:
Educational Value - e.g. the text includes clear explanations, step-by-step reasoning, or questions and answers
Facts & Trivia - how much factual and trivia knowledge the text… See the full description on the dataset page: https://huggingface.co/datasets/princeton-nlp/QuRatedPajama-1B_tokens_for_analysis.wikipedia-2023-11-pt-tokens
Dataset Card for "wikipedia-2023-11-pt-tokens"
More Information needed
pgn-dataset-including-special-tokens
PGN Dataset Including Special Tokens
Last Updated: 2025-01-26
Description
This is a dataset of chess games in Portable Game Notation (PGN) format. The dataset was created by cleaning and formatting the milesh1/35-million-chess-games dataset from Kaggle.
This version of the pgn-dataset includes the [g_start] and [g_end] special tokens that PGNTokenizer uses to denote the start and end of a game.
Notes
This was the training dataset for PGNTokenizer. For more… See the full description on the dataset page: https://huggingface.co/datasets/InterwebAlchemy/pgn-dataset-including-special-tokens.open-thoughts-4-30k-code-qwen3-30b-a3B-thinking-2507-annotated-32768-tokens-n8
Open Thoughts 4 - Code (Qwen3-30B-A3B-Thinking-2507, 32K tokens, n=8)
This dataset contains code reasoning problems with 8 independent responses generated by Qwen3-30B-A3B-Thinking-2507.
Overview
Source: marin-community/open-thoughts-4-30k-code-qwen3-32b-annotated (prompts only)
Model: Qwen/Qwen3-30B-A3B-Thinking-2507
Temperature: 0.8
Max tokens: 32,768
Columns
Column
Description
instruction_seed
The code problem prompt
_source
Source dataset… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/open-thoughts-4-30k-code-qwen3-30b-a3B-thinking-2507-annotated-32768-tokens-n8.ru-big-russian-dataset-16k-tokens-limitdata-32k-200b-tokens
TR-HASH 32K · 200B Token Mixture
Pretokenized training mixture for compact TR-HASH language-model research.
The Dataset Viewer displays one summary row per source. The actual training data
is stored as packed uint16 token shards under corpora/<source>/tokens-*.bin.
Each sequence contains 1,024 token IDs produced by the project 32K tokenizer.
Mixture
Source
Weight
Training tokens
DCLM
45%
90B
FineWeb-Edu deduplicated
30%
60B
Stack-Edu
10%
20B… See the full description on the dataset page: https://huggingface.co/datasets/Pacific-i64/data-32k-200b-tokens.open-thoughts-4-30k-math-qwen3-30b-a3B-thinking-2507-annotated-32768-tokens-n8
Open Thoughts 4 - Math (Qwen3-30B-A3B-Thinking-2507, 32K tokens, n=8)
This dataset contains math reasoning problems with 8 independent responses generated by Qwen3-30B-A3B-Thinking-2507.
Overview
Source: marin-community/open-thoughts-4-30k-math-qwen3-32b-annotated (base prompts)
Model: Qwen/Qwen3-30B-A3B-Thinking-2507
Temperature: 0.8
Max tokens: 32,768
Columns
Column
Description
instruction_seed
The math problem prompt
_source
Source dataset… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/open-thoughts-4-30k-math-qwen3-30b-a3B-thinking-2507-annotated-32768-tokens-n8.everyayah-with-tajweed-tokensopen-thoughts-4-30k-math-qwen3-30b-a3B-thinking-2507-annotated-32768-tokens-n8-reformatted
Dataset Card for Open-Thoughts-4-30K-Math-Qwen3-30B-A3B-Thinking-2507-Annotated-32768-Tokens-N8-Reformatted
Overview
This dataset is a reformatted version of marin-community/open-thoughts-4-30k-math-qwen3-30b-a3B-thinking-2507-annotated-32768-tokens-n8. The original dataset contained 29,963 samples, each with 8 responses generated by the same model with different random seeds (stored in generated_text, generated_text2, ..., generated_text8 columns). This reformatted… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/open-thoughts-4-30k-math-qwen3-30b-a3B-thinking-2507-annotated-32768-tokens-n8-reformatted.fineweb-sample-100BT_over-1024-tokens