datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
jora_corpus1_tokenized_128kfineweb-tokenized
FineWeb Tokenized
> 4 trillion tokens of the pre-tokenized data the 🌐 web has to offer
What is it?
This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards.
By distributing the pre-tokenized corpus, we eliminate… See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.swallow-code-v2
SwallowCode-v2
Resources
📑 arXiv: Read our paper for detailed methodology and results at arXiv:2505.02881.
🤗 Sister Dataset: Discover SwallowMath-v2, our companion dataset for mathematical reasoning.
💻 What is it?
SwallowCode-v1 was a high-quality Python code dataset generated through an LLM-based rewriting pipeline.
However, it had two significant limitations:
(1) it was distributed under the Llama 3.3 Community License, and
(2) its size was limited to… See the full description on the dataset page: https://huggingface.co/datasets/tokyotech-llm/swallow-code-v2.hrm-tokenized-bpe65ktokenizers-dependents
tokenizers metrics
This dataset contains metrics about the huggingface/tokenizers package.
Number of repositories in the dataset: 11460
Number of packages in the dataset: 124
Package dependents
This contains the data available in the used-by
tab on GitHub.
Package & Repository star count
This section shows the package and repository star count, individually.
Package
Repository
There are 14 packages that have more than 1000 stars.
There are 41… See the full description on the dataset page: https://huggingface.co/datasets/open-source-metrics/tokenizers-dependents.seamless-align-enA-jaA.tokenized.encodecsubliminal-transfer-token-replacement
Subliminal transfer: token replacement vs masking (artifacts)
Teachers, training data, per-token divergence scores and evaluation outputs for
brendanlong/subliminal-transfer-token-replacement.
The experiment asks whether replacing attribution-flagged tokens suppresses a
subliminally transmitted trait better than masking them from the loss, and
whether any advantage is specific to those tokens. Everything here is for the
one studied cell: Llama-3.2-1B-Instruct, target animal… See the full description on the dataset page: https://huggingface.co/datasets/brendanlong/subliminal-transfer-token-replacement.fineweb-tokenized-fake
What is it?
It's similar to anisolai/fineweb-tokenized but fake.
I don't understand why I did that :)
WARNING:
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN… See the full description on the dataset page: https://huggingface.co/datasets/mondk/fineweb-tokenized-fake.seamless-align-enA-hiA.tokenized.encodecloracle-pretrain-v5-qwen14b-tokensmeta-llama-Llama-3.2-1B-toksuite-detokenizedTraining data of the model detokenized in the exact order seen by the model.
The training data is partitioned into 8 chunks (chunk-0 through chunk-7), based on the GPU rank that generated the data. Each chunk contains detokenized text files in JSON Lines format (.jsonl).
RULER-8192-Qwen2.5-3B-tokenizerseamless-align-enA-esA.tokenized.encodecseamless-align-enA-viA.tokenized.encodecseamless-align-deA-enA.tokenized.encodecparler-tts_mls_eng_10k_snac_token_old
Dataset Card for Dataset Name
This dataset card aims to be a base template for new datasets. It has been generated using this raw template.
Dataset Details
Dataset Description
Curated by: [More Information Needed]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Language(s) (NLP): [More Information Needed]
License: [More Information Needed]
Dataset Sources [optional]
Repository: [More… See the full description on the dataset page: https://huggingface.co/datasets/blanchon/parler-tts_mls_eng_10k_snac_token_old.swallow-code
SwallowCode
Notice
May 21, 2025: We have deleted ablation/exp1-the-stack-v2-train-smol-ids-python because it was flagged as potentially containing unsafe data collected from the Python subset of https://huggingface.co/datasets/bigcode/the-stack-v2-train-smol-ids. However, since this dataset can be reconstructed from the-stack-v2-train-smol-ids, there is no issue in terms of reproducibility.
May 21, 2025: ClamAV has flagged “Win.Trojan.MSShellcode-88” in… See the full description on the dataset page: https://huggingface.co/datasets/tokyotech-llm/swallow-code.carbon-tokenized-corpus
Dataset Summary
AINovice2005/carbon-tokenized-corpus is the tokenized sample of AINovice2005/carbon-cpu-enriched-sequences-sampled .
Schema
The current dataset contains the following fields:
Field
Type
Description
record_id
string
Source/reference sequence identifier
start
int64
Start coordinate of the sequence interval
end
int64
End coordinate of the sequence interval
token_ids
list
Integer token IDs produced by the tokenizer
token_mask
list… See the full description on the dataset page: https://huggingface.co/datasets/AINovice2005/carbon-tokenized-corpus.0399-tv-valid-clean-sft-tokenized-llmjp4-8bToken-To-Token-Prices-OHLC-Ethereum-Cryptocurrency-Data
Token-To-Token-Prices-OHLC-Ethereum-Cryptocurrency-Data
Hive-partitioned Parquet export of BlockDB token_to_token_prices_ohlc (Ethereum).
Load
from datasets import load_dataset
ds = load_dataset("BlockDB/Token-To-Token-Prices-OHLC-Ethereum-Cryptocurrency-Data", split="train")
Files live under data/year=YYYY/month=MM/part-NNNN.parquet.
Range: 2015-08 .. 2026-06 (UTC calendar months).
Schema
column
type
block_timestamp
timestamp… See the full description on the dataset page: https://huggingface.co/datasets/BlockDB/Token-To-Token-Prices-OHLC-Ethereum-Cryptocurrency-Data.clt_pretrain_data_qwen_tokenizedaya-expanse-8b-toksuite-detokenizedTraining data of the model detokenized in the exact order seen by the model.
The training data is partitioned into 8 chunks (chunk-0 through chunk-7), based on the GPU rank that generated the data. Each chunk contains detokenized text files in JSON Lines format (.jsonl).
gpt-4o-toksuite-detokenizedTraining data of the model detokenized in the exact order seen by the model.
The training data is partitioned into 8 chunks (chunk-0 through chunk-7), based on the GPU rank that generated the data. Each chunk contains detokenized text files in JSON Lines format (.jsonl).
Token-To-Token-VWAP-Ethereum-Cryptocurrency-Data
Token-To-Token-VWAP-Ethereum-Cryptocurrency-Data
Hive-partitioned Parquet export of BlockDB token_to_token_vwap (Ethereum).
Load
from datasets import load_dataset
ds = load_dataset("BlockDB/Token-To-Token-VWAP-Ethereum-Cryptocurrency-Data", split="train")
Files live under data/year=YYYY/month=MM/part-NNNN.parquet.
Range: 2015-08 .. 2026-06 (UTC calendar months).
Schema
column
type
block_timestamp
timestamp
bucket_start
timestamp… See the full description on the dataset page: https://huggingface.co/datasets/BlockDB/Token-To-Token-VWAP-Ethereum-Cryptocurrency-Data.clt-pretrain-data-tokenized-Qwen3-1024tokenised_subsetof_erickfmm__red_pajama_es_hq_35exp-pool-repository-code-dolma2-tokenized
Locus EXP Repository Code - Dolma 2 tokenized
Pretokenized experiment pool for reproducible proxy-training runs.
MANIFEST.json is the authoritative schema, provenance, checksums, and train/holdout assignment.
shards/shard-NNNNN/tokens.bin stores little-endian int32 token IDs.
offsets.bin stores little-endian int64 document boundaries.
index.parquet stores document IDs, offsets, and compact filter fields.
metadata.parquet stores complete source metadata and is downloaded only… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/exp-pool-repository-code-dolma2-tokenized.urls-tokenized
URLs (tokenized)
ks46/urls-sampled run through a byte-level
BPE built for URLs, stored as flat uint16 token streams that memory-map
directly into a training loop.
Shards
512
URLs
18,729,786,698
Tokens
664,731,047,208
Vocabulary
8,192
Token dtype
uint16, little-endian
There is no parquet here and the dataset viewer will not render it. These
are raw token bins; see Reading the data below.
Layout
tokenizer/ the exact vocabulary… See the full description on the dataset page: https://huggingface.co/datasets/ks46/urls-tokenized.jora_corpus1_FR_tokenized_128ktokenized_sample