tokenization
tokenization-multiplicity-data
Dataset: Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service
This dataset contains the official experiment inference traces for the paper Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service by Ivi Chatzi, Nina Corvelo Benz, Stratis Tsirtsis and Manuel Gomez-Rodriguez.
📂 Dataset Structure
The dataset is organized into folders as follows:
.\{model}\{task}\{lang}\{seed}_{10*temperature}.jsonl
where {model}… See the full description on the dataset page: https://huggingface.co/datasets/Human-Centric-Machine-Learning/tokenization-multiplicity-data.retok-noncanonical-tokenization
Non-canonical tokenization in LLM generations
Per-generation records from seven language models, capturing the token IDs each
model actually emitted alongside the canonical re-encoding of its own output —
plus the trained toy-model checkpoints from the accompanying controlled
experiment.
Code, writeup and full run log:
https://github.com/brendanlong/tokenization-hidden-computation-experiment
Tokenization is many-to-one: many token sequences decode to the same string, but… See the full description on the dataset page: https://huggingface.co/datasets/brendanlong/retok-noncanonical-tokenization.dpk-tokenization-sample
DPK tokenization sample input
Five small Parquet files used as the input artifact for the
DPK_Tokenize_Skypilot
template. Total size ~44 KB, so it is committed directly rather than fetched at
build time — the template runs offline apart from the tokenizer download.
Provenance
Copied verbatim from the Data Prep Kit
project (Apache-2.0), release 1.1.8:
transforms/universal/tokenization/test-data/tkn2arrow-ds01/input/
These are DPK's own test fixtures for the… See the full description on the dataset page: https://huggingface.co/datasets/shalisha-witherspoon/dpk-tokenization-sample.tokenization_robustness_v102
Dataset Card for Tokenization Robustness
A comprehensive evaluation dataset for testing robustness of different tokenization strategies.
Dataset Details
Dataset Description
This dataset evaluates how robust language models are to different tokenization strategies and edge cases. It includes questions with multiple choice answers designed to test various aspects of tokenization handling.
Curated by: R3
Funded by [optional]: [More Information Needed]
Shared… See the full description on the dataset page: https://huggingface.co/datasets/r-three/tokenization_robustness_v102.refusal-data-tokenizationcyrillic-vs-latin-tokenization
Cyrillic Tokenization Overhead Benchmark
This dataset accompanies the paper "Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems" submitted to the MRL Workshop at EMNLP 2026. It contains all the data needed to reproduce the paper's three studies, along with a balanced BPE tokenizer trained as part of the research.
What's inside
Directory
What it contains
study01_corpus_benchmark/
Tokenization fertility measured on the BrUK corpus (1.34M… See the full description on the dataset page: https://huggingface.co/datasets/idobrovolskyi/cyrillic-vs-latin-tokenization.