datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
dpk-tokenization-sample
DPK tokenization sample input
Five small Parquet files used as the input artifact for the
DPK_Tokenize_Skypilot
template. Total size ~44 KB, so it is committed directly rather than fetched at
build time — the template runs offline apart from the tokenizer download.
Provenance
Copied verbatim from the Data Prep Kit
project (Apache-2.0), release 1.1.8:
transforms/universal/tokenization/test-data/tkn2arrow-ds01/input/
These are DPK's own test fixtures for the… See the full description on the dataset page: https://huggingface.co/datasets/shalisha-witherspoon/dpk-tokenization-sample.tokenization_robustness_v102
Dataset Card for Tokenization Robustness
A comprehensive evaluation dataset for testing robustness of different tokenization strategies.
Dataset Details
Dataset Description
This dataset evaluates how robust language models are to different tokenization strategies and edge cases. It includes questions with multiple choice answers designed to test various aspects of tokenization handling.
Curated by: R3
Funded by [optional]: [More Information Needed]
Shared… See the full description on the dataset page: https://huggingface.co/datasets/r-three/tokenization_robustness_v102.refusal-data-tokenizationprotein-tokenization-experiments-minitokenization_test_data
Dataset Card for Dataset Name
This dataset card aims to be a base template for new datasets. It has been generated using this raw template.
Dataset Details
Dataset Description
Curated by: [More Information Needed]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Language(s) (NLP): [More Information Needed]
License: [More Information Needed]
Dataset Sources [optional]
Repository: [More… See the full description on the dataset page: https://huggingface.co/datasets/hf-internal-testing/tokenization_test_data.nlprepl-nkjp-with-char-level-tokenizationclinical-healing-trajectory-tokenization-phase-segmentation-v0.1What this dataset tests
Whether a model can segment high-frequency recovery datainto interpretable healing phases.
Required outputs
phase_sequence
phase_boundaries
phase_confidence_0_100
Token labels
acute_drop
early_rebound
consolidation_plateau
oscillatory_instability
secondary_drop
delayed_rebound
steady_ascent
maladaptive_plateau
recovery_lock_in
Boundary format
Use day indicesexampleacute_drop d0-d2
Typical failures
naming phases without boundaries… See the full description on the dataset page: https://huggingface.co/datasets/ClarusC64/clinical-healing-trajectory-tokenization-phase-segmentation-v0.1.tokenization-corpusgood_chats_dataset_pre_tokenization
Dataset Card for "good_chats_dataset_pre_tokenization"
More Information needed
test_tokenization
Orpheus PT-BR SNAC 8192
Data Fields
input_ids: List[int] length 8192
attention_mask: List[int] length 8192
metadata: Dict[str, Any] containing original dataset, config, split, audio_length, text
Usage
from datasets import load_dataset
ds = load_dataset("matheusfpinto/orpheus-ptbr-snac-8192", split="train", streaming=True)
sample = next(iter(ds))
assert len(sample["input_ids"]) == 8192
Citation
Please cite the original data sources… See the full description on the dataset page: https://huggingface.co/datasets/matheusfpinto/test_tokenization.blockchain-tokenization-qatest_tokenizationbengali-tokenization-corpus
Bengali Tokenization Corpus (25k Sentences)
Dataset Description
A balanced 25,000-sentence Bengali corpus designed for tokenization benchmarking.
Dataset Summary
This dataset is used in the manuscript:
Quantifying the Tokenization Tax on Bengali: A Multi-Tokenizer Audit Across 25k Sentences
MD. Muktadirul Haque Maruf, Israt Jahan Munny, Moutithi Sarker MouManuscript in preparation
Domains
Academic
News
Literary
Colloquial
Dialectal… See the full description on the dataset page: https://huggingface.co/datasets/M-H-MARUF/bengali-tokenization-corpus.blockchain-tokenization-qa_datasblockchain-tokenization-datas_for_QA