CoolFace
Datasetpublic

hf-internal-testing/tokenizers-test-data

tokenizers-test-data Test and benchmark fixtures for huggingface/tokenizers, pulled on demand by the repo Makefiles (make test / make bench / make fixtures via hf download). Layout fixtures/ — multilingual + modality corpora for cross-language encode benchmarks. Organized, documented, and reproducible: see fixtures/FIXTURES.md for provenance and fixtures/fixtures_manifest.json for exact sources, pinned revisions, and sizes. Rebuild any file with… See the full description on the dataset page: https://huggingface.co/datasets/hf-internal-testing/tokenizers-test-data.

sourceHugging Faceupdated 11d agoView on Hugging Face
0likes17kdownloads
Dataset Card

tokenizers-test-data

Test and benchmark fixtures for huggingface/tokenizers, pulled on demand by the repo Makefiles (make test / make bench / make fixtures via hf download).

Layout

  • —fixtures/ — multilingual + modality corpora for cross-language encode benchmarks. Organized, documented, and reproducible: see `fixtures/FIXTURES.md` for provenance and `fixtures/fixtures_manifest.json` for exact sources, pinned revisions, and sizes. Rebuild any file with `fixtures/fetch_fixtures.py`.
  • —fixtures/lang/<iso639-3>_<iso15924>.txt — ~5 MB of real web text per language (FineWeb-2 / FineWeb), 13 non-Latin scripts + English baseline
  • —fixtures/modalities/ — source code, math/LaTeX, and coding-agent trajectories
  • —Root files (legacy) — tokenizer JSONs, vocab/merges files, and text corpora referenced by their flat paths from the tokenizers repo Makefiles (Rust, Python, and Node bindings) on main and release branches. Do not move or rename them; new additions should go into organized subdirectories instead.

Licensing

Corpora under fixtures/ are small excerpts of public datasets, each under its own license — FineWeb/FineWeb-2 (ODC-By), open-web-math (ODC-By), SWE-smith-trajectories (MIT), and permissively-licensed OSS repos for code (see the manifest for the exact repo list and commit SHAs). Root-level files predate this README and carry the licenses of their original sources.