test-data
swe-bench-dummy-test-datasettokenizers-test-data
tokenizers-test-data
Test and benchmark fixtures for huggingface/tokenizers,
pulled on demand by the repo Makefiles (make test / make bench / make fixtures
via hf download).
Layout
fixtures/ — multilingual + modality corpora for cross-language encode
benchmarks. Organized, documented, and reproducible: see
fixtures/FIXTURES.md for provenance and
fixtures/fixtures_manifest.json for
exact sources, pinned revisions, and sizes. Rebuild any file with… See the full description on the dataset page: https://huggingface.co/datasets/hf-internal-testing/tokenizers-test-data.emit-test-dataset
Dataset Card for EMIT-MSeg Dataset
If you use this dataset, please cite our article:
@misc{herec2026fastmethanedetectionpipeline,
title={A Fast Methane Detection Pipeline on Board Satellites Based on Mag1c-SAS and LinkNet},
author={Jonáš Herec and Vít Růžička and Rado Pitoňák and Jan Sedmidubsky},
year={2026},
eprint={2606.03675},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2606.03675},
}… See the full description on the dataset page: https://huggingface.co/datasets/onboard-coop/emit-test-dataset.dataset-test-1Scicode-test-data-h5hindi_audio_dataset_test