toksuite/tokenizer-comparison

Citation

If you use this space, please cite:

@article{toksuite2025, title={TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior}, author={Altıntaş, Gul Sena and Ehghaghi, Malikeh and Lester, Brian and Liu, Fengyuan and Zhao, Wanru and Ciccone, Marco and Raffel, Colin}, year={2025}, arxiv={arxiv.org/abs/2512.20757} }