toksuite/toksuite_general

Dataset Card for Tokenization Robustness TokSuite Bonus Benchmarks (General Collection) This is a bonus TokSuite dataset containing a small set of high-signal examples that highlight surface-form variations known to affect tokenization robustness. It includes canonical questions alongside perturbations such as abbreviations, character deletion, currency symbols, diverse date formats, and unusual formatting. These examples focus on tokenization challenges that… See the full description on the dataset page: https://huggingface.co/datasets/toksuite/toksuite_general.