toksuite/toksuite_turkish

Dataset Card for Tokenization Robustness

<!-- Provide a quick summary of the dataset. -->

<img src="toksuite-logo.png" alt="TokSuite Logo" width="250px" style="margin-left:'auto' margin-right:'auto' display:'block'"/>

TokSuite Benchmark (Turkish Collection)

Dataset Description

This dataset is part of TokSuite, a comprehensive benchmark designed to measure how different tokenization strategies affect language model performance and robustness. This specific subset contains Turkish language multiple-choice text completion questions with various real-world perturbations that test tokenizer robustness.

• —Curated by: R3 Research Team
• —Language(s): Turkish (Tr)
• —License: MIT License

Dataset Summary

TokSuite addresses a fundamental challenge in language model research: understanding how tokenization choices impact model behavior in isolation. The Turkish subset specifically measures model performance on canonical questions and various perturbations.

Key Features:

• —40 canonical questions covering general knowledge, geography, science, and language understanding
• —Multiple perturbation types reflecting real-world text variations in Turkish
• —Parallel structure with TokSuite benchmark (available in English, Italian, Farsi, Chinese)
• —Native speaker curation ensuring linguistic authenticity

Supported Tasks

• —Multiple-Choice Question Answering: Text completion format with 4 answer choices
• —Tokenizer Robustness Evaluation: Measuring performance degradation under various text perturbations
• —Multilingual NLP Benchmarking: Evaluating language models on Turkish text understanding

Languages

The dataset contains text in Turkish (language code: tur_Latn / tr).

Dataset Structure

Data Fields

Dataset Creation

Curation Rationale

This dataset was created to:

1. 1.Systematically evaluate how different tokenization strategies handle Turkish
2. 2.Measure robustness against real-world text perturbations specific to Turkish
3. 3.Support research into the impact of tokenization on language model behavior
4. 4.Provide standardized benchmarks for Turkish language models

The questions were designed to be straightforward with high baseline accuracy, allowing researchers to cleanly measure performance degradation when perturbations are applied.

Source Data

Data Collection and Processing

• —Canonical Questions: 40 baseline questions created in English
• —Translation: Native Turkish speakers translated questions
• —Perturbations: Each question underwent targeted perturbations designed to reflect Turkish characteristics
• —Validation: Model-in-the-loop process ensured high baseline accuracy

Perturbation Categories

1. 1.Canonical The baseline Turkish text written in standard, grammatically correct Turkish with no perturbations. This serves as the reference condition for evaluating the impact of all other perturbations.

1. 1.Code / Language / Script Switching Mixes Turkish with English words or phrases within the same sentence, reflecting real-world code-switching common in technical, academic, or online Turkish text.

1. 1.Colloquial Uses colloquial versions of the words and phrases (e.g. Bir to Bi).

1. 1.Date Formats Varies date representations (e.g., 12.03.2022, 12 Mart 2022, 03/12/22), testing sensitivity to formatting and punctuation variation.

1. 1.Derivations Suffixes added to a stem to create new words (e.g. Kedi to Kediciǧim).

1. 1.Dialects Introduces regional Turkish dialectal or colloquial variants that preserve meaning but differ lexically or morphologically from Standard Turkish.

1. 1.English Keyboard Simulates Turkish text typed on an English keyboard, leading to missing or substituted Turkish-specific characters (e.g., cok instead of çok, saglik instead of sağlık).

1. 1.Equivalent Expressions Uses equivalent expressions, like synonyms, historical words, similar words, etc. (e.g. paspas to halı, işe başladım to çalışıyorum.)

1. 1.Grammatical Errors Injects plausible grammatical mistakes such as incorrect suffix usage, agreement errors, or case marking issues, reflecting non-standard or learner Turkish.

1. 1.Inflections Adds suffixes marking case, possession, tense, person, and number on nouns and verbs, often without needing separate pronouns (e.g. Dünyadaki to Dünyamızdaki (on our Earth)).

1. 1.Keyboard Proximity Errors Introduces typos caused by pressing adjacent keys on a keyboard, simulating realistic typing errors without intentionally changing word choice.

1. 1.Orthographic Errors Applies spelling mistakes that violate standard Turkish orthography (e.g., incorrect consonant usage or misspelled suffixes) while remaining plausible to native readers.

1. 1.Similar Words Substitutes words with closely related or easily confusable alternatives (e.g., near-synonyms or minimal lexical contrasts), preserving sentence plausibility.

1. 1.Spelled-Out Forms Replaces numerals, abbreviations, or symbols with fully spelled-out Turkish equivalents, increasing sequence length and altering token boundaries.

1. 1.Typographical Errors Introduces general typographical mistakes such as duplicated letters, missing characters, or minor corruption commonly found in fast or careless typing.

1. 1.Web Search Query Rewrites questions in the style of Turkish web search queries, using keyword-heavy phrasing, omitted function words, and informal structure typical of search engine inputs.

1. 1.Word Re-ordering Re-ordering of the words, often used in informal settings.

Who are the source data producers?

Native Turkish speakers curated and validated all questions and perturbations. The TokSuite research team at R3 designed the overall benchmark framework.

Annotations

Annotation process

Questions were manually created and translated by native speakers. Each perturbation was carefully designed to reflect authentic variations encountered in real-world Turkish text processing.

Who are the annotators?

Native Turkish speakers with expertise in linguistics and NLP, working as part of the TokSuite project.

Personal and Sensitive Information

The dataset contains only general knowledge questions and does not include any personal or sensitive information.

Considerations for Using the Data

Social Impact of Dataset

This dataset contributes to improving language technology for Turkish speakers by enabling better understanding of tokenization challenges and supporting more robust multilingual models.

Discussion of Biases

• —Language variety: he dataset uses Standard Turkish (Türkiye Türkçesi) and may not fully represent regional or dialectal variations.
• —Script focus: Only the Latin script is used; Turkish-specific diacritics and keyboard-related variations are included as perturbations.
• —Domain coverage: Questions focus on general knowledge and may not represent domain-specific Turkish language use.
• —Question simplicity: Designed for high baseline accuracy, which may not reflect real-world task complexity.

Other Known Limitations

• —Relatively small dataset size (evaluation-only)
• —Multiple-choice format
• —Language-specific perturbations
• —Results may differ at larger model scales

Additional Information

Dataset Curators

The dataset was curated by the TokSuite research team at R3.

Licensing Information

MIT license

Citation Information

If you use this dataset in your research, please cite the TokSuite paper:

@inproceedings{toksuite2026,
  title={TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior},
  author={Altıntaş, Gül Sena and Ehghaghi, Malikeh and Lester, Brian and Liu, Fengyuan and Zhao, Wanru and Ciccone, Marco and Raffel, Colin},
  booktitle={Preprint.},
  year={2026},
  arxiv={https://arxiv.org/abs/2512.20757},
  url={TBD}
}

Paper: TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Contributions

This dataset is part of TokSuite, which includes:

• —14 language models with identical architectures but different tokenizers
• —Multilingual benchmark datasets (English, Turkish, Italian, Farsi, Chinese)
• —Comprehensive analysis of tokenization's impact on model behavior

Contact

For questions or issues related to this dataset, please refer to the TokSuite project or contact the authors of the paper.

<div align="center">

Part of the [TokSuite Project](TBD)

Understanding Tokenization's Role in Language Model Behavior

</div>