toksuite/toksuite_italian

Dataset Card for Tokenization Robustness

<!-- Provide a quick summary of the dataset. -->

<img src="toksuite-logo.png" alt="TokSuite Logo" width="250px" style="margin-left:'auto' margin-right:'auto' display:'block'"/>

TokSuite Benchmark (Italian Collection)

Dataset Description

This dataset is part of TokSuite, a comprehensive benchmark designed to measure how different tokenization strategies affect language model performance and robustness. This specific subset contains Italian language multiple-choice text completion questions with various real-world perturbations that test tokenizer robustness.

• —Curated by: R3 Research Team
• —Language(s): Italian (It)
• —License: MIT License

Dataset Summary

TokSuite addresses a fundamental challenge in language model research: understanding how tokenization choices impact model behavior in isolation. The Italian subset specifically measures model performance on canonical questions and various perturbations. Key Features:

• —40 canonical questions covering general knowledge, geography, science, and language understanding
• —Multiple perturbation types reflecting real-world text variations in Italian
• —Parallel structure with TokSuite benchmark (available in English, Turkish, Farsi, Chinese)
• —Native speaker curation ensuring linguistic authenticity

Supported Tasks

• —Multiple-Choice Question Answering: Text completion format with 4 answer choices
• —Tokenizer Robustness Evaluation: Measuring performance degradation under various text perturbations
• —Multilingual NLP Benchmarking: Evaluating language models on Italian text understanding

Languages

The dataset contains text in Italian (language code: ita_Latn / it).

Dataset Structure

Data Fields

Dataset Creation

Curation Rationale

This dataset was created to:

1. 1.Systematically evaluate how different tokenization strategies handle Italian
2. 2.Measure robustness against real-world text perturbations specific to Italian
3. 3.Support research into the impact of tokenization on language model behavior
4. 4.Provide standardized benchmarks for Italian language models

The questions were designed to be straightforward with high baseline accuracy, allowing researchers to cleanly measure performance degradation when perturbations are applied.

Source Data

Data Collection and Processing

• —Canonical Questions: 40 baseline questions created in English
• —Translation: Native Italian speakers translated questions
• —Perturbations: Each question underwent targeted perturbations designed to reflect Italian characteristics
• —Validation: Model-in-the-loop process ensured high baseline accuracy

Perturbation Categories

1. 1.Canonical The original Italian question written in standard, well-formed Italian with correct spelling, grammar, accents, capitalization, and formatting. All other perturbations are derived from this version and preserve its meaning.

1. 1.Abbreviations Words or expressions in the canonical sentence are replaced with common Italian abbreviations (e.g., titles like Dr., shortened forms such as ecc. or n.). The semantic content remains unchanged, but surface length and token boundaries are altered.

1. 1.Capitalization Capital letters are altered relative to the canonical form (e.g., sentence-level lowercasing, random capitalization, or improper casing of proper nouns). The lexical content is the same, but casing information is corrupted or inconsistent.

1. 1.Code / Language / Script Switching Italian sentences contain inserted English words or phrases (often technical terms or borrowed expressions). The script remains Latin, but language identity switches mid-sentence, reflecting realistic bilingual or mixed-language usage.

1. 1.Contractions Italian elisions and contractions are introduced or modified (e.g., l’amico, dell’acqua, all’università). Apostrophes merge words that are separate in canonical form, changing token segmentation while preserving meaning.

1. 1.Date Formats Dates are rewritten using alternative Italian or international formats (e.g., numeric dates, month-name formats, different separators). The temporal meaning is preserved, but punctuation and numeric structure vary.

1. 1.Dialects Standard Italian words or constructions are replaced with dialect-influenced variants (e.g., regional lexical or morphological forms). These versions remain interpretable to native speakers but diverge from standardized Italian orthography.

1. 1.English Keyboard Italian text is written as if typed on an English keyboard, resulting in missing or simplified accented characters (e.g., perche instead of perché). Unicode accents are dropped or normalized, stressing tokenizer handling of diacritics.

1. 1.Grammatical Errors The sentence includes plausible grammatical mistakes such as incorrect agreement, article misuse, or tense errors. The sentence remains understandable, but violates formal Italian grammar rules.

1. 1.Keyboard Proximity Errors Introduces typos caused by pressing adjacent keys on a keyboard, simulating realistic typing errors without altering intended meaning.

1. 1.Numerical Formats Numbers are rewritten using different Italian-appropriate formats (e.g., thousand separators, decimal symbols, or spacing). The numeric value is preserved while its surface representation changes.

1. 1.Orthographic Errors Spelling errors are introduced that violate standard Italian orthography (e.g., incorrect consonant doubling, wrong letter choice). These errors are visually or phonetically plausible but formally incorrect.

1. 1.Phonetic Spelling Words are spelled according to pronunciation rather than standard orthography, often resembling informal or speech-based writing. This alters character sequences while preserving phonetic identity.

1. 1.Plausible Diacritics Errors Introduces missing, incorrect, or misplaced diacritics (e.g., e vs. è, perché vs. perche), testing tokenizer sensitivity to accent marks that affect meaning.

1. 1.Similar Words Canonical words are replaced with closely related or confusable alternatives (e.g., near-synonyms or minimal lexical contrasts). The sentence remains plausible and grammatical but is lexically altered.

1. 1.Spelled-Out Forms Digits, abbreviations, or compact expressions are replaced with their fully spelled-out Italian equivalents (e.g., numerals written as words). This increases token length and changes lexical composition without changing meaning.

1. 1.Typographical Errors General typing mistakes are introduced, such as duplicated letters, missing characters, or minor corruptions. These errors are less systematic than keyboard-proximity errors and reflect careless typing.

1. 1.Web Search Query The question is rewritten in the style of an Italian web search query: function words may be dropped, word order simplified, and phrasing becomes keyword-like rather than sentence-like, while retaining the same informational intent.

Who are the source data producers?

Native Italian speakers curated and validated all questions and perturbations. The TokSuite research team at R3 designed the overall benchmark framework.

Annotations

Annotation process

Questions were manually created and translated by native speakers. Each perturbation was carefully designed to reflect authentic variations encountered in real-world Italian text processing.

Who are the annotators?

Native Italian speakers with expertise in linguistics and NLP, working as part of the TokSuite project.

Personal and Sensitive Information

The dataset contains only general knowledge questions and does not include any personal or sensitive information.

Considerations for Using the Data

Social Impact of Dataset

This dataset contributes to improving language technology for Italian speakers by enabling better understanding of tokenization challenges and supporting more robust multilingual models.

Discussion of Biases

• —Language variety:The dataset uses Standard Italian (Italiano standard) and may not fully represent regional or dialectal variations.
• —Script focus: Only the Latin script is used; accent and keyboard-related variations are included as perturbations.
• —Domain coverage: Questions focus on general knowledge and may not represent domain-specific Italian language use.
• —Question simplicity: Designed for high baseline accuracy, which may not reflect real-world task complexity

Other Known Limitations

• —Relatively small dataset size (evaluation-only)
• —Multiple-choice format
• —Language-specific perturbations
• —Results may differ at larger model scales

Additional Information

Dataset Curators

The dataset was curated by the TokSuite research team at R3.

Licensing Information

MIT license

Citation Information

If you use this dataset in your research, please cite the TokSuite paper:

@inproceedings{toksuite2026,
  title={TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior},
  author={Altıntaş, Gül Sena and Ehghaghi, Malikeh and Lester, Brian and Liu, Fengyuan and Zhao, Wanru and Ciccone, Marco and Raffel, Colin},
  booktitle={Preprint.},
  year={2026},
  arxiv={https://arxiv.org/abs/2512.20757},
  url={TBD}
}

Paper: TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Contributions

This dataset is part of TokSuite, which includes:

• —14 language models with identical architectures but different tokenizers
• —Multilingual benchmark datasets (English, Turkish, Italian, Farsi, Chinese)
• —Comprehensive analysis of tokenization's impact on model behavior

Contact

For questions or issues related to this dataset, please refer to the TokSuite project or contact the authors of the paper.

<div align="center">

Part of the [TokSuite Project](TBD)

Understanding Tokenization's Role in Language Model Behavior

</div>