Datasetpublic

toksuite/toksuite_farsi

Dataset Card for Tokenization Robustness TokSuite Benchmark (Farsi Collection) Dataset Description This dataset is part of TokSuite, a comprehensive benchmark designed to measure how different tokenization strategies affect language model performance and robustness. This specific subset contains Farsi (Persian) language multiple-choice text completion questions with various real-world perturbations that test tokenizer robustness. Curated by: R3… See the full description on the dataset page: https://huggingface.co/datasets/toksuite/toksuite_farsi.

sourceHugging Facemitupdated 8mo agoView on Hugging Face

0likes134downloads

root

revisionmain 1.0.3

dirtoksuite_farsi_arabic_keyboard_for_farsi/

dirtoksuite_farsi_canonical/

dirtoksuite_farsi_code_language_script_switching/

dirtoksuite_farsi_colloquial/

dirtoksuite_farsi_dialects/

dirtoksuite_farsi_equivalent_expressions/

dirtoksuite_farsi_keyboard_proximity_errors/

dirtoksuite_farsi_number_romanization/