toksuite/toksuite_turkish
Dataset Card for Tokenization Robustness TokSuite Benchmark (Turkish Collection) Dataset Description This dataset is part of TokSuite, a comprehensive benchmark designed to measure how different tokenization strategies affect language model performance and robustness. This specific subset contains Turkish language multiple-choice text completion questions with various real-world perturbations that test tokenizer robustness. Curated by: R3… See the full description on the dataset page: https://huggingface.co/datasets/toksuite/toksuite_turkish.
Updated the examples
Updated paths
Delete tokenizer_robustness_completion_turkish_historical_spelling
Delete tokenizer_robustness_completion_turkish_space_additions_for_natural_split
Delete tokenizer_robustness_completion_turkish_synonyms
Delete tokenizer_robustness_completion_turkish_word_spacing_zero-width_characters_extra_space
Rename tokenizer_robustness_completion_turkish_word_reordering/test-00000-of-00001.parquet to toksuite_turkish_word_reordering/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_web_search_query/test-00000-of-00001.parquet to toksuite_turkish_web_search_query/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_typographical_errors/test-00000-of-00001.parquet to toksuite_turkish_typographical_errors/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_spelled_out/test-00000-of-00001.parquet to toksuite_turkish_spelled_out/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_similar_words/test-00000-of-00001.parquet to toksuite_turkish_similar_words/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_orthographic_errors/test-00000-of-00001.parquet to toksuite_turkish_orthographic_errors/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_keyboard_proximity_errors/test-00000-of-00001.parquet to toksuite_turkish_keyboard_proximity_errors/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_inflections/test-00000-of-00001.parquet to toksuite_turkish_inflections/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_grammatical_errors/test-00000-of-00001.parquet to toksuite_turkish_grammatical_errors/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_equivalent_expressions/test-00000-of-00001.parquet to toksuite_turkish_equivalent_expressions/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_english_keyboard/test-00000-of-00001.parquet to toksuite_turkish_english_keyboard/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_dialects/test-00000-of-00001.parquet to toksuite_turkish_dialects/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_derivations/test-00000-of-00001.parquet to toksuite_turkish_derivations/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_date_formats/test-00000-of-00001.parquet to toksuite_turkish_date_formats/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_colloquial/test-00000-of-00001.parquet to toksuite_turkish_colloquial/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_code_language_script_switching/test-00000-of-00001.parquet to toksuite_turkish_code_language_script_switching/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_turkish_canonical/test-00000-of-00001.parquet to toksuite_turkish_canonical/test-00000-of-00001.parquet
Delete tokenizer_robustness_completion_turkish_cannonical
Delete tokenizer_robustness_completion_turkish_borrowing
Delete tokenizer_robustness_completion_turkish_
Update README.md
Update README.md
Update README.md
Update README.md
Update README.md
Update README.md
Upload toksuite-logo.png
Uploading tokenizer_robustness_completion_turkish_word_reordering subset
Uploading tokenizer_robustness_completion_turkish_web_search_query subset
Uploading tokenizer_robustness_completion_turkish_typographical_errors subset
Uploading tokenizer_robustness_completion_turkish_spelled_out subset
Uploading tokenizer_robustness_completion_turkish_similar_words subset
Uploading tokenizer_robustness_completion_turkish_orthographic_errors subset
Uploading tokenizer_robustness_completion_turkish_keyboard_proximity_errors subset
Uploading tokenizer_robustness_completion_turkish_inflections subset
Uploading tokenizer_robustness_completion_turkish_grammatical_errors subset
Uploading tokenizer_robustness_completion_turkish_equivalent_expressions subset
Uploading tokenizer_robustness_completion_turkish_english_keyboard subset
Uploading tokenizer_robustness_completion_turkish_dialects subset
Uploading tokenizer_robustness_completion_turkish_derivations subset
Uploading tokenizer_robustness_completion_turkish_date_formats subset
Uploading tokenizer_robustness_completion_turkish_colloquial subset
Uploading tokenizer_robustness_completion_turkish_code_language_script_switching subset
Uploading tokenizer_robustness_completion_turkish_canonical subset