toksuite/toksuite_general
Dataset Card for Tokenization Robustness TokSuite Bonus Benchmarks (General Collection) This is a bonus TokSuite dataset containing a small set of high-signal examples that highlight surface-form variations known to affect tokenization robustness. It includes canonical questions alongside perturbations such as abbreviations, character deletion, currency symbols, diverse date formats, and unusual formatting. These examples focus on tokenization challenges that… See the full description on the dataset page: https://huggingface.co/datasets/toksuite/toksuite_general.
Updated the paths to simplified name
Rename tokenizer_robustness_completion_general_date_formats/test-00000-of-00001.parquet to toksuite_general_date_formats/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_general_unusual_formatting/test-00000-of-00001.parquet to toksuite_general_unusual_formatting/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_general_currency_symbol/test-00000-of-00001.parquet to toksuite_general_currency_symbol/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_general_character_deletion/test-00000-of-00001.parquet to toksuite_general_character_deletion/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_general_canonical/test-00000-of-00001.parquet to toksuite_general_canonical/test-00000-of-00001.parquet
Rename tokenizer_robustness_completion_general_abbreviations/test-00000-of-00001.parquet to toksuite_general_abbreviations/test-00000-of-00001.parquet
Update README.md
Update README.md
Update README.md
Update README.md
Update README.md
Update README.md
Upload toksuite-logo.png
Uploading tokenizer_robustness_completion_general_unusual_formatting subset
Uploading tokenizer_robustness_completion_general_date_formats subset
Uploading tokenizer_robustness_completion_general_currency_symbol subset
Uploading tokenizer_robustness_completion_general_character_deletion subset
Uploading tokenizer_robustness_completion_general_canonical subset
Uploading tokenizer_robustness_completion_general_abbreviations subset
Upload README.md with huggingface_hub
initial commit