Datasetpublic

Polygl0t/tokenizers

Polygl0t Tokenizers Dataset Summary This dataset contains several subsets for training multilingual tokenizers. Every subset possesses a collection of curated text samples in different languages. Supported Tasks and Leaderboards This dataset can be used for the task of text generation, specifically for training and evaluating tokenizers in multiple languages. Languages Hindi, Bengali, English, Portuguese, and Code (a mixture of 36… See the full description on the dataset page: https://huggingface.co/datasets/Polygl0t/tokenizers.

sourceHugging Faceotherupdated 7mo agoView on Hugging Face

0likes462downloads

root

dirbengali/

dircode/

direnglish/

dirhindi/

dirportuguese/

file.gitattributes2 KBdownload

filebengali_test.txt255 KBdownload

filecode_test.txt20 KBdownload

fileenglish_test.txt90 KBdownload

filehindi_test.txt198 KBdownload

fileportuguese_test.txt93 KBdownload

fileREADME.md7 KBdownload

Polygl0t/tokenizers · main · files are served by the source, never re-hosted here