Polygl0t/tokenizers

1---2dataset_info:3- config_name: portuguese4  features:5  - name: text6    dtype: string7  splits:8  - name: train9    num_bytes: 2130779939610    num_examples: 200000011  download_size: 798108240312  dataset_size: 2130779939613- config_name: bengali14  features:15  - name: text16    dtype: string17  splits:18  - name: train19    num_bytes: 1322291382920    num_examples: 200000021  download_size: 380445318522  dataset_size: 1322291382923- config_name: code24  features:25  - name: text26    dtype: string27  splits:28  - name: train29    num_bytes: 685428824230    num_examples: 97500031  download_size: 219475506332  dataset_size: 685428824233- config_name: english34  features:35  - name: text36    dtype: string37  splits:38  - name: train39    num_bytes: 1567076860640    num_examples: 200000041  download_size: 570059232542  dataset_size: 1567076860643- config_name: hindi44  features:45  - name: text46    dtype: string47  splits:48  - name: train49    num_bytes: 2257921692750    num_examples: 200000051  download_size: 619481035052  dataset_size: 2257921692753configs:54- config_name: portuguese55  data_files:56  - split: train57    path: portuguese/train-*58- config_name: bengali59  data_files:60  - split: train61    path: bengali/train-*62- config_name: code63  data_files:64  - split: train65    path: code/train-*66- config_name: english67  default: true68  data_files:69  - split: train70    path: english/train-*71- config_name: hindi72  data_files:73  - split: train74    path: hindi/train-*75language:76- hi77- bn78- en79- pt80license: other81task_categories:82- text-generation83tags:84- tokenizer85- tokenization86- english87- code88- bengali89- hindi90- portuguese91pretty_name: Polygl0t tokenizers92size_categories:93- 1M<n<10M94---95# Polygl0t Tokenizers96 97## Table of Contents98 99- [Dataset Description](#dataset-description)100  - [Dataset Summary](#dataset-summary)101  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)102  - [Languages](#languages)103- [Dataset Structure](#dataset-structure)104  - [Data Instances](#data-instances)105  - [Data Fields](#data-fields)106  - [Subsets and Splits](#subsets-and-splits)107- [Dataset Creation](#dataset-creation)108  - [Source Data](#source-data)109- [Additional Information](#additional-information)110  - [Dataset Maintainers](#dataset-maintainers)111  - [Licensing Information](#licensing-information)112  - [Citation Information](#citation-information)113  - [Acknowledgments](#acknowledgments)114  - [Contributions](#contributions)115 116## Dataset Description117 118- **Homepage:** https://huggingface.co/datasets/Polygl0t/tokenizers119- **Repository:** https://huggingface.co/datasets/Polygl0t/tokenizers120- **Point of Contact:** [Polyg0t](mailto:kluge@uni-bonn.de)121 122### Dataset Summary123 124This dataset contains several subsets for training multilingual tokenizers. Every subset possesses a collection of curated text samples in different languages. 125 126### Supported Tasks and Leaderboards127 128This dataset can be used for the task of text generation, specifically for training and evaluating tokenizers in multiple languages.129 130### Languages131 132Hindi, Bengali, English, Portuguese, and Code (a mixture of 36 programming languages).133 134<details>135<summary><b>All programming languages</b></summary>136 <code>137  fortran, jupyter, cpp, solidity, python, cmake, assembly, ruby, perl, lua, typescript, c, java, html, powershell, php, haskell, shell, scala, sql, visual_basic, ada, julia, markdown, batchfile, rust, cuda, json, kotlin, go, r, javascript, pascal, yaml, css, c_sharp138 </code>139</details>140 141## Dataset Structure142 143### Data Instances144 145The dataset consists of the following features:146 147- **text:** a string of text in the respective language of the subset.148 149### Data Fields150 151```json152{153  "text": "Olá, como vai você?"154}155```156 157### Subsets and Splits158 159The dataset includes the following subsets:160 161- **Portuguese:** This subset contains 2,000,000 text samples in Portuguese.162- **Hindi:** This subset contains 2,000,000 text samples in Hindi.163- **Bengali:** This subset contains 2,000,000 text samples in Bengali164- **English:** This subset contains 2,000,000 text samples in English.165- **Code:** This subset contains 975,000 text samples in various programming languages.166 167The `txt` files (e.g., [`hindi_test.txt`](hindi_test.txt)) are for testing/evaluation purposes.168 169### Dataset Creation170 171### Source Data172 173- **Bengali:** The Bengali text samples were sourced from [Polygl0t/gigakriya-v1](https://huggingface.co/datasets/Polygl0t/gigakriya-v1).174- **English:** The English text samples were sourced from [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu).175- **Hindi:** The Hindi text samples were sourced from [Polygl0t/gigalekh-v1](https://huggingface.co/datasets/Polygl0t/gigalekh-v1).176- **Portuguese:** The Portuguese text samples were sourced from [Polygl0t/gigaverbo-v2](https://huggingface.co/datasets/Polygl0t/gigaverbo-v2).177- **Code:** The code samples were sourced from [bigcode/starcoderdata](https://huggingface.co/datasets/bigcode/starcoderdata).178 179## Additional Information180 181### Dataset Maintainers182 183- [Nicholas Kluge Corrêa](mailto:kluge@uni-bonn.de).184- [Shiza Fatimah](mailto:shizafatimah15@gmail.com).185- [Aniket Sen](mailto:sen@hiskp.uni-bonn.de).186 187### Licensing Information188 189Please refer to the individual licenses of the source datasets used to create this corpus, as listed in the "Source Data" section above. The combined dataset does not have a single unified license, and users should ensure compliance with the terms of each source dataset when utilizing this corpus.190 191### Citation Information192 193```latex194@misc{correa2026tucano2cool,195      title={{Tucano 2 Cool: Better Open Source LLMs for Portuguese}}, 196      author={Nicholas Kluge Corr{\^e}a and Aniket Sen and Shiza Fatimah and Sophia Falk and Lennard Landgraf and Julia Kastner and Lucie Flek},197      year={2026},198      eprint={2603.03543},199      archivePrefix={arXiv},200      primaryClass={cs.CL},201      url={https://arxiv.org/abs/2603.03543}, 202}203 204@misc{shiza2026lilmoo,205      title={{Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi}}, 206      author={Shiza Fatimah and Aniket Sen and Sophia Falk and Florian Mai and Lucie Flek and Nicholas Kluge Corr{\^e}a},207      year={2026},208      eprint={2603.03508},209      archivePrefix={arXiv},210      primaryClass={cs.CL},211      url={https://arxiv.org/abs/2603.03508}, 212}213 214@misc{fatimah2026liltii,215  title={{LilTii: A 0.6B Bengali Language Model that Outperforms Qwen}},216  author={Shiza Fatimah and Aniket Sen and Sophia Falk and Florian Mai and Lucie Flek and Nicholas Kluge Corr{\^e}a},217  year={2026},218  howpublished={\url{https://hf.co/blog/Polygl0t/liltii}}219}220```221 222### Acknowledgments223 224Polyglot is a project funded by the Federal Ministry of Education and Research (BMBF) and the Ministry of Culture and Science of the State of North Rhine-Westphalia (MWK) as part of TRA Sustainable Futures (University of Bonn) and the Excellence Strategy of the federal and state governments.225 226We also gratefully acknowledge the granted access to the [Marvin cluster](https://www.hpc.uni-bonn.de/en/systems/marvin) hosted by [University of Bonn](https://www.uni-bonn.de/en) along with the support provided by its High Performance Computing & Analytics Lab.227 228 229### Contributions230 231If you want to contribute, contact us at [polyglot@uni-bonn.de](mailto:polyglot@uni-bonn.de)!