bigscience/xP3mt

1---2annotations_creators:3- expert-generated4- crowdsourced5language:6- ak7- ar8- as9- bm10- bn11- ca12- code13- en14- es15- eu16- fon17- fr18- gu19- hi20- id21- ig22- ki23- kn24- lg25- ln26- ml27- mr28- ne29- nso30- ny31- or32- pa33- pt34- rn35- rw36- sn37- st38- sw39- ta40- te41- tn42- ts43- tum44- tw45- ur46- vi47- wo48- xh49- yo50- zh51- zu52programming_language: 53- C54- C++55- C#56- Go57- Java58- JavaScript59- Lua60- PHP61- Python62- Ruby63- Rust64- Scala65- TypeScript66license:67- apache-2.068multilinguality:69- multilingual70pretty_name: xP371size_categories:72- 100M<n<1B73task_categories:74- other75---76 77# Dataset Card for xP378 79## Table of Contents80- [Table of Contents](#table-of-contents)81- [Dataset Description](#dataset-description)82  - [Dataset Summary](#dataset-summary)83  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)84  - [Languages](#languages)85- [Dataset Structure](#dataset-structure)86  - [Data Instances](#data-instances)87  - [Data Fields](#data-fields)88  - [Data Splits](#data-splits)89- [Dataset Creation](#dataset-creation)90  - [Curation Rationale](#curation-rationale)91  - [Source Data](#source-data)92  - [Annotations](#annotations)93- [Additional Information](#additional-information)94  - [Licensing Information](#licensing-information)95  - [Citation Information](#citation-information)96  - [Contributions](#contributions)97 98## Dataset Description99 100- **Repository:** https://github.com/bigscience-workshop/xmtf101- **Paper:** [Crosslingual Generalization through Multitask Finetuning](https://arxiv.org/abs/2211.01786)102- **Point of Contact:** [Niklas Muennighoff](mailto:niklas@hf.co)103 104### Dataset Summary105 106> xP3 (Crosslingual Public Pool of Prompts) is a collection of prompts & datasets across 46 of languages & 16 NLP tasks. It is used for the training of BLOOMZ and mT0, multilingual language models capable of following human instructions in dozens of languages zero-shot.107 108- **Creation:** The dataset can be recreated using instructions available [here](https://github.com/bigscience-workshop/xmtf#create-xp3). We provide this version to save processing time and ease reproducibility.109- **Languages:** 46 (Can be extended by [recreating with more splits](https://github.com/bigscience-workshop/xmtf#create-xp3))110- **xP3 Dataset Family:**111 112<table>113  <tr>114<th>Name</th>115<th>Explanation</th>116<th>Example models</th>117</tr>118<tr>119<td><a href=https://huggingface.co/datasets/Muennighoff/xP3x>xP3x</a></t> 120<td>Mixture of 17 tasks in 277 languages with English prompts</td>121<td>WIP - Join us at Project Aya @<a href=https://cohere.for.ai/>C4AI</a> to help!</td>122</tr>123<tr>124<td><a href=https://huggingface.co/datasets/bigscience/xP3>xP3</a></t> 125<td>Mixture of 13 training tasks in 46 languages with English prompts</td>126<td><a href=https://huggingface.co/bigscience/bloomz>bloomz</a> & <a href=https://huggingface.co/bigscience/mt0-xxl>mt0-xxl</a></td>127</tr>128<tr>129<td><a href=https://huggingface.co/datasets/bigscience/xP3mt>xP3mt</a></t> 130<td>Mixture of 13 training tasks in 46 languages with prompts in 20 languages (machine-translated from English)</td>131<td><a href=https://huggingface.co/bigscience/bloomz-mt>bloomz-mt</a> & <a href=https://huggingface.co/bigscience/mt0-xxl-mt>mt0-xxl-mt</a></td>132</tr>133<tr>134<td><a href=https://huggingface.co/datasets/bigscience/xP3all>xP3all</a></t> 135<td>xP3 + evaluation datasets adding an additional 3 tasks for a total of 16 tasks in 46 languages with English prompts</td>136<td></td>137</tr>138<tr>139<td><a href=https://huggingface.co/datasets/bigscience/xP3megds>xP3megds</a></t> 140<td><a href=https://github.com/bigscience-workshop/Megatron-DeepSpeed>Megatron-DeepSpeed</a> processed version of xP3</td>141<td><a href=https://huggingface.co/bigscience/bloomz>bloomz</a></td>142</tr>143<tr>144<td><a href=https://huggingface.co/datasets/Muennighoff/P3>P3</a></t> 145<td>Repreprocessed version of the English-only <a href=https://huggingface.co/datasets/bigscience/P3>P3</a> with 8 training tasks</td>146<td><a href=https://huggingface.co/bigscience/bloomz-p3>bloomz-p3</a> & <a href=https://huggingface.co/bigscience/mt0-xxl-p3>mt0-xxl-p3</a></td>147</tr>148</table>149 150## Dataset Structure151 152### Data Instances153 154An example of "train" looks as follows:155```json156{157"inputs": "Oración 1: Fue académico en literatura metafísica, teología y ciencias clásicas.\Oración 2: Fue académico en literatura metafísica, teología y ciencia clásica.\nPregunta: ¿La oración 1 parafrasea la oración 2? ¿Si o no?",158"targets": "Sí" 159}160```161 162### Data Fields163 164The data fields are the same among all splits:165- `inputs`: the natural language input fed to the model166- `targets`: the natural language target that the model has to generate167 168### Data Splits169 170The below table summarizes sizes per language (computed from the `merged_{lang}.jsonl` files). Due to languages like `tw` only being single sentence translation samples from Flores, their byte percentage is significantly lower than their sample percentage. We machine-translated prompts for monolingual datasets, thus languages with only crosslingual datasets (e.g. Translation) do not have non-English prompts. Languages without non-English prompts are equivalent to [xP3](https://huggingface.co/datasets/bigscience/xP3).171 172|Language|Kilobytes|%|Samples|%|Non-English prompts|173|--------|------:|-:|---:|-:|-:|174|tw|106288|0.11|265071|0.33| |175|bm|107056|0.11|265180|0.33| |176|ak|108096|0.11|265071|0.33| |177|ca|110608|0.11|271191|0.34| |178|eu|113008|0.12|281199|0.35| |179|fon|113072|0.12|265063|0.33| |180|st|114080|0.12|265063|0.33| |181|ki|115040|0.12|265180|0.33| |182|tum|116032|0.12|265063|0.33| |183|wo|122560|0.13|365063|0.46| |184|ln|126304|0.13|365060|0.46| |185|as|156256|0.16|265063|0.33| |186|or|161472|0.17|265063|0.33| |187|kn|165456|0.17|265063|0.33| |188|ml|175040|0.18|265864|0.33| |189|rn|192992|0.2|318189|0.4| |190|nso|229712|0.24|915051|1.14| |191|tn|235536|0.24|915054|1.14| |192|lg|235936|0.24|915021|1.14| |193|rw|249360|0.26|915043|1.14| |194|ts|250256|0.26|915044|1.14| |195|sn|252496|0.26|865056|1.08| |196|xh|254672|0.26|915058|1.14| |197|zu|263712|0.27|915061|1.14| |198|ny|272128|0.28|915063|1.14| |199|ig|325440|0.33|950097|1.19|✅|200|yo|339664|0.35|913021|1.14|✅|201|ne|398144|0.41|315754|0.39|✅|202|pa|529632|0.55|339210|0.42|✅|203|sw|561392|0.58|1114439|1.39|✅|204|gu|566576|0.58|347499|0.43|✅|205|mr|674000|0.69|417269|0.52|✅|206|bn|854864|0.88|428725|0.54|✅|207|ta|943440|0.97|410633|0.51|✅|208|te|1384016|1.42|573354|0.72|✅|209|ur|1944416|2.0|855756|1.07|✅|210|vi|3113184|3.2|1667306|2.08|✅|211|code|4330752|4.46|2707724|3.38| |212|hi|4469712|4.6|1543441|1.93|✅|213|id|4538768|4.67|2582272|3.22|✅|214|zh|4604112|4.74|3571636|4.46|✅|215|ar|4703968|4.84|2148970|2.68|✅|216|fr|5558912|5.72|5055942|6.31|✅|217|pt|6130016|6.31|3562772|4.45|✅|218|es|7579424|7.8|5151349|6.43|✅|219|en|39252528|40.4|32740750|40.87| |220|total|97150128|100.0|80100816|100.0|✅|221 222## Dataset Creation223 224### Source Data225 226#### Training datasets227 228- Code Miscellaneous229  - [CodeComplex](https://huggingface.co/datasets/codeparrot/codecomplex)230  - [Docstring Corpus](https://huggingface.co/datasets/teven/code_docstring_corpus)231  - [GreatCode](https://huggingface.co/datasets/great_code)232  - [State Changes](https://huggingface.co/datasets/Fraser/python-state-changes)233- Closed-book QA234  - [Hotpot QA](https://huggingface.co/datasets/hotpot_qa)235  - [Trivia QA](https://huggingface.co/datasets/trivia_qa)236  - [Web Questions](https://huggingface.co/datasets/web_questions)237  - [Wiki QA](https://huggingface.co/datasets/wiki_qa)  238- Extractive QA239  - [Adversarial QA](https://huggingface.co/datasets/adversarial_qa)240  - [CMRC2018](https://huggingface.co/datasets/cmrc2018)241  - [DRCD](https://huggingface.co/datasets/clue)242  - [DuoRC](https://huggingface.co/datasets/duorc)243  - [MLQA](https://huggingface.co/datasets/mlqa)      244  - [Quoref](https://huggingface.co/datasets/quoref)245  - [ReCoRD](https://huggingface.co/datasets/super_glue)  246  - [ROPES](https://huggingface.co/datasets/ropes)247  - [SQuAD v2](https://huggingface.co/datasets/squad_v2)248  - [xQuAD](https://huggingface.co/datasets/xquad)249  - TyDI QA250    - [Primary](https://huggingface.co/datasets/khalidalt/tydiqa-primary)251    - [Goldp](https://huggingface.co/datasets/khalidalt/tydiqa-goldp)252- Multiple-Choice QA253  - [ARC](https://huggingface.co/datasets/ai2_arc)254  - [C3](https://huggingface.co/datasets/c3)  255  - [CoS-E](https://huggingface.co/datasets/cos_e)256  - [Cosmos](https://huggingface.co/datasets/cosmos)257  - [DREAM](https://huggingface.co/datasets/dream)258  - [MultiRC](https://huggingface.co/datasets/super_glue)259  - [OpenBookQA](https://huggingface.co/datasets/openbookqa)260  - [PiQA](https://huggingface.co/datasets/piqa)  261  - [QUAIL](https://huggingface.co/datasets/quail)262  - [QuaRel](https://huggingface.co/datasets/quarel)263  - [QuaRTz](https://huggingface.co/datasets/quartz)264  - [QASC](https://huggingface.co/datasets/qasc)265  - [RACE](https://huggingface.co/datasets/race)266  - [SciQ](https://huggingface.co/datasets/sciq)    267  - [Social IQA](https://huggingface.co/datasets/social_i_qa)268  - [Wiki Hop](https://huggingface.co/datasets/wiki_hop)269  - [WiQA](https://huggingface.co/datasets/wiqa)  270- Paraphrase Identification271  - [MRPC](https://huggingface.co/datasets/super_glue)272  - [PAWS](https://huggingface.co/datasets/paws)273  - [PAWS-X](https://huggingface.co/datasets/paws-x)  274  - [QQP](https://huggingface.co/datasets/qqp)  275- Program Synthesis276  - [APPS](https://huggingface.co/datasets/codeparrot/apps)277  - [CodeContests](https://huggingface.co/datasets/teven/code_contests)278  - [JupyterCodePairs](https://huggingface.co/datasets/codeparrot/github-jupyter-text-code-pairs)279  - [MBPP](https://huggingface.co/datasets/Muennighoff/mbpp)280  - [NeuralCodeSearch](https://huggingface.co/datasets/neural_code_search)281  - [XLCoST](https://huggingface.co/datasets/codeparrot/xlcost-text-to-code)  282- Structure-to-text283  - [Common Gen](https://huggingface.co/datasets/common_gen)284  - [Wiki Bio](https://huggingface.co/datasets/wiki_bio)285- Sentiment286  - [Amazon](https://huggingface.co/datasets/amazon_polarity)287  - [App Reviews](https://huggingface.co/datasets/app_reviews)288  - [IMDB](https://huggingface.co/datasets/imdb)289  - [Rotten Tomatoes](https://huggingface.co/datasets/rotten_tomatoes)290  - [Yelp](https://huggingface.co/datasets/yelp_review_full)291- Simplification292  - [BiSECT](https://huggingface.co/datasets/GEM/BiSECT)293- Summarization294  - [CNN Daily Mail](https://huggingface.co/datasets/cnn_dailymail)295  - [Gigaword](https://huggingface.co/datasets/gigaword)296  - [MultiNews](https://huggingface.co/datasets/multi_news)297  - [SamSum](https://huggingface.co/datasets/samsum)298  - [Wiki-Lingua](https://huggingface.co/datasets/GEM/wiki_lingua)299  - [XLSum](https://huggingface.co/datasets/GEM/xlsum)300  - [XSum](https://huggingface.co/datasets/xsum)301- Topic Classification302  - [AG News](https://huggingface.co/datasets/ag_news)303  - [DBPedia](https://huggingface.co/datasets/dbpedia_14)304  - [TNEWS](https://huggingface.co/datasets/clue)  305  - [TREC](https://huggingface.co/datasets/trec)306  - [CSL](https://huggingface.co/datasets/clue) 307- Translation308  - [Flores-200](https://huggingface.co/datasets/Muennighoff/flores200)309  - [Tatoeba](https://huggingface.co/datasets/Helsinki-NLP/tatoeba_mt)310- Word Sense disambiguation311  - [WiC](https://huggingface.co/datasets/super_glue)312  - [XL-WiC](https://huggingface.co/datasets/pasinit/xlwic)313  314#### Evaluation datasets (included in [xP3all](https://huggingface.co/datasets/bigscience/xP3all) except for NLI & HumanEval)315  316- Natural Language Inference (NLI)317  - [ANLI](https://huggingface.co/datasets/anli)318  - [CB](https://huggingface.co/datasets/super_glue)319  - [RTE](https://huggingface.co/datasets/super_glue)320  - [XNLI](https://huggingface.co/datasets/xnli)321- Coreference Resolution322  - [Winogrande](https://huggingface.co/datasets/winogrande)323  - [XWinograd](https://huggingface.co/datasets/Muennighoff/xwinograd)324- Program Synthesis325  - [HumanEval](https://huggingface.co/datasets/openai_humaneval)326- Sentence Completion327  - [COPA](https://huggingface.co/datasets/super_glue)328  - [Story Cloze](https://huggingface.co/datasets/story_cloze)329  - [XCOPA](https://huggingface.co/datasets/xcopa)  330  - [XStoryCloze](https://huggingface.co/datasets/Muennighoff/xstory_cloze)331 332## Additional Information333 334### Licensing Information335 336The dataset is released under Apache 2.0.337 338### Citation Information339 340```bibtex341@misc{muennighoff2022crosslingual,342      title={Crosslingual Generalization through Multitask Finetuning}, 343      author={Niklas Muennighoff and Thomas Wang and Lintang Sutawika and Adam Roberts and Stella Biderman and Teven Le Scao and M Saiful Bari and Sheng Shen and Zheng-Xin Yong and Hailey Schoelkopf and Xiangru Tang and Dragomir Radev and Alham Fikri Aji and Khalid Almubarak and Samuel Albanie and Zaid Alyafeai and Albert Webson and Edward Raff and Colin Raffel},344      year={2022},345      eprint={2211.01786},346      archivePrefix={arXiv},347      primaryClass={cs.CL}348}349```350 351### Contributions352 353Thanks to the contributors of [promptsource](https://github.com/bigscience-workshop/promptsource/graphs/contributors) for adding many prompts used in this dataset.