Salesforce/wikitext

1---2annotations_creators:3- no-annotation4language_creators:5- crowdsourced6language:7- en8license:9- cc-by-sa-3.010- gfdl11multilinguality:12- monolingual13size_categories:14- 1M<n<10M15source_datasets:16- original17task_categories:18- text-generation19- fill-mask20task_ids:21- language-modeling22- masked-language-modeling23paperswithcode_id: wikitext-224pretty_name: WikiText25dataset_info:26- config_name: wikitext-103-raw-v127  features:28  - name: text29    dtype: string30  splits:31  - name: test32    num_bytes: 130508833    num_examples: 435834  - name: train35    num_bytes: 54650094936    num_examples: 180135037  - name: validation38    num_bytes: 115928839    num_examples: 376040  download_size: 31546639741  dataset_size: 54896532542- config_name: wikitext-103-v143  features:44  - name: text45    dtype: string46  splits:47  - name: test48    num_bytes: 129557549    num_examples: 435850  - name: train51    num_bytes: 54514191552    num_examples: 180135053  - name: validation54    num_bytes: 115475155    num_examples: 376056  download_size: 31309383857  dataset_size: 54759224158- config_name: wikitext-2-raw-v159  features:60  - name: text61    dtype: string62  splits:63  - name: test64    num_bytes: 130508865    num_examples: 435866  - name: train67    num_bytes: 1106171768    num_examples: 3671869  - name: validation70    num_bytes: 115928871    num_examples: 376072  download_size: 774736273  dataset_size: 1352609374- config_name: wikitext-2-v175  features:76  - name: text77    dtype: string78  splits:79  - name: test80    num_bytes: 127094781    num_examples: 435882  - name: train83    num_bytes: 1091811884    num_examples: 3671885  - name: validation86    num_bytes: 113412387    num_examples: 376088  download_size: 737128289  dataset_size: 1332318890configs:91- config_name: wikitext-103-raw-v192  data_files:93  - split: test94    path: wikitext-103-raw-v1/test-*95  - split: train96    path: wikitext-103-raw-v1/train-*97  - split: validation98    path: wikitext-103-raw-v1/validation-*99- config_name: wikitext-103-v1100  data_files:101  - split: test102    path: wikitext-103-v1/test-*103  - split: train104    path: wikitext-103-v1/train-*105  - split: validation106    path: wikitext-103-v1/validation-*107- config_name: wikitext-2-raw-v1108  data_files:109  - split: test110    path: wikitext-2-raw-v1/test-*111  - split: train112    path: wikitext-2-raw-v1/train-*113  - split: validation114    path: wikitext-2-raw-v1/validation-*115- config_name: wikitext-2-v1116  data_files:117  - split: test118    path: wikitext-2-v1/test-*119  - split: train120    path: wikitext-2-v1/train-*121  - split: validation122    path: wikitext-2-v1/validation-*123---124 125# Dataset Card for "wikitext"126 127## Table of Contents128- [Dataset Description](#dataset-description)129  - [Dataset Summary](#dataset-summary)130  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)131  - [Languages](#languages)132- [Dataset Structure](#dataset-structure)133  - [Data Instances](#data-instances)134  - [Data Fields](#data-fields)135  - [Data Splits](#data-splits)136- [Dataset Creation](#dataset-creation)137  - [Curation Rationale](#curation-rationale)138  - [Source Data](#source-data)139  - [Annotations](#annotations)140  - [Personal and Sensitive Information](#personal-and-sensitive-information)141- [Considerations for Using the Data](#considerations-for-using-the-data)142  - [Social Impact of Dataset](#social-impact-of-dataset)143  - [Discussion of Biases](#discussion-of-biases)144  - [Other Known Limitations](#other-known-limitations)145- [Additional Information](#additional-information)146  - [Dataset Curators](#dataset-curators)147  - [Licensing Information](#licensing-information)148  - [Citation Information](#citation-information)149  - [Contributions](#contributions)150 151## Dataset Description152 153- **Homepage:** [https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/)154- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)155- **Paper:** [Pointer Sentinel Mixture Models](https://arxiv.org/abs/1609.07843)156- **Point of Contact:** [Stephen Merity](mailto:smerity@salesforce.com)157- **Size of downloaded dataset files:** 391.41 MB158- **Size of the generated dataset:** 1.12 GB159- **Total amount of disk used:** 1.52 GB160 161### Dataset Summary162 163 The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified164 Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License.165 166Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over167110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation168and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models169that can take advantage of long term dependencies.170 171Each subset comes in two different variants:172- Raw (for character level work) contain the raw tokens, before the addition of the <unk> (unknown) tokens.173- Non-raw (for word level work) contain only the tokens in their vocabulary (wiki.train.tokens, wiki.valid.tokens, and wiki.test.tokens).174  The out-of-vocabulary tokens have been replaced with the the <unk> token.175 176 177### Supported Tasks and Leaderboards178 179[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)180 181### Languages182 183[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)184 185## Dataset Structure186 187### Data Instances188 189#### wikitext-103-raw-v1190 191- **Size of downloaded dataset files:** 191.98 MB192- **Size of the generated dataset:** 549.42 MB193- **Total amount of disk used:** 741.41 MB194 195An example of 'validation' looks as follows.196```197This example was too long and was cropped:198 199{200    "text": "\" The gold dollar or gold one @-@ dollar piece was a coin struck as a regular issue by the United States Bureau of the Mint from..."201}202```203 204#### wikitext-103-v1205 206- **Size of downloaded dataset files:** 190.23 MB207- **Size of the generated dataset:** 548.05 MB208- **Total amount of disk used:** 738.27 MB209 210An example of 'train' looks as follows.211```212This example was too long and was cropped:213 214{215    "text": "\" Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to..."216}217```218 219#### wikitext-2-raw-v1220 221- **Size of downloaded dataset files:** 4.72 MB222- **Size of the generated dataset:** 13.54 MB223- **Total amount of disk used:** 18.26 MB224 225An example of 'train' looks as follows.226```227This example was too long and was cropped:228 229{230    "text": "\" The Sinclair Scientific Programmable was introduced in 1975 , with the same case as the Sinclair Oxford . It was larger than t..."231}232```233 234#### wikitext-2-v1235 236- **Size of downloaded dataset files:** 4.48 MB237- **Size of the generated dataset:** 13.34 MB238- **Total amount of disk used:** 17.82 MB239 240An example of 'train' looks as follows.241```242This example was too long and was cropped:243 244{245    "text": "\" Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to..."246}247```248 249### Data Fields250 251The data fields are the same among all splits.252 253#### wikitext-103-raw-v1254- `text`: a `string` feature.255 256#### wikitext-103-v1257- `text`: a `string` feature.258 259#### wikitext-2-raw-v1260- `text`: a `string` feature.261 262#### wikitext-2-v1263- `text`: a `string` feature.264 265### Data Splits266 267|       name        | train |validation|test|268|-------------------|------:|---------:|---:|269|wikitext-103-raw-v1|1801350|      3760|4358|270|wikitext-103-v1    |1801350|      3760|4358|271|wikitext-2-raw-v1  |  36718|      3760|4358|272|wikitext-2-v1      |  36718|      3760|4358|273 274## Dataset Creation275 276### Curation Rationale277 278[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)279 280### Source Data281 282#### Initial Data Collection and Normalization283 284[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)285 286#### Who are the source language producers?287 288[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)289 290### Annotations291 292#### Annotation process293 294[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)295 296#### Who are the annotators?297 298[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)299 300### Personal and Sensitive Information301 302[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)303 304## Considerations for Using the Data305 306### Social Impact of Dataset307 308[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)309 310### Discussion of Biases311 312[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)313 314### Other Known Limitations315 316[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)317 318## Additional Information319 320### Dataset Curators321 322[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)323 324### Licensing Information325 326The dataset is available under the [Creative Commons Attribution-ShareAlike License (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/).327 328### Citation Information329 330```331@misc{merity2016pointer,332      title={Pointer Sentinel Mixture Models},333      author={Stephen Merity and Caiming Xiong and James Bradbury and Richard Socher},334      year={2016},335      eprint={1609.07843},336      archivePrefix={arXiv},337      primaryClass={cs.CL}338}339```340 341 342### Contributions343 344Thanks to [@thomwolf](https://github.com/thomwolf), [@lewtun](https://github.com/lewtun), [@patrickvonplaten](https://github.com/patrickvonplaten), [@mariamabarham](https://github.com/mariamabarham) for adding this dataset.