EleutherAI/wikitext_document_level

1---2configs:3- config_name: wikitext-103-raw-v14  data_files:5  - split: train6    path: wikitext-103-raw-v1/*-train.parquet7  - split: validation8    path: wikitext-103-raw-v1/*-validation.parquet9  - split: test10    path: wikitext-103-raw-v1/*-test.parquet11- config_name: wikitext-103-v112  data_files:13  - split: train14    path: wikitext-103-v1/*-train.parquet15  - split: validation16    path: wikitext-103-v1/*-validation.parquet17  - split: test18    path: wikitext-103-v1/*-test.parquet19- config_name: wikitext-2-raw-v120  data_files:21  - split: train22    path: wikitext-2-raw-v1/*-train.parquet23  - split: validation24    path: wikitext-2-raw-v1/*-validation.parquet25  - split: test26    path: wikitext-2-raw-v1/*-test.parquet27- config_name: wikitext-2-v128  data_files:29  - split: train30    path: wikitext-2-v1/*-train.parquet31  - split: validation32    path: wikitext-2-v1/*-validation.parquet33  - split: test34    path: wikitext-2-v1/*-test.parquet35license: cc-by-sa-3.036---37 38 39# Wikitext Document Level40 41This is a modified version of [https://huggingface.co/datasets/wikitext](https://huggingface.co/datasets/wikitext) that returns Wiki pages instead of Wiki text line-by-line. The original readme is contained below.42 43# Dataset Card for "wikitext"44 45## Table of Contents46- [Dataset Description](#dataset-description)47  - [Dataset Summary](#dataset-summary)48  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)49  - [Languages](#languages)50- [Dataset Structure](#dataset-structure)51  - [Data Instances](#data-instances)52  - [Data Fields](#data-fields)53  - [Data Splits](#data-splits)54- [Dataset Creation](#dataset-creation)55  - [Curation Rationale](#curation-rationale)56  - [Source Data](#source-data)57  - [Annotations](#annotations)58  - [Personal and Sensitive Information](#personal-and-sensitive-information)59- [Considerations for Using the Data](#considerations-for-using-the-data)60  - [Social Impact of Dataset](#social-impact-of-dataset)61  - [Discussion of Biases](#discussion-of-biases)62  - [Other Known Limitations](#other-known-limitations)63- [Additional Information](#additional-information)64  - [Dataset Curators](#dataset-curators)65  - [Licensing Information](#licensing-information)66  - [Citation Information](#citation-information)67  - [Contributions](#contributions)68 69## Dataset Description70 71- **Homepage:** [https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/)72- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)73- **Paper:** [Pointer Sentinel Mixture Models](https://arxiv.org/abs/1609.07843)74- **Point of Contact:** [Stephen Merity](mailto:smerity@salesforce.com)75- **Size of downloaded dataset files:** 373.28 MB76- **Size of the generated dataset:** 1072.25 MB77- **Total amount of disk used:** 1445.53 MB78 79### Dataset Summary80 81 The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified82 Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License.83 84Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over85110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation86and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models87that can take advantage of long term dependencies.88 89### Supported Tasks and Leaderboards90 91[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)92 93### Languages94 95[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)96 97## Dataset Structure98 99### Data Instances100 101#### wikitext-103-raw-v1102 103- **Size of downloaded dataset files:** 183.09 MB104- **Size of the generated dataset:** 523.97 MB105- **Total amount of disk used:** 707.06 MB106 107An example of 'validation' looks as follows.108```109This example was too long and was cropped:110 111{112    "text": "\" The gold dollar or gold one @-@ dollar piece was a coin struck as a regular issue by the United States Bureau of the Mint from..."113}114```115 116#### wikitext-103-v1117 118- **Size of downloaded dataset files:** 181.42 MB119- **Size of the generated dataset:** 522.66 MB120- **Total amount of disk used:** 704.07 MB121 122An example of 'train' looks as follows.123```124This example was too long and was cropped:125 126{127    "text": "\" Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to..."128}129```130 131#### wikitext-2-raw-v1132 133- **Size of downloaded dataset files:** 4.50 MB134- **Size of the generated dataset:** 12.91 MB135- **Total amount of disk used:** 17.41 MB136 137An example of 'train' looks as follows.138```139This example was too long and was cropped:140 141{142    "text": "\" The Sinclair Scientific Programmable was introduced in 1975 , with the same case as the Sinclair Oxford . It was larger than t..."143}144```145 146#### wikitext-2-v1147 148- **Size of downloaded dataset files:** 4.27 MB149- **Size of the generated dataset:** 12.72 MB150- **Total amount of disk used:** 16.99 MB151 152An example of 'train' looks as follows.153```154This example was too long and was cropped:155 156{157    "text": "\" Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to..."158}159```160 161### Data Fields162 163The data fields are the same among all splits.164 165#### wikitext-103-raw-v1166- `text`: a `string` feature.167 168#### wikitext-103-v1169- `text`: a `string` feature.170 171#### wikitext-2-raw-v1172- `text`: a `string` feature.173 174#### wikitext-2-v1175- `text`: a `string` feature.176 177### Data Splits178 179|       name        | train |validation|test|180|-------------------|------:|---------:|---:|181|wikitext-103-raw-v1|1801350|      3760|4358|182|wikitext-103-v1    |1801350|      3760|4358|183|wikitext-2-raw-v1  |  36718|      3760|4358|184|wikitext-2-v1      |  36718|      3760|4358|185 186## Dataset Creation187 188### Curation Rationale189 190[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)191 192### Source Data193 194#### Initial Data Collection and Normalization195 196[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)197 198#### Who are the source language producers?199 200[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)201 202### Annotations203 204#### Annotation process205 206[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)207 208#### Who are the annotators?209 210[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)211 212### Personal and Sensitive Information213 214[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)215 216## Considerations for Using the Data217 218### Social Impact of Dataset219 220[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)221 222### Discussion of Biases223 224[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)225 226### Other Known Limitations227 228[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)229 230## Additional Information231 232### Dataset Curators233 234[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)235 236### Licensing Information237 238The dataset is available under the [Creative Commons Attribution-ShareAlike License (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/).239 240### Citation Information241 242```243@misc{merity2016pointer,244      title={Pointer Sentinel Mixture Models},245      author={Stephen Merity and Caiming Xiong and James Bradbury and Richard Socher},246      year={2016},247      eprint={1609.07843},248      archivePrefix={arXiv},249      primaryClass={cs.CL}250}251```252 253 254### Contributions255 256Thanks to [@thomwolf](https://github.com/thomwolf), [@lewtun](https://github.com/lewtun), [@patrickvonplaten](https://github.com/patrickvonplaten), [@mariamabarham](https://github.com/mariamabarham) for adding this dataset.