bertin-project/mc4-sampling

1---2annotations_creators:3- no-annotation4language_creators:5- found6language:7- af8- am9- ar10- az11- be12- bg13- bn14- ca15- ceb16- co17- cs18- cy19- da20- de21- el22- en23- eo24- es25- et26- eu27- fa28- fi29- fil30- fr31- fy32- ga33- gd34- gl35- gu36- ha37- haw38- hi39- hmn40- ht41- hu42- hy43- id44- ig45- is46- it47- iw48- ja49- jv50- ka51- kk52- km53- kn54- ko55- ku56- ky57- la58- lb59- lo60- lt61- lv62- mg63- mi64- mk65- ml66- mn67- mr68- ms69- mt70- my71- ne72- nl73- 'no'74- ny75- pa76- pl77- ps78- pt79- ro80- ru81- sd82- si83- sk84- sl85- sm86- sn87- so88- sq89- sr90- st91- su92- sv93- sw94- ta95- te96- tg97- th98- tr99- uk100- und101- ur102- uz103- vi104- xh105- yi106- yo107- zh108- zu109license:110- odc-by111multilinguality:112- multilingual113size_categories:114- n<1K115- 1K<n<10K116- 10K<n<100K117- 100K<n<1M118- 1M<n<10M119- 10M<n<100M120- 100M<n<1B121- 1B<n<10B122source_datasets:123- original124task_categories:125- text-generation126- fill-mask127task_ids:128- language-modeling129paperswithcode_id: mc4130pretty_name: mC4-sampling131language_bcp47:132- bg-Latn133- el-Latn134- hi-Latn135- ja-Latn136- ru-Latn137- zh-Latn138---139 140# Dataset Card for mC4-sampling141 142## Table of Contents143 144- [Dataset Card for mC4-sampling](#dataset-card-for-mc4-sampling)145  - [Table of Contents](#table-of-contents)146  - [Dataset Description](#dataset-description)147    - [Dataset Summary](#dataset-summary)148    - [Dataset Sampling](#dataset-sampling)149    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)150    - [Languages](#languages)151  - [Dataset Structure](#dataset-structure)152    - [Data Instances](#data-instances)153    - [Data Fields](#data-fields)154    - [Data Splits](#data-splits)155  - [Additional Information](#additional-information)156    - [Dataset Curators](#dataset-curators)157    - [Licensing Information](#licensing-information)158    - [Citation Information](#citation-information)159    - [Contributions](#contributions)160 161## Dataset Description162 163- **Homepage:** https://huggingface.co/bertin-project/bertin-roberta-base-spanish164 165### Dataset Summary166 167This dataset builds upon the AllenAI version of the original [mC4](https://huggingface.co/datasets/allenai/c4) and adds sampling methods to perform perplexity-based filtering on the fly. Please, refer to [BERTIN Project](https://huggingface.co/bertin-project/bertin-roberta-base-spanish).168 169The original dataset is mC4, the multilingual colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org".170 171108 languages are available and are reported in the [`mc4` dataset](https://huggingface.co/datasets/allenai/c4#dataset-summary).172 173You can load the mC4 subset of any language like this (with default *random* sampling):174 175```python176from datasets import load_dataset177 178en_mc4 = load_dataset("bertin-project/mc4-sampling", "en")179```180 181And if you can even specify a list of languages:182 183```python184from datasets import load_dataset185 186mc4_subset_with_five_languages = load_dataset("bertin-project/mc4-sampling", languages=["en", "fr", "es", "de", "zh"])187```188 189### Dataset Sampling190 191There are 3 main different ways of getting sampled versions of mc4 using this dataset.192 193#### Random194 195Arguably, the simplest of methods. It keeps a document based on a probability threshold we called `factor`. It defaults to `0.5` for random sampling:196 197```python198def _should_keep_doc_random(self, doc, factor=None, **kwargs):199    factor = 0.5 if factor is None else factor200    return self.rng.uniform() <= factor201```202 203The way to use this sampling method is by adding an extra parameter to the instantiation of the dataset:204 205```python206from datasets import load_dataset207 208mc4random = load_dataset(209    "bertin-project/mc4-sampling", "es",210    split="train",211    streaming=True,212    sampling_method="random",213    factor=0.5,214)215for sample in mc4random:216    print(sample)217    break218```219 220#### Gaussian221 222This sampling method tries to adjust to the underlying distribution while oversampling the central quartiles of the perplexity distribution of the documents in mC4 for a given language. Two parameters control the shape of the approximation, `factor` (peakness of the exponential function) and `width` (spread). Default values are selected for Spanish.223 224```python225def _should_keep_doc_gaussian(self, doc, factor=None, width=None, boundaries=None, **kwargs):226    perplexity = self.get_perplexity(doc)227    width = (9 / 2) if width is None else width228    factor = 0.78 if factor is None else factor229    median = 662247.50212365 if boundaries is None else boundaries[1]230    exponential = np.exp((-1 / width) * ((perplexity - median) / median) ** 2)231    weighted_perplexity = factor * exponential232    return self.rng.uniform() < weighted_perplexity233```234 235In order to use this sampling methods, information about the quartile boundaries of the underlying distribution need to be calculated beforehand and passed in to the instantiation of the dataset. Moreover, the path to a [KenLM model](https://github.com/kpu/kenlm/) (5-gram language model) or an object with a method `.score(text:str) -> float` need to also be passed in for the calculation of the perplexity value of a document. KenLM can be installed with pip:236 237```bash238pip install https://github.com/kpu/kenlm/archive/master.zip239```240 241```python242from datasets import load_dataset243 244mc4gaussian = load_dataset(245    "bertin-project/mc4-sampling",246    "es",247    split="train",248    streaming=True,249    sampling_method="gaussian",250    perplexity_model="./es.arpa.bin",251    boundaries=[536394.99320948, 662247.50212365, 919250.87225178],252    factor=0.78,253    width=9/2,254)255for sample in mc4gaussian:256    print(sample)257    break258```259 260Facebook has created and released 5-gram Kneser-Ney models for 100 languages available to download and use within the KenLM library. To download your own Kneser-Ney language model, chose a language code from the next list:261 262```bash263af,ar,az,be,bg,bn,ca,cs,da,de,el,en,es,et,fa,fi,fr,gu,he,hi,hr,hu,hy,id,is,it,ja,ka,kk,km,kn,ko,lt,lv,mk,ml,mn,mr,my,ne,nl,no,pl,pt,ro,ru,uk,zh264```265 266And run the next download command replacing `lang` with your own language code:267 268```bash269wget http://dl.fbaipublicfiles.com/cc_net/lm/lang.arpa.bin270```271 272### Stepwise273 274The stepwise sampling method uses a simple criteria by oversampling from the central quartiles inversely proportionally their range. Only `boundaries`, `factor` (strength of the oversampling), and `perplexity_model` are needed:275 276```python277def _should_keep_doc_step(self, doc, factor=None, boundaries=None, **kwargs):278    perplexity = self.get_perplexity(doc)279    factor = 1.5e5 if factor is None else factor280    if boundaries is None:281        boundaries = [536394.99320948, 662247.50212365, 919250.87225178]282    if perplexity <= boundaries[0]:283        quartile_range = boundaries[0]284    elif boundaries[0] < perplexity < boundaries[1]:285        quartile_range = boundaries[1] - boundaries[0]286    elif boundaries[1] < perplexity < boundaries[2]:287        quartile_range = boundaries[2] - boundaries[1]288    elif perplexity >= boundaries[2]:289        quartile_range = 10 * boundaries[2]290    probability = factor / quartile_range291    return self.rng.uniform() < probability292```293 294In order to use this sampling method, a similar invocation is needed:295 296```python297mc4stepwsie = load_dataset(298    "bertin-project/mc4-sampling",299    "es",300    split="train",301    streaming=True,302    sampling_method="stepwise",303    perplexity_model="./es.arpa.bin",304    boundaries=[536394.99320948, 662247.50212365, 919250.87225178],305    factor=1.5e5,306)307for sample in mc4stepwsie:308    print(sample)309    break310```311 312### Supported Tasks and Leaderboards313 314mC4-sampling is mainly intended to pretrain language models and word representations on a budget.315 316### Languages317 318The dataset supports 108 languages.319 320## Dataset Structure321 322### Data Instances323 324An example form the `en` config is:325 326```327{'timestamp': '2018-06-24T01:32:39Z',328 'text': 'Farm Resources in Plumas County\329Show Beginning Farmer Organizations & Professionals (304)\330There are 304 resources serving Plumas County in the following categories:\331Map of Beginning Farmer Organizations & Professionals serving Plumas County\332Victoria Fisher - Office Manager - Loyalton, CA\333Amy Lynn Rasband - UCCE Plumas-Sierra Administrative Assistant II - Quincy , CA\334Show Farm Income Opportunities Organizations & Professionals (353)\335There are 353 resources serving Plumas County in the following categories:\336Farm Ranch And Forest Retailers (18)\337Map of Farm Income Opportunities Organizations & Professionals serving Plumas County\338Warner Valley Wildlife Area - Plumas County\339Show Farm Resources Organizations & Professionals (297)\340There are 297 resources serving Plumas County in the following categories:\341Map of Farm Resources Organizations & Professionals serving Plumas County\342There are 57 resources serving Plumas County in the following categories:\343Map of Organic Certification Organizations & Professionals serving Plumas County',344 'url': 'http://www.californialandcan.org/Plumas/Farm-Resources/'}345```346 347### Data Fields348 349The data have several fields:350 351- `url`: url of the source as a string352- `text`: text content as a string353- `timestamp`: timestamp as a string354 355### Data Splits356 357The same splits as in [mC4 are available](https://huggingface.co/datasets/mc4#data-splits).358 359## Additional Information360 361### Licensing Information362 363BERTIN Project is releasing this dataset under the same terms AllenAI released mC4, that is, those of the ODC-BY. By using this, you are also bound by the Common Crawl terms of use in respect of the content contained in the dataset.364 365### Citation Information366 367To cite this dataset:368```bibtex369@article{BERTIN,370	author = {Javier De la Rosa y Eduardo G. Ponferrada y Manu Romero y Paulo Villegas y Pablo González de Prado Salas y María Grandury},371	title = {{BERTIN}: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling},372	journal = {Procesamiento del Lenguaje Natural},373	volume = {68},374	number = {0},375	year = {2022},376	keywords = {},377	abstract = {The pre-training of large language models usually requires massive amounts of resources, both in terms of computation and data. Frequently used web sources such as Common Crawl might contain enough noise to make this pretraining sub-optimal. In this work, we experiment with different sampling methods from the Spanish version of mC4, and present a novel data-centric technique which we name perplexity sampling that enables the pre-training of language models in roughly half the amount of steps and using one fifth of the data. The resulting models are comparable to the current state-of-the-art, and even achieve better results for certain tasks. Our work is proof of the versatility of Transformers, and paves the way for small teams to train their models on a limited budget.},378	issn = {1989-7553},379	url = {http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6403},380	pages = {13--23}381}382```383 384If you use this dataset, we would love to hear about it! Reach out on twitter, GitHub, Discord, or shoot us an email.385 386To cite the original `mc4` dataset:387```388@article{2019t5,389    author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},390    title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},391    journal = {arXiv e-prints},392    year = {2019},393    archivePrefix = {arXiv},394    eprint = {1910.10683},395}396```397 398### Contributions399 400Dataset contributed by [@versae](https://github.com/versae).401 402Thanks to [@dirkgr](https://github.com/dirkgr) and [@lhoestq](https://github.com/lhoestq) for adding the original mC4 dataset.403