Polygl0t/mc4-50k
mC4 (50K samples per language) Dataset Summary This is the multilingual version of the C4 dataset. We only keep 50,000 samples per language for all the 108 languages available. This is intended for educational and experimentation purposes. For the full dataset, please refer to the multilingual config of the allenai/c4 dataset. How do I download this? Using 🤗 Datasets from datasets import load_dataset # Portuguese only pt =… See the full description on the dataset page: https://huggingface.co/datasets/Polygl0t/mc4-50k.
mC4 (50K samples per language)
Dataset Description
- Paper: https://arxiv.org/abs/1910.10683
Dataset Summary
This is the multilingual version of the C4 dataset. We only keep 50,000 samples per language for all the 108 languages available. This is intended for educational and experimentation purposes. For the full dataset, please refer to the `multilingual` config of the `allenai/c4` dataset.
How do I download this?
Using 🤗 Datasets
from datasets import load_dataset
# Portuguese only
pt = load_dataset("Polygl0t/mc4-50k", "pt")
# All languages (108)
mc4 = load_dataset("Polygl0t/mc4-50k")You can also load and mix multiple languages:
from datasets import concatenate_datasets, interleave_datasets, load_dataset
es = load_dataset("Polygl0t/mc4-50k", "es", streaming=True)
fr = load_dataset("Polygl0t/mc4-50k", "fr", streaming=True)
# Concatenate both datasets
concatenated = concatenate_datasets([es, fr])
# Or interleave them (alternates between one and the other)
interleaved = interleave_datasets([es, fr])Languages
Note that the languages that end with "-Latn" are simply romanized variants, i.e. written using the Latin script.
Dataset Structure
Data Instances
An example form the en config is:
{
'text': 'Beginners BBQ Class Taking Place in Missoula!\nDo you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.\nHe will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.\nThe cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.'
}Data Fields
The data have several fields:
text: text content as a string
Source Data
Initial Data Collection and Normalization
The C4 and mC4 datasets are collections text sourced from the public Common Crawl web scrape. It includes heuristics to extract only natural language (as opposed to boilerplate and other gibberish) in addition to extensive deduplication.
Licensing Information
We are releasing this dataset under the terms of ODC-BY. By using this, you are also bound by the Common Crawl terms of use in respect of the content contained in the dataset.