Rijgersberg/common_corpus_nl

Common Corpus v2 NL

This is a version of Common Corpus v2 filtered to keep only the rows where language is "Dutch".

Common Corpus is a very large open and permissible licensed text dataset created by Pleias. Please be sure to acknowledge the creators of the original dataset when using this filtered version.

Filtering

Common Corpus is a collection of disparate datasets. Note that filtering the entire collection for rows where the language is "Dutch" is not the same as filtering for entire datasets that are supposed to be Dutch-language. Since language classification is an automated process, there may be false positives and false negatives.

Examples are:

• —false positives: code from StackExchange, French from French-PD-diverse are included in Common Corpus NL when they shouldn't be.
• —false negatives: rows from Dutch-PD that were misclassified, for example as English, are not included in Common Corpus NL when they should be.

If you want to use entire datasets, you either have to look up what the source was for including it into Common Corpus, or filter Common Corpus yourself.

Usage

from datasets import load_dataset

# load the full dataset
dataset = load_dataset('Rijgersberg/common_corpus_nl', 'all', split='train')

# load only a specific subset
wikipedia = load_dataset('Rijgersberg/common_corpus_nl', 'Open Web-Wikipedia', split='train')

Contents

The dataset has the following content. Tokens are measured by the robbert-2023-dutch-base Dutch tokenizer on the text column only. word_count is taken directly from the word_count column of the dataset.

Code

In principle it is very easy to create Common Corpus NL by filtering Common Corpus using Hugging Face datasets' dataset.filter() functionality. However, Common Corpus is larger than my available disk space.

A possible solution is to process Common Corpus streaming and relying on the fact that the Dutch subset will be much, much smaller than the full dataset. The code for that solution is below. However, I was having trouble streaming the entire dataset without any connection errors along the way.

from datasets import load_dataset, Dataset


common_corpus = load_dataset('PleIAs/common_corpus', split='train', streaming=True)

def nl():
    for row in common_corpus:
        if row['language'] == 'Dutch':
            yield row

common_corpus_nl = Dataset.from_generator(nl)

common_corpus_nl.push_to_hub('Rijgersberg/common_corpus_nl')

Therefore I took the approach for every one of ten subfolders of Common Corpus:

• —downloading the subfolder in a fault-tolerant way
• —doing the filtering to Dutch rows only
• —uploading that by itself to the Hugging Face hub
• —deleting all the downloaded files and datasets cache files (around 1.5 TB for every subfolder)

Then finally I concatenated the ten Dutch datasets into a single one, which is the one you are looking at.

import shutil

from datasets import concatenate_datasets, load_dataset
from huggingface_hub import snapshot_download
from huggingface_hub.errors import LocalEntryNotFoundError
from requests import ReadTimeout


local_dir = '/path/to/downloadfolder/commoncorpus'

for i in range(1, 10+1):
    success = False
    while not success:
        try:
            # download one common corpus folder at a time to a local directory
            snapshot_download(  # will skip files that have already been downloaded
                repo_id='PleIAs/common_corpus',
                repo_type='dataset',
                allow_patterns=f'common_corpus_{i}/*',
                local_dir=local_dir
            )
            success = True
        except (LocalEntryNotFoundError, ReadTimeout) as e:
            print(e)

    subdataset = load_dataset(local_dir, split='train')
    subdataset = subdataset.filter(lambda lang: lang == 'Dutch', input_columns=['language'])

    subdataset.push_to_hub(f'Rijgersberg/common_corpus_nl_{i}')

    # remove local copies of the data to free up disk space
    shutil.rmtree(local_dir)
    shutil.rmtree('/path/to/cache/huggingface/datasets/commoncorpus')

# concatenate all (much smaller) Dutch datasets into a single dataset
common_corpus_nl = concatenate_datasets([load_dataset(f'Rijgersberg/common_corpus_nl_{i}', split='train')
                                         for i in range(1, 10+1)])
common_corpus_nl.push_to_hub('Rijgersberg/common_corpus_nl')