CoolFace
Datasetpublic

mosesdaudu/translation_dataset

Loading and Splitting Dataset To Various Languages In this example, I will show you how to load the dataset and split by language for your downstream task. >>> from datasets import load_dataset >>> # load dataset >>> dataset = load_dataset("mosesdaudu/translation_dataset") >>> dataset DatasetDict({ train: Dataset({ features: ['english_text', 'language', 'translated_text', 'split'], num_rows: 198084 }) test: Dataset({ features:… See the full description on the dataset page: https://huggingface.co/datasets/mosesdaudu/translation_dataset.

sourceHugging Faceupdated 2y agoView on Hugging Face
0likes17downloads
Dataset Card

Loading and Splitting Dataset To Various Languages

In this example, I will show you how to load the dataset and split by language for your downstream task.

python
>>> from datasets import load_dataset

>>> # load dataset
>>> dataset = load_dataset("mosesdaudu/translation_dataset")

>>> dataset
DatasetDict({
    train: Dataset({
        features: ['english_text', 'language', 'translated_text', 'split'],
        num_rows: 198084
    })
    test: Dataset({
        features: ['english_text', 'language', 'translated_text', 'split'],
        num_rows: 22009
    })
})

>>> # Filter Dataset To Pidgin Language Only
>>> pidgin_dataset = dataset.filter(lambda example: example['language'] == 'pidgin')

>>> pidgin_dataset
DatasetDict({
    train: Dataset({
        features: ['english_text', 'language', 'translated_text', 'split'],
        num_rows: 22476
    })
    test: Dataset({
        features: ['english_text', 'language', 'translated_text', 'split'],
        num_rows: 2497
    })
})