Datasetpublic

tartuNLP/pale-madlad-data

license: mit PaLe-MADLAD Data Data used for training the PaLe-MADLAD model to translate from Proper Karelian, Livvi, Ludian, and Veps to Russian and vice versa. Every dataset entry represents a single text and comes as a list of sentences supplemented (where possible) with a list of translations into Russian. Our sources include: VepKar: various articles, Biblical texts, folklore, and more in Proper Karelian, Livvi, Ludian, and Veps, mostly translated into… See the full description on the dataset page: https://huggingface.co/datasets/tartuNLP/pale-madlad-data.

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes26downloads

Dataset Card

license: mit ---

PaLe-MADLAD Data

Data used for training the PaLe-MADLAD model to translate from Proper Karelian, Livvi, Ludian, and Veps to Russian and vice versa. Every dataset entry represents a single text and comes as a list of sentences supplemented (where possible) with a list of translations into Russian. Our sources include:

—VepKar: various articles, Biblical texts, folklore, and more in Proper Karelian, Livvi, Ludian, and Veps, mostly translated into Russian
—Omamedia: news articles in Proper Karelian, Livvi, and Veps
—Wikipedia: multi-domain articles in Livvi and Veps
—yle.fi: news articles in Livvi

Example Usage

from datasets import load_dataset

# Options for loading: 'vepkar', 'omamedia', 'wikipedia', 'ylefi'
load_dataset('tartuNLP/pale-madlad-data', 'vepkar')

Please cite the following paper if you use these datasets in your work:

@inproceedings{
pashchenko2024paragraphlevel,
title={Paragraph-Level Machine Translation for Low-Resource Finno-Ugric Languages},
author={Dmytro Pashchenko and Lisa Yankovskaya and Mark Fishel},
booktitle={The Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies},
year={2024},
url={https://openreview.net/forum?id=uTFJsQpNZk}
}

If you use 'Vepkar', please also cite the following paper:

@article{Boyko_2022_vepkar,
    title={The Open Corpus of the Veps and Karelian Languages: Overview and Applications},
    volume={7},
    number={3},
    journal={KnE Social Sciences},
    author={Boyko, Tatyana and Zaitseva, Nina and Krizhanovskaya, Natalia and Krizhanovsky, Andrew and Novak, Irina and Pellinen, Nataliya and Rodionova, Aleksandra},
    year={2022},
    month={Feb.},
    pages={29–40}}