CoolFace
Datasetpublic

Helsinki-NLP/multiun

Dataset Card for OPUS MultiUN Dataset Summary The MultiUN parallel corpus is extracted from the United Nations Website , and then cleaned and converted to XML at Language Technology Lab in DFKI GmbH (LT-DFKI), Germany. The documents were published by UN from 2000 to 2009. This is a collection of translated documents from the United Nations originally compiled by Andreas Eisele and Yu Chen (see http://www.euromatrixplus.net/multi-un/). This corpus is available in… See the full description on the dataset page: https://huggingface.co/datasets/Helsinki-NLP/multiun.

sourceHugging Faceunknownupdated 3y agoView on Hugging Face
12likes18kdownloads
Dataset Card

Dataset Card for OPUS MultiUN

Table of Contents

Dataset Description

Dataset Summary

The MultiUN parallel corpus is extracted from the United Nations Website , and then cleaned and converted to XML at Language Technology Lab in DFKI GmbH (LT-DFKI), Germany. The documents were published by UN from 2000 to 2009.

This is a collection of translated documents from the United Nations originally compiled by Andreas Eisele and Yu Chen (see http://www.euromatrixplus.net/multi-un/).

This corpus is available in all 6 official languages of the UN consisting of around 300 million words per language

Supported Tasks and Leaderboards

The underlying task is machine translation.

Languages

Parallel texts are present in all six official languages: Arabic (ar), Chinese (zh), English (en), French (fr), Russian (ru) and Spanish (es), with a small part of the documents available also in German (de).

Dataset Structure

Data Instances

{
  "translation": {
    "ar": "قرار اتخذته الجمعية العامة",
    "de": "Resolution der Generalversammlung"
  }
}

Data Fields

  • —translation (dict): Parallel sentences for the pair of languages.

Data Splits

The dataset contains a single "train" split for each language pair.

Dataset Creation

Curation Rationale

[More Information Needed]

Source Data

Initial Data Collection and Normalization

Original MultiUN source data: http://www.euromatrixplus.net/multi-unp

Who are the source language producers?

[More Information Needed]

Annotations

Annotation process

[More Information Needed]

Who are the annotators?

[More Information Needed]

Personal and Sensitive Information

[More Information Needed]

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

[More Information Needed]

Licensing Information

[More Information Needed]

Citation Information

If you use this corpus in your work, please cite the paper:

@inproceedings{eisele-chen-2010-multiun,
    title = "{M}ulti{UN}: A Multilingual Corpus from United Nation Documents",
    author = "Eisele, Andreas  and
      Chen, Yu",
    booktitle = "Proceedings of the Seventh International Conference on Language Resources and Evaluation ({LREC}'10)",
    month = may,
    year = "2010",
    address = "Valletta, Malta",
    publisher = "European Language Resources Association (ELRA)",
    url = "http://www.lrec-conf.org/proceedings/lrec2010/pdf/686_Paper.pdf",
    abstract = "This paper describes the acquisition, preparation and properties of a corpus extracted from the official documents of the United Nations (UN). This corpus is available in all 6 official languages of the UN, consisting of around 300 million words per language. We describe the methods we used for crawling, document formatting, and sentence alignment. This corpus also includes a common test set for machine translation. We present the results of a French-Chinese machine translation experiment performed on this corpus.",
}

If you use any part of the corpus (hosted in OPUS) in your own work, please cite the following article:

@inproceedings{tiedemann-2012-parallel,
    title = "Parallel Data, Tools and Interfaces in {OPUS}",
    author = {Tiedemann, J{\"o}rg},
    editor = "Calzolari, Nicoletta  and
      Choukri, Khalid  and
      Declerck, Thierry  and
      Do{\u{g}}an, Mehmet U{\u{g}}ur  and
      Maegaard, Bente  and
      Mariani, Joseph  and
      Moreno, Asuncion  and
      Odijk, Jan  and
      Piperidis, Stelios",
    booktitle = "Proceedings of the Eighth International Conference on Language Resources and Evaluation ({LREC}'12)",
    month = may,
    year = "2012",
    address = "Istanbul, Turkey",
    publisher = "European Language Resources Association (ELRA)",
    url = "http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper.pdf",
    pages = "2214--2218",
    abstract = "This paper presents the current status of OPUS, a growing language resource of parallel corpora and related tools. The focus in OPUS is to provide freely available data sets in various formats together with basic annotation to be useful for applications in computational linguistics, translation studies and cross-linguistic corpus studies. In this paper, we report about new data sets and their features, additional annotation tools and models provided from the website and essential interfaces and on-line services included in the project.",
}

Contributions

Thanks to @patil-suraj for adding this dataset.