aiana94/polynews-parallel

Dataset Card for PolyNewsParallel

Table of Contents

• —Table of Contents
• —Dataset Description
• —Dataset Summary
• —Uses
• —Languages
• —Dataset Structure
• —Data Instances
• —Data Fields
• —Data Splits
• —Dataset Creation
• —Curation Rationale
• —Source Data
• —Data Collection and Processing
• —Annotations
• —Personal and Sensitive Information
• —Considerations for Using the Data
• —Social Impact of Dataset
• —Discussion of Biases
• —Other Known Limitations
• —Additional Information
• —Licensing Information
• —Citation Information

Dataset Description

• —Homepage: https://huggingface.co/datasets/aiana94/polynews-parallel
• —Repository: https://github.com/andreeaiana/nase
• —Paper: News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings for Cross-lingual News Recommendation
• —Point of Contact: Andreea Iana
• —License: CC-BY-4.0-NC

Dataset Summary

PolyNewsParallel is a multilingual paralllel dataset containing news titles for 833 language pairs. It covers 64 languages and 17 scripts.

Uses

This dataset can be used for machine translation or text retrieval.

Languages

There are 64 languages avaiable:

The heatmap shows the language pairs available, as well as the number of articles per language pair.

<center> <img src="https://huggingface.co/datasets/aiana94/polynews-parallel/resolve/main/polynews_parallel.png" alt="PolyNewsParallel: Number of texts per language pair"> </center>

Dataset Structure

Data Instances

>>> from datasets import load_dataset
>>> data = load_dataset('aiana94/polynews-parallel', 'eng_Latn-ron_Latn')

# Please, specify the language code,

# A data point example is below:

{
"src": "They continue to support the view that this decision will have a lasting negative impact on the rule of law in the country. ",
"tgt": "Ei continuă să creadă că această decizie va avea efecte negative pe termen lung asupra statului de drept în țară. ",
"provenance": "globalvoices"
}

Data Fields

• —src (string): source news text
• —tgt (string): target news text
• —provenance (string) : source dataset for the news example

Data Splits

For all languages, there is only the train split.

Dataset Creation

Curation Rationale

Multiple multilingual, human-translated, datasets containing news texts have been released in recent years. However, these datasets are stored in different formats and various websites, and many contain numerous near duplicates. With PolyNewsParallel, we aim to provide an easily-accessible, unified and deduplicated parallel dataset that combines these disparate data sources. It can be used for machine translation or text retrieval in both high-resource and low-resource languages.

Source Data

The source data consists of five multilingual news datasets.

• —GlobalVoices (v2018q4)
• —WMT-News (v2019)
• —MAFAND (train split)

Data Collection and Processing

We processed the data using a working script which covers the entire processing pipeline. It can be found here.

The data processing pipeline consists of:

1. 1.Downloading the WMT-News and GlobalVoices News from OPUS.
2. 2.Loading MAFAND datasets from Hugging Face Hub (only the train splits).
3. 3.Concatenating, per language, all news texts from the source datasets.
4. 4.Data cleaning (e.g., removal of exact duplicates, short texts, texts in other scripts)
5. 5.MinHash near-deduplication per language.

Annotations

We augment the original samples with the provenance annotation which specifies the original data source from which a particular examples stems.

Personal and Sensitive Information

The data is sourced from newspaper sources and contains mentions of public figures and individuals.

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

Users should keep in mind that the dataset contains short news texts (e.g., mostly titles), which might limit the applicability of the developed systems to other domains.

Additional Information

Licensing Information

The dataset is released under the CC BY-NC Attribution-NonCommercial 4.0 International license.

Citation Infomation

BibTeX:

@misc{iana2024news,
      title={News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings for Cross-lingual News Recommendation}, 
      author={Andreea Iana and Fabian David Schmidt and Goran Glavaš and Heiko Paulheim},
      year={2024},
      eprint={2406.12634},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2406.12634}
}