aiana94/polynews

Dataset Card for PolyNews

Table of Contents

• —Table of Contents
• —Dataset Description
• —Dataset Summary
• —Uses
• —Languages
• —Dataset Structure
• —Data Instances
• —Data Fields
• —Data Splits
• —Dataset Creation
• —Curation Rationale
• —Source Data
• —Data Collection and Processing
• —Annotations
• —Personal and Sensitive Information
• —Considerations for Using the Data
• —Social Impact of Dataset
• —Discussion of Biases
• —Other Known Limitations
• —Additional Information
• —Licensing Information
• —Citation Information

Dataset Description

• —Homepage: https://huggingface.co/datasets/aiana94/polynews
• —Repository: https://github.com/andreeaiana/nase
• —Paper: News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings for Cross-lingual News Recommendation
• —Point of Contact: Andreea Iana
• —License: CC-BY-4.0-NC

Dataset Summary

PolyNews is a multilingual dataset containing news titles in 77 languages and 19 scripts.

Uses

This dataset can be used for domain adaptation of language models, language modeling or text generation.

Languages

There are 77 languages available:

Dataset Structure

Data Instances

>>> from datasets import load_dataset
>>> data = load_dataset('aiana94/polynews', 'ron_Latn')

# Please, specify the language code,

# A data point example is below:

{
"text": "Un public numeros. Este uimitor succesul după doar trei ediții . ",
"provenance": "globalvoices"
}

Data Fields

• —text (string): news text
• —provenance (string) : source dataset for the news example

Data Splits

For all languages, there is only the train split.

Dataset Creation

Curation Rationale

Multiple multilingual, human-translated, datasets containing news texts have been released in recent years. However, these datasets are stored in different formats and various websites, and many contain numerous near duplicates. With PolyNews, we aim to provide an easily-accessible, unified and deduplicated dataset that combines these disparate data sources. It can be used for domain adaptation of language models, language modeling or text generation in both high-resource and low-resource languages.

Source Data

The source data consists of five multilingual news datasets.

• —Wikinews (latest dump available in May 2024)
• —GlobalVoices (v2018q4)
• —WMT-News (v2019)
• —MasakhaNews (train split)
• —MAFAND (train split)

Data Collection and Processing

We processed the data using a working script which covers the entire processing pipeline. It can be found here.

The data processing pipeline consists of:

1. 1.Downloading the WMT-News and GlobalVoices News from OPUS.
2. 2.Downloading the latest dump from WikiNews.
3. 3.Loading the MasakhaNews and MAFAND datasets from Hugging Face Hub (only the train splits).
4. 4.Concatenating, per language, all news texts from the source datasets.
5. 5.Data cleaning (e.g., removal of exact duplicates, short texts, texts in other scripts)
6. 6.MinHash near-deduplication per language.

Annotations

We augment the original samples with the provenance annotation which specifies the original data source from which a particular examples stems.

Personal and Sensitive Information

The data is sourced from newspaper sources and contains mentions of public figures and individuals.

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

Users should keep in mind that the dataset contains short news texts (e.g., mostly titles), which might limit the applicability of the developed systems to other domains.

Additional Information

Licensing Information

The dataset is released under the CC BY-NC Attribution-NonCommercial 4.0 International license.

Citation Infomation

BibTeX:

@misc{iana2024news,
      title={News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings for Cross-lingual News Recommendation}, 
      author={Andreea Iana and Fabian David Schmidt and Goran Glavaš and Heiko Paulheim},
      year={2024},
      eprint={2406.12634},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2406.12634}
}