CoolFace
Datasetpublic

locailabs/eubookshop_welsh

🏴󠁧󠁢󠁷󠁬󠁳󠁿🇬🇧 Welsh-English EUbookshop Translation Dataset Part of the Welsh parallel corpora collection. Contains 2,124 Welsh-English translation pairs in chat format. Please find a blog on the data curation process here. Dataset Description This dataset provides Welsh-English translation pairs from EUbookshop. Corpus of documents from the EU bookshop. The data has been processed through a multi-stage quality pipeline and formatted for instruction-based… See the full description on the dataset page: https://huggingface.co/datasets/locailabs/eubookshop_welsh.

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face
0likes16downloads
Dataset Card

🏴󠁧󠁢󠁷󠁬󠁳󠁿🇬🇧 Welsh-English EUbookshop Translation Dataset

Part of the Welsh parallel corpora collection. Contains 2,124 Welsh-English translation pairs in chat format.

Please find a blog on the data curation process here.

Dataset Description

This dataset provides Welsh-English translation pairs from EUbookshop. Corpus of documents from the EU bookshop. The data has been processed through a multi-stage quality pipeline and formatted for instruction-based fine-tuning.

Format

Each entry is in OpenAI chat format with balanced bidirectional translations (~50% English→Welsh, ~50% Welsh→English).

Source

Data sourced from OPUS - EUbookshop.

Original source: http://bookshop.europa.eu

Processing Pipeline

  1. 1.Length Filtering: Removed pairs < 20 characters
  2. 2.Semantic Deduplication: MinHash LSH with multilingual embeddings (threshold: 0.85)
  3. 3.Quality Filtering: Removed URLs, emojis, and excessive repetition
  4. 4.Bidirectional Balancing: Equal representation of both translation directions

Citation

Related Datasets