CoolFace
Datasetpublic

locailabs/tatoeba_welsh

🏴󠁧󠁢󠁷󠁬󠁳󠁿🇬🇧 Welsh-English Tatoeba Translation Dataset Part of the Welsh parallel corpora collection. Contains 3,337 Welsh-English translation pairs in chat format. Please find a blog on the data curation process here. Dataset Description This dataset provides Welsh-English translation pairs from Tatoeba. Collection of sentences and translations from Tatoeba community (combined v2 through v2023-04-12). The data has been processed through a multi-stage… See the full description on the dataset page: https://huggingface.co/datasets/locailabs/tatoeba_welsh.

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face
0likes11downloads
Dataset Card

🏴󠁧󠁢󠁷󠁬󠁳󠁿🇬🇧 Welsh-English Tatoeba Translation Dataset

Part of the Welsh parallel corpora collection. Contains 3,337 Welsh-English translation pairs in chat format.

Please find a blog on the data curation process here.

Dataset Description

This dataset provides Welsh-English translation pairs from Tatoeba. Collection of sentences and translations from Tatoeba community (combined v2 through v2023-04-12). The data has been processed through a multi-stage quality pipeline and formatted for instruction-based fine-tuning.

Format

Each entry is in messages format with balanced bidirectional translations (~50% English→Welsh, ~50% Welsh→English).

Source

Data sourced from OPUS - Tatoeba.

Original source: https://tatoeba.org/en/

Processing Pipeline

  1. 1.Length Filtering: Removed pairs < 20 characters
  2. 2.Semantic Deduplication: MinHash LSH with multilingual embeddings (threshold: 0.85)
  3. 3.Quality Filtering: Removed URLs, emojis, and excessive repetition
  4. 4.Bidirectional Balancing: Equal representation of both translation directions

Citation

Related Datasets