Datasetpublic

techiaith/cardiff-university-tm-en-cy

Dataset Card for Cardiff University Translation Memories Dataset Summary This dataset consists of English-Welsh sentence pairs extracted from Cardiff University translation memories. Supported Tasks and Leaderboards language-modeling parsing semantic-segmentation semantic-similarity-classification semantic-similarity-scoring sentiment-analysis sentiment-classification sentiment-scoring Languages English Welsh Dataset… See the full description on the dataset page: https://huggingface.co/datasets/techiaith/cardiff-university-tm-en-cy.

sourceHugging Facecc-by-4.0updated 10mo agoView on Hugging Face

2likes34downloads

Dataset Card

Dataset Card for Cardiff University Translation Memories

Dataset Description

—Homepage: https://huggingface.co/datasets/mgrbyte/cardiff-university-tm-en-cy
—Repository: https://huggingface.co/datasets/mgrbyte/cardiff-university-tm-en-cy
—Point of Contact: techiaith@bangor.ac.uk

Dataset Summary

This dataset consists of English-Welsh sentence pairs extracted from Cardiff University translation memories.

Supported Tasks and Leaderboards

—language-modeling
—parsing
—semantic-segmentation
—semantic-similarity-classification
—semantic-similarity-scoring
—sentiment-analysis
—sentiment-classification
—sentiment-scoring

Languages

—English
—Welsh

Dataset Structure

Data Fields

—text_en
—text_cy

Data Splits

—train

Dataset Creation

The original source data consists of English-Welsh human generated translations, stored as translation memory files in the TMX format.

This dataset was produced by processing the original translated content through a text pre-processing pipeline, which includes the following stages:

—Sentence splitting
—Deduplication
—Normalisation (encoding, whitespace, removal of formatting elements)
—LID (Language Identifcation Detection) filtering

Source Data

TMX file(s) prvoided by Cardiff University, pre-processed by the Language Technologies Unit, Bangor University.

Who are the source language producers?

Cardiff University