Datasetpublic

atahanuz/setimes-en-tr-aligned-corpus-model-answers

SETimes EN-TR — Model Answers (Test Set) Translation outputs from two NMT architectures (a Transformer and an RNN seq2seq) on the 1,000-sentence test split of the SETimes EN-TR aligned corpus. Both models were trained on the same data with a joint 32k BPE vocabulary, and each was run in both directions (Turkish→English and English→Turkish). Each row pairs the human reference translations with all four model hypotheses, so the file is self-contained for re-scoring. 1,000… See the full description on the dataset page: https://huggingface.co/datasets/atahanuz/setimes-en-tr-aligned-corpus-model-answers.

sourceHugging Facecc-by-4.0updated 4mo agoView on Hugging Face

0likes8downloads

Dataset Card

SETimes EN-TR — Model Answers (Test Set)

Translation outputs from two NMT architectures (a Transformer and an RNN seq2seq) on the 1,000-sentence test split of the SETimes EN-TR aligned corpus. Both models were trained on the same data with a joint 32k BPE vocabulary, and each was run in both directions (Turkish→English and English→Turkish). Each row pairs the human reference translations with all four model hypotheses, so the file is self-contained for re-scoring.

1,000 sentences × 4 model outputs.

Columns

Column	Type	Description
`index`	int	Row id (1–1000)
`english_ref`	string	Human English reference (the TR→EN target)
`turkish_ref`	string	Human Turkish reference (the EN→TR target)
`tr2en_transformer`	string	Transformer output, Turkish→English
`en2tr_transformer`	string	Transformer output, English→Turkish
`tr2en_rnn`	string	RNN output, Turkish→English
`en2tr_rnn`	string	RNN output, English→Turkish

For TR→EN, score the tr2en_* columns against english_ref. For EN→TR, score the en2tr_* columns against turkish_ref.

Results

Corpus BLEU computed with sacreBLEU, using the international tokenizer with lowercasing (--tokenize intl --lc). This tokenization matters for Turkish, where apostrophes and hyphens are grammatical (ABD'nin, Bosna-Hersek). COMET is Unbabel/XCOMET-XL.

Direction	Model	Dev CE	BLEU	COMET	n-gram (1/2/3/4)	BP
TR→EN	Transformer	35.92	42.78	0.7793	72.9 / 49.8 / 36.4 / 27.3	0.98
TR→EN	RNN	52.88	35.10	0.6217	65.5 / 41.3 / 28.2 / 19.9	1.00
EN→TR	Transformer	38.69	42.04	0.7758	69.7 / 49.1 / 36.7 / 27.8	0.97
EN→TR	RNN	56.70	33.93	0.5893	60.1 / 39.6 / 27.9 / 20.0	1.00

The Transformer beats the RNN by ~7.7 BLEU (TR→EN) and ~8.1 BLEU (EN→TR). sacreBLEU signature: nrefs:1|case:lc|eff:no|tok:intl|smooth:exp.

How to load and re-score

python

from datasets import load_dataset
import sacrebleu

ds = load_dataset("atahanuz/setimes-en-tr-aligned-corpus-model-answers")["test"]

# TR -> EN, Transformer
hyps = ds["tr2en_transformer"]
refs = ds["english_ref"]
bleu = sacrebleu.corpus_bleu(hyps, [refs], tokenize="intl", lowercase=True)
print(bleu)

Or directly from pandas:

python

import pandas as pd
df = pd.read_csv("setimes_en_tr_model_answers.csv")

Source and license

References come from the SETimes EN-TR aligned corpus, an LLM-cleaned, sentence-aligned version of the SETimes parallel corpus (Tyers and Alperen, 2010). Distributed under CC-BY-4.0, same as the source.

Tyers, F. M., & Alperen, M. S. (2010). South-East European Times: A parallel corpus of Balkan languages. Proceedings of the LREC workshop on exploitation of multilingual resources and tools for Central and (South-) Eastern European Languages.