LumiOpen/lingsoft-pemt-evals

Lingsoft PEMT translation preference evals

Evaluation sets for translation quality and fluency, built from the Lingsoft-EU-Summaries-PEMT corpus: professional post-edited machine translations of "Summaries of EU Legislation" (2016–2025). Each item pairs a raw machine translation (Raw_MT) with its professional human post-edit (Target_PEMT) for the same English source sentence. Only pairs where post-editing changed the text are included. Intended for use with the LumiOpen lm-evaluation-harness lingsoft_pemt tasks.

Configs

FI_fluency

The current Finnish fluency set for post-trained models. It contains 226 deduplicated RawMT/TargetPEMT pairs selected from the annotation rows supplied by Maria Barrett after retaining samples unique to the fluency workflow. The lm-eval task presents only the Finnish candidates, in both A/B orders; the English source and annotation labels remain metadata and are not shown to the model.

Source annotation SHA-256: 7849da8d4c4ff1f14a9fb6a6821cf0132822d4faafdb290fdc208eac4362680a.

FI_preference_5shot

Pre-rendered blind A/B preference prompts for Finnish (task lingsoft_pemt_fi_mcf_*): the prompt shows the English source and both translations labeled only "Finnish translation A/B", with 5 few-shot examples (sampled from the FI train split, balanced answer labels) baked into the prompt. Every pair appears twice with the A/B order swapped (order = raw_first / target_first), so random chance is exactly 50%. Score the single-token continuations " A" / " B" by loglikelihood; the correct answer is always the post-edited side. Generated by create_fi_base_model_eval.py in the source repo (5 shots, seed 42).

FI_pairs

The underlying changed rows (task lingsoft_pemt_fi_cf_*): source_text, raw_mt, target_pemt plus corpus metadata. id matches pair_id in FI_preference_5shot. Use for completion-form scoring — compare loglikelihoods of the two translations directly as continuations of the source.

Splits

• —test_shared: text units from 6 segments shared across all 23 corpus languages (cross-lingually comparable if more languages are added).
• —test_broad: a broad sample across all segments.

FI_fluency has one test split with 226 canonical pairs.

Provenance and license

Derived from professional translation and post-editing projects by Lingsoft under "Summaries of EU Legislation" (2016–2025). Original editorial content © European Union, re-used via the Publications Office of the European Union under CC-BY 4.0. Proprietary triple-stage alignment and raw MT layers provided by Lingsoft.