Prosho/sentinel-src-25

<div align="center">

<h1 style="font-family: 'Arial', sans-serif; font-size: 28px; font-weight: bold;"> 📊 Estimating Machine Translation Difficulty </h1>

</div>

<div style="display: flex; justify-content: center; align-items: center; gap: 8px; flex-wrap: wrap;"> <a href="https://2025.emnlp.org/" style="display: flex;"> <img src="https://img.shields.io/badge/EMNLP-2025-4b44ce.svg" alt="EMNLP 2025" height="20" style="display: block;"

</a> <a href="https://aclanthology.org/2025.findings-emnlp.1317/" style="display: flex;"> <img src="https://img.shields.io/badge/paper-ACL--anthology-B31B1B.svg" alt="ACL Anthology paper" height="20" style="display: block;"

</a> <a href="https://huggingface.co/collections/Prosho/translation-difficulty-estimators-6816665c008e1d22426eb6c4" style="display: flex;"> <img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Collection-FCD21D" alt="Hugging Face collection" height="20" style="display: block;"

</a> <a href="https://www.apache.org/licenses/LICENSE-2.0" style="display: flex;"> <img src="https://img.shields.io/badge/License-Apache%202.0-green" alt="Apache 2.0 license" height="20" style="display: block;"

</a> </div>

This repository contains the SENTINEL_SRC metric model used for Difficulty Sampling at the WMT25 General Machine Translation Shared Task, and analyzed in our paper Estimating Machine Translation Difficulty.

Usage

To run this model, install the following git repository:

pip install git+https://github.com/prosho-97/guardians-mt-eval

After that, you can use this model within Python in the following way:

from sentinel_metric import download_model, load_from_checkpoint

model_path = download_model("Prosho/sentinel-src-25")
model = load_from_checkpoint(model_path)

data = [
    {"src": "Please sign the form."},
    {"src": "He spilled the beans, then backpedaled—talk about mixed signals!"}
]

output = model.predict(data, batch_size=8, gpus=1)

Output:

# Segment scores
>>> output.scores
[0.5604351758956909, -0.08413456380367279]

# System score
>>> output.system_score
0.23815030604600906

Where the higher the output score, the easier it is to translate the input source text.

Cite this work

This work has been presented at EMNLP 2025. If you use any part, please consider citing our paper as follows:

@inproceedings{proietti-etal-2025-estimating,
    title = "Estimating Machine Translation Difficulty",
    author = "Proietti, Lorenzo  and
      Perrella, Stefano  and
      Zouhar, Vil{\'e}m  and
      Navigli, Roberto  and
      Kocmi, Tom",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-emnlp.1317/",
    doi = "10.18653/v1/2025.findings-emnlp.1317",
    pages = "24261--24285",
    ISBN = "979-8-89176-335-7"
}