eole-nlp/metricx-24-hybrid-xl-v2p6-eole

MetricX-24 Hybrid XL FP32 (EOLE)

This is google/metricx-24-hybrid-xl-v2p6 converted to EOLE format.

Model weights are stored as fp32. This repository contains a format conversion for use with EOLE's native transformer_encoder_decoder_scorer implementation.

This is not an upstream Google MetricX, Transformers, or T5X checkpoint layout. Use it with EOLE.

Model Details

Original model

google/metricx-24-hybrid-xl-v2p6

EOLE architecture

transformer_encoder_decoder_scorer

Weight dtype

fp32

Scoring type

metricx

EOLE scorers

EOLE-METRICX, EOLE-METRICX-QE

Supported input modes

reference, qe

Requires reference by default

Yes for reference mode; no for QE mode

Max input length

1536

Reference input template

source: {src} candidate: {tgt} reference: {ref}

QE input template

source: {src} candidate: {tgt}

Score interpretation

MetricX scores are lower-is-better error scores, clipped by the upstream model to the 0..25 range.

Usage

Requires an EOLE version with native MetricX scorer support.

Validation Metric: Reference Mode

valid_metrics: ["EOLE-METRICX"]
metricx_model: eole-nlp/metricx-24-hybrid-xl-v2p6-eole
metricx_compute_dtype: fp32
metricx_batch_size: 4

Validation Metric: QE Mode

valid_metrics: ["EOLE-METRICX-QE"]
metricx_model: eole-nlp/metricx-24-hybrid-xl-v2p6-eole
metricx_compute_dtype: fp32
metricx_batch_size: 4

Direct Scoring: Reference Mode

eole predict \
  --model_path eole-nlp/metricx-24-hybrid-xl-v2p6-eole \
  --src /path/to/src.txt \
  --tgt /path/to/mt.txt \
  --ref /path/to/ref.txt \
  --output /path/to/scores.txt \
  --with_score \
  --compute_dtype fp32

Direct Scoring: QE Mode

Omit --ref to score in reference-free QE mode:

eole predict \
  --model_path eole-nlp/metricx-24-hybrid-xl-v2p6-eole \
  --src /path/to/src.txt \
  --tgt /path/to/mt.txt \
  --output /path/to/qe-scores.txt \
  --with_score \
  --compute_dtype fp32

By default, direct scoring writes one segment score per input line. To emit a single aggregate system score, add --score_level system.

Conversion

Converted with:

eole convert MetricX \
  --model google/metricx-24-hybrid-xl-v2p6 \
  --dtype fp32 \
  --output metricx-24-hybrid-xl-v2p6-eole

Precision Note

MetricX uses mT5-style encoder-decoder scorer models, which can be numerically sensitive under reduced precision. This repository publishes fp32 converted weights as the reference EOLE artifact. Users may convert fp16 or bf16 variants separately if they accept possible score drift.

Original Model and Attribution

This model is a conversion of google/metricx-24-hybrid-xl-v2p6. The model weights were converted to EOLE format.

The upstream MetricX-24 models were proposed for the WMT 2024 Metrics Shared Task. MetricX-24 models are hybrid reference-based/reference-free metrics: the same model can score with references or in QE mode without references. Google recommends the Large variant for best speed, XL as an intermediate option, and XXL for best agreement with human judgments.

This converted repository follows the upstream model license: Apache-2.0. Please refer to the original model card and license for training data, intended use, limitations, and citation details.

MetricX GitHub repository: https://github.com/google-research/metricx

If you use MetricX-24 in your research, please cite:

@inproceedings{juraska-etal-2024-metricx,
    title = "{M}etric{X}-24: The {G}oogle Submission to the {WMT} 2024 Metrics Shared Task",
    author = "Juraska, Juraj  and
      Deutsch, Daniel  and
      Finkelstein, Mara  and
      Freitag, Markus",
    editor = "Haddow, Barry  and
      Kocmi, Tom  and
      Koehn, Philipp  and
      Monz, Christof",
    booktitle = "Proceedings of the Ninth Conference on Machine Translation",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.wmt-1.35",
    pages = "492--504",
}