alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026-nocollapse

TTLab MARBERTv2 Surface Tagger for Arabic MT Error Spans (AlexandriaX-2026, revised no-collapse)

A fine-tuned MARBERTv2 token classifier that detects character-level error spans in English→Arabic-dialect machine translations and assigns each span an LQM/MQM-style error category. It is TTLab's system for AlexandriaX-2026 Subtask 3 and is described in TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification (Abusaleh, Verma & Mehler, ArabicNLP 2026).

This is the revised system from the paper's ablation: the same tagger, but trained on all six original LQM categories without collapsing the rare ones. In the paper, this configuration raised the official test Overall Score from 40.91 (official submission) to 42.00 (scored by the organizers after the evaluation period).

The exact official-submission checkpoint (rank 3, 40.91) is at alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026.

• —Code: https://github.com/aliabusaleh/arabic-dialectal-mt-error-span-detection
• —Data: `UBC-NLP/AlexandriaX_Subtask_3` (1,125 train / 138 dev sentences)
• —Dialect directions: ENG_EGY (Egyptian), ENG_MAU (Mauritanian), ENG_MOR (Moroccan), ENG_PAL (Palestinian), ENG_UAE (Emirati)

How it works

The model tags each sub-word token of the Arabic MT output only (the English source is not an input) over the labels O, graphetics, morphosyntax, orthography_writing_conventions, pragmatics, semantics, sociolinguistics. At inference, a token is flagged as an error when 1 - P(O) reaches a dialect-specific threshold τ_d, and it takes the most probable error category. Contiguous same-category tokens are merged, mapped back to character offsets with the tokenizer's offset mapping, trimmed of whitespace and punctuation, and spans shorter than 2 characters are dropped.

The head predicts O plus all six original categories directly; no label remapping is applied.

The thresholds are stored in span_decoding.json:

Usage

The span decoding (thresholds + merging) lives in inference.py, which ships in this repo:

from huggingface_hub import hf_hub_download
import importlib.util, sys

spec = importlib.util.spec_from_file_location("inference", hf_hub_download("alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026-nocollapse", "inference.py"))
inference = importlib.util.module_from_spec(spec); spec.loader.exec_module(inference)

tagger = inference.ErrorSpanTagger("alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026-nocollapse")
mt = "يلا خلّك وياها، هذا خرابيط. انت ما تدري إذا هي تأن ولا لا، وإذا هي تأن يسمونه أنين ليلي. نحنا بعدنا وايد بدري."
print(tagger.predict(mt, direction="ENG_UAE"))
# [{'text': ..., 'start': ..., 'end': ..., 'category': ...}, ...]

Batch tagging of a JSONL file (id, direction, source, model_prediction per line), writing the Codabench LQM_prediction_tagged_errors format:

python inference.py alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026-nocollapse test.jsonl predictions.jsonl

To get raw per-token probabilities, the checkpoint loads as a standard BertForTokenClassification:

from transformers import AutoTokenizer, AutoModelForTokenClassification
tok = AutoTokenizer.from_pretrained("alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026-nocollapse")
model = AutoModelForTokenClassification.from_pretrained("alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026-nocollapse")

For an unseen direction, the decoder falls back to τ = 0.4.

Training

Results

Overall = (overlap-span F1 + category F1) / 2, in percent. The official Codabench metric uses macro averages over directions. Our dev "proxy" uses character-overlap F1 + category micro-F1 and is not directly comparable.

Reproducibility note: the weights of the run behind the 42.00 score were not kept. This checkpoint is a fresh full-training-set retrain with the identical configuration, seed and thresholds. Because of GPU non-determinism it is not bit-identical: on the test set its predictions match the scored run's on 138 of 145 sentences (character-level span+category F1 0.95 between the two), and the original run scored 40.1 on dev. Expect a test score close to, but not guaranteed to equal, 42.00. For the exact ranked system, use the official-submission checkpoint.

Limitations

• —Tiny, imbalanced data. About 80% of tokens carry no error; sociolinguistics accounts for 58% of spans and graphetics has 2 training examples. Localisation is reasonable (overlap F1 ≈ 46–48), but category assignment is the bottleneck: morphosyntax, pragmatics and graphetics are rarely or never recovered.
• —Source-blind. Only the Arabic MT output is encoded, so errors that are only detectable by comparison with the English source (omissions, mistranslations of fluent-looking text) are partly out of reach.
• —Domain. Trained on five dialect directions of a single shared-task dataset. Thresholds are tuned per direction, so other dialects or domains will need re-tuning.
• —Intended for research on Arabic MT quality estimation. It is not a substitute for human MT evaluation.

Citation

If you use this model, please cite:

@inproceedings{Abusaleh:et:al:2026:mt,
  title     = {TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic
               Machine-Translation Error-Span Detection and Classification},
  author    = {Ali Abusaleh and Bhuvanesh Verma and Alexander Mehler},
  booktitle = {SIGARAB ArabicNLP 2026 AlexandriaX Shared Task},
  year      = {2026},
  address   = {Budapest, Hungary},
  note      = {accepted}
}

The model builds on MARBERTv2 (Abdul-Mageed et al., ACL 2021) and the AlexandriaX-2026 shared-task data; please cite those as well where appropriate.