alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026

TTLab MARBERTv2 Surface Tagger for Arabic MT Error Spans (AlexandriaX-2026, official submission)

A fine-tuned MARBERTv2 token classifier that detects character-level error spans in English→Arabic-dialect machine translations and assigns each span an LQM/MQM-style error category. It is TTLab's system for AlexandriaX-2026 Subtask 3 and is described in TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification (Abusaleh, Verma & Mehler, ArabicNLP 2026).

This is the exact checkpoint of our official leaderboard submission (focal loss, rare categories collapsed into other), which ranked 3rd in AlexandriaX-2026 Subtask 3 with an official test Overall Score of 40.91. Running the bundled inference.py on the released test set reproduces the submitted predictions exactly (145/145 sentences).

A revised variant without category collapsing, which scored higher post-hoc, is available at alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026-nocollapse.

• —Code: https://github.com/aliabusaleh/arabic-dialectal-mt-error-span-detection
• —Data: `UBC-NLP/AlexandriaX_Subtask_3` (1,125 train / 138 dev sentences)
• —Dialect directions: ENG_EGY (Egyptian), ENG_MAU (Mauritanian), ENG_MOR (Moroccan), ENG_PAL (Palestinian), ENG_UAE (Emirati)

How it works

The model tags each sub-word token of the Arabic MT output only (the English source is not an input) over the labels O, morphosyntax, semantics, sociolinguistics, other. At inference, a token is flagged as an error when 1 - P(O) reaches a dialect-specific threshold τ_d, and it takes the most probable error category. Contiguous same-category tokens are merged, mapped back to character offsets with the tokenizer's offset mapping, trimmed of whitespace and punctuation, and spans shorter than 2 characters are dropped.

The head predicts O, morphosyntax, semantics, sociolinguistics and other. other is the three rarest categories (orthography, pragmatics, graphetics) merged during training, and it is mapped back to orthography_writing_conventions (the most frequent of the three) at decoding time, via label_map in span_decoding.json. As a result, pragmatics and graphetics are never predicted.

The thresholds are stored in span_decoding.json:

Usage

The span decoding (thresholds + merging) lives in inference.py, which ships in this repo:

from huggingface_hub import hf_hub_download
import importlib.util, sys

spec = importlib.util.spec_from_file_location("inference", hf_hub_download("alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026", "inference.py"))
inference = importlib.util.module_from_spec(spec); spec.loader.exec_module(inference)

tagger = inference.ErrorSpanTagger("alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026")
mt = "يلا خلّك وياها، هذا خرابيط. انت ما تدري إذا هي تأن ولا لا، وإذا هي تأن يسمونه أنين ليلي. نحنا بعدنا وايد بدري."
print(tagger.predict(mt, direction="ENG_UAE"))
# [{'text': ..., 'start': ..., 'end': ..., 'category': ...}, ...]

Batch tagging of a JSONL file (id, direction, source, model_prediction per line), writing the Codabench LQM_prediction_tagged_errors format:

python inference.py alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026 test.jsonl predictions.jsonl

To get raw per-token probabilities, the checkpoint loads as a standard BertForTokenClassification:

from transformers import AutoTokenizer, AutoModelForTokenClassification
tok = AutoTokenizer.from_pretrained("alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026")
model = AutoModelForTokenClassification.from_pretrained("alighabusaleh/marbertv2-arabic-dialect-mt-error-span-detection-alexandriax2026")

For an unseen direction, the decoder falls back to τ = 0.4.

Training

Results

Overall = (overlap-span F1 + category F1) / 2, in percent. The official Codabench metric uses macro averages over directions. Our dev "proxy" uses character-overlap F1 + category micro-F1 and is not directly comparable.

The paper's dev number for this system (40.8) comes from the model-selection run in the development notebook. This checkpoint is the one retrained on the full training set for the test submission, so its dev score differs.

Limitations

• —Tiny, imbalanced data. About 80% of tokens carry no error; sociolinguistics accounts for 58% of spans and graphetics has 2 training examples. Localisation is reasonable (overlap F1 ≈ 46–48), but category assignment is the bottleneck: morphosyntax, pragmatics and graphetics are rarely or never recovered.
• —Source-blind. Only the Arabic MT output is encoded, so errors that are only detectable by comparison with the English source (omissions, mistranslations of fluent-looking text) are partly out of reach.
• —Domain. Trained on five dialect directions of a single shared-task dataset. Thresholds are tuned per direction, so other dialects or domains will need re-tuning.
• —Intended for research on Arabic MT quality estimation. It is not a substitute for human MT evaluation.

Citation

If you use this model, please cite:

@inproceedings{Abusaleh:et:al:2026:mt,
  title     = {TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic
               Machine-Translation Error-Span Detection and Classification},
  author    = {Ali Abusaleh and Bhuvanesh Verma and Alexander Mehler},
  booktitle = {SIGARAB ArabicNLP 2026 AlexandriaX Shared Task},
  year      = {2026},
  address   = {Budapest, Hungary},
  note      = {accepted}
}

The model builds on MARBERTv2 (Abdul-Mageed et al., ACL 2021) and the AlexandriaX-2026 shared-task data; please cite those as well where appropriate.