Hailay/VEXMLM-TiQuAD

VEXMLM — TiQuAD

Tigrinya question answering fine-tuned from `Hailay/VEXMLM`, the vocabulary-extended XLM-R for Ge'ez-script languages.

Official implementation: https://github.com/hailaykidu/VEXMLM

TiQuAD is a supplementary task in the VEXMLM paper — a diagnostic evaluation, not a reported paper benchmark. Its 926 development questions make it a more stable measurement than TIGQA's 67, and it is the stronger Tigrinya QA checkpoint of the two.

Five-seed benchmark evaluation

Fine-tuned independently under seeds 42–46 with one configuration (hash ce27cc194946) on an A100-PCIE-40GB. Reported as mean ± standard deviation over the five runs, on the dataset's development split (the fine-tuning script evaluates QA on the validation split when one exists).

These are the paper's verified results. They come from the five-seed evaluation described above — not from interactive use.

Interactive inference vs. benchmark

Benchmark evaluation is the five-seed measurement on the held-out development split, shown in the table above.

Interactive inference is what the usage example below performs: Supply a Tigrinya context and question; the model returns an extracted span. Predictions on arbitrary user input are demonstrations only and do not produce or reproduce the benchmark score.

Repository layout

Five independently fine-tuned checkpoints, one per seed. The reported benchmark score is the mean ± standard deviation over all five; no single seed is the "five-seed model."

seed-42/  seed-43/  seed-44/  seed-45/  seed-46/

Load a specific seed with the subfolder argument, as in the example below.

Fine-tuning

Fine-tuned from `Hailay/VEXMLM`, a vocabulary-extended XLM-R (280,002 subwords, 30,000 Ge'ez tokens merged into the SentencePiece model) after continued MLM pretraining.

Runs are bit-reproducible: enable_full_determinism, CUBLAS_WORKSPACE_CONFIG=:4096:8, dataloader_num_workers=0.

Usage

from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

repo = "Hailay/VEXMLM-TiQuAD"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="seed-42")
model = AutoModelForQuestionAnswering.from_pretrained(repo, subfolder="seed-42")
model.eval()

question = "ኤርትራ ኣብ ኣየናይ ክፍለ ዓለም ትርከብ?"
context  = "ኤርትራ ኣብ አፍሪቃ ክፍለ ዓለም እትርከብ ሃገር እያ።"
enc = tokenizer(question, context, return_tensors="pt", truncation=True, max_length=256)

with torch.no_grad():
    out = model(**enc)

start = out.start_logits.argmax()
end = out.end_logits.argmax()
print(tokenizer.decode(enc.input_ids[0][start:end + 1], skip_special_tokens=True))

Limitations

• —Fine-tuned for Tigrinya on TiQuAD only; performance on other languages, domains or label schemes is not characterised.
• —The base model covers Amharic and Tigrinya; other Ge'ez-script languages were not part of pretraining.
• —The pretraining text is of undocumented origin (licence unknown); samples contain religious translations alongside general web prose, and the model may reflect those distributions and any biases present in them.
• —Single-configuration study: no hyperparameter search was performed, and baseline comparisons in the paper are single-seed.

Reproducibility

The fine-tuning launcher, evaluation code and per-run result records are in the official repository: https://github.com/hailaykidu/VEXMLM

sbatch scripts/slurm_stage2_spm_seeds.sh    # 6 tasks × 5 seeds
python3 evaluation/export_spm_results.py    # regenerates the metrics table

Citation

@inproceedings{teklehaymanot2026vexmlm,
  title     = {Vocabulary Expansion for Low-Resource African Languages:
               A Case Study in Amharic and Tigrinya},
  author    = {Teklehaymanot, Hailay Kidu and Yadeta, Debela Desalegn and
               Nejdl, Wolfgang},
  booktitle = {Proceedings of the Workshop on Language Models for
               Underserved Communities (LM4UC) at IJCAI},
  year      = {2026}
}

Accepted at the LM4UC Workshop, IJCAI 2026.

License

Apache 2.0, following xlm-roberta-base.