emilyyy04/xlm-roberta-base-burmese-nli-v3

mmNLI: Burmese Text Natural Language Inference with XLM-RoBERTa

A Burmese NLI model fine-tuned from xlm-roberta-base, trained on a carefully cleaned and curated dataset, achieving 79% accuracy on the test set.

This model predicts the relationship between a premise and a hypothesis as one of:

• —Entailment
• —Neutral
• —Contradiction

Model & Demo

• —Model >> https://huggingface.co/emilyyy04/xlm-roberta-base-burmese-nli-v3
• —Space Demo Link >> https://huggingface.co/spaces/emilyyy04/burmese-text-nli-xlmroberta

Model Details

• —Base model: xlm-roberta-base
• —Language: Burmese (Myanmar)
• —Task: Natural Language Inference (NLI)
• —Labels: entailment, neutral, contradiction
• —Framework: Transformers / PyTorch ---

Dataset and its Structure

The dataset consists of ~10k [10,443] samples across three classes:

and the dataset is prepared from:

• —By clearning Burmese NLI data (source: [(https://huggingface.co/datasets/akhtet/myanmar-xnli)]) and additional manually created samples.
• —Translated English NLI data for diversity.
• —Most samples follow a 1 premise → 3 hypotheses structure. Each hypothesis has a different NLI label.
• —An `genre` field is included intended for future zero-shot / cross-genre experiments later.

Preprocessing

Steps:

1. 1.Unicode normalization (NFC)
2. 2.Zawgyi detection
3. 3.Automatic conversion to Unicode if Zawgyi text is detected
4. 4.Rely on XLM-R subword tokenizer for tokenization

Data Splitting Strategy

To prevent data leakage caused by shared premises:

• —Train: 70%
• —Validation: 15%
• —Test: 15%

Instead of random shuffling:

• —GroupShuffleSplit is used
• —Samples with the same premise always stay in the same split
• —Prevents:

• —Premise overlap across splits
• —Hypothesis leakage between train / validation / test sets ---

Training Setup

The model was trained using Hugging Face Transformers with the following key configurations:

• —Model: facebook/xlm-roberta-base
• —Epochs: Up to 10 epochs
• —Early Stopping: Enabled (patience = 1)
• —Learning Rate: 1.5e-5
• —Batch Size: 16 (train & evaluation)
• —Weight Decay: 0.02
• —Warmup Ratio: 0.1
• —FP16 Training: Enabled
• —Best Model Selection: Based on F1-score
• —Seed: 42

The best checkpoint was automatically loaded at the end of training using early stopping and F1-based model selection.

Evaluation Metrics

The model performance is evaluated using:

• —Accuracy
• —Macro F1-score

Training Results

Training stopped early at epoch 6 due to validation performance plateau and the best model saved is at the f1 score of 0.7646.

Test Set Performance

{
  "eval_loss": 0.5780,
  "eval_accuracy": 0.7877,
  "eval_f1": 0.7876
}

Confusion Matrix on Test Set

Label order: entailment, neutral, contradiction

[[481  50  18]
 [ 78 418  20]
 [ 14  20 476]]

Rows represent true labels, columns represent predicted labels.

Classification Report (Test Set)

Inference Example

You can use the model as follows:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "emilyyy04/xlm-roberta-base-burmese-nli"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

premise = "ဟုတ်တယ်ဗျခင်ဗျားပြောတာမှန်တယ်အခုခေတ်မှာလူငယ်တွေကနည်းပညာကိုအရင်ကထက်ပိုပြီးကျွမ်းကျင်လာကြတော့အလုပ်အကိုင်အခွင့်အလမ်းသစ်တွေအများကြီးပေါ်လာတာပေါ့"
hypothesis = "နည်းပညာတိုးတက်လာတာကလူငယ်တွေအတွက်အခွင့်အလမ်းကောင်းတွေဖြစ်စေတယ်"

inputs = tokenizer(
    premise,
    hypothesis,
    return_tensors="pt",
    truncation=True,
    padding=True,
    max_length=128
)

outputs = model(**inputs)
predicted_class = torch.argmax(outputs.logits, dim=1).item()

label_map = {0: "entailment", 1: "neutral", 2: "contradiction"}
print("Predicted label:", label_map[predicted_class])
# conf
probs = torch.softmax(outputs.logits, dim=-1)[0]
print("Confidence:", {k: round(float(probs[i]), 3) for i, k in label_map.items()})

Our Team

This project is part of our Semester-8 CS-502 Natural Language Processing course.

Limitations & Future Work

• —Genre-aware and zero-shot classification is planned but not yet implemented
• —Performance may vary for:

• —Very long inputs
• —Out-of-domain or highly informal Burmese
• —Future improvements:

• —Larger native Burmese NLI dataset
• —Explicit genre-based evaluation
• —Domain adaptation