eluve-inc/SEA-LION-ModernBERT-600M-multilingual-NLI-26lang-2mil7

SEA-LION ModernBERT 600M Multilingual NLI 26lang 2mil7

SEA-LION ModernBERT 600M Multilingual NLI 26lang 2mil7 is a natural language inference (NLI) model based on the ModernBERT architecture. The model was originally the pre-trained aisingapore/SEA-LION-ModernBERT-600M model, which is then fine-tuned on MoritzLaurer/multilingual-NLI-26lang-2mil7. The model is then evaluated zero-shot on `IndoNLI`.

Evaluation Results

Zero-shot accuracy on IndoNLI:

How to Use

As Natural Language Inference (NLI) Model

from transformers import pipeline

model_name = "eluve-inc/SEA-LION-ModernBERT-600M-multilingual-NLI-26lang-2mil7"

clf = pipeline("text-classification", model=model_name, tokenizer=model_name, top_k=None, device="cuda:0")

premise = "Minggu lalu saya mulai batuk pilek, dan sekarang saya sudah sembuh."
hypothesis = "Pasien sudah tidak lagi mengalami gejala batuk pilek."

out = clf({"text": premise, "text_pair": hypothesis})
# [{'label': 'entailment', 'score': 0.8801426887512207},
#  {'label': 'neutral', 'score': 0.08782942593097687},
#  {'label': 'contradiction', 'score': 0.03202790021896362}]

Model

Model Configuration

• —labels: entailment (0), neutral (1), contradiction (2)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 2e-05
• —train_batch_size: 256
• —gradient_accumulation_steps: 1
• —seed: 42
• —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• —lr_scheduler_type: cosine
• —warmup_ratio: 0.1
• —weight_decay: 0.01
• —num_epochs: 3
• —mixed_precision_training: bf16

References

[1] Mahendra, R., Aji, A. F., Louvan, S., Rahman, F., & Vania, C. (2021, November). IndoNLI: A Natural Language Inference Dataset for Indonesian. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

[2] Warner, B., Chaffin, A., Clavié, B., Weller, O., Hallström, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., Cooper, N., Adams, G., Howard, J., & Poli, I. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv preprint arXiv:2412.13663.