munkhbayar-batkhuu/Llama-3.2-3B-Instruct-Mongolian
Llama-3.2-3B-Instruct-Mongolian
A LoRA fine-tuned version of meta-llama/Llama-3.2-3B-Instruct for Mongolian language instruction-following and chat.
Model Description
This model adapts Llama 3.2 3B Instruct to understand and generate fluent Mongolian text. It was fine-tuned using LoRA (Low-Rank Adaptation) on the saillab/alpaca-mongolian-cleaned dataset containing ~41,600 Mongolian instruction-following examples.
The base model struggles with Mongolian, producing garbled or incoherent text. After fine-tuning, the model generates fluent, coherent Mongolian responses across a wide range of topics.
Quick Start
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch
BASE_MODEL = "meta-llama/Llama-3.2-3B-Instruct"
ADAPTER = "munkhbayar-batkhuu/Llama-3.2-3B-Instruct-Mongolian"
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
model = AutoModelForCausalLM.from_pretrained(
BASE_MODEL, torch_dtype=torch.float16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()
messages = [
{"role": "system", "content": "You are a helpful assistant that responds in Mongolian."},
{"role": "user", "content": "Монгол улсын нийслэл хаана байдаг вэ?"},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to(model.device)
with torch.no_grad():
output = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.9, do_sample=True)
response = tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(response)
# Output: Монгол улсын нийслэл нь Улаанбаатар хот юм.Training Details
Chat Template
Training used the Llama 3.2 chat template with system prompt: "You are a helpful assistant that responds in Mongolian."
Benchmark Results
MM-Eval (Mongolian Multi-task Evaluation)
MM-Eval (arXiv:2411.09492) is a hierarchical benchmark for evaluating LLMs on Mongolian language tasks across 1,840 items.
Perplexity (on eval split, 2,081 samples)
BLEU / ROUGE-L (200 eval samples)
Example Outputs
Prompt: "Монгол улсын нийслэл хаана байдаг вэ?" (Where is the capital of Mongolia?)
Prompt: "Хүүхдэд зориулж богино үлгэр бичнэ үү." (Write a short story for children.)
Prompt: "Монгол хоолны жор бичнэ үү." (Write a Mongolian food recipe.)
Limitations
- Domain: Trained on general instruction-following data; may not perform well on specialized domains (medical, legal, technical)
- Math/Reasoning: Mathematical reasoning did not improve (slightly declined on MM-Eval reasoning)
- Hallucination: Like all LLMs, may generate plausible but factually incorrect information
- Sequence Length: Trained with max 512 tokens; may degrade on longer inputs
- Model Size: 3B parameters -- larger models would likely achieve better results
Framework Versions
- PEFT: 0.18.1
- TRL: 0.27.2
- Transformers: 5.1.0
- PyTorch: 2.10.0+cu128
- Datasets: 4.5.0
Citation
If you use this model, please cite:
@misc{llama32-3b-mongolian-2026,
title={Llama-3.2-3B-Instruct-Mongolian},
author={Munkhbayar Batkhuu},
year={2026},
url={https://huggingface.co/munkhbayar-batkhuu/Llama-3.2-3B-Instruct-Mongolian}
}