Nimra28/urdu-llama-pakistan

🇵🇰 Urdu LLaMA — Pakistan's First Fine-tuned Urdu LLM

Llama 3.2-1B fine-tuned on a Pakistani corpus — Urdu news, literature, Islamic knowledge, history, culture and code-switching (Urdu+English)

Model Summary

Urdu LLaMA is Pakistan's first open-source instruction-following language model fine-tuned specifically on Pakistani and Urdu content. It is built on top of meta-llama/Llama-3.2-1B-Instruct and fine-tuned using QLoRA (4-bit quantization + LoRA) on a curated Pakistani corpus.

The model understands and generates text in:

• —🇵🇰 Urdu (primary language)
• —🌐 English (secondary)
• —💬 Code-switching (Urdu + English mixed — as spoken by Pakistanis daily)

🚀 Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Nimra28/urdu-llama-pakistan"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

def ask_urdu(question):
    prompt = f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>
آپ ایک مددگار پاکستانی AI اسسٹنٹ ہیں جو اردو اور انگریزی میں جواب دے سکتے ہیں۔<|eot_id|><|start_header_id|>user<|end_header_id|>
{question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=200,
            temperature=0.7,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )
    response = tokenizer.decode(
        outputs[0][inputs["input_ids"].shape[1]:],
        skip_special_tokens=True
    )
    return response.strip()

# Try it
print(ask_urdu("پاکستان کی تاریخ بتائیں"))
print(ask_urdu("علامہ اقبال کون تھے؟"))
print(ask_urdu("Machine learning کیا ہے؟"))

📚 Training Data

The model was fine-tuned on a curated Pakistani corpus covering:

⚙️ Training Details

📊 Evaluation

The model was evaluated on held-out Urdu test examples:

• —Perplexity: Improved significantly vs base model on Urdu text
• —Qualitative evaluation: Model responds correctly in Urdu to Pakistani cultural, historical, and Islamic questions
• —Code-switching: Handles natural Urdu+English mixing as spoken by Pakistanis

💬 Example Outputs

Question: پاکستان کب آزاد ہوا؟

پاکستان 14 اگست 1947 کو برطانوی استعمار سے آزاد ہوا۔ یہ دن ہر سال یوم آزادی کے طور پر منایا جاتا ہے۔ قائداعظم محمد علی جناح پاکستان کے پہلے گورنر جنرل بنے۔

Question: Machine learning کیا ہے؟

Machine learning یعنی مشینی سیکھ ایک ایسی technology ہے جس میں computer خود بخود data سے سیکھتا ہے۔ جیسے ہم experience سے سیکھتے ہیں، ویسے ہی machine بھی examples سے patterns پہچاننا سیکھتی ہے۔

Question: علامہ اقبال کون تھے؟

علامہ اقبال پاکستان کے قومی شاعر اور مفکر تھے۔ انہوں نے 1930 میں مسلمانوں کے لیے ایک الگ وطن کا خواب پیش کیا۔ انہیں شاعر مشرق اور مفکر پاکستان کہا جاتا ہے۔

⚠️ Limitations

• —Fine-tuned on a relatively small dataset (~30 curated examples) — larger dataset will improve quality
• —May occasionally mix Urdu and English even when pure Urdu is requested
• —Islamic knowledge is general — not a substitute for qualified scholars
• —Political opinions reflect training data — use with caution

🔮 Future Work

• —[ ] Scale training data to 10,000+ Urdu examples
• —[ ] Add Urdu news datasets from Dawn, Geo, ARY
• —[ ] Fine-tune on Urdu Wikipedia dump
• —[ ] Add Punjabi, Sindhi, Pashto support
• —[ ] Release larger 7B parameter version
• —[ ] Benchmark on Urdu NLP tasks

📄 Citation

If you use this model, please cite:

@misc{urdu-llama-pakistan-2024,
  title={Urdu LLaMA: Pakistan's First Fine-tuned Urdu Language Model},
  author={Nimra Tariq},
  year={2024},
  publisher={HuggingFace},
  url={https://huggingface.co/Nimra28/urdu-llama-pakistan}
}

👩‍💻 Developed By

Nimra Tariq — AI Engineer & Assistant Professor, Superior University, Pakistan

• —🐙 GitHub: github.com/nimra-pixel
• —🤗 HuggingFace: huggingface.co/Nimra28

📜 License

This model is built on Llama 3.2 and follows the Llama 3.2 Community License.