Modelpublic

nadaashraff/smollm2-135m-squad-lora

sourceHugging Faceapache-2.0updated 10mo agoView on Hugging Face

0likes

Model Card

smollm2-135m-squad-lora

This is a LoRA fine-tuned version of the SmolLM2-135M model trained on 5,000 samples from the SQuAD v1.1 question-answering dataset. The goal of this fine-tune is to improve extractive question answering on short factual passages.

Model Details

—Base model: HuggingFaceTB/smollm2-135m
—Fine-tuning method: LoRA (efficient parameter-efficient fine-tuning)
—Dataset: 5k examples from SQuAD v1.1 (train/dev split)
—Epochs: 2
—Batch size: 4
—Loss after fine-tuning: ~2.21
—Hardware: Google Colab T4 GPU
—Quantization: 4-bit (bitsandbytes)

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "HuggingFaceTB/smollm2-135m"
LORA = "nadaashraff/smollm2-135m-squad-lora"

tokenizer = AutoTokenizer.from_pretrained(BASE)
base_model = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto")
model = PeftModel.from_pretrained(base_model, LORA)

prompt = """Read the following context and answer the question.

Context: Egypt ranks as the fifth worst country in the world for religious freedom. According to a 2010 Pew survey, 84% of Egyptians polled supported the death penalty for those leaving Islam.

Question: What percentage of Egyptians polled supported the death penalty for those leaving Islam?
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))