Modelpublic

greta44/albanian-spelling-lora

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes10downloads

Model Card

Albanian Spelling Exercise LoRA

LoRA adapter for automatic generation of Albanian spelling exercises (primary-school orthography), trained for PhD research in educational NLP.

—Base model: `Qwen/Qwen2.5-0.5B-Instruct`
—Method: PEFT LoRA (r=16, alpha=32)
—Language: Albanian (sq)
—Related dataset: `greta44/albanian-error-augmentation`

Safety note: Treat model output as a proposal only. For children, the final correct answer must come from deterministic Albanian rules / a curated database — not from the LLM alone.

Intended use

Generate short Albanian orthography exercise proposals, for example:

—missing_letter — plotëso shkronjën
—find_error — gjej gabimin
—explain_error — shpjego gabimin (pedagogjikisht)

Usage (Python demo)

1) Install

bash

pip install torch transformers peft accelerate sentencepiece

2) Load the adapter and generate

python

import json
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

ADAPTER_ID = "greta44/albanian-spelling-lora"
BASE_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_ID, use_fast=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(BASE_MODEL)
model = PeftModel.from_pretrained(model, ADAPTER_ID)
model.eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

seed_word = "mirë"
grade = 3
exercise_type = "missing_letter"  # or: find_error, explain_error

payload = {
    "seed_word": seed_word,
    "grade": grade,
    "difficulty": "easy",
    "exercise_type": exercise_type,
    "safety": "Kthe vetëm propozim; përgjigjja finale kontrollohet nga rregullat.",
}

prompt = (
    "### Instruksion:\n"
    f"Gjenero një ushtrim të sigurt për drejtshkrimin shqip. Kategoria: {exercise_type}. "
    f"Klasa: {grade}. Vështirësia: easy. Fjala bazë: {seed_word}.\n\n"
    "### Input:\n"
    + json.dumps(payload, ensure_ascii=False)
    + "\n\n### Përgjigje:\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.4,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )

text = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(text.strip())

3) Or use the included script

bash

python generate_exercises.py \
  --adapter greta44/albanian-spelling-lora \
  --seed-word mirë \
  --grade 3 \
  --type missing_letter

Training summary

Item	Value
Train / eval split	686 / 77 (from 763 AlbLingo exercises, seed=42)
LoRA rank / alpha / dropout	16 / 32 / 0.05
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`
Max length	256
Effective batch size	4
Hardware	Apple M1 (MPS, float32)
Protocol target	3 epochs
Reported run	instrumented Phase A + Phase B continuation (~1.4 epoch-equivalent)
Train loss	3.149 → min 0.276
Held-out eval_loss	0.345
Approx. wall-clock	~62 minutes (logged span)

See training_manifest.json in this repository for the full measured summary, and training_loss.png for the loss curve.

Files in this repo

—adapter_model.safetensors + adapter_config.json — LoRA weights
—tokenizer* / vocab.json / merges.json — tokenizer files
—generate_exercises.py — standalone demo script
—training_manifest.json — hyperparameters + measured metrics
—training_loss.png — training loss curve
—instruction_dataset.jsonl — sample instruction pairs

Citation

If you use this model, please cite the related PhD research on Albanian educational NLP / AlbLingo and link this repository.

License

Apache 2.0