tachelhit-lab/tash_tfngh_pissa_medical

tashtfnghpissa_medical

PEFT PiSSA adapter (rank 32) for facebook/nllb-200-distilled-600M, English → Tachelhit (Tifinagh).

Continues training from the Stage 1 general-domain PiSSA adapter (`ilyasbejja/tash_tfngh_translator`) on 4,182 medical sentence pairs, with a reduced learning rate (1e-5, 5× lower than Stage 1) and KL anti-forgetting regularization against the Stage 1 checkpoint (weight 0.1, temperature 2.0).

⚠️ Language code caveat: tzm_Tfng is an NLLB routing proxy, not an established Tachelhit-specific language code. Documented limitation, not a labeling error.

⚠️ Note on `summary.json`: the hyperparams.rank field in that file reads 16, but this is stale — the adapter's actual weights (and adapter_config.json) confirm rank 32. Everything else in summary.json (losses, metrics, effective-rank diagnostics) is accurate.

Results

Medical test set (520 sentences):

General-domain held-out set (14,741 sentences) — catastrophic forgetting check:

Medical chrF++ gain is bootstrap-significant (95% CI before: [10.03, 10.76], after: [21.04, 23.15], p ≈ 0). The general-domain drop reflects a real specialization/forgetting trade-off at this configuration — see summary.json for the full metrics, effective-rank diagnostics, and per-layer stable-rank breakdown.

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")
model = PeftModel.from_pretrained(base, "tachelhit-lab/tash_tfngh_pissa_medical")
tokenizer = AutoTokenizer.from_pretrained(
    "tachelhit-lab/tash_tfngh_pissa_medical", src_lang="eng_Latn", tgt_lang="tzm_Tfng"
)

inputs = tokenizer("Take two tablets twice a day.", return_tensors="pt")
tgt_id = tokenizer.convert_tokens_to_ids("tzm_Tfng")
out = model.generate(
    **inputs, forced_bos_token_id=tgt_id,
    no_repeat_ngram_size=3, repetition_penalty=1.3,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Training details

• —Base model: facebook/nllb-200-distilled-600M
• —PEFT method: PiSSA, rank 32
• —Stage 1 → Stage 2 LR: 5e-5 → 1e-5 (5× reduction)
• —Epochs: 5, batch size 4, grad accumulation 4
• —Anti-forgetting KL: weight 0.1, temperature 2.0, reference = Stage 1 checkpoint
• —Part of the IWSIF 2026 Tachelhit NMT paper comparing LoRA / MiLoRA / PiSSA.

License

Inherits cc-by-nc-4.0 from the base NLLB-200 model — non-commercial use only.