SatorTenet/gemma-4-E2B-it-serbian-dpo-lora

gemma-4-E2B-it — Serbian LoRA (SFT + DPO)

The first published Serbian fine-tune of gemma-4-E2B (and, at the time of publishing, of any gemma-4 model). A 101 MB–268 MB LoRA adapter that makes the 2B edge-class Gemma write measurably better Serbian — both scripts, clean form, no script mixing — while keeping base knowledge intact (paired McNemar guard on every training step) and partially repairing the overconfidence that instruction tuning itself introduces.

Runs in ~3.8 GB VRAM with full 131k context under llama.cpp.

• —Base: google/gemma-4-E2B-it · Method: QLoRA SFT (form) → DPO (confidence calibration) · 62M trainable params (1.55 %)
• —Trained on: a single RTX 3060 12 GB (DPO stage: 30 min)
• —Data: fully synthetic, generated with Google models (local Gemma-4-12B + Gemini); no Anthropic/OpenAI model outputs in the training data
• —License: Apache 2.0 (same as the base model)

What it does (and honestly does not)

Examples (greedy, actual outputs):

Q: Koje godine je Nikola Tesla dobio Nobelovu nagradu za fiziku? A: Nikola Tesla nije dobio Nobelovu nagradu za fiziku. …

Q: U kojoj srpskoj pokrajini se nalazi grad Dubrovnik? A: Grad Dubrovnik se ne nalazi u Srbiji. On se nalazi u Hrvatskoj…

The measured story

We tracked 20 false-premise trap questions (own SRZ-1 eval, frozen classifier + mandatory manual review) across every method. Confabulation count, lower is better:

Three findings we believe transfer beyond this project:

1. 1.SFT teaches form but destroys epistemic caution and cannot restore it. Abstention went 7/20 → 0/20 after any SFT; even 13 % diverse denial examples in the SFT mix left confabulation at ~17/20.
2. 2.Fine-tuning does not inject retrievable facts into a 2B model. In a controlled run (3,000 facts, dose ladder, untrained control arm) the control arm moved most; trained arms stayed flat.
3. 3.DPO is the only method that moved the confidence prior — with real generalization (new denials are on entities that were banned from the training set) — but it plateaus: +73 % pairs and +1 epoch bought ~2 traps; epochs 2 and 3 deny an identical trap set.

Knowledge guard: paired McNemar on MC accuracy after every step — final Δ latin −0.012 (p=0.42), cyrillic −0.004 (p=0.23) vs. the SFT checkpoint.

Usage

llama.cpp (recommended — 101 MB GGUF adapter included)

llama-server -m gemma-4-E2B-it-Q6_K.gguf \
  --lora gguf/lima_dpo_adapter_v2_f16.gguf \
  -ngl 99 -c 131072 --jinja --reasoning off

`--reasoning off` is required. Without it the gemma-4 chat template puts the model in thinking mode and message.content comes back empty (everything lands in reasoning_content). Alternatively pass "chat_template_kwargs": {"enable_thinking": false} per request.

transformers + PEFT

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("google/gemma-4-E2B-it",
                                            device_map="auto", load_in_4bit=True)
model = PeftModel.from_pretrained(base, "SatorTenet/gemma-4-E2B-it-serbian-dpo-lora")
tok = AutoTokenizer.from_pretrained("SatorTenet/gemma-4-E2B-it-serbian-dpo-lora")

msgs = [{"role": "user", "content": "Napiši mi tri rečenice o Beogradu."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids.to(model.device), max_new_tokens=200)[0]))

Training details

• —Stage 1 — SFT (form): LIMA-style instruction set with masked-prompt loss, 2 epochs; fixed script mixing, repetition and formatting.
• —Stage 2 — DPO (calibration): 465 preference pairs — chosen = audited denials/abstentions, rejected = the SFT model's own greedy confabulations on the same premises; β=0.1, lr 5e-6, 3 epochs, manual DPO loop with reference log-probs precomputed. An entity guard (with Cyrillic transliteration) ensured zero overlap between training entities and eval traps — measured denials are generalization, not memorization.

Limitations

• —Factual knowledge is that of a 2B model: it still confabulates on 12/20 trap questions. Do not use it for factual QA without retrieval.
• —MC accuracy end-to-end: 0.478 latin / 0.406 cyrillic vs 0.514 / 0.432 base — a small cumulative tax from the SFT stage (guard-clean per step).
• —Occasional invented detail or ijekavian slip in free prose.
• —Own evaluation (SRZ-1, 20 traps) — small samples, wide intervals.

Files