Modelpublic

randomDude26/gemma_mktg_sft

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

0likes4downloads

Model Card

Gemma-3-4B-IT — joint SFT for Mizo, Khasi, Tulu, Gondi

Gemma-3-4B-IT supervised fine-tuned jointly on four low-resource Indian language pairs. Instruction-style prompting; no continued pretraining.

A LoRA adapter — load it on top of the base model google/gemma-3-4b-it.

Training data

Fine-tuned jointly on four low-resource pairs from `randomDude26/mi_kh_tulu_dataset`:

Pair	Train	Val	Test
`en-mizo` (English → Mizo, `lus`)	49,496	464	800
`en-khasi` (English → Khasi, `kha`)	22,358	482	800
`en-tulu` (English → Tulu, `tcy`)	16,218	488	800
`hi-gondi` (Hindi → Gondi, `gon`)	15,500	389	737

The validation and test splits are decontaminated: no exact or near duplicate of any test/val sentence appears in training (exact pair, exact source, exact target, ≥0.90 character similarity, ≥0.80 4-gram containment, ≥0.80 token Jaccard). All scores below are on those decontaminated test sets.

Gondi has 737 test sentences rather than 800 — after decontamination only 737 clean sentences remain.

Results

Decontaminated test sets. BLEU and chrF++ via sacrebleu (BLEU(effective_order=True), CHRF(word_order=2)).

Direction	n	BLEU	chrF++
`eng→mizo`	800	31.56	57.70
`eng→khasi`	800	34.48	58.89
`eng→tulu`	800	34.36	62.45
`hi→gondi`	737	5.55	29.72

Checkpoint used for these scores: gemma-sft-joint-20260713_170412/checkpoint-3500

Against the zero/few-shot baselines (BLEU)

Direction	Gemma 0-shot	Gemma 3-shot	this model
`eng→mizo`	6.99	10.86	31.56
`eng→khasi`	3.16	3.69	34.48
`eng→tulu`	4.32	4.16	34.36
`hi→gondi`	2.44	2.64	5.55

Configuration

LoRA


rank `r`	64
`lora_alpha`	128
`lora_dropout`	0.05
target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
`modules_to_save`	—

Training


batch size	16 (per device), grad-accum 2 → effective 32
learning rate	2e-4, cosine, 200 warmup steps
weight decay	0.01
max sequence length	512 (prompt + response)
precision	bf16, gradient checkpointing
decoding	greedy

Checkpoints are saved every 500 steps. Checkpoint 3500 is reported: validation loss bottoms out earlier (~step 2500), but validation BLEU keeps improving to 3500 — the two diverge, and checkpoints were selected on BLEU.

Usage

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

REPO = "randomDude26/gemma_mktg_sft"

tokenizer = AutoTokenizer.from_pretrained(REPO)
tokenizer.padding_side = "left"

base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-3-4b-it", dtype=torch.bfloat16
)
model = PeftModel.from_pretrained(base, REPO).merge_and_unload().cuda().eval()

# the prompt MUST match the training format
prompt = (
    "<start_of_turn>user\n"
    "Translate the following English sentence into Tulu.\n"
    "Output only the translated sentence. Do not include explanations.\n\n"
    "The temple is very old.<end_of_turn>\n"
    "<start_of_turn>model\n"
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Citation

Paper under anonymous review. Dataset: `randomDude26/mi_kh_tulu_dataset`.