Modelpublic

randomDude26/mbart_mktg_sft

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

0likes3downloads

Model Card

mBART-50 — joint SFT for Mizo, Khasi, Tulu, Gondi

mBART-50 supervised fine-tuned jointly on four low-resource Indian language pairs, with three new language tokens added to the tokenizer.

A LoRA adapter — load it on top of the base model facebook/mbart-large-50.

Training data

Fine-tuned jointly on four low-resource pairs from `randomDude26/mi_kh_tulu_dataset`:

Pair	Train	Val	Test
`en-mizo` (English → Mizo, `lus`)	49,496	464	800
`en-khasi` (English → Khasi, `kha`)	22,358	482	800
`en-tulu` (English → Tulu, `tcy`)	16,218	488	800
`hi-gondi` (Hindi → Gondi, `gon`)	15,500	389	737

The validation and test splits are decontaminated: no exact or near duplicate of any test/val sentence appears in training (exact pair, exact source, exact target, ≥0.90 character similarity, ≥0.80 4-gram containment, ≥0.80 token Jaccard). All scores below are on those decontaminated test sets.

Gondi has 737 test sentences rather than 800 — after decontamination only 737 clean sentences remain.

Results

Decontaminated test sets. BLEU and chrF++ via sacrebleu (BLEU(effective_order=True), CHRF(word_order=2)).

Direction	n	BLEU	chrF++
`eng→mizo`	800	11.79	31.00
`eng→khasi`	800	19.59	38.96
`eng→tulu`	800	26.93	53.95
`hi→gondi`	737	6.45	31.03

Checkpoint used for these scores: mbart-sft-joint-upsample-20260525_223852/final_adapter

Against the zero/few-shot baselines (BLEU)

Direction	NLLB 0-shot	this model
`eng→mizo`	20.40	11.79
`eng→khasi`	4.50	19.59
`eng→tulu`	8.51	26.93
`hi→gondi`	2.22	6.45

Configuration

LoRA


rank `r`	64
`lora_alpha`	128
`lora_dropout`	0.1
target modules	`q_proj`, `k_proj`, `v_proj`, `out_proj`, `fc1`, `fc2`
`modules_to_save`	`embed_tokens`, `lm_head`

The embedding matrix and LM head are trained in full (modules_to_save) because three new language tokens — kha_Latn, tcy_Knda, gon_Deva — were added to the tokenizer. None of the four target languages is native to mBART-50.

Training


max steps	10,000
batch size	24 (per device), grad-accum 1
learning rate	3e-4, cosine, 200 warmup steps
weight decay	0.01
max sequence length	256
precision	bf16
decoding	beam search, 4 beams

Trained jointly on all four pairs, with upsampling of the lower-resource pairs.

Usage

python

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

REPO = "randomDude26/mbart_mktg_sft"

tokenizer = AutoTokenizer.from_pretrained(REPO)
base = AutoModelForSeq2SeqLM.from_pretrained(
    "facebook/mbart-large-50", dtype=torch.bfloat16
)
base.resize_token_embeddings(len(tokenizer), mean_resizing=False)

model = PeftModel.from_pretrained(base, REPO).merge_and_unload().cuda().eval()

tokenizer.src_lang = "en_XX"
inputs = tokenizer("The temple is very old.", return_tensors="pt").to("cuda")
out = model.generate(
    **inputs,
    forced_bos_token_id=tokenizer.convert_tokens_to_ids("tcy_Knda"),
    num_beams=4, max_new_tokens=256,
)
print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])

Citation

Paper under anonymous review. Dataset: `randomDude26/mi_kh_tulu_dataset`.