Modelpublic

sinhala-nlp/SinLlama-PaliSinhala-Pi2Si-en

sourceHugging Facellama3updated 2mo agoView on Hugging Face

0likes7downloads

Model Card

SinLlama-PaliSinhala-Pi2Si-en

Built with Meta Llama 3.

A LoRA adapter for Pali to Sinhala machine translation, trained on sinhala-nlp/pali-sinhala as part of the SinGen Sinhala text generation benchmark.

The task adapter is stacked on top of polyglots/SinLlama_v01, a Sinhala continual-pretraining adapter over meta-llama/Meta-Llama-3-8B with an extended Sinhala tokenizer.

Usage

The base model must be loaded with the extended tokenizer and resized embeddings, the SinLlama adapter merged in, and only then this task adapter applied:

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("polyglots/Extended-Sinhala-LLaMA")
base = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B", dtype="auto", device_map="auto")
base.resize_token_embeddings(len(tok))
model = PeftModel.from_pretrained(base, "polyglots/SinLlama_v01").merge_and_unload()
model = PeftModel.from_pretrained(model, "sinhala-nlp/SinLlama-PaliSinhala-Pi2Si-en")

Prompts follow the Alpaca format used during training, ending at ### Response: with the model continuing from the Translation: prefix. Leading verse/section numbers were stripped from both sides of the corpus before training.

Training


Training pairs used	27383
Pairs dropped (target exceeded the sequence budget)	12
Instruction language	`en`
Epochs	3.0
Effective batch size	16
Learning rate	0.0002
Max sequence length	8192
LoRA r / alpha / dropout	16 / 32 / 0.05
Target modules	all-linear

Evaluation

Held-out tail of the corpus (1000 segments), whitespace-tokenized (sacreBLEU's default 13a tokenizer splits Sinhala conjuncts and vowel signs):

Metric	Score
Corpus sacreBLEU	0.51
Sentence-level BLEU mean	1.37

Caveats

—The split is positional (last 1000 rows). The corpus is ordered by canonical text and aligned at uneven granularity, so the test tail contains far longer segments than the training body: it is not a sample of the same distribution.
—10 of 1000 references exceed the generation budget of 3000 tokens; those predictions are cut off and penalised by BLEU's brevity term.
—The Pali canon repeats stock passages verbatim, so near-duplicate overlap between splits is possible even where exact overlap is zero.

Licence

Derived from Meta Llama 3 and governed by the Meta Llama 3 Community License.