pradipbasnet68/gemma-3-4b-nepali-lyrics-lora

Gemma-3-4B Nepali Lyrics LoRA

A parameter-efficient LoRA adapter for google/gemma-3-4b-it, fine-tuned specifically for generating Nepali song lyrics in Devanagari script.

This model was developed as the final lyric-generation component of an undergraduate final-year project investigating end-to-end AI-based Nepali folk music generation.

The generated Nepali lyrics are subsequently passed to a separately fine-tuned ACE-Step 1.5 music-generation model to produce complete Nepali folk songs.

Model Details

• —Developed by: Pradip Raj Basnet
• —Base model: google/gemma-3-4b-it
• —Model type: PEFT LoRA adapter
• —Fine-tuning approach: QLoRA / parameter-efficient supervised fine-tuning
• —Primary language: Nepali
• —Writing system: Devanagari
• —Primary task: Nepali lyric generation
• —Framework: Hugging Face Transformers + PEFT
• —Adapter format: Safetensors
• —License: Gemma license, following the upstream base model

This repository contains the LoRA adapter weights only.

The original google/gemma-3-4b-it base model must also be loaded when using this adapter.

Project Context

This model is part of an end-to-end Nepali folk music generation pipeline.

The final project architecture is:

User Request → Gemma-3-4B + Lyrics LoRA → Nepali Lyrics → ACE-Step 1.5 + Nepali Folk LoRA → Generated Nepali Folk Song

The lyric model is responsible only for producing the Nepali textual lyrics.

Music and vocal generation are handled separately by the ACE-Step component.

Intended Use

The model is intended for experimental and academic generation of original Nepali song lyrics.

Example themes include:

• —village life
• —memories
• —nostalgia
• —romance
• —separation
• —love
• —happiness
• —emotional experiences
• —rural Nepali life
• —folk-oriented storytelling

The model was specifically optimized to generate text primarily in Nepali Devanagari script while reducing unwanted English or Roman-script contamination.

Out-of-Scope Use

This model is not intended for:

• —factual question answering
• —general-purpose conversational assistance
• —translation
• —medical, legal, or financial advice
• —safety-critical applications
• —authoritative cultural or linguistic judgement
• —generation of guaranteed publication-quality poetry

The model should be treated as a creative generation system rather than a factual knowledge source.

Training Data

The adapter was fine-tuned on a curated dataset of Nepali song lyrics collected and prepared for the associated academic project.

The dataset was cleaned with particular attention to:

• —Nepali Devanagari text
• —removal of unwanted Romanized lyrics
• —reduction of non-Nepali text
• —removal of unsuitable or malformed samples
• —lyric-oriented text formatting

The final training data focused on generating coherent Nepali lyrics suitable for downstream music generation.

Training Method

The model was fine-tuned using parameter-efficient LoRA adaptation rather than updating all parameters of Gemma.

The base model was quantized during training to reduce GPU-memory requirements while trainable LoRA matrices were attached to selected transformer modules.

LoRA Configuration

Only the adapter parameters are distributed in this repository.

Evaluation

The final fine-tuned model was compared against the original Gemma-3-4B-IT base model using held-out test data.

Quantitative Evaluation

The fine-tuned adapter achieved:

• —lower test loss
• —substantially lower perplexity
• —higher token-level accuracy

than the unmodified base model on the project test set.

Generation Quality Evaluation

The project also evaluated generated lyrics using prompt-based diagnostic testing.

The evaluation examined:

• —Devanagari-script consistency
• —presence of Latin characters
• —foreign-character contamination
• —repetition
• —unwanted meta-text
• —general lyric structure

A final benchmark using 20 generation prompts showed:

This means all outputs in that final test remained fully within the expected Devanagari script according to the project's automated character-based evaluation.

These measurements do not fully measure creativity, poetic quality, emotional quality, or cultural authenticity.

Recommended Inference Configuration

The following decoding configuration was used in the final project pipeline:

The project also used a strict Nepali system instruction requesting Devanagari-only lyric generation.

Example Usage

Install the required packages:

pip install transformers peft accelerate bitsandbytes

Then load the base model and adapter:

import torch

from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    BitsAndBytesConfig,
)

from peft import PeftModel


BASE_MODEL = "google/gemma-3-4b-it"

ADAPTER_MODEL = (
    "pradipbasnet68/"
    "gemma-3-4b-nepali-lyrics-lora"
)




bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float32,
)




tokenizer = AutoTokenizer.from_pretrained(
    BASE_MODEL
)




base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    quantization_config=bnb_config,
    device_map="auto",
)




model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_MODEL,
)


model.eval()

Example Nepali Lyric Generation

SYSTEM_PROMPT = """
तपाईं नेपाली लोकगीतका शब्द लेख्ने सहायक हुनुहुन्छ।

नियमहरू:
- केवल नेपाली देवनागरी लिपिमा गीत लेख्नुहोस्।
- अंग्रेजी वा रोमन अक्षर प्रयोग नगर्नुहोस्।
- स्वाभाविक नेपाली लोकगीत शैली प्रयोग गर्नुहोस्।
- अनावश्यक व्याख्या, शीर्षक वा निर्देशन नलेख्नुहोस्।
- केवल गीतका पङ्क्तिहरू दिनुहोस्।
- अत्यधिक दोहोरिने शब्द वा पङ्क्ति प्रयोग नगर्नुहोस्।
""".strip()


USER_PROMPT = """
धेरै वर्षपछि आफ्नो गाउँ फर्किएको युवकको
सम्झनाबारे भावुक नेपाली लोकगीत लेख्नुहोस्।
""".strip()


messages = [
    {
        "role": "system",
        "content": SYSTEM_PROMPT,
    },
    {
        "role": "user",
        "content": USER_PROMPT,
    },
]


prompt_text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)


inputs = tokenizer(
    prompt_text,
    return_tensors="pt",
)


inputs = {
    key: value.to(model.device)
    for key, value in inputs.items()
}


input_length = inputs[
    "input_ids"
].shape[-1]


torch.manual_seed(42)


with torch.inference_mode():

    output = model.generate(
        **inputs,
        max_new_tokens=300,
        do_sample=True,
        temperature=0.55,
        top_p=0.85,
        repetition_penalty=1.20,
        no_repeat_ngram_size=4,
        renormalize_logits=True,
        pad_token_id=tokenizer.eos_token_id,
    )


generated_tokens = output[0][
    input_length:
]


lyrics = tokenizer.decode(
    generated_tokens,
    skip_special_tokens=True,
).strip()


print(lyrics)

Example Output

A model output may follow a structure similar to:

गाउँको बाटो सम्झनामा आयो
पुरानो चौतारी मनमा छायो
हावाले ल्यायो माटोको बास्ना
फर्केर आएँ बोकी धेरै सम्झना

डाँडामाथि घाम अझै उस्तै
खोलाको आवाज लाग्छ मलाई आफ्नै
टाढा बितेका ती दिन सम्झिँदा
मन फेरि पुग्छ बालापनतिर

Generation is stochastic, so outputs will vary depending on the prompt and decoding settings.

Relationship to Other Project Models

Earlier Lyric Model

An earlier stage of the project used:

pradipbasnet68/nepali-lyrics-bloomz-3b-lora

The BLOOMZ model demonstrated that LoRA fine-tuning could improve Nepali lyric generation, but the later Gemma-based system was selected as the final lyric-generation model.

Music Generation Model

Generated lyrics are passed to:

pradipbasnet68/nepali-folk-acestep-1.5-lora

which adapts ACE-Step 1.5 toward Nepali folk-style music generation.

Limitations

The model can still produce:

• —repetitive lyric lines
• —awkward grammar
• —unusual word combinations
• —weak narrative continuity
• —semantically inconsistent lines
• —culturally imperfect expressions
• —occasionally unnatural poetic structures

Generation quality also depends heavily on:

• —prompt wording
• —temperature
• —sampling parameters
• —generation length

The model has not been evaluated as a general-purpose Nepali language model.

Automatic metrics such as loss, perplexity, and token accuracy also do not directly measure artistic quality.

Human judgement remains important when evaluating creative lyrics.

Ethical and Responsible Use

Generated lyrics should be treated as AI-generated creative content.

Users should manually review generated material before publication or commercial use.

The model may reproduce linguistic or stylistic patterns present in its training data and should not be assumed to represent all Nepali cultures, dialects, communities, or musical traditions.

Base Model

This adapter requires:

google/gemma-3-4b-it

The base model is not included in this repository.

Users must separately obtain the base model and comply with Google's Gemma license and terms.

Adapter Repository

pradipbasnet68/gemma-3-4b-nepali-lyrics-lora

Final Project Pipeline

User Song Request
        ↓
Gemma-3-4B-IT
+ Nepali Lyrics LoRA
        ↓
Generated Nepali Lyrics
        ↓
Lyric Cleaning / Validation
        ↓
ACE-Step 1.5 Turbo
+ Nepali Folk LoRA
        ↓
Generated Nepali Folk Song

Author

Pradip Raj Basnet

Developed as part of an undergraduate final-year project investigating AI-based Nepali folk music generation.

Disclaimer

This repository contains an experimental research adapter developed for academic purposes.

Performance may vary across prompts, environments, hardware configurations, and generation settings.