GaborMadarasz/gemma-4-E2B-it-CV24_hun

gemma-4-E2B-it-CV24_hun

Fine-tuned google/gemma-4-E2B-it for Hungarian automatic speech recognition (ASR) using LoRA adapters, trained with Unsloth.

Model Description

This model is a LoRA fine-tune of Gemma 4 E2B (2B parameter Mixture-of-Experts multimodal variant), adapted for Hungarian speech transcription using the multimodal chat format. The audio encoder and language model backbone are fine-tuned jointly via low-rank adaptation.

• —Base model: google/gemma-4-E2B-it
• —Architecture: Gemma 4 multimodal (text + audio)
• —Fine-tuning method: LoRA (r=8, alpha=16) via Unsloth FastModel
• —Saved format: Merged float16 weights (via push_to_hub_merged)

Eval results

Training Details

Dataset

• —Training data: Mozilla Common Voice 24 Hungarian split (local preprocessed copy)
• —Dataset size: ~44,350 audio-text pairs used for training
• —Audio sampling rate: 16,000 Hz

LoRA Configuration

Training Hyperparameters

Training Hardware

Single RTX3060 12GB

5.5 GPU hours

Data Formatting

Each sample is formatted into a multimodal chat conversation:

<bos><|turn>system You are an assistant that transcribes speech accurately.<turn|> <|turn>user <|audio|>Please transcribe this audio.<turn|> <|turn>model [transcription]<turn|>

Usage

from unsloth import FastModel
from transformers import TextStreamer

model, processor = FastModel.from_pretrained(
    model_name="GaborMadarasz/gemma-4-E2B-it-CV24_hun",
    max_seq_length=1024,
    load_in_4bit=True,
)

def transcribe(audio_array):
    messages = [
        {
            "role": "system",
            "content": [{"type": "text", "text": "You are an assistant that transcribes speech accurately."}],
        },
        {
            "role": "user",
            "content": [
                {"type": "audio", "audio": audio_array},
                {"type": "text", "text": "Please transcribe this audio."}
            ]
        }
    ]
    model.generate(
        **processor.apply_chat_template(
            messages,
            add_generation_prompt=True,
            tokenize=True,
            return_dict=True,
            return_tensors="pt",
        ).to("cuda"),
        max_new_tokens=256,
        do_sample=False,
        streamer=TextStreamer(processor, skip_prompt=True),
    )

Note: Audio input must be resampled to 16,000 Hz before passing to the processor.

Limitations

• —Optimized specifically for Hungarian speech; performance on other languages is not guaranteed.
• —Trained on 1 epoch of Common Voice HU data; rare vocabulary, dialects, and heavy accents may degrade accuracy.
• —The model was fine-tuned with load_in_4bit=True; the merged float16 weights are provided for full-precision inference.

Framework

• —Unsloth — efficient LoRA fine-tuning
• —TRL SFTTrainer + UnslothVisionDataCollator
• —transformers==5.5.0