BettySara/betty-malayalam-whisper-large-v3-qlora-r32
Whisper Large V3 Turbo Malayalam ASR – QLoRA Adapter
Model Description
This model is a QLoRA-based parameter-efficient fine-tuned adapter for openai/whisper-large-v3-turbo, adapted for Malayalam Automatic Speech Recognition (ASR). The model uses 4-bit quantization with LoRA adapters to reduce GPU memory requirements while improving Malayalam transcription performance over the zero-shot Whisper baseline.
The model was trained on the Malayalam subset of the AI4Bharat IndicVoices dataset and is suitable for research on low-resource Indic ASR and efficient adaptation of large speech models.
Model Details
Intended Use
This model can be used for:
- Malayalam speech-to-text transcription
- Research on parameter-efficient fine-tuning for ASR
- Low-resource Indic language ASR experiments
- Comparing full fine-tuning with PEFT methods such as LoRA and QLoRA
- Deployment experiments where GPU memory is limited
Dataset
The model was trained using the Malayalam subset of the AI4Bharat IndicVoices dataset. Audio was converted to 16 kHz and transformed into Whisper-compatible log-Mel input features. Malayalam text was tokenized using the Whisper tokenizer configured for Malayalam transcription.
The preprocessing pipeline included:
- Loading Malayalam train and validation splits from
ai4bharat/IndicVoices - Selecting a training subset for QLoRA experimentation
- Removing unused metadata columns
- Resampling audio to 16 kHz
- Extracting input features using
WhisperFeatureExtractor - Tokenizing labels using the Malayalam Whisper tokenizer
- Filtering samples that exceed the maximum target token length
Training Configuration
model_id = "openai/whisper-large-v3-turbo"
USE_QLORA = True
epochs = 6
lora_config = {
"r": 32,
"lora_alpha": 64,
"lora_dropout": 0.05,
"target_modules": ["q_proj", "k_proj", "v_proj", "out_proj"],
"bias": "none"
}
quantization_config = {
"load_in_4bit": True,
"bnb_4bit_quant_type": "nf4",
"bnb_4bit_use_double_quant": True,
"bnb_4bit_compute_dtype": "bfloat16"
}
training_args = {
"per_device_train_batch_size": 8,
"gradient_accumulation_steps": 2,
"per_device_eval_batch_size": 8,
"gradient_checkpointing": True,
"optim": "adamw_bnb_8bit",
"learning_rate": 5e-5,
"warmup_steps": 500,
"num_train_epochs": 6,
"eval_strategy": "epoch",
"save_strategy": "epoch",
"predict_with_generate": True,
"generation_max_length": 448,
"metric_for_best_model": "wer",
"greater_is_better": False,
"lr_scheduler_type": "constant",
"seed": 42,
"data_seed": 42
}Evaluation
The model was evaluated using Word Error Rate (WER), computed with the evaluate and jiwer libraries.
Note: Replace the WER value above with the exact final eval_wer from the completed training run before final publication if needed.Loading the QLoRA Adapter
import torch
import librosa
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from peft import PeftModel
BASE_MODEL_ID = "openai/whisper-large-v3-turbo"
ADAPTER_ID = "BettySara/betty-malayalam-whisper-large-v3-qlora-r32"
processor = WhisperProcessor.from_pretrained(
BASE_MODEL_ID,
language="Malayalam",
task="transcribe"
)
base_model = WhisperForConditionalGeneration.from_pretrained(
BASE_MODEL_ID,
torch_dtype=torch.float16,
device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER_ID)
model.eval()
forced_decoder_ids = processor.get_decoder_prompt_ids(
language="ml",
task="transcribe"
)
def transcribe(audio_path):
speech, sr = librosa.load(audio_path, sr=16000)
inputs = processor(
speech,
sampling_rate=16000,
return_tensors="pt"
)
input_features = inputs.input_features.to(model.device)
input_features = input_features.to(model.dtype)
predicted_ids = model.generate(
input_features,
forced_decoder_ids=forced_decoder_ids,
max_length=448
)
return processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcribe("sample_malayalam_audio.wav"))Merged Model and Faster-Whisper Conversion
The adapter can be merged with the base Whisper model using PEFT:
model = model.merge_and_unload()
model.save_pretrained("whisper-large-v3-malayalam-merged")
processor.save_pretrained("whisper-large-v3-malayalam-merged")It can also be converted to CTranslate2 format for faster inference:
ct2-transformers-converter \
--model BettySara/whisper-large-v3-malayalam-merged \
--output_dir whisper-large-v3-malayalam-ct2 \
--copy_files tokenizer_config.json preprocessor_config.json \
--quantization int8_float16Example Faster-Whisper loading:
from faster_whisper import WhisperModel
model = WhisperModel(
"whisper-large-v3-malayalam-ct2",
device="cuda",
compute_type="int8_float16"
)
segments, info = model.transcribe(
"sample_malayalam_audio.wav",
language="ml",
beam_size=5,
vad_filter=True
)
for segment in segments:
print(segment.text)Limitations
- This is a PEFT adapter and requires the base model unless merged.
- The model is specialized for Malayalam and may not generalize to other languages.
- Accuracy may vary for noisy audio, code-mixed Malayalam-English speech, dialectal variation, long recordings, or overlapping speakers.
- QLoRA is memory-efficient but may not always match the accuracy of full fine-tuning.
- The model should be evaluated on a larger independent test set before production deployment.
Ethical Considerations
Users should obtain consent before transcribing private speech. ASR outputs may contain errors and should not be used without human verification in sensitive contexts such as healthcare, law, finance, or official documentation.
Citation
If you use this model, please cite Whisper, LoRA/QLoRA-related PEFT work, and the IndicVoices dataset.
@article{radford2022whisper,
title={Robust Speech Recognition via Large-Scale Weak Supervision},
author={Radford, Alec and others},
journal={arXiv preprint arXiv:2212.04356},
year={2022}
}
@article{hu2021lora,
title={LoRA: Low-Rank Adaptation of Large Language Models},
author={Hu, Edward J. and others},
journal={arXiv preprint arXiv:2106.09685},
year={2021}
}Author
Developed by Betty Sara Santhosh as part of research on Malayalam ASR using Whisper and PEFT-based fine-tuning.