Ghost3454/translynx-pakistani-punjabi-whisper-small

TransLynx Pakistani Punjabi Whisper (Small)

A fine-tuned version of openai/whisper-small for Pakistani Punjabi speech recognition in Shahmukhi script (the Perso-Arabic script used to write Punjabi in Pakistan).

Model description

This model transcribes spoken Pakistani Punjabi into written Shahmukhi text. It was fine-tuned from OpenAI's multilingual Whisper Small checkpoint, which had a baseline Word Error Rate (WER) of roughly 70% on this language before fine-tuning.

Because Shahmukhi shares its character set with Urdu, the model uses the Urdu language token (`ur`) internally rather than the Whisper pa token, which corresponds to Eastern Punjabi in Gurmukhi script. The language and task are baked into the generation config, so you do not need to pass them at inference time.

Results

The test WER is measured on a speaker whose voice was never seen during training, which reflects real-world generalization to new voices.

Intended use

• —Transcription of Pakistani Punjabi audio into Shahmukhi text
• —Voice interfaces, subtitles, and speech data pipelines for Pakistani Punjabi
• —A starting checkpoint for further fine-tuning on additional Punjabi data

How to use

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa
import torch

REPO_ID = "Ghost3454/translynx-pakistani-punjabi-whisper-small"

processor = WhisperProcessor.from_pretrained(REPO_ID)
model     = WhisperForConditionalGeneration.from_pretrained(REPO_ID)
device    = "cuda" if torch.cuda.is_available() else "cpu"
model     = model.to(device).eval()

# load a 16 kHz audio file
audio, _ = librosa.load("sample.wav", sr=16000)

inputs = processor(
    audio, sampling_rate=16000, return_tensors="pt"
).input_features.to(device)

with torch.no_grad():
    generated_ids = model.generate(inputs)

text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(text)

The language token (ur), task (transcribe), and a repetition guard are already set in the generation config, so no extra generation arguments are required.

Training data

• —3,500 recordings total
• —7 speakers, each recording the same 500 Punjabi sentences
• —Sentences written in Shahmukhi script
• —One speaker held out entirely as an unseen test set; the remaining six split 90/10 into train and validation

Training procedure

The model was fully fine-tuned (all parameters trained, no LoRA or quantization) on a single consumer GPU.

Key hyperparameters:

• —Base model: openai/whisper-small
• —Learning rate: 1e-5 with 100 warmup steps
• —Effective batch size: 16 (batch size 4 with gradient accumulation of 4)
• —Mixed precision: fp16
• —Max steps: 2000, evaluating every 200 steps
• —Best checkpoint selected by validation WER
• —Language token forced to ur for stable generation
• —no_repeat_ngram_size=3 to prevent decoding loops

Limitations

• —Trained on a controlled set of 500 sentences across 7 speakers, so vocabulary and acoustic diversity are limited compared to large-scale datasets.
• —Performance may drop on spontaneous speech, heavy background noise, dialects far from the training speakers, or vocabulary outside the training sentences.
• —Optimized for Shahmukhi output; it is not intended for Gurmukhi (Eastern Punjabi) transcription.

Team & credits

Developed by the TransLynx team:

• —Ali Hamza — model fine-tuning, training pipeline, and evaluation
• —Farwa — project planning, data collection, and recording
• —Jhangir — project planning, data collection, and recording

Thanks to all seven speakers who contributed recordings to the training dataset, without whom this model would not exist.

Citation

If you use this model, please credit the TransLynx Pakistani Punjabi ASR project.