CoolFace
Modelpublic

Grenmango/whisper-medium-en-hindi-accent

sourceHugging Faceapache-2.0updated 7d agoView on Hugging Face
0likes41downloads
Model Card

Whisper Medium (English) - Fine-Tuned for Hindi-Accented English

This model is a fine-tuned, standalone merged version of `openai/whisper-medium.en` specifically adapted for English speech spoken with a Hindi accent.

It was trained on native Hindi speakers from the L2-ARCTIC speech corpus using Parameter-Efficient Fine-Tuning (LoRA rank=32, alpha=64), with the learned adapters permanently merged into the base model weights.

Benchmark & Performance Evaluation

Evaluated against the baseline whisper-medium.en and whisper-large-v3-turbo on clean held-out read speech (test split):

ModelTest Split WERTest Split CER
This Model (`whisper-medium-en-hindi-accent`)2.64%1.30%
openai/whisper-medium.en (Zero-Shot)4.54%2.07%
openai/whisper-large-v3-turbo (Zero-Shot)3.94%2.07%
  • —Relative WER reduction on held-out Hindi-accented English: ~41.9% improvement over zero-shot whisper-medium.en.

Training Data & Configuration

  • —Base Model: openai/whisper-medium.en (769M parameters)
  • —Dataset: L2-ARCTIC (Hindi subset)
  • —Training Utterances: 4,071 utterances (~3.5h)
  • —Speakers in Corpus: ASI (Male), RRBI (Male), SVBI (Female), TNI (Female)
  • —Acoustic Input: 80-channel log-Mel spectrogram, 16 kHz mono audio
  • —Training Method: LoRA ($r=32, \alpha=64$, targeting q_proj, k_proj, v_proj, out_proj, fc1, fc2)
  • —Precision: FP16 merged weights (compatible with standard WhisperForConditionalGeneration)

Quickstart & Usage

This is a standalone model. You can load and use it directly with Hugging Face transformers without needing peft or any extra setup.

1. Using pipeline (Recommended)

python
from transformers import pipeline

# Initialize the pipeline
transcriber = pipeline(
    "automatic-speech-recognition",
    model="Grenmango/whisper-medium-en-hindi-accent",
    chunk_length_s=30,
    device="cuda",  # or "cpu"
)

# Transcribe an audio file (automatically resampled to 16kHz)
result = transcriber("path/to/audio.wav")
print(result["text"])

2. Direct Model & Processor Usage

python
import torch
import soundfile as sf
from transformers import WhisperProcessor, WhisperForConditionalGeneration

model_id = "Grenmango/whisper-medium-en-hindi-accent"
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

# Load 16kHz audio
audio_data, sample_rate = sf.read("path/to/audio.wav")
if sample_rate != 16000:
    import soxr
    audio_data = soxr.resample(audio_data, sample_rate, 16000)

input_features = processor(audio_data, sampling_rate=16000, return_tensors="pt").input_features.to("cuda", torch.float16)

# Generate transcription
predicted_ids = model.generate(input_features, max_new_tokens=128)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print("Transcription:", transcription)

Multi-Accent & Personalized Collection

This model is part of a complete multi-accent & personalized English fine-tuning suite on Hugging Face: