Grenmango/whisper-medium-en-vi-hqtv-personalized

Whisper Medium (English) - Personalized for Speaker HQTV (Vietnamese Accent)

This model is a fine-tuned, standalone merged version of `openai/whisper-medium.en` specifically adapted for Speaker HQTV (Male, native Vietnamese accent) from the L2-ARCTIC speech corpus.

It was trained using Parameter-Efficient Fine-Tuning (LoRA rank=32, alpha=64), with the learned speaker-adapted adapters permanently merged into the base model weights.

Benchmark & Performance Evaluation

Evaluated on clean held-out read speech (test split) specifically spoken by HQTV:

• —Relative WER reduction over zero-shot baseline (`whisper-medium.en`): 55.0% improvement (21.85% -> 9.84%).
• —Relative WER reduction over multi-speaker accent model: 26.5% improvement (13.39% -> 9.84%).
• —Outperforms zero-shot `whisper-large-v3-turbo` by 9.06 percentage points on this speaker.

Training Data & Configuration

• —Base Model: openai/whisper-medium.en (769M parameters)
• —Dataset: L2-ARCTIC (Speaker HQTV, Vietnamese English)
• —Training Utterances: 1,018 utterances (~1.1h)
• —Target Speaker: HQTV (Male, native Vietnamese accent)
• —Acoustic Input: 80-channel log-Mel spectrogram, 16 kHz mono audio
• —Training Method: LoRA ($r=32, \alpha=64$, targeting q_proj, k_proj, v_proj, out_proj, fc1, fc2)
• —Precision: FP16 merged weights (compatible with standard WhisperForConditionalGeneration)

Quickstart & Usage

This is a standalone model. You can load and use it directly with Hugging Face transformers without needing peft or any extra setup.

1. Using pipeline (Recommended)

from transformers import pipeline

# Initialize the pipeline
transcriber = pipeline(
    "automatic-speech-recognition",
    model="Grenmango/whisper-medium-en-vi-hqtv-personalized",
    chunk_length_s=30,
    device="cuda",  # or "cpu"
)

# Transcribe an audio file (automatically resampled to 16kHz)
result = transcriber("path/to/audio.wav")
print(result["text"])

2. Direct Model & Processor Usage

import torch
import soundfile as sf
from transformers import WhisperProcessor, WhisperForConditionalGeneration

model_id = "Grenmango/whisper-medium-en-vi-hqtv-personalized"
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

# Load 16kHz audio
audio_data, sample_rate = sf.read("path/to/audio.wav")
if sample_rate != 16000:
    import soxr
    audio_data = soxr.resample(audio_data, sample_rate, 16000)

input_features = processor(audio_data, sampling_rate=16000, return_tensors="pt").input_features.to("cuda", torch.float16)

# Generate transcription
predicted_ids = model.generate(input_features, max_new_tokens=128)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print("Transcription:", transcription)

Multi-Accent & Personalized Collection

This model is part of a complete multi-accent & personalized English fine-tuning suite on Hugging Face:

• —🇻🇳 whisper-medium-en-vi-accent (Vietnamese)
• —🇸🇦 whisper-medium-en-arabic-accent (Arabic)
• —🇨🇳 whisper-medium-en-chinese-accent (Chinese)
• —🇮🇳 whisper-medium-en-hindi-accent (Hindi)
• —🇰🇷 whisper-medium-en-korean-accent (Korean)
• —🇪🇸 whisper-medium-en-spanish-accent (Spanish)
• —👤 whisper-medium-en-vi-hqtv-personalized (Personalized - HQTV)