specialv/whisper-small-merged-hi-en

Whisper Small - Merged Hindi & English (Task Arithmetic)

This model is a bilingual version of OpenAI's Whisper Small, specifically optimized for Hindi transcription while recovering original English capabilities using Task Arithmetic merging.

Model Details

Model Description

Standard fine-tuning on a specific language often causes "Catastrophic Forgetting," where the model loses its accuracy in the original languages (like English). This model solves that by merging a Hindi-fine-tuned model with the original base model.

• —Developed by: specialv
• —Model type: Speech-to-Text (ASR)
• —Language(s): Hindi (hi), English (en)
• —Finetuned from model: openai/whisper-small
• —Finetuning Dataset: google/fleurs (Hindi subset)
• —Merging Method: Task Arithmetic ($W{new} = W{base} + \lambda(W{fine} - W{base})$)

Model Sources

• —Base Model: openai/whisper-small
• —Hindi Task Vector source: specialv/whisper-small-hi-fleurs

Uses

Direct Use

• —Bilingual Transcription: Accurately transcribes both pure Hindi and pure English audio.
• —Hinglish Support: Improved performance on code-switched speech (mixed Hindi and English) compared to the base or purely fine-tuned versions.
• —Transcription of Long-form Audio: Supports chunking for files longer than 30 seconds.

How to Get Started with the Model

Use the code below to transcribe audio in either Hindi or English:

from transformers import pipeline
import torch

pipe = pipeline(
    "automatic-speech-recognition",
    model="specialv/whisper-small-merged-hi-en",
    device="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
)

# For Hindi Transcription
result_hi = pipe("audio_hi.wav", generate_kwargs={"language": "hindi"})
print(result_hi["text"])

# For English Transcription
result_en = pipe("audio_en.wav", generate_kwargs={"language": "english"})
print(result_en["text"])

Training Details

Training Data

The task vector was derived from fine-tuning on the Google FLEURS Hindi (hi_in) dataset.

Merging Procedure

The model was created using Task Arithmetic. This involves calculating the "Hindi task vector" (the difference between the fine-tuned weights and the base weights) and adding it back to the original English-capable base model. Scaling Factor (λ=W openai/whisper−small+0.6×(W specialv/whisper−small−hi−fleurs−W openai/whisper−small)

Evaluation

Quantitative Comparison

Qualitative Comparison (Sample Transcription)

Environmental Impact

• —Hardware Type: Tesla T4 GPU (Google Colab)
• —Hours used for fine-tuning: ~1.5 hours
• —Merging Time: < 5 minutes

How to apply this to your repo:

1. 1.Go to your model page: https://huggingface.co/specialv/whisper-small-merged-hi-en
2. 2.Click the "Edit model card" button.
3. 3.Delete the existing text and paste the block above.
4. 4.Click "Commit changes".