Oriserve/Whisper-Hindi2Hinglish-Prime

A better version of this model is available: Oriserve/Whisper-Hindi2Hinglish-Apex

Whisper-Hindi2Hinglish-Prime:

• —GITHUB LINK: github link
• —SPEECH-TO-TEXT ARENA: Speech-To-Text Arena

Table of Contents:

• —Key Features
• —Training
• —Data
• —Finetuning
• —Usage
• —Performance Overview
• —Qualitative Performance Overview
• —Quantitative Performance Overview
• —Miscellaneous

Key Features:

1. 1.Hinglish as a language: Added ability to transcribe audio into spoken Hinglish language reducing chances of grammatical errors
2. 2.Whisper Architecture: Based on the whisper architecture making it easy to use with the transformers package
3. 3.Better Noise handling: The model is resistant to noise and thus does not return transcriptions for audios with just noise
4. 4.Hallucination Mitigation: Minimizes transcription hallucinations to enhance accuracy.
5. 5.Performance Increase: ~39% average performance increase versus pretrained model across benchmarking datasets

Training:

Data:

• —Duration: A total of ~550 Hrs of noisy Indian-accented Hindi data was used to finetune the model.
• —Collection: Due to a lack of ASR-ready hinglish datasets available, a specially curated proprietary dataset was used.
• —Labelling: This data was then labeled using a SOTA model and the transcriptions were improved by human intervention.
• —Quality: Emphasis was placed on collecting noisy data for the task as the intended use case of the model is in Indian environments where background noise is abundant.
• —Processing: It was ensured that the audios are all chunked into chunks of length <30s, and there are at max 2 speakers in a clip. No further processing steps were done so as to not change the quality of the source data.

Finetuning:

• —Novel Trainer Architecture: A custom trainer was written to ensure efficient supervised finetuning, with custom callbacks to enable higher observability during the training process.
• —Custom Dynamic Layer Freezing: Most active layers were identified in the model by running inference on a subset of the training data using the pre-trained models. These layers were then kept unfrozen during the training process while all the other layers were kept frozen. This enabled faster convergence and efficient finetuning
• —Deepspeed Integration: Deepspeed was also utilized to speed up, and optimize the training process.

Performance Overview

Qualitative Performance Overview

Quantitative Performance Overview

*Note*:

• —The below WER scores are for Hinglish text generated by our model and the original whisper model
• —To check our model's real-world performance against other SOTA models please head to our [Speech-To-Text Arena](https://huggingface.co/spaces/Oriserve/ASR_arena) arena space.

Usage:

Using Transformers

• —To run the model, first install the Transformers library

- The model can be used with the [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline)
class to transcribe audios of arbitrary length:

import torch from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline from datasets import load_dataset

Set device (GPU if available, otherwise CPU) and precision

device = "cuda:0" if torch.cuda.isavailable() else "cpu" torchdtype = torch.float16 if torch.cuda.is_available() else torch.float32

Specify the pre-trained model ID

model_id = "Oriserve/Whisper-Hindi2Hinglish-Prime"

Load the speech-to-text model with specified configurations

model = AutoModelForSpeechSeq2Seq.frompretrained( modelid, torchdtype=torchdtype, # Use appropriate precision (float16 for GPU, float32 for CPU) lowcpumemusage=True, # Optimize memory usage during loading usesafetensors=True # Use safetensors format for better security ) model.to(device) # Move model to specified device

Load the processor for audio preprocessing and tokenization

processor = AutoProcessor.frompretrained(modelid)

Create speech recognition pipeline

pipe = pipeline( "automatic-speech-recognition", model=model, tokenizer=processor.tokenizer, featureextractor=processor.featureextractor, torchdtype=torchdtype, device=device, generate_kwargs={ "task": "transcribe", # Set task to transcription "language": "en" # Specify English language } )

Process audio file and print transcription

sample = "sample.wav" # Input audio file path result = pipe(sample) # Run inference print(result["text"]) # Print transcribed text

#### Using Flash Attention 2

Flash-Attention 2 can be used to make the transcription fast. If your GPU supports Flash-Attention you can use it by, first installing Flash Attention:

• —Once installed you can then load the model using the below code:

model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, attn_implementation="flash_attention_2")

Using the OpenAI Whisper module

• —First, install the openai-whisper library

- Convert the huggingface checkpoint to a pytorch model

import torch from transformers import AutoModelForSpeechSeq2Seq import re from tqdm import tqdm from collections import OrderedDict import json

Load parameter name mapping from HF to OpenAI format

with open('converthf2openai.json', 'r') as f: reversetranslation = json.load(f)

reversetranslation = OrderedDict(reversetranslation)

def savemodel(model, savepath): def reversetranslate(currentparam): # Convert parameter names using regex patterns for pattern, repl in reversetranslation.items(): if re.match(pattern, currentparam): return re.sub(pattern, repl, current_param)

# Extract model dimensions from config config = model.config modeldims = { "nmels": config.nummelbins, # Number of mel spectrogram bins "nvocab": config.vocabsize, # Vocabulary size "naudioctx": config.maxsourcepositions, # Max audio context length "naudiostate": config.dmodel, # Audio encoder state dimension "naudiohead": config.encoderattentionheads, # Audio encoder attention heads "naudiolayer": config.encoderlayers, # Number of audio encoder layers "ntextctx": config.maxtargetpositions, # Max text context length "ntextstate": config.dmodel, # Text decoder state dimension "ntexthead": config.decoderattentionheads, # Text decoder attention heads "ntextlayer": config.decoderlayers, # Number of text decoder layers }

# Convert model state dict to Whisper format originalmodelstatedict = model.statedict() newstatedict = {}

for key, value in tqdm(originalmodelstatedict.items()): key = key.replace("model.", "") # Remove 'model.' prefix newkey = reversetranslate(key) # Convert parameter names if newkey is not None: newstatedict[new_key] = value

# Create final model dictionary pytorchmodel = {"dims": modeldims, "modelstatedict": newstatedict}

# Save converted model torch.save(pytorchmodel, savepath)

Load Hugging Face model

modelid = "Oriserve/Whisper-Hindi2Hinglish-Prime" model = AutoModelForSpeechSeq2Seq.frompretrained( modelid, lowcpumemusage=True, # Optimize memory usage use_safetensors=True # Use safetensors format )

Convert and save model

modelsavepath = "Whisper-Hindi2Hinglish-Prime.pt" savemodel(model,modelsave_path)

- Transcribe

import whisper

Load converted model with Whisper and transcribe

model = whisper.load_model("Whisper-Hindi2Hinglish-Prime.pt") result = model.transcribe("sample.wav") print(result["text"])

### Miscellaneous
This model is from a family of transformers-based ASR models trained by Oriserve. To compare this model against other models from the same family or other SOTA models please head to our [Speech-To-Text Arena](https://huggingface.co/spaces/Oriserve/ASR_arena). To learn more about our other models, and other queries regarding AI voice agents you can reach out to us at our email [ai-team@oriserve.com](ai-team@oriserve.com)