vishal98m/whisper-small-malayalam

Model Details

Model Description

This model is finetuned version of whisper-small on malayalam language for speech to text translation<!-- Provide a longer summary of what this model is. -->

Bias, Risks, and Limitations

This model is purely for learning purposes<!-- This section is meant to convey both technical and sociotechnical limitations. --> This model in its present form can't be used for live translation, as its only trained for 4000 steps due to compute limitations as its fine-tuned on 8 GB VRAM. For Better results its recomended to train with a larger dataset and possibly upto 10000 steps.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

model_id = "vishal98m/whisper-small-malayalam"

device = "cuda" if torch.cuda.is_available() else "cpu"

# Load model and processor
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id).to(device)
processor = AutoProcessor.from_pretrained(model_id)

# Create pipeline
pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    device=0 if device == "cuda" else -1,
)

# Run inference (Malayalam speech → English translation)
result = pipe(
    "sample_audio.wav",
    generate_kwargs={
        "task": "translate",
        "language": "malayalam"
    }
)

print("Translation:")
print(result["text"])

Evaluation

• —Eval Loss: 1.384
• —Word Error Rate (WER): 60.8
• —Orthographic WER: 63.29
• —BLEU Score: 31.77 <!-- This section describes the evaluation protocols and provides the results. -->

Framework versions

• —Transformers 5.1.0
• —Pytorch 2.10.0+cu128
• —Datasets 2.16.1
• —Tokenizers 0.22.2