Modelpublic

waveletdeboshir/whisper-base-ser-dusha

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes4.9kdownloads

Model Card

Whisper-base for Speech Emotion Recognition in Russian on Dusha dataset

Whisper-base encoder with classification head for speech emotion recognition.

Dusha dataset: https://github.com/salute-developers/golos/tree/master/dusha

Multiclass classification into 5 classes:

—angry 0
—sad 1
—neutral 2
—positive 3
—other 4

Model was fine-tuned on full Dusha-crowd with

—augmentations Time Shift, Time Masking and Colored Noise;
—WeightedRandomSampler.

Usage

python

import torch
import torchaudio
from transformers import WhisperForAudioClassification, WhisperFeatureExtractor

# load model and feature extractor
model = WhisperForAudioClassification.from_pretrained("waveletdeboshir/whisper-base-ser-dusha")
model.eval()
feature_extractor = WhisperFeatureExtractor.from_pretrained("waveletdeboshir/whisper-base-ser-dusha")

# load audio and resample if necessary
wav, sr = torchaudio.load("audio.wav")
if sr != 16000:
    wav = torchaudio.functional.resample(wav, sr, 16000)

# compute predictions
features = feature_extractor(wav[0], sampling_rate=16000, return_tensors="pt").input_features
with torch.no_grad():
    preds = model(features)

# get emotion and its probability
probs = torch.nn.functional.softmax(preds.logits, dim=-1)
print(f"Predicted emotion: {model.config.id2label[probs.argmax().item()]} with probability {probs.max().item():.4f}")