Ghazouaniwala/emotions_speech

emotions_speech — speaker-independent speech emotion recognition

A wav2vec2-large-XLSR-53 fine-tune that classifies emotional tone from raw audio alone — from how something is said, not what is said.

Classes (7): neutral · happy · sad · angry · fearful · disgust · surprised

The evaluation protocol is the point

This model is evaluated with speaker-independent `GroupKFold`, and the split is asserted at runtime rather than merely intended:

assert len(train_actors & test_actors) == 0, "Speaker leakage!"

A random split places the same actors in both train and test, so the model learns to recognise voices rather than emotions. That is how speech-emotion models reach the 90%-plus accuracy figures commonly reported for this task — and why those figures collapse on the first unseen speaker.

Per-fold metrics live in the training notebook. No headline figure is quoted here: the only figures worth quoting are the speaker-disjoint ones, and a single averaged number hides how much fold-to-fold variance a 7-class problem over a small pool of actors actually carries.

Architecture

Beyond a plain classification head:

• —Learned weighted sum over all 25 transformer hidden states — nn.Parameter(torch.ones(25)) — instead of taking only the final layer. Emotion is carried largely in the mid-layers, and letting the model learn the mixture beats guessing which layer to read.
• —Projected residual path: LayerNorm → 768→384 → GELU → 384→128, summed with a linear proj_skip shortcut before the classifier.
• —Input: raw 16 kHz waveform, 4-second clips, up to 6 chunks aggregated per turn.

Because the pooling layer is custom, this checkpoint does not load through a stock AutoModelForAudioClassification call.

Usage

<!-- Paste the loading code you have actually run — the custom head means the model class must be defined before the state dict is loaded. The setup script in the Solace repo (backend/scripts/setupemotionmodel.py) is a working reference. -->

Training

• —Data: RAVDESS, split speaker-independently with GroupKFold
• —Loss: focal loss, to handle class imbalance
• —Regularisation: mixup, label smoothing
• —Schedule: cosine annealing with warm restarts
• —Augmentation: ESC-50 environmental noise
• —Second view: a complementary EfficientNet-B2 over multi-channel spectrograms (mel + MFCC + delta-MFCC) was trained for ensembling

Intended use

Conversational systems that need vocal tone as a signal independent of transcript content.

Deployed in Solace, where it runs alongside a facial-expression classifier and the two channels are deliberately kept separate. When a user's wording reads neutral while their voice reads fearful, that disagreement is surfaced to the user rather than averaged into a single score — the gap is the most informative thing the system can notice, and averaging destroys it.

Limitations

• —RAVDESS is North American English acted speech. Acted emotion is more exaggerated than spontaneous emotion, and the cultural distribution does not match Tunisian or other non-Western speakers. Expect degradation outside that distribution.
• —Seven categorical labels are a coarse model of affect. Real emotion is continuous, mixed, and context-dependent.
• —Outputs are classifier predictions, not measurements of a person's internal state. Any system built on this should present them that way — "your tone read as frustrated", never "you were frustrated".
• —Not for clinical, diagnostic, hiring, or surveillance use.

Related work

Citation

@misc{ghazouani2025emotionsspeech,
  author = {Ghazouani, Wala Eddine},
  title  = {emotions_speech: speaker-independent speech emotion recognition},
  year   = {2025},
  url    = {https://huggingface.co/Ghazouaniwala/emotions_speech}
}

Built by Wala Eddine Ghazouani · GitHub · LinkedIn