hugoaslm/multimodal-emotion-recognition

Multimodal Emotion Recognition System

A state-of-the-art multimodal emotion recognition model combining Wav2Vec2 (audio) and RoBERTa (text) encoders with cross-attention fusion and label smoothing regularization.

🎯 Results

Key Findings:

• —Cross-attention fusion significantly outperforms simple concatenation (+6.4% F1)
• —Label smoothing (0.1) provides +2.5% test accuracy improvement
• —Audio carries the primary emotional signal; text provides complementary context
• —The 219M parameter model achieves SOTA-level performance on the benchmark

📊 Dataset

[stapesai/ssi-speech-emotion-recognition](https://huggingface.co/datasets/stapesai/ssi-speech-emotion-recognition)

• —Source Datasets: CREMA-D, TESS, RAVDESS, SAVEE
• —Splits: 10,000 train / 1,999 validation / 163 test
• —Emotions (8 classes): angry, calm, disgust, fear, happy, neutral, sad, surprise
• —Modalities: Audio (speech) + Text (transcription)
• —Note: Test set has no "calm" samples (7 classes evaluated)

Class Distribution (Train)

🏗️ Architecture

Input Audio ──► Wav2Vec2-Base ──► Mean Pooling ──► Audio Features (768-dim)
                                                              │
                                                              ▼
                                                    Cross-Attention Fusion
                                                    (text queries audio)
                                                              │
                                                              ▼
Input Text ──► RoBERTa-Base ──► [CLS] Token ──► Text Features (768-dim)
                                                              │
                                                              ▼
                                                   Concatenate + MLP
                                                              │
                                                              ▼
                                                   Classification Head
                                                              │
                                                              ▼
                                              8 Emotion Classes

Key Components

1. 1.Audio Encoder: facebook/wav2vec2-base (95M params)
2. 2.Pre-trained on speech data
3. 3.Mean pooling over time dimension

1. 1.Text Encoder: roberta-base (125M params)
2. 2.Pre-trained on large text corpus
3. 3.[CLS] token as sentence representation

1. 1.Fusion Module: Cross-Attention
2. 2.Text features query audio features
3. 3.4 attention heads, 256-dim fusion space
4. 4.Residual connection + LayerNorm

1. 1.Classification Head:
2. 2.2-layer MLP with GELU activation
3. 3.Dropout (0.3)

1. 1.Training Improvements:
2. 2.Label smoothing (0.1)
3. 3.Gradient checkpointing
4. 4.Mixed precision (fp16)
5. 5.Early stopping (patience=3)

🚀 Training Details

🔬 Ablation Studies

Fusion Strategy Comparison

Cross-attention fusion provides significant improvement over simpler fusion methods, demonstrating the importance of modeling interactions between modalities.

Label Smoothing Impact

Label smoothing improves both validation and test performance, indicating better generalization.

📚 References

This implementation is based on:

1. 1.arXiv:2406.17667 - Early Feature Fusion with Wav2Vec2-MSP + RoBERTa for emotion recognition
2. 2.arXiv:2503.06805 - RoBERTa + Wav2Vec2 Feature Fusion for MELD benchmark
3. 3.arXiv:2505.06685 - Emotion-Qwen: Multimodal LLM for emotion understanding
4. 4.arXiv:2406.11161 - Emotion-LLaMA: Instruction-tuned emotion recognition

🛠️ Usage

from transformers import AutoModel, AutoFeatureExtractor, AutoTokenizer
import torch
import torch.nn.functional as F

# Load model components
audio_encoder = AutoModel.from_pretrained("facebook/wav2vec2-base")
text_encoder = AutoModel.from_pretrained("roberta-base")
feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
tokenizer = AutoTokenizer.from_pretrained("roberta-base")

# The fusion head and classifier need to be loaded from the checkpoint
# See the training script for the full model definition

🔮 Future Improvements

1. 1.SER-Pretrained Audio Encoder: Use audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim for better audio emotion features
2. 2.Visual Modality: Add face/video encoding for full multimodal recognition
3. 3.Instruction Tuning: Convert to instruction-following format for zero-shot generalization
4. 4.Class Balancing: Oversample rare classes (calm, surprise) or use focal loss
5. 5.Data Augmentation: Speed perturbation, noise injection for audio robustness

📄 License

Apache 2.0

🙏 Acknowledgments

• —Hugging Face Transformers for the pre-trained models
• —The creators of CREMA-D, TESS, RAVDESS, and SAVEE datasets
• —The authors of the referenced papers for their valuable insights