iamtokarev/emotion-distilbert-finetuned

Emotion Classification with DistilBERT

A fine-tuned DistilBERT model for emotion classification trained on the dair-ai/emotion dataset. This model classifies text into 6 emotion categories: sadness, joy, love, anger, fear, and surprise.

Model Details

• —Model type: DistilBERT for sequence classification
• —Base model: distilbert-base-uncased
• —Language: English
• —Number of classes: 6
• —Task: Text Classification (Emotion Recognition)
• —Architecture: 6 layers, 12 attention heads, 768 hidden dimensions

Emotion Categories

The model classifies text into the following 6 emotions:

• —sadness (0)
• —joy (1)
• —love (2)
• —anger (3)
• —fear (4)
• —surprise (5)

Training Details

Training Data

• —Dataset: dair-ai/emotion
• —Training samples: ~16,000
• —Validation samples: ~2,000
• —Test samples: ~2,000

Training Configuration

• —Epochs: 3
• —Batch size: 16 (train), 32 (eval)
• —Learning rate: 2e-5
• —Weight decay: 0.01
• —Warmup ratio: 0.06
• —Evaluation strategy: epoch
• —Save strategy: epoch
• —Best model metric: F1 score (macro-averaged)

Training Results

• —Best validation F1: Model saved based on best F1 score during training
• —Best validation accuracy: Model performance optimized for F1 metric

Usage

Using Transformers

from transformers import pipeline

# Using the pipeline
classifier = pipeline("text-classification", model="your-username/emotion-distilbert-finetuned")
result = classifier("I feel great today!")
print(result)
# Output: [{'label': 'joy', 'score': 0.9876}]

Direct Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("your-username/emotion-distilbert-finetuned")
model = AutoModelForSequenceClassification.from_pretrained("your-username/emotion-distilbert-finetuned")

# Emotion labels
emotion_labels = {
    0: "sadness",
    1: "joy",
    2: "love",
    3: "anger",
    4: "fear",
    5: "surprise"
}

# Classify emotion
text = "I feel great today!"
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)

with torch.no_grad():
    outputs = model(**inputs)
    predicted_class = torch.argmax(outputs.logits, dim=1).item()

print(f"Predicted emotion: {emotion_labels[predicted_class]}")

Inference API

# Get all probabilities
with torch.no_grad():
    outputs = model(**inputs)
    probabilities = torch.softmax(outputs.logits, dim=1).squeeze()

for i, prob in enumerate(probabilities):
    print(f"{emotion_labels[i]}: {prob:.4f}")

Model Architecture

• —Base Model: DistilBERT (distilled version of BERT-base)
• —Hidden Size: 768
• —Number of Layers: 6
• —Attention Heads: 12
• —Vocabulary Size: 30,522
• —Max Position Embeddings: 512
• —Classification Head: Linear layer with 6 outputs

Training Infrastructure

• —Framework: PyTorch with Hugging Face Transformers
• —Optimizer: AdamW
• —Learning Rate Scheduler: Linear warmup followed by linear decay
• —Loss Function: Cross-entropy loss for multi-class classification
• —Mixed Precision: Not used (trained in FP32)

Performance Notes

• —Model was trained on a dataset balanced across emotion categories
• —Best model checkpoint was selected based on validation F1 score
• —Model performs well on various text lengths and domains
• —May require domain adaptation for very specific use cases

Limitations

• —Trained primarily on English text
• —May not generalize well to other languages
• —Performance may vary across different text domains
• —Model predictions should be used with appropriate confidence thresholds

Citation

If you use this model, please cite the original dataset:

@inproceedings{saravia-etal-2018-carer,
    title = "{CARER}: Contextualized Affect Representations for Emotion Recognition",
    author = "Saravia, Elvis  and
      Liu, Hsien-Chi Toby  and
      Huang, Yen-Hao  and
      Wu, Junlin  and
      Chen, Yi-Shin",
    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",
    month = oct # "-" # nov,
    year = "2018",
    address = "Brussels, Belgium",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D18-1404",
    doi = "10.18653/v1/D18-1404",
    pages = "3687--3697"
}