shae2977/xlm-roberta-hinglish-sentiment-analysis

XLM-RoBERTa Hinglish Sentiment Analysis

A fine-tuned XLM-RoBERTa model for sentiment classification of Hinglish text — the code-mixed Hindi-English language used by hundreds of millions of Indians online.

Most sentiment models are trained on clean English or formal Hindi. They fail badly on Hinglish because it is neither; it is a fluid mix of both, written in Roman script, full of slang, abbreviations, and cultural references. This model is trained specifically to handle that.

Model details

Performance

Evaluated on a held-out test set of Hinglish YouTube comments.

VADER drops to 0.39 on Hinglish because it has no concept of Roman-script Hindi words like acha, bahut, zabardast, or mixed constructions like "yaar this song is too good na?". XLM-RoBERTa's multilingual pretraining gives it a foundation to handle this naturally.

Per-class results

Neutral is the hardest class as neutral comments are often ambiguous even for human annotators.

Dataset

The training data is a self-annotated dataset of 3,000+ Hinglish YouTube comments scraped from Indian music, entertainment, and pop culture videos. Comments were manually labeled as Positive, Negative, or Neutral.

Dataset: shae2977/hinglish-youtube-sentiment-dataset

Usage

from transformers import XLMRobertaTokenizer, XLMRobertaForSequenceClassification
import torch

model_name = "shae2977/xlm-roberta-hinglish-sentiment-analysis"
tokenizer = XLMRobertaTokenizer.from_pretrained(model_name)
model = XLMRobertaForSequenceClassification.from_pretrained(model_name)
model.eval()

labels = {0: "Negative", 1: "Neutral", 2: "Positive"}

def predict(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
    with torch.no_grad():
        outputs = model(**inputs)
    predicted = torch.argmax(outputs.logits, dim=1).item()
    return labels[predicted]

predict("yaar ye song bahut zabardast hai")   # Positive
predict("bilkul bekar tha")                   # Negative
predict("theek tha, kuch khaas nahi")         # Neutral

Training details

Limitations

• —Trained on YouTube comments from Indian entertainment content — may not generalize well to other domains like politics or sports
• —Neutral class performance is weaker than Positive/Negative
• —Dataset size is limited (~3,000 samples) — a larger annotated dataset would improve performance
• —Overall weighted F1 of 0.67 means that comments may be miscalssified

Author

Built by shae2977