AakashakaAkku/vibe-check-emotion

Vibe Check — a 7-emotion Transformer trained from scratch 🎭

🔗 Try the live app — chat with Buddy & Wit and run a Vibe Check, nothing to install: https://civilaakash.github.io/buddymini/

Vibe Check is a compact (~8.3M parameter) Transformer text classifier that reads the emotional tone of a sentence across 7 classes: sadness · joy · love · anger · fear · surprise · neutral.

It was built entirely from scratch as a hands-on learning project — my own word tokenizer, my own architecture, trained from random initialization (no pretrained weights, no fine-tuning of a large model) on public data only.

Not a fine-tune

Unlike most models on the Hub, this is not a fine-tune of a large pretrained model. Every weight was learned from zero on public data. The tokenizer, architecture, training config and inference code are all included in this repo.

Architecture

• —Word-level tokenizer (regex [a-z']+), vocab 20,002
• —Token embedding + learned positional embedding (max length 64)
• —4 Transformer encoder layers · embedding dim 256 · 8 heads · FF 1024 · GELU
• —Padding-masked mean pooling → linear classification head
• —8,297,735 parameters · runs on CPU in milliseconds

Evaluation

Measured on the test splits only. The model was trained on train + validation, so none of these 6,590 rows were seen during training. The benchmark script is in this repo (bench.py) so every number below can be reproduced.

Headline

The baseline matters: a model that ignores its input entirely and always says "joy" scores 30.3% on this data. 70.7% is 2.3× that floor, which is where the claim "it is actually learning" comes from. A number without a baseline means nothing.

Accuracy and macro-F1 are close (70.7 vs 68.2), which says the model is not ignoring the rare classes. That is the payoff of capping every class at 6,500 examples during training.

Per class

Performance is not uniform — it splits by source

A single headline number hides a 25-point gap. Short, emotionally explicit tweets are easy. Long, conversational Reddit comments are much harder. If your text looks like Reddit rather than Twitter, expect the lower number.

⚠️ Known limitations

1. Anger and neutral collapse into each other

The largest single failure. From the confusion matrix:

• —230 genuinely angry messages were labelled neutral
• —222 genuinely neutral messages were labelled anger

That is 452 errors in one confusion pair, and every one of them comes from the GoEmotions portion — dair-ai has no neutral class at all.

The cause is a labelling decision in this project, not a training failure. GoEmotions' annoyance and disapproval were mapped to anger, while confusion and curiosity were mapped to neutral. Mild annoyance and neutral commentary look nearly identical in text, so the model cannot separate categories that were merged by hand. A future version should give mild-negative its own class or drop it.

2. It follows emotional keywords past the point of truth

When an emotion word appears in the sentence, the model tends to predict that emotion far more often than it should:

The classic failure: "my team threw me a surprise party" is joy, and the model answers surprise with 87.6% confidence.

3. Implied emotion reads as neutral

The model keys off explicit emotional language. When feeling is only implied by a factual statement, it falls back to neutral:

4. Surprise is the weakest class

43.7% F1, and it under-predicts — 202 predictions against 242 real cases. Partly because GoEmotions' realization was mapped into surprise, which widened the class beyond its ordinary meaning.

5. The reference labels are themselves noisy

GoEmotions is crowd-labelled Reddit text and some gold labels are questionable. Example from the test set, labelled joy by the annotator:

"I don't think that would be an issue with [NAME]. He doesn't..."

Part of the 63.3% GoEmotions score is the answer key, not the model.

Intended use

Best used as a conversation-tone reader (see analyze_day), aggregating emotion across many messages, rather than as a single-sentence oracle. Treat it as an educational / entertainment model, not a clinical or production sentiment system.

Reproducing these numbers

python3 bench.py     # downloads the public test splits, runs the model, writes results.json

Prints accuracy, macro-F1, per-class precision/recall/F1, the full confusion matrix and the keyword probe. No PyTorch required — it runs the ONNX export.

Usage — ONNX (no PyTorch needed)

import json, re, numpy as np, onnxruntime as ort
vocab  = json.load(open("emotion_vocab.json"))
labels = ["sadness","joy","love","anger","fear","surprise","neutral"]
sess   = ort.InferenceSession("vibe-int8.onnx")
name   = sess.get_inputs()[0].name

ids = [vocab.get(w, 1) for w in re.findall(r"[a-z']+", "i love this so much".lower())][:64]
logits = sess.run(None, {name: np.array([ids], dtype=np.int64)})[0][0]
p = np.exp(logits - logits.max()); p /= p.sum()
print(labels[int(p.argmax())], f"{p.max()*100:.1f}%")

Usage — PyTorch

from emotion import predict_emotion, analyze_day

predict_emotion("i feel so happy right now")
# {'label': 'joy', 'confidence': 97.6, 'distribution': {...}, 'words': [...]}

analyze_day([
    "work made me feel sad and invisible today",
    "but then my friend called and i felt loved",
])
# {'dominant': 'love', 'valence': 'mixed', 'distribution': {...}, ...}

Training data

• —dair-ai/emotion — 6 emotions, tweets
• —GoEmotions — 27 emotions, Reddit, single-label rows only, mapped down to these 7 classes

Classes were capped at 6,500 examples each during training to prevent the joy bias.

Licence

Apache 2.0.