Modelpublic

EsferSami/meme-sentiment-dual-cross-attention-v1

sourceHugging Facemitupdated 5mo agoView on Hugging Face

0likes

Model Card

Meme Sentiment Analysis — Dual Cross-Attention (V1)

Author: EsferSami Task: Multilingual Multimodal Meme Sentiment Analysis Dataset: MemoSen — Kaggle Version: V1 — Frozen Encoders (Pure Pretrained Baseline)

Overview

MemoSen is a multilingual multimodal meme sentiment dataset containing Bengali, English, and code-mixed meme captions paired with meme images. Each sample has a sentiment label: Positive, Negative, or Neutral.

This model resolves cross-modal conflict between meme text and image using a dual cross-attention framework. Meme understanding is challenging because text and image often conflict — especially in sarcastic, ironic, or culturally specific memes.

Model Architecture

Component	Details
Text Encoder	XLM-RoBERTa base (frozen)
Image Encoder	EVA02-CLIP-L/14 from QuanSun (frozen)
Feature Dim	768 projected to 512
Fusion	Dual Cross-Attention + Adaptive Gating
Conflict Detection	Cosine similarity + MLP
Reliability Weighting	Per-modality sigmoid scoring
Classifier	MLP (512 -> 256 -> 128 -> 3)

Dataset

Dataset: MemoSen on Kaggle

Split	Ratio
Train	70%
Validation	15%
Test	15%

Classes: Positive, Negative, Neutral Languages: Bengali, English, Code-mixed (Banglish)

Training Configuration

Parameter	Value
Epochs	50 (early stopped at 26)
Optimizer	AdamW (lr=1e-4)
Scheduler	CosineAnnealingLR
Loss	Focal Loss + Class Weights
Early Stopping	Patience = 7
Batch Size	32
Max Text Length	128 tokens
Image Size	224 x 224
GPU	T4 (Kaggle)

Results (Validation Set)

Metric	Score
Macro-F1	0.5584

Full test set metrics will be updated after evaluation. Main metric: Macro-F1

Training Curves

[image]

Notes

—V1 uses fully frozen encoders — only fusion and classification layers are trained
—Trainable parameters are limited to adaptation, cross-attention, conflict detection, reliability weighting, gated fusion, and MLP classifier
—V2 will experiment with unfreezing encoders and architecture modifications

Citation

If you use MemoSen dataset, please refer to the original dataset on Kaggle: https://www.kaggle.com/datasets/arifkaggle979/memosen-dataset