luciacpassaro/MM-Emo

Dataset Card for MM-Emo

Dataset Summary

MM-Emo is an English multimodal multi-label emotion recognition dataset built from posts on X (formerly Twitter). It is designed for emotion recognition from paired text and image social media content.

The dataset contains a column with the post ID and a set of binary emotion label columns. Label columns starting with M_ correspond to multimodal labels, while columns starting with T_ correspond to text-only labels.

For privacy reasons, the original contents of the posts are not distributed with the dataset. To reconstruct the dataset used in experiments, users must retrieve the original post text and associated image(s) using the X API.

A subset of posts contains multiple images. In those cases, the reconstructed experimental dataset should be expanded into multiple samples that share the same post text and labels, but differ in the associated image.

According to the associated paper, MM-Emo is a gold-labelled multimodal dataset of 900 tweets, annotated for multilabel emotion recognition using Plutchik’s emotion taxonomy plus a neutral class.

Supported Tasks

This dataset is intended for:

• —Multimodal multi-label emotion classification
• —Text-only multi-label emotion classification
• —Social media emotion analysis
• —Cross-modal emotion modelling

Languages

The dataset language is English.

Dataset Structure

Data Instances

Each row in the released files contains:

• —a social media post ID
• —one or more text-only binary emotion labels (T_*)
• —one or more multimodal binary emotion labels (M_*)

Because raw text and images are not redistributed, a usable training or evaluation instance must be reconstructed externally from the post ID.

A simplified example:

{
  "post_id": "1234567890123456789",
  "T_anger": 0,
  "T_anticipation": 1,
  "T_disgust": 0,
  "T_fear": 0,
  "T_joy": 1,
  "T_sadness": 0,
  "T_surprise": 0,
  "T_trust": 1,
  "T_neutral": 0,
  "M_anger": 0,
  "M_anticipation": 1,
  "M_disgust": 0,
  "M_fear": 0,
  "M_joy": 1,
  "M_sadness": 0,
  "M_surprise": 1,
  "M_trust": 1,
  "M_neutral": 0
}

Data Fields

• —post_id: unique identifier of the X post
• —T_anger, T_anticipation, T_disgust, T_fear, T_joy, T_sadness, T_surprise, T_trust, T_neutral: binary labels for text-only emotion interpretation
• —M_anger, M_anticipation, M_disgust, M_fear, M_joy, M_sadness, M_surprise, M_trust, M_neutral: binary labels for multimodal emotion interpretation based on the combined text+image post

Reconstructing the Experimental Dataset

To rebuild the dataset used in the experiments, two elements are required for each post ID:

1. 1.the text of the post
2. 2.the associated image or images

Because some posts contain multiple images, they should be expanded into multiple samples during reconstruction. Each expanded sample should:

• —keep the same text
• —keep the same labels
• —use one different image from the original post

This means that the number of reconstructed training or evaluation samples may be larger than the number of rows in the CSV files.

Data Splits

The dataset files are:

• —train_MM-Emo.csv: training set
• —test_MM-Emo.csv: test set
• —val_MM-Emo.csv: validation set
• —silver_MM-Emo.csv: portion of silver-label-only samples, used to augment the training set in some experiments

Label Space

The dataset uses nine binary emotion dimensions:

• —anger
• —anticipation
• —disgust
• —fear
• —joy
• —sadness
• —surprise
• —trust
• —neutral

This is a multi-label setup: a single post may express more than one emotion at the same time.

The associated paper describes the task as multilabel emotion recognition from paired image-text posts.

Dataset Creation

Curation Rationale

MM-Emo was created to support research on multimodal multilabel emotion recognition in realistic social media settings. Compared with unimodal or single-label benchmarks, this dataset targets the richer setting where users communicate affect through a combination of short text, imagery, and social-media-specific cues.

Source Data

The source data comes from posts on X / Twitter containing paired textual and visual content.

The released dataset does not include the original text or images. Instead, it distributes post identifiers and emotion annotations, and users must reconstruct the original content independently via the platform API.

Annotations

The associated paper describes MM-Emo as a gold-labelled dataset with multilabel annotations collected from five annotators per tweet. The label space follows Plutchik’s taxonomy plus Neutral, and labels are binarized through score thresholding.

In addition to the gold-labelled train/validation/test portions, the release also includes a silver-labelled file intended for data augmentation experiments.

Personal and Sensitive Information

Because the source material comes from public social media, original posts may contain usernames, personal references, or other sensitive information. To reduce redistribution and privacy risks, the dataset does not directly include raw post contents.

Users reconstructing the dataset should ensure compliance with:

• —X platform terms of service
• —applicable privacy and data protection requirements
• —their local institutional or research ethics guidelines

Considerations for Using the Data

Social Impact of Dataset

This dataset may support progress in:

• —multimodal affective computing
• —emotion-aware social media analysis
• —human-centered multimodal NLP
• —low-resource multimodal classification research

Potential applications include social listening, content understanding, and computational social science. However, emotion recognition can also be misused for profiling, surveillance, or manipulative targeting, so downstream uses should be considered carefully.

Biases

Possible sources of bias include:

• —annotator subjectivity in emotion interpretation
• —demographic and cultural differences in emotional expression
• —platform-specific communication norms
• —temporal drift in language, memes, and visual conventions
• —survivorship bias from posts that remain available through the API

Because reconstruction depends on current API access and content availability, the practical dataset may change over time.

Limitations

• —The dataset is small
• —Original text and images are not distributed
• —Rehydration depends on API access and content availability
• —Some posts may no longer be available
• —Posts with multiple images require additional reconstruction logic
• —Results may not generalize across platforms, domains, time periods, or languages
• —Low-support emotions may be difficult to model reliably because of class imbalance

Recommended Uses

• —benchmarking multimodal multi-label emotion classifiers
• —comparing text-only and multimodal emotion recognition
• —studying the contribution of visual information in social media emotion analysis
• —experiments on threshold tuning, class imbalance, and weak supervision
• —silver-label data augmentation experiments

Discouraged Uses

• —inferring stable psychological traits about individuals
• —surveillance or profiling of social media users
• —high-stakes decision-making based on inferred emotion
• —uses that violate X platform policies or privacy expectations

Citation

If you use MM-Emo, please cite:

Passaro, L., Amadei, D., & Bacciu, D. (2026). Emotion recognition in multimodal social data. In ESANN 2026 Proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. Bruges, Belgium and online. https://doi.org/10.14428/esann/2026.ES2026-287

In-text citation: (Passaro et al., 2026)

@inproceedings{passaro2026emotion,
  title={Emotion Recognition in Multimodal Social Data},
  author={Passaro, Lucia and Amadei, Davide and Bacciu, Davide},
  booktitle={ESANN 2026 Proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning},
  year={2026},
  address={Bruges, Belgium and online},
  isbn={9782875870964},
  doi={10.14428/esann/2026.ES2026-287}
}

Contact

For questions about the dataset, please contact the dataset authors through the channels provided in the associated publication or repository.