ukr-detect/ukr-emotions-binary

EmoBench-UA: Emotions Detection Dataset in Ukrainian Texts

<img alt="EmoBench-UA" style="width: 75%; height: 75%" src="intro_logo.png">

EmoBench-UA: the first of its kind emotions detection dataset in Ukrainian texts. This dataset covers the detection of basic emotions: Joy, Anger, Fear, Disgust, Surprise, Sadness, or None. Any text can contain any amount of emotion -- only one, several, or none at all. The texts with None emotions are the ones where the labels per emotions classes are 0.

Binary: specifically this dataset contains binary labels indicating simply presence of any emotion in the text.

Data Collection

The data collection was done via Toloka.ai crowdsourcing platform. For original Ukrainian texts, we used the opensourced corpus of Ukrainian tweets.

Firstly, we did data pre-filtering:

Length We applied a length-based filter, discarding texts that were too short (N words < 5), as such samples often consist of hashtags or other non-informative tokens. Similarly, overly long texts (N words >= 50) were excluded, as longer sequences tend to obscure the central meaning and make it challenging to accurately identify the expressed emotions.

Toxicity While toxic texts can carry quite strong emotions, to ensure annotators well-being and general appropriateness of our corpus, we filtered out too toxic instances using our opensourced toxicity classifier.

Emotional Texts Pre-selection To avoid an excessive imbalance toward emotionless texts, we performed a pre-selection step aimed at identifying texts likely to express emotions. Specifically, we applied the English emotion classifier DistillRoBERTa-Emo-EN on translated Ukrainian texts with NLLB model.

Then, we utilized several control strategies to ensure the quality of the data:

• —annotators were native Ukrainian speakers;
• —annotators were obliged to finish training and examination before being allowed to perform annotation;
• —annotators were permanently banned if they submitted the last three task pages in under 15 seconds each, indicating low engagement;
• —a one-day ban was triggered if three consecutive pages were skipped;
• —annotators were asked to take a 30-minute break after completing 25 consecutive pages;
• —control tasks were randomly injected to check.

Finally, each sample were annotated by 5 annotators. We took for datasets only instances with 90% confidence score.

Splits Statistics

Overall, the datasets contains of 4949 labelled instances. Krippendorff's alpha annotation agreement score is 0.85. Then, we partitioned the dataset into fixed train/development/test subsets following a 50/5/45% split ratio.

<img alt="Splits Statistic" style="width: 50%; height: 50%" src="data_stats.png">

Citation

If you would like to acknowledge our work, please, cite the following manuscript:

@inproceedings{dementieva-etal-2025-emobench,
    title = "{E}mo{B}ench-{UA}: A Benchmark Dataset for Emotion Detection in {U}krainian",
    author = "Dementieva, Daryna  and
      Babakov, Nikolay  and
      Fraser, Alexander",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-emnlp.107/",
    doi = "10.18653/v1/2025.findings-emnlp.107",
    pages = "2025--2048",
    ISBN = "979-8-89176-335-7",
    abstract = "While Ukrainian NLP has seen progress in many texts processing tasks, emotion classification remains an underexplored area with no publicly available benchmark to date. In this work, we introduce \textbf{EmoBench-UA}, the first annotated dataset for emotion detection in Ukrainian texts. Our annotation schema is adapted from the previous English-centric works on emotion detection (Mohammad et al., 2018; Mohammad, 2022) guidelines. The dataset was created through crowdsourcing using the Toloka.ai platform ensuring high-quality of the annotation process. Then, we evaluate a range of approaches on the collected dataset, starting from linguistic-based baselines, synthetic data translated from English, to large language models (LLMs). Our findings highlight the challenges of emotion classification in non-mainstream languages like Ukrainian and emphasize the need for further development of Ukrainian-specific models and training resources."
}

Contacts

Daryna Dementieva, Nikolay Babakov