MCINext/synthetic-persian-chatbot-conversational-sentiment-analysis-surprise

Dataset Summary

Synthetic Persian Chatbot Conversational SA – Surprise is a Persian (Farsi) dataset for the Classification task, focused on detecting the expression of "surprise" in user-chatbot conversations. It is part of the FaMTEB (Farsi Massive Text Embedding Benchmark). The dataset was synthetically generated using GPT-4o-mini and is a subset of the Synthetic Persian Chatbot Conversational Sentiment Analysis dataset.

• —Language(s): Persian (Farsi)
• —Task(s): Classification (Emotion Classification – Surprise)
• —Source: Synthetic, generated using GPT-4o-mini
• —Part of FaMTEB: Yes

Supported Tasks and Leaderboards

This dataset is used to evaluate the ability of text embedding models to identify and classify the presence of the emotion "surprise" in chatbot dialogues. Model performance is evaluated on the Persian MTEB Leaderboard on Hugging Face Spaces (filter by language: Persian).

Construction

This dataset was built using the following process:

• —175 distinct conversation topics were predefined
• —9 combinations of user and chatbot tones (e.g., formal, casual, childish)
• —One of 9 predefined emotions was chosen (including surprise)
• —An emotion intensity was assigned: neutral, moderate, or high
• —GPT-4o-mini generated the final conversation samples

Labeling Strategy:

• —Positive Label: Emotion intensity is moderate or high
• —Negative Label: Emotion intensity is neutral

The FaMTEB paper (Table 1) reports a 93% average accuracy for the parent dataset based on human evaluation of LLM-generated emotion labels.

Data Splits

The full parent dataset (SynPerChatbotConvSAClassification) is split as follows:

• —Train: 4,496 samples
• —Dev: 0 samples
• —Test: 1,499 samples

This surprise-specific subset contains 530 examples, but its internal split is not separately specified. It is assumed to be part of the above aggregate totals.