MCINext/synthetic-persian-chatbot-rag-tone-chatbot-classification

Dataset Summary

Synthetic Persian Chatbot RAG Tone Chatbot Classification (SynPerChatbotRAGToneChatbotClassification) is a Persian (Farsi) dataset designed for the Classification task, specifically to identify the chatbot's tone in user-chatbot conversations within Retrieval-Augmented Generation (RAG) contexts. It is part of the FaMTEB (Farsi Massive Text Embedding Benchmark). The dataset was synthetically generated using the GPT-4o-mini Large Language Model and derived from the Synthetic Persian Chatbot RAG Dataset. Each entry includes a multi-turn conversation, and the goal is to classify the chatbot's tone (e.g., formal, casual, childish, aggressive, street-style).

• —Language(s): Persian (Farsi)
• —Task(s): Classification (Chatbot Tone Classification in RAG Chatbots)
• —Source: Synthetic, created using GPT-4o-mini
• —Part of FaMTEB: Yes

Supported Tasks and Leaderboards

This dataset is used to evaluate model performance on tone classification in RAG-style chatbot conversations. Benchmark results are available on the Persian MTEB Leaderboard on Hugging Face Spaces (filterable by language: Persian).

Construction

The dataset was constructed using:

• —175 unique conversation topics
• —19 combinations of user and chatbot tones (e.g., formal, casual, childish, aggressive, street-style)
• —Randomized conversation history lengths before each new chatbot response
• —The chatbot tone used during generation is assigned as the classification label

According to the FaMTEB paper (Table 1), the dataset achieved an 86% accuracy based on human evaluation of the GPT-4o-mini-generated chatbot tone annotations.

Data Splits

As specified in the FaMTEB paper (Table 5):

• —Train: 3,261 samples
• —Development (Dev): 0 samples
• —Test: 1,087 samples