MCINext/synthetic-persian-chatbot-summary-retrieval

Dataset Summary

Synthetic Persian Chatbot Summary Retrieval (SynPerChatbotSumSRetrieval) is a Persian (Farsi) dataset designed for the novel Summary Retrieval task, introduced as part of the FaMTEB (Farsi Massive Text Embedding Benchmark). This dataset was synthetically generated using the GPT-4o-mini Large Language Model and is derived from the Synthetic Persian Chatbot Dataset. The core task is to retrieve the correct, pre-generated summary that corresponds to a given user–chatbot conversation.

• —Language(s): Persian (Farsi)
• —Task(s): Summary Retrieval
• —Source: Synthetically generated using GPT-4o-mini, derived from the Synthetic Persian Chatbot Dataset
• —Part of FaMTEB: Yes

Supported Tasks and Leaderboards

This dataset is used to evaluate the performance of text embedding models on the Summary Retrieval task. This involves embedding a full conversation and a set of candidate summaries, then retrieving the most appropriate summary. Model performance is benchmarked on the Persian MTEB Leaderboard (filter by language: Persian).

Construction

The dataset originates from the Synthetic Persian Chatbot Dataset, where GPT-4o-mini was used to generate user–chatbot conversations across 175 predefined topics and 19 tone combinations. For each conversation, a corresponding summary was also generated using GPT-4o-mini.

As outlined in the FaMTEB paper, the quality of this synthetic dataset was verified by human annotators. For SynPerChatbotSumSRetrieval, summaries were evaluated for their relevance to the conversations, achieving a 100% accuracy rating, indicating strong alignment between conversation and summary.

Data Splits

Data split details from the FaMTEB paper (Table 5):

• —Train: 8,709 samples
• —Development (Dev): 0 samples
• —Test: 1,537 samples