MCINext/synthetic-persian-chatbot-rag-summary-retrieval

Dataset Summary

Synthetic Persian Chatbot RAG Summary Retrieval (SynPerChatbotRAGSumSRetrieval) is a Persian (Farsi) dataset for the Summary Retrieval task, specifically built for Retrieval-Augmented Generation (RAG) systems. It is part of the FaMTEB (Farsi Massive Text Embedding Benchmark). The dataset was synthetically generated using GPT-4o-mini and is derived from the Synthetic Persian Chatbot RAG Dataset. It evaluates the ability of models to match conversations—possibly incomplete or context-dependent—to their correct summaries.

• —Language(s): Persian (Farsi)
• —Task(s): Summary Retrieval
• —Source: Synthetically generated using GPT-4o-mini, from Synthetic Persian Chatbot RAG Dataset
• —Part of FaMTEB: Yes

Supported Tasks and Leaderboards

This dataset benchmarks how effectively a model can retrieve the correct summary for a Persian chatbot conversation in a RAG-like scenario. It emphasizes real-world complexities like partial history and conversational context. Model performance is listed on the Persian MTEB Leaderboard.

Construction

1. 1.GPT-4o-mini generated RAG-style Persian chatbot conversations with varying context depths.
2. 2.Each conversation was paired with an LLM-generated summary.
3. 3.The dataset includes these conversation–summary pairs for supervised evaluation.
4. 4.Human validation confirmed the summaries’ alignment with conversations, achieving 99% accuracy in relevance evaluation.

Data Splits

• —Train: 3,261 samples
• —Development (Dev): 0 samples
• —Test: 1,087 samples