MCINext/synthetic-persian-chatbot-topics-retrieval

Dataset Summary

Synthetic Persian Chatbot Topics Retrieval (SynPerChatbotTopicsRetrieval) is a Persian (Farsi) dataset for the Retrieval task, focused on chatbot topic identification. It is part of the FaMTEB (Farsi Massive Text Embedding Benchmark). The dataset was synthetically created using the GPT-4o-mini language model to simulate user queries and retrieve topic-relevant chatbot responses.

• —Language(s): Persian (Farsi)
• —Task(s): Retrieval (Chatbot Topic Retrieval)
• —Source: Synthetically generated using GPT-4o-mini
• —Part of FaMTEB: Yes

Supported Tasks and Leaderboards

This dataset is designed to evaluate the retrieval performance of text embedding models in chatbot applications. It benchmarks how well a model can retrieve topic-relevant chatbot responses from a large candidate set. Model performance is visible on the Persian MTEB Leaderboard on Hugging Face Spaces.

Construction

1. 1.A list of Persian chatbot topics (e.g., “سفر”، “سلامتی”) was compiled.
2. 2.GPT-4o-mini was used to generate natural language questions or phrases that would be representative of users initiating those topics.
3. 3.Each query was matched with its correct topic or response group in the dataset.
4. 4.Distractor responses from unrelated topics were added to simulate realistic retrieval difficulty.

The dataset was validated by human annotators to ensure that queries were aligned to their correct topic labels and that retrieval was meaningful.

Data Splits

Following the FaMTEB benchmark structure:

• —Train: 93,478 samples
• —Development (Dev): 0 samples
• —Test: 35,946 samples