SajjadAyoubi/persian_qa

PersianQA: a dataset for Persian Question Answering

Persian Question Answering (PersianQA) Dataset is a reading comprehension dataset on Persian Wikipedia. The crowd-sourced dataset consists of more than 9,000 entries. Each entry can be either an impossible to answer or a question with one or more answers spanning in the passage (the context) from which the questioner proposed the question. Much like the SQuAD2.0 dataset, the impossible or unanswerable questions can be utilized to create a system which "knows that it doesn't know the answer". On top of that, the dataset has 900 test data available. Moreover, the first models trained on the dataset, Transformers, are available. All the crowd workers of the dataset are native Persian speakers. Also, it worth mentioning that the contexts are collected from all categories of the Wiki (Historical, Religious, Geography, Science, etc.) At the moment, each context has 7 pairs of questions with one answer and 3 impossible questions.

Dataset

Access/Download

• —You can find the data under the `dataset/` directory. and use it like this

import read_qa # is avalible at src/read_ds.py
train_ds = read_qa('pqa_train.json')
test_ds  = read_qa('pqa_test.json')

• —Alternatively, you can also access the data through the HuggingFace🤗 datasets library
• —First, you need to install datasets using this command in your terminal:

pip install -q datasets

• —Then import persian_qa dataset using load_dataset:

from datasets import load_dataset
dataset = load_dataset("SajjadAyoubi/persian_qa")

Examples

Statistic

The lengths are on the token level.

• —for more about data and more example see here

Models

Currently, two models (baseline) on HuggingFace🤗 model hub are using the dataset.

Citation

Yet, we didn't publish any papers on the work. However, if you did, please cite us properly with an entry like one below.

@misc{PersianQA,
  author          = {Ayoubi, Sajjad \& Davoodeh, Mohammad Yasin},
  title           = {PersianQA: a dataset for Persian Question Answering},
  year            = 2021,
  publisher       = {GitHub},
  journal         = {GitHub repository},
  howpublished    = {\url{https://github.com/SajjjadAyobi/PersianQA}},
}