IMISLab/DemosQA

DemosQA

We introduce DemosQA (δῆμος), a novel Greek QA dataset, which is constructed using social media user questions and community-reviewed answers to better capture the Greek social and cultural zeitgeist. It comprises questions extracted from the “r/greece” subreddit, each accompanied by four candidate answers, the selected best answer and its index, the date of posting, and the corresponding Reddit post ID. Candidate answers are ranked based on community voting, with the highest-upvoted response designated as the reference answer. This community-driven ranking mechanism not only ensures that the dataset captures genuine user preferences but also establishes a meaningful benchmark for assessing how closely large language models align with human judgments of response quality. For information about dataset creation, limitations etc. see the arxiv preprint.

<img src="demosqa.png" width="400"/>

Supported Task

This dataset supports evaluation of LLMs for Question Answering.

Language

All dataset samples are written in Greek.

Dataset Structure

The dataset is structured as a .csv file of 600 rows. The following data fields are provided:

id: (str) The post id. question: (str) The post question and its context. answers: (str): The string containing the four candidate answers. best_answer: (str) The best answer text selected by the community. best_answer_index: (str) The letter index of the best answer. date: (str) The post publication date.

Example code

from datasets import load_dataset

# Load the dataset.
test_split = load_dataset('IMISLab/DemosQA', split = 'test')
print(test_split[0])

Contact

If you have any questions/feedback about the dataset please e-mail one of the following authors:

giarelis@ceid.upatras.gr
cmastrokostas@ac.upatras.gr
karacap@upatras.gr

Citation

@misc{
  mastrokostas2026evaluatingmonolingualmultilinguallarge,
  title = {Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark}, 
  author = {Charalampos Mastrokostas and Nikolaos Giarelis and Nikos Karacapilidis},
  year = {2026},
  eprint = {2602.16811},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2602.16811}, 
}