ibm-research/PopQA_robustness

Dataset Card for "PopQA-robustness"

Dataset Summary

PopQS-robustness is an expanded version of the PopQA dataset (https://aclanthology.org/2023.acl-long.546/) but with perturbations of the original input questions. It is intended for use as a benchmark for evaluating model robustness on question-answering to these perturbations.

Data Instances

popqa_robustness

• —Size of downloaded dataset file: 26.4 MB

Data Fields

boolq_robustness

• —id (integer): original question grouping ID
• —question (string): variant of question from BoolQ.
• —variant_id (integer): identifier of the variant. 0 indicates it is the original unperturbed question.
• —variant_type (string): name of the expansion variant type. "original" is the original question; "simple" is a superficial non-semantic perturbation; "paraphrase" is a semantic paraphrase of the question.
• —possible_answers (string): list of strings of possible answers.

Citation Information

@misc{ackerman2024novelmetricmeasuringrobustness,
      title={A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios}, 
      author={Samuel Ackerman and Ella Rabinovich and Eitan Farchi and Ateret Anaby-Tavor},
      year={2024},
      eprint={2408.01963},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2408.01963}, 
}