OpenDFM/AirQA

AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation

This repository contains the test set, the metadata, processed_data and papers for the AirQA dataset introduced in our paper **AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation** accepted to ICLR 2026. Detailed instructions for using the dataset will soon be publicly available in our official repository.

AirQA is a human-annotated multi-modal multitask Artificial Intelligence Research Question Answering dataset, which encompasses 1,246 examples and 13,956 papers, aiming at evaluating an agent’s research capabilities in realistic scenarios. It is the first dataset that encompasses multiple question types, also the first to bring function-based evaluation into QA domain, enabling convenient and systematic assessment of research capabilities.

🔍 Quick Start

Load the AirQA dataset in one line using Hugging Face datasets:

from datasets import load_dataset

dataset = load_dataset("OpenDFM/AirQA")

However, we recommend referring to our official repository for complete usage instructions, including the data format and evaluation scripts.

📂 Folder Structure

AirQA
|── data/
|   |── test.parquet # test set (simple, for minimal usage)
|   |── test_data.jsonl # test set (complete, including function-based evaluation)
|   └── uuid2title.json # mapping from paper UUID to title
|── metadata/
|   |── 000ab6db-4b65-5dc0-8393-fbc2c05843c8.json
|   └── ... # more metadata dicts
|── papers/
|   |── acl2016/
|   |   └── 16c3a7ad-d638-5ebf-a72a-bd58f06c16d7.pdf
|   |── acl2019/
|   |   └── c7563d97-695f-5c77-8021-334bf2ff9ddb.pdf
|   |── acl2023/
|   |   |── 001ab93b-7665-5d56-a28e-eac95d2a9d7e.pdf
|   |   └── ... # more .pdf published in ACL 2023
|   └── ... # other sub-folders of paper collections
|── processed_data/
|   |── 000ab6db-4b65-5dc0-8393-fbc2c05843c8.json # cached data for PDF parsing
|   └── ... # more cached data for PDFs
└── README.md

Due to Hugging Face's limit on the number of files in a single folder, we packaged metadata and processed_data into archives.

📊 Dataset Statistics

Our dataset encompasses papers from 34 volumes, spanning 7 conferences over 16 years. The detailed distribution is summarized below.

<details><summary>👇🏻 Click to view the paper distribution of dataset</summary>

</details>

✍🏻 Citation

If you find this dataset useful, please cite our work:

@misc{huang2025airqacomprehensiveqadataset,
      title={AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation}, 
      author={Tiancheng Huang and Ruisheng Cao and Yuxin Zhang and Zhangyi Kang and Zijian Wang and Chenrun Wang and Yijie Luo and Hang Zheng and Lirong Qian and Lu Chen and Kai Yu},
      year={2025},
      eprint={2509.16952},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.16952}, 
}