qinchuanhui/UDA-QA

Dataset Card for Dataset Name

[NIPS-2024] UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis (https://arxiv.org/abs/2406.15187)

UDA (Unstructured Document Analysis) is a benchmark suite for Retrieval Augmented Generation (RAG) in real-world document analysis. Each entry in the UDA dataset is organized as a document-question-answer triplet, where a question is raised from the document, accompanied by a corresponding ground-truth answer. The documents are retained in their original file formats without parsing or segmentation; they consist of both textual and tabular data, reflecting the complex nature of real-world analytical scenarios.

Dataset Details

Dataset Description

• —Curated by: Yulong Hui, Tsinghua University
• —Language(s) (NLP): English
• —License: CC-BY-SA-4.0
• —Repository: https://github.com/qinchuanhui/UDA-Benchmark

Uses

Direct Use

Question-answering tasks on complete unstructured documents.

After loading the dataset, you should also download the sourc document files from the folder `src_doc_files`.

More usage guidelines please refer to https://github.com/qinchuanhui/UDA-Benchmark

Extended Use

Evaluate the effectiveness of retrieval strategies using the evidence provided in the extended_qa_info folder.

Directly assess the performance of LLMs in numerical reasoning and table reasoning, using the evidence in the extended_qa_info folder as context.

Assess the effectiveness of parsing strategies on unstructured PDF documents.

Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

Descriptive Statistics

Data Fields

Additional Notes: Some sub-datasets may have multiple groundtruth answers, where the answers are organized as `answer1, answer2` (in FinHybrid, PaperTab and PaperText) or `shortanswer, longanswer` (in NqText); In sub-dataset TatHybrid, the answer is organized as a sequence, due to the involvement of the multi-span Q&A type. Additionally, some sub-datasets may have unique data fields. For example, `docurl in FetaTab and NqText describes the Wikipedia URL page, while answertype` and `answerscale` in TatHybrid provide extended answer references.

Dataset Creation

Source Data

<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

Data Collection and Processing

We collect the Q&A labels from the open-released datasets (i.e., source datasets), which are all annotated by human participants. Then we conduct a series of essential constructing actions, including source-document identification, categorization, filtering, data transformation.

Who are the source data producers?

[1] CHEN, Z., CHEN, W., SMILEY, C., SHAH, S., BOROVA, I., LANGDON, D., MOUSSA, R., BEANE, M., HUANG, T.-H., ROUTLEDGE, B., ET AL. Finqa: A dataset of numerical reasoning over financial data. arXiv preprint arXiv:2109.00122 (2021).

[2] ZHU, F., LEI, W., FENG, F., WANG, C., ZHANG, H., AND CHUA, T.-S. Towards complex document understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference on Multimedia (2022), pp. 4857–4866.

[3] DASIGI, P., LO, K., BELTAGY, I., COHAN, A., SMITH, N. A., AND GARDNER, M. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011 (2021).

[4] NAN, L., HSIEH, C., MAO, Z., LIN, X. V., VERMA, N., ZHANG, R., KRYS ́ CIN ́ SKI, W., SCHOELKOPF, H., KONG, R., TANG, X., ET AL. Fetaqa: Free-form table question answering. Transactions of the Association for Computational Linguistics 10 (2022), 35–49.

[5] KWIATKOWSKI, T., PALOMAKI, J., REDFIELD, O., COLLINS, M., PARIKH, A., ALBERTI, C., EPSTEIN, D., POLOSUKHIN, I., DEVLIN, J., LEE, K., ET AL. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7 (2019), 453–466.

Considerations for Using the Data

Personal and Sensitive Information

The dataset doesn't contain data that might be considered personal, sensitive, or private. The sources of data are publicly available reports, papers and wikipedia pages, which have been commonly utilized and accepted by the broader community.

Citation

Please kindly cite our paper if helps your research:

@article{hui2024uda,
  title={UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis},
  author={Hui, Yulong and Lu, Yao and Zhang, Huanchen},
  journal={arXiv preprint arXiv:2406.15187},
  year={2024}
}

Dataset Card Contact

qinchuanhui@gmail.com