yyyyyyyyy111/MMDocIR_Evaluation_Dataset

Evaluation Datasets

Evaluation Set Overview

MMDocIR evaluation set includes 313 long documents averaging 65.1 pages, categorized into ten main domains: research reports, administration&industry, tutorials&workshops, academic papers, brochures, financial reports, guidebooks, government documents, laws, and news articles. Different domains feature distinct distributions of multi-modal information. Overall, the modality distribution is: Text (60.4%), Image (18.8%), Table (16.7%), and other modalities (4.1%).

MMDocIR evluation set encompasses 1,658 questions, 2,107 page labels, and 2,638 layout labels. The modalities required to answer these questions distribute across four categories: Text (44.7%), Image (21.7%), Table (37.4%), and Layout/Meta (11.5%). The ``Layout/Meta'' category encompasses questions related to layout information and meta-data statistics. Notably, the dataset poses several challenges: 254 questions necessitate cross-modal understanding, 313 questions demand evidence across multiple pages, and 637 questions require reasoning based on multiple layouts. These complexities highlight the need for advanced multi-modal reasoning and contextual understanding.

Dataset Format

1. Overall Annotation

`MMDocIR_annotations.jsonl` contains 313 json lines, each for the annotations corresponding to a long document.

Each QA item consists of :

Each layout item consists of:

2. MMDocIR Pages related data

`MMDocIR_pages.parquet` contains 20,395 document page screenshots from 313 documents. The parquet file is formatted as:

3. MMDocIR Layouts related data

`MMDocIR_layouts.parquet` contains 170,338 document layouts from 313 documents. The parquet file is formatted as:

Miscellaneous Document Files

The miscellaneous document-related files can be found in: `doc_miscellaneous`. They are not required in MMDocIR inference and encoding. But, it can be helpful to facilitate the understanding and customizations on MMDocIR.

1. MMDocIR Original PDF files

Documents are mainly collected from [DocBench](https://github.com/Anni-Zou/DocBench) and [MMLongBench-Doc](https://github.com/mayubo2333/MMLongBench-Doc). We process/filter/fix these files to get the final 313 documents as in `doc_pdfs.rar`.

2. MMDocIR Page-related files

• —`page_images.rar` contains 20,395 document screenshots in JPEG format.

• —`page_content.rar` contains 313 files, each comprising the page-level content in json-line format:

3. MMDocIR Layout-related files

• —layout_images.rar contains 14,826 layout (table/figure) images cropped from page screenshots, in JPEG format.

• —layout_text_images.rar contains 155,512 layout (text/equation) images cropped from page screenshots, in JPEG format.

• —layout_content.rar contains 313 files, each comprising the layout-level content in json-line format:

Citation Information

If you use this dataset in your research, please cite the original dataset as follows:

@misc{dong2025mmdocirbenchmarkingmultimodalretrieval,
      title={MMDocIR: Benchmarking Multi-Modal Retrieval for Long Documents}, 
      author={Kuicai Dong and Yujing Chang and Xin Deik Goh and Dexun Li and Ruiming Tang and Yong Liu},
      year={2025},
      eprint={2501.08828},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2501.08828}, 
}