AmazonScience/document-haystack

1---2tags:3- long-context4- multimodal5- llm6- vlm7- benchmark8- text9- image10- vision11- document12- pdf13task_categories:14- question-answering15- visual-question-answering16- document-question-answering17language:18- en19viewer: false20---21# Document Haystack Dataset22 23This repository contains the dataset for the paper “[Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark](https://arxiv.org/abs/2507.15882)”.24 25---26 27## 📑 Abstract Paper28 29The proliferation of multimodal Large Language Models has significantly advanced the ability to analyze and understand complex data inputs from different modalities. However, the processing of long documents remains under-explored, largely due to a lack of suitable benchmarks. To address this, we introduce Document Haystack, a comprehensive benchmark designed to evaluate the performance of Vision Language Models (VLMs) on long, visually complex documents. Document Haystack features documents ranging from 5 to 200 pages and strategically inserts pure text or multimodal text+image "needles" at various depths within the documents to challenge VLMs' retrieval capabilities. Comprising 400 document variants and a total of 8,250 questions, it is supported by an objective, automated evaluation framework. We detail the construction and characteristics of the Document Haystack dataset, present results from prominent VLMs and discuss potential research avenues in this area.30 31---32 33## 🗂️ Overview34 35**Document Haystack** is a comprehensive benchmark dataset designed to evaluate the **long-context retrieval** and **multimodal document understanding** capabilities of Vision Language Models (VLMs).36 37It expands on the *Needle in a Haystack* concept by embedding *needles* — short key-value statements in pure text or as multimodal text+image snippets — within real-world long documents (5–200 pages). These needles test whether models can **locate specific information** hidden deep inside long, complex documents with textual, visual or mixed content.38 39---40 41## 🎯 Benchmark Design42 43- **Key-value pairs:** Each needle follows the pattern “The secret *KEY* is *VALUE*.” where `VALUE` appears as either text or an image. For example: “The secret sport is *basketball*.”. The keys span diverse categories including sports, animals, currencies, fruits, musical instruments, and more (see Table 3 in the paper for the complete category list).44- **Tasks:** Each needle has an associated retrieval question: “What is the secret *KEY* in the document?”45- **Objective scoring:** The VLM’s answer is checked for the correct `VALUE` (or acceptable aliases for text+image needles).46 47---48 49## ✅ Key Features50 51- **Document Lengths:** 5, 10, 25, 50, 75, 100, 150, 200 pages52- **Total Documents:** 400 document variants53- **Total Questions:** 8,250 unique retrieval queries54- **Needle Types:**55  - *Text Needles*: Pure text (e.g., “The secret sport is *basketball*.”)56  - *Text+Image Needles*: The value is shown as an image (e.g., “The secret sport is *\<image of basketball\>*.”)57- **Formats Provided:**58  - Original PDF59  - Page-wise images (200 DPI)60  - Parsed plain text (for text needles only)61 62---63 64## 🔬 Use Cases65 66- Stress-test multimodal VLMs for long-context understanding67- Compare retrieval from parsed pdf text vs images vs original pdfs68- Explore text vs image vs mixed retrieval challenges69- Measure performance drop with increasing context length70 71---72 73## 📦 Document Haystack Format Variants74 75| Benchmark Set | Format | Description | Use Case |76|----------------|--------|-------------------------------|-------------------------------|77| (1) Text needles | PDF | Original document format | VLMs supporting PDF input |78|  | Image | 200 DPI page-wise images | VLMs requiring image input |79|  | Text | Extracted plain text | Text-only LLMs |80| (2) Text+Image needles | PDF | Original document format | VLMs supporting PDF input |81|  | Image | 200 DPI page-wise images | VLMs requiring image input |82 83---84 85## 📊 Document Haystack Characteristics86 87| # Pages | 5 | 10 | 25 | 50 | 75 | 100 | 150 | 200 | Total |88|----------------|-----|-----|------|------|------|-------|-------|-------|-------|89| Text Needles # Documents | 25 | 25 | 25 | 25 | 25 | 25 | 25 | 25 | 200 |90| Text Needles # Questions | 125 | 250 | 625 | 625 | 625 | 625 | 625 | 625 | 4125 |91| Text+Image Needles # Documents | 25 | 25 | 25 | 25 | 25 | 25 | 25 | 25 | 200 |92| Text+Image Needles # Questions | 125 | 250 | 625 | 625 | 625 | 625 | 625 | 625 | 4125 |93| **Total Documents** | **50** | **50** | **50** | **50** | **50** | **50** | **50** | **50** | **400** |94| **Total Questions** | **250** | **500** | **1250** | **1250** | **1250** | **1250** | **1250** | **1250** | **8250** |95 96---97 98## 🗄️ Dataset Structure99 100Below is an example of the dataset’s folder layout:101 102```103DocumentHaystack/104├── AIG/105│   ├── AIG_5Pages/106│   │   ├── AIG_5Pages_ImageNeedles.pdf107│   │   ├── AIG_5Pages_TextNeedles.pdf108│   │   ├── Images_TextImageNeedles/109│   │   │   ├── AIG_5Pages_ImageNeedles_page_1.jpg110│   │   │   ├── AIG_5Pages_ImageNeedles_page_2.jpg111│   │   │   ├── ...112│   │   ├── Images_TextNeedles/113│   │   │   ├── AIG_5Pages_TextNeedles_page_1.jpg114│   │   │   ├── AIG_5Pages_TextNeedles_page_2.jpg115│   │   │   ├── ...116│   │   ├── Text_TextNeedles/117│   │   │   ├── AIG_5Pages_TextNeedles_page_1.txt118│   │   │   ├── AIG_5Pages_TextNeedles_page_2.txt119│   │   │   ├── ...120│   │   ├── needles_info.csv121│   │   ├── needles.csv122│   │   ├── prompt_questions.txt123│   ├── AIG_10Pages/124│   ├── AIG_25Pages/125│   ├── AIG_50Pages/126│   ├── AIG_75Pages/127│   ├── AIG_100Pages/128│   ├── AIG_150Pages/129│   ├── AIG_200Pages/130│131│   needles.csv132│   prompt_questions.txt133│134├── AmericanAirlines/135├── APA/136├── BankOfMontreal/137...138```139 140---141 142## 📁 File and Folder Descriptions143 144Below is an explanation of the files inside the **AIG_5Pages** subfolder:145 146| File/Folder | Description |147|-------------------------------|--------------------------------------------------------------------------|148| **AIG_5Pages_ImageNeedles.pdf** | PDF version with hidden Text+Image needles |149| **AIG_5Pages_TextNeedles.pdf** | PDF version with hidden Text-only needles |150| **Images_TextImageNeedles/** | Folder with page-wise JPGs of the Text+Image needles PDF |151| **Images_TextNeedles/** | Folder with page-wise JPGs of the Text-only needles PDF |152| **Text_TextNeedles/** | Folder with plain `.txt` files per page for Text-Needles version |153| **needles.csv** | Lists the key-value pairs inserted in the document variant |154| **needles_info.csv** | Detailed placement metadata for each needle (page, coordinates, font, etc.) |155| **prompt_questions.txt** | Contains the questions the model must answer for needle retrieval |156 157Within the main **AIG folder**, you'll find two key files: *needles.csv*, which lists all 25 needles that are utilized across the different AIG variants, and *prompt_questions.txt*, which contains the complete set of 25 prompts used throughout the AIG variants.158 159There are 25 top-level subfolders in total, each referring to a different document (e.g., **AIG/**, **AmericanAirlines/**, **APA/**, **BankOfMontreal/**), each structured the same way.160 161---162 163## 📌 `needles_info.csv`164 165Each document variant includes a `needles_info.csv` detailing every needle’s properties:166 167**Example rows:**168```169The secret office supply is a "pencil".,2,8,gray,white,0.339,0.931,times-bold,90170The secret sport is "basketball".,3,10,green,white,0.917,0.746,times-bolditalic,112171```172 173| Column | Description |174|---------------------|----------------------------------------------|175| **Needle** | Needle text statement (the hidden key-value pair) |176| **Page** | Page number where needle is inserted |177| **Font Size** | Font size |178| **Text Color** | Foreground color |179| **Background Color** | Background color |180| **X** | X coordinate (normalized 0–1) |181| **Y** | Y coordinate (normalized 0–1) |182| **Font** | Font type |183| **Scale** | Image scale (for text+image needles) |184 185**Placement:**186- Needles are randomly placed across equal, non-overlapping page ranges to ensure coverage throughout the document.187- Same locations are reused for both text-only and text+image sets.188 189---190 191## 📏 Reference Evaluation192 193Use with the [Document Haystack Benchmark Code](https://github.com/amazon-science/document-haystack) for:194- Fully automated inference & scoring pipelines195- Heatmap generation for depth-based performance196 197---198 199## 📝 License200 201This project is licensed under the **CC-BY-NC-4.0** License - see the GitHub LICENSE file for details.202 203---204 205## 📚 Citation206 207If you use this work in any way, please cite:208 209```bibtex210@article{huybrechts2025document,211  title={Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark},212  author={Huybrechts, Goeric and Ronanki, Srikanth and Jayanthi, Sai Muralidhar and Fitzgerald, Jack and Veeravanallur, Srinivasan},213  journal={arXiv preprint arXiv:2507.15882},214  year={2025}215}216```217 218---219 220## 👥 Authors221 222Amazon AGI223- **Goeric Huybrechts**224- **Srikanth Ronanki**225- **Sai Muralidhar Jayanthi**226- **Jack Fitzgerald**227- **Srinivasan Veeravanallur**