AmazonScience/document-haystack

Document Haystack Dataset

This repository contains the dataset for the paper “Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark”.

📑 Abstract Paper

The proliferation of multimodal Large Language Models has significantly advanced the ability to analyze and understand complex data inputs from different modalities. However, the processing of long documents remains under-explored, largely due to a lack of suitable benchmarks. To address this, we introduce Document Haystack, a comprehensive benchmark designed to evaluate the performance of Vision Language Models (VLMs) on long, visually complex documents. Document Haystack features documents ranging from 5 to 200 pages and strategically inserts pure text or multimodal text+image "needles" at various depths within the documents to challenge VLMs' retrieval capabilities. Comprising 400 document variants and a total of 8,250 questions, it is supported by an objective, automated evaluation framework. We detail the construction and characteristics of the Document Haystack dataset, present results from prominent VLMs and discuss potential research avenues in this area.

🗂️ Overview

Document Haystack is a comprehensive benchmark dataset designed to evaluate the long-context retrieval and multimodal document understanding capabilities of Vision Language Models (VLMs).

It expands on the Needle in a Haystack concept by embedding needles — short key-value statements in pure text or as multimodal text+image snippets — within real-world long documents (5–200 pages). These needles test whether models can locate specific information hidden deep inside long, complex documents with textual, visual or mixed content.

🎯 Benchmark Design

• —Key-value pairs: Each needle follows the pattern “The secret KEY is VALUE.” where VALUE appears as either text or an image. For example: “The secret sport is basketball.”. The keys span diverse categories including sports, animals, currencies, fruits, musical instruments, and more (see Table 3 in the paper for the complete category list).
• —Tasks: Each needle has an associated retrieval question: “What is the secret KEY in the document?”
• —Objective scoring: The VLM’s answer is checked for the correct VALUE (or acceptable aliases for text+image needles).

✅ Key Features

• —Document Lengths: 5, 10, 25, 50, 75, 100, 150, 200 pages
• —Total Documents: 400 document variants
• —Total Questions: 8,250 unique retrieval queries
• —Needle Types:
• —Text Needles: Pure text (e.g., “The secret sport is basketball.”)
• —Text+Image Needles: The value is shown as an image (e.g., “The secret sport is \<image of basketball\>.”)
• —Formats Provided:
• —Original PDF
• —Page-wise images (200 DPI)
• —Parsed plain text (for text needles only)

🔬 Use Cases

• —Stress-test multimodal VLMs for long-context understanding
• —Compare retrieval from parsed pdf text vs images vs original pdfs
• —Explore text vs image vs mixed retrieval challenges
• —Measure performance drop with increasing context length

📦 Document Haystack Format Variants

📊 Document Haystack Characteristics

🗄️ Dataset Structure

Below is an example of the dataset’s folder layout:

DocumentHaystack/
├── AIG/
│   ├── AIG_5Pages/
│   │   ├── AIG_5Pages_ImageNeedles.pdf
│   │   ├── AIG_5Pages_TextNeedles.pdf
│   │   ├── Images_TextImageNeedles/
│   │   │   ├── AIG_5Pages_ImageNeedles_page_1.jpg
│   │   │   ├── AIG_5Pages_ImageNeedles_page_2.jpg
│   │   │   ├── ...
│   │   ├── Images_TextNeedles/
│   │   │   ├── AIG_5Pages_TextNeedles_page_1.jpg
│   │   │   ├── AIG_5Pages_TextNeedles_page_2.jpg
│   │   │   ├── ...
│   │   ├── Text_TextNeedles/
│   │   │   ├── AIG_5Pages_TextNeedles_page_1.txt
│   │   │   ├── AIG_5Pages_TextNeedles_page_2.txt
│   │   │   ├── ...
│   │   ├── needles_info.csv
│   │   ├── needles.csv
│   │   ├── prompt_questions.txt
│   ├── AIG_10Pages/
│   ├── AIG_25Pages/
│   ├── AIG_50Pages/
│   ├── AIG_75Pages/
│   ├── AIG_100Pages/
│   ├── AIG_150Pages/
│   ├── AIG_200Pages/
│
│   needles.csv
│   prompt_questions.txt
│
├── AmericanAirlines/
├── APA/
├── BankOfMontreal/
...

📁 File and Folder Descriptions

Below is an explanation of the files inside the AIG_5Pages subfolder:

Within the main AIG folder, you'll find two key files: needles.csv, which lists all 25 needles that are utilized across the different AIG variants, and prompt_questions.txt, which contains the complete set of 25 prompts used throughout the AIG variants.

There are 25 top-level subfolders in total, each referring to a different document (e.g., AIG/, AmericanAirlines/, APA/, BankOfMontreal/), each structured the same way.

📌 needles_info.csv

Each document variant includes a needles_info.csv detailing every needle’s properties:

Example rows:

The secret office supply is a "pencil".,2,8,gray,white,0.339,0.931,times-bold,90
The secret sport is "basketball".,3,10,green,white,0.917,0.746,times-bolditalic,112

Placement:

• —Needles are randomly placed across equal, non-overlapping page ranges to ensure coverage throughout the document.
• —Same locations are reused for both text-only and text+image sets.

📏 Reference Evaluation

Use with the Document Haystack Benchmark Code for:

• —Fully automated inference & scoring pipelines
• —Heatmap generation for depth-based performance

📝 License

This project is licensed under the CC-BY-NC-4.0 License - see the GitHub LICENSE file for details.

📚 Citation

If you use this work in any way, please cite:

@article{huybrechts2025document,
  title={Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark},
  author={Huybrechts, Goeric and Ronanki, Srikanth and Jayanthi, Sai Muralidhar and Fitzgerald, Jack and Veeravanallur, Srinivasan},
  journal={arXiv preprint arXiv:2507.15882},
  year={2025}
}

👥 Authors

Amazon AGI

• —Goeric Huybrechts
• —Srikanth Ronanki
• —Sai Muralidhar Jayanthi
• —Jack Fitzgerald
• —Srinivasan Veeravanallur