Doctor5benchmark/DocToR-5

DocToR 5 documents Tracing of Robustness

Dataset Summary

We introduce the DOCTOR 5 benchmark to systematically test the visual and logical robustness of multimodal models. The dataset features approximately 13k+ document images and over 65000 visual question answering pairs. It covers 12 diverse domains including scientific papers financial reports and business presentations. This resource provides a standardized testbed for evaluating models against generation aware perturbations.

Taxonomy of Robustness

The benchmark categorizes cognitive and generative perturbations into five exclusive layers across 17 subcategories.

• —L1 Perception focuses on font rendering noise character ghosting and physical artifacts like handwritten pen strokes and watermarks.
• —L2 Language evaluates factual hallucinations semantic drift and key information removal.
• —L3 Alignment targets structural layout mismatches and severe image text contradictions.
• —L4 Reasoning injects multi hop logic failures fact inversions and numerical absurdities.
• —L5 Agent traps models using double bind instructions and authority role mixing prompt injections.

Data Generation Pipeline

We employ a hybrid data generation pipeline. The pristine document pool is filtered dynamically using language models to ensure sufficient structural complexity. We combine programmatic pixel rendering for low level visual noise with automated language model injections for high level semantic errors. All textual interventions are strictly constrained to the original domain context. We do not apply uniform image resizing keeping the natural diversity of the original document resolutions.

Evaluation Metrics

We measure model performance using a set of standardized evaluation tools.

• —Textual Tasks We utilize Exact Match and F1 scores for general question answering.
• —Table Extraction We calculate tree structural similarity using the TEDS metric.
• —Formula Extraction We evaluate the accuracy of generated formulas using the Edit Distance metric. We apply a standardized preprocessing and normalization pipeline to all predicted LaTeX code strings before calculating the edit distance. This standardization removes format variations and ensures objective evaluation.

Limitations and Broader Impacts

The synthetic noise generated in this benchmark may not perfectly capture all entirely random human errors occurring in the wild. The prompt injection techniques demonstrated in the Agent layer carry potential dual use risks. We release this dataset to accelerate community development of defensive mechanisms against adversarial document generation and to promote the creation of safer multimodal AI systems.

Usage

You can load the dataset directly using the Hugging Face datasets library.

from datasets import load_dataset

dataset = load_dataset("Doctor5benchmark/DocToR-5")
print(dataset)