uclanlp/MRAG-Bench
MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models 🌐 Homepage | 📖 Paper | 💻 Evaluation Intro MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios, providing a robust and systematic evaluation of Large Vision Language Model (LVLM)’s vision-centric multimodal retrieval-augmented generation (RAG) abilities. Results Evaluated upon 10 open-source and 4… See the full description on the dataset page: https://huggingface.co/datasets/uclanlp/MRAG-Bench.
MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models
**🌐 Homepage** | **📖 Paper** | **💻 Evaluation**
Intro
MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios, providing a robust and systematic evaluation of Large Vision Language Model (LVLM)’s vision-centric multimodal retrieval-augmented generation (RAG) abilities.
<img src="https://gordonhu608.github.io/images/mragbench_teaser.png" width="1000" />
Results
Evaluated upon 10 open-source and 4 proprietary LVLMs, our results show that all LVLMs exhibit greater improvements when augmented with images compared to textual knowledge. Notably, the top-performing model, GPT-4o, faces challenges in effectively leveraging retrieved knowledge, achieving only a 5.82% improvement with ground-truth information, in contrast to a 33.16% improvement observed in human participants. These findings highlight the importance of MRAG-Bench in encouraging the community to enhance LVLMs' ability to utilize retrieved visual knowledge more effectively.
<img src="https://gordonhu608.github.io/images/mragbench_qual.png" width="800" />
Load Dataset
The data/ directory contains the full dataset annotations and images pre-loaded for processing with HF Datasets. It can be loaded as follows:
from datasets import load_dataset
mrag_bench = load_dataset("uclanlp/MRAG-Bench", split="test")Dataset Description
The dataset contains the following fields:
<br>
We release the image corpus here for retrieval.
<br>
Contact
- Wenbo Hu: whu@cs.ucla.edu
Citation
@article{hu2024mragbench,
title={MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models},
author={Hu, Wenbo and Gu, Jia-Chen and Dou, Zi-Yi and Fayyaz, Mohsen and Lu, Pan and Chang, Kai-Wei and Peng, Nanyun},
journal={arXiv preprint arXiv:2410.08182},
year={2024}
}