Datasetpublic

TIGER-Lab/Mantis-Eval

Overview This is a newly curated dataset to evaluate multimodal language models' capability to reason over multiple images. More details are shown in https://tiger-ai-lab.github.io/Mantis/. Statistics This evaluation dataset contains 217 human-annotated challenging multi-image reasoning problems. Leaderboard We list the current results as follows: Models Size Mantis-Eval LLaVA OneVision 72B 77.60 LLaVA OneVision 7B 64.20 GPT-4V -… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/Mantis-Eval.

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face

6likes1kdownloads

Dataset Card

Overview

This is a newly curated dataset to evaluate multimodal language models' capability to reason over multiple images. More details are shown in https://tiger-ai-lab.github.io/Mantis/.

Statistics

This evaluation dataset contains 217 human-annotated challenging multi-image reasoning problems.

Leaderboard

We list the current results as follows:

Models	Size	Mantis-Eval
LLaVA OneVision	72B	77.60
LLaVA OneVision	7B	64.20
GPT-4V	-	62.67
Mantis-SigLIP	8B	59.45
Mantis-Idefics2	8B	57.14
Mantis-CLIP	8B	55.76
VILA	8B	51.15
BLIP-2	13B	49.77
Idefics2	8B	48.85
InstructBLIP	13B	45.62
LLaVA-V1.6	7B	45.62
CogVLM	17B	45.16
LLaVA OneVision	0.5B	39.60
Qwen-VL-Chat	7B	39.17
Emu2-Chat	37B	37.79
VideoLLaVA	7B	35.04
Mantis-Flamingo	9B	32.72
LLaVA-v1.5	7B	31.34
Kosmos2	1.6B	30.41
Idefics1	9B	28.11
Fuyu	8B	27.19
OpenFlamingo	9B	12.44
Otter-Image	9B	14.29

Citation

If you are using this dataset, please cite our work with

@article{Jiang2024MANTISIM,
  title={MANTIS: Interleaved Multi-Image Instruction Tuning},
  author={Dongfu Jiang and Xuan He and Huaye Zeng and Cong Wei and Max W.F. Ku and Qian Liu and Wenhu Chen},
  journal={Transactions on Machine Learning Research},
  year={2024},
  volume={2024},
  url={https://openreview.net/forum?id=skLtdUVaJa}
}