datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
MMEB-eval
Massive Multimodal Embedding Benchmark
We compile a large set of evaluation tasks to understand the capabilities of multimodal embedding models. This benchmark covers 4 meta tasks and 36 datasets meticulously selected for evaluation.
The dataset is published in our paper VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks.
Dataset Usage
For each dataset, we have 1000 examples for evaluation. Each example contains a query and a set of… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/MMEB-eval.MMEB-train
Massive Multimodal Embedding Benchmark
The training data split used for training VLM2Vec models in the paper VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025).
MMEB benchmark covers 4 meta tasks and 36 datasets meticulously selected for evaluating capabilities of multimodal embedding models.
During training, we utilize 20 out of the 36 datasets.
For evaluation, we assess performance on the 20 in-domain (IND) datasets and the remaining 16… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/MMEB-train.MMEB-V3
MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models
🌐 Website |
GitHub |
🏆 Leaderboard |
📖 MMEB-V3 Paper |
📖 MMEB-V2 Paper |
📖 MMEB-V1 Paper |
🤗 Models
Introduction
MMEB-V3 is a comprehensive benchmark for evaluating omni-modality embedding models across text, image, video, audio, visual-document, and agent-centric retrieval scenarios.
Building upon MMEB-V1 and MMEB-V2, MMEB-V3 adds 111 new tasks, resulting in 190 evaluation tasks in… See the full description on the dataset page: https://huggingface.co/datasets/VLM2Vec/MMEB-V3.MMEB-train-DocVQA-images
MMEB-train — DocVQA / Train images
Backup copy of the DocVQA Train image split used in MMEB-train (VLM2Vec) training.
Original folder structure is preserved: files live under DocVQA/Train/.
Files: 78,926 JPG images
Size: ~15 GB
MMEB_train_with_imageMME-Benchmark-pt
Avaliação - MME-Perception
Estrutura do Diretório
main
├── MME_Benchmark
│ ├── artwork
│ │ ├── images
│ │ │ ├── 1.jpg
│ │ │ ├── 2.jpg
│ │ │ ├── ...
│ │ ├── question_answers_YN
│ │ │ ├── 1.txt
│ │ │ ├── 2.txt
│ │ │ ├── ...
│ ├── celebrity
│ ├── code_reasoning
│ ├── ...
├── calculation.py
├── translate_MME
Estrutura dos Arquivos TXT
Cada arquivo num.txt contém as perguntas correspondentes à imagem num.jpg.… See the full description on the dataset page: https://huggingface.co/datasets/LucasLima/MME-Benchmark-pt.TIGER-Lab_MMEB-train
MMEB Training Dataset (Lance Format)
This is a Lance-format version of the TIGER-Lab/MMEB-train dataset, optimized for efficient storage and fast random access.
The original dataset is used for training VLM2Vec models in the paper VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025).
Directory Structure
TIGER-Lab_MMEB-train/
└── data/
├── A-OKVQA/
│ ├── train.lance
│ ├── original.lance
│ └── diverse.lance… See the full description on the dataset page: https://huggingface.co/datasets/laughatwill/TIGER-Lab_MMEB-train.MMEB-eval-ChartQA-beir-v3MMEB-eval-ChartQA-beirMMEB-eval-DocVQA-beirMMEB-eval-Wiki-SS-NQ-beir-v3MMEB-eval-VisDial-beirMMEB-eval-OVEN-beir-v3MMEB-eval-A-OKVQA-beirMMEB-eval-MSCOCO_i2t-beirMMEB-eval-ObjectNet-beirMMEB-eval-GQA-beir-v2MMEB-eval-OK-VQA-beirMMEB-eval-ImageNet-R-beir-v3MMEB-eval-FashionIQ-beir-v3MMEB-eval-RefCOCO-Matching-beir-v2MMEB-eval-GQA-beir-v3MMEB-train-subsampledMMEB-eval-ImageNet-A-beirMMEB-eval-TextVQA-beir-v3MMEB-eval-WebQA-beirMMEB-eval-ScienceQA-beirMMEB-eval-TextVQA-beirMMEB-eval-WebQA-beir-v2MMEB-eval-VisDial-beir-v3