mmeb
Datasets
All datasets matching “mmeb”MMEB_Test_InstructVideo-MME-v2
🔥 News
2026.06.11 Videos re-encoded to H265, maintaining consistent evaluation scores. Fixed 2 incorrect MP4s & 3 mismatched URLs. Original data preserved in the original branch.
2026.05.22 Task types are now available for Q1-Q3 in coherence (logic) groups.
🤗 About This Repo
This repository contains annotation data for "Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding". It mainly consists of three… See the full description on the dataset page: https://huggingface.co/datasets/MME-Benchmarks/Video-MME-v2.MMEB-eval
Massive Multimodal Embedding Benchmark
We compile a large set of evaluation tasks to understand the capabilities of multimodal embedding models. This benchmark covers 4 meta tasks and 36 datasets meticulously selected for evaluation.
The dataset is published in our paper VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks.
Dataset Usage
For each dataset, we have 1000 examples for evaluation. Each example contains a query and a set of… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/MMEB-eval.MMEB-train
Massive Multimodal Embedding Benchmark
The training data split used for training VLM2Vec models in the paper VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025).
MMEB benchmark covers 4 meta tasks and 36 datasets meticulously selected for evaluating capabilities of multimodal embedding models.
During training, we utilize 20 out of the 36 datasets.
For evaluation, we assess performance on the 20 in-domain (IND) datasets and the remaining 16… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/MMEB-train.MMEB-V3
MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models
🌐 Website |
GitHub |
🏆 Leaderboard |
📖 MMEB-V3 Paper |
📖 MMEB-V2 Paper |
📖 MMEB-V1 Paper |
🤗 Models
Introduction
MMEB-V3 is a comprehensive benchmark for evaluating omni-modality embedding models across text, image, video, audio, visual-document, and agent-centric retrieval scenarios.
Building upon MMEB-V1 and MMEB-V2, MMEB-V3 adds 111 new tasks, resulting in 190 evaluation tasks in… See the full description on the dataset page: https://huggingface.co/datasets/VLM2Vec/MMEB-V3.MMEB-V2
MMEB-V2 (Massive Multimodal Embedding Benchmark)
Website |Github | 🏆Leaderboard | 📖MMEB-V2/VLM2Vec-V2 Paper | | 📖MMEB-V1/VLM2Vec-V1 Paper |
Introduction
Building upon on our original MMEB, MMEB-V2 expands the evaluation scope to include five new tasks: four video-based tasks — Video Retrieval, Moment Retrieval, Video Classification, and Video Question Answering — and one task focused on visual documents, Visual Document Retrieval. This comprehensive suite enables… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/MMEB-V2.