mmv
Datasets
All datasets matching “mmv”MMVP
MMVP Benchmark Datacard
Basic Information
Title: MMVP Benchmark
Description: The MMVP (Multimodal Visual Patterns) Benchmark focuses on identifying “CLIP-blind pairs” – images that are perceived as similar by CLIP despite having clear visual differences. MMVP benchmarks the performance of state-of-the-art systems, including GPT-4V, across nine basic visual patterns. It highlights the challenges these systems face in answering straightforward questions, often leading to… See the full description on the dataset page: https://huggingface.co/datasets/MMVP/MMVP.MMVU
MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
🌐 Homepage •
🥇 Leaderboard •
📖 Paper •
🤗 Data
📰 News
2025-01-21: We are excited to release the MMVU paper, dataset, and evaluation code!
👋 Overview
Why MMVU Benchmark?
Despite the rapid progress of foundation models in both text-based and image-based expert reasoning, there is a clear gap in evaluating these models’ capabilities in specialized-domain video understanding.… See the full description on the dataset page: https://huggingface.co/datasets/yale-nlp/MMVU.MMVP
MMVP (Multimodal Visual Patterns) Benchmark
This is a corrected version of the MMVP benchmark, re-hosted by lmms-lab-eval for use with lmms-eval.
Why this copy?
The original MMVP/MMVP dataset was uploaded in imagefolder format, which only exposes the image column. The text annotations (Question, Options, Correct Answer, Index) from the accompanying Questions.csv were not loaded into the dataset, making it unusable for evaluation.
This version reconstructs the complete… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab-eval/MMVP.MMVet
Large-scale Multi-modality Models Evaluation Suite
Accelerating the development of large-scale multi-modality models (LMMs) with lmms-eval
🏠 Homepage | 📚 Documentation | 🤗 Huggingface Datasets
This Dataset
This is a formatted version of MM-Vet. It is used in our lmms-eval pipeline to allow for one-click evaluations of large multi-modality models.
@misc{yu2023mmvet,
title={MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities}… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab-encoder/MMVet.mm-vetPaper: https://arxiv.org/abs/2308.02490
MMVP_VLM
MMVP-VLM Benchmark Datacard
Basic Information
Title: MMVP-VLM Benchmark
Description: The MMVP-VLM (Multimodal Visual Patterns - Visual Language Models) Benchmark is designed to systematically evaluate the performance of recent CLIP-based models in understanding and processing visual patterns. It distills a subset of questions from the original MMVP benchmark into simpler language descriptions, categorizing them into distinct visual patterns. Each visual pattern is… See the full description on the dataset page: https://huggingface.co/datasets/MMVP/MMVP_VLM.