Datasetpublic

JinyuLiu/MUMU-Eval-6000

MUMU Eval 6000 This repository contains the 6,000-image source-data evaluation set used for the Florence-2 and LFM2.5-VL-450M baselines in the MUMU evaluation repository. It is an independently prepared research split, not an official MUMU Challenge release. Splits Split Images Ground truth in manifest validation 1,000 Yes test 5,000 Yes The split contains 2,001 Task A samples, 2,000 Task B samples, and 1,999 Task C samples. All 6,000 image… See the full description on the dataset page: https://huggingface.co/datasets/JinyuLiu/MUMU-Eval-6000.

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes56downloads

Dataset Card

MUMU Eval 6000

This repository contains the 6,000-image source-data evaluation set used for the Florence-2 and LFM2.5-VL-450M baselines in the MUMU evaluation repository. It is an independently prepared research split, not an official MUMU Challenge release.

Splits

Split	Images	Ground truth in manifest
`validation`	1,000	Yes
`test`	5,000	Yes

The split contains 2,001 Task A samples, 2,000 Task B samples, and 1,999 Task C samples. All 6,000 image SHA-256 values are unique. Sampling seed: 20260817.

Layout

text

images/valid/                 # 1,000 validation images
images/public_test/           # 5,000 public-test images
manifests/valid.jsonl         # metadata and validation ground truth
manifests/public_test.jsonl   # metadata and public-test ground truth
summary.json                  # counts and integrity summary

Each manifest row records the stable ID, task, source dataset, relative image path, dimensions, format, SHA-256, source ID, source split, and prompt. The validation and public-test manifests both contain source-data ground truth for fully reproducible local evaluation.

Composition

Task	Source dataset	Validation	Test
A	KonIQ-10k	67	334
A	SPAQ	67	334
A	LIVE Challenge	67	333
A	Places365	67	333
A	NUS-WIDE	66	333
B	Objects365	111	556
B	LVIS	111	556
B	Visual Genome	111	555
C	COCO Captions	111	556
C	Flickr30k	111	555
C	Conceptual Captions	111	555

Loading

python

from datasets import load_dataset

dataset = load_dataset("JinyuLiu/MUMU-Eval-6000")
print(dataset["validation"][0])

To download files through the Hugging Face mirror:

bash

HF_ENDPOINT=https://hf-mirror.com \
  hf download JinyuLiu/MUMU-Eval-6000 \
  --repo-type dataset --local-dir ./mumu_eval_v1

Licensing and attribution

This is a multi-source evaluation collection. Copyright and licenses remain with each original dataset and image owner; no new license is asserted over third-party content. Users must follow the terms of KonIQ-10k, SPAQ, LIVE Challenge, Places365, NUS-WIDE, Objects365, LVIS, Visual Genome, COCO, Flickr30k, and Conceptual Captions. source_id and dataset are retained for traceability. Use is intended for research evaluation.

Important limitation

Scores produced from this split are source-data proxy metrics, not official MUMU Challenge scores. The challenge taxonomy, novel-category split, and efficiency coefficients were not available when this split was prepared.