datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
RealWorldQAmorpheus-real-world
Morpheus — Real-World Physics Videos
Real-world reference footage for Morpheus, a benchmark that tests whether
video generative models (Wan, CogVideo, LTX-Video, COSMOS-predict1/2,
Pyramid-Flow, Veo3, Kling-Turbo, ...) obey Newtonian mechanics. Object
trajectories are extracted via SAM2 tracking and tested against physical laws
(energy/momentum conservation, equations of motion) rather than pixel-matched
to a single "correct" video.
This repo contains the filmed real-world… See the full description on the dataset page: https://huggingface.co/datasets/physics-from-video/morpheus-real-world.RealworldQA
RealWorldQA
RealWorldQA is a benchmark designed for real-world understanding. The dataset consists of anonymized images taken from vehicles, in addition to other real-world images. We are excited to release RealWorldQA to the community, and we intend to expand it as our multimodal models improve.
The initial release of the RealWorldQA consists of over 700 images, with a question and easily verifiable answer for each image. See the announcement of Grok-1.5 Vision Preview.… See the full description on the dataset page: https://huggingface.co/datasets/xai-org/RealworldQA.realworldqa
Real World QA Dataset
This is a benchmark dataset released by xAI under CC-by-nd-4.0 license along with Grok-1.5 Vision Announcement.
This benchmark is designed to evaluate basic real-world spatial understanding capabilities of multimodal models.
While many of the examples in the current benchmark are relatively easy for humans, they often pose a challenge for frontier models.
This release of the RealWorldQA consists of 765 images, with a question and easily verifiable answer for… See the full description on the dataset page: https://huggingface.co/datasets/nirajandhakal/realworldqa.realworld_task820MME-RealWorld-Lite
2024.11.14 🌟 MME-RealWorld now has a lite version (50 samples per task, or all if fewer than 50) for inference acceleration, which is also supported by VLMEvalKit and Lmms-eval.
2024.09.03 🌟 MME-RealWorld is now supported in the VLMEvalKit and Lmms-eval repository, enabling one-click evaluation—give it a try!"
2024.08.20 🌟 We are very proud to launch MME-RealWorld, which contains 13K high-quality images, annotated by 32 volunteers, resulting in 29K question-answer pairs that cover 43… See the full description on the dataset page: https://huggingface.co/datasets/yifanzhang114/MME-RealWorld-Lite.realworldqa
RealWorldQA dataset
This is the benchmark dataset released by xAI along with the Grok-1.5 Vision announcement.
This benchmark is designed to evaluate basic real-world spatial understanding capabilities of multimodal models.
While many of the examples in the current benchmark are relatively easy for humans, they often pose a challenge for frontier models.
This release of the RealWorldQA consists of 765 images, with a question and easily verifiable answer for each image.
The… See the full description on the dataset page: https://huggingface.co/datasets/visheratin/realworldqa.mme_realworld_train_21690Real_World_Multi_Taskrealworld_replayed_task820MME-RealWorld-Literace-realworld
RACE — real-robot planner data
Two tasks on one 7-DoF arm, chosen to sit on different axes. Both give a high-level planner the
same contract: read the scene, decide the subtask the controller should execute next, and say
whether the accepted one has finished.
shell_game memory the answer is only in frames that have already scrolled past
solve_equation reasoning the answer is in view but has to be computed
Same rig for both: 7-DoF arm plus gripper, three stereo… See the full description on the dataset page: https://huggingface.co/datasets/ghkim-rlwrld/race-realworld.MMRC_Real_World_Conversation
MMRC - Multi-Modal Open-Ended Conversation Dataset
Overview:
MMRC is a benchmark dataset designed for evaluating Multi-Modal Large Language Models (MLLMs) in open-ended, multi-turn conversations. It provides diverse, real-world conversational data that integrates both textual and visual modalities, aiming to push the boundaries of MLLM performance in practical settings.
Dataset Details:
The MMRC dataset is composed of multi-turn conversations with integrated… See the full description on the dataset page: https://huggingface.co/datasets/WUUE/MMRC_Real_World_Conversation.real-world-samplesreal_world_sampleMultipanelVQA_real-worldReal-world data in MultipanelVQA
Paper: Muffin or Chihuahua? Challenging Large Vision-Language Models with Multipanel VQA (arXiv)
Website: https://sites.google.com/view/multipanelvqa/home
MultipanelVQA includes both real-world data and synthetic data.
mme_realworld_lite_1919_vstar_191_sum_2110realworld_replay_task820_firm_gentle_mixed_currentChartNet_RealWorldChart
2077AI ChartNet — Real-World Chart Caption Dataset
A collection of 30,000 real-world chart images paired with detailed natural-language captions,
intended for chart understanding and image-to-text research.
Dataset Structure
Column
Type
Description
image_name
string
Original filename stem
image
image
Chart image (PNG)
caption
string
Detailed descriptive caption
Usage
from datasets import load_dataset
ds =… See the full description on the dataset page: https://huggingface.co/datasets/2077AIDataFoundation/ChartNet_RealWorldChart.real_world_data_smallrealworld_replay_task820_firm_mixedRealWorldQAsketchrefiner-real-world-test-protocolFrontierBias-realworld
FrontierBias: real-world images
Real-photo image set used for the real-image transferability experiment in Fairness
Failure Modes of Multimodal LLMs. Each image pairs two real photographs side by side,
mirroring the two-person layout of the synthetic
MLL-Lab/FrontierBias images. Evaluating on
this set checks that conclusions drawn from the synthetic benchmark carry over to real
photos: in the paper, fairness scores on synthetic and real images are highly… See the full description on the dataset page: https://huggingface.co/datasets/MLL-Lab/FrontierBias-realworld.realworld_replayed_task820_firmrealworld-lerobot-2.1realworld_task820_currentrlt-realworld-ee-peginsertion-100-succrealworldqaDataset from here: https://x.ai/blog/grok-1.5v
Script used to push the dataset:
import json
from datasets import load_dataset
# load annotations
with open('realworldqa/annotations.json') as f:
data = json.load(f)
# create metadata for the dataset
metadata = [{"file_name": i["image"], "question": i["question"], "answer": i["answer"]} for i in data]
# write metadata to json lines file
with open('realworldqa/images/metadata.jsonl', 'w') as outfile:
for entry in metadata:… See the full description on the dataset page: https://huggingface.co/datasets/nielsr/realworldqa.realworld_replayed_task820_firm