datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
RealWorldQARealworldQA
RealWorldQA
RealWorldQA is a benchmark designed for real-world understanding. The dataset consists of anonymized images taken from vehicles, in addition to other real-world images. We are excited to release RealWorldQA to the community, and we intend to expand it as our multimodal models improve.
The initial release of the RealWorldQA consists of over 700 images, with a question and easily verifiable answer for each image. See the announcement of Grok-1.5 Vision Preview.… See the full description on the dataset page: https://huggingface.co/datasets/xai-org/RealworldQA.apex-r1-real-world-documents
Apex-R1 Real-World Benchmark Documents
This dataset stores real-world document/data assets collected for Apex-R1 synthetic long-horizon agentic RL workspace generation.
The files are intended as seed workspace materials, not as benchmark task labels. They can be injected into APEX-style filesystem/ or .apps_data/ environments to create more realistic and diverse professional-domain tasks.
Contents
benchmark_documents/
EnterpriseBench/ # CRM invoices… See the full description on the dataset page: https://huggingface.co/datasets/mtybilly/apex-r1-real-world-documents.MME-RealWorld-Lmms-evalreal-world-swe-problems
SYNTHETIC-1
This is a subset of the task data used to construct SYNTHETIC-1. You can find the full collection here
realworldqa
Real World QA Dataset
This is a benchmark dataset released by xAI under CC-by-nd-4.0 license along with Grok-1.5 Vision Announcement.
This benchmark is designed to evaluate basic real-world spatial understanding capabilities of multimodal models.
While many of the examples in the current benchmark are relatively easy for humans, they often pose a challenge for frontier models.
This release of the RealWorldQA consists of 765 images, with a question and easily verifiable answer for… See the full description on the dataset page: https://huggingface.co/datasets/nirajandhakal/realworldqa.Sekai2_Real_World
Sekai2 Real World
This repository releases the reproducible URL/timestamp metadata and paired
camera-pose/caption annotations for the perspective-video portion of
Sekai2. See the paper: Sekai2: From World Exploration to Interactive World Modeling.
Resources: 🌐 Project Page · 💻 GitHub · 📄 Paper
The perspective MP4 clips are not redistributed here. Each row in
sekai2_clips.csv provides the source URL and the exact half-open frame range
[start_frame, end_frame) in a canonical 30… See the full description on the dataset page: https://huggingface.co/datasets/Kangverse/Sekai2_Real_World.MME-RealWorld-CN-Lmms-evalMME-RealWorld-lite-lmms-eval
2024.11.14 🌟 MME-RealWorld now has a lite version (50 samples per task, or all if fewer than 50) for inference acceleration, which is also supported by VLMEvalKit and Lmms-eval.
2024.09.03 🌟 MME-RealWorld is now supported in the VLMEvalKit and Lmms-eval repository, enabling one-click evaluation—give it a try!"
2024.08.20 🌟 We are very proud to launch MME-RealWorld, which contains 13K high-quality images, annotated by 32 volunteers, resulting in 29K question-answer pairs that cover 43… See the full description on the dataset page: https://huggingface.co/datasets/yifanzhang114/MME-RealWorld-lite-lmms-eval.MME-RealWorld-Base64
MME-RealWorld Dataset
This dataset contains multiple JSON files split into chunks. It includes information such as questions, images encoded in base64, and other related metadata.
Usage
You can load the dataset using the datasets library:
from datasets import load_dataset
dataset = load_dataset('yifanzhang114/MME-RealWorld-Base64', data_dir='MME-RealWorld')
dataset = load_dataset('yifanzhang114/MME-RealWorld-Base64', data_dir='MME-RealWorld-CN')
## the image can be… See the full description on the dataset page: https://huggingface.co/datasets/yifanzhang114/MME-RealWorld-Base64.realworldqa
RealWorldQA dataset
This is the benchmark dataset released by xAI along with the Grok-1.5 Vision announcement.
This benchmark is designed to evaluate basic real-world spatial understanding capabilities of multimodal models.
While many of the examples in the current benchmark are relatively easy for humans, they often pose a challenge for frontier models.
This release of the RealWorldQA consists of 765 images, with a question and easily verifiable answer for each image.
The… See the full description on the dataset page: https://huggingface.co/datasets/visheratin/realworldqa.mme_realworld_train_21690MME-RealWorld-LiteMMRC_Real_World_Conversation
MMRC - Multi-Modal Open-Ended Conversation Dataset
Overview:
MMRC is a benchmark dataset designed for evaluating Multi-Modal Large Language Models (MLLMs) in open-ended, multi-turn conversations. It provides diverse, real-world conversational data that integrates both textual and visual modalities, aiming to push the boundaries of MLLM performance in practical settings.
Dataset Details:
The MMRC dataset is composed of multi-turn conversations with integrated… See the full description on the dataset page: https://huggingface.co/datasets/WUUE/MMRC_Real_World_Conversation.senseshift-rl-real-world-active-sensing
SenseShift-RL
SenseShift-RL is a replayable active-sensing world built from real,
quality-screened U.S. EPA ozone measurements. On each day an agent activates
at most two of eight monitoring sites, observes only those valid measurements,
and is rewarded for reconstructing the withheld network while paying an
explicit authored sensing cost.
This is an observation-control benchmark, not an air-quality control
simulator. Actions reveal recorded measurements; they do not alter… See the full description on the dataset page: https://huggingface.co/datasets/haidang2405/senseshift-rl-real-world-active-sensing.real_world_sampleMultipanelVQA_real-worldReal-world data in MultipanelVQA
Paper: Muffin or Chihuahua? Challenging Large Vision-Language Models with Multipanel VQA (arXiv)
Website: https://sites.google.com/view/multipanelvqa/home
MultipanelVQA includes both real-world data and synthetic data.
mme_realworld_lite_1919_vstar_191_sum_2110ChartNet_RealWorldChart
2077AI ChartNet — Real-World Chart Caption Dataset
A collection of 30,000 real-world chart images paired with detailed natural-language captions,
intended for chart understanding and image-to-text research.
Dataset Structure
Column
Type
Description
image_name
string
Original filename stem
image
image
Chart image (PNG)
caption
string
Detailed descriptive caption
Usage
from datasets import load_dataset
ds =… See the full description on the dataset page: https://huggingface.co/datasets/2077AIDataFoundation/ChartNet_RealWorldChart.RealWorldQAreal_world_data_annotated
Real-World Data — Camera-3 Annotations
This dataset provides perception annotations for 1,000 real robot demonstration
episodes across five tasks. It is the annotation companion to
mzxuan/real_world_data,
which contains the original recordings.
Download both datasets for visual robot-learning experiments. This repository
does not contain the original RGB, depth, robot state, or action commands. It
contains masks, object identities and roles, 2D/3D boxes, annotation-only HDF5… See the full description on the dataset page: https://huggingface.co/datasets/mzxuan/real_world_data_annotated.realworld_dataworld_model_real_rollout_genrlt-realworld-ee-peginsertion-100-succrealworldqaDataset from here: https://x.ai/blog/grok-1.5v
Script used to push the dataset:
import json
from datasets import load_dataset
# load annotations
with open('realworldqa/annotations.json') as f:
data = json.load(f)
# create metadata for the dataset
metadata = [{"file_name": i["image"], "question": i["question"], "answer": i["answer"]} for i in data]
# write metadata to json lines file
with open('realworldqa/images/metadata.jsonl', 'w') as outfile:
for entry in metadata:… See the full description on the dataset page: https://huggingface.co/datasets/nielsr/realworldqa.real-world-agent-benchmark
Real-World Agent Benchmark (RAB)
Paper: Orchestrator and Task-Framing Effects Dominate Fine-Tuning in Real-World Agent Evaluation of a Quantized 31B ModelAuthors: Kiko Cisneros, Claude Sonnet 4.6 · Utopia IA, May 2026Code: github.com/KikoCisBot/gemma4-31b-study
📄 See paper4_orchestrator_dominance.pdf in the Files tab.
TL;DR
Standard benchmarks (BFCL, HumanEval) do not predict real-world agent capability. A model scoring 95% BFCL scores 0/10 on a real autonomous task… See the full description on the dataset page: https://huggingface.co/datasets/KikoCis/real-world-agent-benchmark.privacy-preserving-real-world-human-motion-sample
Privacy-Preserving Real-World Human Motion Sample
A market-validation sample of anonymous 2D skeleton/pose observations derived from a real-world indoor CCTV stream.
Why this sample exists
We are validating demand for continuously collected, privacy-oriented real-world human-motion data before expanding to multi-camera releases.
Current public sample
750 public observations
derived pose/skeleton data
anonymous track identifiers
no raw RGB video
no… See the full description on the dataset page: https://huggingface.co/datasets/Ragab-Adel/privacy-preserving-real-world-human-motion-sample.a1_code_primeintellect_real_world_swe_eval_636d
mlfoundations-dev/a1_code_primeintellect_real_world_swe_eval_636d
Precomputed model outputs for evaluation.
Evaluation Results
Summary
Metric
AIME24
AMC23
MATH500
MMLUPro
JEEBench
GPQADiamond
LiveCodeBench
CodeElo
CodeForces
Accuracy
12.3
52.2
72.4
24.4
36.3
38.6
15.9
7.8
8.4
AIME24
Average Accuracy: 12.33% ± 1.42%
Number of Runs: 10
Run
Accuracy
Questions Solved
Total Questions
1
23.33%
7
30
2
10.00%
3
30
3
10.00%
3… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations-dev/a1_code_primeintellect_real_world_swe_eval_636d.realWorldQARealWorldQuestioning
RealWorldQuestioning Benchmark
RealWorldQuestioning is a benchmark dataset of 400+ real-world user questions collected from public discussion forums (e.g., Reddit, Quora), designed to support evaluation of gender bias and information disparity in Large Language Models (LLMs). The dataset spans four business-relevant domains: Education, Jobs, Investment, and Health.
Each question is annotated with:
User persona (Male or Female framing)
Source forum
Domain category
Four anonymized… See the full description on the dataset page: https://huggingface.co/datasets/SonalPrabhune/RealWorldQuestioning.