datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Molmo2-SynMultiImageQA
Molmo2-SynMultiImageQA
Molmo2-SynMultiImageQA is a collection of synthetic multi-image question-answer pairs about various kinds of text-rich images, including charts, tables, documents, diagrams, etc.
The synthetic data is generated by extending the CoSyn framework into multi-image settings,
with Claude-sonnet-4-5 as the coding LLM to generate code that can be executed to render an image.
Then, we use GPT-5 to generate question-answer pairs with code (without using the rendered… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-SynMultiImageQA.Molmo2-ER-VST-P
Molmo2-ER · rayruiyang/vst_500k
500K perception QA over images normalized to a uniform virtual camera (single + multi-view).
This is a re-hosted, loader-ready subset of the upstream dataset, used to train allenai/Molmo2-ER-4B. Files mirror the upstream layout; nothing in the data has been modified.
⚠️ This dataset is released for non-commercial research use only, inheriting the most-restrictive license among its upstream sources. See the upstream repository for details.… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-ER-VST-P.Molmo2-ER-SenseNova-SI
Molmo2-ER · sensenova/SenseNova-SI-800K
832K multi-image spatial-intelligence conversations grounded in 3D scene annotations.
This is a re-hosted, loader-ready subset of the upstream dataset, used to train allenai/Molmo2-ER-4B. Files mirror the upstream layout; nothing in the data has been modified.
Upstream source
Original dataset: sensenova/SenseNova-SI-800K
Paper: Scaling Spatial Intelligence with Multimodal Foundation Models (arXiv:2511.13719)
License: apache-2.0… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-ER-SenseNova-SI.molmo2-tulu4-classifiedMolmo2-ER-RoboPoint
Molmo2-ER · wentao-yuan/robopoint-data
1.43M robotics affordance instruction-tuning examples (pointing + detection + VQA).
This is a re-hosted, loader-ready subset of the upstream dataset, used to train allenai/Molmo2-ER-4B. Files mirror the upstream layout; nothing in the data has been modified.
Upstream source
Original dataset: wentao-yuan/robopoint-data
Paper: RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics (arXiv:2406.10721)
License:… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-ER-RoboPoint.Molmo2-ER-VSI-590K
Molmo2-ER · nyu-visionx/VSI-590K
590K spatial QA samples (image+video) propagated from 3D ground truth and CV pseudo-labels.
This is a re-hosted, loader-ready subset of the upstream dataset, used to train allenai/Molmo2-ER-4B. Files mirror the upstream layout; nothing in the data has been modified.
Upstream source
Original dataset: nyu-visionx/VSI-590K
Paper: Cambrian-S: Towards Spatial Supersensing in Video (arXiv:2511.04670)
License: apache-2.0 (inherits from upstream)… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-ER-VSI-590K.gdufs-molmo2-sftMolmo2-VideoPoint
Molmo2-VideoPoint
Molmo2-VideoPoint is a dataset of video pointing data collected from human annotators.
It can be used to fine-tune vision-language models for video grounding by pointing.
Molmo2-VideoPoint is part of the Molmo2 dataset collection and was used to train the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Usage
from datasets import load_dataset
# Load entire dataset
ds = load_dataset("allenai/Molmo2-VideoPoint", split="train")
#… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-VideoPoint.Molmo2-ER-RoboVQA
Molmo2-ER · Google DeepMind RoboVQA
Human-annotated long-horizon robotics video QA across three embodiments.
This is a re-hosted, loader-ready subset of the upstream dataset, used to train allenai/Molmo2-ER-4B. Files mirror the upstream layout; nothing in the data has been modified.
Upstream source
Original dataset: Google DeepMind RoboVQA
Paper: RoboVQA: Multimodal Long-Horizon Reasoning for Robotics (arXiv:2311.00899)
License: cc-by-4.0 (inherits from upstream)
If you… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-ER-RoboVQA.Molmo2-Cap
Molmo2-Cap
Molmo2-Cap is a dataset of very long (average 900 words), detailed video captions.
It can be used to pre-train and fine-tune vision-language models.
Molmo2-Cap is part of the Molmo2 dataset collection and was used to train the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Data Format
Videos are stored as Youtube video ID that will need to be downloaded separately.
Clip Annotations
clip_timestamps: a list of video… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-Cap.Molmo2-ER-RefSpatial
Molmo2-ER · JingkunAn/RefSpatial
2.5M spatial-referring corpus (web + indoor + simulated) covering 31 spatial relations.
This is a re-hosted, loader-ready subset of the upstream dataset, used to train allenai/Molmo2-ER-4B. Files mirror the upstream layout; nothing in the data has been modified.
Upstream source
Original dataset: JingkunAn/RefSpatial
Paper: RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics (arXiv:2506.04308)… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-ER-RefSpatial.Molmo2-MultiImagePoint
Molmo2 Multi-Image Pointing
This dataset contains multi-image pointing/counting metadata.
This dataset is generated by extending PixMo-Points using a semantic grouping algorithm designed to maximize coverage.
Molmo2-MultiImagePoint is a part of the Molmo2 dataset collection and was used to
provide the multi-image pointing capabilities of the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Columns
image_urls: list of image URLs (original source… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-MultiImagePoint.Molmo2-VideoCapQA
Molmo2-VideoCapQA
Molmo2-VideoCapQA is a dataset of multiple-choice video QA that only requires visual content.
It can be used to fine-tune vision-language models.
Molmo2-VideoCapQA is part of the Molmo2 dataset collection and was used to train the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Data Format
Videos are stored as Youtube video ID that will need to be downloaded separately. We provide a mapping from their IDs to the original YouTube… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-VideoCapQA.Molmo2-ER-CLEVR
Molmo2-ER · CLEVR v1.0 (Stanford)
Compositional VQA over rendered 3D primitives (train split only).
This is a re-hosted, loader-ready subset of the upstream dataset, used to train allenai/Molmo2-ER-4B. Files mirror the upstream layout; nothing in the data has been modified.
Upstream source
Original dataset: CLEVR v1.0 (Stanford)
Paper: CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (arXiv:1612.06890)
License: cc-by-4.0 (inherits… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-ER-CLEVR.molmo2-moments
Molmo-2 Moments (M2M)
Long-video QA dataset where every question is anchored to a specific
[start, end] clip interval in seconds. Released alongside the ToolMerge
paper, "Decomposing Queries into Tool Calls for Long-Video Keyframe
Retrieval".
⚠️ Source videos & ownership
The videos/*.mp4 files in this repository were collected from YouTube.
We do not own these videos and claim no copyright over them. All rights to
the video content remain with the original… See the full description on the dataset page: https://huggingface.co/datasets/michalsr/molmo2-moments.Molmo2-VideoTrack
Molmo2-VideoTrack
Molmo2-VideoTrack is a dataset of video point tracking annotations collected from human annotators across 16 video datasets.
It can be used to fine-tune vision-language models for video object tracking via point trajectories.
Molmo2-VideoTrack is part of the Molmo2 dataset collection and was used to train the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Usage
from datasets import load_dataset
# Load entire dataset
ds =… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-VideoTrack.molmo2-sft-valmolmo2-single-object-track
Single Object Tracking Dataset with Point as Input
Video object tracking annotations for single object tracking, where the input is a single point.
This dataset is derived from the original datasets by converting bbox annotations to point tracks.
Each config corresponds to a different source dataset. All share the same task (single_point_track) and schema.
Configs
Config
Description
Splits
all (default)
All source datasets combined
train
lvosv1
lvosv1 single… See the full description on the dataset page: https://huggingface.co/datasets/allenai/molmo2-single-object-track.Molmo2-TVQAMolmo2-AskModelAnything
Molmo2-AskModelAnything
Molmo2-AskModelAnything is a dataset of open-ended video QA generated by human annotators.
It can be used to fine-tune vision-language models.
Molmo2-AskModelAnything is part of the Molmo2 dataset collection and was used to train the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Data Format
Videos are stored as Youtube video ID that will need to be downloaded separately. We provide a mapping from their IDs to the… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-AskModelAnything.Molmo2-ER-SIMS-VSI
Molmo2-ER · ellisbrown/SIMS-VSI
203K simulated video spatial QA pairs from AI2-THOR/ProcTHOR agent trajectories.
This is a re-hosted, loader-ready subset of the upstream dataset, used to train allenai/Molmo2-ER-4B. Files mirror the upstream layout; nothing in the data has been modified.
Upstream source
Original dataset: ellisbrown/SIMS-VSI
Paper: SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding (arXiv:2511.04668)
License: apache-2.0 (inherits from… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-ER-SIMS-VSI.Molmo2-VideoSubtitleQA
Molmo2-VideoSubtitleQA
Molmo2-VideoSubtitleQA is a dataset of multiple-choice video QA that requires both visual content and audio transcript.
It can be used to fine-tune vision-language models.
Molmo2-VideoSubtitleQA is part of the Molmo2 dataset collection and was used to train the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Data Format
Videos are stored as Youtube video ID that will need to be downloaded separately. We provide a mapping… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-VideoSubtitleQA.Molmo2-VideoTrackEval
Molmo2-VideoTrackEval
Molmo2-VideoTrackEval is an evaluation benchmark for video point tracking, containing human-annotated ground truth expressions. It includes segmentation masks for evaluating whether predicted points fall within the correct object regions. Currently, there are five categories for evaluation:
animal
dance
sports
person
misc
This benchmark is part of the Molmo2 dataset collection and is used to evaluate the Molmo2 family of models on video object tracking via… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-VideoTrackEval.Molmo2-MultiImageQA
Molmo2-MultiImageQA
Molmo2-MultiImageQA is a multi-image extension of PixMo-AskModelAnything, an instruction-tuning dataset for vision-language models. It contains human-authored question-answer pairs over multiple images with long-form answers.
Molmo2-MultiImageQA is part of the Molmo2 dataset collection and was used to train the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Loading
import datasets
train_dataset =… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-MultiImageQA.Molmo2-VideoPointEval
Molmo2-VideoPointEval
Molmo2-VideoPointEval is a validation set of Molmo2-VideoPoint video pointing data collected from human annotators.
It can be used to evaluate vision-language models' video pointing capability.
Molmo2-VideoPointEval is part of the Molmo2 dataset collection and was used to evaluate the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Data Format
video_source: There are three video sources: youtube, generated and MammalNet.… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-VideoPointEval.molmo2-text-vqaMolmo2-SynMultiImageQA-tinymolmo2-vicas
vicas Tracking Dataset
Video object tracking annotations for vicas.
Configs
Config
Description
Splits
track (default)
Track points across all frames
train
ground
First and last appearance of points
train
single_point_track
Given starting point, track to end
train
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-vicas", split="train")
# Specific config
ds = load_dataset("allenai/molmo2-vicas"… See the full description on the dataset page: https://huggingface.co/datasets/allenai/molmo2-vicas.molmo2-track-instructionmolmo2-yt-vis
yt-vis Tracking Dataset
Video object tracking annotations for yt-vis.
Configs
Config
Description
Splits
track (default)
Track points across all frames
train
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-yt-vis", split="train")
# Specific config
ds = load_dataset("allenai/molmo2-yt-vis", "ground", split="train")