datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Molmo2-VideoPoint
Molmo2-VideoPoint
Molmo2-VideoPoint is a dataset of video pointing data collected from human annotators.
It can be used to fine-tune vision-language models for video grounding by pointing.
Molmo2-VideoPoint is part of the Molmo2 dataset collection and was used to train the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Usage
from datasets import load_dataset
# Load entire dataset
ds = load_dataset("allenai/Molmo2-VideoPoint", split="train")
#… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-VideoPoint.molmo2-moments
Molmo-2 Moments (M2M)
Long-video QA dataset where every question is anchored to a specific
[start, end] clip interval in seconds. Released alongside the ToolMerge
paper, "Decomposing Queries into Tool Calls for Long-Video Keyframe
Retrieval".
⚠️ Source videos & ownership
The videos/*.mp4 files in this repository were collected from YouTube.
We do not own these videos and claim no copyright over them. All rights to
the video content remain with the original… See the full description on the dataset page: https://huggingface.co/datasets/michalsr/molmo2-moments.Molmo2-VideoTrack
Molmo2-VideoTrack
Molmo2-VideoTrack is a dataset of video point tracking annotations collected from human annotators across 16 video datasets.
It can be used to fine-tune vision-language models for video object tracking via point trajectories.
Molmo2-VideoTrack is part of the Molmo2 dataset collection and was used to train the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Usage
from datasets import load_dataset
# Load entire dataset
ds =… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-VideoTrack.Molmo2-TVQAmolmo2-single-object-track
Single Object Tracking Dataset with Point as Input
Video object tracking annotations for single object tracking, where the input is a single point.
This dataset is derived from the original datasets by converting bbox annotations to point tracks.
Each config corresponds to a different source dataset. All share the same task (single_point_track) and schema.
Configs
Config
Description
Splits
all (default)
All source datasets combined
train
lvosv1
lvosv1 single… See the full description on the dataset page: https://huggingface.co/datasets/allenai/molmo2-single-object-track.Molmo2-VideoTrackEval
Molmo2-VideoTrackEval
Molmo2-VideoTrackEval is an evaluation benchmark for video point tracking, containing human-annotated ground truth expressions. It includes segmentation masks for evaluating whether predicted points fall within the correct object regions. Currently, there are five categories for evaluation:
animal
dance
sports
person
misc
This benchmark is part of the Molmo2 dataset collection and is used to evaluate the Molmo2 family of models on video object tracking via… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-VideoTrackEval.Molmo2-VideoPointEval
Molmo2-VideoPointEval
Molmo2-VideoPointEval is a validation set of Molmo2-VideoPoint video pointing data collected from human annotators.
It can be used to evaluate vision-language models' video pointing capability.
Molmo2-VideoPointEval is part of the Molmo2 dataset collection and was used to evaluate the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Data Format
video_source: There are three video sources: youtube, generated and MammalNet.… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-VideoPointEval.molmo2-mevis
mevis Tracking Dataset
Video object tracking annotations for mevis.
Configs
Config
Description
Splits
track (default)
Track points across all frames
train, valid_u
ground
First and last appearance of points
train
single_point_track
Given starting point, track to end
train
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-mevis", split="train")
# Specific config
ds =… See the full description on the dataset page: https://huggingface.co/datasets/allenai/molmo2-mevis.molmo2-vicas
vicas Tracking Dataset
Video object tracking annotations for vicas.
Configs
Config
Description
Splits
track (default)
Track points across all frames
train
ground
First and last appearance of points
train
single_point_track
Given starting point, track to end
train
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-vicas", split="train")
# Specific config
ds = load_dataset("allenai/molmo2-vicas"… See the full description on the dataset page: https://huggingface.co/datasets/allenai/molmo2-vicas.molmo2-track-instructionmolmo2-yt-vis
yt-vis Tracking Dataset
Video object tracking annotations for yt-vis.
Configs
Config
Description
Splits
track (default)
Track points across all frames
train
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-yt-vis", split="train")
# Specific config
ds = load_dataset("allenai/molmo2-yt-vis", "ground", split="train")
Molmo2-VideoCountEval
Molmo2-VideoCountEval
Molmo2-VideoCountEval is a validation set of video pointing data collected from human annotators.
It can be used to evaluate vision-language models' video counting capability.
Molmo2-VideoCountEval is part of the Molmo2 dataset collection and was used to evaluate the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Data Format
video_source: There are two video sources: youtube, and MammalNet. For YouTube videos, you need to… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-VideoCountEval.molmo2-ref-davis17
ref-davis17 Tracking Dataset
Video object tracking annotations for ref-davis17.
Configs
Config
Description
Splits
track (default)
Track points across all frames
train, valid
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-ref-davis17", split="train")
# Specific config
ds = load_dataset("allenai/molmo2-ref-davis17", "ground", split="train")
molmo2-academic-video-points
AcademicTrackingPoints
Video object tracking point annotations derived from academic video segmentation
and tracking benchmarks. Each row contains per-frame point locations for tracked
objects across video clips, along with category labels, frame metadata, and object counts.
Subsets
Subset
Description
Train
Val
lvvis
Large Vocabulary Video Instance Segmentation
yes
yes
ovis
Occluded Video Instance Segmentation
yes
burst
Benchmark for Unifying Recognition… See the full description on the dataset page: https://huggingface.co/datasets/allenai/molmo2-academic-video-points.molmo2-ref-yt-vos
ref-yt-vos Tracking Dataset
Video object tracking annotations for ref-yt-vos.
Configs
Config
Description
Splits
track (default)
Track points across all frames
train, valid
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-ref-yt-vos", split="train")
# Specific config
ds = load_dataset("allenai/molmo2-ref-yt-vos", "ground", split="train")
molmo2-burst
burst Tracking Dataset
Video object tracking annotations for burst.
Configs
Config
Description
Splits
track (default)
Track points across all frames
train
ground
First and last appearance of points
train
single_point_track
Given starting point, track to end
train
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-burst", split="train")
# Specific config
ds = load_dataset("allenai/molmo2-burst"… See the full description on the dataset page: https://huggingface.co/datasets/allenai/molmo2-burst.molmo2-lv-vis
lv-vis Tracking Dataset
Video object tracking annotations for lv-vis.
Configs
Config
Description
Splits
track (default)
Track points across all frames
train
ground
First and last appearance of points
train
single_point_track
Given starting point, track to end
train
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-lv-vis", split="train")
# Specific config
ds =… See the full description on the dataset page: https://huggingface.co/datasets/allenai/molmo2-lv-vis.Molmo2-CapEval
Molmo2-CapEval
Molmo2-CapEval is a dataset of very long, detailed video captions from multiple annotators per video.
It can be used to test the caption capability of vision-language models.
Molmo2-Cap is part of the Molmo2 dataset collection and was used to test the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Evaluation code
Please check out the caption_eval.py file for caption evaluation used in Molmo2 paper.
Prepare videos… See the full description on the dataset page: https://huggingface.co/datasets/allenai/Molmo2-CapEval.molmo2-revos
revos Tracking Dataset
Video object tracking annotations for revos.
Configs
Config
Description
Splits
track (default)
Track points across all frames
train
ground
First and last appearance of points
train
single_point_track
Given starting point, track to end
train
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-revos", split="train")
# Specific config
ds = load_dataset("allenai/molmo2-revos"… See the full description on the dataset page: https://huggingface.co/datasets/allenai/molmo2-revos.molmo2-reasonvos
reasonvos Tracking Dataset
Video object tracking annotations for reasonvos.
Configs
Config
Description
Splits
track (default)
Track points across all frames
test
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-reasonvos", split="train")
# Specific config
ds = load_dataset("allenai/molmo2-reasonvos", "ground", split="train")
molmo2-moca
moca Tracking Dataset
Video object tracking annotations for moca.
Configs
Config
Description
Splits
track (default)
Track points across all frames
train
ground
First and last appearance of points
train
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-moca", split="train")
# Specific config
ds = load_dataset("allenai/molmo2-moca", "ground", split="train")
molmo2-mevis-valid
mevis-valid Tracking Dataset
Video object tracking annotations for mevis-valid.
Configs
Config
Description
Splits
track (default)
Track points across all frames
valid
Usage
from datasets import load_dataset
# Default (track)
ds = load_dataset("allenai/molmo2-mevis-valid", split="train")
# Specific config
ds = load_dataset("allenai/molmo2-mevis-valid", "ground", split="train")
Molmo2-CountixMolmo2-NewsVideoQAMolmo2-VideoPoint
Molmo2-VideoPoint
Molmo2-VideoPoint is a dataset of video pointing data collected from human annotators.
It can be used to fine-tune vision-language models for video grounding by pointing.
Molmo2-VideoPoint is part of the Molmo2 dataset collection and was used to train the Molmo2 family of models.
Quick links:
📃 Paper
🎥 Blog with Videos
Usage
from datasets import load_dataset
# Load entire dataset
ds = load_dataset("allenai/Molmo2-VideoPoint", split="train")
#… See the full description on the dataset page: https://huggingface.co/datasets/lessen1024/Molmo2-VideoPoint.
