allenai/Molmo2-VideoTrackEval

Molmo2-VideoTrackEval

Molmo2-VideoTrackEval is an evaluation benchmark for video point tracking, containing human-annotated ground truth expressions. It includes segmentation masks for evaluating whether predicted points fall within the correct object regions. Currently, there are five categories for evaluation:

• —animal
• —dance
• —sports
• —person
• —misc

This benchmark is part of the Molmo2 dataset collection and is used to evaluate the Molmo2 family of models on video object tracking via point trajectories.

Quick links:

• —📃 Paper
• —🎥 Blog with Videos

Usage

from datasets import load_dataset

# Load entire evaluation dataset
ds = load_dataset("allenai/Molmo2-VideoTrackEval", split="test")

# Load a specific benchmark subset by config name
animal = load_dataset("allenai/Molmo2-VideoTrackEval", "animal", split="test")
dance = load_dataset("allenai/Molmo2-VideoTrackEval", "dance", split="test")
sports = load_dataset("allenai/Molmo2-VideoTrackEval", "sports", split="test")
person = load_dataset("allenai/Molmo2-VideoTrackEval", "person", split="test")
misc = load_dataset("allenai/Molmo2-VideoTrackEval", "misc", split="test")

Available Configs

Data Format

Each row contains tracking annotations for one or more objects in a video clip:

Important: start_frame and end_frame indicate which portion of the source video to use. You need to trim the video to this range — the annotations correspond to frames within [start_frame, end_frame], not the entire video.

Evaluation with Masks

The masks field contains ground truth segmentation masks that can be used to evaluate tracking predictions. A predicted point is considered correct if it falls within the segmentation mask of the target object for that frame.

Folder Structure

Molmo2-VideoTrackEval/
├── README.md
└── data/
    ├── animal/
    │   └── APTv2_point_tracks_with_masks.parquet
    ├── dance/
    │   └── dancetrack_point_tracks_with_masks.parquet
    └── sports/
        └── sportsmot_point_tracks_with_masks.parquet
    ├── person/
    │   └── personpath22_point_tracks_with_masks.parquet
    ├── misc/
    │   └── sav_point_tracks_with_masks.parquet

Video Sources

The table below contains information on the sources of the third party datasets used or referenced in curating the benchmark data for Molmo2-VideoTrackEval. We do not provide video files or share the original raw data from datasets with restrictions on use and distribution according to the source license.

License

This dataset is licensed under ODC-BY-1.0. It is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines. Please refer to the Video Sources section for the original datasets that provide the videos used to generate the segmentations and point tracks for this dataset. All use of the videos and original data from these datasets are subject to the licenses and terms of use provided by the sources. Please check the sources to determine if they are appropriate for your use case.