augmentedcognitionlab/MotionBlind

MotionBlind

A contrastive benchmark for physical-motion perception in Video-LLMs.


📄 Paper: MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs

A Video-LLM can watch two clips of the same person in the same room and name every object in both, yet fail to tell you which one moves faster, which way a hand travels, or how far a box slides. MotionBlind is a contrastive, minimal-pairs benchmark built to expose exactly that gap: pairs of near-identical clips that differ only in motion, each paired with two complementary yes/no questions.

<p align="center"> <img src="assets/overview.png" width="90%"> </p>

Why it is hard

Each instance is a 2×2: two near-identical videos (\(i0, i1\)) crossed with two complementary questions (\(q0, q1\)), with a diagonal gold pattern (\(q0\) is *yes* for \(i0\) and no for \(i1\); vice-versa for \(q1\)). Because the paired clips share static content and differ only in motion, a model that ignores the temporal signal cannot satisfy the pattern no matter how good its object recognition is. Single-frame, appearance, and language-only shortcuts all collapse toward the 6.25 % chance floor.

What is inside

All categories are physical / kinematic — precisely the variables that are hard to source and label from uncontrolled internet video, but directly controllable in a lab:

Metric

We use the TimeBlind / Winoground-style nested scores (primary = Instance Accuracy):

• —Acc — fraction of individual items correct (chance 50 %).
• —Q_Acc — a question scores only if correct on both videos (chance 25 %).
• —V_Acc — a video scores only if both its questions are correct (chance 25 %).
• —I_Acc — an instance scores only if all four items are correct (chance 6.25 %).

Human ceiling vs. models

Per-item accuracy looks competent, but instance-level discrimination sits near the floor — the illusion the benchmark is named for. See the paper for the full study of open and frontier Video-LLMs.

Dataset structure

The default viewer config is a videofolder — each row is a clip with an inline player plus its two complementary questions (videos/metadata.jsonl):

{"file_name": "02_00_0.mp4", "instance_id": "02_00", "category": "Speed",
 "subtype": "successive", "variant": 0,
 "question_1": "Does the person stir faster the first time compared to the second time?",
 "question_2": "Does the person stir slower the first time compared to the second time?"}

The annotations config exposes the flat item view — data/motionblind.jsonl, one row per item (164 rows):

{"instance_id": "02_00", "category": "Speed", "subtype": "successive",
 "video": "02_00_0.mp4", "variant": 0, "question_idx": 1,
 "question": "Does the person stir faster the first time compared to the second time?"}

data/instances.jsonl — one row per 2×2 instance (41 rows), with video_0/1 and question_1/2. Videos live in videos/.

Gold labels. Ground-truth answers are withheld from this release for now. Files are named <catID>_<pair>_<variant>.<ext>; the scoring follows the diagonal minimal-pairs pattern described above.

Usage

from datasets import load_dataset
from huggingface_hub import snapshot_download

# default config: clips with an inline video column + the two questions
ds = load_dataset("augmentedcognitionlab/MotionBlind", split="test")
ds[0]["video"]        # decoded video
ds[0]["question_1"]   # first question

# flat item view (one row per video x question)
items = load_dataset("augmentedcognitionlab/MotionBlind", "annotations", split="test")

# raw files on disk
path = snapshot_download("augmentedcognitionlab/MotionBlind", repo_type="dataset")

License & intended use

Released under CC-BY-4.0 for research on motion perception in video models. Human faces in all clips are automatically blurred. Please use responsibly.

Citation

If you use MotionBlind, please cite:

@article{bhatia2026motionblind,
  title   = {MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs},
  author  = {Bhatia, Dhairya and Galoaa, Bishoy and Fritsche, Oliver and Kamal, Shahid and
             Salam, Muhammad Obaidullah Abdul and Saleem, Umer and Rastogi, Om and
             Chettiar, Frania Felix and Erdogmus, Nesli and Ostadabbas, Sarah},
  journal = {arXiv preprint arXiv:2609.09528},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.09528}
}

Maintained by the Augmented Cognition Lab (ACLab), Northeastern University.