nyu-visionx/vstat

VSTAT: Visual State Tracking Benchmark

VSTAT is a video-based benchmark for evaluating the visual state tracking capability of Multimodal Large Language Models (MLLMs). It contains 834 video clips paired with 1,500 questions whose answers cannot be inferred from any single keyframe or short segment.

Dataset Composition

Files

• —vstat_qa_clean.json — all 1,500 question-answer pairs with taxonomy labels
• —youtube_metadata.json — YouTube URLs + start/end timestamps (one entry per chunk)
• —youtube_resolutions.json — per-clip target (W, H, fps) used by the downloader to reproduce the official release pixel layout
• —redactions.json — declarative privacy-redaction regions applied after trim
• —scripts/download_youtube.py — fetches & trims the 304 YouTube clips
• —scripts/redact.sh — applies the privacy black-boxes from redactions.json
• —scripts/build_resolution_map.py — utility to (re)build youtube_resolutions.json from a reference render
• —videos/synthetic/<category>/<id>.mp4 — Blender-rendered videos (hosted)
• —videos/self_recorded/<category>/<id>.mp4 — author-recorded clips, hands only, audio removed (hosted)
• —videos/youtube/<category>/<id>.mp4 — NOT redistributed; you must download these yourself with the provided script (see Quick start below)

Quick start

1. Get the repo

Pick whichever method you prefer:

# A. huggingface-cli (recommended, supports LFS)
pip install -U "huggingface_hub[cli]"
huggingface-cli download nyu-visionx/vstat \
  --repo-type=dataset \
  --local-dir vstat
cd vstat

# B. git clone (requires git-lfs installed)
git lfs install
git clone https://huggingface.co/datasets/nyu-visionx/vstat vstat
cd vstat

After this, you have all annotations and the synthetic + self_recorded videos. The YouTube clips are still missing — fetch them next.

2. Download and redact the YouTube clips

The downloader reads youtube_metadata.json and downloads each source video once with yt-dlp, then trims it into the chunks expected by vstat_qa_clean.json. Pass --resolution-map youtube_resolutions.json so each chunk lands at the exact (width, height, fps) of the official release. After trimming, scripts/redact.sh applies the privacy black-boxes (matches redactions.json) to the affected clips in place.

Important — reproducing the official release. The benchmark numbers in our paper were obtained on the clips produced by exactly this two-step pipeline (download_youtube.py --resolution-map … → redact.sh). The downloader picks the smallest YouTube format that matches each clip's target dimensions and frame rate so the trim avoids any resampling drift. Skip the resolution map only for ablations on input resolution.

# Install dependencies
pip install -U yt-dlp
# macOS:    brew install ffmpeg
# Ubuntu:   sudo apt install ffmpeg

# 1. Fetch and trim every YouTube clip to its release-spec dims
python scripts/download_youtube.py --resolution-map youtube_resolutions.json

# 2. Apply privacy redactions in place (idempotent)
bash scripts/redact.sh

Common flags for the downloader:

# Faster: 4 parallel downloads
python scripts/download_youtube.py --resolution-map youtube_resolutions.json --workers 4

# Test on a few videos first
python scripts/download_youtube.py --resolution-map youtube_resolutions.json --limit 5

# Keep the full source videos around (faster re-trim, more disk)
python scripts/download_youtube.py --resolution-map youtube_resolutions.json --keep-fulls

# Print plan without doing anything
python scripts/download_youtube.py --resolution-map youtube_resolutions.json --dry-run

# Cap source download size (default uncapped — required for portrait sources)
python scripts/download_youtube.py --resolution-map youtube_resolutions.json --source-cap 1080

Re-running the downloader is safe: it skips clips that already exist on disk and writes a download_report.json listing any failures (rare, usually due to YouTube link rot — affected clips can be reported to the authors via the dataset issue tracker). Re-running redact.sh is also idempotent and replaces any earlier redaction with the canonical set defined in redactions.json.

3. Load the data

import json

with open("vstat_qa_clean.json") as f:
    data = json.load(f)

for cat, entries in data["data"].items():
    for e in entries:
        print(e["video_id"], e["video_path"], e["video_source"])

Each entry has these fields:

4. Run an evaluation

A minimal MCQ scoring loop (numeric questions are scored with mean relative accuracy in our paper; see Section 3.1 for details):

def score(entry, model_pred):
    if entry["answer_type"] == "mcq":
        return int(model_pred.strip().upper() == entry["answer"])
    # numeric
    try:
        return int(int(model_pred) == int(entry["answer"]))
    except ValueError:
        return 0

Taxonomy

Each question is annotated with:

• —perceptual_complexity (multi-label, paper Section 2.2): action_ambiguity, camera_motion, homogeneity, multi_entity_attribution, occlusion, symbolic_decoding
• —state_element_type (single label): count, location, attribute
• —state_structure (single label): atomic, sequence, set, dictionary

License

• —Annotations and self-recorded / synthetic videos: CC BY 4.0
• —YouTube videos: NOT redistributed; subject to original uploader's license
• —See LICENSE for full terms

Privacy & consent

• —Self-recorded videos contain only the authors' hands; no faces, voices, or other identifiable persons. Audio tracks were stripped before release.
• —Authors consented to public release of their hand footage.
• —For YouTube clips, only URLs and timestamps are redistributed; original uploaders retain control over their content. The redact.sh step applies black-boxes over scoreboards / on-screen text in a small number of clips per redactions.json, matching the official release.

Citation

@article{vstat2026,
    title={Benchmarking Visual State Tracking in Multimodal Video Understanding},
    author={Sihyun Yu and Nanye Ma and Pinzhi Huang and Hyunseok Lee and Shusheng Yang and June Suk Choi and Ellis Brown and Oscar Michel and Boyang Zheng and Jinwoo Shin and Saining Xie},
    year={2026},
    journal={arXiv preprint arXiv:2606.03920},
}