PaulineLi/QuantiPhy

QuantiPhy

Dataset Summary

QuantiPhy is a benchmark for evaluating whether vision–language models (VLMs) can perform quantitative physical inference from visual evidence, rather than producing plausible but ungrounded numerical guesses.

This repository contains the official test set of the QuantiPhy benchmark, consisting of 3,289 video–question (QA) pairs across 568 videos. Ground-truth answers are withheld to ensure fair evaluation.

Each instance requires a model to output a single continuous numerical value (e.g., object size, velocity, or acceleration) in real-world units, given a short video and a natural-language question.

Looking for the validation set? A 159-sample validation split with ground-truth answers is available at PaulineLi/QuantiPhy-validation for model development, prompt tuning, and ablation studies.

🏆 QuantiPhy Challenge @ NeurIPS 2026

QuantiPhy has been accepted to the [NeurIPS 2026 Competition Track](https://quantiphy.stanford.edu/competition/index.html)! The challenge is live — run your model on this test set, submit your predictions, and get scored on the official leaderboard.

• —Competition website: https://quantiphy.stanford.edu/competition/index.html
• —Submission deadline: October 23, 2026, 23:59 AoE
• —How to submit: Register a team (up to 5 members), run your model on this test set, format predictions as a single CSV matching the reference submission template, and upload through the submission portal (up to 3 scored submissions per day).
• —Tracks: Track A (Main) — any model permitted, ranked by raw numerical accuracy. Track B (Open-Weight) — restricted to publicly available model weights and tools for reproducibility.
• —Metric: Mean Relative Accuracy (MRA).
• —Prizes (per track): 1st — \$1,000 · 2nd — \$500 · 3rd — \$250.

Evaluation code and a starter kit for running a VLM on QuantiPhy are available in the GitHub repository.

Supported Tasks

• —Video-based numerical regression
• —Quantitative visual reasoning
• —Vision–language model evaluation

Tasks cover three core kinematic properties:

• —Size
• —Velocity
• —Acceleration

All questions are open-ended and require predicting a real-valued scalar.

Dataset Structure

Each instance is represented as a structured video–text record with the following fields:

Videos are short (typically 2–3 seconds) and recorded with a static camera to ensure well-defined kinematic inference.

Video Type Code

Each video_type is a 4-character code [P][D][O][B]:

Inference Type

Task Design Overview

Each instance provides the model with:

• —a short video depicting object motion, and
• —one physical prior in world units (object size, velocity at a given timestamp, or acceleration at a given timestamp).

The model is then asked to infer a target kinematic quantity—possibly for a different object—expressed in real-world units.

Tasks vary along four axes:

1. 1.Physical prior: Size (S), Velocity (V), Acceleration (A)
2. 2.Dimensionality: 2D (planar motion) or 3D (with depth variation)
3. 3.Object setting: Single-object (S) or multi-object (M)
4. 4.Background complexity: Plain (X), Simple (S), Complex (C)

Dataset Statistics

By source:

By inference type:

Videos

Videos are provided in two resolutions:

• —quantiphy_fullset_videos/ — original resolution
• —quantiphy_fullset_videos_480p/ — 480p (for faster download and lower-resolution evaluation)

Each video filename corresponds to the video_id field in the dataset (e.g., simulation_0007.mp4).

Usage

from datasets import load_dataset

ds = load_dataset("PaulineLi/QuantiPhy", split="test")
print(ds[0])
# {'video_id': 'simulation_0007', 'video_source': 'simulation', ...}

For the validation set with ground-truth answers:

ds_val = load_dataset("PaulineLi/QuantiPhy-validation", split="validation")

Data Sources and Quality Control

• —Simulation: Blender-rendered scenes with precise physical ground truth.
• —Laboratory capture: Real-world recordings using calibrated depth and multi-view setups.
• —Internet / author-recorded videos: Carefully curated monocular videos meeting strict physical constraints.
• —Segmentation: Videos with segmented objects for controlled evaluation.

All videos undergo manual review to remove:

• —excessive motion blur,
• —severe occlusion,
• —untrackable motion,
• —personally identifiable information (PII).

License

The annotations and metadata in this repository are released under the Creative Commons Attribution 4.0 (CC BY 4.0) license.

Videos originate from simulated environments, laboratory recordings, and publicly available sources. Each video remains subject to its original license and terms of use.

This release is intended for research and evaluation purposes.

Authors

Puyin Li\, Tiange Xiang\, Ella Mao\***, Shirley Wei, Xinye Chen, Adnan Masood, Li Fei-Fei†, Ehsan Adeli†

\* Equal contribution.

Citation

If you use QuantiPhy in your work, please cite:

@article{li2025quantiphy,
  title   = {QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models},
  author  = {Li, Puyin and Xiang, Tiange and Mao, Ella and Wei, Shirley and Chen, Xinye and Masood, Adnan and Li, Fei-Fei and Adeli, Ehsan},
  journal = {arXiv preprint arXiv:2512.19526},
  year    = {2025}
}