CoolFace
Datasetpublic

nvidia/video_to_data_challenge

Video to Data (V2D) Challenge Dataset Dataset Description The Video to Data (V2D) Challenge Dataset is an NVIDIA-developed benchmark for studying the complete path from human demonstration video to physics-grounded robot behavior. It supports three coupled challenge tracks over shared manipulation tasks: 4D human-object interaction reconstruction, robotic grounding, and end-to-end egocentric transfer. Challenge website Starter toolkit Dataset repository Contact:… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/video_to_data_challenge.

sourceHugging Facecc-by-4.0updated 2d agoView on Hugging Face
11likes771downloads
Dataset Card

Video to Data (V2D) Challenge Dataset

Dataset Description

The Video to Data (V2D) Challenge Dataset is an NVIDIA-developed benchmark for studying the complete path from human demonstration video to physics-grounded robot behavior. It supports three coupled challenge tracks over shared manipulation tasks: 4D human-object interaction reconstruction, robotic grounding, and end-to-end egocentric transfer.

Challenge Tracks

TrackInputGoalEvaluation summary
Track 1: ReconstructionMonocular third-person RGB videoRecover the human, object pose, and object geometry as a metric 4D human-object interaction scene in a consistent world frameReconstruction accuracy and physical plausibility relative to the multi-view reference
Track 2: Robotic GroundingThird-person 4D human-object interaction trajectories at different input-noise tiersRetarget the demonstration and learn an executable policy for the robotic embodiment and simulatorObject-tracking performance at each input tier
Track 3: EgocentricEgocentric human demonstration videoProduce an executable robot policy through either an explicit reconstruct-and-retarget pipeline or an implicit end-to-end methodFinal results produced with the official evaluation script

Track 1: Reconstruction

Track 1 evaluates monocular 4D human-object interaction reconstruction under challenging conditions including occlusion, bimanual coordination, and long-horizon manipulation.

Participants reconstruct:

  • —the human body and hands;
  • —object pose trajectories;
  • —object geometry; and
  • —metric scale.

Track 1 is evaluated along two equally weighted axes:

  1. 1.Accuracy
  2. 2.Chamfer distance to the multi-view human mesh
  3. 3.Chamfer distance to the multi-view object mesh
  4. 4.Physical plausibility
  5. 5.Human-joint acceleration error
  6. 6.Object acceleration error
  7. 7.Contact penetration error

Track 2: Robotic Grounding

Track 2 measures how upstream reconstruction quality affects human-to-robot transfer and downstream policy learning.

The dataset provides three input tiers:

  1. 1.Tier 1 — Clean multi-view capture: an upper-bound input for upstream reconstruction.
  2. 2.Tier 2 — Synthetic corruption: trajectories with jitter, dropout, and contact errors sampled from Track 1 error distributions.
  3. 3.Tier 3 — Off-the-shelf reconstruction: trajectories produced by current reconstruction methods.

Each tier is scored separately. Metrics include AUC, SP-SR, MP-SR, and MPPE, as defined by the challenge evaluation protocol.

Track 3: Egocentric

Track 3 evaluates the full pipeline from egocentric human video to robot behavior. The track is method-agnostic: participants may use an explicit reconstruction-and-retargeting pipeline, an end-to-end model, a pretrained vision-language-action model, a world-action model, or a hybrid approach.

The NVIDIA-collected data includes human-object manipulation recordings and associated assets made available for development and evaluation. Depending on the released split, these assets may include:

  • —egocentric videos;
  • —motion-capture trajectories;
  • —sequence metadata;
  • —textured 3D object meshes; and
  • —URDF object descriptions.

Use eval_e2e.py from the starter toolkit to package the required reconstructions and recorded policy evaluations.

Download

Install the Hugging Face Hub client:

bash
python -m pip install -U "huggingface_hub"

Download the complete dataset repository while preserving its file structure:

bash
hf download nvidia/video_to_data_challenge \
  --repo-type dataset \
  --local-dir ./video_to_data_challenge

If authentication is requested, first run:

bash
hf auth login

The same operation can be performed from Python:

python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="nvidia/video_to_data_challenge",
    repo_type="dataset",
    local_dir="./video_to_data_challenge",
)

Large assets are stored using Hugging Face's large-file infrastructure. Make sure sufficient disk space is available before downloading the complete repository.

Support

For challenge or dataset questions, contact v2d_challenge@nvidia.com. For software issues, use the issue tracker in the Video to Data repository.