nvidia/video_to_data_challenge

1---2license: cc-by-4.03pretty_name: Video to Data (V2D) Challenge Dataset4tags:5  - robotics6  - robot-learning7  - manipulation8  - human-object-interaction9  - motion-capture10  - video11  - 4d-reconstruction12  - reinforcement-learning13  - egocentric-video14---15 16# Video to Data (V2D) Challenge Dataset17 18## Dataset Description19 20The **Video to Data (V2D) Challenge Dataset** is an NVIDIA-developed benchmark for studying the complete path from human demonstration video to physics-grounded robot behavior. It supports three coupled challenge tracks over shared manipulation tasks: **4D human-object interaction reconstruction**, **robotic grounding**, and **end-to-end egocentric transfer**.21 22- [Challenge website](https://nvidia-isaac.github.io/video_to_data/v2d_challenge/)23- [Starter toolkit](https://github.com/nvidia-isaac/video_to_data)24- [Dataset repository](https://huggingface.co/datasets/nvidia/video_to_data_challenge)25- Contact: [v2d_challenge@nvidia.com](mailto:v2d_challenge@nvidia.com)26 27## Challenge Tracks28 29| Track | Input | Goal | Evaluation summary |30| --- | --- | --- | --- |31| **Track 1: Reconstruction** | Monocular third-person RGB video | Recover the human, object pose, and object geometry as a metric 4D human-object interaction scene in a consistent world frame | Reconstruction accuracy and physical plausibility relative to the multi-view reference |32| **Track 2: Robotic Grounding** | Third-person 4D human-object interaction trajectories at different input-noise tiers | Retarget the demonstration and learn an executable policy for the robotic embodiment and simulator | Object-tracking performance at each input tier |33| **Track 3: Egocentric** | Egocentric human demonstration video | Produce an executable robot policy through either an explicit reconstruct-and-retarget pipeline or an implicit end-to-end method | Final results produced with the official evaluation script |34 35### Track 1: Reconstruction36 37Track 1 evaluates monocular 4D human-object interaction reconstruction under challenging conditions including occlusion, bimanual coordination, and long-horizon manipulation.38 39Participants reconstruct:40 41- the human body and hands;42- object pose trajectories;43- object geometry; and44- metric scale.45 46Track 1 is evaluated along two equally weighted axes:47 481. **Accuracy**49   - Chamfer distance to the multi-view human mesh50   - Chamfer distance to the multi-view object mesh512. **Physical plausibility**52   - Human-joint acceleration error53   - Object acceleration error54   - Contact penetration error55 56### Track 2: Robotic Grounding57 58Track 2 measures how upstream reconstruction quality affects human-to-robot transfer and downstream policy learning.59 60The dataset provides three input tiers:61 621. **Tier 1 — Clean multi-view capture:** an upper-bound input for upstream reconstruction.632. **Tier 2 — Synthetic corruption:** trajectories with jitter, dropout, and contact errors sampled from Track 1 error distributions.643. **Tier 3 — Off-the-shelf reconstruction:** trajectories produced by current reconstruction methods.65 66Each tier is scored separately. Metrics include **AUC**, **SP-SR**, **MP-SR**, and **MPPE**, as defined by the challenge evaluation protocol.67 68### Track 3: Egocentric69 70Track 3 evaluates the full pipeline from egocentric human video to robot behavior. The track is method-agnostic: participants may use an explicit reconstruction-and-retargeting pipeline, an end-to-end model, a pretrained vision-language-action model, a world-action model, or a hybrid approach.71 72The NVIDIA-collected data includes human-object manipulation recordings and associated assets made available for development and evaluation. Depending on the released split, these assets may include:73 74- egocentric videos;75- motion-capture trajectories;76- sequence metadata;77- textured 3D object meshes; and78- URDF object descriptions.79 80Use `eval_e2e.py` from the starter toolkit to package the required reconstructions and recorded policy evaluations.81 82## Download83 84Install the Hugging Face Hub client:85 86```bash87python -m pip install -U "huggingface_hub"88```89 90Download the complete dataset repository while preserving its file structure:91 92```bash93hf download nvidia/video_to_data_challenge \94  --repo-type dataset \95  --local-dir ./video_to_data_challenge96```97 98If authentication is requested, first run:99 100```bash101hf auth login102```103 104The same operation can be performed from Python:105 106```python107from huggingface_hub import snapshot_download108 109snapshot_download(110    repo_id="nvidia/video_to_data_challenge",111    repo_type="dataset",112    local_dir="./video_to_data_challenge",113)114```115 116Large assets are stored using Hugging Face's large-file infrastructure. Make sure sufficient disk space is available before downloading the complete repository.117 118## Support119 120For challenge or dataset questions, contact [v2d_challenge@nvidia.com](mailto:v2d_challenge@nvidia.com). For software issues, use the issue tracker in the [Video to Data repository](https://github.com/nvidia-isaac/video_to_data/issues).121