nvidia/PhysicalAI-VANTAGE-Bench

1---2license: other3license_name: nvidia-evaluation-data-license4license_link: LICENSE.md5task_categories:6- image-text-to-text7- video-text-to-text8dataset_info:9  splits:10  - name: test11    num_examples: 3502712---13 14# VANTAGE-BENCH15 16*Video ANalysis Tasks Across Generalized Environments*17 18**Paper:** [VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models](https://huggingface.co/papers/2609.09396)19 20## Dataset Description21 22VANTAGE-BENCH is the first public benchmark purpose-built for evaluating visual understanding on video captured by fixed infrastructure cameras. It spans three real-world domains — warehouse, smart city / Intelligent Transportation Systems (ITS), and smart spaces — across six spatio-temporal video understanding tasks including video question answering (VQA), temporal grounding, dense video captioning, event verification, spatial grounding, and spatio-temporal tracking.23 24This dataset is for evaluation purposes only.25 26## Dataset Owner(s)27 28NVIDIA Corporation29 30## Dataset Creation Date31 32April 24, 202633 34## License/Terms of Use35 36This dataset is released under the [NVIDIA Evaluation Data License](./LICENSE.md).37 38## Dataset Characterization39 40**Data Collection Method**<br>41Hybrid: Human, Synthetic, Automated. Video data is sourced from vendor-provided footage (GoPro captures of warehouse and smart space environments), synthetic generation (DriveSim collision and multi-camera scenarios), and publicly scraped sources (Dubuque highway/ITS footage).42 43**Labeling Method**<br>44Hybrid: Human, Synthetic, Pseudolabeled. Annotations for VQA, dense video captions, and temporal localization are primarily human-authored. Spatial grounding labels (2D/3D bounding boxes, referring expressions) use a combination of human annotation and pseudolabeling pipelines (detection + SAM for spatial pointing). Event verification labels are human-curated. Annotations are held server-side for evaluation only.45 46### Directory Structure47 48```text49VANTAGE-BENCH/50├── vqa/                     # Video question answering51├── dense_captioning/        # Dense video captioning52├── temporal_localization/   # Temporal localization53├── event_verification/      # Event verification54├── 2dbbox/                  # 2D object localization55├── referring/               # 2D referring expressions56├── pointing/                # 2D spatial pointing57├── tracking/                # Spatio-temporal tracking58└── README.md                # Dataset documentation and submission instructions59```60 61## Evaluation62 63### Tasks and Submission Formats64 65| Category | Task | Metric |66|----------|------|--------|67| Semantic | VQA | Accuracy |68| Semantic | Event Verification | Macro F1 |69| Temporal | Dense Video Captioning | SODA-c |70| Temporal | Temporal Localization | mIoU |71| Spatial | 2D Object Localization | F1@0.5 |72| Spatial | 2D Referring Expressions | mIoU |73| Spatial | 2D Spatial Pointing | Accuracy |74| Spatio-Temporal | Single Object Tracking | AUC |75 76Submit predictions via the [VANTAGE-Bench submission portal](https://vantage-bench.org/submit) and track results on the [official leaderboard](https://huggingface.co/spaces/clemson-computing/VANTAGE-Bench-Leaderboard).77 78### Metric Notes79 80- **Accuracy**: Percentage of correct predictions.81- **SODA-c**: Metric for dense video captioning quality across event coverage and language quality.82- **Macro F1**: Unweighted mean of per-class F1 scores.83- **F1@0.5**: F1 score at an IoU threshold of 0.5.84- **mIoU**: Mean Intersection over Union — average overlap between predicted and ground-truth regions (spatial bounding boxes or temporal segments, depending on the task).85- **AUC**: Area under the ROC curve, measuring the model's ability to distinguish correct detections or tracks from incorrect ones across varying confidence thresholds.86 87### Evaluation Server88 89The [VANTAGE-Bench GitHub repository](https://github.com/Clemson-Capstone/VANTAGE-Bench) provides a sample evaluation pipeline for generating model predictions. Predictions are submitted through the [submission portal](https://vantage-bench.org/submit) and scored on the [official leaderboard](https://huggingface.co/spaces/clemson-computing/VANTAGE-Bench-Leaderboard).90 91## Dataset Format92 93Video (mp4) and Images (jpg).94 95## Dataset Quantification96 97| Category | Task | Media | Entries |98|----------|------|-------|---------|99| Semantic | VQA | 282 videos | 1,195 QAs (MCQ) |100| Semantic | Event Verification | 163 videos | 163 QAs (BCQ) |101| Temporal | Dense Video Captioning | 104 videos | 717 Events |102| Temporal | Temporal Localization | 203 videos | 1,067 Segments / Spans |103| Spatial | 2D Object Localization | 628 images (3 video sequences) | 27,404 Bboxes |104| Spatial | 2D Referring Expressions | 1,503 images | 3,276 Expressions |105| Spatial | 2D Spatial Pointing | 361 images | 1,005 QAs (MCQ) |106| Spatio-Temporal | Single Object Tracking | 102 video clips | 200 Trajectories |107 108**Total Entries (Annotations):** 35,027109**Total Media Samples (across tasks, with overlaps):** 3,346110**Total Data Storage:** 42 GB111 112## Potential Known Risks113 114- Ground truth annotations are not publicly released. All evaluation is performed server-side.115- Some warehouse videos are concatenated clips from longer recording sessions.116 117## Citations118 119```bibtex120@inproceedings{Fujita2020SODA,121  author    = {Soichiro Fujita and Tsutomu Hirao and Hidetaka Kamigaito and Manabu Okumura and Masaaki Nagata},122  title     = {{SODA}: Story Oriented Dense Video Captioning Evaluation Framework},123  booktitle = {Proc. ECCV},124  year      = {2020}125}126 127@inproceedings{Fu2024BLINK,128  author    = {Xingyu Fu and Yushi Hu and Bangzheng Li and Yu Feng and Haoyu Wang and Xudong Lin and Dan Roth and Noah A. Smith and Wei-Chiu Ma and Ranjay Krishna},129  title     = {{BLINK}: Multimodal Large Language Models Can See but Not Perceive},130  booktitle = {Proc. ECCV},131  year      = {2024}132}133 134@article{Sun2025RefDrone,135  author    = {Zhichao Sun and Yuda Zou and Xian Sun and Yingchao Feng and Wenhui Diao and Menglong Yan and Kun Fu},136  title     = {{RefDrone}: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenes},137  journal   = {arXiv preprint arXiv:2502.00392},138  year      = {2025}139}140```141 142## References143 144- HuggingFace dataset: [nvidia/PhysicalAI-VANTAGE-Bench](https://huggingface.co/datasets/nvidia/PhysicalAI-VANTAGE-Bench)145- Project website: [vantage-bench.org](https://vantage-bench.org/)146 147<img src="./assets/vantage_bench_tasks.png" alt="VANTAGE-BENCH task overview across Semantic, Temporal, Spatial, and Spatio-Temporal understanding categories" width="100%">148 149## Ethical Considerations150 151NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal developer teams to ensure this dataset meets requirements for the relevant industry and use case and addresses unforeseen product misuse.152Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).153 154## Changelog155 156See [CHANGELOG.md](./CHANGELOG.md) for release history.