nvidia/Cosmos-Reason1-Benchmark

1---2configs:3  - config_name: bridgev24    data_files:5      - split: benchmark6        path: bridgev2/bridgev2_benchmark_qa_pairs.json7  - config_name: robovqa8    data_files:9      - split: benchmark10        path: robovqa/robovqa_benchmark_qa_pairs.json11  - config_name: agibot12    data_files:13      - split: benchmark14        path: agibot/agibot_benchmark_qa_pairs.json15  - config_name: holoassist16    data_files:17      - split: benchmark18        path: holoassist/holoassist_benchmark_qa_pairs.json19  - config_name: robofail20    data_files:21      - split: benchmark22        path: robofail/robofail_benchmark_qa_pairs.json23language:24- en25task_categories:26- visual-question-answering27- video-text-to-text28tags:29- video30---31 32 33## Dataset Description:34 35The data format is a pair of video and text annotations. We summarize the data and annotations in Table 4 (SFT), Table 5 (RL), and Table 6 (Benchmark) of the Cosmos-Reason1 paper. ​​ We release the annotations for embodied reasoning tasks for BridgeDatav2, RoboVQA, Agibot, HoloAssist, AV, and the videos for the RoboVQA and AV datasets. We additionally release the annotations and videos for the RoboFail dataset for benchmarks. By releasing the dataset, NVIDIA supports the development of open embodied reasoning models and provides benchmarks to evaluate the progress. 36 37This dataset is ready for commercial/non-commercial use.38 39## Dataset Owner(s):40NVIDIA Corporation41 42## Dataset Creation Date:432025/05/1744 45## License/Terms of Use: 46The use of this dataset is governed by [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/deed.en). Additional Information: [Apache License 2.0](https://github.com/google-deepmind/robovqa/blob/main/LICENSE); [MIT](https://github.com/real-stanford/reflect/blob/main/LICENSE).47 48## Intended Usage:49This dataset is intended to demonstrate and facilitate understanding and usage of the Cosmos-Reason1 models.  It should primarily be used for educational and demonstration purposes.50 51## Dataset Characterization52The embodied reasoning datasets and benchmarks focus on the following areas: robotics (RoboVQA, BridgeDataV2, Agibot, RobFail), ego-centric human demonstration (HoloAssist), and Autonomous Vehicle (AV) driving video data.53**The AV data is currently unavailable and will be uploaded soon!**54 55**Data Collection Method**:56* RoboVQA:  Hybrid:  Automatic/Sensors57* BridgeDataV2: Automatic/Sensors58* AgiBot: Automatic/Sensors59* RoboFail: Automatic/Sensors60* HoloAssist: Human61* AV: Automatic/Sensors62 63**Labeling Method**:64* RoboVQA:  Hybrid:  Human,Automated65* BridgeDataV2: Hybrid:  Human,Automated66* AgiBot: Hybrid:  Human,Automated67* RoboFail: Hybrid:  Human,Automated68* HoloAssist: Hybrid:  Human,Automated69* AV: Hybrid:  Human,Automated70 71## Dataset Format72* Modality: Video (mp4) and Text73 74## Dataset Quantification75We release the embodied reasoning data and benchmarks. Each data sample is a pair of video and text. The text annotations include understanding and reasoning annotations described in the Cosmos-Reason1 paper. Each video may have multiple text annotations. The quantity of the video and text pairs is described in the table below.76 77| Dataset       | SFT Data | RL Data | Benchmark Data |78|--------------|---------:|--------:|---------------:|79| [RoboVQA](https://robovqa.github.io/)       | 1.14m    | 252    | 110            | 80| AV              | 24.7k    | 200    | 100            |81| [BridgeDataV2](https://rail-berkeley.github.io/bridgedata/) | 258k     | 240    | 100            |82| [Agibot](https://github.com/OpenDriveLab/AgiBot-World)        | 38.9k    | 200    | 100            |83| [HoloAssist](https://holoassist.github.io/)  | 273k     | 200    | 100            |84| [RoboFail](https://robot-reflect.github.io/)      | N/A      | N/A    | 100            |85| **Total Storage Size** | **300.6GB** | **2.6GB** | **1.5GB** | |86 87 88We release text annotations for all embodied reasoning datasets and videos for RoboVQA and AV datasets. For other datasets, users may download the source videos from the original data source and find corresponding video sources via the video names. The held-out RoboFail benchmark is released for measuring the generalization capability.89 90## Reference(s):91[[2503.15558] Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning](https://arxiv.org/abs/2503.15558)92 93## Ethical Considerations:94NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.   95 96Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).