OpenDriveLab/SparseVideoNav

1---2license: cc-by-nc-sa-4.03language:4- en5pretty_name: SparseVideoNav Datasets6task_categories:7- robotics8- visual-question-answering9tags:10- embodied-ai11- vision-language-navigation12- robot-navigation13- video14- trajectory15- sparsevideonav16- opendrivelab17configs:18- config_name: bvn19  data_files:20  - split: train21    path: frames/bvn/data.jsonl22- config_name: ifn23  data_files:24  - split: train25    path: frames/ifn/data.jsonl26---27 28# SparseVideoNav Datasets29 30![SparseVideoNav dataset overview](assets/dataset_mosaic.png)31 32This repository contains the real-world navigation datasets released with [OpenDriveLab/SparseVideoNav](https://github.com/OpenDriveLab/SparseVideoNav):33 34- **BVN**: Beyond-the-View Navigation.35- **IFN**: Instruction-Following Navigation.36 37Project links:38 39- Project page: https://opendrivelab.com/SparseVideoNav40- GitHub: https://github.com/OpenDriveLab/SparseVideoNav41- Paper: https://arxiv.org/abs/2602.0582742 43## Dataset Summary44 45SparseVideoNav studies real-world vision-language navigation with sparse future video generation. The datasets contain language instructions, RGB frame sequences, and low-level navigation actions. The number of actions matches the number of RGB frames for every released episode.46 47This repository version contains the processed IFN and BVN subsets used by SparseVideoNav. The complete dataset contains about 140 hours; due to regional policy restrictions, the currently open-sourced portion is approximately 121.74 hours.48 49| Subset | Episodes | RGB frames | Duration @ 4 fps | Task |50| --- | ---: | ---: | ---: | --- |51| `bvn` | 5,433 | 825,786 | 57.35 h | Beyond-the-View Navigation |52| `ifn` | 6,260 | 927,268 | 64.39 h | Instruction-Following Navigation |53| **Total** | **11,693** | **1,753,054** | **121.74 h** | - |54 55Duration is computed as `num_frames / 4 / 3600`.56 57## Repository Structure58 59Images are stored in compressed tar shards to avoid hundreds of thousands of small files in the Hugging Face repository. Each shard preserves the original relative paths.60 61```text62.63├── README.md64├── assets/65│   └── dataset_mosaic.png66├── frames/67│   ├── bvn/68│   │   ├── annotations.json69│   │   ├── data.jsonl70│   │   ├── merge_info.json71│   │   ├── shard_manifest.jsonl72│   │   └── shards/73│   │       ├── bvn-00000.tar.zst74│   │       └── ...75│   └── ifn/76│       ├── annotations.json77│       ├── data.jsonl78│       ├── merge_info.json79│       ├── shard_manifest.jsonl80│       └── shards/81│           ├── ifn-00000.tar.zst82│           └── ...83└── raw_videos/84    ├── README.md85    ├── manifest.json86    ├── bvn/87    │   ├── metadata.jsonl88    │   └── <episode_id>.mp489    └── ifn/90        ├── metadata.jsonl91        └── <episode_id>.mp492```93 94Current shard counts:95 96| Subset | Shards | Compressed shard bytes |97| --- | ---: | ---: |98| `bvn` | 8 | 14,597,684,355 |99| `ifn` | 9 | 16,623,840,035 |100 101## Privacy-Processed Episode Videos102 103The episode video release is temporarily unavailable while video-only MP4 files are rebuilt and reuploaded. The `frames/` release remains available.104 105## Data Format106 107Each line in `frames/bvn/data.jsonl` or `frames/ifn/data.jsonl` is an episode-level JSON object.108 109| Field | Type | Description |110| --- | --- | --- |111| `dataset` | string | Dataset subset name, either `bvn` or `ifn`. |112| `subset` | string | Release subset marker. The current release uses `main`. |113| `episode_id` | string | Unique episode identifier. This matches the `id` field in `annotations.json`. |114| `instruction` | string | Primary natural-language navigation instruction. |115| `instructions` | list[string] | Instruction list. Current records contain one instruction. |116| `task_type` | string | Task label, e.g. `beyond_the_view_navigation` or `instruction_following_navigation`. |117| `split` | string | Dataset split. Current release uses `train`. |118| `image_dir` | string | Relative episode image directory after extraction. |119| `rgb_dir` | string | Relative RGB frame directory after extraction. |120| `num_frames` | integer | Number of RGB frames in the episode. |121| `num_actions` | integer | Number of low-level actions. This matches `num_frames`. |122| `actions` | list[object] | Per-frame low-level navigation actions. Each action has `dx`, `dy`, and `dyaw`. |123 124Each action object contains:125 126| Field | Type | Description |127| --- | --- | --- |128| `dx` | float | Relative forward/backward displacement for the corresponding step. |129| `dy` | float | Relative lateral displacement for the corresponding step. |130| `dyaw` | float | Relative yaw change for the corresponding step. |131 132Example:133 134```json135{136  "dataset": "ifn",137  "episode_id": "<episode_id>",138  "instruction": "please go along with the rail until you are near by a red cone.",139  "num_frames": 177,140  "num_actions": 177,141  "rgb_dir": "images/<episode_dir>/rgb",142  "actions": [{"dx": 0.0429, "dy": -0.0311, "dyaw": 0.0271}]143}144```145 146`annotations.json` stores the annotation records with the core fields `id`, `video`, `actions`, and `instructions`. `shard_manifest.jsonl` stores shard-level metadata, including the shard path, episode ids, raw byte size, compressed byte size, and frame count.147 148## Usage149 150Load episode metadata with Hugging Face Datasets:151 152```python153from datasets import load_dataset154 155bvn = load_dataset("OpenDriveLab/SparseVideoNav", "bvn")156ifn = load_dataset("OpenDriveLab/SparseVideoNav", "ifn")157```158 159Download and inspect shards:160 161```bash162tar -I zstd -tf frames/ifn/shards/ifn-00000.tar.zst | head163tar -I zstd -xf frames/ifn/shards/ifn-00000.tar.zst164```165 166After extraction, image paths resolve to paths such as:167 168```text169images/<episode_dir>/rgb/000.jpg170```171 172## License173 174The dataset is released under CC BY-NC-SA 4.0.175 176## Citation177 178```bibtex179@article{zhang2026sparse,180  title={Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation},181  author={Zhang, Hai and Liang, Siqi and Chen, Li and Li, Yuxian and Xu, Yukuan and Zhong, Yichao and Zhang, Fu and Li, Hongyang},182  journal={arXiv preprint arXiv:2602.05827},183  year={2026}184}185```186