Datasetpublic

OpenDriveLab/SparseVideoNav

SparseVideoNav Datasets This repository contains the real-world navigation datasets released with OpenDriveLab/SparseVideoNav: BVN: Beyond-the-View Navigation. IFN: Instruction-Following Navigation. Project links: Project page: https://opendrivelab.com/SparseVideoNav GitHub: https://github.com/OpenDriveLab/SparseVideoNav Paper: https://arxiv.org/abs/2602.05827 Dataset Summary SparseVideoNav studies real-world vision-language navigation with sparse future… See the full description on the dataset page: https://huggingface.co/datasets/OpenDriveLab/SparseVideoNav.

sourceHugging Facecc-by-nc-sa-4.0updated 26d agoView on Hugging Face

4likes6.1kdownloads

Dataset Card

SparseVideoNav Datasets

[image]

This repository contains the real-world navigation datasets released with OpenDriveLab/SparseVideoNav:

—BVN: Beyond-the-View Navigation.
—IFN: Instruction-Following Navigation.

Project links:

—Project page: https://opendrivelab.com/SparseVideoNav
—GitHub: https://github.com/OpenDriveLab/SparseVideoNav
—Paper: https://arxiv.org/abs/2602.05827

Dataset Summary

SparseVideoNav studies real-world vision-language navigation with sparse future video generation. The datasets contain language instructions, RGB frame sequences, and low-level navigation actions. The number of actions matches the number of RGB frames for every released episode.

This repository version contains the processed IFN and BVN subsets used by SparseVideoNav. The complete dataset contains about 140 hours; due to regional policy restrictions, the currently open-sourced portion is approximately 121.74 hours.

Subset	Episodes	RGB frames	Duration @ 4 fps	Task
`bvn`	5,433	825,786	57.35 h	Beyond-the-View Navigation
`ifn`	6,260	927,268	64.39 h	Instruction-Following Navigation
Total	11,693	1,753,054	121.74 h	-

Duration is computed as num_frames / 4 / 3600.

Repository Structure

Images are stored in compressed tar shards to avoid hundreds of thousands of small files in the Hugging Face repository. Each shard preserves the original relative paths.

text

.
├── README.md
├── assets/
│   └── dataset_mosaic.png
├── frames/
│   ├── bvn/
│   │   ├── annotations.json
│   │   ├── data.jsonl
│   │   ├── merge_info.json
│   │   ├── shard_manifest.jsonl
│   │   └── shards/
│   │       ├── bvn-00000.tar.zst
│   │       └── ...
│   └── ifn/
│       ├── annotations.json
│       ├── data.jsonl
│       ├── merge_info.json
│       ├── shard_manifest.jsonl
│       └── shards/
│           ├── ifn-00000.tar.zst
│           └── ...
└── raw_videos/
    ├── README.md
    ├── manifest.json
    ├── bvn/
    │   ├── metadata.jsonl
    │   └── <episode_id>.mp4
    └── ifn/
        ├── metadata.jsonl
        └── <episode_id>.mp4

Current shard counts:

Subset	Shards	Compressed shard bytes
`bvn`	8	14,597,684,355
`ifn`	9	16,623,840,035

Privacy-Processed Episode Videos

The episode video release is temporarily unavailable while video-only MP4 files are rebuilt and reuploaded. The frames/ release remains available.

Data Format

Each line in frames/bvn/data.jsonl or frames/ifn/data.jsonl is an episode-level JSON object.

Field	Type	Description
`dataset`	string	Dataset subset name, either `bvn` or `ifn`.
`subset`	string	Release subset marker. The current release uses `main`.
`episode_id`	string	Unique episode identifier. This matches the `id` field in `annotations.json`.
`instruction`	string	Primary natural-language navigation instruction.
`instructions`	list[string]	Instruction list. Current records contain one instruction.
`task_type`	string	Task label, e.g. `beyond_the_view_navigation` or `instruction_following_navigation`.
`split`	string	Dataset split. Current release uses `train`.
`image_dir`	string	Relative episode image directory after extraction.
`rgb_dir`	string	Relative RGB frame directory after extraction.
`num_frames`	integer	Number of RGB frames in the episode.
`num_actions`	integer	Number of low-level actions. This matches `num_frames`.
`actions`	list[object]	Per-frame low-level navigation actions. Each action has `dx`, `dy`, and `dyaw`.

Each action object contains:

Field	Type	Description
`dx`	float	Relative forward/backward displacement for the corresponding step.
`dy`	float	Relative lateral displacement for the corresponding step.
`dyaw`	float	Relative yaw change for the corresponding step.

Example:

json

{
  "dataset": "ifn",
  "episode_id": "<episode_id>",
  "instruction": "please go along with the rail until you are near by a red cone.",
  "num_frames": 177,
  "num_actions": 177,
  "rgb_dir": "images/<episode_dir>/rgb",
  "actions": [{"dx": 0.0429, "dy": -0.0311, "dyaw": 0.0271}]
}

annotations.json stores the annotation records with the core fields id, video, actions, and instructions. shard_manifest.jsonl stores shard-level metadata, including the shard path, episode ids, raw byte size, compressed byte size, and frame count.

Usage

Load episode metadata with Hugging Face Datasets:

python

from datasets import load_dataset

bvn = load_dataset("OpenDriveLab/SparseVideoNav", "bvn")
ifn = load_dataset("OpenDriveLab/SparseVideoNav", "ifn")

Download and inspect shards:

bash

tar -I zstd -tf frames/ifn/shards/ifn-00000.tar.zst | head
tar -I zstd -xf frames/ifn/shards/ifn-00000.tar.zst

After extraction, image paths resolve to paths such as:

text

images/<episode_dir>/rgb/000.jpg

License

The dataset is released under CC BY-NC-SA 4.0.

Citation

bibtex

@article{zhang2026sparse,
  title={Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation},
  author={Zhang, Hai and Liang, Siqi and Chen, Li and Li, Yuxian and Xu, Yukuan and Zhong, Yichao and Zhang, Fu and Li, Hongyang},
  journal={arXiv preprint arXiv:2602.05827},
  year={2026}
}