nvidia/ffs_stereo4d

1---2license: cc-by-nc-4.03size_categories:4- 1M<n<10M5task_categories:6- depth-estimation7pretty_name: FFS Stereo4D8tags:9- stereo-matching10- disparity11- stereo4d12- foundationstereo13---14 15# FFS Stereo4D16 17[[Project Page]](https://nvlabs.github.io/Fast-FoundationStereo/) [[Paper]](https://huggingface.co/papers/2512.11130) [[Code]](https://github.com/NVlabs/Fast-FoundationStereo)18 19Disparity maps for stereo matching, generated from the [Stereo4D](https://github.com/niconielsen32/Stereo4D) dataset using [FoundationStereo](https://github.com/NVlabs/FoundationStereo).20 21## Dataset Structure22 23```24data/train/25  metadata.csv26  0000000.zip   (first 50,000 images)27  0000001.zip   (next 50,000 images)28  ...29  0000025.zip30```31 32Each zip contains disparity PNG files named `{vid_id}_frame_{frame_idx:06d}.png`.33 34- **Disparity images**: 3-channel uint8 784×784 PNG files encoding per-pixel disparity. Decode with: `disp = (R * 255*255 + G * 255 + B) / 1000.0`. See also: https://github.com/NVlabs/FoundationStereo/blob/master/scripts/vis_dataset.py35- **metadata.csv**: Links each disparity image back to its source YouTube video, with a `zip_file` column indicating which zip contains the image.36 37### Metadata Columns38 39| Column | Description |40|---|---|41| `file_name` | Disparity image filename (inside the zip) |42| `zip_file` | Which zip file contains this image |43| `vid_id` | Clip identifier (matches the `.npz` calibration file) |44| `frame_idx` | Frame index in the rectified stereo output |45| `youtube_video_id` | YouTube video ID of the source 360 video |46| `timestamp_us` | Timestamp in microseconds in the original video |47| `timestamp_sec` | Timestamp in seconds |48| `video_frame_index` | Estimated frame number in the original video |49| `fps` | FPS of the source video |50 51## Retrieving Source RGB Frames52 53This dataset contains **disparity maps only**. Due to the copyrights of these videos, users need to download on your own behalf. The corresponding left/right RGB stereo pairs can be recovered by:54 551. Following [stereo4d toolkit](https://github.com/Stereo4d/stereo4d-code) to download the YouTube video using `youtube_video_id`.562. Seek to `timestamp_sec` (or `video_frame_index`) to locate the source frame.573. Apply equirectangular rectification using the Stereo4D calibration `.npz` files to obtain the left and right perspective images.58 59## Generation Pipeline60 611. **Source**: YouTube 360 videos from the Stereo4D dataset.622. **Rectification**: Equirectangular frames are rectified and cropped to 1024×1024 perspective stereo pairs.633. **Disparity estimation**: FoundationStereo computes dense disparity at 784×784 resolution (resized by `scale=0.765625` of the 1024×1024 input).64 65### Camera Parameters66 67The rectified stereo pairs are generated at 1024×1024 with the following pinhole camera model:68 69| Parameter | Value (1024×1024 rectified) | Value (784×784 disparity) | Formula |70|---|---|---|---|71| HFOV | 60° | 60° | `output_hfov` in `batch_rectify.py` |72| Baseline | 0.063 m | 0.063 m | Assumed interpupillary distance for VR180 cameras |73| fx, fy | 886.8 px | 678.8 px | `size * 0.5 / tan(0.5 * HFOV * pi/180)` |74| cx, cy | 512 px | 392 px | Image center |75 76Depth is derived as: `depth = fx * baseline / disparity`.77 78Since disparity is computed at 784×784 resolution (scale factor 784/1024 = 0.765625 of the 1024×1024 input), use the 784×784 camera parameters when converting disparity to depth:79 80```python81import numpy as np82hfov = 60  # degrees83baseline = 0.063  # meters84imw = 78485fx = imw * 0.5 / np.tan(0.5 * np.radians(hfov))  # 678.8 px86depth = fx * baseline / disparity87```88 89 90## Citation91 92If you use this dataset, please consider cite:93 94```bibtex95@article{wen2026fastfoundationstereo,96  title={Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matching},97  author={Bowen Wen and Shaurya Dewan and Stan Birchfield},98  journal={CVPR},99  year={2026}100}101@article{wen2025foundationstereo,102  title={FoundationStereo: Zero-Shot Stereo Matching},103  author={Wen, Bowen and Trepte, Matthew and Aribido, Joseph and Kautz, Jan and Birchfield, Stan and Wan, Yao},104  journal={CVPR},105  year={2025}106}107@inproceedings{jin2025stereo4d,108  title={{Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos}},109  author={Jin, Linyi and Tucker, Richard and Li, Zhengqi and Fouhey, David and Snavely, Noah and Holynski, Aleksander},110  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},111  year={2025},112}113```