Datasetpublic

ACERobotics/Puffin-16M

Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States 📖 Project Page   |   💻 GitHub   |   🤗 Models   |   📄 HF Paper   |   🤗 HF Blog Dataset Details Datasets and benchmarks that span vision, language, and camera modalities remain scarce in the domain of spatial multimodal intelligence. Puffin-16M is a large-scale, camera-centric dataset that substantially scales up Puffin-4M… See the full description on the dataset page: https://huggingface.co/datasets/ACERobotics/Puffin-16M.

sourceHugging Faceupdated 18d agoView on Hugging Face

16likes11kdownloads

Dataset Card

Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

📖 <a href="https://kangliao929.github.io/projects/puffin-16m/">Project Page</a>   |   💻 <a href="https://github.com/KangLiao929/Puffin">GitHub</a>   |   🤗 <a href="https://huggingface.co/ACERobotics/Puffin-World">Models</a>   |   📄 <a href="https://huggingface.co/papers/2609.04196">HF Paper</a>   |   🤗 <a href="https://huggingface.co/blog/KangLiao/puffin-world">HF Blog</a>

Dataset Details

Datasets and benchmarks that span vision, language, and camera modalities remain scarce in the domain of spatial multimodal intelligence. Puffin-16M is a large-scale, camera-centric dataset that substantially scales up Puffin-4M, comprising ~16.5 million samples across two complementary parts:

—Puffin-Cam-15M — 15,338,221 single-view image–caption–camera triplets. Each perspective image (rendered from panoramas across diverse indoor and outdoor scenes) is paired with a caption that combines a scene description, precise camera parameters (roll, pitch, field-of-view, and radial distortion), and a camera-height category.
—Puffin-Traj-1M — 1,136,696 camera trajectories. Each trajectory is a 90-frame sequence with dense per-frame camera annotations (Euler angles, intrinsics, and camera pose), supporting camera-controlled world exploration and trajectory-aware generation and understanding.

Two small held-out benchmarks accompany the training corpus for evaluation, rendered from panoramas disjoint from the training scenes and packaged in the same format as their training counterparts:

—Puffin-Cam-15M-Bench — 600 single-view image–caption–camera pairs (<hash>.jpg ↔ <hash>.json, one shard each). Captions keep the scene-and-camera description; instead of camera_height, each JSON records the image width / height.
—Puffin-Traj-1M-Bench — 100 camera trajectories (90 frames + cameras.json per scene, one shard) with the same per-frame annotation schema as Puffin-Traj-1M.


Developed by	Kang Liao, Yihang Luo, Xiao-Ming Wu, Linyi Jin, Size Wu, Chunyu Lin, Yao Zhao, Fei Wang, Wei Li, Chen Change Loy
Affiliations	S-Lab, Nanyang Technological University; University of Michigan; Beijing Jiaotong University; ACE Robotics
First released	arXiv preprint, 2026
Dataset type	Camera-centric multimodal dataset for physical-world perception and 3D world modeling
Modality	Image → Text+Camera; Text+Camera → Image; Image+Camera → Image; Image+Camera → Text

Data Format

Puffin-Cam-15M — captions

Each caption is a JSON file, named by a random hash that matches its paired image (<hash>.jpg ↔ <hash>.json). It keeps the original scene-and-camera caption and adds a camera_height field:

json

{
  "caption": "The image shows a rocky foreground with a clear blue sky and scattered clouds above ... The camera parameters (roll, pitch, field-of-view, and radial distortion) are: 0.0977, 0.2316, 1.0756, 0.0000.",
  "camera_height": "Eye-level shot"
}

camera_height is one of: Eye-level shot, Low-position shot, High-position shot, Aerial shot, Underwater shot.

Puffin-Traj-1M — trajectories

Each trajectory scene is a folder (named by a random hash) containing its frames and a single cameras.json with per-frame camera annotations:

json

{
  "motion_type": "pitch", "image_height": 640, "image_width": 640,
  "vfov_deg": 84.47, "focal_px": 352.49, "images_per_segment": 90,
  "frames": [
    {
      "file_path": "000001.jpg",
      "roll_deg": 43.27, "pitch_deg": 44.5, "yaw_deg": 3.38,
      "camera_intrinsics": [[fx, 0, cx], [0, fy, cy], [0, 0, 1]],
      "camera_pose": [[ ... 4x4 camera-to-world ... ]]
    }
  ]
}

Camera Parameters and Ranges

Puffin-Cam-15M (single-view)

Each perspective view is rendered from a panorama with camera parameters sampled as:

Parameter	Range / Distribution	Notes
Roll	[−45°, +45°]	uniform
Pitch	[−45°, +45°]	uniform
Vertical FoV	[20°, 105°]	uniform
Radial distortion k₁	0	pinhole camera model (no distortion)

In each caption, the four numbers following "The camera parameters (roll, pitch, field-of-view, and radial distortion) are:" are given in radians; the radial-distortion term is always 0 under the pinhole model.

The paired camera_height field further categorizes the shooting height into five levels ordered by increasing altitude, spanning underwater scenes up to aerial and space-like views:

`camera_height`	Viewpoint
`Underwater shot`	below the water surface (ocean depths)
`Low-position shot`	low, near ground / sea level (looking up)
`Eye-level shot`	human eye level (the most common case)
`High-position shot`	elevated viewpoint (looking down from height)
`Aerial shot`	aerial / bird's-eye / space-like overhead view

The altitude figures in the illustration (e.g. ~10 km, ~50 km, 100 km+) are indicative only and do not define hard numeric cut-offs for the labels.

Puffin-Traj-1M (trajectory)

Each trajectory is a continuous single-axis camera motion rendered from a panorama:

Parameter	Range / Options	Notes
Motion type	`pitch` / `roll` / `yaw`	one axis varies per segment
Sub-pattern	`mono_pos`, `mono_neg`, `bi_pos_neg`, `bi_neg_pos`	monotonic (up/down, cw/ccw) or bidirectional
Roll	[−45°, +45°]	initial; held constant unless it is the moving axis
Pitch	[−45°, +45°]	initial; held constant unless it is the moving axis
Yaw	base ∈ [−180°, +180°]	swept when `yaw` is the moving axis
Vertical FoV	[60°, 100°]	uniform, constant within a trajectory
Frames / step	90 frames, 1° per frame	each segment covers ~90° of motion

Per frame, cameras.json stores roll_deg / pitch_deg / yaw_deg (degrees), the 3×3 camera_intrinsics, and the 4×4 camera_pose (camera-to-world).

Directory Structure

DATA_PATH/
├─ Puffin-Cam-15M/            # 15,338,221 single-view image-caption-camera samples
│  ├─ image_folder/           # perspective images (rendered from panoramas)
│  │  ├─ 000000.zip           # 10,000 images per shard, named by random hash
│  │  └─ ...                  # 000000.zip ... 001533.zip
│  ├─ cap_folder/             # captions: scene description + camera parameters + camera height
│  │  ├─ 000000.zip           # <hash>.json, paired with the same hash in image_folder
│  │  └─ ...
│  ├─ summary.json            # index mapping each image to its caption
├─ Puffin-Cam-15M-Bench/      # held-out benchmark: 600 image-caption-camera pairs
│  ├─ image_folder/000000.zip # 600 images, named by hash
│  └─ cap_folder/000000.zip   # 600 captions, <hash>.json paired with image_folder
├─ Puffin-Traj-1M/            # 1,136,696 camera trajectories (90-frame sequences)
│  ├─ data/
│  │  ├─ 000000.zip           # 500 scenes per shard; each scene: <hash>/000001.jpg ... 000090.jpg + cameras.json
│  │  └─ ...
│  ├─ summary.json            # index of trajectory scenes
├─ Puffin-Traj-1M-Bench/      # held-out benchmark: 100 trajectories
│  └─ data/000001.zip         # 100 scenes; each scene: <hash>/000001.jpg ... 000090.jpg + cameras.json
└─ README.md

Both summary.json files index the (hashed) samples so that images and their annotations can be located within the shard archives.

Dataset Download

You can download the entire Puffin-16M dataset using the following command:

bash

hf download KangLiao/Puffin-16M --repo-type dataset

Puffin-Cam-15M is approximately 2.4 TB, and Puffin-Traj-1M is approximately 9.2 TB (~11.6 TB in total); the two held-out benchmarks are small (~100 MB and ~1.1 GB). You may also download a single part, e.g.:

bash

hf download KangLiao/Puffin-16M --repo-type dataset --include "Puffin-Cam-15M/*"
# benchmarks only
hf download KangLiao/Puffin-16M --repo-type dataset --include "Puffin-Cam-15M-Bench/*" "Puffin-Traj-1M-Bench/*"

Pixel-wise camera maps are omitted due to their large total size, but can be generated from the provided camera parameters using ``scripts/camera/cam_dataset.py`` in our GitHub repository.

Citation

If Puffin-16M or Puffin-World supports your research, please cite our paper:

bibtex

@article{liao2026puffinworld,
  title   = {Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States},
  author  = {Liao, Kang and Luo, Yihang and Wu, Xiao-Ming and Jin, Linyi and Wu, Size and Lin, Chunyu and Zhao, Yao and Wang, Fei and Li, Wei and Loy, Chen Change},
  journal = {arXiv preprint arXiv:2609.04196},
  year    = {2026}
}

License

This project is licensed under NTU S-Lab License 1.0.