kamwoh/mini-carla-192x320-wan-2p2-vae

mini-carla-192x320-wan-2p2-vae

Wan2.2-VAE-encoded latents of a small CARLA driving dataset (192x320, native resolution, no resize). Produced for training miniworld, a minimal flow-matching world-model framework, by caching pixel clips through the frozen pretrained Wan2.2 video VAE instead of a locally-trained one.

Data size

Each latent frame covers 4 pixel frames (except the very first latent of an episode, which covers 1 — see "Clip layout" below), so 48,000 latents ≈ 191,000 pixel frames' worth of driving footage compressed to a 30x smaller footprint.

Files

• —latents.npy — (48000, 48, 12, 20) float16, per-channel normalised to zero mean / unit variance (see channel_mean / channel_std in meta.json to invert).
• —actions.npy — (48000, 12) float32. Each row is 4 pixel frames' worth of (throttle, steer, brake) concatenated in time order (one latent = 4 pixel frames = 4x3 action values), not averaged — averaging would make two different steering profiles inside one window indistinguishable.
• —ep_ids.npy — (48000,) int64. Groups latent frames into clips; a clip never straddles two ep_ids.
• —meta.json — provenance and everything needed to invert normalisation: channel_mean, channel_std, temporal_downsample (4), clip_frames (97), lat_per_clip (25), and vae_kwargs (the exact VAE config that produced this cache — models.wan_vae.WanVAE wrapping Wan-AI/Wan2.2-TI2V-5B-Diffusers's VAE, frozen, vae_checkpoint: null since nothing local was trained).

Clip layout

Encoding streams each 600-frame episode through the VAE continuously (not per-clip), so every latent keeps the causal encoder's full receptive field, then slices the resulting per-episode latent sequence into non-overlapping 25-latent groups — each becoming one clip with its own ep_id. Only the first group of an episode starts at the VAE's single-frame latent (index 0, encoded alone against replicate padding); every later group starts at a latent that already covers 4 pixel frames. All latents are still fully causal (streamed from the episode's real start), just not all "first-latent" clips begin at a literal frame 1.

Loading

Plain numpy, no extra dependencies:

import json
import numpy as np
from huggingface_hub import snapshot_download

local_dir = snapshot_download("kamwoh/mini-carla-192x320-wan-2p2-vae", repo_type="dataset")

latents = np.load(f"{local_dir}/latents.npy", mmap_mode="r")  # (48000, 48, 12, 20) float16
actions = np.load(f"{local_dir}/actions.npy")                  # (48000, 12) float32
ep_ids  = np.load(f"{local_dir}/ep_ids.npy")                    # (48000,) int64
with open(f"{local_dir}/meta.json") as f:
    meta = json.load(f)

# undo per-channel normalisation to get raw Wan2.2 latents back
mean = np.asarray(meta["channel_mean"], dtype=np.float32)
std = np.asarray(meta["channel_std"], dtype=np.float32)
raw = latents[:4].astype(np.float32) * std[None, :, None, None] + mean[None, :, None, None]

To decode back to pixels, feed raw (channels-first, time axis inserted) to models.wan_vae.WanVAE.decode_from_latent in the miniworld repo — it applies Wan's own latents_mean / latents_std before calling diffusers.AutoencoderKLWan.decode.

With miniworld's own dataset class

Drop-in for `datasets/carla_latent.py`'s CarlaLatentDataset — just point data_dir at the downloaded snapshot:

from datasets.carla_latent import CarlaLatentDataset

ds = CarlaLatentDataset(data_dir=local_dir, frames_per_clip=25)  # 25 = full 97-frame clip
item = ds[0]  # {"image": (25, 48, 12, 20), "action": (25, 12)}

frames_per_clip can be smaller (e.g. 1 for single-latent-frame training) since every clip is sliced from the same cache; see configs/experiment/carla_latent_97*.yaml in the miniworld repo for the training configs this cache was built for.

Provenance

Built with encode_latents.py from the miniworld repo:

python encode_latents.py model=wan_vae dataset.data_dir=<mini_192x320_low pixel dir> \
    out_dir=<this cache> clip_frames=97

No local VAE checkpoint — models.wan_vae.WanVAE wraps the frozen, pretrained Wan-AI/Wan2.2-TI2V-5B-Diffusers VAE (16x spatial / 4x temporal compression, 48 latent channels), used purely as an encoder here.