Datasetpublic

Haosonnn/wan22-incontext-control-data

Wan2.2 In-Context Control — derived training data Derived metadata/pose NPZs for the in-context camera + audio control fork of DiffSynth-Studio (training Wan2.2-TI2V-5B). This repo holds only the small derived files needed to reproduce the camera and audio (e11h) runs. It does not rehost source videos — those come from the original datasets linked below. Companion code: the DiffSynth-Studio fork (see its README for the full reproduction walkthrough). Files… See the full description on the dataset page: https://huggingface.co/datasets/Haosonnn/wan22-incontext-control-data.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes36downloads

Dataset Card

Wan2.2 In-Context Control — derived training data

Derived metadata/pose NPZs for the in-context camera + audio control fork of DiffSynth-Studio (training Wan2.2-TI2V-5B). This repo holds only the small derived files needed to reproduce the camera and audio (e11h) runs. It does not rehost source videos — those come from the original datasets linked below.

Companion code: the DiffSynth-Studio fork (see its README for the full reproduction walkthrough).

Files

File	Size	Used by
`orbit_lr_first49_thr0.97.npz`	~102 MB	Camera training (orbit-left/right subset, 5881 clips)
`train_data_openhumanvid_megasam_001-030_w2c.npz`	~163 MB	Audio (e11h) + joint training (OpenHumanVid, 8,989 clips)
`sa5b_200subset/metadata.csv`	—	e11h 200-clip subset (long captions)
`sa5b_200subset/metadata_generic.csv`	—	e11h 200-clip subset (generic prompt — the run used this)

Download (huggingface_hub 1.x CLI is hf):

bash

hf download Haosonnn/wan22-incontext-control-data --repo-type dataset --local-dir data/

This lands data/orbit_lr_first49_thr0.97.npz, data/train_data_openhumanvid_megasam_001-030_w2c.npz, and data/sa5b_200subset/*.csv.

Changed 2026-08-06 — MonST3R poses replaced by MegaSaM. train_data_openhumanvid_monst3r_001-040.npz (32,176 clips) has been removed and superseded by train_data_openhumanvid_megasam_001-030_w2c.npz (8,989 clips). This is not a drop-in swap — read Migrating from the MonST3R NPZ before pointing a training run at it.

NPZ schema

Both NPZs are a NumPy object array under key arr_0 — a Python list of per-clip dicts.

`orbit_lr_first49_thr0.97.npz` (camera; derived from RealCam-Vid):

key	shape / type	meaning
`dataset_source`	str	`RealEstate10K` / `DL3DV-10K` / `MiraData9K`
`video_path`	str	path relative to the RealCam-Vid video root
`short_caption`, `long_caption`	str	captions
`camera_intrinsics`	(4,)	normalized `fx, fy, cx, cy`
`camera_extrinsics`	(F,4,4)	world-to-camera; full length, not truncated
`align_factor`	float	per-clip translation scale
`camera_scale`, `vtss_score`	float	scene scale / quality score

5881 clips = orbit-left ∪ orbit-right, selected by motion-cosine ≥ 0.97 among clips with ≥ 49 frames ("first49" is the ≥49-frame selection filter — extrinsics are kept at full length).

`train_data_openhumanvid_megasam_001-030_w2c.npz` (audio / joint; OpenHumanVid + MegaSaM poses):

key	shape / type	meaning
`video_path`, `audio_path`	str	same mp4 (audio is in-stream), relative to the OpenHumanVid root
`short_caption`, `long_caption`	str	captions
`camera_intrinsics`	(4,)	normalized `fx, fy, cx, cy`
`camera_extrinsics`	(T,4,4)	keyframe world-to-camera, metric (metres); T varies per clip (72–130, median 106)
`camera_extrinsics_kf_inds`	(T,)	keyframe frame indices → interpolate to sampled frames
`align_factor`	float	`1.0` — MegaSaM trajectories are already metric, no rescale needed
`face_bbox`, `lip_bbox`	(4,)	pixel coords `x0,y0,x1,y1` → normalize by `video_width/height`
`face_conf`, `lip_conf`	float	detector confidence
`face_bbox_seq`, `lip_bbox_seq`	(T,4)	per-frame talking-speaker boxes, pixel coords, index-aligned with `camera_extrinsics`
`conf_seq`	(T,)	per-frame detector confidence
`match_seq`	(T,) int8	`1` = frame matched the TalkNet speaker, `0` = no match (boxes are blank/zero)
`camera_scale`, `video_width`, `video_height`	—	scene scale / resolution

8,989 clips over parts 001–030, 935,004 keyframes, of which 850,238 (91.0 %) carry a TalkNet-speaker match. Poses come from MegaSaM (DepthAnything → UniDepth → DROID-SLAM) with whole-person masking, and are metric because UniDepth anchors the scale. camera_extrinsics[0] ≈ I (the first camera is anchored at the origin, up to BA refinement — median deviation 5e-4).

Migrating from the MonST3R NPZ

The MegaSaM file is a strict superset of the retired MonST3R schema — every key the old file had is still present with the same name, dtype and units, plus the four *_seq per-frame keys. Code that read the MonST3R NPZ will load this one unchanged. What differs materially:

	MonST3R `001-040` (removed)	MegaSaM `001-030_w2c`
clips	32,176	8,989
parts	001–040	001–030
`camera_extrinsics` length	12–30 (median ~20)	72–130 (median 106)
pose convention	world-to-camera	world-to-camera (unchanged)
scale	non-metric	metric (metres)
`align_factor`	`0.0` (unused)	`1.0`
per-frame face/lip	✗	✓ `face_bbox_seq` / `lip_bbox_seq` / `match_seq`

—Clip count drops 72 %. The MegaSaM set is filtered to clips with real, well-conditioned camera motion (static clips and rough/incoherent trajectories are dropped); the MonST3R pack was not filtered this way. Expect fewer steps per epoch.
—Sequence length grows ~5×. Anything that hard-codes 20 keyframes, or preallocates on that assumption, needs to handle variable T. camera_extrinsics_kf_inds remains the mapping to source frames.
—`align_factor` changes from `0.0` to `1.0`. Code that multiplied translations by it produced zeros before and will now pass them through unchanged — check whether it was special-cased.
—Parts 031–040 are not yet included. They are being reconstructed with MegaSaM and can be appended later.

⚠️ `sa5b_200subset` CSVs are stale with respect to this NPZ

The sample column is a row index into the OpenHumanVid NPZ, and those indices were assigned against the retired MonST3R pack. They are not valid for the MegaSaM pack: the indices (0–209) all still fall inside its 8,989 rows, so a training run will not error — it will silently train on the wrong clips. Only 47 of the 200 referenced clips exist in the MegaSaM set at all.

Use these CSVs only with the retired MonST3R NPZ (recoverable from this repo's git history), or regenerate the subset against the MegaSaM pack, preserving video → row-index correspondence rather than reusing the old sample numbers. metadata_generic.csv (the one the e11h run used) replaces captions with a fixed generic prompt.

Provenance & licenses