Modelpublic

hmkang/wam_wan22_dit4dit_robocasa300_hwabl_3latin

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes

Model Card

wamwan22dit4ditrobocasa300hwabl_3latin

Video-only continuation of huiwon's 4knob-A recipe, run as a history-length ablation: how much does the clean-latent context length matter?


backbone	Wan2.2-TI2V-5B (`wan22_diffusers`)
context	3 latent slots (9 stride-2 pixel frames, 288 tokens)
prediction	2 latent slots (192 tokens)
`num_frames_in` / `num_frames_out`	17 / 33
`training_mode`	`video`
block-causal	no
global batch	64 (8 GPU x per-device 8 x GA 1)
schedule	`WAN_FM_TRAIN_SHIFT=5.0`, bell time weighting, wd 0.01, lr 1e-4, min-lr ratio 0.01
other	caption dropout 0.1, image aug, zero text pad, video EMA (`warmup`) saved

Every checkpoint carries both the live weights (backbone.extractor.transformer.*) and the EMA copy (backbone.extractor._video_ema_model.*, 825 tensors). All numbers below are measured on the EMA weights.

Sibling arms

The arms share every setting except the context length, so they are directly comparable at equal step count. Note that 6latin reaches the same global batch 64 with 4 GPUs x GA 2 instead of 8 GPUs x GA 1.

arm	context	`num_frames_in`/`out`	cond pixel frames	cond tokens
`hwhist5` (2latin base)	2 latent	9 / 25	5	192
`hwabl_3latin`	3 latent	17 / 33	9	288
`hwabl_4latin`	4 latent	25 / 41	13	384
`hwabl_6latin`	6 latent	41 / 57	21	576
`hwabl_8latin`	8 latent	57 / 73	29	768

2latin is huiwon's original run, not one of these arms: it trains with training_mode: joint rather than video. It is included as the starting point of the ladder, not as a like-for-like fifth arm. A 5-latent arm was also trained under the same recipe but stopped before 100k steps, so it is not part of this comparison.

Video metrics (1-step, N=2048 robocasa val windows, EMA)

checkpoint	PSNR	SSIM	LPIPS	FVD
`checkpoint-20000`	20.822	0.7488	0.2459	139.06
`checkpoint-40000`	21.116	0.7547	0.2332	126.94
`checkpoint-60000`	21.374	0.7601	0.2185	114.90
`checkpoint-80000`	21.612	0.7652	0.2040	102.08
`checkpoint-100000`	21.681	0.7675	0.1921	89.60

Reference, same protocol: the 2-latent base run scores 20.772 / 0.7461 / 0.2458 / 142.56 at 20k and 21.513 / 0.7628 / 0.1968 / 94.48 at 100k.

Attentive probing (frozen features, 15-way primitive classification)

A single attentive-probe head is trained on frozen features from the checkpoint (condition = step1, N = 20,964 validation samples, 5 epochs, best epoch reported).

checkpoint	best epoch	accuracy	macro recall
`checkpoint-20000`	e3	0.6722	0.5804
`checkpoint-40000`	e3	0.6886	0.6031
`checkpoint-60000`	e2	0.7084	0.6266
`checkpoint-80000`	e1	0.7152	0.6233
`checkpoint-100000`	e2	0.7237	0.6323

Cross-arm comparison

FVD (lower is better) and probing accuracy (higher is better) at equal step count. * marks a probing run that has not finished its 5 epochs yet.

step	2latin FVD	3latin FVD	4latin FVD	6latin FVD	8latin FVD	2latin acc	3latin acc	4latin acc	6latin acc	8latin acc
20k	—	139.06	140.45	138.89	139.19	—	0.6722	0.6681	0.6806	0.6781
40k	—	126.94	126.97	126.88	126.99	—	0.6886	0.6901	0.7023	0.7057
60k	—	114.90	114.10	115.13	115.32	—	0.7084	0.7069	0.7162	0.7169
80k	—	102.08	97.55	99.53	99.96	—	0.7152	0.7254	0.7248	0.7276
100k	—	89.60	86.26	—	88.34	—	0.7237	0.7282	—	0.7318

The two metrics disagree, consistently. On probing, the longer-context arms lead at every step that has been measured - 8latin is best at 40k, 60k, 80k and 100k, 6latin at 20k - and the gap over 3latin/4latin is roughly 1 to 1.5 points throughout. On FVD the same arms do not win: at 20k-60k all four sit within about 1 point of each other (40k spans 126.88-126.99, a 0.11 range), and once FVD does spread at 80k and 100k it is 4latin that leads, with 8latin second and 3latin last - no monotone relation to context length.

So a longer clean-latent context buys representation quality that a frozen probe can read out, without buying 1-step generation quality on this val set. Read probing and FVD as answers to different questions rather than as two views of the same ranking.

Machine-readable copies of every number above, for all four arms, are in `metrics/eval_metrics.json`.

Provenance and caveats

Derived from huiwon/wam_wan22_dit4dit_robocasa300_b64_4knobA_hist5 (run isaac_wan22_joint_b64_4knobA_plus_hist5in_emasaveW_actemaW_tdrop01_pd8ga1, 2 latent context). These arms are not a reproduction of it - the one deliberate change is training_mode: joint -> video, since the action flow-matching loss never reaches the video DiT in either mode. Everything else in the recipe is carried over unchanged.

Note when comparing against that base run at equal step count: these arms train at global batch 64 from step 0, so 100k steps here is 6.4M samples.

Optimizer state is not included - these are inference/eval checkpoints.