WitneyWW/tactile-world-model-rollouts

Tactile Diffusion-Forcing world model: inference rollouts

Ground truth vs. model prediction for tac_only_wan22_fps6act — a tactile-only Diffusion-Forcing world model over Wan2.2 latents, conditioned on 6 fps sensor actions (val_loss = 0.0354, best epoch 54). 56 rollouts, 336 clips, decoded through the Wan2.2 VAE with the same LoRA adapter used to encode the latents.

Findings

Short-horizon prediction generalizes. Held-out 44.15 ± 1.94 dB vs train 43.11 ± 3.42 dB (n = 16 each). The +1.04 dB gap is smaller than its 2×SEM of 1.97 — indistinguishable from noise. No memorization.

Long autoregressive rollout fails immediately, then plateaus. Latent error jumps to ~0.122 at the first predicted frame and grows only 1.4× across the following 14 seconds. It saturates rather than compounding — which is why the 8 s and 16 s horizons score the same (+0.07 dB, 2×SEM 0.20).

Every train-vs-test and 8 s-vs-16 s comparison is within noise at these sample sizes.

Caveat carried on the page

Long-rollout ground truth is stitched from different VAE encode windows than the rollout's seed: the seed frame matches exactly (MSE 0.0000) but the second given context frame reads 2.2–4.6e-02. That offset inflates every long-rollout error, so the ~44 → ~32 dB step overstates true failure. It does not affect the within-regime comparisons.

Each sample_XXX/ has {tl,tr}_gt.mp4, {tl,tr}_rollout.mp4 and {tl,tr}_gt_vs_rollout.mp4 for the left and right tactile sensors.