WitneyWW/tactile-wm-actionflow-rollouts

Tactile Diffusion-Forcing world model with action-motion conditioning: inference rollouts

Ground truth vs. model prediction for tac_only_wan22_fps6act_actionflow — a tactile-only Diffusion-Forcing world model over Wan2.2 latents, conditioned on 6 fps sensor actions plus a frozen action-motion encoder (inverse-action maps + physics-flow priors injected into the DiT per token group). Checkpoint best-epoch=35-val_loss=0.2080 (val loss includes motion/pred-flow auxiliary terms — not comparable to runs without them). 30 rollouts, 180 clips, decoded through the Wan2.2 VAE with the same LoRA adapter used to encode the latents.

Results

Short-horizon prediction generalizes. Held-out 38.28 ± 1.88 dB vs train 40.40 ± 2.10 dB (n = 5 each): the −2.12 dB gap is smaller than its 2×SEM of 2.52 — within noise, no memorization signal.

Long autoregressive rollout fails immediately, then plateaus. Latent error jumps to ~0.11 at the first predicted frame and grows only ~1.5× over the following 14 seconds; 8 s and 16 s score the same on test (31.26 vs 31.04 dB).

Notes. The long-rollout train cells cover five distinct episodes across four capture sessions; the 0519 episode is a consistent outlier (~21–26 dB left sensor) and widens the train sd. The inference scripts were patched for this run to rebuild the action-motion encoder from the checkpoint and feed the per-window action-map / physics-flow tensors into sampling — without the patch the trained spatial conditioning is silently dropped. Long-rollout ground truth is stitched from different VAE encode windows than the seed (second context frame reads ~5e-02 MSE), which inflates long-rollout error relative to the short-window cells.

Each sample_XXX/ has {tl,tr}_gt.mp4, {tl,tr}_rollout.mp4 and {tl,tr}_gt_vs_rollout.mp4 for the left and right tactile sensors.