WitneyWW/visuotactile-mot-world-model-rollouts

Visuo-tactile Mixture-of-Transformers world model: inference rollouts

Ground-truth vs. rollout videos and metrics for mot_df_vt_actionflow — a two-expert Mixture-of-Transformers diffusion-forcing world model over Wan2.2 VAE latents.

The model

A VideoDiT over the middle camera latent and a TactileDiT over both GelSight sensors, fused by MoT joint mixed attention (256 + 512 = 768 tokens). What distinguishes it:

• —Per-modality action conditioning. The 36-D action splits by observation space — the video expert cross-attends action[0:18], the sensors' positions projected into the camera image (absolute u, v, z); the tactile expert cross-attends action[18:36], the per-sensor body-frame SE(3) step (dp = Rᵀ(p₁−p₀), dR = R₀ᵀR₁ → Rot6D).
• —Cross-attention before fusion. Each expert absorbs its own action context first; only then are the tokens concatenated for the single joint self-attention. Tokens that mix across modalities are already action-conditioned.
• —Independent noise schedulers. Diffusion forcing with per-frame noise levels drawn separately for the visual and tactile branches, so the two sit at unrelated points of their schedules.
• —Temporally causal joint mask. Camera frame t attends tactile frames ≤ t and vice versa.
• —Actionflow prior on the tactile branch. A frozen action-motion encoder reads the inverse-action displacement map on the gel plane plus a calibrated physics flow prior, and injects a zero-initialised motion feature into the future tactile tokens. A frozen latent-flow probe scores the predicted clean latents against measured AllTracker optical flow.

240.8M trainable parameters, 100 epochs on 2×H100.

What is here

Four cells — {short window, 16 s long rollout} × {test, train} — six rollouts each (two episodes × three start offsets).

Rows are the same (episode, start) pairs as the view_tactile_wan22_action36 Space, so the two are directly comparable; the page tabulates the delta.

Reading the numbers

• —PSNR rewards blur. A model predicting a smooth mean-ish gel image scores well. Watch the clips alongside the table.
• —n = 6 per cell. The sem is indicative, not a significance test.
• —*Held-out episode, not held-out condition.* Test episodes share board and session with training episodes, so this measures "unseen trajectory, seen conditions".
• —*`last.ckpt`, not `best-`.** This model's validation loss is ~56% the frozen-flow-probe term, which oscillates and does not track denoising quality; a fixed step count is also the fairer comparison against the baseline.