RoMALab/video2vla-robotwin15-ground-truth-latent-slice-idm

Video2VLA RoboTwin-15 Ground-Truth Latent-Slice IDM

A compact inverse-dynamics action head trained on frozen ground-truth HunyuanVideo 1.5 causal-VAE latents from 15 RoboTwin tasks. This is an action model, not a video generator.

Model

• —Input: BF16/FP32 latent tensor [B, 32, 6, H, W] (training cache used H=19, W=38).
• —Temporal layout: two context latent slices followed by four future latent slices.
• —Output: normalized joint targets [B, 24, 14].
• —Architecture: the same 4.408M-parameter spatial decoder is applied independently to every latent slice; each slice predicts four actions.
• —No proprioception input.

The training cache was created before Hunyuan posttraining and contains clean VAE encodings. This run did not use generated inference latents, a posttrained LoRA, text encoders, or online VAE encoding. Training added independent flow-style noise sampled uniformly from [0.0, 0.2]; validation used fixed noise 0.1.

Training

• —Dataset: 15-task RoboTwin randomized-500 wide corpus.
• —Usable indexed windows: 69,704 train / 8,821 validation.
• —Steps: 100,000.
• —Effective batch: 4 on one NVIDIA GH200.
• —Optimizer: AdamW, LR 3e-4, weight decay 0.05, 500 warmup steps.
• —Runtime: BF16; fused optimizer.
• —Source commit: d76f4a79a8a7f38f3e280dcc9d112d7c3592f25b.
• —Weights & Biases run

Validation

Metrics are computed on normalized joint targets. They should not be interpreted as task success rates.

Files

• —checkpoint_100000.pt: model, optimizer, scheduler, resolved config, and step; suitable for resuming with Video2VLA.
• —checkpoint_030000.pt: earlier 30k checkpoint retained for comparison.
• —model.safetensors: model state only.
• —config.yaml: source training configuration.
• —resolved_config.json: fully resolved configuration embedded in the checkpoint.
• —action_stats.json: joint-action normalization statistics.
• —training_metadata.json: provenance, metrics, and SHA-256 hashes.
• —training_code/: relevant source snapshot used by this cache-backed run.

Loading the model-only weights

from safetensors.torch import load_file
from video2vla.action_heads import LatentSliceHead

model = LatentSliceHead(
    latent_channels=32,
    action_dim=14,
    hidden_channels=128,
    spatial_pool=8,
    mlp_dim=512,
    actions_per_latent=4,
    dropout=0.05,
)
model.load_state_dict(load_file("model.safetensors"))
model.eval()

The outputs are normalized joint positions; use action_stats.json to invert z-score normalization.

Additional mixed-latent checkpoint

The repository also contains the newer 60% clean / 40% generated-latent IDM, trained for 100,000 steps. It is stored separately so checkpoint_100000.pt at the repository root continues to identify the original clean-only model.