Modelpublic

unitreerobotics/UnifoLM-ER-Flow

sourceHugging Faceapache-2.0updated 11d agoView on Hugging Face

12likes519downloads

Model Card

UnifoLM-ER-Flow

UnifoLM-ER-Flow extends UnifoLM-ER-1 with interaction-centric world modeling and discrete action learning. It jointly aligns visual observations, language conditions, predicted future dynamic regions, and robot actions within a single vision-language model.

Dynamic Region Prediction

We use optical flow to extract dynamic regions that capture future scene changes, then train a VQ-VAE to encode them into fixed-length sequences of discrete tokens. Conditioned on the current image and a task description or action, the VLM directly predicts mask tokens for future dynamic regions, focusing on interaction subjects and the scene changes they induce to enable interaction-centric world modeling.

[image]

Demonstrations

<table> <tr> <td width="50%" align="center"> <video src="assets/videos/flow-demo-web-2x/cleantable.mp4" controls muted loop playsinline width="100%"></video> 01 · Clean table </td> <td width="50%" align="center"> <video src="assets/videos/flow-demo-web-2x/foldtowel.mp4" controls muted loop playsinline width="100%"></video> 02 · Fold towel </td> </tr> <tr> <td width="50%" align="center"> <video src="assets/videos/flow-demo-web-2x/installphone.mp4" controls muted loop playsinline width="100%"></video> 03 · Packaging phone </td> <td width="50%" align="center"> <video src="assets/videos/flow-demo-web-2x/placeplates.mp4" controls muted loop playsinline width="100%"></video> 04 · Place plates </td> </tr> </table>

Discrete Action Learning

We partition the unified action space into three components: end-effector (EEF) poses, end-effector joints, and lower-body joints. A separate residual vector quantization (RVQ) model is trained for each component to discretize action sequences. Building on UnifoLM-ER-1, we introduce discrete action tokens and mask tokens for future dynamic regions, jointly aligning visual, language, and action representations within a single VLM to obtain the UnifoLM-ER-Flow model.

Action Encoding Pipeline

Each continuous motion component is encoded independently with its own RVQ model. The resulting token sequences share the same timesteps and are synchronously fed into the VLM.

[image]

Unified Training Signals

Signal	Representation	Role
Visual observations	Image tokens	Describe the current environment
Task or action condition	Language or action tokens	Specify the intended interaction
Future dynamic regions	Mask tokens	Represent interaction-induced scene changes
EEF poses	Discrete EEF tokens	Encode end-effector trajectories
End-effector joints	Discrete hand tokens	Encode gripper or dexterous-hand motion
Lower-body joints	Discrete lower-body tokens	Encode whole-body coordination

Together, these signals connect spatial understanding, future-change prediction, and action representation in a unified multimodal model.