allenai/molmomotion

MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction

MolmoMotion is a 4B vision-language model that forecasts 3D point trajectories under natural-language action instructions. Given a short RGB observation history, a set of user-specified 2D query points with their initial 3D positions, and a language description of the intended action, the model predicts each query point's 3D trajectory in camera-frame coordinates.

Demo

This Space uses the allenai/MolmoMotion-4B-H3-F30 checkpoint (history H=3, future horizon F=30, predicting ~2 seconds at 15 fps).

Upload your own short video + a 3D-points bundle (.npz with points_2d, points_3d_history, intrinsics_K — see the in-app format note, and the data-generation pipeline for how to produce 3D point tracks from raw video), or pick one of the bundled gr.Examples clips below the inputs. Click "Preview Input" to sanity-check the t₀ frame + query points before running inference, enter an action instruction, and click "Predict Trajectory" to see the model's predicted 3D point trajectories projected onto the t₀ frame as an animated video.

• —Paper: arXiv:2606.18558
• —Code: github.com/allenai/molmo-motion
• —Models: MolmoMotion collection