Salesforce/FOFPred

1---2license: apache-2.03library_name: diffusers4pipeline_tag: image-to-image5tags:6- optical-flow prediction7- motion prediction8- diffusion9---10 11# FOFPred: Language-Driven Future Optical Flow Prediction12 13 14[Paper](https://arxiv.org/abs/2601.10781) | [Project Site](https://fofpred.github.io) | [Demo](https://fofpred.salesforceresearch.ai)15 16**FOFPred** is a diffusion-based model that predicts future optical flow from a single image guided by natural language instructions. Given an input image and a text prompt describing a desired action (e.g., *"Moving the water bottle from right to left"*), FOFPred generates 4 sequential optical flow frames showing how objects would move.17 18## Usage19 20```python21import einops22import numpy as np23import torch24from diffusers import DiffusionPipeline25from PIL import Image26 27# Load pipeline with trust_remote_code28pipeline = DiffusionPipeline.from_pretrained(29    "Salesforce/FOFPred",30    torch_dtype=torch.bfloat16,31    trust_remote_code=True,32).to("cuda")33 34# Run inference35results = pipeline(36    prompt="Moving the water bottle from right to left.",37    input_images=[Image.open("your_image.jpg")],38    width=256,39    height=256,40    num_inference_steps=1,41    num_images_per_prompt=4,42    frame_count=4,43    generator=torch.Generator(device="cuda").manual_seed(42),44    output_type="pt",45)46 47flow_frames = results.images  # [B, F, C, H, W]48 49output_tensor = flow_frames[0]  # [F, C, H, W]50output_np = pipeline.image_processor.pt_to_numpy(output_tensor)  # [F, H, W, C]51reshaped = einops.rearrange(output_np, "f h w c -> h (f w) c")52img = Image.fromarray((reshaped * 255).astype(np.uint8))53img.save("output_combined.png")54```55 56## Architecture57 58| Component | Model |59|-----------|-------|60| **V-LLM** | Qwen2.5-VL-3B-Instruct |61| **DiT** | OmniGen2Transformer3DModel |62| **VAE** | FLUX.1-dev AutoencoderKL |63| **Scheduler** | FlowMatchEulerDiscreteScheduler |64 65## Acknowledgements66 67- [OmniGen2](https://github.com/VectorSpaceLab/OmniGen2)68- [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL)69- [Flux VAE](https://huggingface.co/black-forest-labs/FLUX.1-dev)70 71## License72 73Our code and weights are released under the [CC by-NC 4.0 license](https://creativecommons.org/licenses/by-nc/4.0/deed.en).74