Modelpublic

morphic/Wan2.2-frames-to-video

sourceHugging Faceapache-2.0updated 11mo agoView on Hugging Face

42likes

Model Card

Wan2.2 based Transitions and Frames-2-video

🎬 Morphic Frames to Video

High-quality video generation from image frames using Wan2.2

![GitHub](https://github.com/morphicfilms/MorphicFrames2Video.git) ![Hugging Face](https://huggingface.co/morphicmlteam/Wan2.2-I2V-A14B-frames) ![Website](https://www.morphic.com)

</div>

Video	Input Images & Prompt
<video src="https://external-cdn.morphic.com/assets-production/0193449d-bd5f-7034-a819-90186076c7d4/0191fef9-bf7e-7ddf-b6b1-20cacfb7f121/019a28d7-e369-7882-9fe0-30530497e5a8/019a28d7-e3d0-7ea3-93cf-92dbfc464b93.mp4" controls width="640"></video>	<img src="transition91.png" width="200"> → <img src="transition92.png" width="200"><br><br>Prompt: "A clown, slowly transforms into a poster."<br><br>Type: Two-frame transition
<video src="https://external-cdn.morphic.com/assets-production/0193449d-bd5f-7034-a819-90186076c7d4/0191fef9-bf7e-7ddf-b6b1-20cacfb7f121/019a28d7-e372-71f2-bde4-1ad6144c475f/019a28d7-e3cd-7060-9f39-3fa030dcd020.mp4" controls width="640"></video>	<img src="pink1.png" width="150"> → <img src="pink2.png" width="150"> → <img src="pink3.png" width="150"> → <img src="pink4.png" width="150"><br><br>Prompt: "The animated girl rises up from her chair and waves hi to the camera as the camera zooms in."<br><br>Type: Multi-frame interpolation

Setting up the repository

First clone the Morphic Interpolation repo:

git clone https://github.com/morphicfilms/frames-to-video.git

To install the environment, we recommend following the Wan2.2 installation guide.

Or you could alternatively run : bash setup_env.sh -> we recommend using the flash-attn version listed in the bash file for hassle free install.

Downloading the weights

First: download Wan2.2 I2V weights:

huggingface-cli download Wan-AI/Wan2.2-I2V-A14B --local-dir ./Wan2.2-I2V-A14B

Second : download the Morphic Frames to Video lora weights :

huggingface-cli download morphic/Wan2.2-frames-to-video --local-dir ./morphic-frames-lora-weights

Running Frames to Video

For multi node run for 2 frame interpolation :

torchrun --nproc_per_node=8 generate.py \
    --task i2v-A14B \
    --size 1280*720 \
    --frame_num 81 \
    --ckpt_dir ./Wan2.2-I2V-A14B-Interpolation \
    --high_noise_lora_weights_path ./morphic-frames-lora-weights/lora_interpolation_high_noise_final.safetensors \
    --dit_fsdp \
    --t5_fsdp \
    --ulysses_size 8 \
    --image examples/transition9_1.png \
    --prompt "Aa clown, slowly transforms into a poster." \
    --img_end examples/transition9_2.png \

For multi node run for multi frame interpolation :

torchrun --nproc_per_node=8 generate.py \
    --task i2v-A14B \
    --size 1280*720 \
    --frame_num 81 \
    --ckpt_dir ./Wan2.2-I2V-A14B-Interpolation \
    --high_noise_lora_weights_path ./morphic-frames-lora-weights/lora_interpolation_high_noise_final.safetensors \
    --dit_fsdp \
    --t5_fsdp \
    --ulysses_size 8 \
    --image examples/pink_1.png \
    --prompt "The animated girl rises up from her chair and waves hi to the camera as the camer zooms in." \
    --img_end examples/pink_4.png \
    --middle_images examples/pink_2.png examples/pink_3.png \
    --middle_images_timestamps 0.4 0.7

Note:

1.--middleimagestimestamps : should be used if multiple intermediate frames are provided, the numbers indicate the the location the intermediate frame is provided (0.5 -> midway, 0.33, 0.66 -> 2 equally spaced intermediate frames, 0.25, 0.5, 0.75 -> 3 equally spaced intermediate frames)
2.Number of middleimages must be equal to number of middleimages_timestamps