Lightricks/LTX-Video-0.9.7-dev
247.4k
1---2tags:3- ltx-video4- image-to-video5pinned: true6language:7- en8license: other9pipeline_tag: text-to-video10library_name: diffusers11base_model: Lightricks/LTX-Video12---13 14# LTX-Video 0.9.7 Model Card15This model card focuses on the model associated with the LTX-Video model, codebase available [here](https://github.com/Lightricks/LTX-Video).16 17LTX-Video is the first DiT-based video generation model capable of generating high-quality videos in real-time. It produces 30 FPS videos at a 1216×704 resolution faster than they can be watched. Trained on a large-scale dataset of diverse videos, the model generates high-resolution videos with realistic and varied content.18We provide a model for both text-to-video as well as image+text-to-video usecases19 20<img src="./media/trailer.gif" alt="trailer" width="512">21 22 23| | | | |24|:---:|:---:|:---:|:---:|25| <br><details style="max-width: 300px; margin: auto;"><summary>A woman with long brown hair and light skin smiles at another woman...</summary>A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage.</details> | <br><details style="max-width: 300px; margin: auto;"><summary>A woman walks away from a white Jeep parked on a city street at night...</summary>A woman walks away from a white Jeep parked on a city street at night, then ascends a staircase and knocks on a door. The woman, wearing a dark jacket and jeans, walks away from the Jeep parked on the left side of the street, her back to the camera; she walks at a steady pace, her arms swinging slightly by her sides; the street is dimly lit, with streetlights casting pools of light on the wet pavement; a man in a dark jacket and jeans walks past the Jeep in the opposite direction; the camera follows the woman from behind as she walks up a set of stairs towards a building with a green door; she reaches the top of the stairs and turns left, continuing to walk towards the building; she reaches the door and knocks on it with her right hand; the camera remains stationary, focused on the doorway; the scene is captured in real-life footage.</details> | <br><details style="max-width: 300px; margin: auto;"><summary>A woman with blonde hair styled up, wearing a black dress...</summary>A woman with blonde hair styled up, wearing a black dress with sequins and pearl earrings, looks down with a sad expression on her face. The camera remains stationary, focused on the woman's face. The lighting is dim, casting soft shadows on her face. The scene appears to be from a movie or TV show.</details> | <br><details style="max-width: 300px; margin: auto;"><summary>The camera pans over a snow-covered mountain range...</summary>The camera pans over a snow-covered mountain range, revealing a vast expanse of snow-capped peaks and valleys.The mountains are covered in a thick layer of snow, with some areas appearing almost white while others have a slightly darker, almost grayish hue. The peaks are jagged and irregular, with some rising sharply into the sky while others are more rounded. The valleys are deep and narrow, with steep slopes that are also covered in snow. The trees in the foreground are mostly bare, with only a few leaves remaining on their branches. The sky is overcast, with thick clouds obscuring the sun. The overall impression is one of peace and tranquility, with the snow-covered mountains standing as a testament to the power and beauty of nature.</details> |26| <br><details style="max-width: 300px; margin: auto;"><summary>A woman with light skin, wearing a blue jacket and a black hat...</summary>A woman with light skin, wearing a blue jacket and a black hat with a veil, looks down and to her right, then back up as she speaks; she has brown hair styled in an updo, light brown eyebrows, and is wearing a white collared shirt under her jacket; the camera remains stationary on her face as she speaks; the background is out of focus, but shows trees and people in period clothing; the scene is captured in real-life footage.</details> | <br><details style="max-width: 300px; margin: auto;"><summary>A man in a dimly lit room talks on a vintage telephone...</summary>A man in a dimly lit room talks on a vintage telephone, hangs up, and looks down with a sad expression. He holds the black rotary phone to his right ear with his right hand, his left hand holding a rocks glass with amber liquid. He wears a brown suit jacket over a white shirt, and a gold ring on his left ring finger. His short hair is neatly combed, and he has light skin with visible wrinkles around his eyes. The camera remains stationary, focused on his face and upper body. The room is dark, lit only by a warm light source off-screen to the left, casting shadows on the wall behind him. The scene appears to be from a movie.</details> | <br><details style="max-width: 300px; margin: auto;"><summary>A prison guard unlocks and opens a cell door...</summary>A prison guard unlocks and opens a cell door to reveal a young man sitting at a table with a woman. The guard, wearing a dark blue uniform with a badge on his left chest, unlocks the cell door with a key held in his right hand and pulls it open; he has short brown hair, light skin, and a neutral expression. The young man, wearing a black and white striped shirt, sits at a table covered with a white tablecloth, facing the woman; he has short brown hair, light skin, and a neutral expression. The woman, wearing a dark blue shirt, sits opposite the young man, her face turned towards him; she has short blonde hair and light skin. The camera remains stationary, capturing the scene from a medium distance, positioned slightly to the right of the guard. The room is dimly lit, with a single light fixture illuminating the table and the two figures. The walls are made of large, grey concrete blocks, and a metal door is visible in the background. The scene is captured in real-life footage.</details> | <br><details style="max-width: 300px; margin: auto;"><summary>A woman with blood on her face and a white tank top...</summary>A woman with blood on her face and a white tank top looks down and to her right, then back up as she speaks. She has dark hair pulled back, light skin, and her face and chest are covered in blood. The camera angle is a close-up, focused on the woman's face and upper torso. The lighting is dim and blue-toned, creating a somber and intense atmosphere. The scene appears to be from a movie or TV show.</details> |27| <br><details style="max-width: 300px; margin: auto;"><summary>A man with graying hair, a beard, and a gray shirt...</summary>A man with graying hair, a beard, and a gray shirt looks down and to his right, then turns his head to the left. The camera angle is a close-up, focused on the man's face. The lighting is dim, with a greenish tint. The scene appears to be real-life footage. Step</details> | <br><details style="max-width: 300px; margin: auto;"><summary>A clear, turquoise river flows through a rocky canyon...</summary>A clear, turquoise river flows through a rocky canyon, cascading over a small waterfall and forming a pool of water at the bottom.The river is the main focus of the scene, with its clear water reflecting the surrounding trees and rocks. The canyon walls are steep and rocky, with some vegetation growing on them. The trees are mostly pine trees, with their green needles contrasting with the brown and gray rocks. The overall tone of the scene is one of peace and tranquility.</details> | <br><details style="max-width: 300px; margin: auto;"><summary>A man in a suit enters a room and speaks to two women...</summary>A man in a suit enters a room and speaks to two women sitting on a couch. The man, wearing a dark suit with a gold tie, enters the room from the left and walks towards the center of the frame. He has short gray hair, light skin, and a serious expression. He places his right hand on the back of a chair as he approaches the couch. Two women are seated on a light-colored couch in the background. The woman on the left wears a light blue sweater and has short blonde hair. The woman on the right wears a white sweater and has short blonde hair. The camera remains stationary, focusing on the man as he enters the room. The room is brightly lit, with warm tones reflecting off the walls and furniture. The scene appears to be from a film or television show.</details> | <br><details style="max-width: 300px; margin: auto;"><summary>The waves crash against the jagged rocks of the shoreline...</summary>The waves crash against the jagged rocks of the shoreline, sending spray high into the air.The rocks are a dark gray color, with sharp edges and deep crevices. The water is a clear blue-green, with white foam where the waves break against the rocks. The sky is a light gray, with a few white clouds dotting the horizon.</details> |28| <br><details style="max-width: 300px; margin: auto;"><summary>The camera pans across a cityscape of tall buildings...</summary>The camera pans across a cityscape of tall buildings with a circular building in the center. The camera moves from left to right, showing the tops of the buildings and the circular building in the center. The buildings are various shades of gray and white, and the circular building has a green roof. The camera angle is high, looking down at the city. The lighting is bright, with the sun shining from the upper left, casting shadows from the buildings. The scene is computer-generated imagery.</details> | <br><details style="max-width: 300px; margin: auto;"><summary>A man walks towards a window, looks out, and then turns around...</summary>A man walks towards a window, looks out, and then turns around. He has short, dark hair, dark skin, and is wearing a brown coat over a red and gray scarf. He walks from left to right towards a window, his gaze fixed on something outside. The camera follows him from behind at a medium distance. The room is brightly lit, with white walls and a large window covered by a white curtain. As he approaches the window, he turns his head slightly to the left, then back to the right. He then turns his entire body to the right, facing the window. The camera remains stationary as he stands in front of the window. The scene is captured in real-life footage.</details> | <br><details style="max-width: 300px; margin: auto;"><summary>Two police officers in dark blue uniforms and matching hats...</summary>Two police officers in dark blue uniforms and matching hats enter a dimly lit room through a doorway on the left side of the frame. The first officer, with short brown hair and a mustache, steps inside first, followed by his partner, who has a shaved head and a goatee. Both officers have serious expressions and maintain a steady pace as they move deeper into the room. The camera remains stationary, capturing them from a slightly low angle as they enter. The room has exposed brick walls and a corrugated metal ceiling, with a barred window visible in the background. The lighting is low-key, casting shadows on the officers' faces and emphasizing the grim atmosphere. The scene appears to be from a film or television show.</details> | <br><details style="max-width: 300px; margin: auto;"><summary>A woman with short brown hair, wearing a maroon sleeveless top...</summary>A woman with short brown hair, wearing a maroon sleeveless top and a silver necklace, walks through a room while talking, then a woman with pink hair and a white shirt appears in the doorway and yells. The first woman walks from left to right, her expression serious; she has light skin and her eyebrows are slightly furrowed. The second woman stands in the doorway, her mouth open in a yell; she has light skin and her eyes are wide. The room is dimly lit, with a bookshelf visible in the background. The camera follows the first woman as she walks, then cuts to a close-up of the second woman's face. The scene is captured in real-life footage.</details> |29 30# Models31 32| Name | Notes | inference.py config | ComfyUI workflow (Recommended) |33|----------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|34| ltxv-13b-0.9.7-dev | Highest quality, requires more VRAM | [ltxv-13b-0.9.7-dev.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.7-dev.yaml) | [ltxv-13b-i2v-base.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/ltxv-13b-i2v-base.json) |35| [ltxv-13b-0.9.7-mix](https://app.ltx.studio/motion-workspace?videoModel=ltxv-13b) | Mix ltxv-13b-dev and ltxv-13b-distilled in the same multi-scale rendering workflow for balanced speed-quality | N/A | [ltxv-13b-i2v-mix.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/ltxv13b-i2v-mixed-multiscale.json) |36| [ltxv-13b-0.9.7-distilled](https://app.ltx.studio/motion-workspace?videoModel=ltxv) | Faster, less VRAM usage, slight quality reduction compared to 13b. Ideal for rapid iterations | [ltxv-13b-0.9.7-distilled.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.7-dev.yaml) | [ltxv-13b-dist-i2v-base.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/13b-distilled/ltxv-13b-dist-i2v-base.json) |37| [ltxv-13b-0.9.7-distilled-lora128](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-13b-0.9.7-distilled-lora128.safetensors) | LoRA to make ltxv-13b-dev behave like the distilled model | N/A | N/A |38| ltxv-13b-0.9.7-fp8 | Quantized version of ltxv-13b | Coming soon | [ltxv-13b-i2v-base-fp8.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/ltxv-13b-i2v-base-fp8.json) |39| ltxv-13b-0.9.7-distilled-fp8 | Quantized version of ltxv-13b-distilled | Coming soon | [ltxv-13b-dist-fp8-i2v-base.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/13b-distilled/ltxv-13b-dist-fp8-i2v-base.json) |40| ltxv-2b-0.9.6 | Good quality, lower VRAM requirement than ltxv-13b | [ltxv-2b-0.9.6-dev.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-2b-0.9.6-dev.yaml) | [ltxvideo-i2v.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/low_level/ltxvideo-i2v.json) |41| ltxv-2b-0.9.6-distilled | 15× faster, real-time capable, fewer steps needed, no STG/CFG required | [ltxv-2b-0.9.6-distilled.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-2b-0.9.6-distilled.yaml) | [ltxvideo-i2v-distilled.json](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/low_level/ltxvideo-i2v-distilled.json) |42 43 44## Model Details45- **Developed by:** Lightricks46- **Model type:** Diffusion-based text-to-video and image-to-video generation model47- **Language(s):** English48 49 50## Usage51 52### Direct use53You can use the model for purposes under the license:54- 2B version 0.9: [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltx-video-2b-v0.9.license.txt)55- 2B version 0.9.1 [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltx-video-2b-v0.9.1.license.txt)56- 2B version 0.9.5 [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltx-video-2b-v0.9.5.license.txt)57- 2B version 0.9.6-dev [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/LTX-Video-Open-Weights-License-0.X.txt)58- 2B version 0.9.6-distilled [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/LTX-Video-Open-Weights-License-0.X.txt)59- 13B version 0.9.7-dev [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/LTX-Video-Open-Weights-License-0.X.txt)60- 13B version 0.9.7-dev-fp8 [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/LTX-Video-Open-Weights-License-0.X.txt)61- 13B version 0.9.7-distilled [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/LTX-Video-Open-Weights-License-0.X.txt)62- 13B version 0.9.7-distilled-fp8 [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/LTX-Video-Open-Weights-License-0.X.txt)63- 13B version 0.9.7-distilled-lora128 [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/LTX-Video-Open-Weights-License-0.X.txt)64- Temporal upscaler version 0.9.7 [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/LTX-Video-Open-Weights-License-0.X.txt)65- Spatial upscaler version 0.9.7 [license](https://huggingface.co/Lightricks/LTX-Video/blob/main/LTX-Video-Open-Weights-License-0.X.txt)66 67 68### General tips:69* The model works on resolutions that are divisible by 32 and number of frames that are divisible by 8 + 1 (e.g. 257). In case the resolution or number of frames are not divisible by 32 or 8 + 1, the input will be padded with -1 and then cropped to the desired resolution and number of frames.70* The model works best on resolutions under 720 x 1280 and number of frames below 257.71* Prompts should be in English. The more elaborate the better. Good prompt looks like `The turquoise waves crash against the dark, jagged rocks of the shore, sending white foam spraying into the air. The scene is dominated by the stark contrast between the bright blue water and the dark, almost black rocks. The water is a clear, turquoise color, and the waves are capped with white foam. The rocks are dark and jagged, and they are covered in patches of green moss. The shore is lined with lush green vegetation, including trees and bushes. In the background, there are rolling hills covered in dense forest. The sky is cloudy, and the light is dim.`72 73### Online demo74The model is accessible right away via the following links:75- [LTX-Studio image-to-video](https://app.ltx.studio/ltx-video)76- [Fal.ai text-to-video](https://fal.ai/models/fal-ai/ltx-video)77- [Fal.ai image-to-video](https://fal.ai/models/fal-ai/ltx-video/image-to-video)78- [Replicate text-to-video and image-to-video](https://replicate.com/lightricks/ltx-video)79 80### ComfyUI81To use our model with ComfyUI, please follow the instructions at a dedicated [ComfyUI repo](https://github.com/Lightricks/ComfyUI-LTXVideo/).82 83### Run locally84 85#### Installation86 87The codebase was tested with Python 3.10.5, CUDA version 12.2, and supports PyTorch >= 2.1.2.88 89```bash90git clone https://github.com/Lightricks/LTX-Video.git91cd LTX-Video92 93# create env94python -m venv env95source env/bin/activate96python -m pip install -e .\[inference-script\]97```98 99#### Inference100 101To use our model, please follow the inference code in [inference.py](https://github.com/Lightricks/LTX-Video/blob/main/inference.py):102 103##### For text-to-video generation:104 105```bash106python inference.py --prompt "PROMPT" --height HEIGHT --width WIDTH --num_frames NUM_FRAMES --seed SEED --pipeline_config configs/ltxv-13b-0.9.7-dev.yaml107```108 109##### For image-to-video generation:110 111```bash112python inference.py --prompt "PROMPT" --input_image_path IMAGE_PATH --height HEIGHT --width WIDTH --num_frames NUM_FRAMES --seed SEED --pipeline_config configs/ltxv-13b-0.9.7-dev.yaml113```114 115### Diffusers 🧨116 117LTX Video is compatible with the [Diffusers Python library](https://huggingface.co/docs/diffusers/main/en/index). It supports both text-to-video and image-to-video generation.118 119Make sure you install `diffusers` before trying out the examples below.120 121```bash122pip install -U git+https://github.com/huggingface/diffusers123```124 125Now, you can run the examples below (note that the upsampling stage is optional but reccomeneded):126 127### text-to-video:128```py129import torch130from diffusers import LTXConditionPipeline, LTXLatentUpsamplePipeline131from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXVideoCondition132from diffusers.utils import export_to_video133 134pipe = LTXConditionPipeline.from_pretrained("Lightricks/LTX-Video-0.9.7-dev", torch_dtype=torch.bfloat16)135pipe_upsample = LTXLatentUpsamplePipeline.from_pretrained("Lightricks/ltxv-spatial-upscaler-0.9.7", vae=pipe.vae, torch_dtype=torch.bfloat16)136pipe.to("cuda")137pipe_upsample.to("cuda")138pipe.vae.enable_tiling()139 140def round_to_nearest_resolution_acceptable_by_vae(height, width):141 height = height - (height % pipe.vae_spatial_compression_ratio)142 width = width - (width % pipe.vae_spatial_compression_ratio)143 return height, width144 145prompt = "The video depicts a winding mountain road covered in snow, with a single vehicle traveling along it. The road is flanked by steep, rocky cliffs and sparse vegetation. The landscape is characterized by rugged terrain and a river visible in the distance. The scene captures the solitude and beauty of a winter drive through a mountainous region."146negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"147expected_height, expected_width = 512, 704148downscale_factor = 2 / 3149num_frames = 121150 151# Part 1. Generate video at smaller resolution152downscaled_height, downscaled_width = int(expected_height * downscale_factor), int(expected_width * downscale_factor)153downscaled_height, downscaled_width = round_to_nearest_resolution_acceptable_by_vae(downscaled_height, downscaled_width)154latents = pipe(155 conditions=None,156 prompt=prompt,157 negative_prompt=negative_prompt,158 width=downscaled_width,159 height=downscaled_height,160 num_frames=num_frames,161 num_inference_steps=30,162 generator=torch.Generator().manual_seed(0),163 output_type="latent",164).frames165 166# Part 2. Upscale generated video using latent upsampler with fewer inference steps167# The available latent upsampler upscales the height/width by 2x168upscaled_height, upscaled_width = downscaled_height * 2, downscaled_width * 2169upscaled_latents = pipe_upsample(170 latents=latents,171 output_type="latent"172).frames173 174# Part 3. Denoise the upscaled video with few steps to improve texture (optional, but recommended)175video = pipe(176 prompt=prompt,177 negative_prompt=negative_prompt,178 width=upscaled_width,179 height=upscaled_height,180 num_frames=num_frames,181 denoise_strength=0.4, # Effectively, 4 inference steps out of 10182 num_inference_steps=10,183 latents=upscaled_latents,184 decode_timestep=0.05,185 image_cond_noise_scale=0.025,186 generator=torch.Generator().manual_seed(0),187 output_type="pil",188).frames[0]189 190# Part 4. Downscale the video to the expected resolution191video = [frame.resize((expected_width, expected_height)) for frame in video]192 193export_to_video(video, "output.mp4", fps=24)194```195 196### For image-to-video:197 198```py199import torch200from diffusers import LTXConditionPipeline, LTXLatentUpsamplePipeline201from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXVideoCondition202from diffusers.utils import export_to_video, load_image, load_video203 204pipe = LTXConditionPipeline.from_pretrained("Lightricks/LTX-Video-0.9.7-dev", torch_dtype=torch.bfloat16)205pipe_upsample = LTXLatentUpsamplePipeline.from_pretrained("Lightricks/ltxv-spatial-upscaler-0.9.7", vae=pipe.vae, torch_dtype=torch.bfloat16)206pipe.to("cuda")207pipe_upsample.to("cuda")208pipe.vae.enable_tiling()209 210def round_to_nearest_resolution_acceptable_by_vae(height, width):211 height = height - (height % pipe.vae_spatial_compression_ratio)212 width = width - (width % pipe.vae_spatial_compression_ratio)213 return height, width214 215image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/penguin.png")216video = load_video(export_to_video([image])) # compress the image using video compression as the model was trained on videos217condition1 = LTXVideoCondition(video=video, frame_index=0)218 219prompt = "A cute little penguin takes out a book and starts reading it"220negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"221expected_height, expected_width = 480, 832222downscale_factor = 2 / 3223num_frames = 96224 225# Part 1. Generate video at smaller resolution226downscaled_height, downscaled_width = int(expected_height * downscale_factor), int(expected_width * downscale_factor)227downscaled_height, downscaled_width = round_to_nearest_resolution_acceptable_by_vae(downscaled_height, downscaled_width)228latents = pipe(229 conditions=[condition1],230 prompt=prompt,231 negative_prompt=negative_prompt,232 width=downscaled_width,233 height=downscaled_height,234 num_frames=num_frames,235 num_inference_steps=30,236 generator=torch.Generator().manual_seed(0),237 output_type="latent",238).frames239 240# Part 2. Upscale generated video using latent upsampler with fewer inference steps241# The available latent upsampler upscales the height/width by 2x242upscaled_height, upscaled_width = downscaled_height * 2, downscaled_width * 2243upscaled_latents = pipe_upsample(244 latents=latents,245 output_type="latent"246).frames247 248# Part 3. Denoise the upscaled video with few steps to improve texture (optional, but recommended)249video = pipe(250 conditions=[condition1],251 prompt=prompt,252 negative_prompt=negative_prompt,253 width=upscaled_width,254 height=upscaled_height,255 num_frames=num_frames,256 denoise_strength=0.4, # Effectively, 4 inference steps out of 10257 num_inference_steps=10,258 latents=upscaled_latents,259 decode_timestep=0.05,260 image_cond_noise_scale=0.025,261 generator=torch.Generator().manual_seed(0),262 output_type="pil",263).frames[0]264 265# Part 4. Downscale the video to the expected resolution266video = [frame.resize((expected_width, expected_height)) for frame in video]267 268export_to_video(video, "output.mp4", fps=24)269```270 271### For video-to-video: 272 273```py274import torch275from diffusers import LTXConditionPipeline, LTXLatentUpsamplePipeline276from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXVideoCondition277from diffusers.utils import export_to_video, load_video278 279pipe = LTXConditionPipeline.from_pretrained("Lightricks/LTX-Video-0.9.7-dev", torch_dtype=torch.bfloat16)280pipe_upsample = LTXLatentUpsamplePipeline.from_pretrained("Lightricks/ltxv-spatial-upscaler-0.9.7", vae=pipe.vae, torch_dtype=torch.bfloat16)281pipe.to("cuda")282pipe_upsample.to("cuda")283pipe.vae.enable_tiling()284 285def round_to_nearest_resolution_acceptable_by_vae(height, width):286 height = height - (height % pipe.vae_spatial_compression_ratio)287 width = width - (width % pipe.vae_spatial_compression_ratio)288 return height, width289 290video = load_video(291 "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cosmos/cosmos-video2world-input-vid.mp4"292)[:21] # Use only the first 21 frames as conditioning293condition1 = LTXVideoCondition(video=video, frame_index=0)294 295prompt = "The video depicts a winding mountain road covered in snow, with a single vehicle traveling along it. The road is flanked by steep, rocky cliffs and sparse vegetation. The landscape is characterized by rugged terrain and a river visible in the distance. The scene captures the solitude and beauty of a winter drive through a mountainous region."296negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"297expected_height, expected_width = 768, 1152298downscale_factor = 2 / 3299num_frames = 161300 301# Part 1. Generate video at smaller resolution302downscaled_height, downscaled_width = int(expected_height * downscale_factor), int(expected_width * downscale_factor)303downscaled_height, downscaled_width = round_to_nearest_resolution_acceptable_by_vae(downscaled_height, downscaled_width)304latents = pipe(305 conditions=[condition1],306 prompt=prompt,307 negative_prompt=negative_prompt,308 width=downscaled_width,309 height=downscaled_height,310 num_frames=num_frames,311 num_inference_steps=30,312 generator=torch.Generator().manual_seed(0),313 output_type="latent",314).frames315 316# Part 2. Upscale generated video using latent upsampler with fewer inference steps317# The available latent upsampler upscales the height/width by 2x318upscaled_height, upscaled_width = downscaled_height * 2, downscaled_width * 2319upscaled_latents = pipe_upsample(320 latents=latents,321 output_type="latent"322).frames323 324# Part 3. Denoise the upscaled video with few steps to improve texture (optional, but recommended)325video = pipe(326 conditions=[condition1],327 prompt=prompt,328 negative_prompt=negative_prompt,329 width=upscaled_width,330 height=upscaled_height,331 num_frames=num_frames,332 denoise_strength=0.4, # Effectively, 4 inference steps out of 10333 num_inference_steps=10,334 latents=upscaled_latents,335 decode_timestep=0.05,336 image_cond_noise_scale=0.025,337 generator=torch.Generator().manual_seed(0),338 output_type="pil",339).frames[0]340 341# Part 4. Downscale the video to the expected resolution342video = [frame.resize((expected_width, expected_height)) for frame in video]343 344export_to_video(video, "output.mp4", fps=24)345```346 347 348To learn more, check out the [official documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video). 349 350Diffusers also supports directly loading from the original LTX checkpoints using the `from_single_file()` method. Check out [this section](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video#loading-single-files) to learn more.351 352## Limitations353- This model is not intended or able to provide factual information.354- As a statistical model this checkpoint might amplify existing societal biases.355- The model may fail to generate videos that matches the prompts perfectly.356- Prompt following is heavily influenced by the prompting-style.