lite-infer/LTX-2.3-Distilled-v1.1-Diffusers-nunchaku-lite-nvfp4-bnb4-text-encoder

LTX-2.3 Distilled v1.1 — Nunchaku Lite nvfp4 (Diffusers)

Nunchaku Lite nvfp4 SVDQuant quantization of `rootonchair/LTX-2.3-Distilled-v1.1-Diffusers`, a Diffusers conversion of the v1.1 distilled checkpoint of Lightricks/LTX-2.3 — a DiT-based foundation model that jointly generates synchronized video and audio. Runs in 8 steps with CFG = 1.

Quantization

• —Transformer: SVDQuant nvfp4 (rank 32) with GPTQ residual rounding, quantized with `diffuse_compressor` and exported via examples/convert_nunchaku_lite_diffusers.py to Diffusers' native Nunchaku Lite format. Loading requires Diffusers' NunchakuLiteQuantizer (baked into transformer/config.json's quantization_config) plus the `kernels` package for the compiled CUDA kernels.
• —Text encoder (Gemma3): additionally quantized to BNB4 NF4 via bitsandbytes (baked into text_encoder/config.json's quantization_config), to reduce host RAM/VRAM footprint alongside the quantized transformer.
• —Connectors (LTX2TextConnectors, the text-conditioning adapter run once per generation before the denoising loop — memory-bound like the text encoder, not compute-bound like the DiT): also quantized to BNB4 NF4 via bitsandbytes (baked into connectors/config.json's quantization_config), shrinking it from ~6GB to ~1.7GB.

Everything else (VAE, audio VAE, tokenizer, vocoder, scheduler config) is unchanged from the source repo.

VRAM

Measured on a 32GB GPU, no CPU offload, at the default 768x512 / 121-frame / 8-step settings:

• —After pipe.to("cuda") (weights only): ~23.9 GB
• —Peak during generation: ~29.3 GB allocated / ~32.6 GB reserved

This still uses most of the capacity of a 32GB-class GPU — it will not comfortably fit on a 24GB card without CPU offload (enable_model_cpu_offload() / enable_sequential_cpu_offload()).

Latency

Measured on a single Blackwell GPU, no CPU offload, at the default 768x512 / 121-frame / 8-step settings (pipeline weights already resident on-device, i.e. per-generation cost after the one-time pipeline load):

• —Pipeline load (from_pretrained + pipe.to("cuda"), locally cached): ~5s (one-time)
• —8-step denoising loop: ~11s (~1.4s/step)
• —VAE decode + video encode: <1s
• —Total per 5-second 768x512 clip: ~16-17s

Usage

Requires a Diffusers build with the native Nunchaku Lite quantizer, plus kernels and bitsandbytes:

pip install -U git+https://github.com/huggingface/diffusers kernels bitsandbytes

import torch
from diffusers import LTX2Pipeline
from diffusers.pipelines.ltx2.export_utils import encode_video
from diffusers.pipelines.ltx2.utils import DEFAULT_NEGATIVE_PROMPT, DISTILLED_SIGMA_VALUES

pipe = LTX2Pipeline.from_pretrained(
    "lite-infer/LTX-2.3-Distilled-v1.1-Diffusers-nunchaku-lite-nvfp4-bnb4-text-encoder",
    torch_dtype=torch.bfloat16,
)
pipe = pipe.to("cuda")

prompt = "A flowing river in a forest at golden hour, gentle wind in the leaves."
frame_rate = 24.0

video, audio = pipe(
    prompt=prompt,
    negative_prompt=DEFAULT_NEGATIVE_PROMPT,
    width=768,
    height=512,
    num_frames=121,
    frame_rate=frame_rate,
    num_inference_steps=8,
    sigmas=DISTILLED_SIGMA_VALUES,
    guidance_scale=1.0,
    output_type="np",
    return_dict=False,
)

encode_video(
    video[0],
    fps=frame_rate,
    audio=audio[0].float().cpu(),
    audio_sample_rate=pipe.vocoder.config.output_sampling_rate,
    output_path="ltx2_distilled_v1_1_nvfp4.mp4",
)

Notes

• —width/height must be divisible by 32; num_frames must equal 8k + 1.
• —Always pass sigmas=DISTILLED_SIGMA_VALUES, num_inference_steps=8, guidance_scale=1.0 for this distilled checkpoint.
• —The quantization config is embedded in transformer/config.json, text_encoder/config.json, and connectors/config.json, so no PipelineQuantizationConfig is needed at load time.
• —See the Diffusers LTX-2 docs for multimodal guidance, prompt enhancement, and the upscaling/refinement pipeline.

License

These weights are released under the LTX Video 2 Open Source License.