Modelpublic

benjiaiplayground/HY-OmniWeaving-FP8

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face

4likes

Model Card

OmniWeaving

This model repo built to run with <a href="https://github.com/benjiyaya/OmniWeaving/">OmniWeaving forked by Benji</a>

<b>FP8 Quantized Checkpoints</b> These are FP8-quantized versions of the original OmniWeaving checkpoints, converted from bf16 to torch.float8_e4m3fn precision.

File Size Comparison File Original (bf16) FP8 Reduction File : textencodermodel.safetensors 15.8 GB 7.9 GB 50% diffusionpytorchmodel.safetensors 31.8 GB 8.0 GB 75% Total 47.6 GB 15.9 GB ~67%

The transformer achieves greater compression because many non-floating-point tensors (int/bool indices, embeddings) are kept at full precision, while the text encoder is dominated by float weights.

Performance

Metric bf16 (original) FP8 Inference speed Baseline Same (weights auto-cast to bf16 at load time) Video quality Reference Near-identical (minimal precision loss) Loading RAM ~48 GB peak ~16 GB peak Inference VRAM ~50-60 GB ~50-60 GB (unchanged)

Who Should Use This Users with limited system RAM who can't load the full bf16 checkpoint Users who want faster download times (16 GB vs 48 GB) Users who want a smaller disk footprint

Who Should NOT Use This Users expecting lower VRAM usage — use --quantize_4bit instead Users who need bit-exact reproducibility with the original bf16 model

How to Use Replace the original checkpoint files with these FP8 versions:

ckpts/

├── textencoder/ckpt/textencoder_model.safetensors ← replace with FP8 version

└── transformer/diffusionpytorchmodel.safetensors ← replace with FP8 version

Or rename the _fp8 files to match the originals. The loading code is identical — no code changes needed.

<img src="./assets/weaving-mark.svg" alt="icon" height="30"> OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning

</div>

Kaihang Pan<sup>&ast;1,2</sup>, Qi Tian<sup>&ast;2</sup>, Jianwei Zhang<sup>2</sup>, Weijie Kong<sup>2</sup>, Jiangfeng Xiong<sup>2</sup>, Yanxin Long<sup>2</sup>, Shixue Zhang<sup>2</sup>, Haiyi Qiu<sup>1</sup>, Tan Wang<sup>3</sup>, Zheqi Lv<sup>1</sup>, Yue Wu<sup>§2</sup>, Liefeng Bo<sup>2</sup>, Siliang Tang<sup>§1</sup>, Zhao Zhong<sup>&dagger;2</sup>

<sup>1</sup>Zhejiang University   <sup>2</sup>Tencent Hunyuan   <sup>3</sup>Nanyang Technological University <sup>&ast;</sup>Equal Contribution   <sup>§</sup>Corresponding Authors   <sup>&dagger;</sup>Project Leader Work done during Kaihang Pan's internship at Tencent Hunyuan

</div>

</div>

</div>

🔥🔥🔥 News

—📌 OmniWeaving is developed by the HunyuanVideo team and is built upon the latest [HunyuanVideo-1.5](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5) as the backbone. If you find our work useful, please consider giving this repository a like ❤️ and citing our paper~
—🚀 April 3, 2026: We release the code and model weights of OmniWeaving.
—🚀 April 3, 2026: We release the IntelligentVBench.
—📖 Mar 26, 2026: We release the OmniWeaving paper on Arxiv.
—👋 Mar 25, 2026: We release the webpage of OmniWeaving.

📖 Abstract

While proprietary systems such as Seedance-2.0 have achieved remarkable success in omni-capable video generation, open-source alternatives significantly lag behind. To bridge this gap, we propose <img src="./assets/weaving-mark.svg" alt="OmniWeaving" height="18px"> OmniWeaving, an omni-level video generation model featuring powerful multimodal composition and reasoning-informed capabilities. By leveraging a massive-scale pretraining dataset that encompasses diverse compositional and reasoning-augmented scenarios, OmniWeaving learns to temporally bind interleaved text, multi-image, and video inputs while acting as an intelligent agent to infer complex user intentions for sophisticated video creation. Furthermore, we introduce IntelligentVBench, the first comprehensive benchmark designed to rigorously assess next-level intelligent unified video generation. Extensive experiments demonstrate that OmniWeaving achieves SoTA performance among open-source unified models.

🏗 Model Architecture

Following the paper, OmniWeaving is built as an integrated MLLM + MMDiT + VAE framework for unified free-form video generation. The MLLM serves as the semantic parser for interleaved text, images, and video inputs, mapping them into a high-level semantic space and forwarding its hidden states through an MLP connector. The VAE acts as the visual tokenizer, compressing visual inputs into low-level latents, while the MMDiT uses these semantic conditions together with latent noise to generate semantically aligned, high-fidelity videos.

On this basis, we further introduce two extra improvements tailored for advanced reasoning and composition.

—(1) Activating Thinking Mode of the MLLM: Direct MLLM encoding of interleaved visual-text inputs often yields semantic ambiguity due to weak intra-correlations and unclear video creation intents. We elevate the MLLM from a passive feature extractor to an active reasoner. By activating the thinking mode to generate intermediate reasoning steps, it autonomously deduces a semantically precise, enhanced prompt. The hidden states of this enhanced prompt are then forwarded alongside the original MLLM features to condition the MMDiT, effectively bridging the cognitive gap between abstract user intent and pixel-level generation.
—(2) Hidden States DeepStacking: Compositional video generation involving multiple subjects or intricate scenes often relies on both low- and high-level semantic representations. Drawing inspiration from the DeepStacking mechanism in Qwen3-VL, we extract hidden states from a broader range of intermediate MLLM layers to capture a rich semantic spectrum spanning from fine-grained details to high-level abstractions. An MLP connector projects these multi-level features into the MMDiT embedding space. These projected features are then directly added to the corresponding hidden states within the first three layers of the MMDiT conditioning branch, effectively injecting multi-granular semantic guidance into the generative process.

Figure 1. Overview of the OmniWeaving architecture, which consists of an MLLM for multimodal understanding and an MMDiT for generation. </div>

🚀 Supported Tasks

OmniWeaving is flexible in its input and output configurations, supporting a wide range of unified video generation tasks:

<table> <thead> <tr> <th>Task</th> <th>Input Type</th> <th>Output</th> <th>Description</th> <th>Demo Input</th> <th>Demo Output</th> </tr> </thead> <tbody>  <tr> <td><b>Text-to-Video (T2V)</b></td> <td>Text 📝</td> <td>Video 🎬</td> <td>Generating a video from text prompts.</td> <td align="center"> <img src="assets/prompt-badge.svg" height="18"/> </td> <td align="center"> <img src="assets/cases/t2v/1.gif" width="140"/> </td> </tr>  <tr> <td><b>First-Frame-to-Video (I2V)</b></td> <td>Image 🖼 + Text 📝</td> <td>Video 🎬</td> <td>Generating a video based on the first frame.</td> <td align="center"> <img src="assets/cases/i2v/1.png" width="140"/><br/> <img src="assets/prompt-badge.svg" height="18"/> </td> <td align="center"> <img src="assets/cases/i2v/1.gif" width="140"/> </td> </tr>  <tr> <td><b>Key-Frames-to-Video</b></td> <td>2 × Images 🖼 + Text 📝</td> <td>Video 🎬</td> <td>Generating a video conditioned on start and end frames.</td> <td align="center"> <img src="assets/cases/interpolation/1first.png" width="60"/> <img src="assets/cases/interpolation/1last.png" width="60"/><br/> <img src="assets/prompt-badge.svg" height="18"/> </td> <td align="center"> <img src="assets/cases/interpolation/1.gif" width="140"/> </td> </tr>  <tr> <td><b>Video-to-Video Editing</b></td> <td>Video 🎬 + Text 📝</td> <td>Video 🎬</td> <td>Instruction-based video manipulation and stylization.</td> <td align="center"> <img src="assets/cases/editing/1before.gif" width="140"/><br/> <img src="assets/prompt-badge.svg" height="18"/> </td> <td align="center"> <img src="assets/cases/editing/1after.gif" width="140"/> </td> </tr>  <tr> <td><b>Reference-to-Video</b></td> <td>Image 🖼 + Text 📝</td> <td>Video 🎬</td> <td>Single-subject reference-driven video generation.</td> <td align="center"> <img src="assets/cases/reference2v/1.png" width="100"/><br/> <img src="assets/prompt-badge.svg" height="18"/> </td> <td align="center"> <img src="assets/cases/reference2v/1.gif" width="140"/> </td> </tr>  <tr> <td><b>Compositional Multi-Image-to-Video</b></td> <td>2–4 × Images 🖼 + Text 📝</td> <td>Video 🎬</td> <td>Multi-subject compositional video generation.</td> <td align="center"> <img src="assets/cases/compositional/11.png" width="40"/> <img src="assets/cases/compositional/12.png" width="40"/> <img src="assets/cases/compositional/13.png" width="40"/><br/> <img src="assets/prompt-badge.svg" height="18"/> </td> <td align="center"> <img src="assets/cases/compositional/1.gif" width="140"/> </td> </tr>  <tr> <td><b>Text-Image-Video-to-Video</b></td> <td>Video 🎬 + Image 🖼 + Text 📝</td> <td>Video 🎬</td> <td>Generating a video conditioned on text, image, and video inputs.</td> <td align="center"> <img src="assets/cases/tiv2v/1ref.png" width="50"/><br/> <img src="assets/cases/tiv2v/1before.gif" width="140"/><br/> <img src="assets/prompt-badge.svg" height="18"/> </td> <td align="center"> <img src="assets/cases/tiv2v/1after.gif" width="140"/> </td> </tr>  <tr> <td><b>Reasoning-Augmented Video Generation</b></td> <td>Image(s) 🖼 + Text 📝</td> <td>Reasoning 💭 + Video 🎬</td> <td>Reasoning over user intent before generating the video.</td> <td align="center"> <img src="assets/cases/reasoning/1.png" width="140"/><br/> <img src="assets/prompt-badge.svg" height="18"/> </td> <td align="center"> <img src="assets/reasoning-badge.svg" height="18"/><br/> <img src="assets/cases/reasoning/1.gif" width="140"/> </td> </tr> </tbody> </table>

🛠 Preparation

Step 1: Clone the Repository

bash

git clone https://github.com/Tencent-Hunyuan/OmniWeaving
cd OmniWeaving

Step 2: Install Dependencies

OmniWeaving is built upon HunyuanVideo-1.5. The way to install dependencies is similar to HunyuanVideo-1.5. Specifically, you should install basic dependencies:

bash

pip install -r requirements.txt

Additionally, install the attention libraries as needed (we use Flash Attention in practice):

—Flash Attention: Install for faster inference and reduced GPU memory consumption. See Flash Attention for details.

—Flex-Block-Attention: Required only for sparse attention to achieve faster inference:

bash

  git clone https://github.com/Tencent-Hunyuan/flex-block-attn.git
  cd flex-block-attn
  git submodule update --init --recursive
  python3 setup.py install

—SageAttention: For faster inference (will automatically disable Flex-Block-Attention):

bash

  git clone https://github.com/cooper1637/SageAttention.git
  cd SageAttention 
  export EXT_PARALLEL=4 NVCC_APPEND_FLAGS="--threads 8" MAX_JOBS=32 # Optional
  python3 setup.py install

Step 3: Download Models

Detailed download instructions are available at download-checkpoint.md.

🔑 Inference

In our inference code, we define six task flags corresponding to the Supported Tasks. Their mapping is as follows:

Task Flag	Full Name	Description
`t2v`	Text-to-Video	Generate videos from text prompts.
`i2v`	First-Frame-to-Video	Animate a static image into a video guided by text.
`interpolation`	Key-Frames-to-Video	Generate a video conditioned on start and end frames.
`reference2v`	Reference-to-Video / Compositional Multi-Image-to-Video	Single- or multi-subject reference-driven video generation.
`editing`	Video-to-Video Editing	Instruction-based video manipulation and stylization.
`tiv2v`	Text-Image-Video-to-Video	Generate a video conditioned on text, image, and video inputs.

Among these, `t2v`, `i2v`, and `interpolation` can optionally enable thinking mode (--think) for Reasoning-Augmented Video Generation, where the MLLM first reasons over user intent before generating the video.

Common Configuration

All tasks share the following hyperparameters (configured at the top of generate.sh):

bash

N_INFERENCE_GPU=8
SEED=0
ASPECT_RATIO=16:9
MODEL_PATH=/path/to/OmniWeaving

SAGE_ATTN=false ### Use Flash Attention
### SAGE_ATTN=true ### Use SageAttention
SPARSE_ATTN=false
OVERLAP_GROUP_OFFLOADING=false
ENABLE_CACHE=false
CACHE_TYPE=deepcache

Tips: If your GPU memory is limited and you encounter OOM errors, try: ``bash export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:128 ` If you have limited CPU memory, disable overlapped group offloading by setting OVERLAPGROUPOFFLOADING=false`.

Task-Specific Inference Scripts

1. Text-to-Video (`t2v`)

Generate a video from a text prompt.

bash

PROMPT="Put Your Prompt Here"
NEGATIVE_PROMPT="overexposed, low quality, deformation, a poor composition, bad hands, bad teeth, bad eyes, bad limbs, distortion"
OUTPUT_PATH=./outputs/t2v.mp4

torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \
  --task t2v \
  --prompt "$PROMPT" \
  --negative_prompt "$NEGATIVE_PROMPT" \
  --aspect_ratio $ASPECT_RATIO \
  --seed $SEED \
  --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN \
  --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE \
  --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING \
  --output_path $OUTPUT_PATH \
  --model_path $MODEL_PATH \
  # --think \          # Optional: enable reasoning-augmented generation (see note below)

The --think flag activates the MLLM's thinking mode, in which it reasons over user intent and generates an enriched prompt before video generation. The --think flag is supported by t2v, i2v, and interpolation tasks.

2. First-Frame-to-Video (`i2v`)

Animate a first-frame image into a video guided by a text prompt.

bash

PROMPT="Put Your Prompt Here"
IMAGE_PATH=/path/to/reference.png
OUTPUT_PATH=./outputs/i2v.mp4

torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \
  --task i2v \
  --prompt "$PROMPT" \
  --image_path $IMAGE_PATH \
  --aspect_ratio $ASPECT_RATIO \
  --seed $SEED \
  --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN \
  --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE \
  --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING \
  --output_path $OUTPUT_PATH \
  --model_path $MODEL_PATH \
  # --think \          # Optional: enable reasoning-augmented generation (see note below)

3. Key-Frames-to-Video (`interpolation`)

Generate a video that bridges two key frames, guided by a text prompt.

bash

PROMPT="Put Your Prompt Here"
REF_IMAGE_PATHS=(/path/to/first_frame.png /path/to/last_frame.png)
OUTPUT_PATH=./outputs/interpolation.mp4

torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \
  --task interpolation \
  --prompt "$PROMPT" \
  --ref_image_paths "${REF_IMAGE_PATHS[@]}" \
  --aspect_ratio $ASPECT_RATIO \
  --seed $SEED \
  --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN \
  --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE \
  --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING \
  --output_path $OUTPUT_PATH \
  --model_path $MODEL_PATH \
  # --think \          # Optional: enable reasoning-augmented generation

4. Reference-to-Video / Compositional Multi-Image-to-Video (`reference2v`)

Generate a video featuring one or more reference subjects. Provide one or more reference images via --ref_image_paths.

bash

PROMPT="Put Your Prompt Here"
# Supports 1–4 reference images.
# For best results with multiple images, use the same aspect ratio across all images,
# as they will be center-cropped to match the size of the first image.
REF_IMAGE_PATHS=(/path/to/img1.png /path/to/img2.png ... /path/to/img4.png)  # up to 4 input images
OUTPUT_PATH=./outputs/reference2v.mp4

torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \
  --task reference2v \
  --prompt "$PROMPT" \
  --ref_image_paths "${REF_IMAGE_PATHS[@]}" \
  --aspect_ratio $ASPECT_RATIO \
  --seed $SEED \
  --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN \
  --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE \
  --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING \
  --output_path $OUTPUT_PATH \
  --model_path $MODEL_PATH

5. Video-to-Video Editing (`editing`)

Edit an existing video according to the text instruction (e.g., style transfer, object replacement).

bash

PROMPT="Put Your Prompt Here"
CONDITION_VIDEO_PATH=/path/to/source_video.mp4
OUTPUT_PATH=./outputs/editing.mp4

# If you have pre-extracted VAE latents for the condition video, pass them via
# --condition_video_latents_path /path/to/latents.pt to skip VAE encoding at inference.
torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \
  --task editing \
  --prompt "$PROMPT" \
  --condition_video_paths $CONDITION_VIDEO_PATH \
  --aspect_ratio $ASPECT_RATIO \
  --seed $SEED \
  --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN \
  --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE \
  --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING \
  --output_path $OUTPUT_PATH \
  --model_path $MODEL_PATH \
  # --condition_video_latents_path /path/to/latents.pt  # Optional: skip VAE encoding by providing pre-extracted latents

6. Text-Image-Video-to-Video (`tiv2v`)

Edit a video while incorporating reference subject images (e.g., insert a character from a reference image into a source video).

bash

PROMPT="Put Your Prompt Here"
CONDITION_VIDEO_PATH=/path/to/source_video.mp4
# Only one reference image is supported for tiv2v.
# For best results, use a reference image whose aspect ratio is close to the output video's aspect ratio.
REF_IMAGE_PATHS=(/path/to/ref_image.png)
OUTPUT_PATH=./outputs/tiv2v.mp4

# If you have pre-extracted VAE latents for the condition video, pass them via
# --condition_video_latents_path /path/to/latents.pt to skip VAE encoding at inference.
torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \
  --task tiv2v \
  --prompt "$PROMPT" \
  --condition_video_paths $CONDITION_VIDEO_PATH \
  --ref_image_paths "${REF_IMAGE_PATHS[@]}" \
  --aspect_ratio $ASPECT_RATIO \
  --seed $SEED \
  --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN \
  --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE \
  --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING \
  --output_path $OUTPUT_PATH \
  --model_path $MODEL_PATH \
  # --condition_video_latents_path /path/to/latents.pt  # Optional: skip VAE encoding by providing pre-extracted latents

Other Optional Arguments

The arguments below can be appended to any of the task commands above for further customization:

Argument	Type	Default	Description
`--negative_prompt`	str	`""`	Negative prompt for video generation. Default is empty. Setting a negative prompt (e.g., `'overexposed, low quality, deformation, a poor composition, bad hands, bad teeth, bad eyes, bad limbs, distortion'`) can improve quality, especially for tasks like `t2v`.
`--num_inference_steps`	int	`50`	Number of denoising steps
`--video_length`	int	`81`	Number of frames to generate
`--fps`	int	Auto	Output FPS (default: 16 for ≤81 frames, 24 for >81 frames)
`--dtype`	str	`bf16`	Data type: `bf16` or `fp32`
`--offloading`	bool	`true`	Enable CPU offloading
`--group_offloading`	bool	`None`	Enable group offloading (auto-enabled with offloading)
`--pipeline_config`	str	`omniweaving`	Pipeline configuration preset that controls `guidance_scale` and `flow_shift`. Available presets: `omniweaving` (`guidance_scale=6.0, flow_shift=7.0`), `omniweaving2` (`guidance_scale=6.0, flow_shift=5.0`).

Tuning `guidance_scale` / `flow_shift`: You can switch presets via --pipeline_config (e.g., --pipeline_config omniweaving2). If the available presets do not meet your needs, you can add a new key to the `PIPELINE_CONFIGS` dict in `hyvideo/commons/__init__.py` with your desired values. We recommend guidance_scale=6.0 with flow_shift=5.0 or 7.0.

📚 Citation

If you find our work helpful, please consider giving us a like ❤️ on this repo and citing our papers as follows:

OmniWeaving

bibtex

@article{pan2026omniweaving,
  title={OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning},
  author={Pan, Kaihang and Tian, Qi and Zhang, Jianwei and Kong, Weijie and Xiong, Jiangfeng and Long, Yanxin and Zhang, Shixue and Qiu, Haiyi and Wang, Tan and Lv, Zheqi and others},
  journal={arXiv preprint arXiv:2603.24458},
  year={2026}
}

HunyuanVideo 1.5

bibtex

@article{wu2025hunyuanvideo,
  title={Hunyuanvideo 1.5 technical report},
  author={Wu, Bing and Zou, Chang and Li, Changlin and Huang, Duojun and Yang, Fang and Tan, Hao and Peng, Jack and Wu, Jianbing and Xiong, Jiangfeng and Jiang, Jie and others},
  journal={arXiv preprint arXiv:2511.18870},
  year={2025}
}

🙏 Acknowledgements

We would like to thank the contributors to HunyuanVideo 1.5, Transformers, Diffusers, HuggingFace and Qwen-VL, for their open research and exploration.

OmniWeaving

<img src="./assets/weaving-mark.svg" alt="icon" height="30"> OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning

🔥🔥🔥 News

📋 Table of Contents

📖 Abstract

🏗 Model Architecture

🚀 Supported Tasks

🛠 Preparation

Step 1: Clone the Repository

Step 2: Install Dependencies

Step 3: Download Models

🔑 Inference

Common Configuration

Task-Specific Inference Scripts

1. Text-to-Video (t2v)

2. First-Frame-to-Video (i2v)

3. Key-Frames-to-Video (interpolation)

4. Reference-to-Video / Compositional Multi-Image-to-Video (reference2v)

5. Video-to-Video Editing (editing)

6. Text-Image-Video-to-Video (tiv2v)

Other Optional Arguments

📚 Citation

🙏 Acknowledgements

1. Text-to-Video (`t2v`)

2. First-Frame-to-Video (`i2v`)

3. Key-Frames-to-Video (`interpolation`)

4. Reference-to-Video / Compositional Multi-Image-to-Video (`reference2v`)

5. Video-to-Video Editing (`editing`)

6. Text-Image-Video-to-Video (`tiv2v`)