feizhai123/MiniMax-H3-ModelOpt-Mixed9-Dynamic-FP8

1---2pipeline_tag: image-text-to-video3library_name: vllm-omni4license: other5license_name: minimax-h3-community-license-agreement6license_link: LICENSE7base_model: MiniMaxAI/MiniMax-H38base_model_relation: quantized9tags:10  - minimax-h311  - text-to-video12  - image-to-video13  - text-to-audio-video14  - fp815  - modelopt16  - mixed-precision17  - vllm-omni18---19 20# MiniMax-H3 ModelOpt Mixed Dynamic FP821 22This repository contains mixed BF16/FP8 derivatives of both MiniMax-H3 video23generation partitions:24 25- **FL2VA**, under `FL2VA/`, supports T2VA/FL2VA and targets an26  effective precision of 9.0 bits.27- **Ref2VA**, under `Ref2VA/`, supports reference-conditioned video and audio28  generation and targets an effective transformer precision of 9.25 bits.29 30The checkpoints target single-GPU vLLM-Omni serving. CPU component offload is31recommended on GPUs that cannot hold all resident components.32 33> **License:** these derivatives remain subject to the MiniMax H3 Community34> License Agreement in [LICENSE](LICENSE), including its territorial,35> redistribution, notice, and acceptable-use requirements. Review that license36> before using or redistributing this model.37 38## Runtime format39 40| Setting | Value |41|---|---|42| Weight format | FP8 E4M3 for selected Linear weights; BF16 otherwise |43| Weight scaling | Per-output-channel |44| Activation scaling | Dynamic per-token |45| ModelOpt algorithm | `FP8_PER_CHANNEL_PER_TOKEN` |46| Kernel | CUTLASS FP8 in vLLM-Omni |47 48“Mixed9” and “Mixed9.25” are parameter-weighted effective precisions. They do49not denote scalar 9-bit dtypes.50 51## FL2VA Mixed952 53The `FL2VA/` checkpoint was selected using BF16-vs-dynamic-FP8 relative MSE.54 55| Component | Achieved precision | FP8 groups | FP8 parameters | BF16 groups | BF16 parameters |56|---|---:|---:|---:|---:|---:|57| H3 transformer | 8.997527 bits | 203 | 28,988,080,128 | 63 | 4,129,456,128 |58| Qwen3-VL language encoder | 8.994409 bits | 166 | 21,349,007,360 | 34 | 3,030,384,640 |59 60### FL2VA transformer scope61 62Kept in BF16:63 64- `proj_in`, `audio_proj_in`, `context_embedder`, both timestep embedding65  linears, `proj_out`, and `audio_proj_out`.66- Attention output projections in transformer blocks 29-48.67- FFN input/gate-up projections in blocks 28, 31, 32, 33, 41, 45, and 46.68- FFN output projections in blocks 5, 6, 8-10, 12-14, 19, 20, and 30-48.69- Biases, normalization parameters, RoPE buffers, and other non-linear state.70 71Quantized to dynamic FP8:72 73- All Q/K/V projections and AdaLN linears in transformer blocks 0-49.74- All Linear projections in both token-refiner blocks and `norm_out.linear`.75- Attention and FFN projections not listed in the BF16 sets above.76 77### Shared Qwen3-VL text encoder scope78 79Kept in BF16:80 81- Token embeddings, RMSNorms, rotary state, all non-linear parameters, and the82  complete Qwen3-VL vision encoder.83- Attention `o_proj` in language layers 24, 25, 30, 31, 35, 37, 38, 40, and84  42-49.85- MLP `down_proj` in language layers 17-19, 21-30, 34, 36, 46, 48, and 49.86 87Quantized to dynamic FP8:88 89- All Q/K/V and MLP gate/up projections in language layers 0-49.90- Every attention `o_proj` and MLP `down_proj` not listed above.91 92The video VAE, audio VAE, tokenizer, processor, embeddings, normalization93layers, and vision encoder remain unquantized.94 95Exact selections and sensitivity scores are stored in:96 97- `FL2VA/transformer/transformer_mixed_precision_config.json`98- `FL2VA/transformer/transformer_sensitivity_ranking.tsv`99- `FL2VA/text_encoder/text_encoder_mixed_precision_config.json`100- `FL2VA/text_encoder/text_encoder_sensitivity_ranking.tsv`101 102## Ref2VA GlobalGrad Mixed9.25103 104The Ref2VA transformer was independently selected from BF16 using an105output-probed global-gradient sensitivity score. This is a custom search106inspired by mixed-precision AutoQuant; it is not an NVIDIA ModelOpt AutoQuant107export. ModelOpt performs the dynamic FP8 conversion and export.108 109| Setting | Value |110|---|---:|111| Target effective precision | 9.25 bits |112| Achieved effective precision | **9.248862 bits** |113| FP8 groups / Linear modules | 212 / 296 |114| FP8 candidate parameters | 27,947,630,592 |115| BF16 groups / Linear modules | 54 / 74 |116| BF16 candidate parameters | 5,169,905,664 |117 118### Ref2VA transformer scope119 120| Projection family | Dynamic FP8 blocks | BF16 blocks |121|---|---|---|122| Q/K/V | 0, 2-40 | 1, 41-49 |123| Attention output | 0-4, 6-38 | 5, 39-49 |124| FFN input/gate-up | 1-38, 40 | 0, 39, 41-49 |125| FFN output | 0, 2-40 | 1, 41-49 |126| AdaLN Linear | 2-47 | 0-1, 48-49 |127 128Also quantized to dynamic FP8:129 130- All Q/K/V, attention-output, and FFN Linear projections in both token-refiner131  blocks.132- `norm_out.linear`.133 134Always retained in BF16:135 136- `context_embedder`, `proj_in`, `audio_proj_in`, both timestep embedding137  linears, `proj_out`, and `audio_proj_out`.138- Biases, normalization parameters, RoPE buffers, and all other non-linear139  state.140 141Ref2VA reuses the Mixed9 Qwen3-VL text encoder described above. Its VAEs,142tokenizer, processor, embeddings, normalization layers, and vision encoder are143not quantized. The exact transformer decisions are stored in:144 145- `Ref2VA/transformer/transformer_mixed_precision_config.json`146- `Ref2VA/transformer/transformer_sensitivity_ranking.tsv`147 148## Similarity samples149 150These are deterministic short regression samples, not comprehensive151perceptual-quality benchmarks.152 153| Partition | Task | Resolution | Steps | Seed | Video SSIM vs BF16 | Video PSNR | Audio spectral cosine |154|---|---|---:|---:|---:|---:|---:|---:|155| FL2VA Mixed9 | T2VA | 672 x 384 | 10 | 1101 | **0.858496** | **24.7968 dB** | not measured |156| Ref2VA Mixed9.25 | Ref2VA | 672 x 384 | 10 | 3101 | **0.758005** | **22.234802 dB** | **0.996300151** |157 158For the Ref2VA sample, FP8 peak GPU memory was 84,842 MiB versus 131,308 MiB159for BF16, saving 46,466 MiB (35.39%). The request used the same extracted160reference frame/audio and saved BF16 baseline, produced 107 frames, and used a1614-second requested duration. Full machine-readable results are included in:162 163- `evaluation/t2va_bf16_similarity.json`164- `evaluation/ref2va_globalgrad9p25_bf16_similarity.json`165 166## vLLM-Omni serving167 168This checkpoint requires vLLM-Omni with MiniMax-H3 ModelOpt mixed-FP8 loading169support.170 171FL2VA/T2VA:172 173~~~bash174CUDA_VISIBLE_DEVICES=0 \175vllm-omni serve feizhai123/MiniMax-H3-ModelOpt-Mixed9-Dynamic-FP8 \176  --omni \177  --host 0.0.0.0 \178  --port 8000 \179  --trust-remote-code \180  --task-type t2va \181  --enforce-eager \182  --force-cutlass-fp8 \183  --enable-cpu-offload \184  --stage-init-timeout 1800 \185  --init-timeout 2400186~~~187 188Ref2VA:189 190~~~bash191hf download feizhai123/MiniMax-H3-ModelOpt-Mixed9-Dynamic-FP8 \192  --local-dir ./MiniMax-H3-ModelOpt-Mixed9-Dynamic-FP8193 194CUDA_VISIBLE_DEVICES=0 \195vllm-omni serve ./MiniMax-H3-ModelOpt-Mixed9-Dynamic-FP8/Ref2VA \196  --omni \197  --host 0.0.0.0 \198  --port 8000 \199  --trust-remote-code \200  --enforce-eager \201  --force-cutlass-fp8 \202  --enable-cpu-offload \203  --stage-init-timeout 1800 \204  --init-timeout 2400205~~~206 207## Modification notice208 209Selected H3 transformer and Qwen3-VL language-model Linear weights were210modified from the original MiniMax-H3 checkpoints by mixed BF16/FP8211quantization. The VAEs, Qwen3-VL vision encoder, tokenizer, processor, and212other explicitly retained parameters remain in their original precision.213