ModelsLab/Flux2-Klein-9B-True-V3-fp8

1---2license: other3license_name: flux-non-commercial-license-v2.14license_link: https://bfl.ai/licensing5base_model:6  - wikeeyang/Flux2-Klein-9B-True-V37  - black-forest-labs/FLUX.2-klein-9B8base_model_relation: quantized9library_name: diffusers10pipeline_tag: text-to-image11tags:12  - flux13  - flux214  - klein15  - fp816  - quantized17  - optimum-quanto18language:19  - en20---21 22# Flux2-Klein-9B-True-V3-fp823 24An FP8 build of [`wikeeyang/Flux2-Klein-9B-True-V3`](https://huggingface.co/wikeeyang/Flux2-Klein-9B-True-V3), saved in the [optimum-quanto](https://github.com/huggingface/optimum-quanto) `QuantizedDiffusersModel` layout so it loads directly into a `diffusers` `Flux2KleinPipeline` with no conversion step.25 26**This is a format and precision conversion only.** Nothing was retrained, distilled or merged. The weights are wikeeyang's True-V3.27 28> ### ⚠️ Licence — read before use29>30> These weights derive from [`black-forest-labs/FLUX.2-klein-9B`](https://huggingface.co/black-forest-labs/FLUX.2-klein-9B), released under the **FLUX Non-Commercial License v2.1**. Commercial or revenue-generating deployment requires a separate licence from Black Forest Labs — see [bfl.ai/licensing](https://bfl.ai/licensing).31>32> Only FLUX.2-klein-**4B** is Apache-2.0. The 9B models are not. The upstream True-V3 repository is tagged `apache-2.0`; that tag does not reflect the terms of the base model, and a third party cannot relicense weights they do not own. This repository is labelled to match the actual upstream terms.33 34## Why this exists35 36True-V3 ships ComfyUI single-file formats only — `bf16`, `fp8mixed`, `int8mixedrow`, GGUF, `nvfp4`, `int4_convrot`. None of them is a quanto checkpoint: `QuantizedDiffusersModel.from_pretrained()` expects quanto's own directory layout (`_data`/`_scale` tensors plus a quanto config), and quanto cannot read ComfyUI's scaled-FP8 tensor layout.37 38So a diffusers/quanto serving stack could not use True-V3 without converting it first. This repository is that conversion, published so nobody has to redo it.39 40## Usage41 42```python43import torch44from diffusers import Flux2KleinPipeline, Flux2Transformer2DModel45from optimum.quanto import QuantizedDiffusersModel46 47# quanto's Marlin FP8 kernel rejects a non-contiguous activation, which the48# Flux2 attention path produces. Apply before loading.49from optimum.quanto.nn.qlinear import QLinear50_orig = QLinear.forward51QLinear.forward = lambda self, x: _orig(self, x.contiguous())52 53 54class QuantizedFlux2KleinTransformer(QuantizedDiffusersModel):55    base_class = Flux2Transformer2DModel56 57 58pipe = Flux2KleinPipeline.from_pretrained(59    "black-forest-labs/FLUX.2-klein-9B",60    transformer=None,61    torch_dtype=torch.bfloat16,62).to("cuda")63 64pipe.transformer = QuantizedFlux2KleinTransformer.from_pretrained(65    "ModelsLab/Flux2-Klein-9B-True-V3-fp8"66).to("cuda")67 68image = pipe(69    prompt="a vintage storefront window with a hand-painted gold sign that reads "70           '"KLEIN COFFEE ROASTERS EST 1974", rainy street reflection',71    height=1024, width=1024,72    num_inference_steps=4,73    generator=torch.Generator("cuda").manual_seed(1234),74).images[0]75image.save("out.png")76```77 78### Tighter VRAM79 80The transformer is 9.08 GB. The bf16 text encoder is another ~16 GB, which will not fit alongside it on a 24 GB card. Quantise it the same way:81 82```python83from optimum.quanto import QuantizedModelForCausalLM, qfloat884from transformers import AutoModelForCausalLM85 86te = AutoModelForCausalLM.from_pretrained(87    "black-forest-labs/FLUX.2-klein-9B", subfolder="text_encoder",88    torch_dtype=torch.bfloat16)89pipe.text_encoder = QuantizedModelForCausalLM.quantize(90    te, weights=qfloat8).to("cuda")91```92 93FP8 transformer + FP8 text encoder is ~18 GB resident. Measured peak allocation on an RTX 3090 at 1024×1024, 4 steps, with both resident: **20.66 GB** — so it fits a 24 GB card, with roughly 3 GB spare.94 95## Measured against the stock FP8 checkpoint96 97![stock klein-9B FP8 vs True-V3 FP8, same prompt and seed](comparison.jpg)98 99*Same prompt, same seed, same 4 steps, same quantiser. Top row is the stock FP8 checkpoint, bottom row is this one. Note the misspelled "COFFFEE" in the stock render.*100 101Both checkpoints were run through an identical harness: same architecture, same quantiser, same 4 steps, same guidance, same seeds, and prompt embeddings encoded once and cached so both saw bit-identical conditioning. 12 prompts × 3 seeds = **36 timed renders each**, 1024×1024, on an RTX 3090.102 103| | stock klein-9B FP8 | True-V3 FP8 | Δ |104|---|---|---|---|105| Aesthetic (LAION v2) | 5.996 | **6.167** | +0.171 |106| CLIP adherence (×100) | 28.98 | **29.65** | +0.67 |107| Reference retention¹ | 56.6 | **60.6** | +4.0 |108| Median latency / 1024² | 6.29 s | 6.31 s | +0.2% |109| On-disk size | 9.08 GB | 9.08 GB | — |110 111¹ CLIP image–image cosine ×100 between each edit and its reference images, over a two-reference editing run (four scenes × two seeds), scored on the render set shared by every variant tested.112 113The latency difference is inside run-to-run noise — across eight variants benchmarked the same way, the whole spread was 0.06 s. Same architecture, same step count, same FP8 kernels, so this is a quality change at fixed cost rather than a trade.114 115Qualitatively, the clearest wins over stock are **rendered text** (the sign prompt above is misspelled "COFFFEE" by the stock checkpoint and correct here), **hands**, and **product/material detail**. Colour is slightly less saturated and more filmic.116 117**Caveat on the metrics:** the LAION aesthetic predictor rewards contrast and saturation, so it is not a neutral judge of photographic realism — a crunchier model can score well while rendering worse text. Treat the table as a ranking aid and look at your own prompts.118 119## Verification120 121This checkpoint was checked by regenerating four benchmark prompts at a fixed seed and diffing them122against the renders produced by the original bf16 True-V3 weights in the earlier benchmark run:123 124| prompt | max abs diff | mean abs diff | PSNR |125|---|---|---|---|126| text | 0 | 0.0000 | ∞ |127| portrait | 0 | 0.0000 | ∞ |128| hands | 0 | 0.0000 | ∞ |129| product | 0 | 0.0000 | ∞ |130 131**Bit-identical** on all four. The conversion changes the container, not the output.132 133## How this was built134 135Reproducible in about a minute of compute once the 18 GB source file is local (63 s measured: load, quantise, save):136 137```python138import torch139from diffusers import Flux2Transformer2DModel140from optimum.quanto import QuantizedDiffusersModel, qfloat8141from huggingface_hub import hf_hub_download142 143 144class QuantizedFlux2KleinTransformer(QuantizedDiffusersModel):145    base_class = Flux2Transformer2DModel146 147 148path = hf_hub_download("wikeeyang/Flux2-Klein-9B-True-V3",149                       "Flux2-Klein-9B-True-V3-bf16.safetensors")150 151tr = Flux2Transformer2DModel.from_single_file(152    path,153    config="black-forest-labs/FLUX.2-klein-9B", subfolder="transformer",154    torch_dtype=torch.bfloat16,155)156QuantizedFlux2KleinTransformer.quantize(tr, weights=qfloat8).save_pretrained("out")157```158 159## Notes and caveats160 161- **Ampere (RTX 3090 / A100):** these cards have no native FP8 tensor cores. quanto routes through its Marlin FP8 GEMM, which does run on `sm_80`/`sm_86` — so FP8 here buys memory, and the speed is comparable to the equivalent bf16 path rather than dramatically faster. Ada and Hopper get more from it.162- **The contiguity patch in the usage snippet is required**, not optional. Without it the Marlin kernel raises `RuntimeError: A is not contiguous` on the first forward pass.163- **LoRAs must be fused before quantisation.** Calling `load_lora_weights()` on an already-quantised transformer does not work. Load the bf16 transformer, `fuse_lora()`, then quantise.164- **Only the transformer is replaced.** Text encoder, VAE, scheduler and tokenizer come from the base `FLUX.2-klein-9B` repo unchanged.165- **`guidance_scale` is ignored.** Klein is step-distilled, and `diffusers` emits *"Guidance scale 4.0 is ignored for step-wise distilled models"* and drops the value. Passing it does nothing; do not expect it to behave as a tuning knob.166 167## Credits168 169- [**wikeeyang**](https://huggingface.co/wikeeyang) — the True-V3 finetune. All of the quality improvement is theirs; this repository only changes the container.170- [**Black Forest Labs**](https://huggingface.co/black-forest-labs) — FLUX.2-klein-9B.171- [**optimum-quanto**](https://github.com/huggingface/optimum-quanto) — the quantiser and the checkpoint format.172 173Converted and published by [ModelsLab](https://huggingface.co/ModelsLab).174