Eviation/flux-imatrix

Supported?

Expect broken or faulty items for the time being. Use at your own discretion.

• —ComfyUI-GGUF: all? (CPU/CUDA)
• —Fast dequant: BF16, Q80, Q51, Q50, Q41, Q40, Q6K, Q5K, Q4K, Q3K, Q2K
• —Slow dequant: others via GGUF/NumPy
• —Forge: TBC
• —stable-diffusion.cpp: llama.cpp Feature-matrix
• —CPU: all
• —Cuda: all?
• —Vulkan: >= Q3KS, > IQ4S; [PR IQ1S, IQ1M](https://github.com/ggerganov/llama.cpp/pull/11528) [PR IQ4XS](https://github.com/ggerganov/llama.cpp/pull/11501)
• —other: ?

Disco

Dynamic quantization:

• —timein.inlayer: Q80/Q6K
• —finallayer, vectorin.inlayer, guidancein: Q8_0
• —vectorin.outlayer, timein.outlayer, txtin, imgin: F16
• —single_blocks.[> 10 && < 37].modulation.lin: one down?

Caesar

Combined imatrix multiple images 512x512 and 768x768, 25, 30 and 50 steps city96/flux1-dev-Q8_0 euler

data: load_imatrix: loaded 314 importance matrix entries from imatrix_caesar.dat computed on 475 chunks

Using llama.cpp quantize cae9fb4 with modified lcpp.patch.

Dynamic quantization:

• —imgin, guidancein.inlayer, finallayer.linear: f32/bf16/f16
• —guidancein, finallayer: bf16/f16
• —img_attn.qkv, linear1: some layers two bits up
• —txtmod.lin, txtmlp, txt_attn.proj: some layers one bit down

Experimental from f16

Observations

• —More imatrix data doesn't necessarily result in better quants
• —I-quants worse than same bits k-quants?
• —Quant-dequant loss

Bravo

Combined imatrix multiple images 512x512 25 and 50 steps city96/flux1-dev-Q8_0 euler

Using llama.cpp quantize cae9fb4 with modified lcpp.patch.

Experimental from f16

Observations

• —Bravo IQ1_S worse than Alpha?
• —Latent loss
• —Per layer quantization cost from chrisgoringe/casting_cost
• —Per layer quantization cost 2 from Freepik/flux.1-lite-8B: double blocks and single blocks
• —Ablation latent loss per weight type
• —Pareto front loss vs. size

Alpha

Simple imatrix: 512x512 single image 8/20 steps city96/flux1-dev-Q3_K_S euler

data: load_imatrix: loaded 314 importance matrix entries from imatrix.dat computed on 7 chunks.

Using llama.cpp quantize cae9fb4 with modified lcpp.patch.

Experimental from q8

Observations

Sub-quants not diferentiated as expected: IQ2XS == IQ2S, IQ3XS == IQ3S == IQ3M, Q3KM == Q3K_L.

• —Check if lcpp_sd3.patch includes more specific quant level logic
• —Extrapolate the existing level logic