MetaphoricalCode/DeepSeek-V4-Flash-Vision-Uncensored-3.87bpw-Q3_K-GGUF

Llamacpp imatrix Quantization of DeepSeek-V4-Flash-Vision-Uncensored by orcarouter

Using <a href="https://github.com/ggml-org/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggml-org/llama.cpp/releases/tag/b10809">b10809</a> for quantization.

Original model: https://huggingface.co/orcarouter/DeepSeek-V4-Flash-Vision-Uncensored

Quant details

• —This is a 3.87bpw quant meant to be run on a 128GB RAM machine with at least 16, or more likely 24GB VRAM GPU. ffn_(?:gate|up)_exps were quantized from MXFP4 down to Q3_K with the rest of the weights being intact, similar to <a href="https://huggingface.co/orcarouter/DeepSeek-V4-Flash-Vision-Uncensored-GGUF">orcarouter's MXFP4 GGUF quant</a>. The idea of using Q3_K over IQ3_S is to win some speed. IQ quants are known to be slower, especially when used with CPU, though the extent of it varies. On my machine exactly the same quant but with IQ3_S layers instead of Q3_K runs ~30% slower in terms of decode speed. In this particular case, the difference in efficiency between the two is negligible: both scale down to 880 MiB and seem to be similar in quality.
• —The following <a href="https://huggingface.co/tarruda/DeepSeek-V4-Flash-Vision-Exp-GGUF/blob/main/imatrix.gguf">imatrix</a> was used to create this quant, provided by one of llama.cpp maintainers, <a href="https://huggingface.co/tarruda">tarruda</a>.
• —Abliteration left the vision tower intact, so it's okay to use any vision tower compatible with the original Vision-Exp model. BF16 was provided in this repo just in case.

If you're looking for a quant that shares the same recipe and idea behind it, but for the original weights (without abliteration), then there's this <a href="https://huggingface.co/ddh0/DeepSeek-V4-Flash-Vision-Exp-GGUF/blob/main/DeepSeek-V4-Flash-Vision-Exp-3.86bpw-Q3_K.gguf">quant</a> by <a href="https://huggingface.co/ddh0">ddh0</a>.