prometheusAIR/DeepSeek-V4-Flash-Vision-Exp-GGUF

DeepSeek-V4-Flash-Vision-Exp — GGUF (multimodal, imatrix)

Imatrix GGUFs of deepseek-ai/DeepSeek-V4-Flash-Vision-Exp that actually do vision — four rungs from 66.9 to 108.7 GiB. It also ships the importance matrix all four rungs were built with.

### ⚠️ Build llama.cpp from master — 2026-09-02 or later Upstream support landed in two halves on 2026-09-02: #28133 (vision tower) and #28154 (image-token routing). You need both. Release build b10763 carries only the first and will refuse to load these files; build master at or after `9400c894`. No patch is needed any more — see Why a recent build.

Files

Four rungs, all built from the same importance matrix, which is included.

Shard naming is DeepSeek-V4-Flash-Vision-Exp-<RUNG>-0000{1..4}-of-00004.gguf.

¹ Effective average over the whole file, not the expert type alone. The rung name is the dominant expert type; every rung mixes several — see Recipes. Note general.file_type reads Q6_K in all of them: that field records the base type passed to llama-quantize, and cannot express per-tensor overrides.

Will this run on my GPU?

Weights only — add ~13 GiB for KV/compute at a full 1M context, or far less at shorter contexts. Keep the vision tower on CPU with --no-mmproj-offload unless you have room; it is a one-off cost per image (466M params over ≤384 visual tokens).

Build

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
# vision support merged 2026-09-02; no tagged release contains all of it yet,
# so build master (or check out 9400c894 or later explicitly)
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-mtmd-cli

Run

Substitute your chosen rung for IQ2_XXS below.

./build/bin/llama-server \
  --model DeepSeek-V4-Flash-Vision-Exp-IQ2_XXS-00001-of-00004.gguf \
  --mmproj mmproj-DeepSeek-V4-Flash-Vision-Exp-F16.gguf --no-mmproj-offload \
  -ngl 999 --ctx-size 1048576 -fa on -b 2048 -ub 512 -np 1 --no-mmap --jinja

-ub 512 is needed to reach 1M context on a 96 GB card. -ub 2048 roughly doubles prefill but adds ~4.2 GB of compute buffer, which will not fit alongside a speculative draft; drop the draft if you want the larger ubatch.

Quick check with llama-mtmd-cli:

./build/bin/llama-mtmd-cli \
  -m DeepSeek-V4-Flash-Vision-Exp-IQ2_XXS-00001-of-00004.gguf \
  --mmproj mmproj-DeepSeek-V4-Flash-Vision-Exp-F16.gguf \
  --image photo.jpg -p "What is in this image?"

Why a recent build

DeepSeek-V4-Flash-Vision carries a second MoE router bias, ffn.gate.bias_vl, used only for image tokens. The reference implementation selects experts with

scores + torch.where(input_ids >= vocab_size, bias_vl, bias)

and on the three hash-routed layers image tokens abandon the tid2eid lookup entirely for a learned top-k over scores + bias_vl.

That is the whole reason for the version floor, and it shows up as two distinct failures on an older build:

1. 1.The tensor is not in the architecture yet. These files keep it, as blk.N.exp_probs_b_vl.bias, so any llama.cpp before #28154 reports wrong number of tensors; expected 1371, got 1328 and refuses to load. Note that converters which simply drop bias_vl load anywhere — and route image tokens by text preferences. Measured across all 43 layers the two biases are uncorrelated (mean Pearson r = −0.026), so they encode genuinely different expert choices.

1. 1.#28133 on its own aborts on the first image. An image arrives as an embeddings batch, and llm_graph_input_embd::set_input only writes the token tensor when ubatch->token is set — so the hash layers gather tid2eid[t_inp_tokens] from uninitialised memory and hit GGML_ASSERT(i01 >= 0 && i01 < ne01). #28133 touches no file under src/; #28154 is the half that fixes it.

#28154 adds FFN_EXP_PROBS_B_VL to the deepseek4 architecture and selects the bias per batch. Because llama.cpp delivers an image as its own homogeneous embeddings batch — every position in it is an image token, markers included — that is exact for the batches mtmd produces, not an approximation.

Recipes

All four share the same non-expert treatment: attention, shared experts and dense at Q6_K; token_embd and output at Q8_0; routers (ffn_gate_inp, exp_probs_b, exp_probs_b_vl) at BF16/F32. They differ only in the routed experts:

ffn_down gets the higher rung on the first ten and last ten blocks because that is where quantization damage concentrates.

The importance matrix covers 606,208 tokens (74 × 8192-token chunks) with full expert coverage — no expert went uncalibrated, which matters for a 256-expert MoE.

FP8 source weights were dequantized to BF16, not Q8_0 — BF16 represents the E4M3 significand exactly once the power-of-two E8M0 block scale is applied, so this avoids a second lossy step on 365 attention and shared-expert tensors.

Known limitations

• —In-span attention visibility is brand new and unmeasured here. The reference lets tokens inside an [IMAGE_START, IMAGE_END] span see each other bidirectionally and feeds that to the DFlash sparse-attention indexer. Upstream implements it in #28154 (swa_full_non_causal), which landed after the throughput figures below were taken on a build that fell back to the causal window. Larger images, multiple images and long contexts remain unmeasured either way.
• —No KL-divergence ladder has been measured. The rungs are ordered by bits, which is the expected ordering, but this repo makes no measured quality claim between them. IQ1_M in particular is aggressive and unvalidated behaviourally.
• —Testing so far is single-image. Multi-image and interleaved prompts are untested.
• —This is an experimental upstream model, and its llama.cpp support merged on 2026-09-02. Expect the runtime side to keep moving.

Verification

Re-checked 2026-09-02 against stock llama.cpp `master` (`9cc33944f`) with no patch of any kind. Both locally-held rungs answer image prompts correctly, on the CPU backend and with CUDA offload. Prompts used DeepSeek's own example images.

Throughput was measured earlier on a single RTX PRO 6000 Blackwell (96 GB), fully resident at 1M context, with a Q2_K DSpark speculative draft — on the pre-merge patched build, so treat it as indicative rather than a measurement of the code above:

License

MIT, inherited from the base model.