appmana/deepseek-v4-flash-vision-exp-int4-int8

DeepSeek V4 Flash Vision Exp INT4/INT8 for RTX 30xx

Converted from the public MIT-licensed deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at 6821d6ad3681a4b137b066b76094fa82ebd0a380 with tools/ampere/dsv4_requant_checkpoint.py --vision-format int8-imma from AppMana/forks-vllm-consumer-nvidia-platforms. The checkpoint keeps its own 43 decoder layers, its three native DSpark draft stages and its image routing parameters; nothing is grafted from another model. Stock vLLM cannot load it.

Quantization

Routed experts are symmetric INT4 with group size 32 and MSE-selected scales; dense, shared-expert and attention linears are channel-wise INT8 (AllSpark W8A16), with the inverse-RoPE wo_a kept in BF16; the sparse-MLA cache and indexer use the packed int8_ds_mla layout. The vision patch projection, attention projections, MLPs and aligner are signed INT8 weights with FP32 scales per 32 input channels and dynamic INT8 activations, executed on integer tensor cores with INT32 accumulation; vision attention runs INT8 QK and PV products under an FP32 online softmax. Normalization, RoPE, residuals, learned image vectors and nonlinearities stay in floating point. The vllm block of config.json names every kernel, including vision_int8.VisionInt8LinearMethod and vision_int8.vision_attention_int8.

Serving

PP=11 across RTX 3090s with layer partition 4,4,4,4,4,4,4,4,5,5,1 (VLLM_PP_LAYER_PARTITION): rank 0 owns the vision tower, aligner and image embeddings, the last rank the output head and the DSpark stages. Shards preserve that ownership and hold at most 4 GiB of tensor data each.

VLLM_PP_LAYER_PARTITION=4,4,4,4,4,4,4,4,5,5,1 vllm serve appmana/deepseek-v4-flash-vision-exp-int4-int8 \
  --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 \
  --pipeline-parallel-size 11 --distributed-executor-backend ray \
  --max-model-len 1000000 --enable-chunked-prefill --max-num-batched-tokens 1024 \
  --max-num-seqs 8 --limit-mm-per-prompt '{"image":8}' --async-scheduling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \
  --compilation-config '{"mode":3,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[8,16,32,64]}'

The checkpoint's trained draft block size is unchanged; the serving option selects seven probabilistic speculative tokens. Capture sizes are counted in tokens under model runner v2: with an 8-token speculative step, [8,16,32,64] covers one to eight sequences, and a decode step whose token count is not a captured size is padded up to the next one.

Benchmarks

Per stream is the median request of a batch; aggregate is the whole batch over its wall time. Verbatim: each request hides a unique 1,000-token passage in its prompt and asks for it back; the column counts how many of the C simultaneous requests returned it exactly. Text tables are the output of tools/ampere/dsv4_needle_matrix_report.py --markdown in the fork.

Vision, 11 RTX 3090s

Eleven nodes, each a Ryzen 9 7950X with one RTX 3090 power limited to 250 W, linked by Thunderbolt, serving as above. A fixed set of 100 images (30 five-character OCR codes, 30 coloured-square positions, 20 bar charts, 20 photographs), each asked for an exact answer, four requests at a time:

The 2026-09-14 IMMA latency was the runtime recompiling its vision kernels for every new image size; the kernels no longer specialise on the token count and a startup warmup compiles them once, and the 2026-09-19 run logged no compilation during inference. In the same run: eight concurrent requests with eight distinct images each, all answered correctly; the same image repeated reused 5,120 prompt tokens through the KV connector and a different image at identical geometry reused none; every rank kept at least 0.74 GiB of GPU memory free with the four capture sizes above (1.16 GiB in the 2026-09-14 run with fewer graphs). Production PTX shows signed INT8 tensor-core MMA in both vision roles.

Text, 11 RTX 3090s, 2026-09-19

Same server, 8,000-token prompts with a 1,000-token passage, thinking off, DSpark on, fork 1d8b0eae46 (upstream vLLM 729ebac498 merged 2026-09-18). Cold rows: the prompt is new to every cache.

Warm rows: the same prompts again, served from the prefix cache, one C=1 repeat and four C=8 repeats:

Longer contexts and the speculation-off rows are on the sibling card (appmana/deepseek-v4-int4-int8, measured 2026-09-04 on the pre-merge runtime); the language path is the same code and quantization.

Validation

Before every rollout the fork compares this checkpoint against the BF16 vision reference on the fixture set above, checks pipeline image and text routing and draft decoding, and proves cache isolation for different images with identical text. Full image identities and placeholder positions take part in the external cache key without changing router token ids.