malaiwah/DeepSeek-V2-Lite-Chat-SIQ-K4K3

DeepSeek-V2-Lite-Chat — SIQ K4/K3 (rank-sliced Trellis)

SIQ (SparkInfer Quantization) mixed K3/K4 Trellis quantization of deepseek-ai/DeepSeek-V2-Lite-Chat, built as proxy #2 for developing the vLLM + SparkInfer ("gilded-gnosis") serving stack that runs GLM-5.2 SIQ in production. Proxy #1 is malaiwah/Qwen3.6-35B-A3B-SIQ-K6-K4K3.

This model is not smarter than the original. It exists so that stack changes can be tested on a single consumer GPU (RTX 5090, 32 GB) against the exact production code paths — this proxy specifically covers what the Qwen proxy cannot:

• —the native `deepseek_v2.py` rank-sliced SIQ loader (the original implementation the qwen3_5 patch was mirrored from) — serving needs zero vLLM patches;
• —MLA attention (kvlorarank 512, no q compression) with a quantized MoE, under full CUDA graphs (FULL_AND_PIECEWISE) — a mode the hybrid-GDN Qwen proxy cannot run at all on sm_120.

What is quantized

Measured on RTX 5090 (32 GB, sm_120, gilded-gnosis r25)

• —Decode 209.5 tok/s single-stream (512 tok, temp 0)
• —Prefill ~13,100 tok/s at 26.6k-token prompt
• —Small-prompt battery (1/2/5/8/9-token prefills) clean under full CUDA graphs — the sm120 graph-capture Xid bug that forces the Qwen proxy to `cudagraphmode NONE` does not manifest on this architecture
• —Truncated-KLD vs BF16 (top-512, renormalized, 2,044 positions of the bundled LICENSE): 0.0985 — a data point, not a benchmark; magnitudes are not comparable across models/tokenizers
• —--kv-cache-dtype nvfp4_ds_mla is rejected for this model on sm120: the only nvfp4-MLA backend in r25 is `FLASHINFERMLASPARSESM120 (DSA/sparse-indexer models only, e.g. GLM-5.2). Non-sparse MLA runs TRITON_MLA with **fp8** KV (576 B/token — the pool at default gpumemoryutilization` dwarfs the 163,840-token native context)

How to run

podman run -d --name dsv2-siq -p 8000:8000 --gpus all --shm-size 16g \
  -v /path/to/DeepSeek-V2-Lite-Chat-SIQ-K4K3:/model:ro \
  docker.io/voipmonitor/vllm:gilded-gnosis-v20-vllmf5981f1-si978cdb3-fi801d57a-cu132-20260803-r25 \
  vllm serve /model --served-model-name dsv2-lite --port 8000 \
  --max-num-seqs 16 --kv-cache-dtype fp8

No patches, no environment variables, no quantization flags: the checkpoint's hybrid_tr3_tail metadata auto-selects the exl3 rank-sliced loader, and its quantization_config ignore list keeps every non-expert Linear BF16.

Reproduction

Everything needed is in tools/ (see PORTMAP.md for the full fork map from the Qwen proxy toolchain):

capture_dsv2.py (natural-routing activation capture, salvage/quarantine sealing) → encode_dsv2.py (byte-pinned encode_tr3_v31.py adapter, uniform-K pool encode, verified-read payloads) → k_plan_dsv2.py (router-mass tiering, sealed fingerprint chain) → assemble (source re-hash audit, bijective tensor accounting, MANIFEST.sha256). One-command pipeline: run_campaign_dsv2.sh. The recipe fingerprint chain (capture plan → k-plan → per-layer done records → hybrid_tr3_tail) makes the artifact functionally reproducible from the same source + corpus.

Hard-won operational notes (all hit while building this):

1. 1.`cpu_offload_gb` silently no-ops on the r25 V2 GPU model runner (v1/worker/gpu/model_runner.py never calls set_offloader). Capture of the 29.3 GiB BF16 source on a 32 GB card requires VLLM_USE_V2_MODEL_RUNNER=0 (legacy runner, where UVA offload works).
2. 2.With offloaded weights, the FlashInfer/trtllm MoE autotuner takes ~57 s/profile over PCIe (15+ min per engine start). Capture forces moe_backend=triton, enable_flashinfer_autotune=false.
3. 3.On the build host, SHA-mismatching reads of verified-good NAS payloads were traced to a poisoned cachefilesd/FS-Cache disk object (O_DIRECT reads matched the sealed hashes; fadvise-dropped page-cache reads did not). All payload readers in these tools use multi-attempt cache-dropped verified reads; run integrity-critical campaigns with cachefilesd off.

Credits (prior art)

• —b12x / SparkInfer — the rank-sliced Trellis serving stack in vLLM gilded-gnosis (runtime-dynamic mixed tiers, SparkInfer PR #117 line).
• —brandonmusic — the owner-designated calibration encoder (encode_tr3_v31.py, byte-pinned sha e9a85a47…) and the owner 4-axis calibration corpus, shipped with GLM-5.2-EXL3-TR3-3.0bpw.
• —turboderp — exllamav3 0.0.43 — the Trellis/LDLQ/MCG quantization math.
• —DeepSeek-AI — DeepSeek-V2-Lite-Chat (model license applies; see LICENSE).

Built on aiboss (RTX 5090) 2026-08-04. Capture 1,051,203 tokens ×26 layers in one writeback pass (~104 GiB), encode ~50 s/layer, full campaign well under an hour of GPU time.