cfontes/glm-5.3-flash-dflash2-tp4

GLM-5.3-Flash NVFP4 + DFlash2 on 4× DGX Spark (TP4)

69.3 tok/s single-stream coding peak — matching tonyd2wild's TP4 DFlash2 reference of 68.5 tok/s.

This repo documents a working deployment of GLM-5.3-Flash (320B MoE, 18B active) in NVFP4 quantization with DFlash2 block-diffusion speculative decoding across four NVIDIA DGX Spark (GB10) nodes at tensor-parallel 4.

📊 Benchmark Results

All benchmarks measured with llm-inference-bench v0.4.29.

🏆 Headline: Coding Peak — 69.3 tok/s (temp=0, c1)

• —Prompt: "Write a Python script that implements the Sieve of Eratosthenes"
• —Config: temp=0, 2000 max tokens, single stream, thinking disabled
• —Output: ~7,000 chars of Python code + explanation per run
• —Zero reasoning chars — thinking was off, all tokens went to content

Sustained Decode — Mixed Prompts (default temp)

30s per test, 512 max output tokens, ignore_eos=true.

Sustained Decode — temp=0 (greedy decoding)

🔬 Per-Position Acceptance Rates (c1, ctx=0)

DFlash2 proposes 7 draft tokens per step. Each position has independent acceptance probability — the further from the anchor token, the harder it is to predict.

Fun fact: Position 1 acceptance at temp=0 is 78.6% — nearly 4 out of 5 first-draft tokens are correct. By position 7, only 1 in 15 survives. The drafter is excellent at "obvious next tokens" (code keywords, closing brackets) but loses confidence on creative choices.

📈 Scaling: Concurrency vs Throughput

Fun fact: At c4, the cluster pushes 84.7 tok/s aggregate — but each individual user sees only 21.2 tok/s. DFlash2's acceptance rate stays remarkably stable (2.80–2.93) across concurrency levels, unlike MTP which degrades more. The spec-decode rejection sampler handles batched drafts without quality loss.

🧠 KV Cache & Memory Budget

Fun fact: The KV pool holds 2.86 million tokens — enough for 43 concurrent 64K-context conversations, or 1,100+ concurrent 2K-context chats. FP8 KV cache doubles this vs BF16. The DFlash2 drafter keeps its own KV in BF16 but shares the target's embedtokens/lmhead, saving ~1.5 GiB vs a standalone draft model.

⚡ Prefill Throughput

Fun fact: Prefill throughput is ~1,700 tok/s regardless of context length — the linear attention layers earn their keep at depth. A 128K prompt (roughly a 500-page book) processes in 76 seconds. The GB10's unified memory means no CPU↔GPU transfers for the 182GB weight set.

Comparison vs References

Key insight: DFlash2 throughput is highly prompt-dependent. Code and structured output achieve 48-74 tok/s; mixed/prose prompts sit at 18-40 tok/s. This matches tonyd2wild's and joesinvestments' findings. The win is in single-stream agent traffic where outputs are predictable.

What Did NOT Work (v3 experiment)

We tested a v3 config with --gpu-memory-utilization 0.80, --kv-cache-memory 16GiB (pinned), --enable-chunked-prefill, and --default-chat-template-kwargs '{"enable_thinking": false}'. Results regressed:

Root cause: Acceptance dropped 25% (3.28→2.46). The engine ran at the same steps/s (12.8) but accepted fewer drafts per step. Likely culprits:

1. 1.--kv-cache-memory pin may be too small for TP4, causing KV pressure
2. 2.--enable-chunked-prefill (validated on SGLang, not vLLM) may feed the drafter partial context
3. 3.--default-chat-template-kwargs changes prompt format, affecting draft predictions

Lesson: DFlash2 is sensitive to KV cache configuration and prefill scheduling. Changes that help MTP or SGLang can hurt DFlash2 on vLLM. Test one variable at a time.

Raw benchmark JSONs:

• —`benchmarks/glm53-dflash2-bench.json` — mixed prompts, default temp
• —`benchmarks/glm53-dflash2-bench-temp0.json` — mixed prompts, temp=0
• —`benchmarks/glm53-dflash2-bench-code.json` — coding peak, temp=0
• —`benchmarks/glm53-dflash2-bench-v3.json` — v3 regression experiment

🔧 Hardware Configuration

🏗️ How It Works

DFlash2 Block-Diffusion Speculative Decoding

DFlash2 is a block-diffusion speculative decoding method that uses a lightweight 5-layer Qwen3 drafter to predict 7 tokens per step. The drafter captures auxiliary hidden states from 5 target model layers (indices 5, 14, 24, 33, 42), uses non-causal attention with a sliding window of 2048, and shares the target model's embedding and LM head layers.

Per-step flow:

1. 1.Target model runs one forward pass → 1 token + 5 aux hidden states
2. 2.Drafter runs forward → 7 candidate tokens (using aux states + its own KV)
3. 3.Rejection sampler verifies all 7 in parallel
4. 4.Accepted tokens emitted; rejected tokens discarded
5. 5.Accepted: 1 + accepted_count tokens per step

Why it's fast: The drafter is 5 layers vs the target's 120 layers. Even with 7 draft tokens, the total compute is ~1.04× a single target forward pass. At accept=3, you get 4 tokens per step for ~1.04× compute — a 3.8× speedup on the forward pass.

NVFP4 Quantization

GLM-5.3-Flash uses NVIDIA's NVFP4 (4-bit floating point) quantization:

• —182GB total weight size (vs ~640GB FP8, ~1.3TB BF16)
• —120 safetensors shards, ~1.5GB each
• —MoE experts in NVFP4, shared layers in FP8
• —Marlin MoE backend for fast expert dispatch
• —FP8 e4m3 KV cache (halves KV memory vs BF16)

🚀 Reproduction

Prerequisites

• —4× DGX Spark nodes with RoCE interconnect
• —Docker image radixark/vllm-glm53-flash:sm121-v8-dflash2-tony on all nodes
• —GLM-5.3-Flash NVFP4 weights at /var/tmp/glm-5.3-flash-nvfp4/ (182GB, 120 shards)
• —DFlash2 draft model at /var/tmp/glm-5.3-flash-dflash2/ (2.2GB)

Launch

# Copy launch script to all nodes
scp scripts/launch-glm53-dflash2-tp4.sh admin@<node>:/home/admin/

# Launch workers first (rank 3, 2, 1), then head (rank 0)
ssh admin@node4 'NODE_RANK=3 bash /home/admin/launch-glm53-dflash2-tp4.sh'
ssh admin@node3 'NODE_RANK=2 bash /home/admin/launch-glm53-dflash2-tp4.sh'
ssh admin@node2 'NODE_RANK=1 bash /home/admin/launch-glm53-dflash2-tp4.sh'
# Wait 10s
ssh admin@node1 'NODE_RANK=0 bash /home/admin/launch-glm53-dflash2-tp4.sh'

# Wait ~10 min for model loading + kernel compilation
# Check: curl http://node1:8000/v1/models

Benchmark

# Coding peak (the headline number)
llm_decode_bench.py --host <node1> --port 8000 \
  --model glm-5.3-flash-dflash2 \
  --coding-peak --temperature 0.0 \
  --max-tokens 2000 --display-mode plain

# Sustained decode
llm_decode_bench.py --host <node1> --port 8000 \
  --model glm-5.3-flash-dflash2 \
  --concurrency 1,2,4 --contexts 0,4096 \
  --max-tokens 512 --duration 30 \
  --temperature 0.0 --display-mode plain

📝 Key Learnings from the Community

From joesinvestments/GLM-5.3-Flash-FP8-4x-DGX-Spark

1. 1.DFlash2 acceptance warms slowly — first measurement after boot shows +11%, but warmed acceptance reaches +37%. Never benchmark a cold spec-decode server.
2. 2.DFlash2 is prompt-dependent — structured output: 48 tok/s, math: 46 tok/s, code: 32 tok/s, prose: 18 tok/s. "Acceptance runs cold on prose."
3. 3.Chunked prefill 2048 is a GB10 finding from Light Foundry — helps SGLang prefill but may not transfer to vLLM (our v3 regression confirms this).
4. 4.`--load-format instanttensor` gives 15× load speedup but causes silent rank death in multi-node. tonyd2wild ships it disabled. We confirmed and excluded it.
5. 5.Partial MoE Triton tuning is WORSE than none — joesinvestments' boot 9 regressed -10% because partial configs snap non-decode shapes to wrong configs.
6. 6.`clear_thinking: true` should be the default for chat workloads — Z-AI's own model card says so. Thinking replays prior-turn reasoning into context, burning tokens.
7. 7.Never set small `max_tokens` with thinking on — the budget burns inside the think block with empty content output.

From tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark

1. 1.KV sizing doctrine: grow pool until ~8-10GB residual per node, gate behind concurrent 20K prefills.
2. 2.GB10 "phantom backing": reservations can succeed then die on first touch in warmup. Always verify with actual requests.
3. 3.Block size 2304 is required for DeepGEMM arch-12 64-entry pool pages.
4. 4.FlashInfer ≥0.6.18 mandatory on sm121 — FA2 MLA produces NaN below this version.
5. 5.Cache flusher sidecar during weight load prevents GB10 driver vs page cache conflict.
6. 6.Reboot nodes after many boot cycles — driver alloc-pool degrades silently.

🙏 Credits

tonyd2wild — DFlash2 Overlay Patches

• —Repo: tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark
• —Built the vLLM DFlash2 overlay (4 patch scripts) on top of radixark/vllm-glm53-flash:sm121-v8. Patches register DFlash2DraftModel in the model registry, add Eagle3 aux hidden-state capture to GLM-5.3's model code, fix KV page layout for the drafter, and wire the drafter into the spec decode engine.

incoai — DFlash2 Draft Model

• —Model: incoai/GLM-5.3-Flash-DFlash2
• —Trained the 2.2GB DFlash2 draft model (5-layer Qwen3, hidden 4096, selectorrank 256, selectortop_k 16). Non-causal attention with sliding window 2048. CC BY-NC-ND research eval license.

z-lab — DFlash2 Reference Implementation

• —Repo: z-lab/dflash
• —Original DFlash2 block-diffusion implementation and research.

radixark — Base vLLM Image

• —Image: radixark/vllm-glm53-flash:sm121-v8
• —The sm121-v8 base vLLM image with GLM-5.3-Flash support, FlashInfer, and DeepGEMM for GB10.

zai-org — GLM-5.3-Flash Model

• —Model: zai-org/GLM-5.3-Flash-NVFP4
• —The original GLM-5.3-Flash model in NVFP4 quantization (182GB).

joesinvestments — SGLang FP8 Reference Deployment

• —Repo: joesinvestments/GLM-5.3-Flash-FP8-4x-DGX-Spark
• —Independent 4× DGX Spark TP4 deployment of GLM-5.3-Flash using SGLang (FP8 weights, DFlash2 drafter). Their DFlash2 numbers on SGLang (48.1 tok/s structured, 32.4 tok/s code) provided a valuable cross-engine comparison. Their LEDGER.md is a masterclass in deployment documentation — every flag's provenance, every failed boot, every measurement ruler. Key learnings: DFlash2 acceptance warms slowly, chunked-prefill 2048 helps SGLang, instanttensor is unreliable in multi-node, partial MoE tuning is worse than none.

local-inference-lab — Benchmark Tool

• —Repo: local-inference-lab/llm-inference-bench
• —The benchmark tool used for all measurements (v0.4.29). Supports sustained decode, coding peak, prefill, and burst modes with per-position acceptance tracking.

License

MIT. Model weights are not redistributed here — download from zai-org/GLM-5.3-Flash-NVFP4 and incoai/GLM-5.3-Flash-DFlash2.